A Late Add

I was approached some months ago by a vendor known as Akiban, with a request that was totally new to me. On the initial emails, the idea seemed to be some kind of database that can produce JSON documents from tables, which by itself didn't seem at all very novel; there are already products like HTSQL and SlashDB (friend of SQLAlchemy!) which provide RESTful services around relational databases.

But digging in, it became apparent that this company was taking a much more elaborate path to that goal - this company is producing their own relational database from scratch. The server itself, Akiban server, is written in Java and is designed to act in large part like Postgresql, with some MySQL compatibility added in as well. Within the span of this obviously monumental task, they have built what seems to be exactly one twist, but it is a really interesting twist, which they call a "table grouping".

I was very flattered that Akiban not only wanted me to provide SQLAlchemy support for their database, but to help them come up with their Python DBAPI story overall. And, they wanted not just a SQLAlchemy dialect and Python DBAPI story, they also wanted my thoughts on how modern ORMs can integrate with the unique features of this system.

The reason all three of these areas are up for grabs with Akiban, is that while it acts pretty much like a regular relational database most of the time, it does something strange when the "table grouping" idea is in use. At the DDL level, a "table grouping" is established just like a foreign key:

CREATE TABLE customer (
    id INTEGER PRIMARY KEY,
    name VARCHAR(30) NOT NULL
)

CREATE TABLE cust_order (
    id INTEGER PRIMARY KEY,
    customer_id INTEGER,
    ordernum VARCHAR(30) NOT NULL,
    timestamp DATETIME,
    GROUPING FOREIGN KEY (customer_id) REFERENCES customer
)

This single keyword on GROUPING causes Akiban to organize the data in a way I don't think any other database does, which is that it stores the rows for cust_order interleaved within those of customer. Meaning, it stores the data hierarchically, on disk. This special hierarchy is then made available through SQL. This is the second part that is really surprising, which is in order to support this, there were no changes at all needed to the SQL syntax. The only change made to SQL was a semantic one.

Normally, SQL allows us to query for a correlated SELECT in the columns or WHERE clause of a SELECT statement:

SELECT name, (
                SELECT ordernum
                FROM cust_order
                WHERE customer_id=customer.id
                AND ordernum like 'order1%'
        ) AS ordernum
FROM customer

The correlated SELECT must return exactly one column, and one row, or your database will complain. Here, Akiban merely took away the "complain" part, and when specifying a correlated subquery with multiple columns and potentially multiple rows, you instead get back a result that is hierarchical, assuming the tables you're embedding are related to the parent via a "grouping" - the "nested" rows are fetched in groups along with each corresponding parent row, and delivered inline. We would get such a result from Akiban given a statement like that below:

SELECT name, (
                SELECT ordernum, timestamp
                FROM cust_order
                WHERE customer_id=customer.id
        ) AS orders
FROM customer

But what does a "hierarchical" result look like? Well, for Akiban, since they wanted this to all work over the Postgresql protocol, for now they actually send you back a JSON string per parent row, as one column. Such as this:

{"name":"Some Customer", "orders":[{"ordernum":"some order", "timestamp":"20121005121700"}, {"ordernum":"some other order", "timestamp":"20121012184507"}]}

With that, Akiban asked me what I can do with this. Particularly, how can an ORM like SQLAlchemy take advantage of it?

well it's just nested

The first thing I wanted to do with this data was to make that JSON look like a regular SQL result again. Having worked with "eager loading" for so many years, I already tend to see SQL products as "hierarchies" - a regular JOIN of customer and order would have rows like:

customer.name        ordernum            timestamp
-------------        ----------------    --------------
Some Customer        some order          20121005121700
Some Customer        some other order    20121012184507
Some Other Customer  ...                 ...

The result set from a correlated subquery on a "grouped" foreign key, in my mind, looks pretty similar:

customer.name        orders
-------------        ----------------------------------
Some Customer        ordernum            timestamp
                     ----------------    --------------
                     some order          20121005121700
                     some other order    20121012184507
Some Other Customer  ...                 ...

When we normally have a subquery in the "columns" clause of a SELECT, the result of that SELECT comes back as a single result set column. My idea was that just as they don't have to change the syntax of SQL to express a "grouping" here, we didn't have to change the idea on the result side either - we just have a nested cursor, where a single result set column orders returns a new cursor as its value.

I initially studied Postgresql's protocol quite a bit, mostly by reading the source to the pure-Python pg8000 DBAPI for Postgresql, and proposed to Akiban that they just add some extra messages to the Postgresql protocol to support this concept without the need for JSON being involved. I could extend or fork pg8000 into a new Akiban-specific DBAPI. But for the time being, what I ended up doing was to stick with the DBAPI that already works with Akiban, psycopg2, and extend it on the outside of the JSON to coerce the result back into a cursor shape. With psycopg2's very extensible design, I was able to make use of its Connection extension system to intercept JSON-formatted results back into cursor-like results, as well as to reuse its existing typing system in order to apply the same string interception it uses on raw Postgresql messages to the JSON-encoded fields as well. The result of this stage of the game is Akiban for Python, an extension to psycopg2 that introduces a new result-row datatype, the nested cursor, which is transparently returned when you emit an Akiban "nested" SQL statement - the presense of JSON is entirely concealed.

turtles all the way down (or up, in this case)

With a DBAPI providing nested cursors, the next task was an Akiban dialect for SQLAlchemy which can wrap these nested cursors into SQLAlchemy's cursor-wrapping ResultProxy object. Over the years, I've had to make ResultProxy objects deal with so many curveballs, from cx_Oracle's streaming LOB objects and OUT parameters, to the buffering required with psycopg2's "server side cursors", to pysqlite's quirk with column name descriptions being prepended with the table name when the SQL statement is a UNION of two SELECT statements, that this task was relatively easy to get going. Slightly more tricky was to get the statement compiler to keep track of "nested" select() constructs and to relate them to the nested cursors ultimately produced. The end result is that we can build an Akiban nested query and get its results:

from sqlalchemy_akiban import nested

stmt = nested([order.c.ordernum, order.c.timestamp]).\
            where(order.c.customer_id == customer.c.id)
stmt = select([customer.c.name, stmt.label('orders')])

for customer_row in connection.execute(stmt):
    print "customer name:", customer_row['name']
    for order_row in customer_row['orders']:
        print "ordernum:", order_row['ordernum']
        print "timestamp:", order_row['timestamp']

And with the above, all of the usual SQLAlchemy Core capabilities like result-set typing and column targeting work out the same. At the ORM level, I added similar nesting features into Query (meaning, you can get back nested tuples) and also produced a new "loader strategy" specific for "Akiban" style loading on relationships, so that a "nested" cursor can provide for the simplest eager loading implementation ever. The familiar syntax we use with joinedload() and subqueryload() gets the addition of nestedload(), which we drop in in the same way:

from sqlalchemy_akiban.orm import nestedload_all

result = session.query(Customer).options(
                    orm.nestedload_all(Customer.orders, Order.items)).\
                        filter(Customer.id == 1)

Above, the Customer.orders and Order.items collections are populated directly from nested results, using a single statement that optimizes as fast as a straight select of just the customer table and nothing else:

SELECT customer.id AS customer_id, customer.name AS customer_name,
        (SELECT
            (SELECT item.id, item.order_id, item.price, item.quantity
            FROM item
            WHERE "order".id = item.order_id
        ) AS anon_2,
        "order".id, "order".customer_id, "order".order_info
        FROM "order"
        WHERE customer.id = "order".customer_id
) AS anon_1
FROM customer
WHERE customer.id = %(id_1)s

All of the above was possible with pretty much no changes to SQLAlchemy itself, save for one new dialect hook which allows the cursor context to be available to type objects - since we introduced an extension type called NestedResult which needs more information than most types. The ORM, compiler, etc. needed no changes. The result of this stage is available at SQLAlchemy Akiban.

Onward

In an upcoming post, I'll be describing the new dialect test suite I developed, as a result of my work in supporting the Akiban database.

Links

• Akiban for Python - Akiban extension for the psycopg2 DBAPI
• SQLAlchemy Akiban - Akiban dialect and ORM extension for SQLAlchemy

API Centric Development

Most dynamic web applications we write today have the requirement that they provide APIs - that is, methods which return pure data for the consumption by a wide variety of clients. The trend here is towards organizing web applications from the start to act like APIs. In a nutshell, it means we are moving away from mode of data models injected into templates:

def get_balance(request):
    balance = BankBalance(
        amount = Amount(10000, currency="euro"),
        as_of_date = datetime.datetime(2012, 6, 14, 12, 0, 0)
    )
    return render_template("balance.html", balance=balance)

and instead towards returning JSON-compatible data structures, with the "view" being decided somewhere else:

@view_config('balance', renderer='json')
def balance(request):
    return {
        'amount':10000,
        'currency':'euro',
        'as_of_date':'2012-06-14 14:12:00'
    }

This is a fine trend, allowing us to make a clearer line between server side concepts and rendering concepts, and giving us an API-centric view of our data from day one.

Templates

There's a misunderstanding circulating about API-centric development which says that we have to use client side templates:

API-centric development. If you take a client-side rendering approach, odds are the server side of your web application is going to look more like an API than it would if you were doing entirely server-side rendering. And if/when you have plans to release an API, you'd probably already be 90% of the way there. (http://openmymind.net/2012/5/30/Client-Side-vs-Server-Side-Rendering/#comment-544974348)

and:

Lastly, another issue i can see, assuming you develop using MVC. You need to have the view, model and controller very tightly coupled to make his way work. The controller needs to know how the view works to create html that can slot right in. It's easier if the controller only has to pass data, not layout information. (http://www.reddit.com/r/programming/comments/ufyf3/clientside_vs_serverside_rendering/c4v5vjb)

This second quote is specific to the approach of delivering rendered HTML in an ajax response, to be directly rendered into a DOM element, which we will also demonstrate here. How the "model" is more tightly coupled to anything when you have a server side vs client side template, I have absolutely no idea.

Why might we want to stick with server side templates? As a Python developer, in my view the main reason is that they are still a lot easier to develop with, assuming we aren't developing our server in Javascript as well. Consider if our bank account balance needed locale-specific number, currency and date formatting, and also needed to convert the timestamp from UTC into a preferred timezone. A server side approach allows us to easily inject more functionality into our template as part of its standard environment for all render calls, such as something like this:

def render_template(request, template, **template_ns):
    number_formatter = number_formatter(request.user.preferred_locale)
    template_ns['currency_format'] = currency_formatter(number_formatter)
    template_ns['date_format'] = date_formatter(
                                request.user.preferred_timezone,
                                request.user.preferred_date_format)
    return lookup.get_template(template).render(**template_ns)

The template can, if it needs to, call upon these methods like this:

<ul>
    <li>Balance: ${data['amount'] | currency_format(data['currency'])}</li>
    <li>As of: ${data['as_of_date'] | date_format}</li>
</ul>

With a client side template, we have to implement all of currency_formatter, number_formatter, date_formatter in Javascript. These methods may need to respond to business-specific inputs, such as specific user preferences or other rules, that also need to be pushed up to the client. All the additional Javascript logic we're pushing out to the client brings forth the need for it to be unit tested, which so far means we need to add significant complexity to the testing environment by running a Javascript engine like node.js or something else in order to test all this functionality. Remember, we're not already using node.js to do the whole thing. If we were, then yes everything is different, but I wasn't planning on abandoning Python for web development just yet.

Some folks might argue that elements like date conversion and number formatting should still remain on the server, but just be applied by the API to the data being returned directly. To me, this is specifically the worst thing you can do - it basically means that the client side approach is forcing you to move presentation-level concepts directly into your API's data format. Almost immediately, you'll find yourself having to inject HTML entities for currency symbols and other browser-specific markup into this data, at the very least complicating your API with presentational concerns and in the worst case pretty much ruining the purity of your API data. While the examples here may be a little contrived, you can be sure that more intricate cases come up in practice that present an ongoing stream of hard decisions between complicating/polluting server-generated API data with presentation concepts versus building a much heavier client than initially seemed necessary.

Also, what about performance? Don't client side/server side templates perform/scale/respond worse/better? My position here is "if we're just starting out, then who knows, who cares". If I've built an application and I've observed that its responsiveness or scalability would benefit from some areas switching to client side rendering, then that's an optimization that can be made later. We've seen big sites like LinkedIn switch from server to client side rendering, and Twitter switch from client to server side rendering, both in the name of "performance"! Who knows! Overall, I don't think the difference between pushing out json strings versus HTML fragments is something that warrants concern up front, until the application is more fully formed and specific issues can addressed as needed.

The Alternative

Implementing a larger client-side application than we might have originally preferred is all doable of course, but given the additional steps of building a bootstrap system for a pure client-side approach, reimplementing lots of Python functionality, in some cases significant tasks such as timezone conversion, into Javascript, and figuring out how to unit test it all, is a lot of trouble for something that is pretty much effortless in a server side system - must we go all client-side in order to be API centric?

Of course not!

The example I've produced illustrates a single, fake API method that produces a grid of random integers:

@view_config(route_name='api_ajax', renderer='json')
def api(request):
    return [
        ["%0.2d" % random.randint(1, 99)  for col in xrange(10)]
        for row in xrange(10)
    ]

and delivers it in four ways - two use server side rendering, two use client side rendering. Only one client template and one server side template is used, and there is only one API call, which internally knows nothing whatsoever about how it is displayed. Basically, the design of our server component is un-impacted by what style of rendering we use, save for different routing declarations which we'll see later.

For client side rendering of this data, we'll use a handlebars.js template:

<p>
    API data -
    {{#if is_page}}
        Displayed via server-initiated, client-rendered page
    {{/if}}
    {{#unless is_page}}
        Displayed via client-initiated, client-rendered page
    {{/unless}}
</p>

<table>
    {{#each data}}
        <tr>
            {{#each this}}
                <td>{{this}}</td>
            {{/each}}
        </tr>
    {{/each}}
</table>

and for server side rendering, we'll use a Mako template:

<%inherit file="layout.mako"/>

${display_api(True)}

<%def name="display_api(inline=False)">
    <p>
        API data -
        % if inline:
            Displayed inline within a server-rendered page
        % else:
            Displayed via server-rendered ajax call
        % endif
    </p>
    <table>
        % for row in data:
            <tr>
                % for col in row:
                    <td>${col}</td>
                % endfor
            </tr>
        % endfor
    </table>
</%def>

The Mako template is using a <%def> to provide indirection between the rendering of the full page, and the rendering of the API data. This is not a requirement, but is here because we'll be illustrating also how to dual purpose a single Mako template such that part of it can be used for a traditional full page render as well as for an ajax-delivered HTML fragment, with no duplication. It's essentially a Pyramid port of the same technique I illustrated with Pylons four years ago in my post Ajax the Mako Way, which appears to be somewhat forgotten. Among other things, Pyramid's Mako renderer does not appear integrate the capability to call upon page defs directly, even though I had successfully lobbied to get the critical render_def() into its predecessor Pylons. Here, I've implemented my own Mako renderer for Pyramid.

Key here is that we are separating the concept of how the API interface is constructed, versus what the server actually produces. Above, note we're using the Pyramid "json" renderer for our API data. Note the term "renderer". Interpreting our API method as a JSON API, as a call to a specific client-side template plus JSON API, or as a server side render or ajax call is just a matter of declaration. The way we organize our application in an API-centric fashion has nothing to do with where the rendering takes place. To illustrate four different ways of interpreting the same API method, we just need to add four different @view_config directives:

@view_config(route_name='server_navigate', renderer='home.mako')
@view_config(route_name='server_ajax', renderer='home|display_api.mako')
@view_config(route_name='client_navigate', renderer='display_api.handlebars')
@view_config(route_name='api_ajax', renderer='json')
def api(request):
    return [
        ["%0.2d" % random.randint(1, 99)  for col in xrange(10)]
        for row in xrange(10)
    ]

The rendering methods here are as follows:

• Method One, Server Via Server - The api() view method returns the data, which is received by the home.mako template, which renders the full page, and passes the data to the display_api() def within the same phase for all-at-once server side rendering.
• Method Two, Server Via Client - A hyperlink on the page initiates an ajax call to the server's server_ajax route, which invokes the api() view method, and returns the data directly to the display_api def present in the home.mako template. For this case, I had to create my own Pyramid Mako renderer that receives a custom syntax, where the page name and def name are separated by a pipe character.
• Method Three, Client Via Server - Intrinsic to any client side rendered approach is that the server needs to first deliver some kind of HTML layout, as nothing more than a launching point for all the requisite javascript needed to start rendering the page for real. This method illustrates that, by delivering the handlebars_base.mako template which serves as the bootstrap point for any server-initiated page call that renders with a client side template. In this mode, it also embeds the data returned by api() within the <script> tags at the top of the page, and then invokes the Javascript application to render the display_api.handlebars template, providing it with the embedded data. Another approach here might be to deliver the template in one call, and to have the client invoke the api() method as a separate ajax call, though this takes two requests instead of one and also implies adding another server-side view method.
• Method Four, Client Via Client - A hyperlink on the page illustrates how a client-rendered application can navigate to a certain view, calling upon the server only for raw data (and possibly the client side template itself, until its cached in a client-side collection). The link includes additional attributes which allow the javascript application to call upon the display_api.handlebars template directly, and renders it along with the data returned by calling the api() view method with the json renderer.

Credit goes to Chris Rossi for coming up with the original client-side rendering techniques that I've adapted here.

A screen shot of what we're looking at is as follows:

The demonstration here is hopefully useful not just to illustrate the server techniques I'm talking about, but also as a way to play around with client side rendering as well, including mixing and matching both server and client side rendering together. The hybrid approach is where I predict most applications will be headed.

You can pull out the demo using git at https://bitbucket.org/zzzeek/client_template_demo. Enjoy !