API Centric Development

Most dynamic web applications we write today have the requirement that they provide APIs - that is, methods which return pure data for the consumption by a wide variety of clients. The trend here is towards organizing web applications from the start to act like APIs. In a nutshell, it means we are moving away from mode of data models injected into templates:

def get_balance(request):
    balance = BankBalance(
        amount = Amount(10000, currency="euro"),
        as_of_date = datetime.datetime(2012, 6, 14, 12, 0, 0)
    )
    return render_template("balance.html", balance=balance)

and instead towards returning JSON-compatible data structures, with the "view" being decided somewhere else:

@view_config('balance', renderer='json')
def balance(request):
    return {
        'amount':10000,
        'currency':'euro',
        'as_of_date':'2012-06-14 14:12:00'
    }

This is a fine trend, allowing us to make a clearer line between server side concepts and rendering concepts, and giving us an API-centric view of our data from day one.

Templates

There's a misunderstanding circulating about API-centric development which says that we have to use client side templates:

API-centric development. If you take a client-side rendering approach, odds are the server side of your web application is going to look more like an API than it would if you were doing entirely server-side rendering. And if/when you have plans to release an API, you'd probably already be 90% of the way there. (http://openmymind.net/2012/5/30/Client-Side-vs-Server-Side-Rendering/#comment-544974348)

and:

Lastly, another issue i can see, assuming you develop using MVC. You need to have the view, model and controller very tightly coupled to make his way work. The controller needs to know how the view works to create html that can slot right in. It's easier if the controller only has to pass data, not layout information. (http://www.reddit.com/r/programming/comments/ufyf3/clientside_vs_serverside_rendering/c4v5vjb)

This second quote is specific to the approach of delivering rendered HTML in an ajax response, to be directly rendered into a DOM element, which we will also demonstrate here. How the "model" is more tightly coupled to anything when you have a server side vs client side template, I have absolutely no idea.

Why might we want to stick with server side templates? As a Python developer, in my view the main reason is that they are still a lot easier to develop with, assuming we aren't developing our server in Javascript as well. Consider if our bank account balance needed locale-specific number, currency and date formatting, and also needed to convert the timestamp from UTC into a preferred timezone. A server side approach allows us to easily inject more functionality into our template as part of its standard environment for all render calls, such as something like this:

def render_template(request, template, **template_ns):
    number_formatter = number_formatter(request.user.preferred_locale)
    template_ns['currency_format'] = currency_formatter(number_formatter)
    template_ns['date_format'] = date_formatter(
                                request.user.preferred_timezone,
                                request.user.preferred_date_format)
    return lookup.get_template(template).render(**template_ns)

The template can, if it needs to, call upon these methods like this:

<ul>
    <li>Balance: ${data['amount'] | currency_format(data['currency'])}</li>
    <li>As of: ${data['as_of_date'] | date_format}</li>
</ul>

With a client side template, we have to implement all of currency_formatter, number_formatter, date_formatter in Javascript. These methods may need to respond to business-specific inputs, such as specific user preferences or other rules, that also need to be pushed up to the client. All the additional Javascript logic we're pushing out to the client brings forth the need for it to be unit tested, which so far means we need to add significant complexity to the testing environment by running a Javascript engine like node.js or something else in order to test all this functionality. Remember, we're not already using node.js to do the whole thing. If we were, then yes everything is different, but I wasn't planning on abandoning Python for web development just yet.

Some folks might argue that elements like date conversion and number formatting should still remain on the server, but just be applied by the API to the data being returned directly. To me, this is specifically the worst thing you can do - it basically means that the client side approach is forcing you to move presentation-level concepts directly into your API's data format. Almost immediately, you'll find yourself having to inject HTML entities for currency symbols and other browser-specific markup into this data, at the very least complicating your API with presentational concerns and in the worst case pretty much ruining the purity of your API data. While the examples here may be a little contrived, you can be sure that more intricate cases come up in practice that present an ongoing stream of hard decisions between complicating/polluting server-generated API data with presentation concepts versus building a much heavier client than initially seemed necessary.

Also, what about performance? Don't client side/server side templates perform/scale/respond worse/better? My position here is "if we're just starting out, then who knows, who cares". If I've built an application and I've observed that its responsiveness or scalability would benefit from some areas switching to client side rendering, then that's an optimization that can be made later. We've seen big sites like LinkedIn switch from server to client side rendering, and Twitter switch from client to server side rendering, both in the name of "performance"! Who knows! Overall, I don't think the difference between pushing out json strings versus HTML fragments is something that warrants concern up front, until the application is more fully formed and specific issues can addressed as needed.

The Alternative

Implementing a larger client-side application than we might have originally preferred is all doable of course, but given the additional steps of building a bootstrap system for a pure client-side approach, reimplementing lots of Python functionality, in some cases significant tasks such as timezone conversion, into Javascript, and figuring out how to unit test it all, is a lot of trouble for something that is pretty much effortless in a server side system - must we go all client-side in order to be API centric?

Of course not!

The example I've produced illustrates a single, fake API method that produces a grid of random integers:

@view_config(route_name='api_ajax', renderer='json')
def api(request):
    return [
        ["%0.2d" % random.randint(1, 99)  for col in xrange(10)]
        for row in xrange(10)
    ]

and delivers it in four ways - two use server side rendering, two use client side rendering. Only one client template and one server side template is used, and there is only one API call, which internally knows nothing whatsoever about how it is displayed. Basically, the design of our server component is un-impacted by what style of rendering we use, save for different routing declarations which we'll see later.

For client side rendering of this data, we'll use a handlebars.js template:

<p>
    API data -
    {{#if is_page}}
        Displayed via server-initiated, client-rendered page
    {{/if}}
    {{#unless is_page}}
        Displayed via client-initiated, client-rendered page
    {{/unless}}
</p>

<table>
    {{#each data}}
        <tr>
            {{#each this}}
                <td>{{this}}</td>
            {{/each}}
        </tr>
    {{/each}}
</table>

and for server side rendering, we'll use a Mako template:

<%inherit file="layout.mako"/>

${display_api(True)}

<%def name="display_api(inline=False)">
    <p>
        API data -
        % if inline:
            Displayed inline within a server-rendered page
        % else:
            Displayed via server-rendered ajax call
        % endif
    </p>
    <table>
        % for row in data:
            <tr>
                % for col in row:
                    <td>${col}</td>
                % endfor
            </tr>
        % endfor
    </table>
</%def>

The Mako template is using a <%def> to provide indirection between the rendering of the full page, and the rendering of the API data. This is not a requirement, but is here because we'll be illustrating also how to dual purpose a single Mako template such that part of it can be used for a traditional full page render as well as for an ajax-delivered HTML fragment, with no duplication. It's essentially a Pyramid port of the same technique I illustrated with Pylons four years ago in my post Ajax the Mako Way, which appears to be somewhat forgotten. Among other things, Pyramid's Mako renderer does not appear integrate the capability to call upon page defs directly, even though I had successfully lobbied to get the critical render_def() into its predecessor Pylons. Here, I've implemented my own Mako renderer for Pyramid.

Key here is that we are separating the concept of how the API interface is constructed, versus what the server actually produces. Above, note we're using the Pyramid "json" renderer for our API data. Note the term "renderer". Interpreting our API method as a JSON API, as a call to a specific client-side template plus JSON API, or as a server side render or ajax call is just a matter of declaration. The way we organize our application in an API-centric fashion has nothing to do with where the rendering takes place. To illustrate four different ways of interpreting the same API method, we just need to add four different @view_config directives:

@view_config(route_name='server_navigate', renderer='home.mako')
@view_config(route_name='server_ajax', renderer='home|display_api.mako')
@view_config(route_name='client_navigate', renderer='display_api.handlebars')
@view_config(route_name='api_ajax', renderer='json')
def api(request):
    return [
        ["%0.2d" % random.randint(1, 99)  for col in xrange(10)]
        for row in xrange(10)
    ]

The rendering methods here are as follows:

• Method One, Server Via Server - The api() view method returns the data, which is received by the home.mako template, which renders the full page, and passes the data to the display_api() def within the same phase for all-at-once server side rendering.
• Method Two, Server Via Client - A hyperlink on the page initiates an ajax call to the server's server_ajax route, which invokes the api() view method, and returns the data directly to the display_api def present in the home.mako template. For this case, I had to create my own Pyramid Mako renderer that receives a custom syntax, where the page name and def name are separated by a pipe character.
• Method Three, Client Via Server - Intrinsic to any client side rendered approach is that the server needs to first deliver some kind of HTML layout, as nothing more than a launching point for all the requisite javascript needed to start rendering the page for real. This method illustrates that, by delivering the handlebars_base.mako template which serves as the bootstrap point for any server-initiated page call that renders with a client side template. In this mode, it also embeds the data returned by api() within the <script> tags at the top of the page, and then invokes the Javascript application to render the display_api.handlebars template, providing it with the embedded data. Another approach here might be to deliver the template in one call, and to have the client invoke the api() method as a separate ajax call, though this takes two requests instead of one and also implies adding another server-side view method.
• Method Four, Client Via Client - A hyperlink on the page illustrates how a client-rendered application can navigate to a certain view, calling upon the server only for raw data (and possibly the client side template itself, until its cached in a client-side collection). The link includes additional attributes which allow the javascript application to call upon the display_api.handlebars template directly, and renders it along with the data returned by calling the api() view method with the json renderer.

Credit goes to Chris Rossi for coming up with the original client-side rendering techniques that I've adapted here.

A screen shot of what we're looking at is as follows:

The demonstration here is hopefully useful not just to illustrate the server techniques I'm talking about, but also as a way to play around with client side rendering as well, including mixing and matching both server and client side rendering together. The hybrid approach is where I predict most applications will be headed.

You can pull out the demo using git at https://bitbucket.org/zzzeek/client_template_demo. Enjoy !