Archive for the ‘Web Development’ Category.

Ajax the Mako Way

My previous post (updated!) demonstrated how Mako's "defs with embedded content" feature was used to build a library of form tags, keeping all HTML and layout within templates, as well as a succinct method of linking them to form validation and data state. The "def with embedded content", a feature derived from HTML::Mason, is one feature that makes Mako highly unique in the Python world. The form demo also illustrated another unique feature, which is the ability to "export" the functionality of a def (essentially a subcomponent of a page) to other templates, without any dependency on inheritance or other structural relationship. Defs with embeddable content and exportable defs are two features I would never want to do without, which is why I ported HTML::Mason to Myghty, and later created Mako for Python.

A lesser known capability of the def is that they can be called not just by other templates but by any arbitrary caller, such as a controller. As it turns out, this capability is ideal in conjunction with asynchronous requests as well, a use case that didn't even exist when HTML::Mason was first created. Here I'll demonstrate my favorite way to do Ajax with Pylons and the unbelievably excellent jQuery. We'll introduce a new render() function that IMO should be part of Pylons, the same way as render_mako().

An asynchronous HTTP request is often used to render part of the page while leaving the rest unchanged, typically by taking the output of the HTTP request and rendering it into a DOM element. Jquery code such as the following can achieve this:

$("#some_element").load("/datafeed");

The above statement will render the output of the URI /datafeed into the DOM element with the id some_element. In Pylons, a controller and associated template would provide the output for the /datafeed URI, which would be HTML content forming a portion of the larger webpage.

In our example, we'll build a "pager" display which displays multiple pages of a document, one at a time, using the load() method to load new pages. One way we might do this looks like this:

<div id="display"></div>
<a href="javascript:showpage(1)">1</a> 
<a href="javascript:showpage(2)">2</a> 
<a href="javascript:showpage(3)">3</a> 
 
<script>
    function showpage(num) {
        $("#display").load("/page/read/" + num);
    }
    showpage(1);
</script>

Above, we define display, which is a div where the pages render. Some javascript code defines the showpage() function, which given a page number calls the jQuery load() function to load the content from the page-appropriate URI into the div. Three links to three different pages each link to showpage(), given different page numbers.

In this version, the /page/read controller would probably define a separate template of some kind in order to format a the data, so the layout of what goes inside of display is elsewhere. Additionally, the initial display of the full layout requires two HTTP requests, one to deliver the enclosing layout and another to load the formatted content within display.

When using Mako, we often want to group together related components of display within a single file. The <%def> tag makes this possible - a compound layout of small, highly interrelated components need not be spread across many files with small amounts of HTML in each; they can all be defined together, which can cut down on clutter and speed up development.

Such as, if we built the above display entirely without any asynchronous functionality, we might say:

<div id="display">
    ${showpage(c.page)}
</div>
<a href="/page/read/1">1</a>
<a href="/page/read/2">2</a>
<a href="/page/read/3">3</a>
 
<%def name="showpage(page)">
<div class="page">
    <div class="pagenum">Page: ${page.number}</div>
    <h3>${page.title}</h3>
 
    <pre>${page.content}</pre>
</div>
</%def>

The above approach again defines showpage(), but it's now a server-side Mako def, which receives a single Page object as the thing to be rendered. The output is first displayed using the Page object placed at c.page by the controller, and subsequent controller requests re-render the full layout with the appropriate Page represented.

The missing link here is to use both of the above approaches at the same time - render the first Page object into the div without using an asynchronous request, allow subsequent Page requests to be rendered via Ajax, and finally to have the whole layout defined in a single file. For that, we need a new Pylons render function, which looks like this:

def render_def(template_name, name, **kwargs):
    globs = pylons_globals()
 
    if kwargs:
        globs = globs.copy()
        globs.update(kwargs)
 
    template = globs['app_globals'].mako_lookup.get_template(template_name).get_def(name)
    return template.render(**globs)

The above render_def() function is adapted from the standard Pylons boilerplate for building render functions. It's virtually the same as render_mako() except we're calling the extra get_def() method from the Mako Template object, and we're also passing some **kwargs straight to the def in addition to the standard Pylons template globals. A refined approach might involve building a render_mako() function that has the functionality to render both full Template objects as well as individual <%def> objects based on arguments; but we'll keep them separate for now.

With render_def(), the Ajax version of our page now looks like:

<div id="display">
     ${showpage(c.page)}
</div>
<a href="javascript:showpage(1)">1</a> 
<a href="javascript:showpage(2)">2</a> 
<a href="javascript:showpage(3)">3</a> 
 
<script>
    function showpage(num) {
        $("#display").load("/page/read/" + num);
    }
</script>
 
<%def name="showpage(page)">
<div class="page">
    <div class="pagenum">Page: ${page.number}</div>
    <h3>${page.title}</h3>
 
    <pre>${page.content}</pre>
</div>
</%def>

Note above that there are two showpage functions; one is a Mako def, callable during the server's rendering of the template, the other a Javascript function which uses jQuery to issue a new request to load new content. The /page/read controller calls the showpage() def directly as its returned template. The net effect is that the server-side version of showpage() dual purposes itself in two different contexts; as a server-side component which participates in the composition of an enclosing template render, and as a "standalone" template which delivers new versions of its layout into the same overall display within its own HTTP request.

The controller, which locates Page objects using a simple SQLAlchemy model, in its entirety:

class PageController(BaseController):
    def index(self):
        c.number_of_pages = Session.query(Page).count()
        c.page = Session.query(Page).filter(Page.number==1).one()
        return render("/page.mako")
 
    def read(self, id):
        pagenum = int(id)
 
        page = Session.query(Page).filter(Page.number==pagenum).one()
        return render_def("/page.mako", "showpage", page=page)

I've packaged the whole thing as a demo application (using Pylons 0.9.7 and SQLAlchemy 0.5), which pages through a document we all should be well familiar with.

Better Form Generation with Mako and Pylons

Update - The formhelpers demo is updated and is production ready. See the end of the post for details.

If you're a web developer in New York City, unless you work here, here or maybe here, you're probably not using Python for primary development (if you are, please post your company here). Since I'm not a PHP monkey or a Microsoftie, everywhere I work they're using Struts, which is where the dust has settled around Java Server Pages (with honorable mentions to Spring MVC and JSF).

I'm in a position where I might be able to push Python technology for a project or two, which have until now been built on Struts 1, the most common version of Struts even though Struts 2 is considerably nicer. So I wanted to see where Pylons is at with form rendering and processing these days. Most importantly, I wanted to ensure that layout is accomplished plainly within a template with no module-embedded HTML and no "magic generation" of forms from classes or other datastructures, and that the cycle of data from form to controller back to form again is similarly simple and obvious. Most projects in New York are the kind that get handed off to totally different people when complete, or even just 80% complete since you've been put onto something else, for remaining development and future enhancements. Code handoffs are extremely common, so it can't be overstated how much more important it is to be obvious than to be DRY. This is a big reason that tedious, plodding approaches like Struts and PHP are so popular - through raw verbosity they discourage opaque ways of doing things. Overly clever, data-driven rendering solutions that nobody can understand or extend (and are usually broken, anyway) are basically what gets thrown away and rewritten by the next team.

In developing Mako, a primary goal was to make a super-nice version of a particular "component" pattern which I had used for years primarily with HTML::Mason which for me provides a "sweet spot" of obviousness, agility, and succinctness. The focus is around the ability to create "tag libraries" which interact easily with a server-parsed templating language, and which can be implemented within templates themselves. In JSP development, taglibs are now the standard way to indicate dynamic areas of templates, but while they look pretty clean, they are painful to implement (requiring HTML embedded in hand-crafted classes, a few dozen XML pushups for every tag you add, and the obligatory application restart whenever they change), and the EL and OGNL expressions which are standard within taglibs interact terribly with straight Java code.

Mako allows the creation of tags which can be arbitrarily nested and interactive with one another via the <%def> construct, in combination with the <%call> tag. It's been my observation that the <%call> tag as well as the usage of nesting-aware <%defs> hasn't caught on yet, as the examples in the docs are a little dense, so here I will seek to demystify it a bit.

Pylons currently recommends a decent approach to rendering forms, using Form Tags which are essentially little functions you can embed in your template to render standard form elements. The handling of the form at the controller level uses FormEncode and routes validation errors through htmlfill. My approach modifies this to use Mako tags to build a site-specific taglib around the webhelpers tags and adds an explicit interaction between those tags and the controller, in a manner similar to a Struts form handler, which replaces htmlfill and allows all layout, including the layout of validation error messages, using the same template system. It also adds a preprocessor that illustrates how to build custom tags in Mako which look as nice as the built-in ones.

A tar.gz of the approach, which at this point should be regarded strictly as a proof of concept, can be downloaded here (works against Pylons 0.9.7), which contains three templates each illustrating a different approach to laying out the form. The three approaches are raw webhelpers with htmlfill, the <%call> tag approach, and finally using the "custom tag" approach. The final result, present in the file templates/mako_helpers.html, looks like this:

<%namespace name="form" file="/form_tags.mako"/>
 
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
   "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
  <title>Mako Form Helpers</title>
  <link rel="stylesheet" href="/style.css" type="text/css" />
</head>
<body>
 
<h3>Using Mako Helpers</h3>
 
<%form:form controller='comment' action='mako_post'>
<table>
    <tr>
        <th colspan="2">Submit your Comment</th>
    </tr>
    <tr>
        <td>Your Name:</td>
        <td><%form:text name="name"/></td>
    </tr>
 
    <tr>
        <td>How did you hear about this site ?</td>
        <td>
            <%form:select name="heard">
                <%form:option value="" selected="True">None</%form:option>
                % for desc, value in c.heard_choices:
                    <%form:option value="${value}">${desc}</%form:option>
                % endfor
            </%form:select>
        </td>
    </tr>
 
    <tr>
        <td>Comment:</td>
        <td><%form:textarea name="comment"/></td>
    </tr>
 
    <tr>
        <td colspan="2"><%form:submit/></td>
    </tr>
</table>
</%form:form>
 
</body>
</html>

Where of note are the Mako-like <%form:foo> tags that aren't part of Mako ! A short preprocessor is applied to the source file which turns a tag like <%form:foo> into <%call expr="foo()"> at template compile time. I.e., a tag used above as:

<%form:textarea name="comment"/>

is preprocessed into raw Mako code:

<%call expr="form.textarea(name='comment')"/>

The %call tag invokes the textarea def inside the form namespace, which is defined in the file form_tags.mako. The def for textarea looks like:

<%def name="textarea(name, default=None, **attrs)">\
<%doc>
    Render an HTML <textarea></textarea> tag pair with embedded content.
</%doc>\
${form_errors(name)}\
${h.textarea(name, content=request.params.get(name, default), **attrs)}\
</%def>

Above, the $form_errors(name) is a def call used for reporting validation messages. The point of form_tags.mako is that all the form tags and their layout is plainly visible and easily customized. Multiple versions of the file can be used in one application, providing different form layouts for different areas. The fact that it uses the h helper to render the actual HTML for each form control is also arbitrary; you could just as well implement the <textarea> source directly within the def if some special treatment were needed.

The demo also contains a modified version of Pylons' @validate decorator. Usage is mostly the same, except the form parameter is replaced by the more direct input_controller parameter, which is the method used for input:

from formhelpers.lib.mako_forms import validate as mako_validate
 
class CommentController(BaseController):
    def index(self):
        return render('/mako_helpers.html')
 
    @mako_validate(schema=CommentForm, input_controller=index)
    def mako_post(self):
        c.name = self.form_result['name']
        return render('/thanks.html')

Where above, mako_post hits the validator, and on an invalid exception the index controller method is called instead. Validation errors are placed in self.form_errors as well as c.form_errors for template access. The validator as well as the preprocessor are defined in lib/mako_forms.py.

That's pretty much all there is to it at this point, a few folks on #pylons seem enthused about it, so perhaps this can be turned into something more formally available and/or recommended when implementing a Pylons/Mako application.

UPDATE

I've refined the formhelpers demo with a @validate function tailored to a single usage pattern (download). Each form now has a name, which identifies its place on c, such as:

<%form:form name="comment_form" controller='comment' action='post'>

The controller now places a comment_form dictionary on c in all cases, which the @validate function takes care of on the post side:

@validate("comment_form", CommentForm, input_controller=index)
def post(self):
    c.name = self.form_result['name']
    return render('/thanks.html')

The hacky methodology of relating the <%form:option> tag to its parent <%form:select> tag has also been replaced with something reasonable.

This version of formhelpers is production ready.

As far as comments regarding @rest.dispatch_on and forcing a GET to accomodate it, I think that's kind of ugly since I don't really believe in "virtual requests", which is one reason I rewrote the decorator in the first place (i.e., to remove the "re-get" aspect of it). If you are dual-purposing your index() method to display the form on GET as well as to re-display on POST, then index() needs to accommodate both methods. Otherwise you can break index() out into index() and _display_index() and have _display_index() be your input_controller.

Reddit.com Goes Open Source

Reddit has opened their source up, and we can now see just what they've been up to. It's been known for some time that Reddit was (re)built using Pylons and Mako templates, contrary to their FAQ which still states that they use web.py. As it turns out, they've also built something of their own database layer, which seems to include a homegrown caching layer and ultimately is built on top of SQLAlchemy, using the SQLA expression language to generate queries. Connections are served with the QueuePool, and they use the threadlocal setting, so that they can get implicit access to transactions in progress. They vertically partition their database access among four separate engines across four distinct areas of functionality on the site.

This is currently the highest volume website I'm aware of using SQLAlchemy and Pylons, and is a testament to the stability of our core components (I hope). Python in general is not too prominent in New York City where I work; Java, PHP and .NET are still the default "goto" platforms, and most developers here look at you kind of funny when you mention Python. Look how well-known Java advocate Ted Neward says "even Python!", as though we're the most fringe Java alternative imaginable. I hope examples like Reddit continue to illustrate that Python presents the best mix of performance, stability, and rapid development for web development today, not to mention one of the broadest software ecosystems in the field (which I've always maintained is a good thing).

armin on template engines

Armin Ronacher, who has to be my biggest fan, is a great developer in his own right (and only 18 years old ? wow....). Here he is dropping a few ideas on Mako, Genshi and Jinja - while Jinjia is his own Django-like engine, he remains enthusiastic about the other two and uses all three. What a guy !

Keeping HTML Out of Your Code

The debate that has been raging on, not just for the past six months, nor for the past two years, but at least for the past ten years if not longer, is "how much code should we put in our HTML" ? As dynamic HTML systems come and go, it seems this question gets brought up and answered in some way or another over and over again.

My observation has been that a typical programmer goes through at least two, and sometimes more, stages:

1. Write code, put a bunch of HTML in it - We all know this one since we have all been there. We write a Perl CGI script, then it needs to produce a web page, so, put a big print statement at the end ! Problem solved. The ensuing nightmares of further development and maintenance bring us to stage #2.

2. No code in HTML anywhere - Seeing the evils of embedding HTML and code together lead us to the rational conclusion that they should be forever separate. Tag-based languages are devised to meet the needs of the basic dynamic-HTML trifecta: variable substitution, conditional HTML, and iterative HTML. Most then go beyond. Every programming language has a host of libraries with this task in mind.

When I was reaching stage #2, my company was just beginning to make the shift from Perl CGI scripts into Java servlets. I had already written some small templating languages for Perl (since I didn't know about CPAN) and even one for C (since the high-performance alternative to Perl CGI at that time was, C CGI). Note that I mean servlets, not JSP; JSP hadn't been invented yet. So when the boss wanted a new content management system written in Java, it was time to take all of my template-language skills and apply them to this new world of Java servlets, which until then was the language that I had used to create quirky and buggy client-side applets that talked to chat servers and failed at communicating to Javascript objects via a doomed technology called LiveConnect.

Since that company is long gone, I will announce here that my template language and accompanying CMS was called WebMechanic. WebMechanic achieved the abovementioned trifecta, and for its "beyond" category provided a full expression parser/runtime-evaluator, meaning you could do things like <if a==b && (c!=d || f==g)>MARKUP</if>. WebMechanic was then used for two of the company's client websites, the NFL Players Association and "Gillette Girls", which had something to do with getting teenage girls to start shaving their legs.

WebMechanic obviously does not run anywhere these days and has not for many years. However you can get a taste of WebMechanic by using its official template spinoff, which is known today as FreeMarker, the first version of which was written by a colleague of mine and has since been totally re-done by another group of developers. At least the last time I looked, it remained largely driven off of WM's template philosophy, which of course, being firmly rooted in stage #2, is no Java code in templates !

The immense failure of WebMechanic was that while its template language was quite powerful and featured a runtime stack for arbitrary levels of nesting/expression evaluation, calling includes and allowing all kinds of functions to be performed on the objects sent to it via controllers, it was still a royal pain in the ass. An entire team of programmers had to constantly struggle with every new template concept having a corresponding JavaBean or whatever, and all the while as I continued to hear about JSP, a technology that generated Java source files on the fly, all I could think was what are those guys, crazy ? Embedding Java code and putting it straight into generated source files? What maniacs !

Today, while Freemarker, and its more successful counterparts WebMacro and Velocity are doing just fine, it doesn't need to be said that for every WebMacro or Velocity template written, there are like two gazillion-jillion JSP files created. Why is this ? Because, at the end of the day, tag-based languages that dont let you drop out into code are a pain in the ass. Ask anyone that does XSL all day. Its safe, its clean, sure. But the level of code that has to be constantly devised to support these tag-based languages, in high-speed many-developer/ many-designer/ many-salespeople-bringing-new-crap-to-do-every-day environments is significant. Its view-based work that could often be organized more succinctly and handled by HTML designers completely if a little bit of real code here and there can be used to get you through the day. And I dont mean a lot of code; I just mean the little grease here and there to make certain intricate sections and functions work without having to re-architect a bunch of new objects with HTML fragments embedded into them. JSP with extensive usage of taglibs is possibly a decent mix of a primarily tag-oriented system that isn't forcing it upon you.

Specifically, what ends up happening with an overly rigid template system is that programmers, in order to get more intricate sections of HTML content such as highly iterative layers, Javascript that has to interact with server-side code, etc. into their output is they put the HTML into their application source. Crack open any application that is totally XSL or tag driven, has been under development for over six months by at least four people and I can assure you you'll find some HTML in the application's non-template source code...maybe a CSS name, a composed hyperlink, an <img> tag in a configuration file...I have hardly ever seen an app that doesnt need to break the rules a little bit (and usually, to a moderate degree).

So if it's accepted that a large and serious application needs to break the "rules", why not break the rules in a cleaner way that allows HTML people to have alittle more control over the view? So we go to stage #3.

3. Better we have some code in the HTML, rather than HTML in the code. As we started off, people are obsessed with not having code in HTML. As for me, I am obsessed with not having HTML in my code. Why ? Because I dont want to maintain HTML. There are plenty of HTML authors who can handle it, in fact there is pretty much zero HTML skill these days that doesn't include some Javascript programming, ActionScript programming, etc. HTML designers deal with code just fine. After all of my subsequent years doing JSP and HTML::Mason for at least five companies, I have never seen an HTML designer break embedded code in a template. It just doesn't happen; HTML'ers, while they often don't know how to architect a database-driven applciation, are usually very very good at seeing where your code is and where it isnt. Templates of course must be created with appropriate restraint (restraint which you learned by successfully completing stages #1 and #2); application logic must remain out of templates as much as possible, and database queries are just an urban legend as far as templates are concerned. I guess some people think such restraint is impossible...but excess rigidity is just going to make itself apparent as even uglier hacks somewhere else.

There is still another stage to this though.

4. Tag-based template systems are still good. They are ! Of course they are. What are they good for ? When you want markup written by people you don't know (i.e. outsourcing to India) or who are not really HTML designers at all but due to whatever organizational issue they are stuck editing a little bit of HTML here and there (i.e. Bob in the marketing department needs a section of the site he can author).

Once you go through the four stages of embedding code in HTML, all those folks who are rigidly stuck in stage #2 look pretty silly. Note that it is a skill to differentiate which tag-based template advocates are in stage #4 and which are in stage #2. If you do decide that you need a tag-based template system, its probably best that its developers are stage #4 types, as they are by definition coming from a broader perspective.