2a. Try to replace idioms that break in Py3K with cross-version ones

Easiest is if the code in question can be modified so that it works on both platforms, as run through the 2to3 tool for the Py3k version. This is generally where a lot of the bytes/unicode issues wind up. Such as, code like this:

hexlify(somestring)

...doesn't work in Py3k, hexlify() needs bytes. So a change like this might be appropriate:

hexlify(somestring.encode('utf-8'))

or in Mako, the render() method returns an encoded string, which on Py3k is bytes. A unit test was doing this:

html_error = template.render()
assert "RuntimeError: test" in html_error

We fixed it to instead say this:

html_error = template.render()
assert "RuntimeError: test" in str(html_error)

2b. Use Runtime Version Flags to Handle Usage / Library Incompatibilities

SQLAlchemy has a util package which includes code similar to this:

import sys
py3k = sys.version_info >= (3, 0)
py3k_flag = getattr(sys, 'py3kwarning', False)
py26 = sys.version_info >= (2, 6)
jython = sys.platform.startswith('java')
win32 = sys.platform.startswith('win')

This is basically getting some flags upfront that we can use to select behaviors specific to different platforms. Other parts of the library can say from sqlalchemy.util import py3k if we need to switch off some runtime behavior for Py3k (or Jython, or an older Python version).

In Mako we use this flag to do things like switching among 'unicode' and 'str' template filters:

if util.py3k:
    self.default_filters = ['str']
else:
    self.default_filters = ['unicode']

We use it to mark certain unit tests as unsupported (skip_if() is a decorator we use in our Nose tests which raises SkipTest if the given expression is True):

@skip_if(lambda: util.py3k)
def test_quoting_non_unicode(self):
    # ...

For our previously mentioned issue with callable() (which apparently is coming back in Python 3.2), we have a block in SQLAlchemy's compat.py module like this, which returns to us callable(), cmp(), and reduce():

if py3k:
    def callable(fn):
        return hasattr(fn, '__call__')
    def cmp(a, b):
        return (a > b) - (a < b)

    from functools import reduce
else:
    callable = __builtin__.callable
    cmp = __builtin__.cmp
    reduce = __builtin__.reduce

2c. Use a Preprocessor

The "runtime flags" approach is probably as far as 90% of Python libraries need to go. In SQLAlchemy, we took a more heavy handed approach, which is to bolt a preprocessor onto the 2to3 tool. The advantage here is that you can handle incompatible syntaxes, you don't need to be concerned about whatever latency a runtime boolean flag might introduce into some critical section, and in my opinion its a little easier to read, particularly in class declarations where you can maintain the same level of indentation.

The preprocessor is part of the SQLAlchemy distribution and you can also download it here. It currently uses a monkeypatch approach to work.

I've mentioned the usage of a preprocessor in some other forums and mentioned it in talks, but as yet I don't know of anyone else using this approach. I would welcome suggestions how we could do this better, such as if there's a way to get a regular 2to3 "fixer" to do it without the need for monkeypatching (I couldn't get that to work - the system doesn't read comment lines for one thing), or otherwise some approach that has similar advantages to the preprocessor.

An example is our IdentityMap dict subclass, paraphrased here, where we had to define iteritems() on the Python 2 platform as returning an iterator, but on Python 3 that needs to be the items() method:

class IdentityMap(dict):
    # ...

    def items(self):
    # Py2K
        return list(self.iteritems())

    def iteritems(self):
    # end Py2K
        return iter(self._get_items())

Above, the "# Py2K / # end Py2K" comments are picked up, and when passed to the 2to3 tool, the code looks like this:

class IdentityMap(dict):
    # ...

    def items(self):
    # start Py2K
    #    return list(self.iteritems())
    #
    #def iteritems(self):
    # end Py2K
        return iter(self._get_items())

We also use it in cases where new syntactical features are useful. When we re-throw DBAPI exceptions, its nice for us to use Python3's from keyword to do it so that we can chain the exceptions together, something we can't do in Python 2:

# Py3K
#raise MyException(e) from e
# Py2K
raise MyException(e), None, sys.exc_info()[2]
# end Py2K

The 2to3 tool turns the above into a with_traceback() call, also it does it incorrectly on Python 2.6 (was fixed in 2.7). The from keyword has a slightly different meaning than with_traceback() in that both exceptions are preserved in a "chain". Run through the preprocessor we get:

# start Py3K
raise MyException(e) from e
# end Py3K
# start Py2K
#raise MyException(e), None, sys.exc_info()[2]
# end Py2K

After the preprocessor modifies the incoming text stream, it passes it off to the 2to3 tool where the remaining Python 2 idioms are converted to Python 3. The tool ignores code that's already Python 3 compatible (luckily).