I just halved the running time of one of my test suites.

The tests in question are multi-threaded, and while they perform a lot of IO they still push the CPU pretty hard. For some time now, nose has been reporting a happy little message along these lines:

Ran 35 tests in 24.893s

I wouldn't have though anything of it, but every so often this number would drop dramatically – often down to as little as 15 seconds. After a lot of puzzling, I realised that the tests would run faster whenever I had another test suite running at the same time. Making my computer work harder made these tests run almost twice as fast!

Like most web frameworks, Django provides a convenient mechanism for storing data across requests in a persistent "session" object. Like most web frameworks, Django implements sessions using a simple mapping from a "session key" to a session object stored on the server. And like most web frameworks, Django's default session implementation is trivially vulnerable to session hijacking attacks.

One of my commercial projects requires a space-efficient object serialisation format, and until now I've been using the obvious choice in Google's Protocol Buffers. I'm happy enough with the format itself, but the experience of using the Python bindings was just barely satisfactory. The interface feels quite Java-ish and there are some non-obvious gotchas, such as having to use special methods to manipulate list fields. I ploughed ahead, but was quietly looking around for alternatives.

The last straw came when I tried to establish a deployment scheme using pip requirements files. Both pip install protobuf and easy_install protobuf fail hard: the pypi eggs are out of date, the source download has a non-standard structure, and the setup.py script tries to bootstrap itself using the protobuf compiler that it assumes you have already built. Yuck. This was more pain than I was willing to put up with. Plus it was a good opportunity to take another look around.

Following my previous post on testing Django with Windmill, I quickly ran into a common snag with in-browser web app testing: it's not possible to programmatically set the value of file input fields. This makes it very difficult to test file upload functionality using frameworks such as Windmill or Selenium.

In Firefox it's possible to request elevated permissions for your unit tests, but this is far from ideal. It means the tests are no longer automatic (you have to click "yes, grant this page extra permissions" whenever the tests are run) and it takes other browsers out of the testing loop. Like many things in life, the easiest solution seems to be simply to fake it.

I've been having my mind blown by Django over the course of this week. Not the in flashy one-shining-moment-of-brilliance kind of way, but through a slowly dawning awareness of just how much it makes possible. Or perhaps it's more accurate to say: just how much I need to re-calibrate my expectations of what should be possible, and what should be downright easy. My latest little epiphany has revolved around unit-testing, which back when I was cutting my teeth on PHP4 was far from a trivial undertaking for even a simple web-app.

If you care about these things, you probably already know: Python 3 was released last week to much fanfare. There has been some good-natured debate about the pros and cons of switching from the 2.x series, focused mostly around the big-ticket changes like better Unicode handling (pro) and breaking compatibility with all existing Python libraries (con). Instead, I wanted to share a small joy I've found in Python 3 that I'm already missing in Python 2: extended iterable unpacking.

Yesterday I released a new version of the PyEnchant library with two important forward-looking features.

First, I've switched from generating the C-library binding with SWIG to the awesomeness that is ctypes. The process was very straightforward and the switch brings a couple of significant advantages. PyEnchant is now a pure-python extension module, making it much simpler to distribute and saving me the trouble of creating a separate installer for each python version. More importantly, it means that PyEnchant can now be used with PyPy! There are also ctypes implementations in the works for both Jython and IronPython.

Second, PyEnchant is now upwards-compatible with the upcoming Python 3 series. Fortunately PyEnchant doesn't use too many advanced features of Python, so it's possible to support both Python 2 and Python 3 from a single codebase. However, it does take a little work to manage the differences between string objects in the two versions. These tricks might be useful to others so I'll give a brief overview below.

Inspired by the never-ending stream of "explicit self is really ugly" comments that Python seems to attract, and wanting to hack around a little in the deep bowels of Python, I've put together a new python module called autoself.

As the name suggests, it does one very simple thing: automagically adds 'self' as the first argument in a method definition. It doesn't turn local variables into instance attributes. It doesn't change the semantics of method calls in any way shape or form. It just lets you save five keystrokes when defining a method. Will I be using this in my own code? No. No no no. I love explicit 'self'! But it was a lot of fun, and surprisingly subtle to get right in non-trivial cases such as inner classes. And I can now cross "python bytecode hacking" off my things-to-do list. (Interesting fact: python uses different bytecodes to access local variables, for optimization purposes. So turning 'self' from a free variable reference to a local variable requires rewriting the function's bytecode)