MickBlog: translucentcode

I've been thinking of rewriting my site's code for a while. Apart from the fact I've neglected the code base a bit and I'm too lazy to bring it into Django's post magic removal age, it's always fun to rewrite personal projects. Particularly if they aren't important to a company, you can hack away with aplomb. It's also good for keeping my interest in the site, I find the amount of useful articles I write is proportional to how much code I'm writing in the background.

So, how am I rewriting it? My crazy plan at the moment is to go python old school and write my own site framework! It's not as bad as it sounds, I'm using mako for the templates, I'm using webob for the request/response objects (this actually covers 99% of the fiddly bits in a framework in my opinion), probably SQLAlchemy for storage (though I might play with a non-SQL approach) and my own custom url dispatcher modelled very closely on django's regex pattern system.

The latter is the only real custom code I need to write, and at the moment it's a teeny bit of code. Ironically, with the adoption of WSGI to unite web frameworks, and the move towards consolidating on a few frameworks has opened up the possibility of writing your own custom site using off the shelf components. I think this is a good thing, I suspect I'll only have a few hundred lines of custom code at most, and this will give me a site which works exactly the way I want it to.

For example, I'm adding method arguments to the url dispatcher, so I can say (r'/articles/add', mysite.article_add_form, methods=["GET"]) and (r'/articles/add', mysite.article_add, methods=["POST"]) to handle two completely different actions without having a fiddly if/else in my code.

With doctest and nose I've got a bunch of tests which are pretty sweet too, no mucking about with funny classes.

All in all, it's a good time to be a unorthodox developer, with so many good components to choose from, and increasingly standard ways to glue them together without losing your sanity, it's fun to write your own web framework again :)

It's been a little quiet around here, so to make sure I don't get removed from people's subscriptions I thought I'd post some random thoughts about django.

I've been using it quite happily for a while now but there are definitely niggles and little things which can be fixed. These usually get addressed before I get around to reporting them, 'cos someone else has encountered them too.

There are two big things I'd love to see changed thought.

The first is the data backend. While you aren't restricted to django's sql layer for your data needs, you are if you want to use django's admin interface (which is one of the big pluses in django). With the work going on to integrate SQLAlchemy I'd love to see a generic interface between the admin interface and the data backend developed, so I could plug in my own data storage code (e.g. ZODB or flat files).

The other one is probably harder to implement without disrupting django's users a bit. I would love to see the death of DJANGO_SETTINGS_MODULE and the related global settings. Instead I'd like to see some kind of per app context which you'd plug in your templates, url conf and data backend.

The problem of a global settings module becomes very acute if you want to host multiple django apps in a single app server. In my case the server this site is on has a limit of 20 processes. Combine this with a love of using subdomains on this site and you get a recipe for one fastcgi process per site.

Instead I want to write a twisted server which hosts the entire site, handing off subdomains to individual WSGI django apps.

I've been seeing the odd transient mysql error on this site relating to load on the server. Until I move over to postgresql (which I suspect will be better purely because it's harder to setup on this textdrive server, hence less people using it), I've put a simple django middleware in place to try and reduce the failed page requests slightly.

You can see the middleware here. All it does is look for a set of known exceptions and retries the call a few times until either failure or success.

Personally I'd only recommend this as a stop gap solution, and I'll have to see how well it works on this site.

I've finally gotten around to upgrading this site to the latest post magic-removal django. It was less painful than I thought, the only gotcha was the content_type table. I had to fix the blog entry content type to get the comments appearing again. Other than that it was a case of running their SQL and fixing my table names.

Next: Move to postgresql and fix my dying web server problems.

I've been having problems with my lighttpd process dying every so often, so I've decided to try out Supervisor 2 by Chris McDonough. It's a little daemon which you run, which in turn spawns and controls other daemons. It's like daemontools. The main feature I like is automatic restarts of dead daemons.

The only odd niggle I've encountered was the supervisord process racing and being killed by textdrive's reaper. I've inserted a time.sleep into the main loop but I need to figure out which bit isn't blocking when it's expected to block. I suspect the semantics of select are a little different on the BSD box this site is hosted on.