Planet JCR

It’s been over a year since I deployed Django in production, and I wasn’t looking forward to it. Last time, I had a lot of trouble with mod_python, sessions and decimal objects refuising to pickle.

Thankfully, all this really seems to have grown up in the last year – mod_wsgi is now the recommended way of deploying Django in production, and following the mod_wsgi django instructions, I was in business in 20 minutes. No fuss, no mess, no drama, and best of all, using daemon mode, no noticeable performance hit when serving static files and PHP off the same Apache installation. The ability to run the django project as its own unprivileged user when using daemon mode is also real handy.

2010 will be a bad year for IPv4 – this is exactly why I designated native IPv6 support as fundamental, not merely a “nice-to-have”, when setting up Splice last July. Hats of to Bytemark for supplying IPv6 with their hosting. I’m sure the fact that it’s excluded from their SLA is something that won’t be the case in eighteen months’ time, and meanwhile, I and the rest of the crew are using it quite happily.

Footnote: those of you reaching for the comments button to sarcastically remark that this website appears to not be available over IPv6 will be pleased to hear that I intend to fix this in the immediate future.

My e-mail still isn’t as shiny as I’d like. In particular, my use of Exim Filters to sort incoming mail into folders lacks the ability to mark messages as read (although it’s still miles ahead of the dreaded Procmail). This would be handy for high-traffic mailing lists which I don’t have time to read on a daily basis, but which I find it hard to ignore the “unread” icon next to the folders for.

One day, I should probably move to using the Dovecot LDA and its sieve implementation, which supports the “imap4flags” extension, thus allowing marking messages as read, making them turn purple in Thunderbird, and all sorts of other cool stuff. Sadly, life (or this afternoon) is too short.

In the meantime, I’ve solved the problem in the usual way I deal with life’s imperfections: gratuitious Python – http://pastebin.org/77199 run from a crojob every five minutes.

(Disclaimer: letting scratty little bits of Python anywhere near something as important as your e-mail is probably a Very Bad Idea.)

Earlier this year, I hacked together a stateless RSS reader called Bogroll.

It’s been doing sterling service for me at news.dnorth.net ever since. Today, I’ve sorted out a 0.2 release with the following improvements:

• Now caches etags/Last-Modified headers to avoid fetching a feed if it hasn’t changed since last time (thank you, Mark Pilgrim, for chapter 14 of Dive Into Python 3, which reminded me to be a good citizen in this regard). I was pleased to discover that the Universal Feed Parser it’s built on top of already supports gzip and deflate compression to save bandwidth.
• Now supports just one category per feed, because having articles appear in several categories just seems wrong to me
• Each category now really does contain the most recent X articles from the relevant feeds, because I’ve fixed the severely broken sort-by-date logic

A fair bit of refactoring has gone on under the hood, and the code now looks a bit more like an app and less like a ten-minute bodge. The next round will involve getting some proper unit tests in place, and possibly AJAX magic to load the articles lazily on the page.

You can download the 0.2 zip, or get the latest version from subversion if you like to live dangerously. The cool kids all seem to be using Git or Mercurial these days, but I haven’t found the need (or overcome the inertia) yet.

Enjoy. Feedback welcome to the usual address.

So the turkey is (at least partly) eaten, the mince pies are disappearing fast, the wrapping paper has been picked up off the floor and the presents played with. 2009 is done.

It’s hard to say what I’ll remember 2009 most for, because it’s been such a packed year for me. Perhaps I’ll remeber it as the last of my three happy years at Magdalen, the year I finished my degree, the year I set up a hosting co-operative with six friends, the year I moved away from home, the year I got a job or the year I took over as joint church treasurer.

It wasn’t a perfect year, of course – my personal TODO list is still 39 lines long – but never mind, I’m fairly sure there’s another year just around the corner. One item a week. What could possibly go wrong?

Happy New Year to all 3.5 of my readers (and all those following along via Facebook; do come and read the website this is all automatically imported from), and I look forward to seeing as many of you as possible in 2010.

During the planning of our Christmas card at work this year, a mad idea came up. Do we ignore mad ideas? No, we tackle them head-on.

The idea was to build an internet-controlled snow machine - you'd hit the button on the website, and watch a member of the Torchbox team get pelted with snow. When we first came up with the idea, we dismissed it as being "too complex", but after a while, we came around.

Cue three days of frantic development and phoning round to get the parts. I'll be posting a full build article, with all our source code, once we're done.

The brief summary is that we have a Django app which handles rate-limiting of snow, and tracking who has clicked the button, which then communicates with our snow machine using Artnet and DMX. Ustream is used to stream the video back to the internet.

Still, I imagine you want to see it in action, so head over to snow.torchbox.com, and have a go. It's only online 10 - 5 UK time, and only until Tuesday (we can't fill our office up with paper snow forever), but it's still very good fun, even if you don't know any of us!

Let’s be honest about this. Looking back, I should have known better. Nevertheless, as I describe the problem that ate far too many hours of my weekend, judge for yourself whether I was entirely to blame for How It Went…

The problem

A friend of mine has a laptop. It’s about five years old, and it runs Windows XP. This means, inevitably, that it’s a mess. My personal metric of measuring how rodgered a machine is by the number of icons in its system tray gave it a ten, and that’s pretty nasty. Nevertheless, until last Friday, there was nothing wrong with it that an uninstall fest followed by a defrag wouldn’t have fixed.

Enter the Internet Man

Last Friday, a chap called round to set up some broadband for my friend, who’d previously been on dial-up. I’m not going to name the ISP concerned, since it gives me more freedom to say nasty things about them, but suffice to say that they’re big enough that they really should have done better.

Their engineer fixed the simple(ish) problem with wiring which was stopping the broadband from working, and what he should have done at that point was connect the laptop to the router (less than six inches away on the same desk) using the supplied ethernet cable, and left. What he actually did was shove the supplied CD in the drive, which helpfully installed a few hundred megabytes of crapware onto the machine, then hooked it up to the broadband via wireless. Then he left.

Despite the above totally unnecessary shoving of stuff onto it, the laptop struggled manfully on (system tray count now up to 12) and seemed superficially fine.

Enter Dragon Naturally Speaking

My friend makes extensive use of Dragon NaturallySpeaking, a voice-recognition product which seems to knock the socks off everything else on the market when it comes to actually coping with different accents. And it was here that the problem first manifested itself: trying to use Dragon to voice-control Internet Explorer caused it to crash with an error message along the lines of the one described in this Microsoft knowledgebase article.

Enter the sucker, stage left

At this point, I fetched up, and agreed to see if I could fix the problem. Although the above KB article looked ideal, being the first hit on Google when I exercised the too-useful-to-be-documented Windows feature of “Ctrl-C copies the text of the active dialog box to the clipboard”, the hotfix it supplies claimed to be already present in service pack 3 of Windows XP. Just about the only virtue of this laptop was that it was fully up to date on patches and service packs, so what now?

Don’t press that button

Being rather short on ideas at this point, I decided to fire up the nearest thing broken Win32 boxes have to a magic bullet, namely System Restore. The machine refused to roll back to any of the restore points at first, but restarting into safe mode fixed that, and it was soon rolled back to the Friday, at a time before the problem occured.

Unfortunately, Dragon now seemed completely broken, giving the error message described in this support article. And no, of course there were no backups, I hadn’t taken one before I started, and my friend is no different to most of the non-geeky people I know in not backing up, except for dragging his holiday snaps onto CD once every six months.

To their credit (and they’re about the only players in this story to be awarded any), Nuance’s suggestion in the article of how to manually restore the user files for Dragon did work, after I rolled back the fatal system restore [or rather, didn't, because it didn't seem to have made a pre-restore restore point. Fortunately picking one from the Saturday seemed to work].

So, after two hours feeling my friend’s anxiety at the thought of having to retrain the speech recog from scratch – not a pleasant accompinement to the sick, swoopy feeling we get when we know we’ve just permanently erased some irreplaceable data – we were back at square one with the original problem. One last shot in  the dark, disabling the ISP’s nasty extensions to IE, seemed to fix the issue.

So who do we blame here? Laptop vendors, for selling machines so laden with rubbish before they even leave the factory that DLL hell seems assured the minute they meet with real life usage? Or, we could blame the idiots who seem to write the nasty unsigned drivers for most hardware on the market. We can definitely blame the ISP’s engineer for installing the crapware, but perhaps he’d been trained to, and anyway, why do ISPs think we need a CD full of crap to supplement the TCP/IP standard that’s been around for several decades? Is it really asking too much of Johnny User to plug in a cable or enter some Wifi passwords in to the applet that’s sodding well supplied with Windows, thus making the poorly writtten replacements from laptop manufacturers and ISPs alike completely superfluous? We could also blame Microsoft for making system restore not clever enough to cope with software like Dragon. Or possibly blame Nuance for not fixing or documenting what has apparently been a known incompatability for several versions of Dragon*.

Certainly, we can blame me. I clearly need to have “I will not agree to even slightly ‘fix’ someone else’s computer without taking a full disk-image of it first” tattooed across my forehead. I also clearly need to reimmerse myself in the happy world of properly written software which I’m lucky enough to earn a living in and try to forget the horrors of the last 48 hours.

* The most useful reference I could find on Google was this page. The Nuance KB doesn’t mention system restore. Then again, perhaps their customers simply don’t know this is the cause of the issue, or don’t get round to reporting it. It’s not like I have.

I’ve long been of the opinion that Firefox and Thunderbird are a bit like democracy: aclaimed far and wide as major achievements and bastions of a civilised society, but actually, honestly, a bit crap in many ways. Sadly, we’re stuck with all three until someone manages to come up with some compelling alternatives.

If you’re reaching for your e-mail client at this point to tell me I’m being unduly harsh, look me in the eye and tell me you think the way Firefox cheerfully caches DNS lookups and ignores such things as TTLs is a good idea. Or, even harder, give me one good reason why Thunderbird doesn’t check all IMAP folders for new messages by default. You can’t; in both cases it’s a disgrace.

All this being the case, I wasn’t what you’d call hopeful when, last Sunday, I had to write some e-mails on a train. And to do so, I needed to refer to some other e-mails in my Inbox. Since said train lacked anything as useful as a wireless internet service*, I’d need some sort of offline IMAP facility**.

As I bashed Thunderbird offline IMAP into Google, I was expecting a half-baked plugin at best, and “can’t be done” at worst. What I was actually very pleasantly surprised to find is that this functionality is built into Thunderbird.

What astonished me even more is that it actually worked. Faultlessly. So perhaps democracy can be salvaged after all.

* And, let’s be honest, because I still haven’t got organised and bought a phone with internet capabilities.

** Nobody uses POP3 in the twenty-first century, right?

In my neverending quest to save the time of those using RDBMSen, South 0.6.2 is released.

It mostly contains bugfixes, but there's now support for extending introspection to third-party apps without having to edit their source.

I'll probably start working full steam on South 0.7 soon - there's a refactor of the migration engine (mostly done), of startmigration (not started), and a few changes to command names to make them nicer (the old ones will still work), and to the behaviour of default values.

I'm hoping 0.7 will be the last release before 1.0, at which point I will claim I've been working in an octal version numbering sequence all along.

Dear lazyweb, does anyone else find that Spotify mysteriously hangs at the “logging in” stage when run under Wine? This suddenly happend on both my Linux boxes after it had been working fine for weeks.

Various posts on the Ubuntu forums suggested firewall issues, but nothing had changed, and Spotify continue to work on Windows on both machines.

In the end, I applied the nuclear fix of

$ mv ~/.wine ~/.wine-with-broken-spotify

and reinstalling Spotify from scratch, which worked. Might not be so handy for anyone with other apps under Wine, though.

In my opinion, there's just not enough Subversion servers on the internet.

That's an opinion you'll probably never hear me express - I much rather everyone move over to a DVCS (*ahem* mercurial) - but Subversion has, some would say quite rightly, earned its place as the dominant VCS for nearly every part of the IT community (apart from big, lumbering, financial companies).

Recently, I've been slowly switching away from subversion for my personal projects into Mercurial. This, in itself, a painless process, but with my public projects - particularly South - this has the unintended effect of slightly alienating some of my users. Most are alright with adapting - after all, it's quickly becoming the norm to use a DVCS - but there's still a few people left out.

One of the particular issues I have is with svn:externals. A lot of apps - including some we have at work - rely on svn:externals to pull in external dependencies into a libs folder along with the project itself. Externals is one of the few features of subversion that I thought was pretty much perfect, and it was sad to see my move to Mercurial break it.

This week, at DjangoCon, Chris Wanstrath did a nice talk on DVCSen. One of the questions led him onto hg-git - the awesome git backend plugin for Mercurial, that the GitHub guys wrote - and how they first investigated the idea of an svn gateway to expose their repositories transparently to subversion users. From what I gathered, subversion's wire format proved too tricky to deal with, and so they turned elsewhere.

However, the idea of a subversion gateway intrigued me greatly. What better way to transparently serve South so people can still pull it using svn:externals while still developing in Mercurial? With that in mind, I started looking into Subversion's wire protocols and their delta format (svndiff) sometime yesterday afternoon.

After some digging, reading an obscure academic paper and liberal application of Wireshark, I grew confident enough that I could at least implement something. A day later, and I'd like to present the very first version of what I'm calling Heechee (if you get the pun, ten nerd points).

Heechee is a transparent mercurial-as-subversion gateway. It serves a Mercurial repository as a Subversion WebDAV-based repository. It's still in the early stages, but at the moment it will serve its own mercurial repository to subversion in such a way that you can check out the repository, and update to various revisions within it.

You can check it out at BitBucket. It's pretty alpha code, and make sure you have the dependencies mentioned in the README, but it works, which greatly surprises me. I plan to much improve the code to support more 'advanced' features, like being able to do more than checkout and update, as well as exposing tags and branches correctly. There's even the chance I'll stick Git support in, when I've had a play with Dulwich.

Still, hopefully I'll soon have it so it can serve South's code via svn, and thus restore harmony to my land of VCS users.

So, please, take the code and have a play with it. It's not formally licensed and released yet, but it will probably be Apache licensed. Feedback and suggestions for features, as well as how to linearise trees into svn histories, much welcomed!

South 0.6.1 is now available for public consumption; it fixes quite a few bugs.

If you want to see what's been fixed, have a gander at the Release Notes - there's a few significant bugs that have been fixed, including problems with SQLite, GeoDjango, and M2M fields.

Sorry for the short but sweet post - I'm in the process of writing my talk for DjangoCon! I'm giving a talk about South, with both a good grounding for those who haven't grokked migrations yet, as well as some tips on more advanced usage. Come watch it on Tuesday morning!

It is with great pleasure (and a measurable sense of relief) that I announce the release of South 0.6, a new release bringing quite a few new features, although most of them aren't immediately obvious.

Perhaps the biggest change is the move from parsing source files to get the field definitions (ugh, I know) to introspecting the fields to get their definitions. As well as being more stable and making my skin crawl less, this also means that dynamically-created models and fields are now fully supported.

South still falls back on the parser module for custom fields if it has to - since it doesn't know how to introspect them - but we encourage you to implement a South-readable interface for custom fields; see our wiki page explaining how to do that.

The other noticeable change is that the migrate command is much zippier, thanks mostly to the hard work and lazy-evaluation-patches of Juan and Simon from Akoha. Note that the SQL for migrations is no longer printed out as they're running any more - it was getting a bit verbose - but you can re-enable it with --verbosity=2. You can also completely shut South up using --verbosity=0, if you desire.

For easier conversion, there's a new one-shot convert_to_south command - take care to read the wiki page on that, however, as the caveat is that you must run a migrate --fake on any other computers running the same codebase. We're going to make that a bit more intuitive for the next release.

The release notes contain the rest of the changes - some big, some small - and remember, we've switched to Mercurial for revision control, so if you're still running off Subversion please change your checkouts.

After a long few months, during which I've increased my recommendations to run off trunk, South 0.6 is almost here.

This release contains a whole bunch of new features - more of which will be written up on the Release Notes page soon - but in the meantime, I've decided to adopt a more usual release process.

This means two things: First, I'm releasing a release candidate of South 0.6 for general testing and final critical-bug reporting; please see the Download page and follow the instructions to check out the 'stableish' tag, or if you already have a checkout please 'hg pull' and 'hg up' it.

The Git mirror is also, for once, up to date, and there's also a tar.gz available for those who don't fancy either DVCS.

The second change is that I'll actually maintain a stable branch from now on as well, much like Django. This means there will be a 0.6 branch for any fixes that are easy or trivial to backport, and in the unlikely case someone should manage to conceive of a security vulnerability in a migrations library fixes for that will go in too. This should hopefully mean you and your production servers can track a real release number and not miss out on bugfixes.

Note that a few of the planned features haven't made it into this release in time, but I felt I had to get something out there. My much-vaunted fixture migration didn't make it in, but introspection, the major feature, did.

Still, please, test the release! If things look good, I'll officially tag and release 0.6 early next week.

Just installed Ubuntu Jaunty fresh onto a new ext4 partition on my Advent 4211 netbook. Am delighted to report that WiFi, suspend and hibernate all work flawlessly out of the box, so I can finally stop using Windows on this machine.

It’s amazing what sort of shrinkage – both in size and price – you can miss if you’re not buying in a particular market. Yesterday, out for lunch with some friends, one of them showed off his latest toy, a 2GB USB drive. Which, as you can see even in these amateurish photos, is similar in size to an SD card:

It’s called the Tuff ‘n’ Tiny, apparently, and if you live in the UK, Ryman are selling the 2GB version online for £5.99 and two for a tenner on the high street.

I can also confirm that mine has partitioned quite happily into two FAT32 partitions, and, with prodding, boots Linux off one of them [the second. I had to use usb-creator, followed by install-mbr to get the bootloader to work]. I don’t think I’ll be following John’s suggestion of RAIDing two together just yet, but who knows…

Just over 18 months ago, I signed up for a 256slice from Slicehost to host this website, my email, etc. Later this week, I shall be shutting the machine down and cancelling my account with them.

For the record, this has nothing to do with the level of service I’ve received from them – I’ve always found their support team quick to respond and helpful, and their articles site and wiki are both very handy.

However, a combination of fluctuations of the pound against the dollar, a surge in demand for RAM by my applications and sites, and my getting increasingly fed up with transatlantic ping times of 130ms meant the machine was becoming unfit for purpose and overloaded.

Thus, I have now found a reasonably cheap way to bring “my stuff” home to hosting in the UK. About which, I will be writing more shortly!

So goodbye, Slicehost, and keep up the good work. I’d certainly recommend you to anyone who lives in America and needs a VM.

And Ubuntu gets seven out of ten

Last week, it was clear my parents’ ageing XP box needed some attention – in fact, let’s be honest, it had been crying out for a zap-and-reload since last year.

This time, though, I was determined. With my sister safely moved off onto her new laptop, there should be no reason not to migrate the box to Ubuntu. The only things in use on it were Firefox, Thunderbird and Openoffice…so how hard could it be?

Five days on, I’m in a position to report that the answer is “harder than it should have been” – here’s what happened:

Problem 1: Audio CD autoplay

Dad wanted the machine to start playing his CDs automatically when he put them in, much like Windows does. Seemed perfectly reasonable to me, as I blithely assured him it’d be a matter of a few mouse clicks.

It wasn’t.

Much Googling and swearing later, this hack (fifth comment down) seems to do the necessary, though I had to bump the sleep interval to six seconds.

Problem 2: No matter how close KPatience comes, it’s not “real FreeCell”

…largely because, despite generating the same deals from the same numbers as the Windows version, it counts every loss of the same game number as a loss in the stats, whereas the Microsoft one counts them as one loss.

Fortunately, freecell.exe runs fine under wine, so copy it across and save yourself the bother.

In conclusion

Other than the above minor niggles, all seems well, and hopefully the machine should now do a few more years’ trouble-free duty.

So, as the title of this post says, Windows XP has now been deposed as my OS of choice for “real people”. Whoever would’a thought it?

Update, 18 July 2009

I forgot to say that probably the most impressive feature of the switch was that Ubuntu flawlessly detected and printed a test page on our trusty old Epson Stylus C62. That’s the sort of slickness we like to see.

It's about time I posted an update on what's going on with South development, so here it is.

Firstly, there hasn't been much work done in the last few weeks, due to exams and there being a Ball to help run. Now that's over, development will crack down. 0.6 is going to focus on the following new features:

• Less parsing, more introspection. The default way of getting fields' definitions will be to introspect them; the implementation of this is pretty much complete already, and can be found in the 'noparsing' Mercurial branch. Custom fields (which cannot be introspected withour prior knowledge) have two options; provide a south_field_triple method to tell us their definition (preferred way), or as a fallback the parser will be used. This feature solves many issues, including dynamic models, and also doesn't lose the ability to work with most custom fields right from the start.
• Reverse dependencies. Just brought up by one of the Ella developers, this is a good idea that should have been in South a while back. You'll be able to specify 'this other migration depends on us' as well as 'we depend on that other migration' - this helps pluggable apps around a common core.
• Fixture migration. Since I know this is going to bite me in the ass at some point, I'll be rolling in the fixture-migration someone mentioned at EuroDjangoCon (uses the test runner to make a temporary database, loads an old fixture set, an old schema, migrates, dumps the new fixture). I haven't fully tested this yet, so it might not make it for reasons of impossibility, but I'm reasonably confident.
• Signals. There will be post-migrate and pre-migrate signals added to help developers add in custom code that needs to be run after migrations or before migrations (the latter helps people trying to carry out a deployable app rename, for example)

As well as those new features, there's also around 25 bugs that need fixing, and will be tackled soon, hopefully. I aim to get a beta release of these out in the next month, especially in the light of South's allegedly high penetration rate.

The last piece of news is that I'm moving my main development efforts to Mercurial (and specifically BitBucket), and only pushing mostly-stable trunk phases to Subversion. There's a github mirror of whatever hg-git deems to be main development, but that's for convenience only, and because I can do it in one command.

I guess I might as well park these here as anywhere else:

Some hilarity courtesy of bash.org: #693046, #9214, #105199, #626249, #610952

Funny source code comments – “throw up”, snigger. Nothing like Java-based humour.

Batteries Feel Included: 307 – it’s a bit strange, but it made me nearly fall off my chair laughing

A genius article from 1995 explaining the trouble with the internet.

This post brought to you by the department of “I’ve finished finals and have absolutely naff all to do for the next week”.

I had a lot of interesting chats with people about South last week at EuroDjangoCon, and several eyebrows have been raised at me both parsing models and then storing their definitions as dicts.

While this is a design that is partially born of backwards compatibility, it was mostly a conscious design choice I had to make, and so I'd like to now describe why I made it.

Firstly, for those who aren't aware, South's magic autodetection and auto-writing-migration code is based off of a file called 'modelsparser.py', which does what it says on the tin; it opens models.py, parses it, and extracts definitions directly.

Now, my first response to this is also 'ew', but there's a reason for it: custom fields. If it wasn't for those, introspection of the existing fields would work just fine, given a long list of special cases (I in fact have a file called modelsinspector.py sitting around that does just that - if you're interested, it's in the 'noparsing' branch on BitBucket. It doesn't do much, but it proves the concept).

However, with custom fields in the game, you have two options: hope they follow Django conventions and try to introspect common options, or put the onus on custom field authors to add a hook that makes it spit out a definition. While I would obviously choose the second one (possibly added to some of the first), it simply doesn't provide the drop-in compatability I want South to provide.

However, as Alex Gaynor and several others have surmised, I could just pickle everything and stick that at the bottom. No introspection needed, no weird namespace errors because the models dictionary is simply a collection of eval-able strings.

However, I don't like pickle (the Python kind; the delicious food kind, on the other hand, is more than welcome in my household). The security issues don't matter here - after all, we're passing raw strings to eval instead - but it has two major features I don't like. Firstly, it's not at all human-readable - and while I don't imagine people editing their migrations by hand, I like to be able to see what is going on, and I like the fact that South still doesn't depend on you using startmigration - a design choice I'm very proud of.

Secondly, I've been burned by pickles in the past. If you remove a module that a pickled thing uses, pickle will crash and burn, and if you've not done them right, you won't know quite where the error happened. (For the purposes of argument, of course, I'm assuming I'd do it right, so this isn't really a point. Still...)

There's yet another option (see, my job isn't easy) - custom-pickling, which Alex suggested last Wednesday; it consists of roughly freezing the __dict__ of the field. This has some advantages - it's more human-readable - but you get the errors similar to my current approach (where datetime.datetime wasn't unfreezing if people didn't have datetime imported).

My main gripe, with both this and pickle, is that the internals of fields aren't guaranteed to be consistent between releases of Django, just the external API, and so South uses a method that lets it use the external API - in my mind, this is less hassle, since that's the thing that's really not going to change very much (and even if it does, you can easily see what South is trying to call, and run that in a shell yourself to see where the problem lies).

So, that's why models are frozen as a massive lump of dictionaries and triples. As with many things, it probably wouldn't be necessary if models had been done slightly differently (so there was a method that always spat out a nice, portable way to reincarnate a field), but with no reason for that previously, I would have been thoroughly impressed by the core developers' foresight if it existed.

As for parsing versus introspection... I'm still divided. Parsing always has, and still feels, wrong, but introspection might just be less reliable. If you're interested, go peek at the branch I mentioned above, and see what you make of it.

And now, I will stop blathering and return to my small hole at the base of the mighty Django Mountain, where I am attempting to fit square pegs into round holes. Good day.

Quick tip for anyone like me who’s foolish enough to cram Exim4, Spamassasin, Apache with Wordpress running on it, and a pile of other stuff on to a server with 256MB of RAM: Install one of the Wordpress caching plugins. Otherwise, your server is liable to start swapping frantically if you get a dozen or so hits on one of the Wordpress-powered sites.

Indeed, it was possible to nuke jasper just by mashing the F5 key whilst sitting on the front page of this site.

You can now sign in to leave comments here (yes, dear readers, both of you!) – using an OpenID if you have one, and in related news, I’ve got m’self a shiny new OpenID to go and do likewise on other people’s sites.

For those interested, the slides from my migrations talk at EuroDjangoCon are up. I believe the videos will be around soon.

As I write, the sun is up, the bells are ringing, and Oxford is dragging itself out of bed for another day. It’s slightly sobering to think that I’ve just listened to the college choir for my last May morning as an undergraduate…hopefully not my last time from inside college, though – time to start making friends with lower year groups.

I’m going to miss Magdalen. But meanwhile, eight weeks still to enjoy!

a.k.a. “How to query for all objects not in a given subclass”

I ran across this problem today – do please leave a comment if you know of a better way to solve it than what follows.

You have two Django models, one of which inherits from the other, e.g.:

class Order(models.Model):
    # some fields

class DiningOrder(Order):
    # some more fields

So, how do I write a query (using the Django ORM) which returns all the Orders which are not DiningOrders?

Apparently, this does the trick:

>>> Order.objects.all().count()
762L
>>> DiningOrder.objects.all().count()
93L
>>> Order.objects.exclude(id__in=[d.id for d in DiningOrder.objects.all()]).count()
669L

I'm pleased to announce the release of South 0.5; this has been a long and exciting release, with many new features...

• ORM Freezing, a new feature inspired by Migratory, which lets you freeze the historical state of your current models into a migration so you can then use them via an ORM as if your models.py file existed at that point in time. There's a good description of how exactly this is used in part 3 of the new tutorial.
• Automatic change detection; using the frozen model states, South can now diff your current models.py file against how it looked last time you made a migration, and write a migration to make up the difference. This essentially allows django-evolution style hinting, and means you'll only have to manually write more advanced migrations. It's discussed more in part two of the new tutorial.
• A much more robust models.py parser, using the Python parser module. It's a dirty secret that South automatically makes migrations by directly examining your models' source code; however, the new parsing engine hits 99% of all Django use cases, and has provisions for custom hooks so more wacky fields (such as django-denorm's decorated-function-based-fields) can tell South their equivalent definitions.
• A move into a south/ subdirectory, to fit with Python packaging guidelines, and make South more friendly with environments such as buildout and pip.

I've been very excited about the new features for a while, as those who have talked to me will tell you, and thanks to those people who used the trunk releases and submitted a great set of bug reports, it's finally ready for general consumption!

Additionally, for those going to EuroDjangoCon, don't miss my talk on migrations next Monday; as well as South and its new features, I'll be covering and comparing django-evolution and a few other migration solutions.

Happy migrating!

My main problem with RSS readers is that if I go away for a few days and come back, I don’t really have time to catch up with all the posts which came up whilst I was away. What I needed, therefore, was a stateless reader which would just show the newest 10 entries or so.

Add in category filtering and a web interface – such as it is – and you get Bogroll, which I’ve just released under the GPL. You can grab the 0.1 release, check out the code from subversion, and have a nosey at my installation.

Incidentally, if anyone can think of a better name for it, do get in touch.

Stopping overnight in Reading last weekend, with a complete lack of reading material, I picked up an actual dead-tree copy of PC Pro magazine for the first time in years. I’ve read it on and off since discovering three years’ worth of back issues on a shelf at the back of my school library – the output of Jon Honeyball, in particular, is usually good for a laugh – followed, more often than not, by some serious thinking about the issues raised.

The column that most caught my eye this time, though, was one which explained the Rebel Simcard, which appears to offer a way of running an iPhone 3G on the network of your choice, without having to risk bricking the phone or voiding the warranty.

That’s good, but unfortulately, it’s still not good enough for me to send the £400 or so I have earmarked for a smartphone in Apple’s direction. It still doesn’t support tethering, the inability to run a SIP client in the background would be a huge annoyance, and developing for it would require the purchase of a £68 SDK (OK) and a £1000+ Mac to run it on (not OK, and neither is having to distribute code through the app store).

All of the above are things which the phone is technically capable of doing, but has been prevented from for various reasons. Meanwhile, my personal preference for keyboards – even small ones – over touchscreens which I struggle to operate with my medium-sized fingers, is another negative of the iPhone.

At the moment, I’m hanging on to the money and pinning my hopes on the Palm Pre, which should arrive in the UK later this year. If it’s as good as the hype says it is, supports tethering, and can be purchased on its own without an expensive contract I don’t need or want, it should fit the bill very nicely.

Let’s be honest, shall we: if you’re reading this, you’re probably the sort of person who’s been where I was last week: on the recieving end of a phone call from a friend/relative/friend of a friend/friend’s relative, enquiring if you could help them with a “computer problem”…

In this case, it all seemed pretty simple: the chap on the other end of the phone was rightly concerned that his second-hand Windows XP box had come without any antivirus software installed.

“No problem”, I blithely replied, “just install the free edition of AVG“.

And therein began the difficulty: no matter how carefully I spelled it out, the chap insisted that visiting the URL for AVG’s site resulted in a 404 error. Other sites were visible, but not that one. Emailing the link produced the same result, and so, thoroughly puzzled, I agreed to a site visit the next day.

Sure enough, from both IE and Firefox, the machine would not display pages from avg.com. Further experimentation revealed that most of the other antivirus vendors’ sites were similarly blocked, along with microsoft.com.

This rang faint bells with me – viruses have been blocking such sites for years – and full of confidence that I was going to be able to pull off a quick fix and look like a genius, I pulled up the C:\Windows\System32\Drivers\etc\hosts file, expecting to find a bunch of domains being redirected to 127.0.0.1. What I actually found was nothing which shouldn’t have been there.

So then I broke out the usual tools: Spybot, Rootkit Revealer, a manual installation of AVG via disk from another machine, HijackThis … several hours of the machine wrongly being pronounced clean later, I remembered reading that Microsoft’s OncCare sometimes managed to disinfect machines other products didn’t.

I downloaded the free trial (via another machine as the Microsoft site was being blocked) and fired it up. Its first quick scan found nothing, but it also recommended installing Internet Explorer 7. I think it must have been the running of the Microsoft malicious software removal tool during the IE install which finally disinfected the machine to the point where a full OneCare scan found and removed the last of what it proclaimed to be the Conficker worm.

More Windows-savvy readers might well be saying at this point that I should have recognised the Conficker symptoms earlier. The truth is, though, that after nearly three years using Linux on my machines, it’s starting to grate to have to find my way around Windows and fix knackered installs thereof, and I’ve been cheerfully avoiding articles about the latest threats in the Windows world, smug in the knowledge that it’s not my problem any more. .. until the phone rings.

It turns out that the aspect of the infection which had me confused the most – the blocking of the AV sites – is achieved by hooking into the relevant windows API calls and tampering with their behaviour. The question has to be asked, though – why the hell is it possible for code running as a limited user – or, come to that, any user – to intercept API calls like this?

Asside from the still-woeful state of its security, Windows really does feel prehistoric to me now – no central mechanism for installing and updating software, leading to every app having its own updater (launched at startup for maximum slowdown of the machine, of course), no decent command-line, and all the nonsense of antivirus, antispyware and anti-everything-else.

It must be ten years since I started getting the phone calls from people with broken machines, and yet it’s still happening. I saw the future of home computing last week, and it was a broken Windows box. Time to think the unthinkable and start installing Ubuntu as the standard remedy?

Have you ever felt the burning need to ride aimlessly aimfully around the London Underground? Well, then I have good news for you.

My new in-development project is Mornington Square (holding page only so far!) - it's a adaptation of the classic game of Mornington Crescent, a game universally enjoyed by young and old.

The key part of the adaptation is that, unlike other online versions, this one is played by riding around the real underground. A GPS phone (and a location broker like Fire Eagle) updates the server with where you are (this does mean climbing out of stations each time you make a move).

The rough gameplay outline goes as follows (if you want just the technical nuggets, skip a bit):

First, contestants start either in the same location, or in predetermined equidistant-in-time-from-the-destination locations. They're all heading towards the same place - say, Mornington Crescent.

Before each move, the server analyses each player's location and assigns them a pseudorandom set of rules they must follow for the next move. Rules limit the lines you can travel on - so, for example, if you're told that the Occidental Grouping Restriction applies, you can't use Aldgate or Aldgate East. (The rules all have explanations for new players, and a map of possible moves, including restrictions imposed by rules, is shown)

The choice isn't random; the server will try several tens of sets of rules and select the one it think looks most appropriate (which is usually the one blocking any short routes - it knows.)

Then, each player looks at what moves they're allowed (you're always only allowed to move on one line at a time, so you have to choose an appropriate next stop), and gives the server a time estimate, and then they start moving.

Players must get to their next stop in the time they allocated; if they are late, they suffer a 5-minute time penalty. If they are early, they must wait until the end of the time they originally allocated - this encourages good time estimation!

The whole process repeats until someone makes it to the destination and their time runs out (or their penalty period does) - then they're the winner.

You're required to submit your location at the end of every move, to ensure you're in the right place. One possible measure for stopping location cheating is to ask the person to stand next to a local landmark you know the position of, but this shouldn't be a problem early on.

The server also has an overview page showing all the players, their current moves, and generally useful information.

So, it sounds good, but it actually works, as well! So far the rule-calculation code and map drawing code is all done, so it's just the location and timing left before there's a fully-functional beta, which I shall hopefully be demoing at BarCampLondon6 (possibly by actually playing it that evening).

The site is built using Django, of course, along with my usual use of Cairo for drawing the custom Underground maps for each player's move (they're geographical, and not the proper Underground style yet, however).

Rules are defined by exclusion functions - a rule can exclude any route, set of station names, or stations matching a regular expression. An A* algorithm does the route planning, scoring each ruleset based on the shortest route to each destination, allowing more time for interchanges between lines (I have basically invented a small travel planner).

I am perhaps too excited at the prospect of an excuse to go around riding the underground, and emerging from the surface, only to find that the server has decided that on the next move, wild lifts are high, and so I can't go through Covent Garden, but it'll all come out in testing (which I shall be doing in secret before the next DJUGL).

Work on South has been going full steam ahead lately, with some good results.

Two major features that I've been wanting to implement for ages are now in SVN trunk, and due for imminent release with South 0.5. They are:

• ORM Freezing: Much like the strangely similar feature of Migratory (which is what inspired this), South now has the ability to 'freeze' models to make them accessible inside the migration, so you can write data migrations more easily. Things like fake models needed for the other ends of ForeignKeys (when creating tables and the like) are also now implemented using the same system, to make things cleaner.

• Automatic change detection: Before this was impossible, but now, if you freeze all of an app's models into a migration, South can compare that with the current state and automatically write a migration for the changes that have occurred. There's a setting to automatically freeze the app into every migration to make this more usable, too.

I'm very excited about these new features; they should make the next release of South one not to be missed. There's also other changes under the hood, as well, including a new, much more robust models.py parser. Documentation for these features is up already, and the tutorial will be extended (given a second part) soon, in order to cover them - they're features we want everyone using, since they'll save you time, and, hopefully stress.

If you feel like testing the new features before the next release, just check out trunk and have a play. Your feedback is also more than welcome on our new South mailing list, which you should also have a look at.

I've always been somewhat envious when seeing the various sites with nice maps of crime, prices and other things for non-UK regions.

However, over the past week and a bit me and Martin have been changing that. WhoseTurf rolled out of the coding mill some time last week, and it's been improved a lot since, in our quest for useful (and fun) online geographical statistics for the UK.

GeoDjango has been on my list of things to play with properly for a long time, and so it was good fun to get my teeth into it. It works surprisingly well - the admin interface was really helpful when collating all our various area info (and even drawing some districts by hand - not a fun task when your source map is on a different projection).

At the core of WhoseTurf is a load of 'reviews'. Each review has a certain 'fanout' - the radius in which it can affect the ratings - and, accordingly, most of the work in our custom tile renderer is correctly interpolating between the correct review points and their values.

Having circles affecting values rather than the actual discrete shapes of the various UK districts makes for some slight inaccuracy, but for most variables this makes more sense than drawing it discretely (crime doesn't suddenly drop when you change county, for example). Council tax is one of the few exceptions to this, and so accordingly is drawn discretely.

Another clever trick we use is that, because the heatmaps we serve up just consist of tiles of large squares, we simply serve 40x40px images and let the browser scale them up to 256x256. This cuts down on rendering time and bandwidth, makes our cache (Varnish) happier, and also lets me crow on about how clever it is (even though it's reasonably obvious).

It turns out there's some things GeoDjango just can't do - like this massive query that makes correctly-shaped districts out of the various output areas we have (they don't fill the whole space, nor line up perfectly, so otherwise you get holes or hair-thin 'cracks'):

SELECT st_collect(cpoly) FROM (
        SELECT st_simplify(st_makepolygon(st_exteriorring(
        (st_dump(upoly)).geom )), 0.001) AS cpoly FROM (
            SELECT st_union(spoly) AS upoly FROM (
                SELECT st_snaptogrid(mpoly, 0.01) AS spoly
                FROM core_outputarea
                WHERE id LIKE '""" + code + """%%' AND
                st_isvalid(mpoly) = true)
                AS a
            ) AS b
        )
    AS c;

The other small problem was Firefox not having enough JS stack space to load in the border of the Highlands (it has lots of very silly little islands and glens), but I'm willing to let that one slide.

We'll probably be releasing the code behind the thing as open source; I doubt any of it is _that_ transferable, but there is all the code in there to render standard (spherical mercator) tiles using cairo and/or PIL, and if you too want to show lots of heatmaps, and perhaps kill your server, it could be for you.

One last side note: OpenLayers is, unfortunately, still not really useable. Google maps is a) smaller and b) more efficient (drags more smoothly, scroll zoom, although I did have to write our own layers widget). On the flip side, OpenStreetMap is getting more than useable for a site like ours now.

Next time one of your application or websites goes wrong and the users shout at you, take comfort in the knowledge that even Google sometimes Get It Wrong…

(retrieved at 14:44 GMT on Saturday 31 January 2009)

Edit: As Michael points out in the comments, Oxford users can avoid the mess below by simply disabling hybrid mode.

So there I am, firing up the Advent to read my email between lectures when…

$ sudo vpn-connect oxford
vpnc-connect was built without openssl: Can't do hybrid or cert mode.

Gah? The VPN client has always worked in the past, but following the upgrade to Ubuntu Intrepid, it doesn’t. A quick search lead to a solution which can be summarised as:

$ cd /tmp
$ apt-get source vpnc
$ cd vpnc-0.5.1r275
$ $EDITOR Makefile

Uncomment the lines beginning with OPENSSL
Read this and despair for the future of open source software

$ sudo apt-get build-dep vpnc
$ dpkg-buildpkg

Wonder why it all collapses in a heap
Search and find this.

$ apt-get install libssl-dev
$ dpkg-buildpkg
$ cd ..
$ sudo dpkg -i vpnc_0.5.1r275-1ubuntu1_i386.deb

Wonder why this has changed between Hardy and Intrepid
Seriously consider going back to Windows on this machine.

Sigh.

Welcome to the new aeracode.org, now with a bit more blue, and added Unantistalker™ technology.

As well as a graphical tweak, I've also added a small map. In case you haven't figured it out, it's (supposedly) my current location; the actual age of the data is shown further down the page in the sidebar.

This is accomplished by having a Python script on my phone which uploads its current location to my server every time it moves; if there's a GPS available, it'll use that for exact coordinates, otherwise it just submits the cell ID and I attempt to find it in the OpenCellID database (I submit cell locations to it if a GPS is around, so in theory I need only walk a route once to locate all the cells, then I can use just cell IDs from there on).

While it may make it easier for the stalkers (that I probably wish I was famous enough to have) to find me, I'm not terribly worried; I can always add a bit of random jitter (and besides, location by cell ID alone isn't terribly accurate).

When everything's working, I'll post my code up here; until then, if you're sad enough, you can probably watch me walk to and from lectures. Alternatively, you may want to go get on with your life.

Today I took delivery of one USB Panic Button; I've been wanting one of these for a very long time, and so I'm pleased to finally have one. I don't necessarily have a need to panic a lot, but there's a lot of times a big red button is just so satisfying (launch events, test runs, and so on).

Unfortunately, it comes with a Windows driver CD and the only Linux script I could find was a Perl script. Since I'm not the greatest fan of Perl, and I wanted to try some USB code, I've made my own driver for them.

You can download it here; just edit the COMMAND = line at the top of the script to change what it runs. Alternatively, you can import the file for use as a module; just make a new instance of PanicButton and call .pressed() to see if it was pressed recently (yes, you have to poll it).

Also note it has to be run as root, to prise off the kernel's HID module.

Finally, here's one more picture of it in situ.

After far too many months of quiet feature development and bug fixing, we're happy to finally announce South 0.4, which has a whole host of new features to satiate your every migration need.

The list of new features alone really is quite long; you can see it at http://south.aeracode.org/wiki/ReleaseNotes/0.4, but I'll go through a few of them quickly:

Firstly, there's alter_column, which you just pass a new Field instance into and it will deal with changing the column type, nullity and default value for you. --add-field is also a new option to startmigration, which simplifies the common use case of adding one or more new fields as a migration.

Model inheritance and ManyToMany relationships are both now fully supported, and there's a glut of database support with MySQL being fully matured and SQLite almost there. Even SQL Server has a tiny amount of creation-only support.

So, go check it out, and let us know what you think. If you're in or near London, as well, come along to DJUGL; I'll be talking about South there.

Just a heads-up to any LastGraph users to let you know that, soon, I'll be implementing one-week graph storage expiry (but probably not for premium users). There's now 128GB of PDFs being stored on S3, and it's probably time I started not throwing money away.

What this means is, from around the middle of next week, possibly earlier, when you make a graph, the download links you get will only be valid for seven days. After that, the files on the other end will vanish into the aether, so if you want to show them off you should probably download them and host them on your own blog.

Still, I'd like to give S3 credit; when LastGraph originally launched, I had a server with 10GB of disk space and only a bit more bandwidth. These days, LastGraph is pulling 44GB of transfer a month at S3 alone, and I have 4.4GB of stored user data (and that's compressed, children), so I'd like to thank the guys at State51 and Last.fm for the resources they've donated towards it, and the Premium users for just meeting the S3 bills so far with their signups!

I’ve just finished a Python script to shuffle the names for Secret Santa and email each participant the name of the person they’re giving to. Thus if you’re not organized enough to get everybody together to draw names out of a hat, you can still do it without one person having to know who’s giving to whom.

Sadly, I didn’t get this done in time to give it a try last week, but hopefully somebody will find a use for it.

You can grab the latest version of the script here.

It depends on pytz, because in order to generate emails which stand half a chance of getting through spam filters, it needs to be able to generate proper timezone-aware dates. Using the raw email sending libraries in Python was a bit of an eye-opener - turns out Django takes care of a lot of twiddly bits behind the scenes, which you don’t notice until you have to do it all by hand.

Spotted this on Farcebook Facebook just now…

I’d laugh, but somebody probably sued them when the free pig didn’t turn up.

It generally takes a lot to get me to really like a piece of hardware, thoroughly. For example, my experience of Fujitsu-Siemens' customer service was nothing but excellent, but the only reason I had so much contact with them was that my laptop's inverter went three times (to their credit, it's been fine for the past two years or so now, and my brother is still happily using it, but that laptop still sheds screws at the rate of around one a year).

However, a bit over a year ago, I decided to invest in some bluetooth headphones, and picked a pair of Jabra BT620s (warning: flash music thingy) semi-randomly from the field of available devices. A year later, and I can honestly say they're one of the better things I've bought this decade.

They're well-designed, and have good build quality; they also take and recieve phone calls, with pretty decent sound quality; they also turn into a USB Audio device (headphones and mic) when plugged in and turned on, a feature that is nearly impossible to find out about, and more importantly, last for absolutely ages. I've left these things on in my bag for over a week sometimes, and they'll still be happily flashing blue (they do that when switched on, but you can disable it, else you look a little weird) when I get them out.

I only wish the software support for A2DP was as good. Under Linux, it will work via ALSA, but PulseAudio still segfaults whenever you take it near them, Maemo has okay support, but often stutters slightly and the driver uses a lot of CPU, and Symbian is the only thing that seems to deal with them rather well (except for the occasional confusion when it decides it really should start playing the music over the tinny phone speakers at full volume, instead).

My only complaint is that they're a little large and heavy, but given that this seems to be needed for the batteries and speakers, it's fine, and I have big ears, anyway.

I also, in a strange way, enjoy having both earphones and an alarm clock with mini-USB ports and upgradable firmware; it's obviously the future. In a sick way, I can't wait for the day when I need to spend 30 minutes reflashing my shower so there's more than a millimetre between the "scalding hot" and "arctic cold" settings.

It's been a while since I last talked about South, so it's about time I did a short update, I think. There's been a fair few things going on since last time, which was, err, a long time ago.

Firstly, we now have SQLite support (apart from column renames/alters/deletes; I'll be implementing those with the appropriate workarounds soon). This was one of the features I most wanted to roll in, since SQLite is quite widely used for quick development. SQL Server support is also on the way soon!

Also in the news, TWiD's latest episode has some interesting poll results for schema migrations; of the respondents, over half are using one of the premade tools, and of those about 2/5 are using South - good news for the migration camp overall, and a surprisingly high amount for South, considering I have a limited picture of who's using it.

Other news includes startmigration getting support for multi-table inheritance (finally), some nice new work on indexes to flesh out the database API, and plenty of bugfixes and small tweaks to make things flow more smoothly.

Also in the pipeline is the possibility of adding an option to send post_syncdb signals at the end; several people have asked for this, and for some coding styles it will greatly reduce headaches about functions running too early in the migrations.

Finally, 0.4 will probably be released at some point soon, so keep a look out!

I've always not been one for phones; for years, I had a (relatively) old Nokia 6100. It does the job of phone calls and SMS messaging very well, and lasts absolutely ages on a single charge.

At the beginning of the year, though, I wanted 3G connectivity for my PDA and laptop (yes, I have a whole range of portable devices, and every one has its use, although not necessarily simultaneously), and so liberated a Nokia 6280 from eBay. The 6280 was pretty good - long battery life, a WAP browser, and that all-important connectivity. However, when its screen broke last week, and replacement LCDs were not too cheap, I went back to eBay with two rules: it had to be under £60, and should fit the two 6280 batteries and umpteen small Nokia chargers I have (the Internet Tablets use them too).

As you can imagine, this doesn't leave a great deal of choice. In fact, excluding the 6280, it leaves pretty much only the N73. Having experienced the 6280's wonderful affinity for dust (Nokia didn't quite get the sliding mechanism dustproof - it gets inside the screen...), I decided to go for an N73 and see what they were like; after all, how bad can crazy selection criteria be?

Not too bad at all, it turns out. The N73 is a pretty neat little device, but more importantly, it runs Symbian. I had presumed all the older smartphones were like the 6280, but I was pleased to discover that there's more S60 apps out there than I can shake a stick at, and that the version the N73 runs is still the most current, compatability-wise.

Three days later, and I have a whole set of things, including Google Maps, which gets impressive position accuracy from cell info alone, the QR code reader I would have needed a year ago at Barcamp Brighton, my Google Calendar synced up, an application that changes to and from silent mode depending on my location (using cell towers) and calendar events (i.e. anything marked as a lecture), and more importantly, the Python.

In short, it's lovely. The next step is getting a permanent, cheap data plan; Orange will sell you one for £5 a week (on Pay As You Go; I'm not shelling out hundreds on a contract), but Three have the suspiciously good offer of £5/month for 2GB, which is better than some ADSL packages I've seen, so I'm seriously considering migrating. The only thing holding me back is Orange's excellent customer service, and the fact I have a deal from them which means I get five free texts a day, forever, for £12 that I paid back in 2001 or so. I'll have to see if I can keep that under a new number.

I’ve not been sleeping all that well lately. And let’s be honest, this probably has something to do with being in the middle of a term at Oxford University, with a lot on my mind and far too much to see to.

However.

I’ve noticed recently that my room contains a surprising number of light sources, which I suspect can’t be helping me get to sleep. Reading from left to right, we have:

• Light leaking under the door from the staircase (where the lights are apparently permanently on)
• Blue power LED flashing on my laptop
• Green power/charging LED on my other laptop
• Huge red light on the multiplug under my desk
• Flashing lights on my hub

I’ve therefore come up with a cunning plan:

• Block the base of the door with an old pillow
• Prop a book over the laptop light
• Put the other laptop in a drawer
• Tie an old sock round the multiplug
• Shove the hub into a cardboard box with holes for the wires

Whether or not this helps remains to be seen. But here’s a plea to manufacturers: please think twice about the brightness and location of your LEDs, especially if there’s a chance someone will be trying to sleep in the same room as the device they’re on.

I finally got round to putting Linux on my Advent 4211 yesterday. There are various ways and means of doing this; the ones which worked for me were:

• Use the installation image optimised for small screens
• Follow these instructions to make a bootable USB drive from the image (note: it’s ‘cat’, not ‘zcat’ as the ubuntu image is not zipped. I know they’re Debian instructions, but they seem to work)
• Use the script given in this forum post (run as root) to get the WiFi working.

The end results are really rather impressive; here’s a pretty picture:

Here’s one for all you progammers out there: what happens to the values of x and y if you type the following in, say, Python?


>>> x = 5
>>> y = 6
>>> x,y = y,x


I’d always half-thought, without really considering it, that you’d end up with x=6 and y=6, i.e. that the above was syntactic sugar for:


>>> x = y
>>> y = x


Actually, though, it turns out…


>>> x=5
>>> y=6
>>> x,y=y,x
>>> print x,y
6 5


Do you write Python (not very well?).

Ever wondered why your calls to os.execvp() and its ilk don’t work as expected?

The answer may be that you’re doing this:

os.execvp('ls', ['/home/fred'])

When you actually mean this:

os.execvp('ls', ['ls', '/home/fred'])

As ever, reading the Python docs (help(os.execvp)) is usually better than randomly googling for an answer.

One of the topics that popped up repeatedly at DjangoCon last weekend was how bad purely normalised table structures are, and how denormalisation is good for many things, including making your database cry less.

To that end, during the gaps in PyConUK this weekend, I decided to see how easy it would be to write a new Django field that will automatically denormalise a field in a related table across to another model.

Let's take an example; imagine we have this kind of model:

class Picture(models.model):
    user = models.ForeignKey(User)
    user_username = DenormField(User, "username")
    image = models.ImageField(upload_to="foo")
    title = models.CharField(max_length=100)

The idea is that the user's username (which, let's face it, you're going to need almost every time you need the photo) is stored in the same table, saving you from doing that join to get it from users each time. While this may not seem like much, it can be useful, and besides, it was fun to write.

My DenormField simply listens to a trio of the relevant signals, and then updates this value whenever it detects that the related User object has been updated, so it only costs on write (which, let's face it, is much better than on every read). You can do this kind of copying in normal Django app code as well, but I think this solution is much cleaner. I would do, I guess, since I wrote it.

It's only really a little test exercise in writing Strange New Fields (and I wanted to see how much monkeypatching it would take - none, as it turns out, just some very nasty code tricks). You can get a copy of it if you feel like having a play (warning: only briefly tested; may attach captions to your cat).

There are not very many things that could tempt me to fly 10 hours twice for only a weekend, but DjangoCon was one of them, and I'm glad I went; it was one of the best weekends I've had for quite a while.

There were plenty of good talks; I can't mention them all here, since as we all know, bytes cost money, but in that vein I must at least mention Cal Henderson's talk 'Why I Hate Django' - a brilliantly done talk on the things Django's missing, as well as the fact we don't have a mascot (or do we?) and that we aren't smug enough.

Still, I enjoyed many of the other talks as well (what a shame it is not to be able to see them all in person; still, Google will hopefully release the videos soon), as well as our Schema Evolutions panel, which seems to point towards a bright future for Django and migrations. 

Talking of bright futures, judging by the final session with Adrian and Jacob, there's plenty of stuff to get working on for 1.1, some of which have me quite excited. Multiple database support is going to be one of those important ones, really, but then Malcolm seems to have snuck the bottom layer of it in already, which is quite handy.

All in all, an excellent weekend, and finally a chance to meet a subset of everyone involved in Django (and they really do come from many places), and I can now hardly wait to get over to Prague next year for the promised EuroDjangoCon. If you didn't make it, then I highly recommend watching the videos when Google release them (after editing them on their ridicously great AV system, no doubt).

The world of migrations in Django is definitely warming up now, and with that we're proud to announce the release of South 0.3, the 'intelligent django migrations app', available now from south.aeracode.org.

New in this release are the use of fields rather than dicts for specifying columns in migrations, so migrations now look cleaner, work with all the databases Django supports to a better extent, and even support custom fields.

Also new is dependencies between apps, for those situations where your forum app's Post depends on your accounts app's Profile; South will work out the right way to apply everything so foreign keys don't horribly break.

There's also a nice number of bugfixes and general tweaks Andy and I have been making as South moves towards being used in active development work. If you think you might be needing migrations, or even if you don't, give South a try!

(Also new is a very carefully worded "why we're best" page on our wiki. I'm trying not to step on people's toes, just say why we're using South rather than anything else, but I'm not sure how well that's working)

With the launch of its own site - south.aeracode.org - and Andy McCurdy jumping in to help out, South has been making some good progress. We've made a few backwards incompatable changes, which while annoying will hopefully make things much easier in future.

Firstly, the format of the create_table and add_column calls has been changed to use Fields instead of long dicts of parameters. As well as looking more Django-like, this allows us to reuse the SQL generating features Django already has for fields, as well as playing nicer with custom types.

In line with these changes, startmigration has also seen some changes; as well as generating things using Fields now, it can group models together in migrations (this proves useful for foreign keys referencing tables made after them).

We're still working on finishing this all off, and there'll hopefully be something stable to show off soon. Woop.

Also, if you're going to DjangoCon, may I recommend the Schema Evolutions panel; there'll be all manner of evolutiony and migrationary things discussed therein.

I bought myself an Advent 4211 last week (you might know it as the MSI Wind; they’re essentially the same thing). So far, it’s lived up to expectations; there are just a couple of things to say…

• Mine seems to have the decent Synaptics touchpad - read The Register’s review for the warning about the other one.
• You really need to use headphones with it if apps you’re listening to don’t have a very good sound level - inevitably in a device this size, the speakers aren’t great, especially at full volume.

How the Linux install on it goes is something I shall let y’all know shortly…

I’ve been aware for some time that my DNS isn’t quite as securely configured as I’d like. http://crashrecovery.org/named/ looks pretty good, but the two main issues bugging me were:

1. Anyone could do a ‘dig @ns.dnorth.net dnorth.net AXFR’ to retrieve a listing of all my DNS records - not great from a security point of view. This is a capability that should only be turned on for secondary DNS servers which need to fetch from the master.
2. The server would perform arbitrary lookups [for any domain] on request. This means it’s operating in ‘recursive mode’, which is a Bad Thing for various reasons.

The solutions were:

1. Add “allow-transfer { “slaves”; };” (without the double quotes) to the section of the configuration beginning “zone ‘dnorth.net’”. Then add a section defining the “slaves” access control list to be the local server, plus the secondaries: “acl slaves { 127.0.0.1; 123.45.67.89; }” replacing 123.45.67.89 by the IP address(es) of your secondary nameserver(s).
2. Add “recursion: no;” to the “options” section of the configuration.

Then restart the BIND9 service - on Debian, this is “/etc/init.d/bind9 restart”.

Health warning: Don’t do (2) above if you rely on your server to do its own DNS resolution - follow the crashrecovery tutorial above instead.

I did a lightning talk to CompSoc last night on whether 64bit is ready for use on the desktop - if you’re bored, you can download the slides and the second of my famous sketch and scanned graphs.

gq has been a necessary evil in my life for some time now. I need a graphical LDAP client for use on the CompSoc systems, but gq (to be fair, the versions of gq packaged for Ubuntu) seems to be very buggy, segfaulting all over the place if you try to do anything other than browse with it.

Last week, after upgrading to a 64bit version of Ubuntu for the first time, I finally ditched gq, after running into identical symptoms to this Debian bug.

The good news is that there’s an alternative that actually works: it’s called Luma.

Have fun.

The epic tale of how my new mail setup was born

My personal email has been on a rough ride over the years: from a reasonably nice (Microsoft based!) school email setup in 1999, which sadly got removed when Windows 98 was introduced, I went through three Hotmail accounts. I had a brief flirtation with GMail, but not being all that keen on the means of delivering advertising, I ended up back on Hotmail.

Registering dnorth.net last year at least ensured my email address would no longer change, but the technical capabilities of the two mail servers holding the mail still left much to be desired: flaky, unreliable spam scoring, no facilities for server-side filtering/sorting, sheer lack of customisability…

Last week, I finally did something about it. At my disposal was my VPS, running Debian Linux 4 (’etch’). On the wishlist were:

• Accurate server-side spam scoring with SMTP-time rejection of the most obvious spam
• Sender verification
• Sieve filters for server-side sorting into folders
• All mail stored on the server and accessible over secure IMAP

Thankfully, none of the above is too difficult: some pretty good instructions are Out There for most of it. The ones I used were:

• Exim4 on Debian with SpamAssassin, ClamAV, Virtual Domain Alias Files and Message Size Limits per Domain
• Exim + Sieve + Avelsieve Howto
• Handling mail for multiple virtual domains with exim4

Please remember, I am not responsible for the content of external sites (e.g., the links above), nor can I accept any responsibility for the consequences of acting on the points below…

I ran into a couple of issues:

• Permissions on the .sievesource files generated by AvelSieve - I needed to chmod g+w on /var/lib/squirrelmail/data and chown it to www-data:www-data in order to reach a state where Exim could read the file, and Avelsieve could write it.
• Exim4’s native sieve implementation only has the core features in it, not the extensions defined in RFC 3431. I dodged the issue by matching the number of *s in the X-Spam-Score header using string matching, rather than numeric checks on the X-Spam-Score.

All in all, though, it’s working a treat. Email perfection at last!

Twelve months ago, I installed Linux (Kubuntu) on my laptop.

Twelve minutes ago, I got the wireless card working with my WPA-PSK home network.

A combination of the progress made on drivers over the last year, and finding which of the squillions of instruction sets out there Actually Worked has finally got things going - rather than recount it all here, I’ll point owners of the HP Pavilion dv5157eu to this page, and advise everyone else as to how I found it: do a Google search for some combination of ‘ubuntu’, ‘wireless’ and your laptop’s exact model number.

Edit: My wireless happiness sadly didn’t survive a hibernate or a reboot. Turns out the line of the page which says do ’sudo echo ndiswrapper > /etc/module’ actually means, as far as I can tell, that you should do:

sudo -s
echo ndiswrapper >> /etc/modules

As ever, follow my advice at your own risk.

Musings on public transport, and a look round The Other University

So, full term ended yesterday, and I had to get over to Cambridge for a dinner. The X5 bus proved reasonable, both in terms of cost (£15 open return) and time, though the long stop to change drivers in Bedford lengthened an already long enough journey.

After a very pleasant afternoon being shown round Cambridge, I’ll have to begrudgingly admit to being impressed by it. King’s College Chapel, in particular, was spectacular. Even the weather wasn’t bad.

The train journey home from Oxford was tolerable, though prolonged by a needless half hour stop in Birmingham - it wasn’t so much the stop that annoyed me, but the fact that nobody bothered to tell us about it.

Yesterday only served to reaffirm my belief that public transport is mediocre at best, and ghastly at worst. Our rulers will have to try harder if they want to convince me not to use the driving licence lurking in my wallet, as soon as I can afford to…

A problem which has been bugging me for a while is my Konsole and Yakuake terminal sessions ceasing to respond to input for no apparent reason. The symptoms are always the same: the session seemingly stops responding to key presses, yet if I was running screen inside the session, reattaching it elsewhere reveals the characters from the missing key presses.

The answer turns out to be that hitting scroll lock locks the terminal so that the arrow keys scroll it rather than cycling through previous commands, and this also prevents key presses updating the display. My Yakuake hotkey is F12, which is next to scroll lock on my laptop.

So now you know.