Planet JCR

The problem

A word document (plain old .doc, not 2007) should be received by e-mail, fed to a script, turned into a PDF and published on a website.

At my disposal

My server running Debian ‘Lenny’, which does not have a display of any kind.

How hard can it be?

Harder than it should have been, as ever. Here are my steps:

# aptitude install python-uno xvfb openoffice.org-java-common openoffice.org-writer unoconv

You’ll note the inclusion of Xvfb there, because it turns out that “headless” mode in OpenOffice isn’t really headless at all. Sigh. Also sigh some more at the broken dependencies of the unoconv Debian package.

Now we can write our script to do the actual conversion. Shame it took twice as long as it should have…

In June/July 2010 I spent ten days travelling in Rajasthan, India with friends; this is my diary of the trip (full list of entries here).

Saturday 26 July

Landed at Indira Gandhi International Airport at 9.30AM local time. Walked out of the terminal into searing heat (over 40 degrees in the shade) and a medley of noise and traffic. Dodged our way through the traffic and found our taxi waiting. Saw our bags strapped to the roof and set off into the traffic. As we’d been warned, the only similarity between British driving and Indian is that both countries drive on the left – in India the horn means “get out of my way please, I want to pass” and flashing your lights means “I’m not going to stop, so get out of the way or I’ll ram you”. In the city, people, animals and motorbikes all compete for road space with the cars and lorries, there are no traffic lights to speak of, and the smallest gap is made to be squeezed through.

On the main highway – this first day we’re driving from Dheli to K’s grandparents’ house in Jaipur, approximately 200 miles to the south-east – the sheer volume of traffic and the two-lane road limits progress to a maximum of 50mph. Drove past Debenhams and M&S on the way out of Dheli; wondering just how westernized this country is…

Stopped for lunch at India’s answer to the motorway services, about 1pm. Stepped out of our air-conditioned taxi into searing, dusty mid-day heat and ran for cover in the building, which had just enough air conditioning and fans to reduce the temperature from ‘horrifically hot’ to ‘bearable’. Ate a nice samosa with a coke, then back to the car.

Arrived in Jaipur at about 4.30pm, glad to get out of the car. The drive down wasn’t particularly inspiring; mostly motorway.

Met K’s grandparents – great people, made us feel really welcome – settled into our hotel and had a really nice meal with them before going to bed (or trying to; despite the air conditioning, still pretty warm!).

Editor’s note: more to follow soon, with photos!

After a slightly too long hiatus, I've released the second bugfix release of the 0.7 series.There's not a whole lot new and exciting in this one, just lots of small fixes - bugfixes, removal of some unused variables, a few more command line arguments where they should have been already, that sort of thing. You can read more in the release notes. In other news, I'll be giving a talk at DjangoCon US 2010 which, for once, isn't about South, but instead covers a more broad spectrum of databases and schemas, in particular encompassing schemaless databases (which do still need migrations, in some form), so if you're going to be at the conference, come along, or just catch me outside of a session if you want to talk. I apologise for the slightly slow rate of work on South as of late, but I'm in the middle of moving cities. After next week, I'll be settled in London, and hopefully work will progress faster. I'll also be looking for contract/consulting work after next week, so if you'd like to hire me, see my 'corporate site'.

In June/July 2010 I spent ten days travelling in Rajasthan, India with friends; this is my diary of the trip (full list of entries here).

Friday 25 July

Flew out from Heathrow at 20.30 with Jet Airways, bound for New Delhi. A pretty decent (for 30,000 feet) Indian meal for dinner, then fell asleep over Russia after watching James Bond. India is 4.5 hours ahead of BST, so we’ll arive at about 9AM if all goes according to plan. All of us have had the relevant jabs and are carrying industrial quantities of suncream and insect repellent.

The trip was suggested by my friend K who was born in India and lived there for 15 years, and we plan to start by staying with his grandparents who live in Jaipur, the capital city of Rajasthan. Currently it’s reported to be 40 degrees centigrade in New Delhi; here’s hoping my system can cope…

Editor’s note: it gets a lot more exciting than the above, I promise. Parts 1 to 10 coming over the next week when I get time to write them up.

Five years after the cool kids first started jumping on the bandwagon, I’ve got myself a CurrentCost CC128 (Southern Electric send them to some customers for free, it seems – e.g. my granddad who didn’t want it).

So, with the addition of an eight quid data cable and the Linux box running in my lounge, may I present my electricity usage graphs. Bear in mind that these are (at the time of writing) for a five-bedroom house in central Oxford.

The parser for the XML output of the device I’m using is this one – just swap “COM20″ for “/dev/ttyUSB0″ in their testrun script and fix it to ignore empty lines read from the serial port, and you’re in business. I then hackdapted this RRDTool tutorial to plot the graphs.

After Transport For London released their new live departures API, and after Matthew Somerville made a Thing with it, I felt I had to have a go.Thus, I present Live Line, a visualisation of the trains on Tube lines overlaid on their schematic maps. It currently only does the non-forking lines - the line maps are drawn entirely with canvas, and there's a few code issues - but it still looks quite nice. The idea, and some of the logic, is borrowed from Matthew Somerville's idea, which uses geographic maps for a similar purpose.

For at least a year now, people have been suggesting to me that South should be in Django core.I've always resisted, for good reason - not only because of South's relative immaturity at the time, but also because forcing a single solution would, I think, not be a good thing in general. (South, admittedly, has a few design issues, but it's all about tradeoffs.) However, last week at djangocon.eu, I revealed, in rapid-fire English (apologies to non-native speakers for that, we're working on time dilation) my Grand Plan for the future of South and Django migrations in general, one that seems to have the approval of at least a few members of Django core. The proposal is not, as some expected, to put South completely into Django core; instead, South will be split into two parts. One, the part that implements database abstraction, dependency resolution, and history tracking, would be put into Django itself. The other part - autodetection, model freezing, and so forth, will remain as South. (If you'd like more in-depth details of the split, see my post to django-developers.) The idea behind this is simple - take the parts of South that nearly any migration solution would need (database abstraction especially), and move those into a place where they can be used easily by any migrations solution. More interestingly, because the proposal describes a common idea of dependencies and of what a migration actually is, it means that migrations can be written such that they need no supporting framework at all, and can be just pure Python calls to the database APIs (if Django ever ships any migrations, this is probably how they'd be done), raw SQL files (with a version per database backend if needs be), or a Python file managed by a supporting library (South 2.0 migrations), and that these different formats can all coexist, even on the same application. Some might argue that such a solution is slightly restrictive, but I see it as the job of a framework to provide some level of consistency, and by doing this, it means reusable apps can ship with migrations written using the library of their choice (or no supporting library, as the case may be). What does this mean for South? Well, in the short term, South 1.0 will be released, being mainly a polishing-off release of 0.7, and introducing a few new concepts (like the "rebase"). In the longer term, if my proposal meets with acceptance from the community (but, more importantly, those illustrious figures that make up Django's core developers), I'll start work on a branch to move the relevant parts over and firm up some APIs, and it can hopefully be finished and production-ready in time for 1.4. At the same time, I'll be cleaning up the South codebase to match, and will then release South 2.0 at the same time as the migration APIs are released with Django. South 2.0 will, necessarily, only support that version of Django - previous versions will have to stick with South 1.0, which will get maintenance fixes. I'll also ensure there's a reasonably clean upgrade path from 1.0 to 2.0. So, I'd love to hear your feedback and thoughts on my proposal. The mailing list thread is probably a good place for constructive comments or criticism. (Also, slides for my talk are available here, and there's also a video.)

The first bugfix release of South is now out, and there's plenty of fixes.It's been perhaps a little too long since the 0.7 release, so I'm pleased to announce that 0.7.1 has now arrived. It's really all small changes, as befits a point release - the release notes highlight the few slightly more major changes. In other news, I'll be at djangocon.eu next week, talking about both South up to now, as well as what I think the future holds. The jury's still out on what exactly will happen, but I have a few interesting proposals I'll be talking about, some of which are a bit more grand than others. If you're not going to be able to make it, don't worry, I'm sure you can all be subjected to me in two-dimensional video form later on. It's a once in a lifetime experience, mostly because few people are awake enough to try listening to it a second time. I'll also obviously be posting my proposals here and in other appropriate places, to gauge feedback.

I’ve just read an interesting piece over at The Register on the bloated awkwardness of Visual Studio 2010, and another on the question of whether we need IDEs at all.

The latter is a difficult question for me – on the one hand, there’s a school of thought I have some sympathy with, which says that IDEs are a crutch of the feeble-minded, and allow bad programmers to kid themselves that they’re good, because they can generate lots of code automatically and hit ctrl-space if they run out of ideas.

However.

When I started programming, six years ago (!), the first three languages I used were Turbo Pascal for Windows, PHP and Visual Basic 6. TPW was a good basic language for teaching A-Level Computing, but the built-in editor was scarcely better than Notepad. I can’t remember if it had a compile-and-run button, but I seem to recall not. PHP was slightly better – once I’d worked out how to get Apache onto my Windows machine and sacrificed a chicken to get PHP talking to it – but again, no IDE out of the box, and even the relatively advanced capabilities built into the copy of DreamWeaver  (was I the only student in the country honest enough to cough up over £100 for it?) didn’t feel up to much.

VB6, though, felt magic. In retrospect, it was a horrid language, but not only could I drop controls onto a form and double-click to generate the outline of the method they’d activate, I could actually pause and resume the code while it was running! I could see the values of variables at a point in execution, and even go backwards and forwards. The completion facilities of the IDE were basic, but they were there, and they made things much faster. Writing code to automate Microsoft Office was a particular sweet spot – run the macro recorder to generate code containing roughly the API calls you were after, then drop some VB6 control flow round them, and off we go.

Later, reading Computer Science at Oxford, the practicals we did rarely stipulated an IDE, but we nearly always ended up using gedit + the relevant command-line compiler. Certainly, the existence of IDEs for Haskell was never alluded to – either the ultimate example of clever people thinking IDEs are the preserve of the feeble-minded, or the assumption that we’d be clever enough to go and look for one ourselves, depending on how silly I want to consider myself in retrospect.

I’m happy to say that Java hardly featured at all in our courses (I’m with Joel on that one), but when it did, we were told to use BlueJ, because, Mike Spivey explained, “it has only two buttons, and Eclipse has hundreds of others you don’t need”.

Given the short length of the practicals involved, I only paused to think “pish, how many buttons can it have?”, but I didn’t feel the need to find out for myself until I started writing Java for a living. He was right, there are hundreds of them. Despite which, I use Eclipse every day at work, and would never dream of trying to write code without it. It is, irrefutably, a big, bloated beast, but when you’re working on serious real-world Java, with version control, coding standards, complicated dependencies, hundreds of packages making up one program, and spend far more time reading and debugging code than writing it, you really do need the beast on your side (or so I believe).

So that’s it, then – I’ve converted to the world of IDEs? Well, not quite. The other language I use on a day-to-day basis – though mostly for pleasure rather than business – is Python. And I don’t usually use an IDE, simply bashing out code in Notepad++ on Windows, or KDevelop on Linux. OK, so KDevelop is sort-of an IDE, but it’s very lightweight.

Of course, Python being interpreted rather than compiled makes it easier to just fire up your Python program from the command line after editing it. And that, really, gives us a clue as to the only sane conclusion of the IDE debate: it’s the same as the programming language debate. There are tools (languages and IDEs) and there are jobs. Good programmers pick the best tool for the job, and for a compiled language as verbose as Java, an IDE arguably makes things faster. For Python, on the other hand, it’s not essential (IMO), but it depends on the tastes of the individual.

It’s amazing how many fewer afternoons I seem to spend hacking around on my servers these days. Perhaps I got a life; I certainly got a full-time job. I have however sorted a few long-standing bits and pieces out today…

dnorth.net is now available over IPv6

As are its various satelite sites and www.saintcolumbas.org. Sorry, no, there is no bouncing logo to reward those of you viewing them via such.

A backup system that doesn’t Totally Suck™

I’ve finally retired my creaking “run a shell script to rsync them onto my laptop when I remember (i.e. every six months)” manaul backup system in favour of an encrypted LVM partition on my home server, and rdiff-backup to make nice incremental backups of everything on a nightly basis. The instructions on how to do it are all out there on the interweb, and it’s not too difficult, fortunately. I’m a bit disappointed that backupninja doesn’t support remote rdiff-backup, but I guess I should submit a patch if it bothers me that much…meanwhile, my wrapper script seems to work just fine.

After months of hard work, refactoring, blood, sweat, tears, and improvement, South 0.7 is ready.This release has been probably the biggest internal change in South since it first started. We've removed a lot of old code, and significantly refactored the migration-creating code to make it more extensible (as well as not being mostly in one 3000 line file) There are also a few new user-facing changes: New command names.The old startmigration was getting a little overloaded and wasn't too well-named, so it's been split into schemamigration and datamigration commands. There's also a new graphmigrations command, for those with complex dependency sets. No more issues with missing defaults. South realises you need a default for NOT NULL columns now, and prompts you if you forget to specify one as you're making the migration, instead of dying when you try to apply it. Django 1.2 and MultiDB support. As well as supporting the latest and greatest incarnation of the universe's best web framework, 0.7 also has some limited MultiDB support, in the form of a --database option (that works like the option of the same name on the new syncdb command in Django). Custom fields are no longer magically used. Instead, you now have to actually tell South about them. It's an ease-of-use regression, which I'm not too keen on - but it's needed to stop things mysteriously failing in your migrations a few weeks down the line, and there's a whole new tutorial section on it, as well as plenty of reference documentation (and IRC and the mailing list are always around to help). If you want more information, read the new long-form release notes, which goes into a bit more detail on the various changes, and mentions a few I haven't included here. There'll be a lull in development after the release as I rest, and then work will begin on the next release - 1.0*. Which features are going to appear have not yet been decided, but it's highly likely column rename support and a fixturemigration command will be among them. Anyway, why are you waiting? Go install South 0.7 now! * South has an octal numbering system.

It's that time of the year again, when a new South release rears its well-refactored, database-independent head.If you've been following the development of South at all, you'll know that 0.7 has been quite a while in the making. I've been busy working away at a big code refactor, along with Simon Law, who has gained the title of "Knowing Too Much About Migrations". However, there are some user-facing changes as well, some quite significant. While my release notes go into some detail on what's new, the major changes include: New command names.The old startmigration was getting a little overloaded and wasn't too well-named, so it's been split into schemamigration and datamigration commands. No more issues with missing defaults. South realises you need a default for NOT NULL columns now, and prompts you if you forget to specify one as you're making the migration, instead of dying when you try to apply it. Django 1.2 and MultiDB support. As well as supporting the latest and greatest incarnation of the universe's best web framework, 0.7 also has some limited MultiDB support, in the form of a --database option (that works like the option of the same name on the new syncdb command in Django). Custom fields are no longer magically used. Instead, you now have to actually tell South about them. We're working on docs to make this a bit easier - it's an ease-of-use regression, which I'm not too keen on - but it's needed to stop things mysteriously failing in your migrations a few weeks down the line. As I said above, read the new long-form release notes for much more on the release. If the release candidate proves to have sufficiently few bugs that I can fix them in time, 0.7 will be released next Thursday, the 25^th March. So, please, install the release candidate and try it out!

It’s been over a year since I deployed Django in production, and I wasn’t looking forward to it. Last time, I had a lot of trouble with mod_python, sessions and decimal objects refuising to pickle.

Thankfully, all this really seems to have grown up in the last year – mod_wsgi is now the recommended way of deploying Django in production, and following the mod_wsgi django instructions, I was in business in 20 minutes. No fuss, no mess, no drama, and best of all, using daemon mode, no noticeable performance hit when serving static files and PHP off the same Apache installation. The ability to run the django project as its own unprivileged user when using daemon mode is also real handy.

2010 will be a bad year for IPv4 – this is exactly why I designated native IPv6 support as fundamental, not merely a “nice-to-have”, when setting up Splice last July. Hats of to Bytemark for supplying IPv6 with their hosting. I’m sure the fact that it’s excluded from their SLA is something that won’t be the case in eighteen months’ time, and meanwhile, I and the rest of the crew are using it quite happily.

Footnote: those of you reaching for the comments button to sarcastically remark that this website appears to not be available over IPv6 will be pleased to hear that I intend to fix this in the immediate future.

My e-mail still isn’t as shiny as I’d like. In particular, my use of Exim Filters to sort incoming mail into folders lacks the ability to mark messages as read (although it’s still miles ahead of the dreaded Procmail). This would be handy for high-traffic mailing lists which I don’t have time to read on a daily basis, but which I find it hard to ignore the “unread” icon next to the folders for.

One day, I should probably move to using the Dovecot LDA and its sieve implementation, which supports the “imap4flags” extension, thus allowing marking messages as read, making them turn purple in Thunderbird, and all sorts of other cool stuff. Sadly, life (or this afternoon) is too short.

In the meantime, I’ve solved the problem in the usual way I deal with life’s imperfections: gratuitious Python – http://pastebin.org/77199 run from a crojob every five minutes.

(Disclaimer: letting scratty little bits of Python anywhere near something as important as your e-mail is probably a Very Bad Idea.)

Earlier this year, I hacked together a stateless RSS reader called Bogroll.

It’s been doing sterling service for me at news.dnorth.net ever since. Today, I’ve sorted out a 0.2 release with the following improvements:

• Now caches etags/Last-Modified headers to avoid fetching a feed if it hasn’t changed since last time (thank you, Mark Pilgrim, for chapter 14 of Dive Into Python 3, which reminded me to be a good citizen in this regard). I was pleased to discover that the Universal Feed Parser it’s built on top of already supports gzip and deflate compression to save bandwidth.
• Now supports just one category per feed, because having articles appear in several categories just seems wrong to me
• Each category now really does contain the most recent X articles from the relevant feeds, because I’ve fixed the severely broken sort-by-date logic

A fair bit of refactoring has gone on under the hood, and the code now looks a bit more like an app and less like a ten-minute bodge. The next round will involve getting some proper unit tests in place, and possibly AJAX magic to load the articles lazily on the page.

You can download the 0.2 zip, or get the latest version from subversion if you like to live dangerously. The cool kids all seem to be using Git or Mercurial these days, but I haven’t found the need (or overcome the inertia) yet.

Enjoy. Feedback welcome to the usual address.

So the turkey is (at least partly) eaten, the mince pies are disappearing fast, the wrapping paper has been picked up off the floor and the presents played with. 2009 is done.

It’s hard to say what I’ll remember 2009 most for, because it’s been such a packed year for me. Perhaps I’ll remeber it as the last of my three happy years at Magdalen, the year I finished my degree, the year I set up a hosting co-operative with six friends, the year I moved away from home, the year I got a job or the year I took over as joint church treasurer.

It wasn’t a perfect year, of course – my personal TODO list is still 39 lines long – but never mind, I’m fairly sure there’s another year just around the corner. One item a week. What could possibly go wrong?

Happy New Year to all 3.5 of my readers (and all those following along via Facebook; do come and read the website this is all automatically imported from), and I look forward to seeing as many of you as possible in 2010.

During the planning of our Christmas card at work this year, a mad idea came up. Do we ignore mad ideas? No, we tackle them head-on.

The idea was to build an internet-controlled snow machine - you'd hit the button on the website, and watch a member of the Torchbox team get pelted with snow. When we first came up with the idea, we dismissed it as being "too complex", but after a while, we came around.

Cue three days of frantic development and phoning round to get the parts. I'll be posting a full build article, with all our source code, once we're done.

The brief summary is that we have a Django app which handles rate-limiting of snow, and tracking who has clicked the button, which then communicates with our snow machine using Artnet and DMX. Ustream is used to stream the video back to the internet.

Still, I imagine you want to see it in action, so head over to snow.torchbox.com, and have a go. It's only online 10 - 5 UK time, and only until Tuesday (we can't fill our office up with paper snow forever), but it's still very good fun, even if you don't know any of us!

Let’s be honest about this. Looking back, I should have known better. Nevertheless, as I describe the problem that ate far too many hours of my weekend, judge for yourself whether I was entirely to blame for How It Went…

The problem

A friend of mine has a laptop. It’s about five years old, and it runs Windows XP. This means, inevitably, that it’s a mess. My personal metric of measuring how rodgered a machine is by the number of icons in its system tray gave it a ten, and that’s pretty nasty. Nevertheless, until last Friday, there was nothing wrong with it that an uninstall fest followed by a defrag wouldn’t have fixed.

Enter the Internet Man

Last Friday, a chap called round to set up some broadband for my friend, who’d previously been on dial-up. I’m not going to name the ISP concerned, since it gives me more freedom to say nasty things about them, but suffice to say that they’re big enough that they really should have done better.

Their engineer fixed the simple(ish) problem with wiring which was stopping the broadband from working, and what he should have done at that point was connect the laptop to the router (less than six inches away on the same desk) using the supplied ethernet cable, and left. What he actually did was shove the supplied CD in the drive, which helpfully installed a few hundred megabytes of crapware onto the machine, then hooked it up to the broadband via wireless. Then he left.

Despite the above totally unnecessary shoving of stuff onto it, the laptop struggled manfully on (system tray count now up to 12) and seemed superficially fine.

Enter Dragon Naturally Speaking

My friend makes extensive use of Dragon NaturallySpeaking, a voice-recognition product which seems to knock the socks off everything else on the market when it comes to actually coping with different accents. And it was here that the problem first manifested itself: trying to use Dragon to voice-control Internet Explorer caused it to crash with an error message along the lines of the one described in this Microsoft knowledgebase article.

Enter the sucker, stage left

At this point, I fetched up, and agreed to see if I could fix the problem. Although the above KB article looked ideal, being the first hit on Google when I exercised the too-useful-to-be-documented Windows feature of “Ctrl-C copies the text of the active dialog box to the clipboard”, the hotfix it supplies claimed to be already present in service pack 3 of Windows XP. Just about the only virtue of this laptop was that it was fully up to date on patches and service packs, so what now?

Don’t press that button

Being rather short on ideas at this point, I decided to fire up the nearest thing broken Win32 boxes have to a magic bullet, namely System Restore. The machine refused to roll back to any of the restore points at first, but restarting into safe mode fixed that, and it was soon rolled back to the Friday, at a time before the problem occured.

Unfortunately, Dragon now seemed completely broken, giving the error message described in this support article. And no, of course there were no backups, I hadn’t taken one before I started, and my friend is no different to most of the non-geeky people I know in not backing up, except for dragging his holiday snaps onto CD once every six months.

To their credit (and they’re about the only players in this story to be awarded any), Nuance’s suggestion in the article of how to manually restore the user files for Dragon did work, after I rolled back the fatal system restore [or rather, didn't, because it didn't seem to have made a pre-restore restore point. Fortunately picking one from the Saturday seemed to work].

So, after two hours feeling my friend’s anxiety at the thought of having to retrain the speech recog from scratch – not a pleasant accompinement to the sick, swoopy feeling we get when we know we’ve just permanently erased some irreplaceable data – we were back at square one with the original problem. One last shot in  the dark, disabling the ISP’s nasty extensions to IE, seemed to fix the issue.

So who do we blame here? Laptop vendors, for selling machines so laden with rubbish before they even leave the factory that DLL hell seems assured the minute they meet with real life usage? Or, we could blame the idiots who seem to write the nasty unsigned drivers for most hardware on the market. We can definitely blame the ISP’s engineer for installing the crapware, but perhaps he’d been trained to, and anyway, why do ISPs think we need a CD full of crap to supplement the TCP/IP standard that’s been around for several decades? Is it really asking too much of Johnny User to plug in a cable or enter some Wifi passwords in to the applet that’s sodding well supplied with Windows, thus making the poorly writtten replacements from laptop manufacturers and ISPs alike completely superfluous? We could also blame Microsoft for making system restore not clever enough to cope with software like Dragon. Or possibly blame Nuance for not fixing or documenting what has apparently been a known incompatability for several versions of Dragon*.

Certainly, we can blame me. I clearly need to have “I will not agree to even slightly ‘fix’ someone else’s computer without taking a full disk-image of it first” tattooed across my forehead. I also clearly need to reimmerse myself in the happy world of properly written software which I’m lucky enough to earn a living in and try to forget the horrors of the last 48 hours.

* The most useful reference I could find on Google was this page. The Nuance KB doesn’t mention system restore. Then again, perhaps their customers simply don’t know this is the cause of the issue, or don’t get round to reporting it. It’s not like I have.

I’ve long been of the opinion that Firefox and Thunderbird are a bit like democracy: aclaimed far and wide as major achievements and bastions of a civilised society, but actually, honestly, a bit crap in many ways. Sadly, we’re stuck with all three until someone manages to come up with some compelling alternatives.

If you’re reaching for your e-mail client at this point to tell me I’m being unduly harsh, look me in the eye and tell me you think the way Firefox cheerfully caches DNS lookups and ignores such things as TTLs is a good idea. Or, even harder, give me one good reason why Thunderbird doesn’t check all IMAP folders for new messages by default. You can’t; in both cases it’s a disgrace.

All this being the case, I wasn’t what you’d call hopeful when, last Sunday, I had to write some e-mails on a train. And to do so, I needed to refer to some other e-mails in my Inbox. Since said train lacked anything as useful as a wireless internet service*, I’d need some sort of offline IMAP facility**.

As I bashed Thunderbird offline IMAP into Google, I was expecting a half-baked plugin at best, and “can’t be done” at worst. What I was actually very pleasantly surprised to find is that this functionality is built into Thunderbird.

What astonished me even more is that it actually worked. Faultlessly. So perhaps democracy can be salvaged after all.

* And, let’s be honest, because I still haven’t got organised and bought a phone with internet capabilities.

** Nobody uses POP3 in the twenty-first century, right?

In my neverending quest to save the time of those using RDBMSen, South 0.6.2 is released.

It mostly contains bugfixes, but there's now support for extending introspection to third-party apps without having to edit their source.

I'll probably start working full steam on South 0.7 soon - there's a refactor of the migration engine (mostly done), of startmigration (not started), and a few changes to command names to make them nicer (the old ones will still work), and to the behaviour of default values.

I'm hoping 0.7 will be the last release before 1.0, at which point I will claim I've been working in an octal version numbering sequence all along.

Dear lazyweb, does anyone else find that Spotify mysteriously hangs at the “logging in” stage when run under Wine? This suddenly happend on both my Linux boxes after it had been working fine for weeks.

Various posts on the Ubuntu forums suggested firewall issues, but nothing had changed, and Spotify continue to work on Windows on both machines.

In the end, I applied the nuclear fix of

$ mv ~/.wine ~/.wine-with-broken-spotify

and reinstalling Spotify from scratch, which worked. Might not be so handy for anyone with other apps under Wine, though.

In my opinion, there's just not enough Subversion servers on the internet.

That's an opinion you'll probably never hear me express - I much rather everyone move over to a DVCS (*ahem* mercurial) - but Subversion has, some would say quite rightly, earned its place as the dominant VCS for nearly every part of the IT community (apart from big, lumbering, financial companies).

Recently, I've been slowly switching away from subversion for my personal projects into Mercurial. This, in itself, a painless process, but with my public projects - particularly South - this has the unintended effect of slightly alienating some of my users. Most are alright with adapting - after all, it's quickly becoming the norm to use a DVCS - but there's still a few people left out.

One of the particular issues I have is with svn:externals. A lot of apps - including some we have at work - rely on svn:externals to pull in external dependencies into a libs folder along with the project itself. Externals is one of the few features of subversion that I thought was pretty much perfect, and it was sad to see my move to Mercurial break it.

This week, at DjangoCon, Chris Wanstrath did a nice talk on DVCSen. One of the questions led him onto hg-git - the awesome git backend plugin for Mercurial, that the GitHub guys wrote - and how they first investigated the idea of an svn gateway to expose their repositories transparently to subversion users. From what I gathered, subversion's wire format proved too tricky to deal with, and so they turned elsewhere.

However, the idea of a subversion gateway intrigued me greatly. What better way to transparently serve South so people can still pull it using svn:externals while still developing in Mercurial? With that in mind, I started looking into Subversion's wire protocols and their delta format (svndiff) sometime yesterday afternoon.

After some digging, reading an obscure academic paper and liberal application of Wireshark, I grew confident enough that I could at least implement something. A day later, and I'd like to present the very first version of what I'm calling Heechee (if you get the pun, ten nerd points).

Heechee is a transparent mercurial-as-subversion gateway. It serves a Mercurial repository as a Subversion WebDAV-based repository. It's still in the early stages, but at the moment it will serve its own mercurial repository to subversion in such a way that you can check out the repository, and update to various revisions within it.

You can check it out at BitBucket. It's pretty alpha code, and make sure you have the dependencies mentioned in the README, but it works, which greatly surprises me. I plan to much improve the code to support more 'advanced' features, like being able to do more than checkout and update, as well as exposing tags and branches correctly. There's even the chance I'll stick Git support in, when I've had a play with Dulwich.

Still, hopefully I'll soon have it so it can serve South's code via svn, and thus restore harmony to my land of VCS users.

So, please, take the code and have a play with it. It's not formally licensed and released yet, but it will probably be Apache licensed. Feedback and suggestions for features, as well as how to linearise trees into svn histories, much welcomed!

South 0.6.1 is now available for public consumption; it fixes quite a few bugs.

If you want to see what's been fixed, have a gander at the Release Notes - there's a few significant bugs that have been fixed, including problems with SQLite, GeoDjango, and M2M fields.

Sorry for the short but sweet post - I'm in the process of writing my talk for DjangoCon! I'm giving a talk about South, with both a good grounding for those who haven't grokked migrations yet, as well as some tips on more advanced usage. Come watch it on Tuesday morning!

It is with great pleasure (and a measurable sense of relief) that I announce the release of South 0.6, a new release bringing quite a few new features, although most of them aren't immediately obvious.

Perhaps the biggest change is the move from parsing source files to get the field definitions (ugh, I know) to introspecting the fields to get their definitions. As well as being more stable and making my skin crawl less, this also means that dynamically-created models and fields are now fully supported.

South still falls back on the parser module for custom fields if it has to - since it doesn't know how to introspect them - but we encourage you to implement a South-readable interface for custom fields; see our wiki page explaining how to do that.

The other noticeable change is that the migrate command is much zippier, thanks mostly to the hard work and lazy-evaluation-patches of Juan and Simon from Akoha. Note that the SQL for migrations is no longer printed out as they're running any more - it was getting a bit verbose - but you can re-enable it with --verbosity=2. You can also completely shut South up using --verbosity=0, if you desire.

For easier conversion, there's a new one-shot convert_to_south command - take care to read the wiki page on that, however, as the caveat is that you must run a migrate --fake on any other computers running the same codebase. We're going to make that a bit more intuitive for the next release.

The release notes contain the rest of the changes - some big, some small - and remember, we've switched to Mercurial for revision control, so if you're still running off Subversion please change your checkouts.

After a long few months, during which I've increased my recommendations to run off trunk, South 0.6 is almost here.

This release contains a whole bunch of new features - more of which will be written up on the Release Notes page soon - but in the meantime, I've decided to adopt a more usual release process.

This means two things: First, I'm releasing a release candidate of South 0.6 for general testing and final critical-bug reporting; please see the Download page and follow the instructions to check out the 'stableish' tag, or if you already have a checkout please 'hg pull' and 'hg up' it.

The Git mirror is also, for once, up to date, and there's also a tar.gz available for those who don't fancy either DVCS.

The second change is that I'll actually maintain a stable branch from now on as well, much like Django. This means there will be a 0.6 branch for any fixes that are easy or trivial to backport, and in the unlikely case someone should manage to conceive of a security vulnerability in a migrations library fixes for that will go in too. This should hopefully mean you and your production servers can track a real release number and not miss out on bugfixes.

Note that a few of the planned features haven't made it into this release in time, but I felt I had to get something out there. My much-vaunted fixture migration didn't make it in, but introspection, the major feature, did.

Still, please, test the release! If things look good, I'll officially tag and release 0.6 early next week.

Just installed Ubuntu Jaunty fresh onto a new ext4 partition on my Advent 4211 netbook. Am delighted to report that WiFi, suspend and hibernate all work flawlessly out of the box, so I can finally stop using Windows on this machine.

It’s amazing what sort of shrinkage – both in size and price – you can miss if you’re not buying in a particular market. Yesterday, out for lunch with some friends, one of them showed off his latest toy, a 2GB USB drive. Which, as you can see even in these amateurish photos, is similar in size to an SD card:

It’s called the Tuff ‘n’ Tiny, apparently, and if you live in the UK, Ryman are selling the 2GB version online for £5.99 and two for a tenner on the high street.

I can also confirm that mine has partitioned quite happily into two FAT32 partitions, and, with prodding, boots Linux off one of them [the second. I had to use usb-creator, followed by install-mbr to get the bootloader to work]. I don’t think I’ll be following John’s suggestion of RAIDing two together just yet, but who knows…

Just over 18 months ago, I signed up for a 256slice from Slicehost to host this website, my email, etc. Later this week, I shall be shutting the machine down and cancelling my account with them.

For the record, this has nothing to do with the level of service I’ve received from them – I’ve always found their support team quick to respond and helpful, and their articles site and wiki are both very handy.

However, a combination of fluctuations of the pound against the dollar, a surge in demand for RAM by my applications and sites, and my getting increasingly fed up with transatlantic ping times of 130ms meant the machine was becoming unfit for purpose and overloaded.

Thus, I have now found a reasonably cheap way to bring “my stuff” home to hosting in the UK. About which, I will be writing more shortly!

So goodbye, Slicehost, and keep up the good work. I’d certainly recommend you to anyone who lives in America and needs a VM.

And Ubuntu gets seven out of ten

Last week, it was clear my parents’ ageing XP box needed some attention – in fact, let’s be honest, it had been crying out for a zap-and-reload since last year.

This time, though, I was determined. With my sister safely moved off onto her new laptop, there should be no reason not to migrate the box to Ubuntu. The only things in use on it were Firefox, Thunderbird and Openoffice…so how hard could it be?

Five days on, I’m in a position to report that the answer is “harder than it should have been” – here’s what happened:

Problem 1: Audio CD autoplay

Dad wanted the machine to start playing his CDs automatically when he put them in, much like Windows does. Seemed perfectly reasonable to me, as I blithely assured him it’d be a matter of a few mouse clicks.

It wasn’t.

Much Googling and swearing later, this hack (fifth comment down) seems to do the necessary, though I had to bump the sleep interval to six seconds.

Problem 2: No matter how close KPatience comes, it’s not “real FreeCell”

…largely because, despite generating the same deals from the same numbers as the Windows version, it counts every loss of the same game number as a loss in the stats, whereas the Microsoft one counts them as one loss.

Fortunately, freecell.exe runs fine under wine, so copy it across and save yourself the bother.

In conclusion

Other than the above minor niggles, all seems well, and hopefully the machine should now do a few more years’ trouble-free duty.

So, as the title of this post says, Windows XP has now been deposed as my OS of choice for “real people”. Whoever would’a thought it?

Update, 18 July 2009

I forgot to say that probably the most impressive feature of the switch was that Ubuntu flawlessly detected and printed a test page on our trusty old Epson Stylus C62. That’s the sort of slickness we like to see.

It's about time I posted an update on what's going on with South development, so here it is.

Firstly, there hasn't been much work done in the last few weeks, due to exams and there being a Ball to help run. Now that's over, development will crack down. 0.6 is going to focus on the following new features:

• Less parsing, more introspection. The default way of getting fields' definitions will be to introspect them; the implementation of this is pretty much complete already, and can be found in the 'noparsing' Mercurial branch. Custom fields (which cannot be introspected withour prior knowledge) have two options; provide a south_field_triple method to tell us their definition (preferred way), or as a fallback the parser will be used. This feature solves many issues, including dynamic models, and also doesn't lose the ability to work with most custom fields right from the start.
• Reverse dependencies. Just brought up by one of the Ella developers, this is a good idea that should have been in South a while back. You'll be able to specify 'this other migration depends on us' as well as 'we depend on that other migration' - this helps pluggable apps around a common core.
• Fixture migration. Since I know this is going to bite me in the ass at some point, I'll be rolling in the fixture-migration someone mentioned at EuroDjangoCon (uses the test runner to make a temporary database, loads an old fixture set, an old schema, migrates, dumps the new fixture). I haven't fully tested this yet, so it might not make it for reasons of impossibility, but I'm reasonably confident.
• Signals. There will be post-migrate and pre-migrate signals added to help developers add in custom code that needs to be run after migrations or before migrations (the latter helps people trying to carry out a deployable app rename, for example)

As well as those new features, there's also around 25 bugs that need fixing, and will be tackled soon, hopefully. I aim to get a beta release of these out in the next month, especially in the light of South's allegedly high penetration rate.

The last piece of news is that I'm moving my main development efforts to Mercurial (and specifically BitBucket), and only pushing mostly-stable trunk phases to Subversion. There's a github mirror of whatever hg-git deems to be main development, but that's for convenience only, and because I can do it in one command.

I guess I might as well park these here as anywhere else:

Some hilarity courtesy of bash.org: #693046, #9214, #105199, #626249, #610952

Funny source code comments – “throw up”, snigger. Nothing like Java-based humour.

Batteries Feel Included: 307 – it’s a bit strange, but it made me nearly fall off my chair laughing

A genius article from 1995 explaining the trouble with the internet.

This post brought to you by the department of “I’ve finished finals and have absolutely naff all to do for the next week”.

I had a lot of interesting chats with people about South last week at EuroDjangoCon, and several eyebrows have been raised at me both parsing models and then storing their definitions as dicts.

While this is a design that is partially born of backwards compatibility, it was mostly a conscious design choice I had to make, and so I'd like to now describe why I made it.

Firstly, for those who aren't aware, South's magic autodetection and auto-writing-migration code is based off of a file called 'modelsparser.py', which does what it says on the tin; it opens models.py, parses it, and extracts definitions directly.

Now, my first response to this is also 'ew', but there's a reason for it: custom fields. If it wasn't for those, introspection of the existing fields would work just fine, given a long list of special cases (I in fact have a file called modelsinspector.py sitting around that does just that - if you're interested, it's in the 'noparsing' branch on BitBucket. It doesn't do much, but it proves the concept).

However, with custom fields in the game, you have two options: hope they follow Django conventions and try to introspect common options, or put the onus on custom field authors to add a hook that makes it spit out a definition. While I would obviously choose the second one (possibly added to some of the first), it simply doesn't provide the drop-in compatability I want South to provide.

However, as Alex Gaynor and several others have surmised, I could just pickle everything and stick that at the bottom. No introspection needed, no weird namespace errors because the models dictionary is simply a collection of eval-able strings.

However, I don't like pickle (the Python kind; the delicious food kind, on the other hand, is more than welcome in my household). The security issues don't matter here - after all, we're passing raw strings to eval instead - but it has two major features I don't like. Firstly, it's not at all human-readable - and while I don't imagine people editing their migrations by hand, I like to be able to see what is going on, and I like the fact that South still doesn't depend on you using startmigration - a design choice I'm very proud of.

Secondly, I've been burned by pickles in the past. If you remove a module that a pickled thing uses, pickle will crash and burn, and if you've not done them right, you won't know quite where the error happened. (For the purposes of argument, of course, I'm assuming I'd do it right, so this isn't really a point. Still...)

There's yet another option (see, my job isn't easy) - custom-pickling, which Alex suggested last Wednesday; it consists of roughly freezing the __dict__ of the field. This has some advantages - it's more human-readable - but you get the errors similar to my current approach (where datetime.datetime wasn't unfreezing if people didn't have datetime imported).

My main gripe, with both this and pickle, is that the internals of fields aren't guaranteed to be consistent between releases of Django, just the external API, and so South uses a method that lets it use the external API - in my mind, this is less hassle, since that's the thing that's really not going to change very much (and even if it does, you can easily see what South is trying to call, and run that in a shell yourself to see where the problem lies).

So, that's why models are frozen as a massive lump of dictionaries and triples. As with many things, it probably wouldn't be necessary if models had been done slightly differently (so there was a method that always spat out a nice, portable way to reincarnate a field), but with no reason for that previously, I would have been thoroughly impressed by the core developers' foresight if it existed.

As for parsing versus introspection... I'm still divided. Parsing always has, and still feels, wrong, but introspection might just be less reliable. If you're interested, go peek at the branch I mentioned above, and see what you make of it.

And now, I will stop blathering and return to my small hole at the base of the mighty Django Mountain, where I am attempting to fit square pegs into round holes. Good day.

Quick tip for anyone like me who’s foolish enough to cram Exim4, Spamassasin, Apache with Wordpress running on it, and a pile of other stuff on to a server with 256MB of RAM: Install one of the Wordpress caching plugins. Otherwise, your server is liable to start swapping frantically if you get a dozen or so hits on one of the Wordpress-powered sites.

Indeed, it was possible to nuke jasper just by mashing the F5 key whilst sitting on the front page of this site.

You can now sign in to leave comments here (yes, dear readers, both of you!) – using an OpenID if you have one, and in related news, I’ve got m’self a shiny new OpenID to go and do likewise on other people’s sites.

For those interested, the slides from my migrations talk at EuroDjangoCon are up. I believe the videos will be around soon.

As I write, the sun is up, the bells are ringing, and Oxford is dragging itself out of bed for another day. It’s slightly sobering to think that I’ve just listened to the college choir for my last May morning as an undergraduate…hopefully not my last time from inside college, though – time to start making friends with lower year groups.

I’m going to miss Magdalen. But meanwhile, eight weeks still to enjoy!

a.k.a. “How to query for all objects not in a given subclass”

I ran across this problem today – do please leave a comment if you know of a better way to solve it than what follows.

You have two Django models, one of which inherits from the other, e.g.:

class Order(models.Model):
    # some fields

class DiningOrder(Order):
    # some more fields

So, how do I write a query (using the Django ORM) which returns all the Orders which are not DiningOrders?

Apparently, this does the trick:

>>> Order.objects.all().count()
762L
>>> DiningOrder.objects.all().count()
93L
>>> Order.objects.exclude(id__in=[d.id for d in DiningOrder.objects.all()]).count()
669L

I'm pleased to announce the release of South 0.5; this has been a long and exciting release, with many new features...

• ORM Freezing, a new feature inspired by Migratory, which lets you freeze the historical state of your current models into a migration so you can then use them via an ORM as if your models.py file existed at that point in time. There's a good description of how exactly this is used in part 3 of the new tutorial.
• Automatic change detection; using the frozen model states, South can now diff your current models.py file against how it looked last time you made a migration, and write a migration to make up the difference. This essentially allows django-evolution style hinting, and means you'll only have to manually write more advanced migrations. It's discussed more in part two of the new tutorial.
• A much more robust models.py parser, using the Python parser module. It's a dirty secret that South automatically makes migrations by directly examining your models' source code; however, the new parsing engine hits 99% of all Django use cases, and has provisions for custom hooks so more wacky fields (such as django-denorm's decorated-function-based-fields) can tell South their equivalent definitions.
• A move into a south/ subdirectory, to fit with Python packaging guidelines, and make South more friendly with environments such as buildout and pip.

I've been very excited about the new features for a while, as those who have talked to me will tell you, and thanks to those people who used the trunk releases and submitted a great set of bug reports, it's finally ready for general consumption!

Additionally, for those going to EuroDjangoCon, don't miss my talk on migrations next Monday; as well as South and its new features, I'll be covering and comparing django-evolution and a few other migration solutions.

Happy migrating!

My main problem with RSS readers is that if I go away for a few days and come back, I don’t really have time to catch up with all the posts which came up whilst I was away. What I needed, therefore, was a stateless reader which would just show the newest 10 entries or so.

Add in category filtering and a web interface – such as it is – and you get Bogroll, which I’ve just released under the GPL. You can grab the 0.1 release, check out the code from subversion, and have a nosey at my installation.

Incidentally, if anyone can think of a better name for it, do get in touch.

Stopping overnight in Reading last weekend, with a complete lack of reading material, I picked up an actual dead-tree copy of PC Pro magazine for the first time in years. I’ve read it on and off since discovering three years’ worth of back issues on a shelf at the back of my school library – the output of Jon Honeyball, in particular, is usually good for a laugh – followed, more often than not, by some serious thinking about the issues raised.

The column that most caught my eye this time, though, was one which explained the Rebel Simcard, which appears to offer a way of running an iPhone 3G on the network of your choice, without having to risk bricking the phone or voiding the warranty.

That’s good, but unfortulately, it’s still not good enough for me to send the £400 or so I have earmarked for a smartphone in Apple’s direction. It still doesn’t support tethering, the inability to run a SIP client in the background would be a huge annoyance, and developing for it would require the purchase of a £68 SDK (OK) and a £1000+ Mac to run it on (not OK, and neither is having to distribute code through the app store).

All of the above are things which the phone is technically capable of doing, but has been prevented from for various reasons. Meanwhile, my personal preference for keyboards – even small ones – over touchscreens which I struggle to operate with my medium-sized fingers, is another negative of the iPhone.

At the moment, I’m hanging on to the money and pinning my hopes on the Palm Pre, which should arrive in the UK later this year. If it’s as good as the hype says it is, supports tethering, and can be purchased on its own without an expensive contract I don’t need or want, it should fit the bill very nicely.

Let’s be honest, shall we: if you’re reading this, you’re probably the sort of person who’s been where I was last week: on the recieving end of a phone call from a friend/relative/friend of a friend/friend’s relative, enquiring if you could help them with a “computer problem”…

In this case, it all seemed pretty simple: the chap on the other end of the phone was rightly concerned that his second-hand Windows XP box had come without any antivirus software installed.

“No problem”, I blithely replied, “just install the free edition of AVG“.

And therein began the difficulty: no matter how carefully I spelled it out, the chap insisted that visiting the URL for AVG’s site resulted in a 404 error. Other sites were visible, but not that one. Emailing the link produced the same result, and so, thoroughly puzzled, I agreed to a site visit the next day.

Sure enough, from both IE and Firefox, the machine would not display pages from avg.com. Further experimentation revealed that most of the other antivirus vendors’ sites were similarly blocked, along with microsoft.com.

This rang faint bells with me – viruses have been blocking such sites for years – and full of confidence that I was going to be able to pull off a quick fix and look like a genius, I pulled up the C:\Windows\System32\Drivers\etc\hosts file, expecting to find a bunch of domains being redirected to 127.0.0.1. What I actually found was nothing which shouldn’t have been there.

So then I broke out the usual tools: Spybot, Rootkit Revealer, a manual installation of AVG via disk from another machine, HijackThis … several hours of the machine wrongly being pronounced clean later, I remembered reading that Microsoft’s OncCare sometimes managed to disinfect machines other products didn’t.

I downloaded the free trial (via another machine as the Microsoft site was being blocked) and fired it up. Its first quick scan found nothing, but it also recommended installing Internet Explorer 7. I think it must have been the running of the Microsoft malicious software removal tool during the IE install which finally disinfected the machine to the point where a full OneCare scan found and removed the last of what it proclaimed to be the Conficker worm.

More Windows-savvy readers might well be saying at this point that I should have recognised the Conficker symptoms earlier. The truth is, though, that after nearly three years using Linux on my machines, it’s starting to grate to have to find my way around Windows and fix knackered installs thereof, and I’ve been cheerfully avoiding articles about the latest threats in the Windows world, smug in the knowledge that it’s not my problem any more. .. until the phone rings.

It turns out that the aspect of the infection which had me confused the most – the blocking of the AV sites – is achieved by hooking into the relevant windows API calls and tampering with their behaviour. The question has to be asked, though – why the hell is it possible for code running as a limited user – or, come to that, any user – to intercept API calls like this?

Asside from the still-woeful state of its security, Windows really does feel prehistoric to me now – no central mechanism for installing and updating software, leading to every app having its own updater (launched at startup for maximum slowdown of the machine, of course), no decent command-line, and all the nonsense of antivirus, antispyware and anti-everything-else.

It must be ten years since I started getting the phone calls from people with broken machines, and yet it’s still happening. I saw the future of home computing last week, and it was a broken Windows box. Time to think the unthinkable and start installing Ubuntu as the standard remedy?

Have you ever felt the burning need to ride aimlessly aimfully around the London Underground? Well, then I have good news for you.

My new in-development project is Mornington Square (holding page only so far!) - it's a adaptation of the classic game of Mornington Crescent, a game universally enjoyed by young and old.

The key part of the adaptation is that, unlike other online versions, this one is played by riding around the real underground. A GPS phone (and a location broker like Fire Eagle) updates the server with where you are (this does mean climbing out of stations each time you make a move).

The rough gameplay outline goes as follows (if you want just the technical nuggets, skip a bit):

First, contestants start either in the same location, or in predetermined equidistant-in-time-from-the-destination locations. They're all heading towards the same place - say, Mornington Crescent.

Before each move, the server analyses each player's location and assigns them a pseudorandom set of rules they must follow for the next move. Rules limit the lines you can travel on - so, for example, if you're told that the Occidental Grouping Restriction applies, you can't use Aldgate or Aldgate East. (The rules all have explanations for new players, and a map of possible moves, including restrictions imposed by rules, is shown)

The choice isn't random; the server will try several tens of sets of rules and select the one it think looks most appropriate (which is usually the one blocking any short routes - it knows.)

Then, each player looks at what moves they're allowed (you're always only allowed to move on one line at a time, so you have to choose an appropriate next stop), and gives the server a time estimate, and then they start moving.

Players must get to their next stop in the time they allocated; if they are late, they suffer a 5-minute time penalty. If they are early, they must wait until the end of the time they originally allocated - this encourages good time estimation!

The whole process repeats until someone makes it to the destination and their time runs out (or their penalty period does) - then they're the winner.

You're required to submit your location at the end of every move, to ensure you're in the right place. One possible measure for stopping location cheating is to ask the person to stand next to a local landmark you know the position of, but this shouldn't be a problem early on.

The server also has an overview page showing all the players, their current moves, and generally useful information.

So, it sounds good, but it actually works, as well! So far the rule-calculation code and map drawing code is all done, so it's just the location and timing left before there's a fully-functional beta, which I shall hopefully be demoing at BarCampLondon6 (possibly by actually playing it that evening).

The site is built using Django, of course, along with my usual use of Cairo for drawing the custom Underground maps for each player's move (they're geographical, and not the proper Underground style yet, however).

Rules are defined by exclusion functions - a rule can exclude any route, set of station names, or stations matching a regular expression. An A* algorithm does the route planning, scoring each ruleset based on the shortest route to each destination, allowing more time for interchanges between lines (I have basically invented a small travel planner).

I am perhaps too excited at the prospect of an excuse to go around riding the underground, and emerging from the surface, only to find that the server has decided that on the next move, wild lifts are high, and so I can't go through Covent Garden, but it'll all come out in testing (which I shall be doing in secret before the next DJUGL).

Work on South has been going full steam ahead lately, with some good results.

Two major features that I've been wanting to implement for ages are now in SVN trunk, and due for imminent release with South 0.5. They are:

• ORM Freezing: Much like the strangely similar feature of Migratory (which is what inspired this), South now has the ability to 'freeze' models to make them accessible inside the migration, so you can write data migrations more easily. Things like fake models needed for the other ends of ForeignKeys (when creating tables and the like) are also now implemented using the same system, to make things cleaner.

• Automatic change detection: Before this was impossible, but now, if you freeze all of an app's models into a migration, South can compare that with the current state and automatically write a migration for the changes that have occurred. There's a setting to automatically freeze the app into every migration to make this more usable, too.

I'm very excited about these new features; they should make the next release of South one not to be missed. There's also other changes under the hood, as well, including a new, much more robust models.py parser. Documentation for these features is up already, and the tutorial will be extended (given a second part) soon, in order to cover them - they're features we want everyone using, since they'll save you time, and, hopefully stress.

If you feel like testing the new features before the next release, just check out trunk and have a play. Your feedback is also more than welcome on our new South mailing list, which you should also have a look at.

I've always been somewhat envious when seeing the various sites with nice maps of crime, prices and other things for non-UK regions.

However, over the past week and a bit me and Martin have been changing that. WhoseTurf rolled out of the coding mill some time last week, and it's been improved a lot since, in our quest for useful (and fun) online geographical statistics for the UK.

GeoDjango has been on my list of things to play with properly for a long time, and so it was good fun to get my teeth into it. It works surprisingly well - the admin interface was really helpful when collating all our various area info (and even drawing some districts by hand - not a fun task when your source map is on a different projection).

At the core of WhoseTurf is a load of 'reviews'. Each review has a certain 'fanout' - the radius in which it can affect the ratings - and, accordingly, most of the work in our custom tile renderer is correctly interpolating between the correct review points and their values.

Having circles affecting values rather than the actual discrete shapes of the various UK districts makes for some slight inaccuracy, but for most variables this makes more sense than drawing it discretely (crime doesn't suddenly drop when you change county, for example). Council tax is one of the few exceptions to this, and so accordingly is drawn discretely.

Another clever trick we use is that, because the heatmaps we serve up just consist of tiles of large squares, we simply serve 40x40px images and let the browser scale them up to 256x256. This cuts down on rendering time and bandwidth, makes our cache (Varnish) happier, and also lets me crow on about how clever it is (even though it's reasonably obvious).

It turns out there's some things GeoDjango just can't do - like this massive query that makes correctly-shaped districts out of the various output areas we have (they don't fill the whole space, nor line up perfectly, so otherwise you get holes or hair-thin 'cracks'):

SELECT st_collect(cpoly) FROM (
        SELECT st_simplify(st_makepolygon(st_exteriorring(
        (st_dump(upoly)).geom )), 0.001) AS cpoly FROM (
            SELECT st_union(spoly) AS upoly FROM (
                SELECT st_snaptogrid(mpoly, 0.01) AS spoly
                FROM core_outputarea
                WHERE id LIKE '""" + code + """%%' AND
                st_isvalid(mpoly) = true)
                AS a
            ) AS b
        )
    AS c;

The other small problem was Firefox not having enough JS stack space to load in the border of the Highlands (it has lots of very silly little islands and glens), but I'm willing to let that one slide.

We'll probably be releasing the code behind the thing as open source; I doubt any of it is _that_ transferable, but there is all the code in there to render standard (spherical mercator) tiles using cairo and/or PIL, and if you too want to show lots of heatmaps, and perhaps kill your server, it could be for you.

One last side note: OpenLayers is, unfortunately, still not really useable. Google maps is a) smaller and b) more efficient (drags more smoothly, scroll zoom, although I did have to write our own layers widget). On the flip side, OpenStreetMap is getting more than useable for a site like ours now.

Next time one of your application or websites goes wrong and the users shout at you, take comfort in the knowledge that even Google sometimes Get It Wrong…

(retrieved at 14:44 GMT on Saturday 31 January 2009)

Edit: As Michael points out in the comments, Oxford users can avoid the mess below by simply disabling hybrid mode.

So there I am, firing up the Advent to read my email between lectures when…

$ sudo vpn-connect oxford
vpnc-connect was built without openssl: Can't do hybrid or cert mode.

Gah? The VPN client has always worked in the past, but following the upgrade to Ubuntu Intrepid, it doesn’t. A quick search lead to a solution which can be summarised as:

$ cd /tmp
$ apt-get source vpnc
$ cd vpnc-0.5.1r275
$ $EDITOR Makefile

Uncomment the lines beginning with OPENSSL
Read this and despair for the future of open source software

$ sudo apt-get build-dep vpnc
$ dpkg-buildpkg

Wonder why it all collapses in a heap
Search and find this.

$ apt-get install libssl-dev
$ dpkg-buildpkg
$ cd ..
$ sudo dpkg -i vpnc_0.5.1r275-1ubuntu1_i386.deb

Wonder why this has changed between Hardy and Intrepid
Seriously consider going back to Windows on this machine.

Sigh.

Welcome to the new aeracode.org, now with a bit more blue, and added Unantistalker™ technology.

As well as a graphical tweak, I've also added a small map. In case you haven't figured it out, it's (supposedly) my current location; the actual age of the data is shown further down the page in the sidebar.

This is accomplished by having a Python script on my phone which uploads its current location to my server every time it moves; if there's a GPS available, it'll use that for exact coordinates, otherwise it just submits the cell ID and I attempt to find it in the OpenCellID database (I submit cell locations to it if a GPS is around, so in theory I need only walk a route once to locate all the cells, then I can use just cell IDs from there on).

While it may make it easier for the stalkers (that I probably wish I was famous enough to have) to find me, I'm not terribly worried; I can always add a bit of random jitter (and besides, location by cell ID alone isn't terribly accurate).

When everything's working, I'll post my code up here; until then, if you're sad enough, you can probably watch me walk to and from lectures. Alternatively, you may want to go get on with your life.

Today I took delivery of one USB Panic Button; I've been wanting one of these for a very long time, and so I'm pleased to finally have one. I don't necessarily have a need to panic a lot, but there's a lot of times a big red button is just so satisfying (launch events, test runs, and so on).

Unfortunately, it comes with a Windows driver CD and the only Linux script I could find was a Perl script. Since I'm not the greatest fan of Perl, and I wanted to try some USB code, I've made my own driver for them.

You can download it here; just edit the COMMAND = line at the top of the script to change what it runs. Alternatively, you can import the file for use as a module; just make a new instance of PanicButton and call .pressed() to see if it was pressed recently (yes, you have to poll it).

Also note it has to be run as root, to prise off the kernel's HID module.

Finally, here's one more picture of it in situ.

After far too many months of quiet feature development and bug fixing, we're happy to finally announce South 0.4, which has a whole host of new features to satiate your every migration need.

The list of new features alone really is quite long; you can see it at http://south.aeracode.org/wiki/ReleaseNotes/0.4, but I'll go through a few of them quickly:

Firstly, there's alter_column, which you just pass a new Field instance into and it will deal with changing the column type, nullity and default value for you. --add-field is also a new option to startmigration, which simplifies the common use case of adding one or more new fields as a migration.

Model inheritance and ManyToMany relationships are both now fully supported, and there's a glut of database support with MySQL being fully matured and SQLite almost there. Even SQL Server has a tiny amount of creation-only support.

So, go check it out, and let us know what you think. If you're in or near London, as well, come along to DJUGL; I'll be talking about South there.

Just a heads-up to any LastGraph users to let you know that, soon, I'll be implementing one-week graph storage expiry (but probably not for premium users). There's now 128GB of PDFs being stored on S3, and it's probably time I started not throwing money away.

What this means is, from around the middle of next week, possibly earlier, when you make a graph, the download links you get will only be valid for seven days. After that, the files on the other end will vanish into the aether, so if you want to show them off you should probably download them and host them on your own blog.

Still, I'd like to give S3 credit; when LastGraph originally launched, I had a server with 10GB of disk space and only a bit more bandwidth. These days, LastGraph is pulling 44GB of transfer a month at S3 alone, and I have 4.4GB of stored user data (and that's compressed, children), so I'd like to thank the guys at State51 and Last.fm for the resources they've donated towards it, and the Premium users for just meeting the S3 bills so far with their signups!

I’ve just finished a Python script to shuffle the names for Secret Santa and email each participant the name of the person they’re giving to. Thus if you’re not organized enough to get everybody together to draw names out of a hat, you can still do it without one person having to know who’s giving to whom.

Sadly, I didn’t get this done in time to give it a try last week, but hopefully somebody will find a use for it.

You can grab the latest version of the script here.

It depends on pytz, because in order to generate emails which stand half a chance of getting through spam filters, it needs to be able to generate proper timezone-aware dates. Using the raw email sending libraries in Python was a bit of an eye-opener - turns out Django takes care of a lot of twiddly bits behind the scenes, which you don’t notice until you have to do it all by hand.

Spotted this on Farcebook Facebook just now…

I’d laugh, but somebody probably sued them when the free pig didn’t turn up.

It generally takes a lot to get me to really like a piece of hardware, thoroughly. For example, my experience of Fujitsu-Siemens' customer service was nothing but excellent, but the only reason I had so much contact with them was that my laptop's inverter went three times (to their credit, it's been fine for the past two years or so now, and my brother is still happily using it, but that laptop still sheds screws at the rate of around one a year).

However, a bit over a year ago, I decided to invest in some bluetooth headphones, and picked a pair of Jabra BT620s (warning: flash music thingy) semi-randomly from the field of available devices. A year later, and I can honestly say they're one of the better things I've bought this decade.

They're well-designed, and have good build quality; they also take and recieve phone calls, with pretty decent sound quality; they also turn into a USB Audio device (headphones and mic) when plugged in and turned on, a feature that is nearly impossible to find out about, and more importantly, last for absolutely ages. I've left these things on in my bag for over a week sometimes, and they'll still be happily flashing blue (they do that when switched on, but you can disable it, else you look a little weird) when I get them out.

I only wish the software support for A2DP was as good. Under Linux, it will work via ALSA, but PulseAudio still segfaults whenever you take it near them, Maemo has okay support, but often stutters slightly and the driver uses a lot of CPU, and Symbian is the only thing that seems to deal with them rather well (except for the occasional confusion when it decides it really should start playing the music over the tinny phone speakers at full volume, instead).

My only complaint is that they're a little large and heavy, but given that this seems to be needed for the batteries and speakers, it's fine, and I have big ears, anyway.

I also, in a strange way, enjoy having both earphones and an alarm clock with mini-USB ports and upgradable firmware; it's obviously the future. In a sick way, I can't wait for the day when I need to spend 30 minutes reflashing my shower so there's more than a millimetre between the "scalding hot" and "arctic cold" settings.

It's been a while since I last talked about South, so it's about time I did a short update, I think. There's been a fair few things going on since last time, which was, err, a long time ago.

Firstly, we now have SQLite support (apart from column renames/alters/deletes; I'll be implementing those with the appropriate workarounds soon). This was one of the features I most wanted to roll in, since SQLite is quite widely used for quick development. SQL Server support is also on the way soon!

Also in the news, TWiD's latest episode has some interesting poll results for schema migrations; of the respondents, over half are using one of the premade tools, and of those about 2/5 are using South - good news for the migration camp overall, and a surprisingly high amount for South, considering I have a limited picture of who's using it.

Other news includes startmigration getting support for multi-table inheritance (finally), some nice new work on indexes to flesh out the database API, and plenty of bugfixes and small tweaks to make things flow more smoothly.

Also in the pipeline is the possibility of adding an option to send post_syncdb signals at the end; several people have asked for this, and for some coding styles it will greatly reduce headaches about functions running too early in the migrations.

Finally, 0.4 will probably be released at some point soon, so keep a look out!

I've always not been one for phones; for years, I had a (relatively) old Nokia 6100. It does the job of phone calls and SMS messaging very well, and lasts absolutely ages on a single charge.

At the beginning of the year, though, I wanted 3G connectivity for my PDA and laptop (yes, I have a whole range of portable devices, and every one has its use, although not necessarily simultaneously), and so liberated a Nokia 6280 from eBay. The 6280 was pretty good - long battery life, a WAP browser, and that all-important connectivity. However, when its screen broke last week, and replacement LCDs were not too cheap, I went back to eBay with two rules: it had to be under £60, and should fit the two 6280 batteries and umpteen small Nokia chargers I have (the Internet Tablets use them too).

As you can imagine, this doesn't leave a great deal of choice. In fact, excluding the 6280, it leaves pretty much only the N73. Having experienced the 6280's wonderful affinity for dust (Nokia didn't quite get the sliding mechanism dustproof - it gets inside the screen...), I decided to go for an N73 and see what they were like; after all, how bad can crazy selection criteria be?

Not too bad at all, it turns out. The N73 is a pretty neat little device, but more importantly, it runs Symbian. I had presumed all the older smartphones were like the 6280, but I was pleased to discover that there's more S60 apps out there than I can shake a stick at, and that the version the N73 runs is still the most current, compatability-wise.

Three days later, and I have a whole set of things, including Google Maps, which gets impressive position accuracy from cell info alone, the QR code reader I would have needed a year ago at Barcamp Brighton, my Google Calendar synced up, an application that changes to and from silent mode depending on my location (using cell towers) and calendar events (i.e. anything marked as a lecture), and more importantly, the Python.

In short, it's lovely. The next step is getting a permanent, cheap data plan; Orange will sell you one for £5 a week (on Pay As You Go; I'm not shelling out hundreds on a contract), but Three have the suspiciously good offer of £5/month for 2GB, which is better than some ADSL packages I've seen, so I'm seriously considering migrating. The only thing holding me back is Orange's excellent customer service, and the fact I have a deal from them which means I get five free texts a day, forever, for £12 that I paid back in 2001 or so. I'll have to see if I can keep that under a new number.

I’ve not been sleeping all that well lately. And let’s be honest, this probably has something to do with being in the middle of a term at Oxford University, with a lot on my mind and far too much to see to.

However.

I’ve noticed recently that my room contains a surprising number of light sources, which I suspect can’t be helping me get to sleep. Reading from left to right, we have:

• Light leaking under the door from the staircase (where the lights are apparently permanently on)
• Blue power LED flashing on my laptop
• Green power/charging LED on my other laptop
• Huge red light on the multiplug under my desk
• Flashing lights on my hub

I’ve therefore come up with a cunning plan:

• Block the base of the door with an old pillow
• Prop a book over the laptop light
• Put the other laptop in a drawer
• Tie an old sock round the multiplug
• Shove the hub into a cardboard box with holes for the wires

Whether or not this helps remains to be seen. But here’s a plea to manufacturers: please think twice about the brightness and location of your LEDs, especially if there’s a chance someone will be trying to sleep in the same room as the device they’re on.

I finally got round to putting Linux on my Advent 4211 yesterday. There are various ways and means of doing this; the ones which worked for me were:

• Use the installation image optimised for small screens
• Follow these instructions to make a bootable USB drive from the image (note: it’s ‘cat’, not ‘zcat’ as the ubuntu image is not zipped. I know they’re Debian instructions, but they seem to work)
• Use the script given in this forum post (run as root) to get the WiFi working.

The end results are really rather impressive; here’s a pretty picture:

Here’s one for all you progammers out there: what happens to the values of x and y if you type the following in, say, Python?


>>> x = 5
>>> y = 6
>>> x,y = y,x


I’d always half-thought, without really considering it, that you’d end up with x=6 and y=6, i.e. that the above was syntactic sugar for:


>>> x = y
>>> y = x


Actually, though, it turns out…


>>> x=5
>>> y=6
>>> x,y=y,x
>>> print x,y
6 5


Do you write Python (not very well?).

Ever wondered why your calls to os.execvp() and its ilk don’t work as expected?

The answer may be that you’re doing this:

os.execvp('ls', ['/home/fred'])

When you actually mean this:

os.execvp('ls', ['ls', '/home/fred'])

As ever, reading the Python docs (help(os.execvp)) is usually better than randomly googling for an answer.