Planet JCR

It's about time I posted an update on what's going on with South development, so here it is.

Firstly, there hasn't been much work done in the last few weeks, due to exams and there being a Ball to help run. Now that's over, development will crack down. 0.6 is going to focus on the following new features:

• Less parsing, more introspection. The default way of getting fields' definitions will be to introspect them; the implementation of this is pretty much complete already, and can be found in the 'noparsing' Mercurial branch. Custom fields (which cannot be introspected withour prior knowledge) have two options; provide a south_field_triple method to tell us their definition (preferred way), or as a fallback the parser will be used. This feature solves many issues, including dynamic models, and also doesn't lose the ability to work with most custom fields right from the start.
• Reverse dependencies. Just brought up by one of the Ella developers, this is a good idea that should have been in South a while back. You'll be able to specify 'this other migration depends on us' as well as 'we depend on that other migration' - this helps pluggable apps around a common core.
• Fixture migration. Since I know this is going to bite me in the ass at some point, I'll be rolling in the fixture-migration someone mentioned at EuroDjangoCon (uses the test runner to make a temporary database, loads an old fixture set, an old schema, migrates, dumps the new fixture). I haven't fully tested this yet, so it might not make it for reasons of impossibility, but I'm reasonably confident.
• Signals. There will be post-migrate and pre-migrate signals added to help developers add in custom code that needs to be run after migrations or before migrations (the latter helps people trying to carry out a deployable app rename, for example)

As well as those new features, there's also around 25 bugs that need fixing, and will be tackled soon, hopefully. I aim to get a beta release of these out in the next month, especially in the light of South's allegedly high penetration rate.

The last piece of news is that I'm moving my main development efforts to Mercurial (and specifically BitBucket), and only pushing mostly-stable trunk phases to Subversion. There's a github mirror of whatever hg-git deems to be main development, but that's for convenience only, and because I can do it in one command.

I guess I might as well park these here as anywhere else:

Some hilarity courtesy of bash.org: #693046, #9214, #105199, #626249, #610952

Funny source code comments – “throw up”, snigger. Nothing like Java-based humour.

Batteries Feel Included: 307 – it’s a bit strange, but it made me nearly fall off my chair laughing

A genius article from 1995 explaining the trouble with the internet.

This post brought to you by the department of “I’ve finished finals and have absolutely naff all to do for the next week”.

I had a lot of interesting chats with people about South last week at EuroDjangoCon, and several eyebrows have been raised at me both parsing models and then storing their definitions as dicts.

While this is a design that is partially born of backwards compatibility, it was mostly a conscious design choice I had to make, and so I'd like to now describe why I made it.

Firstly, for those who aren't aware, South's magic autodetection and auto-writing-migration code is based off of a file called 'modelsparser.py', which does what it says on the tin; it opens models.py, parses it, and extracts definitions directly.

Now, my first response to this is also 'ew', but there's a reason for it: custom fields. If it wasn't for those, introspection of the existing fields would work just fine, given a long list of special cases (I in fact have a file called modelsinspector.py sitting around that does just that - if you're interested, it's in the 'noparsing' branch on BitBucket. It doesn't do much, but it proves the concept).

However, with custom fields in the game, you have two options: hope they follow Django conventions and try to introspect common options, or put the onus on custom field authors to add a hook that makes it spit out a definition. While I would obviously choose the second one (possibly added to some of the first), it simply doesn't provide the drop-in compatability I want South to provide.

However, as Alex Gaynor and several others have surmised, I could just pickle everything and stick that at the bottom. No introspection needed, no weird namespace errors because the models dictionary is simply a collection of eval-able strings.

However, I don't like pickle (the Python kind; the delicious food kind, on the other hand, is more than welcome in my household). The security issues don't matter here - after all, we're passing raw strings to eval instead - but it has two major features I don't like. Firstly, it's not at all human-readable - and while I don't imagine people editing their migrations by hand, I like to be able to see what is going on, and I like the fact that South still doesn't depend on you using startmigration - a design choice I'm very proud of.

Secondly, I've been burned by pickles in the past. If you remove a module that a pickled thing uses, pickle will crash and burn, and if you've not done them right, you won't know quite where the error happened. (For the purposes of argument, of course, I'm assuming I'd do it right, so this isn't really a point. Still...)

There's yet another option (see, my job isn't easy) - custom-pickling, which Alex suggested last Wednesday; it consists of roughly freezing the __dict__ of the field. This has some advantages - it's more human-readable - but you get the errors similar to my current approach (where datetime.datetime wasn't unfreezing if people didn't have datetime imported).

My main gripe, with both this and pickle, is that the internals of fields aren't guaranteed to be consistent between releases of Django, just the external API, and so South uses a method that lets it use the external API - in my mind, this is less hassle, since that's the thing that's really not going to change very much (and even if it does, you can easily see what South is trying to call, and run that in a shell yourself to see where the problem lies).

So, that's why models are frozen as a massive lump of dictionaries and triples. As with many things, it probably wouldn't be necessary if models had been done slightly differently (so there was a method that always spat out a nice, portable way to reincarnate a field), but with no reason for that previously, I would have been thoroughly impressed by the core developers' foresight if it existed.

As for parsing versus introspection... I'm still divided. Parsing always has, and still feels, wrong, but introspection might just be less reliable. If you're interested, go peek at the branch I mentioned above, and see what you make of it.

And now, I will stop blathering and return to my small hole at the base of the mighty Django Mountain, where I am attempting to fit square pegs into round holes. Good day.

Quick tip for anyone like me who’s foolish enough to cram Exim4, Spamassasin, Apache with Wordpress running on it, and a pile of other stuff on to a server with 256MB of RAM: Install one of the Wordpress caching plugins. Otherwise, your server is liable to start swapping frantically if you get a dozen or so hits on one of the Wordpress-powered sites.

Indeed, it was possible to nuke jasper just by mashing the F5 key whilst sitting on the front page of this site.

You can now sign in to leave comments here (yes, dear readers, both of you!) – using an OpenID if you have one, and in related news, I’ve got m’self a shiny new OpenID to go and do likewise on other people’s sites.

For those interested, the slides from my migrations talk at EuroDjangoCon are up. I believe the videos will be around soon.

As I write, the sun is up, the bells are ringing, and Oxford is dragging itself out of bed for another day. It’s slightly sobering to think that I’ve just listened to the college choir for my last May morning as an undergraduate…hopefully not my last time from inside college, though – time to start making friends with lower year groups.

I’m going to miss Magdalen. But meanwhile, eight weeks still to enjoy!

a.k.a. “How to query for all objects not in a given subclass”

I ran across this problem today – do please leave a comment if you know of a better way to solve it than what follows.

You have two Django models, one of which inherits from the other, e.g.:

class Order(models.Model):
    # some fields

class DiningOrder(Order):
    # some more fields

So, how do I write a query (using the Django ORM) which returns all the Orders which are not DiningOrders?

Apparently, this does the trick:

>>> Order.objects.all().count()
762L
>>> DiningOrder.objects.all().count()
93L
>>> Order.objects.exclude(id__in=[d.id for d in DiningOrder.objects.all()]).count()
669L

I'm pleased to announce the release of South 0.5; this has been a long and exciting release, with many new features...

• ORM Freezing, a new feature inspired by Migratory, which lets you freeze the historical state of your current models into a migration so you can then use them via an ORM as if your models.py file existed at that point in time. There's a good description of how exactly this is used in part 3 of the new tutorial.
• Automatic change detection; using the frozen model states, South can now diff your current models.py file against how it looked last time you made a migration, and write a migration to make up the difference. This essentially allows django-evolution style hinting, and means you'll only have to manually write more advanced migrations. It's discussed more in part two of the new tutorial.
• A much more robust models.py parser, using the Python parser module. It's a dirty secret that South automatically makes migrations by directly examining your models' source code; however, the new parsing engine hits 99% of all Django use cases, and has provisions for custom hooks so more wacky fields (such as django-denorm's decorated-function-based-fields) can tell South their equivalent definitions.
• A move into a south/ subdirectory, to fit with Python packaging guidelines, and make South more friendly with environments such as buildout and pip.

I've been very excited about the new features for a while, as those who have talked to me will tell you, and thanks to those people who used the trunk releases and submitted a great set of bug reports, it's finally ready for general consumption!

Additionally, for those going to EuroDjangoCon, don't miss my talk on migrations next Monday; as well as South and its new features, I'll be covering and comparing django-evolution and a few other migration solutions.

Happy migrating!

My main problem with RSS readers is that if I go away for a few days and come back, I don’t really have time to catch up with all the posts which came up whilst I was away. What I needed, therefore, was a stateless reader which would just show the newest 10 entries or so.

Add in category filtering and a web interface – such as it is – and you get Bogroll, which I’ve just released under the GPL. You can grab the 0.1 release, check out the code from subversion, and have a nosey at my installation.

Incidentally, if anyone can think of a better name for it, do get in touch.

Stopping overnight in Reading last weekend, with a complete lack of reading material, I picked up an actual dead-tree copy of PC Pro magazine for the first time in years. I’ve read it on and off since discovering three years’ worth of back issues on a shelf at the back of my school library – the output of Jon Honeyball, in particular, is usually good for a laugh – followed, more often than not, by some serious thinking about the issues raised.

The column that most caught my eye this time, though, was one which explained the Rebel Simcard, which appears to offer a way of running an iPhone 3G on the network of your choice, without having to risk bricking the phone or voiding the warranty.

That’s good, but unfortulately, it’s still not good enough for me to send the £400 or so I have earmarked for a smartphone in Apple’s direction. It still doesn’t support tethering, the inability to run a SIP client in the background would be a huge annoyance, and developing for it would require the purchase of a £68 SDK (OK) and a £1000+ Mac to run it on (not OK, and neither is having to distribute code through the app store).

All of the above are things which the phone is technically capable of doing, but has been prevented from for various reasons. Meanwhile, my personal preference for keyboards – even small ones – over touchscreens which I struggle to operate with my medium-sized fingers, is another negative of the iPhone.

At the moment, I’m hanging on to the money and pinning my hopes on the Palm Pre, which should arrive in the UK later this year. If it’s as good as the hype says it is, supports tethering, and can be purchased on its own without an expensive contract I don’t need or want, it should fit the bill very nicely.

Let’s be honest, shall we: if you’re reading this, you’re probably the sort of person who’s been where I was last week: on the recieving end of a phone call from a friend/relative/friend of a friend/friend’s relative, enquiring if you could help them with a “computer problem”…

In this case, it all seemed pretty simple: the chap on the other end of the phone was rightly concerned that his second-hand Windows XP box had come without any antivirus software installed.

“No problem”, I blithely replied, “just install the free edition of AVG“.

And therein began the difficulty: no matter how carefully I spelled it out, the chap insisted that visiting the URL for AVG’s site resulted in a 404 error. Other sites were visible, but not that one. Emailing the link produced the same result, and so, thoroughly puzzled, I agreed to a site visit the next day.

Sure enough, from both IE and Firefox, the machine would not display pages from avg.com. Further experimentation revealed that most of the other antivirus vendors’ sites were similarly blocked, along with microsoft.com.

This rang faint bells with me – viruses have been blocking such sites for years – and full of confidence that I was going to be able to pull off a quick fix and look like a genius, I pulled up the C:\Windows\System32\Drivers\etc\hosts file, expecting to find a bunch of domains being redirected to 127.0.0.1. What I actually found was nothing which shouldn’t have been there.

So then I broke out the usual tools: Spybot, Rootkit Revealer, a manual installation of AVG via disk from another machine, HijackThis … several hours of the machine wrongly being pronounced clean later, I remembered reading that Microsoft’s OncCare sometimes managed to disinfect machines other products didn’t.

I downloaded the free trial (via another machine as the Microsoft site was being blocked) and fired it up. Its first quick scan found nothing, but it also recommended installing Internet Explorer 7. I think it must have been the running of the Microsoft malicious software removal tool during the IE install which finally disinfected the machine to the point where a full OneCare scan found and removed the last of what it proclaimed to be the Conficker worm.

More Windows-savvy readers might well be saying at this point that I should have recognised the Conficker symptoms earlier. The truth is, though, that after nearly three years using Linux on my machines, it’s starting to grate to have to find my way around Windows and fix knackered installs thereof, and I’ve been cheerfully avoiding articles about the latest threats in the Windows world, smug in the knowledge that it’s not my problem any more. .. until the phone rings.

It turns out that the aspect of the infection which had me confused the most – the blocking of the AV sites – is achieved by hooking into the relevant windows API calls and tampering with their behaviour. The question has to be asked, though – why the hell is it possible for code running as a limited user – or, come to that, any user – to intercept API calls like this?

Asside from the still-woeful state of its security, Windows really does feel prehistoric to me now – no central mechanism for installing and updating software, leading to every app having its own updater (launched at startup for maximum slowdown of the machine, of course), no decent command-line, and all the nonsense of antivirus, antispyware and anti-everything-else.

It must be ten years since I started getting the phone calls from people with broken machines, and yet it’s still happening. I saw the future of home computing last week, and it was a broken Windows box. Time to think the unthinkable and start installing Ubuntu as the standard remedy?

Have you ever felt the burning need to ride aimlessly aimfully around the London Underground? Well, then I have good news for you.

My new in-development project is Mornington Square (holding page only so far!) - it's a adaptation of the classic game of Mornington Crescent, a game universally enjoyed by young and old.

The key part of the adaptation is that, unlike other online versions, this one is played by riding around the real underground. A GPS phone (and a location broker like Fire Eagle) updates the server with where you are (this does mean climbing out of stations each time you make a move).

The rough gameplay outline goes as follows (if you want just the technical nuggets, skip a bit):

First, contestants start either in the same location, or in predetermined equidistant-in-time-from-the-destination locations. They're all heading towards the same place - say, Mornington Crescent.

Before each move, the server analyses each player's location and assigns them a pseudorandom set of rules they must follow for the next move. Rules limit the lines you can travel on - so, for example, if you're told that the Occidental Grouping Restriction applies, you can't use Aldgate or Aldgate East. (The rules all have explanations for new players, and a map of possible moves, including restrictions imposed by rules, is shown)

The choice isn't random; the server will try several tens of sets of rules and select the one it think looks most appropriate (which is usually the one blocking any short routes - it knows.)

Then, each player looks at what moves they're allowed (you're always only allowed to move on one line at a time, so you have to choose an appropriate next stop), and gives the server a time estimate, and then they start moving.

Players must get to their next stop in the time they allocated; if they are late, they suffer a 5-minute time penalty. If they are early, they must wait until the end of the time they originally allocated - this encourages good time estimation!

The whole process repeats until someone makes it to the destination and their time runs out (or their penalty period does) - then they're the winner.

You're required to submit your location at the end of every move, to ensure you're in the right place. One possible measure for stopping location cheating is to ask the person to stand next to a local landmark you know the position of, but this shouldn't be a problem early on.

The server also has an overview page showing all the players, their current moves, and generally useful information.

So, it sounds good, but it actually works, as well! So far the rule-calculation code and map drawing code is all done, so it's just the location and timing left before there's a fully-functional beta, which I shall hopefully be demoing at BarCampLondon6 (possibly by actually playing it that evening).

The site is built using Django, of course, along with my usual use of Cairo for drawing the custom Underground maps for each player's move (they're geographical, and not the proper Underground style yet, however).

Rules are defined by exclusion functions - a rule can exclude any route, set of station names, or stations matching a regular expression. An A* algorithm does the route planning, scoring each ruleset based on the shortest route to each destination, allowing more time for interchanges between lines (I have basically invented a small travel planner).

I am perhaps too excited at the prospect of an excuse to go around riding the underground, and emerging from the surface, only to find that the server has decided that on the next move, wild lifts are high, and so I can't go through Covent Garden, but it'll all come out in testing (which I shall be doing in secret before the next DJUGL).

Work on South has been going full steam ahead lately, with some good results.

Two major features that I've been wanting to implement for ages are now in SVN trunk, and due for imminent release with South 0.5. They are:

• ORM Freezing: Much like the strangely similar feature of Migratory (which is what inspired this), South now has the ability to 'freeze' models to make them accessible inside the migration, so you can write data migrations more easily. Things like fake models needed for the other ends of ForeignKeys (when creating tables and the like) are also now implemented using the same system, to make things cleaner.

• Automatic change detection: Before this was impossible, but now, if you freeze all of an app's models into a migration, South can compare that with the current state and automatically write a migration for the changes that have occurred. There's a setting to automatically freeze the app into every migration to make this more usable, too.

I'm very excited about these new features; they should make the next release of South one not to be missed. There's also other changes under the hood, as well, including a new, much more robust models.py parser. Documentation for these features is up already, and the tutorial will be extended (given a second part) soon, in order to cover them - they're features we want everyone using, since they'll save you time, and, hopefully stress.

If you feel like testing the new features before the next release, just check out trunk and have a play. Your feedback is also more than welcome on our new South mailing list, which you should also have a look at.

I've always been somewhat envious when seeing the various sites with nice maps of crime, prices and other things for non-UK regions.

However, over the past week and a bit me and Martin have been changing that. WhoseTurf rolled out of the coding mill some time last week, and it's been improved a lot since, in our quest for useful (and fun) online geographical statistics for the UK.

GeoDjango has been on my list of things to play with properly for a long time, and so it was good fun to get my teeth into it. It works surprisingly well - the admin interface was really helpful when collating all our various area info (and even drawing some districts by hand - not a fun task when your source map is on a different projection).

At the core of WhoseTurf is a load of 'reviews'. Each review has a certain 'fanout' - the radius in which it can affect the ratings - and, accordingly, most of the work in our custom tile renderer is correctly interpolating between the correct review points and their values.

Having circles affecting values rather than the actual discrete shapes of the various UK districts makes for some slight inaccuracy, but for most variables this makes more sense than drawing it discretely (crime doesn't suddenly drop when you change county, for example). Council tax is one of the few exceptions to this, and so accordingly is drawn discretely.

Another clever trick we use is that, because the heatmaps we serve up just consist of tiles of large squares, we simply serve 40x40px images and let the browser scale them up to 256x256. This cuts down on rendering time and bandwidth, makes our cache (Varnish) happier, and also lets me crow on about how clever it is (even though it's reasonably obvious).

It turns out there's some things GeoDjango just can't do - like this massive query that makes correctly-shaped districts out of the various output areas we have (they don't fill the whole space, nor line up perfectly, so otherwise you get holes or hair-thin 'cracks'):

SELECT st_collect(cpoly) FROM (
        SELECT st_simplify(st_makepolygon(st_exteriorring(
        (st_dump(upoly)).geom )), 0.001) AS cpoly FROM (
            SELECT st_union(spoly) AS upoly FROM (
                SELECT st_snaptogrid(mpoly, 0.01) AS spoly
                FROM core_outputarea
                WHERE id LIKE '""" + code + """%%' AND
                st_isvalid(mpoly) = true)
                AS a
            ) AS b
        )
    AS c;

The other small problem was Firefox not having enough JS stack space to load in the border of the Highlands (it has lots of very silly little islands and glens), but I'm willing to let that one slide.

We'll probably be releasing the code behind the thing as open source; I doubt any of it is _that_ transferable, but there is all the code in there to render standard (spherical mercator) tiles using cairo and/or PIL, and if you too want to show lots of heatmaps, and perhaps kill your server, it could be for you.

One last side note: OpenLayers is, unfortunately, still not really useable. Google maps is a) smaller and b) more efficient (drags more smoothly, scroll zoom, although I did have to write our own layers widget). On the flip side, OpenStreetMap is getting more than useable for a site like ours now.

Next time one of your application or websites goes wrong and the users shout at you, take comfort in the knowledge that even Google sometimes Get It Wrong…

Every site has 'may harm your computer' next to it, including Google itself. Nice.

(retrieved at 14:44 GMT on Saturday 31 January 2009)

Edit: As Michael points out in the comments, Oxford users can avoid the mess below by simply disabling hybrid mode.

So there I am, firing up the Advent to read my email between lectures when…

$ sudo vpn-connect oxford
vpnc-connect was built without openssl: Can't do hybrid or cert mode.

Gah? The VPN client has always worked in the past, but following the upgrade to Ubuntu Intrepid, it doesn’t. A quick search lead to a solution which can be summarised as:

$ cd /tmp
$ apt-get source vpnc
$ cd vpnc-0.5.1r275
$ $EDITOR Makefile

Uncomment the lines beginning with OPENSSL
Read this and despair for the future of open source software

$ sudo apt-get build-dep vpnc
$ dpkg-buildpkg

Wonder why it all collapses in a heap
Search and find this.

$ apt-get install libssl-dev
$ dpkg-buildpkg
$ cd ..
$ sudo dpkg -i vpnc_0.5.1r275-1ubuntu1_i386.deb

Wonder why this has changed between Hardy and Intrepid
Seriously consider going back to Windows on this machine.

Sigh.

Welcome to the new aeracode.org, now with a bit more blue, and added Unantistalker™ technology.

As well as a graphical tweak, I've also added a small map. In case you haven't figured it out, it's (supposedly) my current location; the actual age of the data is shown further down the page in the sidebar.

This is accomplished by having a Python script on my phone which uploads its current location to my server every time it moves; if there's a GPS available, it'll use that for exact coordinates, otherwise it just submits the cell ID and I attempt to find it in the OpenCellID database (I submit cell locations to it if a GPS is around, so in theory I need only walk a route once to locate all the cells, then I can use just cell IDs from there on).

While it may make it easier for the stalkers (that I probably wish I was famous enough to have) to find me, I'm not terribly worried; I can always add a bit of random jitter (and besides, location by cell ID alone isn't terribly accurate).

When everything's working, I'll post my code up here; until then, if you're sad enough, you can probably watch me walk to and from lectures. Alternatively, you may want to go get on with your life.

Today I took delivery of one USB Panic Button; I've been wanting one of these for a very long time, and so I'm pleased to finally have one. I don't necessarily have a need to panic a lot, but there's a lot of times a big red button is just so satisfying (launch events, test runs, and so on).

Unfortunately, it comes with a Windows driver CD and the only Linux script I could find was a Perl script. Since I'm not the greatest fan of Perl, and I wanted to try some USB code, I've made my own driver for them.

You can download it here; just edit the COMMAND = line at the top of the script to change what it runs. Alternatively, you can import the file for use as a module; just make a new instance of PanicButton and call .pressed() to see if it was pressed recently (yes, you have to poll it).

Also note it has to be run as root, to prise off the kernel's HID module.

Finally, here's one more picture of it in situ.

After far too many months of quiet feature development and bug fixing, we're happy to finally announce South 0.4, which has a whole host of new features to satiate your every migration need.

The list of new features alone really is quite long; you can see it at http://south.aeracode.org/wiki/ReleaseNotes/0.4, but I'll go through a few of them quickly:

Firstly, there's alter_column, which you just pass a new Field instance into and it will deal with changing the column type, nullity and default value for you. --add-field is also a new option to startmigration, which simplifies the common use case of adding one or more new fields as a migration.

Model inheritance and ManyToMany relationships are both now fully supported, and there's a glut of database support with MySQL being fully matured and SQLite almost there. Even SQL Server has a tiny amount of creation-only support.

So, go check it out, and let us know what you think. If you're in or near London, as well, come along to DJUGL; I'll be talking about South there.

Just a heads-up to any LastGraph users to let you know that, soon, I'll be implementing one-week graph storage expiry (but probably not for premium users). There's now 128GB of PDFs being stored on S3, and it's probably time I started not throwing money away.

What this means is, from around the middle of next week, possibly earlier, when you make a graph, the download links you get will only be valid for seven days. After that, the files on the other end will vanish into the aether, so if you want to show them off you should probably download them and host them on your own blog.

Still, I'd like to give S3 credit; when LastGraph originally launched, I had a server with 10GB of disk space and only a bit more bandwidth. These days, LastGraph is pulling 44GB of transfer a month at S3 alone, and I have 4.4GB of stored user data (and that's compressed, children), so I'd like to thank the guys at State51 and Last.fm for the resources they've donated towards it, and the Premium users for just meeting the S3 bills so far with their signups!

I’ve just finished a Python script to shuffle the names for Secret Santa and email each participant the name of the person they’re giving to. Thus if you’re not organized enough to get everybody together to draw names out of a hat, you can still do it without one person having to know who’s giving to whom.

Sadly, I didn’t get this done in time to give it a try last week, but hopefully somebody will find a use for it.

You can grab the latest version of the script here.

It depends on pytz, because in order to generate emails which stand half a chance of getting through spam filters, it needs to be able to generate proper timezone-aware dates. Using the raw email sending libraries in Python was a bit of an eye-opener - turns out Django takes care of a lot of twiddly bits behind the scenes, which you don’t notice until you have to do it all by hand.

Spotted this on Farcebook Facebook just now…

I’d laugh, but somebody probably sued them when the free pig didn’t turn up.

It generally takes a lot to get me to really like a piece of hardware, thoroughly. For example, my experience of Fujitsu-Siemens' customer service was nothing but excellent, but the only reason I had so much contact with them was that my laptop's inverter went three times (to their credit, it's been fine for the past two years or so now, and my brother is still happily using it, but that laptop still sheds screws at the rate of around one a year).

However, a bit over a year ago, I decided to invest in some bluetooth headphones, and picked a pair of Jabra BT620s (warning: flash music thingy) semi-randomly from the field of available devices. A year later, and I can honestly say they're one of the better things I've bought this decade.

They're well-designed, and have good build quality; they also take and recieve phone calls, with pretty decent sound quality; they also turn into a USB Audio device (headphones and mic) when plugged in and turned on, a feature that is nearly impossible to find out about, and more importantly, last for absolutely ages. I've left these things on in my bag for over a week sometimes, and they'll still be happily flashing blue (they do that when switched on, but you can disable it, else you look a little weird) when I get them out.

I only wish the software support for A2DP was as good. Under Linux, it will work via ALSA, but PulseAudio still segfaults whenever you take it near them, Maemo has okay support, but often stutters slightly and the driver uses a lot of CPU, and Symbian is the only thing that seems to deal with them rather well (except for the occasional confusion when it decides it really should start playing the music over the tinny phone speakers at full volume, instead).

My only complaint is that they're a little large and heavy, but given that this seems to be needed for the batteries and speakers, it's fine, and I have big ears, anyway.

I also, in a strange way, enjoy having both earphones and an alarm clock with mini-USB ports and upgradable firmware; it's obviously the future. In a sick way, I can't wait for the day when I need to spend 30 minutes reflashing my shower so there's more than a millimetre between the "scalding hot" and "arctic cold" settings.

It's been a while since I last talked about South, so it's about time I did a short update, I think. There's been a fair few things going on since last time, which was, err, a long time ago.

Firstly, we now have SQLite support (apart from column renames/alters/deletes; I'll be implementing those with the appropriate workarounds soon). This was one of the features I most wanted to roll in, since SQLite is quite widely used for quick development. SQL Server support is also on the way soon!

Also in the news, TWiD's latest episode has some interesting poll results for schema migrations; of the respondents, over half are using one of the premade tools, and of those about 2/5 are using South - good news for the migration camp overall, and a surprisingly high amount for South, considering I have a limited picture of who's using it.

Other news includes startmigration getting support for multi-table inheritance (finally), some nice new work on indexes to flesh out the database API, and plenty of bugfixes and small tweaks to make things flow more smoothly.

Also in the pipeline is the possibility of adding an option to send post_syncdb signals at the end; several people have asked for this, and for some coding styles it will greatly reduce headaches about functions running too early in the migrations.

Finally, 0.4 will probably be released at some point soon, so keep a look out!

I've always not been one for phones; for years, I had a (relatively) old Nokia 6100. It does the job of phone calls and SMS messaging very well, and lasts absolutely ages on a single charge.

At the beginning of the year, though, I wanted 3G connectivity for my PDA and laptop (yes, I have a whole range of portable devices, and every one has its use, although not necessarily simultaneously), and so liberated a Nokia 6280 from eBay. The 6280 was pretty good - long battery life, a WAP browser, and that all-important connectivity. However, when its screen broke last week, and replacement LCDs were not too cheap, I went back to eBay with two rules: it had to be under £60, and should fit the two 6280 batteries and umpteen small Nokia chargers I have (the Internet Tablets use them too).

As you can imagine, this doesn't leave a great deal of choice. In fact, excluding the 6280, it leaves pretty much only the N73. Having experienced the 6280's wonderful affinity for dust (Nokia didn't quite get the sliding mechanism dustproof - it gets inside the screen...), I decided to go for an N73 and see what they were like; after all, how bad can crazy selection criteria be?

Not too bad at all, it turns out. The N73 is a pretty neat little device, but more importantly, it runs Symbian. I had presumed all the older smartphones were like the 6280, but I was pleased to discover that there's more S60 apps out there than I can shake a stick at, and that the version the N73 runs is still the most current, compatability-wise.

Three days later, and I have a whole set of things, including Google Maps, which gets impressive position accuracy from cell info alone, the QR code reader I would have needed a year ago at Barcamp Brighton, my Google Calendar synced up, an application that changes to and from silent mode depending on my location (using cell towers) and calendar events (i.e. anything marked as a lecture), and more importantly, the Python.

In short, it's lovely. The next step is getting a permanent, cheap data plan; Orange will sell you one for £5 a week (on Pay As You Go; I'm not shelling out hundreds on a contract), but Three have the suspiciously good offer of £5/month for 2GB, which is better than some ADSL packages I've seen, so I'm seriously considering migrating. The only thing holding me back is Orange's excellent customer service, and the fact I have a deal from them which means I get five free texts a day, forever, for £12 that I paid back in 2001 or so. I'll have to see if I can keep that under a new number.

I’ve not been sleeping all that well lately. And let’s be honest, this probably has something to do with being in the middle of a term at Oxford University, with a lot on my mind and far too much to see to.

However.

I’ve noticed recently that my room contains a surprising number of light sources, which I suspect can’t be helping me get to sleep. Reading from left to right, we have:

• Light leaking under the door from the staircase (where the lights are apparently permanently on)
• Blue power LED flashing on my laptop
• Green power/charging LED on my other laptop
• Huge red light on the multiplug under my desk
• Flashing lights on my hub

I’ve therefore come up with a cunning plan:

• Block the base of the door with an old pillow
• Prop a book over the laptop light
• Put the other laptop in a drawer
• Tie an old sock round the multiplug
• Shove the hub into a cardboard box with holes for the wires

Whether or not this helps remains to be seen. But here’s a plea to manufacturers: please think twice about the brightness and location of your LEDs, especially if there’s a chance someone will be trying to sleep in the same room as the device they’re on.

I finally got round to putting Linux on my Advent 4211 yesterday. There are various ways and means of doing this; the ones which worked for me were:

• Use the installation image optimised for small screens
• Follow these instructions to make a bootable USB drive from the image (note: it’s ‘cat’, not ‘zcat’ as the ubuntu image is not zipped. I know they’re Debian instructions, but they seem to work)
• Use the script given in this forum post (run as root) to get the WiFi working.

The end results are really rather impressive; here’s a pretty picture:

Ubuntu Linux running on my Advent 4211

Here’s one for all you progammers out there: what happens to the values of x and y if you type the following in, say, Python?


>>> x = 5
>>> y = 6
>>> x,y = y,x


I’d always half-thought, without really considering it, that you’d end up with x=6 and y=6, i.e. that the above was syntactic sugar for:


>>> x = y
>>> y = x


Actually, though, it turns out…


>>> x=5
>>> y=6
>>> x,y=y,x
>>> print x,y
6 5


Do you write Python (not very well?).

Ever wondered why your calls to os.execvp() and its ilk don’t work as expected?

The answer may be that you’re doing this:

os.execvp('ls', ['/home/fred'])

When you actually mean this:

os.execvp('ls', ['ls', '/home/fred'])

As ever, reading the Python docs (help(os.execvp)) is usually better than randomly googling for an answer.

One of the topics that popped up repeatedly at DjangoCon last weekend was how bad purely normalised table structures are, and how denormalisation is good for many things, including making your database cry less.

To that end, during the gaps in PyConUK this weekend, I decided to see how easy it would be to write a new Django field that will automatically denormalise a field in a related table across to another model.

Let's take an example; imagine we have this kind of model:

class Picture(models.model):
    user = models.ForeignKey(User)
    user_username = DenormField(User, "username")
    image = models.ImageField(upload_to="foo")
    title = models.CharField(max_length=100)

The idea is that the user's username (which, let's face it, you're going to need almost every time you need the photo) is stored in the same table, saving you from doing that join to get it from users each time. While this may not seem like much, it can be useful, and besides, it was fun to write.

My DenormField simply listens to a trio of the relevant signals, and then updates this value whenever it detects that the related User object has been updated, so it only costs on write (which, let's face it, is much better than on every read). You can do this kind of copying in normal Django app code as well, but I think this solution is much cleaner. I would do, I guess, since I wrote it.

It's only really a little test exercise in writing Strange New Fields (and I wanted to see how much monkeypatching it would take - none, as it turns out, just some very nasty code tricks). You can get a copy of it if you feel like having a play (warning: only briefly tested; may attach captions to your cat).

There are not very many things that could tempt me to fly 10 hours twice for only a weekend, but DjangoCon was one of them, and I'm glad I went; it was one of the best weekends I've had for quite a while.

There were plenty of good talks; I can't mention them all here, since as we all know, bytes cost money, but in that vein I must at least mention Cal Henderson's talk 'Why I Hate Django' - a brilliantly done talk on the things Django's missing, as well as the fact we don't have a mascot (or do we?) and that we aren't smug enough.

Still, I enjoyed many of the other talks as well (what a shame it is not to be able to see them all in person; still, Google will hopefully release the videos soon), as well as our Schema Evolutions panel, which seems to point towards a bright future for Django and migrations. 

Talking of bright futures, judging by the final session with Adrian and Jacob, there's plenty of stuff to get working on for 1.1, some of which have me quite excited. Multiple database support is going to be one of those important ones, really, but then Malcolm seems to have snuck the bottom layer of it in already, which is quite handy.

All in all, an excellent weekend, and finally a chance to meet a subset of everyone involved in Django (and they really do come from many places), and I can now hardly wait to get over to Prague next year for the promised EuroDjangoCon. If you didn't make it, then I highly recommend watching the videos when Google release them (after editing them on their ridicously great AV system, no doubt).

The world of migrations in Django is definitely warming up now, and with that we're proud to announce the release of South 0.3, the 'intelligent django migrations app', available now from south.aeracode.org.

New in this release are the use of fields rather than dicts for specifying columns in migrations, so migrations now look cleaner, work with all the databases Django supports to a better extent, and even support custom fields.

Also new is dependencies between apps, for those situations where your forum app's Post depends on your accounts app's Profile; South will work out the right way to apply everything so foreign keys don't horribly break.

There's also a nice number of bugfixes and general tweaks Andy and I have been making as South moves towards being used in active development work. If you think you might be needing migrations, or even if you don't, give South a try!

(Also new is a very carefully worded "why we're best" page on our wiki. I'm trying not to step on people's toes, just say why we're using South rather than anything else, but I'm not sure how well that's working)

With the launch of its own site - south.aeracode.org - and Andy McCurdy jumping in to help out, South has been making some good progress. We've made a few backwards incompatable changes, which while annoying will hopefully make things much easier in future.

Firstly, the format of the create_table and add_column calls has been changed to use Fields instead of long dicts of parameters. As well as looking more Django-like, this allows us to reuse the SQL generating features Django already has for fields, as well as playing nicer with custom types.

In line with these changes, startmigration has also seen some changes; as well as generating things using Fields now, it can group models together in migrations (this proves useful for foreign keys referencing tables made after them).

We're still working on finishing this all off, and there'll hopefully be something stable to show off soon. Woop.

Also, if you're going to DjangoCon, may I recommend the Schema Evolutions panel; there'll be all manner of evolutiony and migrationary things discussed therein.

I bought myself an Advent 4211 last week (you might know it as the MSI Wind; they’re essentially the same thing). So far, it’s lived up to expectations; there are just a couple of things to say…

• Mine seems to have the decent Synaptics touchpad - read The Register’s review for the warning about the other one.
• You really need to use headphones with it if apps you’re listening to don’t have a very good sound level - inevitably in a device this size, the speakers aren’t great, especially at full volume.

How the Linux install on it goes is something I shall let y’all know shortly…

I’ve been aware for some time that my DNS isn’t quite as securely configured as I’d like. http://crashrecovery.org/named/ looks pretty good, but the two main issues bugging me were:

1. Anyone could do a ‘dig @ns.dnorth.net dnorth.net AXFR’ to retrieve a listing of all my DNS records - not great from a security point of view. This is a capability that should only be turned on for secondary DNS servers which need to fetch from the master.
2. The server would perform arbitrary lookups [for any domain] on request. This means it’s operating in ‘recursive mode’, which is a Bad Thing for various reasons.

The solutions were:

1. Add “allow-transfer { “slaves”; };” (without the double quotes) to the section of the configuration beginning “zone ‘dnorth.net’”. Then add a section defining the “slaves” access control list to be the local server, plus the secondaries: “acl slaves { 127.0.0.1; 123.45.67.89; }” replacing 123.45.67.89 by the IP address(es) of your secondary nameserver(s).
2. Add “recursion: no;” to the “options” section of the configuration.

Then restart the BIND9 service - on Debian, this is “/etc/init.d/bind9 restart”.

Health warning: Don’t do (2) above if you rely on your server to do its own DNS resolution - follow the crashrecovery tutorial above instead.

I did a lightning talk to CompSoc last night on whether 64bit is ready for use on the desktop - if you’re bored, you can download the slides and the second of my famous sketch and scanned graphs.

gq has been a necessary evil in my life for some time now. I need a graphical LDAP client for use on the CompSoc systems, but gq (to be fair, the versions of gq packaged for Ubuntu) seems to be very buggy, segfaulting all over the place if you try to do anything other than browse with it.

Last week, after upgrading to a 64bit version of Ubuntu for the first time, I finally ditched gq, after running into identical symptoms to this Debian bug.

The good news is that there’s an alternative that actually works: it’s called Luma.

Have fun.

The epic tale of how my new mail setup was born

My personal email has been on a rough ride over the years: from a reasonably nice (Microsoft based!) school email setup in 1999, which sadly got removed when Windows 98 was introduced, I went through three Hotmail accounts. I had a brief flirtation with GMail, but not being all that keen on the means of delivering advertising, I ended up back on Hotmail.

Registering dnorth.net last year at least ensured my email address would no longer change, but the technical capabilities of the two mail servers holding the mail still left much to be desired: flaky, unreliable spam scoring, no facilities for server-side filtering/sorting, sheer lack of customisability…

Last week, I finally did something about it. At my disposal was my VPS, running Debian Linux 4 (’etch’). On the wishlist were:

• Accurate server-side spam scoring with SMTP-time rejection of the most obvious spam
• Sender verification
• Sieve filters for server-side sorting into folders
• All mail stored on the server and accessible over secure IMAP

Thankfully, none of the above is too difficult: some pretty good instructions are Out There for most of it. The ones I used were:

• Exim4 on Debian with SpamAssassin, ClamAV, Virtual Domain Alias Files and Message Size Limits per Domain
• Exim + Sieve + Avelsieve Howto
• Handling mail for multiple virtual domains with exim4

Please remember, I am not responsible for the content of external sites (e.g., the links above), nor can I accept any responsibility for the consequences of acting on the points below…

I ran into a couple of issues:

• Permissions on the .sievesource files generated by AvelSieve - I needed to chmod g+w on /var/lib/squirrelmail/data and chown it to www-data:www-data in order to reach a state where Exim could read the file, and Avelsieve could write it.
• Exim4’s native sieve implementation only has the core features in it, not the extensions defined in RFC 3431. I dodged the issue by matching the number of *s in the X-Spam-Score header using string matching, rather than numeric checks on the X-Spam-Score.

All in all, though, it’s working a treat. Email perfection at last!

Twelve months ago, I installed Linux (Kubuntu) on my laptop.

Twelve minutes ago, I got the wireless card working with my WPA-PSK home network.

A combination of the progress made on drivers over the last year, and finding which of the squillions of instruction sets out there Actually Worked has finally got things going - rather than recount it all here, I’ll point owners of the HP Pavilion dv5157eu to this page, and advise everyone else as to how I found it: do a Google search for some combination of ‘ubuntu’, ‘wireless’ and your laptop’s exact model number.

Edit: My wireless happiness sadly didn’t survive a hibernate or a reboot. Turns out the line of the page which says do ’sudo echo ndiswrapper > /etc/module’ actually means, as far as I can tell, that you should do:

sudo -s
echo ndiswrapper >> /etc/modules

As ever, follow my advice at your own risk.

Musings on public transport, and a look round The Other University

So, full term ended yesterday, and I had to get over to Cambridge for a dinner. The X5 bus proved reasonable, both in terms of cost (£15 open return) and time, though the long stop to change drivers in Bedford lengthened an already long enough journey.

After a very pleasant afternoon being shown round Cambridge, I’ll have to begrudgingly admit to being impressed by it. King’s College Chapel, in particular, was spectacular. Even the weather wasn’t bad.

The train journey home from Oxford was tolerable, though prolonged by a needless half hour stop in Birmingham - it wasn’t so much the stop that annoyed me, but the fact that nobody bothered to tell us about it.

Yesterday only served to reaffirm my belief that public transport is mediocre at best, and ghastly at worst. Our rulers will have to try harder if they want to convince me not to use the driving licence lurking in my wallet, as soon as I can afford to…

A problem which has been bugging me for a while is my Konsole and Yakuake terminal sessions ceasing to respond to input for no apparent reason. The symptoms are always the same: the session seemingly stops responding to key presses, yet if I was running screen inside the session, reattaching it elsewhere reveals the characters from the missing key presses.

The answer turns out to be that hitting scroll lock locks the terminal so that the arrow keys scroll it rather than cycling through previous commands, and this also prevents key presses updating the display. My Yakuake hotkey is F12, which is next to scroll lock on my laptop.

So now you know.