Planet JCR

There’s an acid test that we as engineers should always subject our creations to: do they make life better for the end user? “Better” is perhaps quite difficult to quantify, but you can always approach the problem from the opposite direction and see if you’ve made things worse.

This is something British Gas’s man clearly failed to do when fixing my grandparents’ central heating recently. I don’t know the full details of the problem, but I do know that their thermostat was broken, so he installed a new one.

Fair enough, but it turns out that the iron march of progress has changed a thermostat from a knob with some numbers on it to something ‘smart’:

 

 

 

 

 

And how does the new wireless thermostat make life worse for my nonagenarian grandparents? Let us count the ways:

• Because it’s wireless, it has batteries in it which need replacing every so often. This is achieved by opening a flimsy plastic door on the bottom of the unit which is fiddly to access once it’s wall mounted, then scrabbling on the carpet as the batteries fall to earth. It also means the thermostat will mysteriously stop working once every n months until someone younger sorts it out for them, since there’s no way they’ll hear a low-battery beep or spot an indicator on the screen.
• Since it’s superglued to the wall just inches away from the hot water tank it controls, the only advantage of wirelessness is to save the drilling of one hole and the running of a six-inch bit of cabling – and even these could presumably have been avoided by replacing the original thermostat instead of leaving it screwed to the wall but not doing anything.
• Instead of reading the numbers round a knob, you see them on an LCD display which is not backlit and not very big, thus making it perfect for people with poor eyesight to see in a not-very-well-lit hallway.
• Pressing the middle of it resets it to a pre-programmed ‘preset temperature’ (“ideal for the poorly sighted”, the manual claims with no sense of irony) – an unnecessary recipe for confusion if you knock the middle by mistake
• It doesn’t go ‘click’ as it passes the current room temperature like an electromechanical thermostat would, so you have to read the screen instead
• By default the display shows the current room temperature, meaning you can’t tell without adjusting the knob what temperature the thermostat is currently set at

Somewhat more subjectively, I think it’s more likely to malfunction than an electromechanical device with two moving parts, and presumably it has to fight for spectrum with all surrounding cordless phones, WiFi units and garage door openers – let’s hope the base station does something sensible in the face of losing contact with the unit.

Well done, lads. Another triumph of engineering.

• It’s been a very busy year
• Still working for CFL and enjoying it very much
• Now living in a much nicer house (not damp! luxury!)
• Still helping balance the books for St Columba’s

Oh, and let’s check on the outcome of that 2011 geek wishlist…

• That Debian Squeeze will release in Q1 2011

Granted, and all my machines have been upgraded.

• That I’ll discover a J2EE Servlet Container which isn’t totally horrible

Tomcat 7 is a considerable improvement over its predecessors.

• That some kind of reconciliation between the JCP and Apache will occur

Nope.

• That Java 7 won’t be delayed forever

Partial victory: it’s arrived, but most of the interesting new features will only be in Java 8.

• That Python 3 will get adopted faster than PHP 5 or Java 1.5 did

Probably still to early to say.

• That we’ll finally see some major UK ISPs offering IPv6

Not a sausage, as far as I’m aware.

Merry Christmas all.

In June/July 2010 I spent ten days travelling in Rajasthan, India with friends; this is my diary of the trip (full list of entries here).

Got up at 9.30 and boarded my taxi to the terminal. Cheeky driver actually asked for his tip, but what the heck, he carried my case for me; better than London!

In the airport; bit of an anticlimax now it’s all over; found an Internet kiosk and caught up on ten days’ worth of e-mail (it’s much clearer when you see it all together how pointless 95% of it is). Gave the last five minutes of my time to an American chap anxious to close a deal…

Flight to London took off dead on time and served an excellent curry half way through. My reaction to the slight turbulence over Russia (wheee! isn’t this exciting; look at all these people looking queasy) led me to believe I was a little out of it, probably a combination of jet lag and being cooler than I’d been for a week.

Landed Heathrow after 7.5 hours (ahead of time!). Took three hours to get back to Oxford, what with being at the back of the plane, hundreds of French schoolgirls in front of me at the E.U. passport control lane (why can’t we have a British lane? Or does asking that make me a racist?), and getting across Oxford from the coach stop.

Collapse into bed and work in the morning; it’s a hard life.
—

Note to my loyal readers
Sorry it’s taken so long to write this all up. I’m a lazy busy man, you know. Expect to read about 2012′s holiday some time during 2014, and a very Merry Christmas to you all.

In June/July 2010 I spent ten days travelling in Rajasthan, India with friends; this is my diary of the trip (full list of entries here).

Rose early to find the storm over and a nice pleasant day forming outside.

Walked into town and went to see an old palace, a throwback to the British empire, now a museum of sorts.

Enjoyed a farewell drink and slice of cake with my travelling companions, then got a taxi out to the airport with K. He departed promptly to spend a few more days with his grandparents, leaving me alone as the only white person in the airport (and probably within miles…).

Bought a Robert Ludlum novel from the tiny airport’s tiny bookshop and settled down to wait.

Eventually caught my connecting flight up to Dheli, where I managed to procure a taxi to an airport hotel and fall asleep.

In June/July 2010 I spent ten days travelling in Rajasthan, India with friends; this is my diary of the trip (full list of entries here).

Today, we drove back as far as Jodhpur, where we dismissed our driver at the airport; there was a slight argument as he tried to gouge us for a bigger fee than originally agreed, but having a native speaker with us put a stop to that.

We caught an internal flight south to Udaipur, southern Rajasthan; this was definitely the smallest plane I’d ever been on (two seats either side of the aisle). It was diverted to Ahmadabad owing to bad weather over the landing strip at Udaipur (anxious moments – we hadn’t b0thered with malaria medication as we weren’t supposed to be going outside the low-risk zone), but we eventually hit the tarmac in Udaipur just in time to catch a taxi into town before the gathering storm broke.

From the taxi, we watched the wind blowing the debris from the side of the road everywhere, before locating a cheap and cheerful hotel. By this point it was raining heavily, with the temperature down to a blessedly manageable 30 degrees (but the humidity way up).

A decent dinner by candlelight (owing to the storm having knocked out the power) – definitely the oddest setting in which I’ll ever recount how I once met the Queen – then bed.

In June/July 2010 I spent ten days travelling in Rajasthan, India with friends; this is my diary of the trip (full list of entries here).

The morning after our camel ride, we woke up on our camp beds in the desert, feeling surprisingly refreshed.

After breakfasting on bananas and cake, we set off back to the city on our camels. We galloped the last quarter of a mile, much to my dismay as I bounced up and down in the saddle and kept landing heavily on my gentleman’s area.

We were then dropped off at the hotel, promising to come in to Mr Dessert’s office in town later to sign his testimonials book (which we didn’t. Apologies if you’re reading this, Mr D, but comfort yourself with the thought that I’m hearby recommending you unreservedly to my global readership of two and a half).

We had a nice lazy day at the hotel by the pool (apparently droughts aren’t a good enough reason to upset the handful of tourists wanting a swim), enjoying the colonial throwback presumably responsible for the flat screen TV having BBC World
but not CNN – the first western TV channel we’d seen all week.

We made a brief sortie into the city in the worst heat we’d experienced so far to be shown round an old merchant rest-house, then beat a hasty retreat back under the air conditioning.

In June/July 2010 I spent ten days travelling in Rajasthan, India with friends; this is my diary of the trip (full list of entries here).

Day six of our epic trip, and we’re off on the third leg, travelling north to Jaisalmer, on the edge of the desert near Pakistan. It’s apparently likely to be even hotter up there, but I’m not convinced that’s possible.

On the upside, everyone’s feeling better and we ma   de good time, arriving in mid afternoon.

We left the car on the outskirts and caught a couple of rickshaws up a very steep hill and through some impossibly narrow gaps, arriving at the offices of the very Indian-
named ‘Mr Dessert’, who we’d been recommended to by someone who’d visited the region last year. We booked ourselves on his camel safari, leaving at 4pm that day and returning the next morning, with a night sleeping under the stars in the
desert in between. It cost the princely sum of 1000 rupees a head – not bad for a trip which turned out to be laid on just for us [the only white tourists crazy enough to be there in the off-season heat?].

The icing on the cake was finding a decent hotel for the next night, which booked us in at half price owing to the off season and agreed to take care of our bags.

We were driven out to the camels, pausing en route to visit an abandoned village. The story, we were told, was that 100 years ago, the prime minister of the state, who was a very bad man with five wives, visited the village and demanded the mayor’s daughter be his sixth wife. The mayor refused to let her go and was given an  ultimatum – hand her over in 24 hours or we destroy the village. So the entire  population upped sticks and fled overnight, and the place has been abandoned ever since.

Whatever the truth of it, the village was definitely abandoned (apart from the old man at the entrance charging RS10 per head to look round!).

We rode out into the dessert on our camels – stopping to see the sunset and arriving at the campsite at 7pm.

Our guides were great – one of them mentioned to K that he’d had no formal education and learnt his (pretty good) English entirely from 20 years of looking after tourists on this trip.

We had a very tasty meal at the camp – no idea what it was, as we’d been forbidden from using any lights in order to avoid attracting mosquitos, but it was delicious. Our camp beds under the stars were just the thing, and we all passed a good night
apart from being disturbed initally by the local stray dog howling at the moon.

All of the above was in no way spoiled by noticing that in the middle of the dessert, my mobile phone had three signal strength bars – more than it does in many areas of Oxford. Clearly the architects of the dreaming spires just didn’t think of the radio waves we’d be trying to get through them 500 years later.

On the upside, everyone’s feeling better and we made good
time, arriving in mid afternoon.

We left the car on the outskirts and caught a couple of
rickshaws up a very steep hill and through some impossibly
narrow gaps, arriving at the offices of the very Indian-
named ‘Mr Dessert’, who we’d been recommended to by
someone who’d visited the region last year. We booked ourselves
on his camel safari, leaving at 4pm that day and returning the
next morning, with a night sleeping under the stars in the
desert in between. It cost the princely sum of 1000 rupees a
head – not bad for a trip which turned out to be laid on just
for us [the only white tourists crazy enough to be there in
the off-season heat?].

The icing on the cake was finding a decent hotel for the next
night, which booked us in at half price owing to the off season
and agreed to take care of our bags.

In June/July 2010 I spent ten days travelling in Rajasthan, India with friends; this is my diary of the trip (full list of entries here).

I was quite ill today and spent it in the hotel feeling sorry for myself. Still; I had company – K also feeling pretty ropey. Happily we were both feeling a bit better by evening and resolved to press on in the morning for the third leg provided we felt no worse.

`Mantrid `_ is the first of hopefully many pieces of Epio's infrastructure that we're open-sourcing.Back when we were first developing `Epio `_, we started off using `HAProxy `_ as our load balancer. HAProxy is a fine piece of software, and performed quite well for a while, but it quickly started having issues as we got more sites. The main issue was that Epio has at least two (and often more) hostname-matching rules for each app we host, and we have rather a large number of apps - numbering in the thousands at the time we're talking about, and more these days. HAProxy has two issues at this scale: One, it's not terribly efficient at matching domains (it looks to do a linear scan through several thousand regular expressions), and two, the state of our sites changes constantly; we were having to reload it every ten to fifteen seconds; this worked surprisingly well (HAProxy has a graceful restart mode), but still meant things were a little delicate. Thus, one day in May, when the HaProxy setup was having a particularly bad day, I sat down and wrote a simple Eventlet-based load balancer. We tested and deployed it that week, and much to our surprise, the latency on requests dropped. Over the following months, we built on that code, integrating lazy-loading for applications (so if you hit an application that was disabled, it would hold the request while it started it) and statistics logging (to measure bandwidth). However, that load balancer, while it performed admirably for over 6 months, was heavily tied into the core Epio codebase, and our set of common libraries and utilities. Thus, I began the work of separating it out and polishing it off, and Mantrid was born. The key features for us are, obviously, the fact we can change the host configuration at runtime, and the ability to "spin" requests until applications are loaded. This feature is, incidentally, also useful even for a single-host site; if you're performing a database upgrade, migration, or server restart, you can do it with zero dropped connections with something like:: mantrid-client set myhost.com spin true # ... do restart ... mantrid-client set myhost.com proxy true backends=1.2.3.4:8000 We deployed Mantrid into production on Epio around two weeks ago, and it's performed flawlessly ever since. We have it exposed directly to the internet on port 80, and also to our SSL-terminating Nginx on another port. We'd love for people to try it out and report bugs, or tell us their experiences; our goal is to get it running with a latency and throughput cost close to that of other software load balancers (in part by running it under `PyPy `_, something we designed it to do). (Note: Several people have asked me what we do if Mantrid crashes, as it's configured at runtime. The answer is simple - it persists to a state file, which it loads from on startup)

There's not been much here lately. Time to change that.There was a time, in the distant past, where I blogged at least once a month - the archives sidebar is a testament to that. However, for a nasty combination of reasons - mainly "having a life" - it hasn't seemed to happen in the last year or so. Well, I've decided it's time to change all that. Two days ago I opened up the codebase of this site for the first time in years, changed the repository away from Subversion, upgraded it to use Django 1.3 (rather than 0.97-pre, which will give you some idea of its age), and actually put my recent projects onto the projects page. I suspect one of the more interesting inspirations may have been my recent acquiring of a hobby that isn't programming, which means my dark days of trying to improve make my speaker bio sound less nerdy are over. Said hobby is, unsurprisingly for me, flying (of small planes, rather than self-propelled levitation) - something I've been interested in for a while, but never really got around to doing. Naturally, I'm already pricing up that nice new twin-engined plane I've apparently always wanted, though at this point I'm only five hours into training, and have accomplished the rather spectacular feat of keeping the plane straight and level, as well as leaning how one narrowly avoids hitting the turbulence caused by a fighter jet streaking over the Thames estuary - an important skill for later in life, I'm sure. Needless to say, expect a lot of boring flying-related posts on here (well, unless something exciting happens, but most exciting things in aviation are also dangerous, so getting them written down is harder). I'll also make sure there's a good set of programming-related posts, to make sure every one of my readers has something to be bored about - I'm a big fan of equality. For now, though, it's back to being slightly lost in Kiev (or Kyiv, or Київ), and pronouncing Cyrillic letters one at a time to try and pronounce things - always a joy, especially to the locals, who are delightfully amused whenever I try to speak to them. Adiós, or, rather, до свидания, for now.

Reader, are you an idiot? Let me quantify that slightly. Are you the sort of person who gets up on a Saturday morning and puts a load of washing on without checking the pockets? And realises that your Fitbug is getting a wash it could have done without? I am.

I’ve done this before, and in the past, got away with replacing the battery. This time, though. pushing a new battery in made all the segments on the screen come on, and brought forth no response to the buttons.

So I took the whole thing apart and found it covered in white crud, presumably washing powder. Spraying the innards with WD40 drove that out, and re-assembling brought it back to life and saved me spending £30 on a new one.

Having booked myself the obvious three days off work in December to join up the bank holiday weekends for Christmas and New Year, I thought I might as well book the train tickets back home too.

Logged on to RSH (one of the few operators still willing to post you the tickets for free) and bashed in my dates and times. The cheapest single from Oxford to Macclesfield on 23 December after 1700 (with Railcard)? £39. No sign of an advance single.

Hmm. Maybe RSH being weird. Try East Coast? Nope, same prices, same problem. Same deal with trying the morning of the 24th.

The National Rail Enquiries journey planner’s calendar doesn’t go up to the 23rd of December, and if you enter that date manually, it says you can only book twelve weeks in advance.

So, do we think RSH and East Coast would have told me that? Or taken my money? Being a regular reader of Private Eye’s Signal Failures, the whole thing smells like shameless profiteering to me, but I’m not about to risk £20 finding out. I’ll report back in two weeks.

The slides (and video) for my talk at DjangoCon US this year are now up.You can find the slides here, and the video is available on the DjangoCon blip.tv account.

Following on from my initial foray into SIP a couple of weeks back, I ordered a Zoom 5801 ATA from Expansys, so I could hook my good old-fashioned cordless phone up to Sipgate.

I wasn’t impressed by the six working days it took to turn up, but it was free delivery.

In terms of hooking it up, the man-traps to avoid are:

• Your Sipgate password for registering a device is NOT the same one you use to log in to their website
• You don’t need STUN enabled or configured
• I’ve got various SIP ports forwarded from my router to the ATA’s IP address, though I’m uncertain if they’re absolutely necessary (Google for the ranges to use).

My router is Virgin Media’s latest Superhub.

Having configured it all, incoming calls work perfectly, and my twelve quid cordless phone from Argos manages to display the caller ID, which beats both Virgin Media and BT’s landlines hands down. Outgoing calls work too, and don’t forget that Sipgate don’t charge for 0800 numbers (unlike your mobile provider).

Lastly, since you’ve got Nagios running on the PC behind your sofa anyway (right?), let’s set up a simple check to scrape the web interface and verify that the ATA is correctly registered with Sipgate:

define command{
 command_name    check_zoom_ata
 command_line    $USER1$/check_http -a administrator:yourpassword -I $HOSTADDRESS$ -r "Ready to make calls"
 }

Now define a service using this command, and associate the service with one or more hosts/hostgroups in the usual way.

Being an organised sort of person (and a sucker who can’t say no), I tend to do a lot of the admin and paperwork for the house I share with a few friends. One of the things I do is draw up the chores rota. I’ve typically done it like this*:

Week      10/09 17/09 24/09 ...
Bathroom  Tom   David Harry ...
Kitchen   Dick  Tom   David ...
Bins      Harry Dick  Tom   ...
Hoovering David Harry Dick  ...

Careful and studious readers will have no difficulty in spotting the pattern.

Inevitably, though, instead of being grateful for the minutes I slaved over my desk putting this together, t’housemates complained. Specifically, Tom complained that ‘I always do something the week after Dick’s supposed to have done it, and he makes a mess of everything’.

Given the choice between speaking to Dick to correct the problem, or writing some code, I wrote some code which starts with the above layout, then shuffles the columns until the following constraints are satisfied:

• A given person never does the same thing two weeks running
• For any given pair of people (p1, p2), this pair never appears twice in a given row

Careful and studious readers (with Maths or CS degrees) will have no difficulty working out the number of weeks of chores rota I’ve limited myself to doing at a time, becuse the constraints become impossible for a greater number.

* No, my housemates aren’t really called Tom, Dick and Harry. After five years at an all-boys school, there’s no way I’d live in an all-male household.

Paying BT or Virgin Media £10-£15 per month for a landline telephone seems like a waste of money when I have a mobile phone contract which gives me 300 free minutes per month. But it still costs more for other people to call me on t’mobile, and not everyone has a contract phone (do they, Mum?). So I’ve been looking around for VOIP providers who could give me a geographic UK phone number which I could take calls on via my home internet connection.

There are quite a few providers who’ll do this for £3-£4 per month, but sipgate are doing their best to disprove the theory that there’s no such thing as a free lunch, offering UK numbers for free (at least for the moment).

Within a couple of minutes of signing up, I was the proud owner of a ‘real’ 01865 Oxford phone number and some SIP credentials for registering a device to make/receive calls via it (obviously outbound calls cost real money, it’s just that there’s no line rental or monthly charge to have the number which people can call at their usual geographic rate).

I managed to set up Linphone as follows:

Under Linphone > Preferences > Manage SIP Accounts, hit ‘Add’ under ‘Proxy Accounts’, set “SIP Identity” to sip:1234@sipgate.co.uk, “SIP proxy address” to sip:sipgate.co.uk. OK out and you should get a prompt for your sipgate password; put it in and the status bar should read ‘Registration on sip:sipgate.co.uk successful’.

I’m pleased to report that incoming calls come with the caller ID information – the incoming caller will be “01234 567 890″ <sip:01234567890@sipgate.co.uk>. As far as I can tell, it only supports one incoming call at once, but for £0/month, that’s not too shabby

Next on my list is to buy an ATA and hook up my real cordless phone to sipgate. Watch this space!

 

I don’t much care for Simon Willison‘s (or in this case, the person he’s linking to’s) implication that Java programmers are misguided fools in need of help (as opposed to engineers trying to earn a living by using the best tool for the job on hand, rather than getting all evangelical about their language of choice).

However, the quote he posted recently makes a good point – using immutability and other functional ideas does make for better-written and less error-prone Java. And many of us have moved in that direction already, mostly without any prompting (although since Haskell was the first language they taught me at Oxford, that doubtless influences my world view).

On Monday 1 August, we moved into our new house.

On Monday 22 August, we got an internet connection.

Fortunately, I have a shiny Android phone with WiFi hotspot capability these days, and it turns out the 500MB of data O2 give you lasts almost exactly three weeks if used carefully. They even send you a text when you hit 80% of it, and don’t automatically bill you when you reach 100%.

All very civilised, which is more than can be said for the process of getting an internet connection in central Oxford. First, we tried to get some ADSL via our BT line. There is a master socket in our lounge (carefully hidden behind a curtain, half way up the wall, natch), but the previous tennants had been on Virgin Media, and the BT socket was dead.

It took several phone calls to BT and the Post Office to establish that re-activating the line would cost us £130 for an engineer’s visit – even though the wiring and socket looked perfectly intact to me. Oh, and it would take three weeks. The process of arguing with BT Openreach about reactivation is apparently so tedious that the vast majority of ISPs offering ADSL don’t bother, insisting on your line having a phone number and a dial tone before they’ll talk to you (even if you want to pay them for the line rental too).

Having eschewed the BT monopoly’s attempt to gouge us for £130, we turned to Virgin. More promisingly, their bog-standard 10Mbps broadband (with no phone or TV) didn’t come with an up-front connection fee, but guess what – it would take three weeks to send their bloke to install it. Why they can’t offer self-installation in cases where the house is already wired is beyond me.

Our connection is currently showing a pretty respectable 9.71Mbps downstream and 0.5Mbps upstream – pretty close to the 10Mbps we’re paying for. The two annoyances so far have been that the engineer was obviously getting a bonus for conserving coax, as they gave us a 60cm length just long enough to put the modem/router on the bottom shelf underneath all our DVDs and books, rather than on the top shelf where it would give a better signal to the top two floors. Luckily some past installation of Virgin had left a lot of coax and splitters in a drawer, so I deployed some of those.

The other fly in the ointment is that our modem/router is Virgin’s superhub, much maligned by some as flaky and unreliable. I’ve not found the wifi range great on ours, but positioning it on the aforementioned top shelf has helped a lot. The main problem so far has been that, after a brief power blip on Monday, it came back on but disabled the WiFi, leaving me to plug into it with an ethernet cable in order to turn it back on.

The noddy web interface on the superhub also worried me, but it turned out there’s a well-hidden ‘advanced settings’ link back to the good old Netgear web interface with all the knobs and buttons for the advanced user.

I’ll report back if we experience any serious problems.

Meanwhile, here’s hoping that by the next time I move house, some geek will have achieved a landmark victory at the European Court of Human Rights, mandating that internet is a basic human right and must be installed within 48 hours of a person moving in to a new house. Not holding my breath, though.

I’d noticed for years that sending outbound e-mail from Thunderbird to my server on port 587 took far longer than it should have – about six seconds of staring at the progress bar.

Today, I was finally bored enough to work out the cause: Exim is configured to perform ident checks, which take 5 seconds to time out. Since port 587 only accepts mail from authenticated users, we can disable the ident checks for it:

rfc1413_hosts = ${if eq{$interface_port}{25}{*}{! $sender_host_address}}

If you’re running Debian, change the above in /etc/exim4/conf.d/main/02_exim4-config_options.

I read with interest the news that ICANN has approved the ability for companies to register .anything domain names (so KFC could register .chicken and have http://chicken as their website). Some are saying that  chaos will follow.

Personally, I’m not too worried. In the last week I’ve seen both an older relative and someone my own age refusing to believe that websites without www in front of them can possibly exist (visiting www.sub.example.com despite being told to go to sub.example.com), so I suspect anyone silly enough to shell out $185,000+ will watch the abysmal traffic to their website and wish they’d spent their money on a leaflet campaign instead…

I gave a talk at the wonderful DjangoCon EU again this year, and the slides are now up.You can see the slides `here `_, and the video is available on the DjangoCon blip.tv page.

Over lunch with some colleagues last month, the subject of why Microsoft is paying $8.5bn for Skype came up. “Simple”, I replied, “for years, governments have been looking for back doors into encrypted communication systems, including Skype. The US government must be secretly subsidizing Microsoft’s takeover in return for access to all the Skype traffic”.

Consensus round the table was that my theory sounded just a bit too plausible for comfort. If I disappear suddenly, you’ll know why…

I’m pleased to report that (having fixed the IPv6 on my home WiFi), my HTC Desire S running Android 2.3 picks up an auto-configured IPv6 address on its WiFi interface. This is then used successfully by the built-in browser, K9 Mail and Irssi Connectbot.

I was hoping to include a screenshot showing the IPv6 address the phone had obtained, but apparently you can only screenshot a non-rooted Android by using the developer tools … sounds like a job for another day.

After years of thinking about it, and even writing about it, I’ve finally bought myself a smartphone.

As I write, I’m coming to the end of my third week with my new HTC Desire S. And it hasn’t disappointed.

Carrying it around has proved less arduous than I imagined – although it weighs more than my clunky old Nokia 1100, it’s thinner and flatter, so it fits nicely in my shirt pocket. It gets quite warm when under heavy use (e.g. acting as a WiFi hotspot), but not unpleasantly so.

They’ve packed quite a lot into such a small case – in no particular order, we have FM radio, a 5 megapixel camera with LED flash which works quite well for basic snaps, a second front-facing camera for video calling (makes me look awful, but friends insist it’s accurate), GPS, a half-decent speaker, a headphone jack, a nice big touchscreen, and I’m told it also has a phone.

Typing on the on-screen keyboard has proven easier than I anticipated – even with my fat fingers, I can peck out a short e-mail with reasonable ease. The predictive/corrective text is actually surprisingly helpful here.

The built-in e-mail client is OK, if a little basic. It wouldn’t send outgoing mail via my Exim 4 server over TLS (a TLS packet with unexpected length was received), but I suspect that’s Debian’s fault for insisting that GNUTLS actually, er, works. I ditched the default client in favour of K9-Mail, which boasts PGP integration and is much more customizable. Apart from a few niggles with the UI, it does the job very well.

The browser works much as one would expect – even sites without a mobile option are surprisingly usable – the screen is a decent resolution for its size, and the usual gestures for zoom work well.

The music player worked straight out of the box when I copied some MP3s over, and the supplied headphones aren’t too shabby.

The ability to read bar codes and search online for the cheapest available version of the barcoded thing has proved endlessly amusing.

Irssi Connectbot deserves special mention for making IRC on the go dead easy.

One of the main reasons for opting for an Android phone were the tethering capabilities – with a couple of well-chosen taps, the phone can share its 3G internet connection by turning itself into a WiFi hotspot. This is one of the operations that makes it a bit warm, but it’s very handy if you’ve got a bigger computer with you, but no internet.

Battery life isn’t too bad given the capabilities of the device – it lasts a heavy day’s usage, and charges over USB from almost anything. They also throw in a standard wall-socket adapter too.

Android claims some IPv6 support, which I’ll report back on when my home IPv6 is raised from the dead. I’ve also got a few ideas for app development, so watch this space!

We‘ve been having a spot of bother with update-grub not working on our Debian Xen guests since upgrading them to squeeze.

The symptom: update-grub (as run after the installation of a new kernel package) fails because it’s ‘unable to find [a] GRUB drive for /dev/sda2 – check your device.map’. This happens using both grub-legacy and the new grub-pc package.

You’ll probably have run into this if your guests were created using xen-tools.

Skipping over the related bugs, the long and short of it seems to be:

• The way Xen pokes /dev/sda1 and /dev/sda2 (which are probably LVMs on your dom0) into your guests without a corresponding block device confuses grub
• grub used to be dumb enough to ignore the mysterious presence of ‘partitions’ with no block device and carry on anyway
• The versions in Debian Squeeze are just clever enough to be dangerous and fall over in this situation

Happily, we’ve worked out the fix. Grub is hard-coded to do the right thing in this situation if your disk devices are called ‘/dev/xvd[a-z]‘, so you need to:

1. Fix all references to ‘/dev/sda2′ to read ‘/dev/xvda’ (and /dev/sda1 to /dev/xvdb). In practice this means grub’s menu.lst and possibly your /etc/fstab if that isn’t by UUID
2. Edit the /etc/xen/guest-name file on the dom0 to rename the partitions (change the ‘root’ specifier and the lines in the ‘disk’ list)
3. Destroy (sudo xm destroy) your guest. Note that you must destroy and recreate it; rebooting won’t give Xen a chance to rename the devices.
4. Bring it up again with xm create -c config-file
5. Profit.

In June/July 2010 I spent ten days travelling in Rajasthan, India with friends; this is my diary of the trip (full list of entries here).

Today we left Jaipur to start on the second leg of our trip; unfortunately all of us are feeling a bit off, but that’s possibly unsurprising after all the rich food we’ve eaten over the last 48 hours coupled with the intense heat. Anyway, we’re OK to travel, and our car and driver to Jodhpur await!

A somewhat dull day in the car, trying not to let the sun dazzle us and stay out of the heat. Quick stop at the services, and into Jodhpur by 4pm. Much faffing before we found a hotel that wasn’t full – apparently some sort of conference is in town, and the guest-house we had in mind took one look at us and said ‘no foreigners’.

I considered asking K to ask the bloke ‘what makes you think my friends are foreign?’ but decided on balance we wouldn’t get away with that…

Finally secured a couple of nights in a decent-ish hotel for not too much; our driver is apparently perfectly happy to sleep in the car. A different world…

Dinner in the hotel and an early night under the best air conditioning we’ve had on the trip so far; turned firmly up to maximum and with ‘turbo mode’ on. Can’t help but reflect that having it on is pumping out CO2 which is likely to make the heat worse in the long term; but when the heat outside is bad enough to kill you if you’re used to colder climes, it’s not the time to entertain such thoughts.

In June/July 2010 I spent ten days travelling in Rajasthan, India with friends; this is my diary of the trip (full list of entries here).

Quite a day today!

First, we drove out (supposedly before the worst of the heat, but certain people didn’t get up early enough…) to see Jaipur’s fort.

Then, back to K’s grandparents’ place for a splendid lunch laid on by caterers for us and a large number of visiting members of his family including my first taste of an authentic Indian curry (chicken, obviously, not beef). Delicious!

After hiding from the heat with my book for the rest of the afternoon; as darkness fell we dressed up and ventured out for one of the highlights of the trip: the last night of a genuine Indian wedding (apparently they typically last several days!). Friends of K’s family were getting married in style with 300 guests, and we caught up with the wedding procession marching up Jaipur’s equivalent of the A34, much to the annoyance of passing traffic.

It had elephants, a man on stilts, a marching band, 300 dancing guests, a lot of electric lights, and a man throwing fireworks into the air overhead.

After going round the block a couple of times, the procession marched into the grounds of the hotel where the dinner was being held. As we gorged ourselves on a wide range of delicious Indian food, TV cameras whizzed overhead on wires, televising the whole thing to screens on stands around the place. I felt quite sorry for the bride and groom, sitting on the grand stage at the front, being congratulated by and photographed with all their guests in turn – no chance to get at all the food!

Returned to our hotel thoroughly full and with lots of happy memories of an amazing night.

I’ve just minted the 0.3 release of my stateless RSS reader, Bogroll.

This version features caching support which actually works, plus deep-linking to categories and browser navigation support courtesy of JQuery BBQ.

The code now lives at Bitbucket.

Edit: Those of you coming across this post by googling for ‘Sixxs rDNS’ should note that Bytemark’s DNS service isn’t free unless you already have a server with them, but there are a number of free alternatives out there.

With IPv4 address space exhaustion practically upon us, I decided it was high time my house got IPv6. This is quite easy to do even if your ISP, like ours, doesn’t support it natively. Get a Sixxs tunnel, apply for a subnet, set up radvd on the Linux box behind your sofa (you do have one, right?) and there you are.

Even Windows XP can be trivially prodded into IPv6ing itself up, and my router seems not to mangle the radvd broadcasts so even wireless clients can have v6 if they support it.

What really impressed me, though, was what happened when I wanted to set up reverse DNS for my new Sixxs subnet. I’ve been assigned 2a01:348:1af::/48 and the Sixxs page tells you that you’ll need your own DNS server to host the necessary records. I don’t run my own DNS; I use Bytemark’s content DNS service.

I wasn’t expecting Bytemark to support adding rDNS records for random IPv6 /64s, however as it turns out, once you work out what the TinyDNS file needs to look like:

# IPv6 rDNS authority for Sixxs subnet 2a01:348:1af::/48

.f.a.1.0.8.4.3.0.1.0.a.2.ip6.arpa::a.ns.bytemark.co.uk
.f.a.1.0.8.4.3.0.1.0.a.2.ip6.arpa::b.ns.bytemark.co.uk
.f.a.1.0.8.4.3.0.1.0.a.2.ip6.arpa::c.ns.bytemark.co.uk

# Machines in the first /64 (home network)
6router.ipv6.dnorth.net:2a01034801af00000000000000000001:8640

… it Just Works:

$ host 2a01:348:1af::1
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.f.a.1.0.8.4.3.0.1.0.a.2.ip6.arpa domain name pointer router.ipv6.dnorth.net.

Nice one, Bytemark!

A fond farewell today to my car, which has been in the family since 1999 and suffered both myself and my sister learning to drive in it. The mighty Fiat served me well, but in the end, living and working as I do in the middle of one of the most anti-car cities in England, it had to go.

I sold it to We Buy Any Car (sponsors motoring on Dave), and they gave me fifty quid, which I shall be putting towards finally getting a smartphone.

For the past month and a bit, I've been working part-time on my new venture.Along with Ben Firshman, I'm working on ep.io, a Python/WSGI hosting service, with resource-based (not instance-based) billing. It's been an interesting ride - designing shared hosting so it's both secure and so you can measure and bill resources accurately is a difficult task - but we're finally out of the woods and have everything working reasonably normally, leaving only the slow crawl of beta-testing and polishing (and working out a payment provider). We've also launched a blog, which only contains an initial post now, but it's where we'll be noting down what we've learnt, and general deployment advice, hopefully making it useful as a general learning resource as well as a corporate mouthpiece. Initially we've focused on supporting Django, as it's the thing Ben and I use most often, but the entire stack was built with general WSGI applications in mind (and, indeed, they do run). The stack we've built is quite flexible, so it could potentially be used for a variety of different languages or frameworks, but we think focusing on doing Python (and doing it well) is a good goal. If you want to keep track of what we're doing over at ep.io, it's probably worth following @epdotio, or subscribing to the blog's RSS feed. There's also a beta invite list you can sign up for.

In no particular order…

• That Debian Squeeze will release in Q1 2011
• That I’ll discover a J2EE Servlet Container which isn’t totally horrible
• That some kind of reconciliation between the JCP and Apache will occur
• That Java 7 won’t be delayed forever
• That Python 3 will get adopted faster than PHP 5 or Java 1.5 did
• That we’ll finally see some major UK ISPs offering IPv6

Leave a comment if you can see anything I’ve missed, and we’ll check on them all at the end of the year…

A wise man once wrote:

“Unlike the precocious child who will taunt you mercilessly, knowing just how to report the beating they deserve to their school teacher in a manner that will have you in Police custody before lunchtime, hardware is sneaky.“

I couldn’t agree more.

Hardware and I have never really got on, but we’ve always maintained a queasy peace – my laptop may not hibernate properly under Ubuntu, but it sort-of works. Likewise the machine behind my sofa refuses to charge its CMOS battery, but as long as I reboot it rather than shutting it down, it manages to come back without manual intervention.

Last weekend, though, hardware declared war. And it did so under the guise of a familiar occurence – a phone call from a family member about some “computer trouble”. Mum and Dad have their internet courtesy of now-defunct Tiscali UK, which was taken over by TalkTalk. For about three years, the Tiscali-branded black box router had been sitting in their study and working. It didn’t feel like the highest-quality piece of kit in the world, but it did manage to keep the connection open and serve one wired client – a desktop PC – and my sister’s laptop over WiFi.

Three years on, it died. No problem, I thought. I ordered a reasonably priced replacement to be shipped over there, and the next weekend I tried to talk Mum through installing it over the phone. No good – we eventually established, after much frustration, that the router was not DOA, but partially-alive OA, and kept dying within 30 seconds of being plugged in.

Sigh. Grit teeth. Pack it up, return it, buy something a bit more expensive from Netgear. Talked Mum through installing that, no need for WiFi just yet, job done.

Fast-forward to last weekend when my sister rocked up wanting to use the WiFi. No problem, I thought (you’d think I’d know better by now…) – the router is configured with the same network name and password as the last one, so it should Just Work™.

Guess what, it didn’t. So I stepped into the breach, fired up Vista on her machine, point it at the right access point, find it asking me for a PIN. Gah? Turns out that since I last played with this stuff, WiFi protected setup has been invented. And for sure, it sounds like a good idea, but in this case, I don’t want it. Please just let me specify the password by hand. Vista, as far as I can tell, won’t let you do this.

OK, disable WPS on the router, change the network name to make devices forget everything they think they know about it, try again. “Wireless authentication failed because of a timeout”. And no further progress was possible. Much Googling suggests that the Atheros wireless chipset in the laptop is incompatible with certain Netgear routers, and nobody has a solution to this.

Great. At this point, the urge to throw all  the blasted computers out of the nearest window was strong enough that I decided it would be a good time to hand my sister the 100 metre Ethernet cable and give up. And it was at this point that Mum wandered in and said “the internet isn’t working”.

Sure enough, the desktop PC had lost its connection. What’s more, it had lost all knowlege of containing a network card, and no amount of poking, prodding or re-seating would make it work. It still had flashing lights on the back, but nobody was home.

It was at this point that I looked up from the screen with bleary eyes and realized I had to be back at work in the morning.

I’m not really interested in aportioning blame for all this, but I do find it depressing that after 10 years, WiFi is still about as friendly as a cornered rat when it goes wrong, and badly implemented all over the place.

Browsing through the URC North Western Synod’s website today, I see they now run it on WordPress. And this is a bit of a relief for me – the site’s previous incarnation was a bunch of hand-crafted PHP written by 16-year-old me, and I’d been having sleepless nights wondering if that code was still in use, and how much pain it had caused. (Not that WordPress is perfect, but if it’s good enough for this blog…)

Not all of my early attempts at computer programming have gone the way of all flesh, though – the Python script I wrote to parse the dinner menus for Magdalen JCR (and the subject of my very first blog post) is apparently still in production and telling people when to expect Chicken Kiev as recently as last week. The time it’s saved various JCR computer reps has presumably now exceeded the two days I spent writing it in the first place. And though I know the code is pretty horrible by my current standards, it does that which got me into computer programming in the first place – makes real people’s lives easier.

Just a quick one to say that the slides and videos from Oxford Geek Night 19 are now up – I very much enjoyed speaking for the second time at OGN, and due thanks must go to TorchBox and the Nokia Ovi Store for making it happen.

Moving a charity’s accounts to a new bank: how hard can it be?

It’s now just over a year since I took over as joint treasurer of Saint Columba’s. And much as my mum (a church treasurer herself for many years) predicted, the first year was the worst.

One of the most tedious bits proved to be moving the church’s accounts to a new bank. CAF specialise in banking for charities and nonprofit organisations, and after years of shoddy service from our current bankers, we were ready to bail out.

I won’t name the current bankers, but let’s call them LlBarcBCWestFax.

Moving our savings was relatively easy – just get the right signatures on a letter authorizing the closure of the account, and asking for the funds to be transferred by BACS to CAF. Sadly, LlBarcBCWestFax’s crack team of account closure specialists managed to ignore my explicit instructions and issue a five-figure cheque, which they posted to the church (addressed to ‘The Directors’!). I found it there after an increasingly anxious few days wondering where the money was.

That was easy, though, compared to moving the current account. Having chased up everyone paying money into it and asked them to move their standing orders elsewhere, we were left with the small matter of 25 direct debits needing switching. No problem, allegedly – CAF supplied a form to fill in asking for them to be moved, and we duly sent it off back to CAF. They in turn made a request to LlBarcBCWestFax, and two weeks later, I got a list of direct debits in the post on which I was asked to highlight the ones to move.

Could have sworn I ticked the box on the original form saying ‘all of them’. Never mind, let’s play the game. Highlighted and sent off.

Two weeks later, things started to move. Out of 25 organisations – ranging from ex-national monopolies to smaller operators, one managed to send a letter saying ‘thanks for moving your direct debit – it’s business as usual’. 20 sent letters saying “sorry to hear you’ve cancelled your direct debit”, followed two days later by “glad to hear you’ve set up a new direct debit!”.

And the other four? Three of them failed miserably to enact the switch and insisted on all sorts of paperwork with the signatures of all people on the bank account being sent. Seeing as the others managed it, I can only put this down to incompetence (one of these suppliers has been running a commercial all over the place lately, and the mere sound of their jingle has me wanting to kick something).

And finally, the one supplier who, two months after the switch, still hasn’t done it? Our ISP. Turns out that all banks are equal, but some are more equal than others when it comes to direct debits. Our ISP “can’t do direct debits from CAF”. I haven’t managed to get a straight answer on why this is, but it looks like our LlBarcBCWestFax current account will survive into 2011. Sigh.

The one positive note in all of this has been CAF’s responsiveness as I’ve chased up issues with them. Being able to phone them and get straight through to an intelligent person who gives me a straight answer is exactly why we took on the challenge of making the switch in the first place. And unlike LlBarcBCWestFax, they don’t charge £12 for cancelling a cheque…

I've released South 0.7.3. It's only a bugfix release; the release notes can be found here.

Just a quick one: I've posted my slides from the talk I gave at PyCon Ukraine.The topic was "Relational & Non-relational databases with Python", and is an improved, expanded and non-Django version of the talk I gave at EuroPython this year, with quite a few more code examples. The slides are available as a PDF (242KB). There was a camera pointing at me during the talk, so I presume there'll be a video at some point.

In June/July 2010 I spent ten days travelling in Rajasthan, India with friends; this is my diary of the trip (full list of entries here).

Sunday 27 June

First full day out here, and I slept reasonably well in spite of the heat – thanks to the air conditioning and the jet-lag, presumably.

In the morning, we drove into the centre of Jaipur to see the city palace:

Behind the palace is the observatory. Measuring the movement and position of the stars and planets is important to Hindus as astrology is part of their religion. The life-size instruments were pretty impressive, and provided shade from the blazing heat:

After lunch and waiting for the worst of the midday heat to subside, three of us ventured out to the Rambagh Palage Hotel – 700 rupees for a White Russian, which is pretty horrific in India, but about the same you’d pay in far worse surroundings in Oxford. Had a couple of those and lazed around enjoying the serious air conditioning.

That evening, K’s family very kindly invited us out to dinner – as it turned out to the very same hotel! Fantastic food and unforgettable surroundings -a great dining room originally concieved by the British and immaculately refurbished.

I think of myself as a right-winger but couldn’t help but reflect on the dark side of capitalism whilst being driven back from that dinner amongst the peacocks and fountains, past India’s poor sleeping in the streets. While we were out there the government of Rajasthan cut the water supply from one hour a day to one hour every two days because of the lateness of the monsoon. If you’re rich enough to have a tank on your roof, you don’t mind, and if you’re not, you go without for 23 hours out of 24…

I was fascinated to read about chancel repair liability the other day. Simply put, some ancient laws that nobody bothered to repeal since the Norman Conquest mean that your local C of E church might be able to slap you with a bill for some of their repairs if you own a house on their patch.

As joint treasurer of a church myself (though a Reformed one, not C of E), the thing that struck me most about this was the appalling arrogance of the church in the cited case which bought all this up back in 2003. Sure, Saint Columba’s could do with ninety grand for a new heating system, but even if there were some archaic law that allowed us to send the bill to our neighbours, would we really do that? Let alone pursue it all the way to court and make them pay our legal costs too. OK, so one of our neighbours is Oxford University’s richest college, but even so I find it impossible to imagine we’d sanction that sort of thing.

And they wonder why people percieve the church as arrogant, stuck in the past and unfriendly…

The problem

A word document (plain old .doc, not 2007) should be received by e-mail, fed to a script, turned into a PDF and published on a website.

At my disposal

My server running Debian ‘Lenny’, which does not have a display of any kind.

How hard can it be?

Harder than it should have been, as ever. Here are my steps:

# aptitude install python-uno xvfb openoffice.org-java-common openoffice.org-writer unoconv

You’ll note the inclusion of Xvfb there, because it turns out that “headless” mode in OpenOffice isn’t really headless at all. Sigh. Also sigh some more at the broken dependencies of the unoconv Debian package.

Now we can write our script to do the actual conversion. Shame it took twice as long as it should have…

In June/July 2010 I spent ten days travelling in Rajasthan, India with friends; this is my diary of the trip (full list of entries here).

Saturday 26 July

Landed at Indira Gandhi International Airport at 9.30AM local time. Walked out of the terminal into searing heat (over 40 degrees in the shade) and a medley of noise and traffic. Dodged our way through the traffic and found our taxi waiting. Saw our bags strapped to the roof and set off into the traffic. As we’d been warned, the only similarity between British driving and Indian is that both countries drive on the left – in India the horn means “get out of my way please, I want to pass” and flashing your lights means “I’m not going to stop, so get out of the way or I’ll ram you”. In the city, people, animals and motorbikes all compete for road space with the cars and lorries, there are no traffic lights to speak of, and the smallest gap is made to be squeezed through.

On the main highway – this first day we’re driving from Dheli to K’s grandparents’ house in Jaipur, approximately 200 miles to the south-east – the sheer volume of traffic and the two-lane road limits progress to a maximum of 50mph. Drove past Debenhams and M&S on the way out of Dheli; wondering just how westernized this country is…

Stopped for lunch at India’s answer to the motorway services, about 1pm. Stepped out of our air-conditioned taxi into searing, dusty mid-day heat and ran for cover in the building, which had just enough air conditioning and fans to reduce the temperature from ‘horrifically hot’ to ‘bearable’. Ate a nice samosa with a coke, then back to the car.

Arrived in Jaipur at about 4.30pm, glad to get out of the car. The drive down wasn’t particularly inspiring; mostly motorway.

Met K’s grandparents – great people, made us feel really welcome – settled into our hotel and had a really nice meal with them before going to bed (or trying to; despite the air conditioning, still pretty warm!).

Editor’s note: more to follow soon, with photos!

After a slightly too long hiatus, I've released the second bugfix release of the 0.7 series.There's not a whole lot new and exciting in this one, just lots of small fixes - bugfixes, removal of some unused variables, a few more command line arguments where they should have been already, that sort of thing. You can read more in the release notes. In other news, I'll be giving a talk at DjangoCon US 2010 which, for once, isn't about South, but instead covers a more broad spectrum of databases and schemas, in particular encompassing schemaless databases (which do still need migrations, in some form), so if you're going to be at the conference, come along, or just catch me outside of a session if you want to talk. I apologise for the slightly slow rate of work on South as of late, but I'm in the middle of moving cities. After next week, I'll be settled in London, and hopefully work will progress faster. I'll also be looking for contract/consulting work after next week, so if you'd like to hire me, see my 'corporate site'.

In June/July 2010 I spent ten days travelling in Rajasthan, India with friends; this is my diary of the trip (full list of entries here).

Friday 25 July

Flew out from Heathrow at 20.30 with Jet Airways, bound for New Delhi. A pretty decent (for 30,000 feet) Indian meal for dinner, then fell asleep over Russia after watching James Bond. India is 4.5 hours ahead of BST, so we’ll arive at about 9AM if all goes according to plan. All of us have had the relevant jabs and are carrying industrial quantities of suncream and insect repellent.

The trip was suggested by my friend K who was born in India and lived there for 15 years, and we plan to start by staying with his grandparents who live in Jaipur, the capital city of Rajasthan. Currently it’s reported to be 40 degrees centigrade in New Delhi; here’s hoping my system can cope…

Editor’s note: it gets a lot more exciting than the above, I promise. Parts 1 to 10 coming over the next week when I get time to write them up.

Five years after the cool kids first started jumping on the bandwagon, I’ve got myself a CurrentCost CC128 (Southern Electric send them to some customers for free, it seems – e.g. my granddad who didn’t want it).

So, with the addition of an eight quid data cable and the Linux box running in my lounge, may I present my electricity usage graphs. Bear in mind that these are (at the time of writing) for a five-bedroom house in central Oxford.

The parser for the XML output of the device I’m using is this one – just swap “COM20″ for “/dev/ttyUSB0″ in their testrun script and fix it to ignore empty lines read from the serial port, and you’re in business. I then hackdapted this RRDTool tutorial to plot the graphs.

After Transport For London released their new live departures API, and after Matthew Somerville made a Thing with it, I felt I had to have a go.Thus, I present Live Line, a visualisation of the trains on Tube lines overlaid on their schematic maps. It currently only does the non-forking lines - the line maps are drawn entirely with canvas, and there's a few code issues - but it still looks quite nice. The idea, and some of the logic, is borrowed from Matthew Somerville's idea, which uses geographic maps for a similar purpose.

For at least a year now, people have been suggesting to me that South should be in Django core.I've always resisted, for good reason - not only because of South's relative immaturity at the time, but also because forcing a single solution would, I think, not be a good thing in general. (South, admittedly, has a few design issues, but it's all about tradeoffs.) However, last week at djangocon.eu, I revealed, in rapid-fire English (apologies to non-native speakers for that, we're working on time dilation) my Grand Plan for the future of South and Django migrations in general, one that seems to have the approval of at least a few members of Django core. The proposal is not, as some expected, to put South completely into Django core; instead, South will be split into two parts. One, the part that implements database abstraction, dependency resolution, and history tracking, would be put into Django itself. The other part - autodetection, model freezing, and so forth, will remain as South. (If you'd like more in-depth details of the split, see my post to django-developers.) The idea behind this is simple - take the parts of South that nearly any migration solution would need (database abstraction especially), and move those into a place where they can be used easily by any migrations solution. More interestingly, because the proposal describes a common idea of dependencies and of what a migration actually is, it means that migrations can be written such that they need no supporting framework at all, and can be just pure Python calls to the database APIs (if Django ever ships any migrations, this is probably how they'd be done), raw SQL files (with a version per database backend if needs be), or a Python file managed by a supporting library (South 2.0 migrations), and that these different formats can all coexist, even on the same application. Some might argue that such a solution is slightly restrictive, but I see it as the job of a framework to provide some level of consistency, and by doing this, it means reusable apps can ship with migrations written using the library of their choice (or no supporting library, as the case may be). What does this mean for South? Well, in the short term, South 1.0 will be released, being mainly a polishing-off release of 0.7, and introducing a few new concepts (like the "rebase"). In the longer term, if my proposal meets with acceptance from the community (but, more importantly, those illustrious figures that make up Django's core developers), I'll start work on a branch to move the relevant parts over and firm up some APIs, and it can hopefully be finished and production-ready in time for 1.4. At the same time, I'll be cleaning up the South codebase to match, and will then release South 2.0 at the same time as the migration APIs are released with Django. South 2.0 will, necessarily, only support that version of Django - previous versions will have to stick with South 1.0, which will get maintenance fixes. I'll also ensure there's a reasonably clean upgrade path from 1.0 to 2.0. So, I'd love to hear your feedback and thoughts on my proposal. The mailing list thread is probably a good place for constructive comments or criticism. (Also, slides for my talk are available here, and there's also a video.)

The first bugfix release of South is now out, and there's plenty of fixes.It's been perhaps a little too long since the 0.7 release, so I'm pleased to announce that 0.7.1 has now arrived. It's really all small changes, as befits a point release - the release notes highlight the few slightly more major changes. In other news, I'll be at djangocon.eu next week, talking about both South up to now, as well as what I think the future holds. The jury's still out on what exactly will happen, but I have a few interesting proposals I'll be talking about, some of which are a bit more grand than others. If you're not going to be able to make it, don't worry, I'm sure you can all be subjected to me in two-dimensional video form later on. It's a once in a lifetime experience, mostly because few people are awake enough to try listening to it a second time. I'll also obviously be posting my proposals here and in other appropriate places, to gauge feedback.

I’ve just read an interesting piece over at The Register on the bloated awkwardness of Visual Studio 2010, and another on the question of whether we need IDEs at all.

The latter is a difficult question for me – on the one hand, there’s a school of thought I have some sympathy with, which says that IDEs are a crutch of the feeble-minded, and allow bad programmers to kid themselves that they’re good, because they can generate lots of code automatically and hit ctrl-space if they run out of ideas.

However.

When I started programming, six years ago (!), the first three languages I used were Turbo Pascal for Windows, PHP and Visual Basic 6. TPW was a good basic language for teaching A-Level Computing, but the built-in editor was scarcely better than Notepad. I can’t remember if it had a compile-and-run button, but I seem to recall not. PHP was slightly better – once I’d worked out how to get Apache onto my Windows machine and sacrificed a chicken to get PHP talking to it – but again, no IDE out of the box, and even the relatively advanced capabilities built into the copy of DreamWeaver  (was I the only student in the country honest enough to cough up over £100 for it?) didn’t feel up to much.

VB6, though, felt magic. In retrospect, it was a horrid language, but not only could I drop controls onto a form and double-click to generate the outline of the method they’d activate, I could actually pause and resume the code while it was running! I could see the values of variables at a point in execution, and even go backwards and forwards. The completion facilities of the IDE were basic, but they were there, and they made things much faster. Writing code to automate Microsoft Office was a particular sweet spot – run the macro recorder to generate code containing roughly the API calls you were after, then drop some VB6 control flow round them, and off we go.

Later, reading Computer Science at Oxford, the practicals we did rarely stipulated an IDE, but we nearly always ended up using gedit + the relevant command-line compiler. Certainly, the existence of IDEs for Haskell was never alluded to – either the ultimate example of clever people thinking IDEs are the preserve of the feeble-minded, or the assumption that we’d be clever enough to go and look for one ourselves, depending on how silly I want to consider myself in retrospect.

I’m happy to say that Java hardly featured at all in our courses (I’m with Joel on that one), but when it did, we were told to use BlueJ, because, Mike Spivey explained, “it has only two buttons, and Eclipse has hundreds of others you don’t need”.

Given the short length of the practicals involved, I only paused to think “pish, how many buttons can it have?”, but I didn’t feel the need to find out for myself until I started writing Java for a living. He was right, there are hundreds of them. Despite which, I use Eclipse every day at work, and would never dream of trying to write code without it. It is, irrefutably, a big, bloated beast, but when you’re working on serious real-world Java, with version control, coding standards, complicated dependencies, hundreds of packages making up one program, and spend far more time reading and debugging code than writing it, you really do need the beast on your side (or so I believe).

So that’s it, then – I’ve converted to the world of IDEs? Well, not quite. The other language I use on a day-to-day basis – though mostly for pleasure rather than business – is Python. And I don’t usually use an IDE, simply bashing out code in Notepad++ on Windows, or KDevelop on Linux. OK, so KDevelop is sort-of an IDE, but it’s very lightweight.

Of course, Python being interpreted rather than compiled makes it easier to just fire up your Python program from the command line after editing it. And that, really, gives us a clue as to the only sane conclusion of the IDE debate: it’s the same as the programming language debate. There are tools (languages and IDEs) and there are jobs. Good programmers pick the best tool for the job, and for a compiled language as verbose as Java, an IDE arguably makes things faster. For Python, on the other hand, it’s not essential (IMO), but it depends on the tastes of the individual.

It’s amazing how many fewer afternoons I seem to spend hacking around on my servers these days. Perhaps I got a life; I certainly got a full-time job. I have however sorted a few long-standing bits and pieces out today…

dnorth.net is now available over IPv6

As are its various satelite sites and www.saintcolumbas.org. Sorry, no, there is no bouncing logo to reward those of you viewing them via such.

A backup system that doesn’t Totally Suck™

I’ve finally retired my creaking “run a shell script to rsync them onto my laptop when I remember (i.e. every six months)” manaul backup system in favour of an encrypted LVM partition on my home server, and rdiff-backup to make nice incremental backups of everything on a nightly basis. The instructions on how to do it are all out there on the interweb, and it’s not too difficult, fortunately. I’m a bit disappointed that backupninja doesn’t support remote rdiff-backup, but I guess I should submit a patch if it bothers me that much…meanwhile, my wrapper script seems to work just fine.

After months of hard work, refactoring, blood, sweat, tears, and improvement, South 0.7 is ready.This release has been probably the biggest internal change in South since it first started. We've removed a lot of old code, and significantly refactored the migration-creating code to make it more extensible (as well as not being mostly in one 3000 line file) There are also a few new user-facing changes: New command names.The old startmigration was getting a little overloaded and wasn't too well-named, so it's been split into schemamigration and datamigration commands. There's also a new graphmigrations command, for those with complex dependency sets. No more issues with missing defaults. South realises you need a default for NOT NULL columns now, and prompts you if you forget to specify one as you're making the migration, instead of dying when you try to apply it. Django 1.2 and MultiDB support. As well as supporting the latest and greatest incarnation of the universe's best web framework, 0.7 also has some limited MultiDB support, in the form of a --database option (that works like the option of the same name on the new syncdb command in Django). Custom fields are no longer magically used. Instead, you now have to actually tell South about them. It's an ease-of-use regression, which I'm not too keen on - but it's needed to stop things mysteriously failing in your migrations a few weeks down the line, and there's a whole new tutorial section on it, as well as plenty of reference documentation (and IRC and the mailing list are always around to help). If you want more information, read the new long-form release notes, which goes into a bit more detail on the various changes, and mentions a few I haven't included here. There'll be a lull in development after the release as I rest, and then work will begin on the next release - 1.0*. Which features are going to appear have not yet been decided, but it's highly likely column rename support and a fixturemigration command will be among them. Anyway, why are you waiting? Go install South 0.7 now! * South has an octal numbering system.

It's that time of the year again, when a new South release rears its well-refactored, database-independent head.If you've been following the development of South at all, you'll know that 0.7 has been quite a while in the making. I've been busy working away at a big code refactor, along with Simon Law, who has gained the title of "Knowing Too Much About Migrations". However, there are some user-facing changes as well, some quite significant. While my release notes go into some detail on what's new, the major changes include: New command names.The old startmigration was getting a little overloaded and wasn't too well-named, so it's been split into schemamigration and datamigration commands. No more issues with missing defaults. South realises you need a default for NOT NULL columns now, and prompts you if you forget to specify one as you're making the migration, instead of dying when you try to apply it. Django 1.2 and MultiDB support. As well as supporting the latest and greatest incarnation of the universe's best web framework, 0.7 also has some limited MultiDB support, in the form of a --database option (that works like the option of the same name on the new syncdb command in Django). Custom fields are no longer magically used. Instead, you now have to actually tell South about them. We're working on docs to make this a bit easier - it's an ease-of-use regression, which I'm not too keen on - but it's needed to stop things mysteriously failing in your migrations a few weeks down the line. As I said above, read the new long-form release notes for much more on the release. If the release candidate proves to have sufficiently few bugs that I can fix them in time, 0.7 will be released next Thursday, the 25^th March. So, please, install the release candidate and try it out!

It’s been over a year since I deployed Django in production, and I wasn’t looking forward to it. Last time, I had a lot of trouble with mod_python, sessions and decimal objects refuising to pickle.

Thankfully, all this really seems to have grown up in the last year – mod_wsgi is now the recommended way of deploying Django in production, and following the mod_wsgi django instructions, I was in business in 20 minutes. No fuss, no mess, no drama, and best of all, using daemon mode, no noticeable performance hit when serving static files and PHP off the same Apache installation. The ability to run the django project as its own unprivileged user when using daemon mode is also real handy.

2010 will be a bad year for IPv4 – this is exactly why I designated native IPv6 support as fundamental, not merely a “nice-to-have”, when setting up Splice last July. Hats of to Bytemark for supplying IPv6 with their hosting. I’m sure the fact that it’s excluded from their SLA is something that won’t be the case in eighteen months’ time, and meanwhile, I and the rest of the crew are using it quite happily.

Footnote: those of you reaching for the comments button to sarcastically remark that this website appears to not be available over IPv6 will be pleased to hear that I intend to fix this in the immediate future.

My e-mail still isn’t as shiny as I’d like. In particular, my use of Exim Filters to sort incoming mail into folders lacks the ability to mark messages as read (although it’s still miles ahead of the dreaded Procmail). This would be handy for high-traffic mailing lists which I don’t have time to read on a daily basis, but which I find it hard to ignore the “unread” icon next to the folders for.

One day, I should probably move to using the Dovecot LDA and its sieve implementation, which supports the “imap4flags” extension, thus allowing marking messages as read, making them turn purple in Thunderbird, and all sorts of other cool stuff. Sadly, life (or this afternoon) is too short.

In the meantime, I’ve solved the problem in the usual way I deal with life’s imperfections: gratuitious Python – http://pastebin.org/77199 run from a crojob every five minutes.

(Disclaimer: letting scratty little bits of Python anywhere near something as important as your e-mail is probably a Very Bad Idea.)

Earlier this year, I hacked together a stateless RSS reader called Bogroll.

It’s been doing sterling service for me at news.dnorth.net ever since. Today, I’ve sorted out a 0.2 release with the following improvements:

• Now caches etags/Last-Modified headers to avoid fetching a feed if it hasn’t changed since last time (thank you, Mark Pilgrim, for chapter 14 of Dive Into Python 3, which reminded me to be a good citizen in this regard). I was pleased to discover that the Universal Feed Parser it’s built on top of already supports gzip and deflate compression to save bandwidth.
• Now supports just one category per feed, because having articles appear in several categories just seems wrong to me
• Each category now really does contain the most recent X articles from the relevant feeds, because I’ve fixed the severely broken sort-by-date logic

A fair bit of refactoring has gone on under the hood, and the code now looks a bit more like an app and less like a ten-minute bodge. The next round will involve getting some proper unit tests in place, and possibly AJAX magic to load the articles lazily on the page.

You can download the 0.2 zip, or get the latest version from subversion if you like to live dangerously. The cool kids all seem to be using Git or Mercurial these days, but I haven’t found the need (or overcome the inertia) yet.

Enjoy. Feedback welcome to the usual address.

So the turkey is (at least partly) eaten, the mince pies are disappearing fast, the wrapping paper has been picked up off the floor and the presents played with. 2009 is done.

It’s hard to say what I’ll remember 2009 most for, because it’s been such a packed year for me. Perhaps I’ll remeber it as the last of my three happy years at Magdalen, the year I finished my degree, the year I set up a hosting co-operative with six friends, the year I moved away from home, the year I got a job or the year I took over as joint church treasurer.

It wasn’t a perfect year, of course – my personal TODO list is still 39 lines long – but never mind, I’m fairly sure there’s another year just around the corner. One item a week. What could possibly go wrong?

Happy New Year to all 3.5 of my readers (and all those following along via Facebook; do come and read the website this is all automatically imported from), and I look forward to seeing as many of you as possible in 2010.

During the planning of our Christmas card at work this year, a mad idea came up. Do we ignore mad ideas? No, we tackle them head-on.

The idea was to build an internet-controlled snow machine - you'd hit the button on the website, and watch a member of the Torchbox team get pelted with snow. When we first came up with the idea, we dismissed it as being "too complex", but after a while, we came around.

Cue three days of frantic development and phoning round to get the parts. I'll be posting a full build article, with all our source code, once we're done.

The brief summary is that we have a Django app which handles rate-limiting of snow, and tracking who has clicked the button, which then communicates with our snow machine using Artnet and DMX. Ustream is used to stream the video back to the internet.

Still, I imagine you want to see it in action, so head over to snow.torchbox.com, and have a go. It's only online 10 - 5 UK time, and only until Tuesday (we can't fill our office up with paper snow forever), but it's still very good fun, even if you don't know any of us!

Let’s be honest about this. Looking back, I should have known better. Nevertheless, as I describe the problem that ate far too many hours of my weekend, judge for yourself whether I was entirely to blame for How It Went…

The problem

A friend of mine has a laptop. It’s about five years old, and it runs Windows XP. This means, inevitably, that it’s a mess. My personal metric of measuring how rodgered a machine is by the number of icons in its system tray gave it a ten, and that’s pretty nasty. Nevertheless, until last Friday, there was nothing wrong with it that an uninstall fest followed by a defrag wouldn’t have fixed.

Enter the Internet Man

Last Friday, a chap called round to set up some broadband for my friend, who’d previously been on dial-up. I’m not going to name the ISP concerned, since it gives me more freedom to say nasty things about them, but suffice to say that they’re big enough that they really should have done better.

Their engineer fixed the simple(ish) problem with wiring which was stopping the broadband from working, and what he should have done at that point was connect the laptop to the router (less than six inches away on the same desk) using the supplied ethernet cable, and left. What he actually did was shove the supplied CD in the drive, which helpfully installed a few hundred megabytes of crapware onto the machine, then hooked it up to the broadband via wireless. Then he left.

Despite the above totally unnecessary shoving of stuff onto it, the laptop struggled manfully on (system tray count now up to 12) and seemed superficially fine.

Enter Dragon Naturally Speaking

My friend makes extensive use of Dragon NaturallySpeaking, a voice-recognition product which seems to knock the socks off everything else on the market when it comes to actually coping with different accents. And it was here that the problem first manifested itself: trying to use Dragon to voice-control Internet Explorer caused it to crash with an error message along the lines of the one described in this Microsoft knowledgebase article.

Enter the sucker, stage left

At this point, I fetched up, and agreed to see if I could fix the problem. Although the above KB article looked ideal, being the first hit on Google when I exercised the too-useful-to-be-documented Windows feature of “Ctrl-C copies the text of the active dialog box to the clipboard”, the hotfix it supplies claimed to be already present in service pack 3 of Windows XP. Just about the only virtue of this laptop was that it was fully up to date on patches and service packs, so what now?

Don’t press that button

Being rather short on ideas at this point, I decided to fire up the nearest thing broken Win32 boxes have to a magic bullet, namely System Restore. The machine refused to roll back to any of the restore points at first, but restarting into safe mode fixed that, and it was soon rolled back to the Friday, at a time before the problem occured.

Unfortunately, Dragon now seemed completely broken, giving the error message described in this support article. And no, of course there were no backups, I hadn’t taken one before I started, and my friend is no different to most of the non-geeky people I know in not backing up, except for dragging his holiday snaps onto CD once every six months.

To their credit (and they’re about the only players in this story to be awarded any), Nuance’s suggestion in the article of how to manually restore the user files for Dragon did work, after I rolled back the fatal system restore [or rather, didn't, because it didn't seem to have made a pre-restore restore point. Fortunately picking one from the Saturday seemed to work].

So, after two hours feeling my friend’s anxiety at the thought of having to retrain the speech recog from scratch – not a pleasant accompinement to the sick, swoopy feeling we get when we know we’ve just permanently erased some irreplaceable data – we were back at square one with the original problem. One last shot in  the dark, disabling the ISP’s nasty extensions to IE, seemed to fix the issue.

So who do we blame here? Laptop vendors, for selling machines so laden with rubbish before they even leave the factory that DLL hell seems assured the minute they meet with real life usage? Or, we could blame the idiots who seem to write the nasty unsigned drivers for most hardware on the market. We can definitely blame the ISP’s engineer for installing the crapware, but perhaps he’d been trained to, and anyway, why do ISPs think we need a CD full of crap to supplement the TCP/IP standard that’s been around for several decades? Is it really asking too much of Johnny User to plug in a cable or enter some Wifi passwords in to the applet that’s sodding well supplied with Windows, thus making the poorly writtten replacements from laptop manufacturers and ISPs alike completely superfluous? We could also blame Microsoft for making system restore not clever enough to cope with software like Dragon. Or possibly blame Nuance for not fixing or documenting what has apparently been a known incompatability for several versions of Dragon*.

Certainly, we can blame me. I clearly need to have “I will not agree to even slightly ‘fix’ someone else’s computer without taking a full disk-image of it first” tattooed across my forehead. I also clearly need to reimmerse myself in the happy world of properly written software which I’m lucky enough to earn a living in and try to forget the horrors of the last 48 hours.

* The most useful reference I could find on Google was this page. The Nuance KB doesn’t mention system restore. Then again, perhaps their customers simply don’t know this is the cause of the issue, or don’t get round to reporting it. It’s not like I have.

I’ve long been of the opinion that Firefox and Thunderbird are a bit like democracy: aclaimed far and wide as major achievements and bastions of a civilised society, but actually, honestly, a bit crap in many ways. Sadly, we’re stuck with all three until someone manages to come up with some compelling alternatives.

If you’re reaching for your e-mail client at this point to tell me I’m being unduly harsh, look me in the eye and tell me you think the way Firefox cheerfully caches DNS lookups and ignores such things as TTLs is a good idea. Or, even harder, give me one good reason why Thunderbird doesn’t check all IMAP folders for new messages by default. You can’t; in both cases it’s a disgrace.

All this being the case, I wasn’t what you’d call hopeful when, last Sunday, I had to write some e-mails on a train. And to do so, I needed to refer to some other e-mails in my Inbox. Since said train lacked anything as useful as a wireless internet service*, I’d need some sort of offline IMAP facility**.

As I bashed Thunderbird offline IMAP into Google, I was expecting a half-baked plugin at best, and “can’t be done” at worst. What I was actually very pleasantly surprised to find is that this functionality is built into Thunderbird.

What astonished me even more is that it actually worked. Faultlessly. So perhaps democracy can be salvaged after all.

* And, let’s be honest, because I still haven’t got organised and bought a phone with internet capabilities.

** Nobody uses POP3 in the twenty-first century, right?