Ok, so I thought I was long past this "hey the server blew up" type posts, but...
Hey, the server blew up :(
About 11:30 last night it went down, and this morning myself and Fred went in early and fought with it but the thing wouldn't even POST. So we luckily have the current server, with a mirror backup from 2am the night before (the crash was inconsiderate enough to happen just before the nightly backup).
So the last day has been fixing what I can and getting services slowly up and running. Here's the status of various bits:
It's been a hell of a day for me even with having a backup server and a full mirror of the original server ready to go on it, mostly due to that the new server is Ubuntu 10.04 not 8.10 (old server) so lots of little compatibility errors are cropping up.
My main concern now is the mail, and I hope to have it up tomorrow after at least a few hours sleep.
Keep an eye for things broken if you have services running on the server, and email me or tweet me at @arcterex or @ufies if you find something.
Thanks for your patience.