Just a quick note to welcome my friend Cthulhu who will be keeping things up to date here on the site while I’m away for a week for a work trip. Make him feel welcome and ignore his anti-iPhone rants (which will no doubt be posted as soon as I leave :)
Sorry for the unexpected downtime this afternoon. My partner in crime and maven over at TDI Club installed a new CPU and more RAM, so now the server is running a whopping 16 cores and 32G of RAM. A lot of power to server up some static content huh? :)
Many thanks to Fred for his hard work and hardware.
Bit of shameless self-promotion here, I blogged about My Favorite Photography Podcasts today and thought ya’ll might be interested. If you’re into Photography this might help you out!
Just wanted to say a huge Merry Christmas from the team here at the UFies.org mega-conglomerate. 2012 has been a great year. Almost 900 posts were written, and more than 21,000 unique visitors hit the site and were (hopefully) entertained or at least redirected to somewhere interesting.
I have a confession to make… this isn’t really a mega-conglomerate. It’s not even a mega-corporation. It’s just me, one guy, posting things I find interesting day by day. I don’t do it for money (I have adsense running on the site, but my earnings for the last year were $121), or the fame (the most I can say is that I’ve gotten a bit of minor notoriety), but because I spend the day finding interesting things online and like to share.
For the upcoming year, here are a few things I’m planning:
- To bring the site out of it’s 2002 era design, and perhaps explore a few web design philosophies that were invented in this decade. Maybe give that new “Cee Ess Ess” thing a try. Responsive design to avoid the horrible mobile site would be great too.
- Upgrade the backend CMS. I’ve used Moveable Type since 2002 or so, and moving to either Wordpress or even a newer version of MT would be great.
- Site improvements like better search, some category reorganization, tags, etc.
- Dedicate myself to actually making the site interesting to look at. Images with posts for example.
- Actually survey my users to find out what they want. Yes, that’s right, soon the first annual UFies.org user survey will be showing up on the site to find out what sort of things interest you.
(I don’t have a lot of budget for the first item, so if you’re a designer who feels generous, or would be interested in some sort of a services trade arrangement, please email me (as a photographer though I know that creative people really don’t love being asked to do stuff for free)).
Long story short, I value being able to write on the site and post interesting things, and I appreciate my readers for continuing to make the site a success.
No email for the last few hours? The fact that that list.dsbl.org is offline might have something to do with it. The issue started around 3pm PST thursday and (as far as I can tell) any mail sent to servers using this blacklist would have been bounced (not delayed) :(
As an aside, I use(d) this on UFies, so if you have a mail account here, this is why you haven’t gotten email from last night :( Sorry folks.
Apparently the server has been unloved and closed for years now, and they finally got tired of still getting requests and made it reject all mail so that lazy sysadmins (ie: yours truly) would actually do something about it. sigh
So what’s the canonical list of good RBLs? Or does everyone just use gmail or some other hosted service?
Well, it took longer than expected, but the ufies.org server has moved to it’s new home on…. well, to you folks everything will look exactly the same, but be faster (hopefully). Anyone logging into the server please use ufies.org as the address, as if you log into the old server no changes will take effect. Many thanks to Fred and tdiclub.com for providing the hardware and the patience with me debugging ARP caches, bad kernels, and the same :)
As always email me or tweet me at @ufies if you see anything wrong.
Just a bit of site housekeeping... I discovered you can get a free (but legit) SSL certificate from StartSSL, so I installed one on the server for the webmail. If you're a UFies.org webmail user, you may be prompted that the server certificate has changed, but you'll see it's no longer a self-signed certificate.
Please email me if you see anything odd about it.
We now return you to your list of random links and fun for the day.
No, not the demise of the site, but I'm finally getting a bit of a break so I'll be staying offline as long as I can, so please don't be surprised if nothing really is posted here in the next week or so.
In the meantime though, send me any interesting stories as usual (easiest way is either directly to me @arcterex or the site @ufies on twitter or by email). If you're bored head over to a cool link blog like Reddit or Digg.
If you're interested in helping this site not get stale when I'm not around by being a writer or contributor, and have an idea of what the "vibe" of the site and the sorts of things that are posted here, please email or tweet me, I'd love to have some help.
Ok, so I thought I was long past this "hey the server blew up" type posts, but...
Hey, the server blew up :(
About 11:30 last night it went down, and this morning myself and Fred went in early and fought with it but the thing wouldn't even POST. So we luckily have the current server, with a mirror backup from 2am the night before (the crash was inconsiderate enough to happen just before the nightly backup).
So the last day has been fixing what I can and getting services slowly up and running. Here's the status of various bits:
- Apache is up and running, most sites seem to be up and running, with exceptions of course. Email me or tweet me at @arcterex or @ufies if you find things broken.
- User logins and home directories are up and running (look for any files in your home dir not owned by you though, some uid/gids might be messed up)
- Incoming mail - working
- Outgoing mail (ie: sending mail from the ufies server out) not working, seems everything gets a "relay denied" error, and I have no idea why (still)
- MySQL database is up and running. Check your tables for errors though.
- FTP is running
- DNS is running
It's been a hell of a day for me even with having a backup server and a full mirror of the original server ready to go on it, mostly due to that the new server is Ubuntu 10.04 not 8.10 (old server) so lots of little compatibility errors are cropping up.
My main concern now is the mail, and I hope to have it up tomorrow after at least a few hours sleep.
Keep an eye for things broken if you have services running on the server, and email me or tweet me at @arcterex or @ufies if you find something.
Thanks for your patience.
Just an FYI that I've done a much more mucking around with the mail server on ufies.org. If you're one of the folks who does have an account there, please keep an eye out for any mail being rejected improperly, or anything out of the ordinary. Seems one of the configurations that had a whole bunch of awesome anti-spam stuff in it was being superseded by a duplicate line farther down the file, but that one didn't have the awesome anti-spam stuff. Fingers crossed this will make things better....
If you do see oddness, please email me at alan at ufies dot org or if that's broken, arcterex at gmail dot com (or if you're all web 2.0, hit me up at @ufies on twitter).
I've made a couple of minor tweaks to the mail server as UFies.org seems to have gotten on an RBL list :( Not much we can do about it it seems, though I might be able to fake things out through our second server's IP. Please keep an eye and let me know (posting here if needed) if you're having any strange mail issues and have an account on the server.
Buddy of mine told me about Woopra today for site tracking and analytics, and since I've been getting a bit more serious about improving the site for people (yes, I'm working on getting something that doesn't look like it was designed in 1998) I've set this up. Please let me know if you see any issues with page load, responsiveness, etc.
I just upgraded UFies version of Spamassassin which came with the following notice, so take heed shell users:
This version of SpamAssassin fixes a bug which caused mails sent
in 2010 to be flagged as suspiciously spammy. If upgrading to this
version, you are recommended to update any per-user caches previously
created by sa-compile, and to check mail already in your spam folder
for false positives more carefully than usual.
Wow, it's going to be weird saying/writing/typing "2010" for a while. Also what do we call it? "twenty ten"? "two thousand ten"? "oh ten"? "two thousand and ten"? Someone let me know please, I don't want to be one of the un-cool kids saying it all wrong.
Anyway, welcome everyone to the new year, I hope that your Christmas and holidays was fun, relaxing and spent with family and friends. I know it can be a bit of a grind getting back into work in the new year (speaking from experience, 5:30 am comes really early when you're used to waking up at the crack of noon), but I hope that I can (continue to) throw a few random links your way during the day to keep you amused and thinking. I figured I'd throw a few random thoughts up here about what I hope the coming year will bring.
The Server
The server setup is pretty strong. There are two machines running, the main server and a backup machine running with a mirror of the current server on it. In the coming year I want to set up the database to be properly master/slave so in the event of a crash there's less chance of losing any data, and also setting up the second server to be properly hot-swap (or at least minimal manual interaction) instead of needing someone to go and change and copy things all around to have it working. The big thing though is it's there so (in theory) nothing will be lost. Anyone storing important data is encouraged to do their own backups and database dumps and store them off site just in case.
The Site
Hopefully I'll be able to put up a new look and feel and features. I'm guessing that most people mostly view the site through RSS from Google Reader, Bloglines or the like, so they never see the site. Lets be honest though, the UFies.org site looks like the holdover from a bad 2002 site built by a programmer because it is. Any graphic designers out there who are interested in some practice please contact me if you think you can make the site look nicer.
I'm going to continue to use the @ufies twitter account to tweet out stories and server updates if you get your news that way, as well as getting a bit more into the web 2.0 world of "conversations", social media, fan pages, and the like. For example, I noticed while hitting the Internet archive that UFies wasn't included in 2009 at all, so I have to go and rectify that :)
Myself
Outside of my own (numerous) personal goals for the new year, I want to keep on posting things of interest. I've sort of disconnected from the Userfriendly community that this site was originally set up for back in 1999, and I want to get back to the roots of the site, and find out what the few readers I have are interested in so I can better serve the people who I risk my job every day to post random links of interestingness.
That's my diatribe for this morning, there's a pot of coffee in the kitchen on the other side of the office calling me. All the best to you and yours in this new year, hopefully it'll be "your year" (or at least suck less than the last one), and I now return you to your random collection of fake Apple slate rumors and video game updates!
tl;dr everything is fine, things continuing as normal.
Look like there is a bug in Spamassassin in the check for "dates grossly in the future" which defines among other things, 2010 as "grossly in the future". The details are here. I've updated the ruleset on UFies.org though, but please check your spam boxes if you use SA for your spam checking.
Just a note that these appear to be continuing and we are working to resolve them.
Update: Looks like it was a rate limiter kicking in for the bandwidth, meaning that a semi-free ride for server bandwidth is over. However things are resolved now and we're back up to speed. As always donations are appreciated (paypal link in the sidebar).
No matter where you are or what you are up to today, hopefully you'll have time to sit back and give thanks to those who gave their lives to keep the current freedom that we enjoy in the world today. For me it was my grandfathers, from different sides of World War II, living in completely different worlds, but both of whom fought in their own way. One I ended up knowing, one I only heard about in stories from my dad. Peter, I wish I could have met you to thank you.
And of course, in UFies.org semi-tradition.
In Flanders Fields
By: Lieutenant Colonel John McCrae, MD (1872-1918)
Canadian Army
In Flanders Fields the poppies blow
Between the crosses row on row,
That mark our place; and in the sky
The larks, still bravely singing, fly
Scarce heard amid the guns below.
We are the Dead. Short days ago
We lived, felt dawn, saw sunset glow,
Loved and were loved, and now we lie
In Flanders fields.
Take up our quarrel with the foe:
To you from failing hands we throw
The torch; be yours to hold it high.
If ye break faith with us who die
We shall not sleep, though poppies grow
In Flanders fields.
Well, as suspected it took a bit longer than advertised, but the server is back up and running. Please let me know if you see anything out of sorts.
Hey everyone, Looks like the UFies.org / TDIClub.com hosting provider is moving their offices and merging with another company. One of the fun bits with this is their current office is closing, and they're moving all their hosted servers to a new location. Luckily the new location is just down the road from the old one, and they are willing to do the move of our servers for us.
What does it mean for you, the user? Well, not much. Basically there will be some downtime where nothing will work and you won't be able to get your mail, website, webmail, or anything like that.
When will this happen? This is scheduled for approximately 21:00PST (that'd be 9:00pm) tomorrow, Thursday the 8th. I'm told about 20 minutes, but somehow I doubt that, so I'd wager 1-2 hours of downtime for everything, mail, DNS, web, etc. Sorry about this.
That's it? Yup.
Will anything go wrong? Well, the IPs are supposed to be all set up at the new location, so they will just be powering down the machines, packing them up, and plugging them back in. In *theory*, when the servers come back up all will be back as it was.
Anything else? Well, not currently. We're getting pretty much the same deal as before, same cost, bandwidth, throughput, and support (pretty sure about the last part anyway). However, it is another company and they may want to integrate us into a new pricing SKU in the future. We'll see what happens when the time comes... I'm going forward hoping the best though.
Please follow the UFies.org twitter stream for any notifications @ufies (as well as interesting links throughout the day) where I'll be posting any updates if things do go south.
Just a quick note to say that after a busy long weekend, the UFies.org staff (that'd be me) is back at it, and ready to face the post-labour-day week, kids going back to school, and so on. Hope everyone had a fun and safe weekend and is all ready to get back to the grind! Hopefully I'll be able to entertain you just a wee bit during the day. Thanks again to the couple of thousand of you who visit the site every day. Remember that the site is now iPhone friendly (if you're visiting on the go) and there's also an auto-twitter that you can follow at @ufies.
Have a great week everyone!
Just a quick note that the UFies.org mail server has switched from a long list of probably-not-all-correct mail servers to a shorter but better list sent over to my movie reviewing buddy from an ISP. Hopefully this will cut down on spam coming into the server. Please keep an eye out for any emails you miss and email me if there are any, and I can see if they were accidentally eaten by the system.
Ok, so I did do a quick upgrade to Movable Type 4.3, and for the most part, it was pretty painless. Most of the pain associated was that the templates for the site are not quite of the same ilk that the modern software is setup for, so I need to figure out how to go and slice and dice this lovely late 90's era design into the more modern header/footer/CSS style.
A couple of new things you might notice:
Thanks for your patience, assuming you're up at 10-11 or so PST!
Just a quick note to anyone who uses UFies.org for their mail. I have messed around more with the SMTP and SMTP auth settings, and (in theory) SMTP auth can now properly be used for sending outgoing mail (again, only if you have a UFies.org account). Just make sure the following settings are in your mail client:
- Port is 25 (or 587 if your ISP blocks outgoing 25)
- Use username and password is set (and username is your own)
- STARTTLS (or in Thunderbird in Windows, "Use TLS if possible") is set
Email me (hehehe) or leave a reply here if there's anything wonky.
Yesterday evening, the UFies server expanded to two servers (yay) which will give us a server to use for the database and for backup. However at the same time (well, a couple of hours later) the main server suffered some stunningly catastrophic hardware failures. The RAID array lost 2 of the 4 drives, and the hot spare never worked. Luckily there is a nightly backup which Fred and I managed to get access to (after some heart stopping "no such device" messages).
I spent today and this evening transferring over settings and data and reconfiguring all the old system stuff (from a gentoo install) to work on the new server (a ubuntu install).
Probably 90% of that is done now. User accounts have been moved over, along with data, mail appears to be flowing, DNS, ftp and some websites are back online. Please email me at alan@ufies.org if you see oddness with your stuff.
Sadly not the controlled transfer from old server to new we were wanting, and no chance to clean up dead user accounts and websites, but better (way better) than losing everything (again).
Remember folks, backup regularily!
Things still needing to be fixed:
-
webmail- working other than some php errors, need to upgrade to a newer squirrelmail version
-
https
- some tweaks in pop/imap
- smtp-authentication
Let me know if I missed anything.
Our provider had it's fiber line(s) cut accidentally by the city of Vancouver (thanks!) and thus the server was off the grid for a few hours today. Things are back up and appear to be all good now. Sorry for any inconvenience!
In case anyone is wondering what happened to the server around 1pm today:
# uptime
14:00:30 up 118 days, 2:48, 5 users, load average: 235.87, 218.42, 206.62
Still investigating what happened, but the box is back and responding now.
Isn't it awesome that the server has been so stable lately? Really, it has been... the various blips in service lately have been all hosting company related (other than the swapped hard drive a few weeks ago, which was a planned outage)....
So sometime yesterday afternoon a router or switch at the hosting company blew up, and continued with odd problems (DNS related, looking up subdomains and such) until 11pm or so when I determined there was nothing I could do, it wasn't my fault, and I figured it'd all be magically fixed in the morning.
And like some Christmas miracles, it was! :) Still waiting to hear what the issue actually was though.
As it seems with all hardware issues when I deal with them, a 10 minute job turned into more like an hour. This time the causes were:
- me figuring out how to get the drive out of the server itself
- me making a typo in the boot config for the new kernel
- me forgetting to compile the newly renamed SATA drivers in the new kernel
- me recompiling the kernel to load in the new modules
- the system taking ages to boot because all the drives had to be
filesystem checked as the last time it rebooted was 253 days ago :)
Other than those minor details we're back up and going on a modern kernel and a proper backup drive. Thanks for your patience everyone!
Hey guys.... today UFies blew a hard drive, just the backup drive, so no worries, but I'll be shutting her down (253 days of uptime(!)) tomorrow (Wednesday July 29) sometime in the 11:00-12:00 PST timeframe to swap the new drive in and maybe test the latest kernel to make sure things all boot properly. Downtime should be minimal, under 30 minutes.
During this time all services will be unavailable.
Sorry for the inconvenience.
Hey everyone, hope you're having a fun New Years Eve. From all of me here on the UFies.org team, hoping you have all the best in 2008! It's been fun posting random links all 2007, and I appreciate your support (the clicky clicky you know what wink-wink). I am looking forward to bringing all 3 of you random amusement in the coming year.
-- Arcterex
Old Post... - Guess this never got posted cause I didn't read the part in the manual about scheduling future posts properly.... OOops! Here it is for your enjoyment before I start posting again tomorrow.
Well I'm off for a bit, my first vacation since 2001, I'm very excited. I've turned off comments for non-authenticated users who don't have typekey to keep the comment spam at bay, sorry if this screws people up.
I've left some good folks to amuse you while I'm gone. Fred, Darren and Iambe will (or probably will) be posting here in my absence, if not, well, you get to amuse yourselves :)
While I'm gone you can hit my daily time-sinks:
- Slashdot
- Digg
- Reddit
- Planet GNOME
- Planet Gentoo
- DPReview Forums
- My shared RSS items (RSS feed)
- The Show with Ze Frank
- Shacknews
- Perez Hilton
- Fark
See ya'll in a week!
I've added some super-magic postfix spam fighting rules, mostly RBL checks, based mostly on these pages. Please email me at alan ayt ufies dawt org if you find mail is broken or not going through for any reason.
Other than that, you should just see less spam in your inboxs, and my bandwidth costs (*cough*points to donate button*cough*) should be less! Everyone wins but the spammers, which means that everyone wins!
The server was down from around 11pm last night till 10am this morning. Sorry about that. Sounds like there was a power glitch or something was loose or... something. Anyway, everything is up and going again now.
Apparently there were some power issues last night at the colo, which were resolved around 11am this morning. Appolagies to all.
And now you can go back to your daily Halloween bad costume linkage :)
Currently there is an issue with the UFies.org upstream provider data-fortress, preventing access from some locations (ie: you probably can't get to this page to read this if you're in the lower mainland), and possibly affecting mail. I'm told they are 'investigating'.
Update: Should be all back up...
Just FYI, php was upgraded to version 5 last night.... only causalty is the webmail which gives an error message when you send email.... however, the mail is still sent. I'll fix this tonight.
Update: Upgraded squirrelmail and it looks like that fixed things. Let me know anyone if you have issues.
Linux master 2.6.15-gentoo-r5 #1 SMP Tue Feb 21 17:49:33 PST 2006 x86_64 Intel(R) Xeon(TM) CPU 3.40GHz GNU/Linuxdecides to randomly kill off some processes (apache, mysql.....). This was an occassional annoyance, but now it's almost a nightly thing and for those of us who receive the alerts (luckily not me :) it's a real pisser. The system has gobs of ram in it and we use pretty much no swap:
master cron.daily # free -m
total used free shared buffers cached
Mem: 5969 5514 455 0 296 2490
-/+ buffers/cache: 2726 3243
Swap: 1906 0 1905
Out of 6G of ram, 2.7G is used with the rest holding cached data. No swap is used. Fred dug up this post indicating this might be a 2.6.15 bug on AMD64 and I'm wondering if just upgrading to 2.6.16 2.6.17 (which just went stable) would fix it all. Anyone have any similar experience? Can you lend a hand?
Read on for the nitty gritty details of the kernel dump.
oom-killer: gfp_mask=0xd1, order=0
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
cpu 1 hot: low 0, high 0, batch 1 used:0
cpu 1 cold: low 0, high 0, batch 1 used:0
cpu 2 hot: low 0, high 0, batch 1 used:0
cpu 2 cold: low 0, high 0, batch 1 used:0
cpu 3 hot: low 0, high 0, batch 1 used:0
cpu 3 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:183
cpu 0 cold: low 0, high 62, batch 15 used:51
cpu 1 hot: low 0, high 186, batch 31 used:176
cpu 1 cold: low 0, high 62, batch 15 used:48
cpu 2 hot: low 0, high 186, batch 31 used:158
cpu 2 cold: low 0, high 62, batch 15 used:21
cpu 3 hot: low 0, high 186, batch 31 used:175
cpu 3 cold: low 0, high 62, batch 15 used:47
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:168
cpu 0 cold: low 0, high 62, batch 15 used:11
cpu 1 hot: low 0, high 186, batch 31 used:156
cpu 1 cold: low 0, high 62, batch 15 used:11
cpu 2 hot: low 0, high 186, batch 31 used:23
cpu 2 cold: low 0, high 62, batch 15 used:1
cpu 3 hot: low 0, high 186, batch 31 used:110
cpu 3 cold: low 0, high 62, batch 15 used:12
HighMem per-cpu: empty
Free pages: 510320kB (0kB HighMem)
Active:329243 inactive:1025889 dirty:193094 writeback:0 unstable:0 free:127580 slab:31606 mapped:212907 pagetables:3911
DMA free:12kB min:16kB low:20kB high:24kB active:0kB inactive:0kB present:10992kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 3512 6037 6037
DMA32 free:286540kB min:5776kB low:7220kB high:8664kB active:760260kB inactive:2358736kB present:3596384kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 2525 2525
Normal free:223768kB min:4152kB low:5188kB high:6228kB active:556712kB inactive:1744820kB present:2585600kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 1*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 12kB
DMA32: 46049*4kB 8683*8kB 561*16kB 211*32kB 8*64kB 8*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 286540kB
Normal: 39766*4kB 5398*8kB 989*16kB 50*32kB 14*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 223768kB
HighMem: empty
Swap cache: add 9900, delete 9900, find 4865/5619, race 0+0
Free swap = 1951164kB
Total swap = 1951888kB
Free swap: 1951164kB
1703936 pages of RAM
175694 reserved pages
794922 pages shared
0 pages swap cached
Out of Memory: Killed process 24143 (apache).
Out of Memory: Killed process 25925 (apache).
Out of Memory: Killed process 24297 (apache).
Out of Memory: Killed process 25934 (apache).
Out of Memory: Killed process 25961 (apache).
Out of Memory: Killed process 25924 (apache).
Out of Memory: Killed process 25960 (apache).
Out of Memory: Killed process 16420 (python).
Out of Memory: Killed process 25262 (python).
Out of Memory: Killed process 26005 (postmaster).
Out of Memory: Killed process 25263 (python).
Out of Memory: Killed process 25264 (python).
Out of Memory: Killed process 25265 (python).
Out of Memory: Killed process 26003 (postmaster).
Out of Memory: Killed process 25266 (python).
Out of Memory: Killed process 26022 (postmaster).
Out of Memory: Killed process 25267 (python).
Out of Memory: Killed process 25268 (python).
printk: 28 messages suppressed.
So PHP5 is stable and PHP4 is out of portage. Just a note to people with sites hosted off of ufies.org that I'm planning to upgrade to PHP 5.1.4 in the next while (say, by the end of the week). Please make sure your sites work with this version of PHP5 (as I understand it's pretty much compatible). Thanks.
Update Gregg of 21st Century Digital Boy points to the list of php5 incompatibilities. Thanks dude.
Just an FYI that I did some looking through accounts on the server that hadn't been logged into for more than a couple of years, or that combined with only having spam in their mailboxes. The users have been disabled, if any of these are in error let me know and I'll fix them up. This probably won't affect anyone, but I figured I'd throw it out anyway.
There are still some things broken, mail (pop and imap) don't seem to be working, so I'll get those going ASAP. Also jabber will be down for a bit, but that's less of an urgent thing IMHO.
Updates soon....
Update #1 Hey, I like fixes that fast! Looke like the mail issue was just a process that wasn't being restarted when I thought I was restarting it.... Based on the logfiles though, you all already knew that :)
It also means that if you have super valuable data you should know that I will be taking every precaution to ensure that everything remains safe. The old server will be either online or un-formatted for at least a few days or week to make sure that all data is over properly.
The important directories (/home, mail spool, databases, etc) will be mirrored when the system is disconnected from the net to ensure that no data is lost.
The new server is what I consider to be hardware porn, and Fred and the people over at TDIClub.com are to thank for this.
- dual hyperthreaded 3.4Ghz cpus
- 6G ram
- 5x72Gb SCSI drives in a RAID5
- super special managment thingy to allow me to un-fubar things that would make the system unbootable
So again, 6 days from now the box will go down, then come back up faster and better.
Any questions or concerns feel free to email me.
Given some of the issues we've had over the last few years, you'll see why 2 years of uptime on a linux box is sometimes a big deal, especially when using "commodity" hardware.
No fear, as some may know already, this box is in for a major refit, with a swanky new 2U dual proc, hyperthreaded, 6G beast waiting to be configured and swapped in place here.
So do we have any UFies who want to set something like this up?
Update - All went well, things all seem to be working, let me know at arcterex awt ufies dawt org if anything is busted. Also, please make sure your php and perl apps/cgis don't blindly execute code, k? :)
Many thanks.
alan@master:~$ uptime 10:15:53 up 366 days, 20:31, 68 users, load average: 0.84, 0.95, 0.93 alan@master:~$
In a fit of madness yesterday I posted a local mirrored copy of the new Episode 3 trailer, and somehow it got around the net as a mirror in the feeding frenzy. You can see the results from the BigBrother image. So sometime this morning Fred noticed that the bandwidth was up at something like 20mb/s when he moved the file away. Of course, we did basically the same amount of bandwidth we do in a month in one day, and while the way the hosting works we're not on a bandwidth-based pay system (ie: NN Gig per month) I'm not sure how the hosting company will react.
So now is as good a time as ever to once again bring out the tin cup and point to the "Help UFies via paypal link to the right. If you've got a bit of extra to spare it would certainly make my life easier and the keeping of this server up and going a bit less of a stress point in my life. Please and thankyou and I really appreciate it.
Next time I'll just keep it to myself!
*makes note to set up bandwidth limiting in apache*
Update: Well, didn't break anything, but in theory might have sped the IMAP up a tiny bit, though I can't tell myself. I tried the tip of removing USERID as according to this and this page, but if anyone knows how to speed up a postfix+uw-imap I'd appreciate it. I was going to move to courier-imap, but moving a system with lots of users and already set up heirarchy of MBox folders to a mailder isn't all that good IMHO :) What'd be nice is a imap server that can deal with both mbox and maildir folders dynamically depending on what the user has set up, so my users can convert over slowly if they want.
Help or suggestions appreciated.
The ads have been put (for now) on a few select pages that seem to have brought in through google for one reason or another (ie: the 'how to hack hotmail passwords' page (which isn't)) and the Iambe comments page. I may put them in the main site at some later time however. We'll see how it goes.
If you really hate them, you're welcome to donate and / or bitch and whine at me.
The second part will help those to access mail via imaps and pop3s... the right cert is now loaded by the system so that it's no longer localhost, but the proper domain name.
A VERY SPECIAL REQUEST
A very dear UFie I know (who will have to remain anonymous because he doesn?t know I am doing this) informed me earlier than he is shy a CD burner because his conked out on him in a cruel twist of fate. Normally when this happens we would just head on out to yon nearest webstore or brickstore and pick up a new one (funds providing) ? however our friend is unable to do so because of not only limited income (disability income) but because of a disability (mobility).
If you happen to have an extra CD burner (say you just recently upgraded to a DVD burner) that is hanging around and wouldn?t mind shipping it to a US address on a random act of kindness, please get in touch with me at iambe at userfriendly d-o*t org(an) and will see if we can?t match your spare hardware with this kind soul.
Thank you,
Iambe
I don't know why this rash of hardware problems has cropped up, but I'm going to guess it's that the box is overheating a bit, or the RAM isn't that good, or the poor little thing is just taking an enormous beating. At this point I don't care because between 3:30 and 6:00 tonight the box will go down, have it's innards put in the new (actually old) system and have that fresh, new and lovely clean box come up.
I'll put a page on my home website of http://arcterex.net/blog with a list of what is working and what isn't, and encourage all to post there (or here if apache is up) to let me know what is working and what isn't. Please expect a couple of days of confusion before things are worked out, but by tonight mail and web at minimum should be up and going for the most part.
So backup, cross your fingers and send good vibes!
Update - I managed to get mail going again by removing the postfix-tls package so if you can't do secure SMTP that's why. Sorry, but it'll be fixed tonight.
The upgrade did not happen because of the hardware problems we found (detailed here
The box is slightly slower and has slightly less drive space (any users logging in will note that /home is pretty full right now :) so if you have large backups or files in your home directory that you can afford to lose I'd appreciate if everyone can do a bit of cleaning.
Much thanks, stay tuned, and appreciate the patience.
-- alan
Anyway, things are back up and going, no harm, no foul. Not sure about when the actual scheduled downtime is going to happen though. Stay tuned.
Some changes will be happening though, a second SCSI drive will go in to act as a mirror, and a firewire harddrive will be set up for use of offsite backups.
Oh, and to everyone who said they didn't mind and they were looking at re-doing their websites anyway, sorry about this :)
Yay!
It looks like some websites content and images will be available through a google search via the "cache" keyword. IE: go to google and type in "cache:<site>" where <site> is the site you are trying to recover. If you're lucky you'll be able to get data and images from a relatively recent snapshot of your site. Hitting the wayback machine might contain bits and pieces as well. I'm just hoping all my users had backups for my own sanity.
Again, I'm really sorry about this folks and I'll be working tonight and tomorrow to try to get the system at least up with a /home on a different drive so people can log in, and hopefully the drive will be ok, but if not... :(
Sorry guys.
I'm trying to recreate home directories for those who had them before, email me if you can't get in, don't get a /home/<yourusername> or whatever. Assuming that the drive is pooched of course.
Thanks for everyone's patience and understanding. Read on for an explanation of what actually happened.
Update - webmail is busted, I'll deal with that in the morning. Fixed.
It was the start of a sunday that I was hoping to take easy and relax... ah, the best laid plans of mice and men. Fred threw me an email to point me to a RAID failure message. We had had an element in one of the arrays fail not that long ago, but the spare kicked in and all was well, and I was turning a blind eye until I had the mental strength to deal with it. Anyway, the co-lo had a scheduled powerdown from the hydro company for last night. Turns out the power company was early though. While the RAID5 was reconstructing the array, it must have hit a bad block or something and kicked out the element. So now we were down to 2/3 :(
I moved the spare partition in from the first array to put in as the 3rd and it seemed to go in ok, but as it was reconstructing something kicked it out and now we had 1/3 going, which as anyone who knows about RAID5 knows, can't really happen. Things were still running, and "ls" was still showing files, but there were I/O errors when I tried to copy files around (in a desparate attempt to save data in case something happened). So I resigned myself to the fact that I was going to have to go in and fix things from the console. An hour and a half later I was there, armed with a new drive.
It's getting late and it's been a long day so I'm going to keep it short, hit my blog sometime tomorrow for an even longer and more boring version of this.
Anyway, I figured out which drive was causing the errors and swapped it out. When it came back up all nicely recostructed and what not I was quite disheartened ("bugger" I think was one of the terms I used) to see that I couldn't mount /var anymore, as it was not recognized as an ext3 filesystem. So I re-created it, copied the files I have backed up from /var every night (mail spool, databases, dpkg lib, etc) and brought the system hobbling back up to life. I managed to recreate the needed files from a friend of mine who also has a debian unstable system (thanks wim!) and have been running various "apt-get install --reinstall <package>" commands since. Everything appears to be running again, and other than losing all the logs and mail/database stuff from 3am till 10am when the system went down ("luckily" the mailserver was down most of the day as well, so mail was sent over the the secondary server and saved, and there was only a 7 hour window of nuked mail).
Anyway, thats about it. Like I said, let me know if anyone sees anything that is broken, and thanks to Fred for getting me the spare drive and the guys at data-fortress for putting up with me there and for keeping me company (though they had to be there anyway dealing with the power issues).
Hopefully things will be back to normal tomorrow night.
Update: Based on this post and this bug I'm confident enough to take back what I said. Ignore me. Rewind. Undo, whatever :) Thanks to Christophe for pointing this out to me.
65.110.12.162
The above will be the new IP for UFies.org, so if you have a nameserver or a registrair that needs this information there it is. I've reset all the DNS settings to have a timeout of 10 minutes, so changes should hopefully happen relatively quickly once things are in motion.
Again, there will be a day or two of turmoil, but hopefully things will settle down soon.
Some here it is again:
65.110.12.162 ufies.org
Fred's servers will be at:
65.110.12.165 tdiclub.com
65.110.12.166 forums.tdiclub.com
65.110.12.167 pics.tdiclub.com
These changes will take effect on Monday, Sept 29th.
So I'm wondering if anyone has any recommendations for places who do co-lo hosting in the vancouver area. I'm not looking for a whole lot, basically
- Vancouver area
- Access to get in for reboots or hardware changes (after hours access a bonus)
- Competant support people or admins
- Not getting completely screwed over for the price or for extra traffic (sadly I don't have my bandwidth stats here right now, so I don't know what I'm looking for per month yet)
Hopefully I can find something good though, and quickly. Thanks for any advice or pointers anyone can give me. Thanks for hanging in there with me.
Everything is fine now. It's all ok. We're all fine here.
On the upside, I finally got a chance to put the new kernel options to access the the extra gig of ram I have had sitting unused. Fred will be happy about that :)
Please bear with and sorry for the inconvenience.
Update Ufies will be shut down at 10am sharp and I'm told downtime should be only for a couple of hours max.
"Believe nothing, no matter where you read it or who has said it, not even if I have said it, unless it agrees with your own reason and your owncommon sense."
Buddha
It works for me(tm), so it should work for you to. Please email me any issues, or a fix for IMP if you know what is going on. Squirrelmail is also available at http://mail.ufies.org for your convenience :)
As always huge kudos to Fred and Ken for their help.
We now return you to your regularly scheduled random links and funny stuff.
On Monday, Jan 20th UFies.org will go down from around 10am to sometime slightly later than that. I'm hoping it'll only take an hour or so. Anyway, we'll be putting the nice fast motherboard/cpu(s) back, along with a more powerful power supply.
If all goes well, there should be no more problems, ever :) However, if things crash again I will go back in Tuesday (the 21st) and put the old (current) motherboard back in, and we'll be back up. But that won't happen, knock on wood (*tock*tock*).
Hey all, just wanted to say thanks for the support throughout the last year and for making UFies.org (and it's affiliated sites) such a great success. See you all in 2003!
A suspected faulty drive has been removed and the nice dual motherboard have been replaced with a slightly slower PIII-450 that was found. Assuming we're still up tomorrow (fingers crossed) that'll mean that the load problems have been fixed (they were causing crashing).
My huge thanks to Fred for troubleshooting and the spare hardware, and Tom (at the co-lo) for putting up with me keeping him there long after he should have been gone home and saving my bacon in a couple of cases (mkraid --dangerous-no-resync is your friend!).
Mail will be back up in a few minutes, as soon as I restore the mail spool from backup... some data (logs mostly it seems) was lost, so please email me if you find anything busted.
UPDATE: OMG! It's still up! Whohoooo!
(Oh, and the server is back up and no worse for wear (yay RAID) than a few bounced emails... will post the grisley details later)
And on another happy note, you can now just go to http://mail.ufies.org and you will be sent to the right place :)
Update: *sigh* looks like squirrelmail is b0rked as well. I'll get it all going one of these days. Lets try another one!
In Flanders Fields
In Flanders fields the poppies blow
Between the crosses, row on row
That mark our place; and in the sky
The larks, still bravely singing, fly
Scarce heard amid the guns below.
We are the Dead. Short days ago
We lived, felt dawn, saw sunset glow,
Loved and were loved, and now we lie
In Flanders fields.
Take up our quarrel with the foe:
To you from failing hands we throw
The torch; be yours to hold it high.
If ye break faith with us who die
We shall not sleep, though poppies grow
In Flanders fields.
Anyway, I'm working on these, cursing all the way, and hopefully it'll all be fixed up RSN.
Things that are busted:
-
php not loading(fixed by commenting out the imap module) - php imap auth for webmail (known bug 163003... hope it's fixed soon!)
-
perl 5.8 probs(at least I think so, the one error turned out to be code error, not a perl 5.8 related error) -
MT db(database upgrade problem, but the solution posted here seems to work. Just back up your db directory, then go in and type db3_upgrade *.db *.idx, and all will be well!) Mucho kudos to foz (and the guy he got it from)) for pointing this out to me! -
imap non-ssl/tls authentication(fixed, though I don't like this... I may/should just force everyone to use SSL/TLS)
Update: Looks like all went well, we're back up with MT 2.51, a database backend, and nicely trackback enabled! Wh00p!
- I will not turn on the television all day, unless it is to watch something specific and amusing.
- I will refuse to participate in watching the endless looping of the horrible images that have been gracing our screens for the past days. I will never forget what I saw one year ago, or what I felt, and I don't need CNN, ABC, NBC, and the lot of media gluts to remind me of it.
- I will love my fellow man, regardless.
- I will not hate.
- I will say a silent prayer to all those lost, all those left behind, and all those affected in one way or another.
- I will love.
- I will respect sites that are not running banner ads today, such as OSDN. It's not much, but it's something.
- I will not be political.
- I will run in the morning.
- I will never forget.
- I will never be the same.
My buddy LIMOS threw me a pretty funky Mozilla wallpaper. I think it's been making the rounds, but this is the first I've seen of it. Tis good, and a nice high res as well.
So how is everyone going to be celebrating?
(Pssst... the root CA for ufies.org (for OE users) can be found here)
Oh, another mail related note. I have a system called pop-before-smtp going now. The way it works is as follows. The system will watch for people authenticating themselves with pop or imap (or pops/imaps) and when it sees that, it will open up their IP as a valid relay for SMTP for a limited amount of time. End result? As long as you get your mail first via pop or imap, you can use ufies.org as an outoing smtp server. This is new, and will be removed if there's any abuse (which I don't expect there will be), and feedback is appreciated :)
Short story is that the drives and some other hardware was replaced, but with some data loss and other messed-up-idness.
Read on for more... First of all a huge thanks to a few people. Yohimbe (userfriendly.org sysadmin extraordinary) for mucking with hardware in his living room till 1am and Tom for coming in on a day off, and staying late another day to let me in to get the box out. Oh, and I won't forget Fred (the TDI guy) who donated some of his own time and $ getting the box up and going with it's new hardware.
The box is now back up and running with it's busted 30G drive replaced with 3 40G Maxtor drives using RAID 5 for speed and redundancy, and a nice Antec AX-1240 tower case (reviews) with a 400w power supply (one theory for the blowout was bad power, as the box was running on a 250w power supply.
The bad part about this crash was twofold. First of all, the backup script was backing up /var/spool/mail, a symlink to /var/mail. The script did exactly what it was supposed to do, and every night backed up a symlinks :( With the last hard drive problems I had mail on a separate partition, so it was all ok, but not this time. So anyone who left any mail in their main INBOX spool, sorry, but it's gone now.
The other thing was the databases, which relates to why you're reading this on a nearly default Movable Type page :) See, phpnuke was driven from a database. Every night the database backed up by script. Well, due to some changes in the way that somethingorother changed, the programs to do the database dump every night were failing. I noticed this, but never got a chance to go and fix it because "hey, it's a new hard drive, what could possibly happen?"
Well, guess I found out huH? Anyway, the mysql database's last good backup was around February somethingorother, and postgres's was around March 7. So a lot of stuff from ufies got lost, as well as other things (peer 2 peer being the main one. That one wasn't all that bad as it was postgres, but still...
Anyway, so that's the story. If something is messed up, or you don't have access to something you used to (shell users) like a database, or whatever, email me at arcterex@ufies.org. If you lost data (by the way, I hope I mentioned that the /home partition is just fine, so any data in your $HOME is exactly as you left it), or mail, I'm sorry.
As you've probably noticed, this is a pretty default thing, so bear with me as I re-do some things, and learn the ins and outs of MT :) Now, back to getting it all back up and running!