Avid readers of my blog and fans of my other associated sites may have noticed that my site was down for almost nine hours today. I'm very sorry about that if you rely on any of my server's services. Believe me, I didn't like it any more than you did.

I figured out that I could get newer versions of software by using the apt-get command. I wasn't happy with the version selection inside the confines of my hosting company's distributions, so I changed my /etc/apt/sources.list to include the main Debian package repositories. Over lunch I performed a major upgrade of a bunch of that software.

During a couple of these updates, it asked me if I wanted to overwrite a file or two. I looked at the diffs of the files and it seemed to be harmless enough, and, well, the apts almost always work well and were coming directly from Debian. So, I loaded up the new file and finished upgrading. I figured it would be a good time to restart the machine just to make sure everything updated correctly That's when I realized it didn't go so well.

My server did not come back up. Luckily, my hosting company, 1&1, has this great tool called a Serial Console. Apparently they have all their machines in their large data center wired up to a big KVM switch that doesn't need to have Ethernet connectivity. So, I logged into the central console and could interact with my server as if I was right inside the data center. Geniuses!

I tried, but I just couldn't figure out why the Internet couldn't communicate. At that time I read something in the FAQ saying that under no terms should you mess with the routing or networking settings on your server because it probably wouldn't work anymore. Well, too late for that. I guess I'll have to find something else.

I called up their technical support, and they were as usual unhelpful. The guy said my server was down because I couldn't ping it and almost ignored the fact that I said I could see it running on the serial console just fine. He recommended another option, which is rebooting the machine over the network in a Debian recovery mode. He then said something about mounting a readable file system or some such crap. I knew I wasn't going to do that.

After a bit of thinking, I started doing some queries on Google. I searched a lot about 1and1 nameservers and was finding mostly the wrong stuff. Then I realized that I wanted the 1and1 dhcp information. There I found quickly that the 1and1 root servers need this weird routing configuration in /etc/network/interfaces:

up ip route add 10.255.255.1/32 dev eth0
up ip route add default via 10.255.255.1

Weird. Once I put those in there and refreshed the connection, all was well. Nice.

So I learned a bit, got frustrated a bit, and am still fairly happy with my hosting decision. I mean, when have you ever bought hosting services and ever been totally satisfied? And when do those support people know what they're talking about? As my dad says, "Never is a long time," but I think in this case it fits.