Just one of those days
We needed to reboot serveral of our web servers today for a kernel upgrade and it should have gone smoothly. Instead, it seems like everything was going wrong. The first box didn’t come back because apparently the kernel decided to switch the ethernet interfaces again (even though I had set something to prevent that!). After that, it looked like we were having problems connecting to the database server. It appears the config file for networking was incorrect and a route had been entered incorrectly. The fun didn’t stop there … we then noticed that some of the ifconfig settings weren’t exactly correct, so we had to work around that. To make things more fun, there’s apparently a bug with stunnel to where it won’t restart correctly. I’ll have to find the time and make a bug and possibly troubleshoot the problem myself. Kudos to Corey Shields for making several trips to the machine room to make things work.
I apologize for the extended downtimes you may have encountered for bugs and forums. These things just happen sometimes! 
Anyways, I look forward to the weekend coming up and possibly nailing down some more infra bugs and issues.