One of Those “Ksplice Saved Our Asses” Moments

Posted by Nessa | Posted in uncategorized | Posted on 15-01-2011

4

There comes a time in every sysadmin’s life when you realize that as long as you have your job, you’re never going to

  1. have a life
  2. have friends
  3. sleep
  4. have a break from all the madness

When you fix server A, server B breaks. When you patch exploit 1, exploit 2 strikes threefold.  And to top it off, our customers are still using passwords like 123456.  Luckily, there’s someone out there that appears to be seeing things at my level.

We started using Ksplice back in September 2010 when one infamous kernel exploit started it all. We used to only have to do kernel updates a few times a year.  Now the RHEL guys decided to start compiling their kernels from sad pandas and french fries, backporting and reintroducing a string of security holes that hackers have been sitting until the opportune moment. When you’re working for one of the best hosts in the industry and basically guaranteeing 99.9% uptime, you try rebooting a dump truck full of servers five or six times a month just to avoid some asshat defacing all the index pages on your server with pictures of babies and swastikas.

I mean, unless you’re into that kind of thing.

During these unsexy times there’s one thing I’ve been continually grateful for – and that’s the fact that the developers of Ksplice apparently have no lives, girlfriends, or any other dreams in life other than to make all our jobs easier.  Here at IMH, we were all slightly depressed when we found out that we couldn’t have reboot parties anymore.  The Ksplice guys usually have new vendor-released kernel patches available within a day  and we can easily and instantly run kernel updates on our entire fleet of servers without having to reboot a single one.  With that, we don’t have to wait until the middle of the night over the span of several days to fix critical security problems, and customers don’t have to complain about downtime.  This lets us focus on what’s important: drinking, karaoke, and nude ice fishing keeping people happy.  If that’s not worth the $2.95 a month, I don’t know what is.