Writing

Software, technology, sysadmin war stories, and more. Feed
Friday, May 11, 2012

Customers who root their own managed devices

In the web hosting biz, there are people who need a lot of hand-holding and then there are those who don't. This is a story about the latter.

We had this customer who had their machines behind this derelict old thing called a Webmux. This was quite a few years ago, but even then we considered them obsolete. Most of our load-balanced customers were behind Cisco devices, but not this one.

They opened a ticket that asked about security patches to their net device. We treated all such hardware as "managed" and didn't give them logins as a result. They wound up showing us a thing or two about running these boxes.

You see, these Webmuxes were just glorified Red Hat boxes, running some version which was ancient for its time. They also had daemons running which were similarly out of date. Our customer had scanned it and discovered it had a remotely exploitable ssh hole.

Trouble is, by the time someone got around to answering their ticket, they had gone ahead and used the hole to log in and snoop around. Yep, the customer cracked root on their own device.

Some of us had a good laugh. Some networking types enjoyed it as well, but some account manager type wasn't having it. They needed to be moved to a (Cisco) CSS as soon as possible.

Of course, to do that, they'd need to be moved behind a firewall, since that's how we sold load balancers at that point in time. The days of running exposed load balancers were over.

I think this customer also had to ditch their white box servers for 2U servers so they could move into a cabinet. This is because we didn't do net devices for "bread rack" customers and white boxes.

Obviously, it was a mess. Somehow, I dodged the bullets of having to clean up the mess and then help them migrate. They were a relatively high profile customer, so telling them to "quit hacking your Redhill" wasn't going to fly.

They may have even gotten that new config for free. It's amazing just how far an exploit on an unsupported device can get you when you're an important marketing partner!