Software, technology, sysadmin war stories, and more. Feed
Saturday, April 14, 2018

Leaving "gifts" behind on dedicated server hosts

I used to work in a place that had a bunch of machines that other people would pay to use. You might call it "dedicated" or "managed" or even "enterprise" hosting. A lot of it was just "whitebox" machines sitting on bread racks, but some of it was actually in proper racks (or "cabinets") with machines which were actually built for it.

Regardless of the hardware used, the model was about the same: you'd pay the company some amount of money for the right to do whatever you wanted with the machine, more or less. Obviously, if you screwed it up too badly, or installed some unsupported OS, you might not get much support beyond hardware replacement or reimaging it with a supported system, but it worked for most people.

I also became a customer of this world way back then, and have been ever since. This page is coming to you from the latest incarnation of this kind of server, for instance: just a box hanging in a rack somewhere in Dallas.

Recently, my mind turned to nefarious ideas. It was probably after writing that post about bad hardware getting into loops a couple of weeks back. That post was about hard drives being used by multiple distinct customers, but what about the rest of the machine? Are there parts that don't get "wiped"? Could you leave behind a little "gift" for the next customer?

This is where I put the "evil hat" on. So, I have root on the box, right? I also have out-of-band access to it so I can mess around with the console and/or "BIOS" settings. Basically, I can do all kinds of stuff on here. Reflashing it probably wouldn't be terribly difficult if I really wanted to.

That brings up a whole new bag of hurt: what are the odds that the hardware is going to enforce some kind of signed integrity check on that firmware? Maybe the main board itself will, but what about things like the NIC? Is it all that paranoid? What if it isn't? Could I put my own code on there? How about the BMC? There's a lot of goofiness out there, so I bet there's at least one hole on a lot of popular hardware.

Let's say I do that. Then I "return" the machine. It goes back into the pool, and then some other customer gets a hold of it. Meanwhile, my evil firmware is still on the box, right? Who actually checks that kind of stuff before they reuse hardware for another customer? I'm guessing the number is effectively zero.

Once the machine goes back online with a new customer, then I exploit the hole I left behind, and boom, it's mine again. Maybe I get lucky and find something particularly tasty and interesting. Maybe not.

I don't have an easy solution for this one. Building your own box and doing the co-lo thing is just far too annoying for a lot of people.