Writing

Feed Software, technology, sysadmin war stories, and more.

Friday, June 22, 2012

Quick tales from the trenches of tech support

I have a bunch of anecdotes and other tales from working tech support and various parts of my career. While many of them are full stories and can turn into a regular post, there are some which are just too short. Still, the punchlines can be amusing. Here are a few.

There was a "senior tech" who was working on a brand new customer. They had ordered a Dell PowerEdge 2850 server with FreeBSD for some reason. It wasn't a supported combination at that place, but someone in sales must have pushed it through anyway.

Anyway, the customer wanted more RAM. The senior tech advised they could fix that with "modprobe".

Yes, that's right, the "senior" tech said they could get more physical hardware in their system by running a command to insert a kernel module. On top of that, modprobe is a Linux command! This was a FreeBSD box!

Some "senior" tech.

...

Another time, a new customer wanted to have their new config installed in a specific data center. This data center was old and had been closed to all but growth of existing accounts for quite some time. Somehow, sales managed to drive it through. I was appalled when I found out why this had happened.

That old data center apparently had the most "diversity" in terms of IP blocks. That way, they could have a whole bunch of addresses all over the place for some reason. When my friends saw it, we all figured "spammer special" or possibly "slumlord". That was a term we made up for those idiotic sites which had zero actual content and just spewed out a bunch of ridiculous terms to serve ads. They thought spreading across IP space would somehow "cloak" them from the anti-spam divisions of certain search engines.

...

Once, a customer shipped in a drive which had been ReiserFS formatted and wanted us to migrate data from it into their machine. Whoever got a hold of it wound up having to tear up that machine to custom-rig a kernel which would support it. Then they had the data center pull it down and stick in the disk. Later, they had to undo all of it.

Apparently it never occurred to them to just have the DC build a temporary whitebox, perhaps by grabbing a recently-churned machine which was waiting to be recycled. Then they could have beaten on it for however long and copied the data over the network. What a concept! The customer's actual server would have remained up the whole time.

...

Then there was the time a customer wanted to compile something for his web server. The ramrod tech who got a hold of the ticket decided this meant removing the stock build of Apache 2.0 which was maintained by Red Hat and got automatic updates. His solution was to compile Apache 1.3 from source on that machine.

Fortunately, someone who could figure out what was actually going on looked at the customer's situation and determined that all that was needed was a quick "up2date -i httpd-devel". That installs the header files and other auxiliary stuff needed to link against Apache (like apxs), and the customer was happy.

Apparently that first tech wanted to show off. That's always what he did. Everything would be rebuilt from source. It might mean rebuilding Apache 1.3 from source (and thus taking them off the automatic maintenance from Red Hat). Then he might install X and Evolution, also from source, just so the customer could "log in as root and read mail directly on the server". No, I'm not making this up.

Obviously "just because you can doesn't mean you should" had never occurred to this guy. He left giant steaming turds behind for anyone else who ever had to work on a machine after he had been there.

Those are just some of the many quick tales from the trenches.