Unreasonable customer demands and the spammer special
Near the end of my days working tech support, I came up with a hypothesis about some of the more demanding customers with unreasonable requests. Now, to be clear, I'm not talking about the people who were type-A, demanding, and had reasonable demands. They are a different story, and I've talked about them before.
No, this story is about the people who are type-A, demanding, and have unreasonable demands. These cases never ended well.
So, what is an unreasonable demand when it comes to web hosting? Usually they are a series of bad decisions, and typically it involves Sales bending over backwards to make a sale. After all, they aren't the ones left holding the bag when it comes time to do all of the custom craziness. That's a job for Support.
The worst sort of custom brew was from the obvious "gearhead" customer. This was an individual who absolutely needed to have the latest version of everything. They'd ask for the newest install of Red Hat just because it had a bigger number. Then they'd demand a version of MySQL beyond the one provided by Red Hat. They'd repeat this for PHP, and all of its other little associated fiddly libraries. Sometimes they'd get really ornery and demand Tomcat installs, which we never supported in the first place. Oh, and they'd usually demand ReiserFS -- because "it's faster". Gee, thanks.
The thing is, individually, we would upgrade people to custom versions of MySQL or PHP if they needed it. This usually happened as a response to an actual problem they were having which could not be solved any other way. That's different. The people I'm talking about here today are the ones who wanted to turn all of the knobs up to 11 just because they could.
When these systems would go online, these "provisioning alerts" would appear as support tickets chock full of custom garbage. Some unlucky tech had to grab these tickets and proceed to blow hours on finding, building, packaging, and installing all of the magic versions which had been conveyed to the sales monkey.
Then, more often than not, they'd come back and say something was wrong with it. One guy actually started complaining, saying that some benchmarking tool we had never heard of was reporting that Tomcat was 8 ms faster than Apache to do something or other. Yes, that's right, 8 milliseconds.
Now, for me, 8 milliseconds is not a meaningless number. I actually have a real-world handle on what that means, and it's not a slice of wire from Grace Hopper. No, for me, 8 ms is the time it took to traverse our T1 line to the local university in town which was giving us connectivity back in the '90s. I remembered the days of ~200 ms latencies across a dialup link, so having one down in the single digits was pretty amazing at the time.
As a result, when someone reported 8 ms of fluctuation from two completely different servers on an already heavily-customized configuration when we didn't even support tuning in the first place, you can imagine how much my eyes were rolling. The whole thing was a farce.
All of this inspired a hypothesis. What if you worked for another hosting company, and wanted to seriously screw up the competition? All you'd have to do is spend a few hundred bucks to open an account, and then start making completely unreasonable demands which were calculated to suck up tech time. You could tie up people for hours with just a few taps at your keyboard, and it wouldn't cost you much.
Every time you did this, you'd lengthen the response times for other customers, and would start dragging down the overall experience. What's more, for the demands you made, you could take the results of those and start writing about them in the nastiest ways possible. You could say how they were incompetent and couldn't do this and couldn't do that. It wouldn't matter that such requests were explicitly unsupported and they were going out on a limb to try to make you happy. You could exploit it for your own purposes.
This was just a hypothesis, and I never was able to verify it or disprove it, since I moved on to other things not too long afterward.
Incidentally, ReiserFS in those days was an interesting signal. It seemed like the only people who really asked for and then used that filesystem were setting up "spammer specials". These people would stand up a config with a machine or two and a handful of additional IP addresses, load their custom software onto it, and then proceed to hose down the Internet with their evil spam.
A few months later, the AUP team would kick them offline, but they had accomplished their evil deeds by then. I wonder how many of those accounts were linked to credit cards which were stolen or otherwise weren't able to pay for the actual service.
Other times, you'd get on a brand new box to work a ticket and notice the customer had already been on long enough to install an IRC connection bouncer. It raised a good question: you spent how much money for this config, and the first thing you do is jump on IRC from it, and all of your files look like they came from a rootkit? I'd flag those accounts as possible fraud, and more often than not, they'd be booted off as well.
With any luck, we'd catch them inside the 30 day window so that Sales had to eat the churn and not Support. Those few times when we could actually smack them back for sending us stupid things were worth the trouble.