Supporting "vhosters" and their config file foibles
One of the products I used to have to support in the web hosting business was called Plesk. It was something which basically latched onto an ordinary Linux box with many tentacles and turned it into a "vhosting solution". In other words, it could be used to host a whole bunch of web sites from a single box. Most of the Plesk installs I saw were used in a specific way: someone would put up a machine with us, then they'd sell client access to their customers, and those customers would host their domains that way.
Five dollars per domain per month might not seem like much, but when you can get 1000 or more domains on a single server and you have a dozen servers, suddenly you're talking about some serious coin. I assume those customers weren't paying us $60K/month for those 12 machines, and so were raking in quite a nice profit for a setup that basically ran itself.
All of this automation came with some costs. You had to give up the notion of being able to just drop in and modify Apache config files to your heart's content. The whole thing lived in a database, and making a change in the control panel really just changed the database. There were then tools which would read the database and would write out Apache configs, qmail configs, and anything else which might be needed underneath.
They had this fairly deep scheme of nested includes to make the web server configs work. First, you had your normal /etc/httpd/conf/httpd.conf, but at the bottom, it would Include /some/path/to/httpd.include. That file was the "top level" Plesk include file, and it was dynamically generated. It would be rewritten any time anything changed with any of the web sites, or basically any time it felt like it.
That top level file would then include individual files for each virtual host, also automatically-generated. Those were also called httpd.include but they lived in the individual home directories which were associated with a given client and domain. The same basic thing applied here: they could and would be updated at any time with no warning.
So, when someone came along and tried to do something weird with their Apache config, life got interesting. A frequent request was when someone wanted to twiddle some aspect of their PHP configuration across their entire web site. Normally you'd just stick some directive in the VirtualHost container in httpd.conf, right? Well, that's no good here. Some people would just edit httpd.include, and it would work for a little while, but then it would eventually be overwritten and would die again.
Sometimes people really bashed their heads against this wall for a while trying to figure it out. I even saw things where someone had used "chattr +i" to make the httpd.include file immutable just so their evil hacks would stay around. Plesk's file creator was able to deal with the inability to write that one file without breaking the whole thing, I guess.
At some point, they introduced this feature where you could create your own little "vhost.conf" and that would then get Included for you. This finally gave people a way to do those magic settings without having to make system-level changes which affected everything in parallel.
I should mention that a lot of these PHP-related config requests were bad news. People would ask for things like "allow_url_fopen", claiming they had done their homework and wouldn't be compromised. Of course, a few days later, their site had been automatically scanned and defaced by yet another worm which then turned around and started launching attacks from there.
I mean, when you can effectively make a script run any command you want on the web server by pointing it at your evil URL, you should expect bad things to happen. You're practically giving them the keys to the kingdom right there.
The other one we'd get a lot back in those days was "open_basedir". I'm not much of a PHP person (phew!) but it seemed to be something which restricted where file operations could happen. Whatever the default setting happened to be wasn't compatible with some scripts people wanted to run on their $5/mo account, and that lead to a lot of manual config mangling by those of us working as web support monkeys.
The last one I remember seeing a fair amount was register_globals. Yes, this is the thing where you can basically let incoming requests define anything they want in your namespace. Some crazy people would ask for this to be enabled, and we'd try to stop them, but a few would persist, and it would go on the box. More often than not, they'd get owned, too.
I guess this could just be selection bias, since people don't tend to open support tickets for stuff which actually works. On the other hand, I sure saw a whole bunch of machines running malicious code over the years. I don't think most of these users had any idea what they were in for when they enabled some of this stuff, and nothing we said would make any difference. They would not live without that guest book.