Writing

Feed Software, technology, sysadmin war stories, and more.

Sunday, March 18, 2012

Snarkiness as a measure of badness in config files

Comments bring life to what might otherwise be a lifeless environment. They advise on the "why" or give a warning to others who might follow you. The hope is that you might save some time down the road for whoever might encounter that source code or config file.

A couple of years ago, I encountered something really irritating. There was a LAMP stack which had a bunch of stuff controlled from a single ini-style config file with [sections] and foo = bar type stuff. I ran face-first into an annoying problem one day when I made a change and it wasn't visible consistently.

It turned out that some of the config details were kept inside the mod_p*whatever* in each Apache child, and were never refreshed from the config file. This meant if you made a change, the existing children would have no clue about it. Worse, if new children were spawned to handle added load, or just replaced old/dead children, they would have the new config and the others would not.

That would give you a wonderful nondeterminism when trying to visit the site. Depending on which one woke up to service your request, you might get completely different behavior from the site.

Once I found this out, I left a huge snarky comment up in the top of that file advising anyone who made a change to make sure they did a full down-and-up of the entire Apache server. Never mind the fact that we were surrounded by systems which had intelligent methods for managing configuration, including picking up changes automatically and having source-controlled files. We had to run this broken thing from the stone age.

My comment was something along the lines of this:

If you make any changes to this file, you *have to* restart the entire Apache server, including the parent process. Otherwise, you will have crazy nondeterministic behavior depending on which child handles a request.

Welcome to 2009. -- rachel

That last bit is my way of saying "this is still a problem? WHAT year is this?"... I say that a lot.

It's three years later, and I'm sure that's still standard operating procedure in more places than they would like to admit. I bet people are also being bitten by the "depends on which child you hit" thing and more than a few only know that "restarting fixes it", without knowing why.

We are still very much in the dark ages with some of this stuff.