Software, technology, sysadmin war stories, and more. Feed
Wednesday, April 11, 2012

Lax practices in the pasture

Imagine a company which is responsible for a bunch of cows. They have thousands and thousands of cows. They aren't a dairy farm or anything, but they use the cows to support other things. They stack stuff up on top of their thousands of cows, and the stuff which runs around on top of the cows is their actual product. You know, cowbirds, and stuff like that. Whatever you put on top of a cow.

Sometimes cows get sick, so they have automated systems which will take the stuff off one cow and move it to another. Then, the sick cow can be taken back to the barn where it will be cleaned up. With that done, the cow will be sent back out to the pasture where it can accept some of the load from its "co-cows".

Also, there's this whole robotic cow management system. People could never keep up with the load or the complexity of handling the cows directly, but they can write a system to drive those robots. A bunch of robot programmers sit down and do exactly that, and the robots handle most of the work.

There are rules about these programs for the robots. You have to write your code in a certain style, and you have to get that code reviewed. You also have to make sure it gets tested. These tests run in an automated fashion so you have a chance to find out when something changes and your code/test fails.

Well, times change, and the company stops caring about quality nearly as much as it used to. It does this by chasing out the really good robot wranglers and also by getting people in the higher ranks who have no taste and thus no skill in determining what is good and what isn't. The old robot-handling code is still there, and it still basically works, but it's showing signs of age.

The company's new corporate overlords have decreed packing far more cows into each pasture than ever before because pastures are expensive. They also want to stack even more things on top of each of the cows, because the cows themselves are expensive. If there's a sliver of space on top of a cow, instead of leaving that open, they want the robot handlers to fill it somehow.

Trouble is, these aren't the same people who built the system. Those people have long since set sail for other places. The institutional knowledge is gone, and the replacement people aren't up to the task anyway. They still try, and they get some partial solution in place, and it helps, but it still introduces more problems. The system is rotting.

Finally, someone declares that Fred Brooks is a crackpot and decides to rebuild the whole robot scheme from scratch. They start chopping up the problem space into divisions, and start hiring people fresh off the street to fill them. There's zero institutional memory on these new teams now. They have no idea that a robot has to warm up its "hands" before touching a cow, or the cow will kick the robot and destroy it. (Clearly, they didn't play Return to Zork.)

Soon, all of the best practices go out the window. You have people who start writing code in this greenfield environment. They haven't been tested on their adherence to company coding policies, so they don't have the magic bit set which says "I know this language and will not cause a disaster by writing something horrible". Since they don't have that bit set, they can't add code to the normal code repositories.

That's okay. They use their own little experimental lab-type directories to store things. Those don't need any bits set, and they also don't need code reviews! That's right, they are totally off in their own little world, doing their own thing with no oversight and no guidance. They're attempting to boil the ocean and rewrite a massive cow-handling project, and they have no idea what they're doing.

It gets better. They have rationalizations for all of this. It's not happening in a vacuum. People notice and call them on it. They have ready answers for all of it, including using those lab/experimental areas and so on. It's just amazing.

I'm sure they will eventually squeeze out a release or two, and it will be just as smelly as some of the stuff you find in those pastures. Personally, I wouldn't touch it with a ten foot cattle prod. To me, it's a No-Go situation.