Software, technology, sysadmin war stories, and more. Feed
Tuesday, December 6, 2011

The Tribal Song of the Ancestors

Tribes have oral traditions. Technology firms are no different.

I arrived at one such place several years ago and found myself in a land of immense complexity. There were many questions to ask and few answers to be found online, so I started writing it all down. I borrowed a term from a coworker at a prior job and called it "The Tribal Song of the Ancestors" -- that which should have been written down but never was.

It was a pretty good fit for how I roll. I'm not afraid to announce that I have no idea what something means, and then ask for someone to explain it. I'm also not afraid to leave behind a tangible record of this event by documenting things. The things which you can discover by forcing these issues into the light are worth it.

It's an interesting situation since everyone has to ask these questions at some point. There's just this bad habit a lot of people have which is to pretend they somehow were better than that. Guess what, everyone. If you didn't create something yourself, odds are, you had to learn about it the hard way just like everyone else.

Now, if you have a Matrix-style "I know kung-fu" implant, by all means please ignore this story and have a good chuckle at the rest of us.

So what goes into the tribal song? A lot of it is explaining names, naming schemes and error messages. This is the sort of thing I faced not long after starting my job there:

pausebot: Beetle boop! Pausing cell rectangle-canada. (abcde123)

Got that? That was a real mail which would be sent out by an automated process whenever a certain class of event happened. You have to take that apart piece by piece to understand it.


"Beetle boop" is a reference to Love's Tiny Robot by Michael Kupperman. The original author of the program which sends that message was a big fan. Indeed, if you look in the source code, you can find many more references to it.

Ask him if he is not a hobo, then why does he have a bindle?

It's a pause bot program. It's a robot. He's a robot. Get it?

"Pausing cell" would suggest that action or movement involving a group of things has been temporarily interrupted. That is in fact what was going on, but it didn't give you any idea of how (or why).

"rectangle-canada" is my sanitized version for the name of one such cell. The actual name was not much more descriptive. For one thing, "canada" was not sufficient to tell you where to start looking to find that cell. It would be like saying "San Antonio" when what you really need to know is "210" (the area code). Without a list to provide that mapping, you're still lost.

Finally, "abcde123" turned out to be where this program happened to be running. It was just another opaque string of data until you learned how the names were built.

You'd eventually learn all of these things and would be able to parse it just by reading it and without looking at any lists. This wouldn't help a brand new member of the team or someone else outside of the team. Also, if something "tragic" happened, like the entire team being hit by a bus, then all of that info would have needed to be reverse-engineered from records: source code, log files, and old e-mails. What a mess.

In any event, this is approximately how my document came together. Further discoveries would be added as I got deeper into various systems. It wasn't perfect, but it was better than having nothing.

Unfortunately, it didn't last. A year or two later, I found a copy of it which had been unhelpfully updated by some other people. They added a reference to something along with a parenthetical note along the lines of "you should know what this is". In other words, they completely missed the point of my document by not understanding my target audience.

Will documentation solve every problem? No way. It rots. Still, that doesn't excuse you from writing something about what you're doing and putting clear date information on it. At least that gives someone a fighting chance of turning it up with a text search.

Of course, for that to be meaningful, you have to have a reasonable way to actually search your company's internal documents. But search is a solved problem, right?