Software, technology, sysadmin war stories, and more. Feed
Tuesday, April 30, 2013

Snippets, change logs, and secret project cabals

When I say "snippets", I'm not usually talking about the little chunks of web pages which matched your search query on a search engine's result pages. No, instead, I'm talking about the short summaries of things which someone did at work in a certain limited span of time, like a week.

Officially, I did them when working for that place in Mountain View between 2006 and 2011. They had given the practice an actual name, and while I doubt they invented it, it was widespread inside that company. The way it worked initially was simple enough. Every Monday morning, you'd get a nag mail from the system. You had until some time that evening to respond. If you replied before the cut-off, it would automatically populate the previous week's entry with the contents of your mail.

I used to keep a flat text file in my home directory and would prepend things as I worked on them. The file grew upwards, in other words, so I didn't have to keep scrolling to find the right place to add something. Then, on Monday morning, I'd encounter the nag mail during my usual mail handling pass, and I'd paste the latest chunk in. It would be happy and that would be that.

What did a typical snippet entry look like? That's easy. It's stuff like this:

- AM oncall
- pushed Foozle package v.123 to nebraska-west
- chased down inter-cluster connectivity bug in IX (b/12345)
- hooked metabot up to production logger

This is just an example, since I don't actually have any of those. They're all locked up in some corporate system which might not even exist any more.

Those four lines documented a lot about what I actually did in that week. I wore a pager and dealt with whatever happened as a result of that. I did a push of some new software, dealt with bug #12345 pertaining to some connectivity issues, and added a feature to an existing program called metabot.

Thinking back, I actually did this before I got to that job. At the dev gig I had before that, someone got wind of what this "magical place in MV" was having its engineers do. They heard about the weekly writeups but missed the part about them being *brief*. As a result, I started writing summaries of what I did, and they were much longer than the above. They were written more like e-mails or some of these posts here, and did not resemble a bunch of bullet points.

Looking back even before that, I found a bunch of old entries which could plausibly be called snippets from the '90s. Yep, back when I was running a flock of machines for a school district, I was keeping notes of what had happened. This actually happened for a couple of years before I got the idea to make a coherent sysadmin notebook for all of my machines. Many of those older entries were used to "backfill" the notebook for systems which had been around for a while, in fact.

What did a "snippet" from back then look like? That's easy.

- RBL measures installed on mx2 (sendmail x.y.z)
- nightly log rotations configured
- sendmail x.y.z on mx1
- added virtual domain mail handling to mx1 for dom2 and dom3
- linux migration work:
- turns out bsd/os shadow passwords work fine on linux
- conclusion: mass password change unnecessary
- mail change: envelope froms must now be fully qualified
- upgraded qpopper to x.y
- sendmail x.y.z on mx3

Yep, the "RBL", as in the original DNS black hole list. Like I said, it was the '90s. This is the slightly-sanitized entry for just one day of sysadmin monkeying I did one summer long ago.

All of this brought together makes one thing obvious: I have always liked to write about the things I'm up to, even if there's no guarantee of it ever being read. If nothing else, it's a terrific way to answer questions like "when did X happen?" or "what were you doing on Y?".


A couple of years into my run of doing snippets for That Place, I had noticed a few things. Some people didn't write anything about what they did. Their managers obviously didn't care, and so on up the line. When it came time to figure out what they had been doing for the purposes of a peer review, you couldn't use that as a hint.

I never figured out what made people ignore these things. Were they "too good" to explicitly tell people what they were up to? Did they do it on purpose so you couldn't see how little they got done in a typical week? Was there some other devious plan? Who knows.


Then there was the time they started screwing around with the way it worked. First, they moved the nag mails to Friday afternoon. Now you had to make sure you didn't whack that mail if you encountered it before Monday morning when you'd normally do your update. I think that only lasted a week, though, and it apparently reverted to Monday.

Not too long after that, they disabled recognition of holidays because, well, "Mountain View is not the world", or something like that. Instead of localizing the system for the user and their set of holidays, it would just pretend none of them existed.

You see, previously, it knew about Mondays which were holidays, and would delay the nag mail until Tuesday. This was great, since you could then reply to it by that magic time on Tuesday and it would still apply to the prior week. It assumed that you didn't log in to work and do mail on the holiday, and gave you an extra day to sync up. It worked just fine in my experience.

When they turned off holiday recognition, it started nagging everyone on Monday no matter what. Of course, this happened around Labor Day, which is a Monday holiday in this part of the world. By the time everyone got to work on Tuesday, it was too late to just reply and have it go to the right place. If you cared about getting it right, you'd have to manually go to the site and drop your update into the prior week.

I always wondered how many people had their snippets shifted due to this. If you think about it, it would make some weeks have no report, and some weeks would get a double dose: the one which showed up "late", plus the next one which showed up on time a week later. It could be rather confusing.


Of course, some projects decided they weren't going to write snippets on purpose. One in particular went off and did their whole thing in secret, and locked everyone out of everything they could control. You couldn't discover what they were up to from snippets, and you couldn't see anything in their code reviews, either. The actual code itself was locked down and secret, and you had to be on this stupid list of "special people" to get around any of that.

They missed locking down a few things, so it became possible to see some of what they were up to just by digging around in a couple of unconventional places. What I found didn't seem to make much sense. It was a big bunch of crazy. When it finally launched, it was slow, confusing, and didn't have much happening in there. Hardly anyone knew what to do with it. They apparently didn't know what to do with it, either.

I wonder if the lack of openness make this happen. By operating in secret like that, any sort of oversight from people who had useful experience in different arenas was forcibly rejected. All of those "what are they doing about X" questions did finally happen all at once when they did their internal demo, but it was too late by then. They launched a couple of weeks later, and all of those problems were still there, naturally. There was no way to address them by this point.

Then again, just flaunting the "we're special and you're not" probably earned them the "evil eye" from more than a few people. If curses really exist, I'm sure more than a few were floated around due to this.


What project do you think I'm talking about? Do you have some idea?

Are you sure?

Do you realize I could be describing several?

Oh yeah. Believe it.