Software, technology, sysadmin war stories, and more. Feed
Wednesday, March 14, 2012

Do your homework before applying for a transfer

I should have done my homework before transferring. I should have been able to sniff out all of the insanity before making the decision to jump onto a new team. But, I didn't, and my sanity paid the price. This is my story of transferring to my last project before I bailed out entirely.

The first thing I noticed was that nobody seemed to know anything about a continuous build and associated tests. I did some digging around and found that a profile had been established, but it wasn't doing anything useful. Oh, sure, it was running stuff, but nobody on the team was watching it, so it was completely pointless.

I decided to do something about that. There was a little browser extension you could add which would give a small color-coded OK or ERROR notation in the margins. This way you would always be able to see what our tree status might be, and whether something had broken during the day.

As soon as I got that running, I saw it was red. Then I pulled up the history and looked at prior runs. Not only was it red (failing) right now, but it had been like that for weeks. Clearly, nobody who had already been on the team prior to my arrival cared about it *at all*. This was a place where they posted signs in the bathroom stalls about testing and continuous builds, so ignorance was no excuse.

It took a long time to do something about this. Many of the tests were incompetently written and did foolish things involving threads and sleep(). Yes, the test would actually start something running in another thread, sleep for a few seconds, and then assume it had finished. Clearly, the notion of writing testable code had never occurred to the authors of this service, since there was no easy way to make it use a semaphore or otherwise run synchronously for testing.

One day, the continuous build went "green" with a big OK. This greatly surprised me, since things were still seriously screwed up. One of the folks who had been on the project when I arrived filled me in: the testing system itself had probably broken.

That's right, seeing "OK" was actually in error. That's how incredibly screwed up things had been, and they were just laughing right along with it. Naturally, I could not stand this. How could you possibly try to make changes to a project if you cannot verify its operation before and after?

I mean, it would be one thing if this was some dumb project which just let you post pictures of cats online or something like that, but this was far more important. Having bad software in this position was equivalent to handing a pipe full of crystal meth and an assault rifle to your favorite great ape. You don't know exactly what's going to happen, but you know it's going to involve a lot of blood and feces.

Then again, given the absurd quality of the software "engineering" which went on there, it's entirely possible that the meth-monkeys had already rolled through at some point in the past. There certainly were enough turds around.