Writing

Feed Software, technology, sysadmin war stories, and more.

Tuesday, December 25, 2012

Unscrupulous people may take advantage of your skills

I used to go to unreasonable lengths to fix things which had broken due to other people skipping necessary steps in their projects. I always rationalized it as being the right thing for the users, but it always seemed to set a bad precedent. The other people on the "team" could just fritter away their time and I'd always come up with some kind of save.

One such example comes from the days when I was running the turnkey CD tower solution for my school district gig. They had been running them for about five or six years at this point, and wanted to start rolling out "version 2": better hardware, faster response times, that sort of thing. They'd also have CD-RW drives so the images could be "reburned" to make copies of things.

One day, I got a mail from the people on site telling me that the hardware for schools G and P were ready. This meant they had dropped in my "remote login disc" image and the machines had come up on the network. I had supplied a small ISO image which would allow a machine to boot from CD and then bring up its network and start a SSH server. I could then jump into it and do anything which had to be done.

This disc primarily existed for rescue purposes, but it turned out to be useful for doing the occasional install, too. They didn't want to have me rig up another automatic installer this time around, so that's how we handled it. They'd boot a new machine off this CD, and then I'd ssh in and paste in a bunch of commands. A minute or two later, it would be ready to go. I installed only the minimum set of software, and you don't need very much to be useful.

A few days after that, I got a phone call. They were about to turn on the new tower at school H. I said that was very interesting, given that we had never installed a machine for that school. That suggested they had grabbed the box for G or P. This wasn't a big deal, since I'd just adjust my notes on who had received which box (MAC addresses and such in my sysadmin notebook at that job) and go on with life.

Then they turned it on. It flailed around a little bit and hung, saying "NO OPERATING SYSTEM COULD BE LOADED". Great. It was a third box they hadn't mentioned. This one was never put online at the shop with my remote login disc, so I had never put my custom Linux-based distribution on it. Now they had a blank machine out at the school and needed to get it fixed, and oh, could I help?

They didn't have the install CD with them, and there was no burner on site at that school. They also didn't want to drive all the way back to the office to get it going. Instead, for the next hour or so, we endured countless floppy disc problems with the old machine (which was still at that location and operational) as I tried to use it to make a floppy-based version of my rescue CD. I already had the image ready to go, but that machine's floppy drive hadn't been used in six years (when it was first installed), and had clearly stopped working in that time.

I finally got them to use a nearby Windows machine to run 'rawrite'. That was also a "fun" exercise, since the old DOS program would not work under Windows. We had to dig around to find one which could run under that particular flavor of Windows and would give a usable result.

Then it turned out that image wasn't actually ready for whatever reason. I had to frob the read-write flag and then run LILO again. Somehow, they blew the dust out of the old tower's floppy drive and that let me mount the floppy and fix things. They then fed that floppy to the new machine and that brought it up on the network at last.

Now that I could ssh into the new machine, I found out just how much they had skipped. It was already clear that they hadn't done my installer thing, but they had also skipped the RAID card setup. This was their only real responsibility, since I had no way to automate that. They were supposed to receive the machine, shove in the drives, set up the array, drop in the "remote login" disc, and call me. Now I had to come up with a way to set up the RAID from inside Linux itself.

Fortunately, I got lucky and found something called tw_cli which do exactly that for this kind of 3ware RAID card. Of course, it was too big to fit on the teeny little rescue filesystem I was using, so I had to manually create a RAM disk with dd and mke2fs. Then after mounting that space somewhere useful, I used wget to pull it across, untarred it, and then finally started the tool to zap the JBOD and create a real array.

Finally, I could run my installer script, and that did the rest. A couple of minutes later, it was ready and the box went online for real, booting off its own local resources at last. Then I had to spend another 20 minutes or so schlepping that school's CD images over from the old machine while explaining how things worked now. My "boots on the ground" people there in town hadn't done any of it.

At the time, I used to think I was being a good little drone because I could use all sorts of wild hacks to get things done. Because I found a way to install it using the tools at hand, they didn't have to cart the machine back across town to the office to reinstall it and then drag it back out there.

The flip side of that is they came to depend on me to "just do" certain things. It's like having a maid who's always there to pick up after you. A certain type of person might deliberately leave trash around just because they know someone else will come along and tidy up.

It's one thing to innocently do this. Some people genuinely have no idea they are doing something bad and could change their ways for the better. This is not about them. I'm talking about the ones who know full well what's going on and keep exploiting it anyway. To me, that's evil.

Beware of excessive wizardry. People may take advantage of you.