Writing

Software, technology, sysadmin war stories, and more. Feed
Wednesday, September 14, 2011

More tales from the trenches working support

There are technical chops, and there's seeing the big picture. When you deal with a company for support, which of them are you getting? Are you getting both at the same time? That's something special.

Here's what happens when things go right.

A customer opened a ticket and pasted in a chunk of an error message. They asked us for help with getting some library installed. I could have just installed the library and let it go, but instead, I looked up that actual error message online and found out it was from some bigger package.

My ticket update reflected this. I said that while I had tracked down the requested package, it looked like they were trying to build X, and in that case, here's a pre-built RPM just in case that's what you need. Think Clippy, only totally accurate and only popping up when requested. They were shocked and pleased that I "read their minds" and were able to get their project done.

Another customer was running BIND on their own server. We didn't really support that configuration since our official answer was "use the hosting company's nameservers", but we'd still answer technical questions about it. Someone wrote in and had a question about adding some kind of record.

I gave them the answer, but while I was looking at their zone file, I noticed something amiss. Up in the top, where you list your zone's nameserver (NS) records, they had something like this:

             IN       NS      ns1.their-domain.whatever.
             IN       NS      ns2.thier-domain.whatever.

See the typo? For whatever reason, that one jumped off the page at me, and I let them know. I told them about it and said, hey, this is probably not what you wanted. Someone who registered the typo version of the domain could make things interesting after an initial query hits your server and picks up this one bogus record.

Their response:

You see, this is why we think you guys are the best! I cannot believe that you caught this and I very much appreciate you letting me know.

I have updated our own zone file () and restarted bind.

THANKS AGAIN, THAT PROBABLY THWARTED A HUGE FUTURE PROBLEM OF SOME SORT!

You have to love responses like that.

Another time, someone was trying to stir libGD into some kind of image modification library for Perl. This was a bit after the whole Compuserve/Unisys LZW patent thing where things like gd removed support for GIF. At this point, gd had just added it back.

Our customer was calling some part of this perl library which was supposed to read a GIF but she was getting an error which said "Not a JPEG file". This made no sense because she was explicitly calling "newFromGif".

I started down the road of thinking gd didn't have GIF support, knowing about the whole LZW thing. That was a dead end, since this particular machine had a build which added it back. Finally, i started looking much more closely at GD.pm, the Perl wrapper. Digging through its source, it turned out it would say that exact error in 6 different places! I couldn't be sure which one was firing.

Being a big fan of "debug by printf", I added numbers to each one. A few seconds later I had my answer: "Not a JPEG file 4". I jumped into that part of the code and found a dubious call right above that error message. It was doing some kind of ALLCAPSFUNCTIONCALL with GIF in the name. That screamed "macro" at me, so I went looking for a matching #define.

Sure thing, up in the top, buried in a sea of #ifdefs, there it was:

#define ALLCAPSFOOBARGIF foobaropenJpeg()

Oops. It looked like someone had done a cut and paste of the JPEG code and forgot to flip both of the words over. I changed it to "Gif", rebuilt and reinstalled over the old mess, and ran the test script. The error disappeared! Bliss! Again, happiness all around.

The environment may have been hellish, but helping real people with real problems was pretty rewarding. Knowing that they actually benefited from the work and were pleased with it went a long way.