Writing

Software, technology, sysadmin war stories, and more. Feed
Monday, October 29, 2012

How far you go before giving up

When it comes to obstacles of a technical nature, are you a "fail fast" sort of person, or do you go digging around to see what's happening? If the officially published methods for getting access to something which you are supposed to have aren't working, do you just file a ticket and wait for someone to respond, or do you find another way in?

In my web hosting support monkey days, we had a fairly boring setup for authentication. The customer would get a single user account created with random password when the machine was installed. We'd also create one for ourselves -- call it "tech". Finally, we'd set the root password. Customers were expected to tell us whenever they changed the root password so we would have it on file in our systems. That way, when they opened a ticket for support, we could use it to 'su' and help out.

Far too often, I'd encounter machines where someone had changed it and had effectively locked us out. A bunch of times, this was because the customer thought that "change root password" in their support portal actually did something on their machine. It didn't. It just filed a new set of passwords in the provisioning system. So, the first time a tech would come along and try to 'su' with that password, it would fail.

Some people would stop right there. They'd go back to the customer and say that they were unable to help due to the lack of a password, and then would ask them to update it in the portal or tell us what it was. This let them set the ticket to some state which put it to sleep for 48 hours, and got it off their plate.

I couldn't abide by that. There were so many more things to try. First of all, there's a [password history] button for a reason. If the customer put bad data in the portal, odds are that the previous password history entry will still have the good data. Given that, it's easy enough to pull it up and try to 'su' with that. If it works, it's no big deal to just update the data to again show the right password on top.

If that one is no good, sometimes even older entries will work. If none of those pan out, there's always sudo. By default, we didn't grant the "tech" account sudoers access, but some machines had it anyway. On those, we'd usually put a note on our side saying "[use sudo]", but some people didn't bother to help others out that way. Of course, if we weren't in sudoers, that wouldn't help... or would it?

I got pretty good at using yet another way into those machines. All of the Linux boxes we installed which did not have a "real" control panel like Plesk from us got something called Webmin. It was a separate daemon which sat on a distinct port and required a username and password to allow admin access. Customers usually forgot about this, and so it would sit there with the username and password combo originally assigned when the machine was first installed.

If we didn't have a good root password and couldn't sudo either, there was still a good chance that Webmin would let us in. It had a facility to run a custom command. With that, you could just paste in a command to append a line for "tech" to /etc/sudoers, and it would run as root. After that, the rest was easy.

Sure, it's fiddly, but it means you can knock out a ticket with only a minute or two of extra messing around. The alternative is throwing the ticket back to the customer and waiting for them to figure out what's wrong with their passwords and come back. That could take hours or even days, and all that time, their request isn't being fulfilled.

Little things like this can make for big differences in customer happiness.