Writing

Feed Software, technology, sysadmin war stories, and more.

Sunday, March 25, 2012

Tickets should be the exception, not the norm

I discovered an interesting notion towards the end of my days as a web hosting support monkey some years back. Someone else out there had come up with the idea that tickets should be the exception, not the norm. For a company which churned through thousands of tickets per day and even had a Y2K-type problem involving their numbering at one point, this was pretty far out there.

With your nose to the grindstone, it's hard to think of anything but just hiring more people to help out in the daily battle against the onslaught. This guy's idea came at it from the architecture astronaut perspective of "look how we could move beyond tickets and look cool". I essentially wanted the same change, but for a different reason.

My motivation came from actually living the life of chugging away at tickets for hours at a time. I wanted them to go away and not come back. There were so many dumb duplicate things going on which really never needed to exist in the first place. In my eyes, anyone halfway sensible would eventually tire of this and would want to go do something else. After all, what's the point of knocking down a ticket queue which is just going to build back up again?

Actually doing something about this requires a shift in priorities. As a company, you have to let your people explore. The ones who are hard-wired to resent boredom need some sense of control over their environment. If all you let them do is sit there and order another helping of gruel, you're not doing them any favors.

So, okay, how do you actually do this? Well, you have to dive in and see what keeps coming up. What kinds of tickets drag on for days at a time? Which ones, no matter how simple they are, keep being created in ridiculous quantities because of a trivial gap in tooling?

One obvious scenario was SSL certificate generation. I've discussed this in the past, and it basically went like this. Our customer would talk to their account manager to say "hey, I want a cert". Their AM would drop into some tool to start the process. This sent a magic one-time URL to the customer via e-mail. When the customer loaded that page, they saw a questionnaire which asked for all of the names and other details we'd need to generate a certificate and signing request. Once they hit [submit] on that form, a ticket would pop up in the support queue.

Some poor tech then had to take that ticket, go to that "product number" (unique for this new cert), and then read what the customer had typed in. Then they got to type those details into OpenSSL. Once they had a key and cert signing request, they'd paste those into the provisioning system for safe keeping and then jump out to the SSL vendor's site. Using some magic credentials which were passed down by the tribal song of the ancestors method, they'd log in and paste it all in.

If everything was in order, then the vendor would accept it for processing and it would go to sleep for a while during verification. Some days later, the vendor would issue a cert and mail this mailing list you had to be on (again, tribal song stuff). You'd check back, then go log into their site, and copy/paste the cert into both the provisioning system and then drop it onto the server. Finally, you'd update the ticket with a comment to the customer and close it.

What a horrible mess. It used humans as the glue between far too many systems. Right up front, the customer should be able to order just by going to their portal and saying they want a new cert. Then they'd get the usual questions and provide the usual answers. The backend would then call OpenSSL itself and use some sort of API to throw the mess at the vendor. It would then be responsible for dealing with the follow-up and resulting certificate.

Only then would it be appropriate to loop in a human for installation of the actual cert. This would then become the new pain point. It's significantly better than what had existed before, but eventually people would get tired of it as well. This is yet another place where programming might be of some assistance.

If the customer had a control panel, then their virtual hosts and certificate locations are well-defined in some backend database. It should be possible to verify the incoming cert for validity (CN, dates, etc) and then push it in and turn it on without having a human in the loop. If it works, it declares victory, and that's it. No support techs are involved.

The only time you should need to fetch people for routine tasks is when something blows up. This is the embodiment of the original concept: tickets should be the exception, not the rule.

Some might cry and howl that adding such things takes away the humanity from the support process, and that's what customers are paying for. To that, I say... false! The fact is, you have a better chance of actually talking to real person for an actual problem if they aren't all tied up doing stupid monkey work which should have been automated long ago. If you take care of everything a computer can do, then the only things which are left by definition are those which call for actual people.

After all, what's a better experience? Spending 30 minutes on the phone with a tech who explains what's actually going on because their queues are looking good, or being rushed through a call in 10 minutes because they have a backlog piling up?

As someone who's been on both sides of that call, I know which one I'd rather have.