Software, technology, sysadmin war stories, and more. Feed
Friday, February 3, 2012

Purposely working with so-called "problem customers"

It seems inevitable that any group of people who work tech support will eventually build a list of customers they'd rather not handle. These are people who, for whatever reason, rub them the wrong way. Maybe they just don't get along, or a support issue went bad somewhere along the way, but there's just no love any more.

I learned about this upon starting my "phone and ticket monkey" job some years ago. On an evening a few weeks after I started working there, one of our front-line phone firewall folks answered a call from one of their "problem customers". This was someone who they all knew and nobody really wanted to talk to him. One of the younger guys said a few unsavory things about him.

I found this curious and wanted to know just what could be so bad. After all, I had no history with this particular customer, so as a result, he would have no beef with me, either. He might have some issue with the company, but that sort of thing doesn't have to be fatal.

I volunteered to take his call to find out what all the fuss was about. I was connected with someone who ran a small web hosting business on his leased server and who lived just outside New York City. He was the stereotypical "type A" person who knows what he wants, is going to ask for it, and isn't going to take any excuses.

I'm okay with that sort of person. I like people who know what they want when it's a reasonable request. Also, he would start off with a calm demeanor and would only escalate if he started getting bad support or other unacceptable handling by the person on our end. As long as you knew what you were doing and told him what was going on, everything was fine.

And so it went. The problem this particular evening was that he could no longer connect to his mail server for some reason. As someone who fought the spam wars in an earlier life, I knew that ISPs were starting to filter port 25 from some of their networks, and he may have fallen into such a situation. I personally approve of this choice, but the way they did it broke service for a fair number of people.

To verify this, I jumped on his server, fired up tcpdump, and had him try to connect with his mail client. Nothing appeared. Then, I got him to drop into his mail config, and in there, we configured it to use some other port number temporarily. This time, it immediately showed up in the packet trace. He wanted to see what I was seeing, so I got him to ssh in, su over, and run "screen -x" which attached itself to my session. Then, as we talked, I started up the sniffer again and let him poke at it.

I know he must have gone back and forth a couple of times on his end just to see what happened. 25, nothing. 26, works. 100, works. 25 again, nothing. At this point, he got it. He realized it wasn't anything we had been doing at the hosting company and that his ISP was to blame. I'm sure they got a nice phone call from him later on.

But, I didn't stop there. As long as I had him on the phone, I said that we can set up a workaround so your server will listen on *two* ports for mail service. Then, you can just leave your Outlook or Eudora, or whatever set to that port if your ISP won't open it up again. Also, if your customers have similar problems and they're on the same ISP, you can just tell them to flip the port over and go on with life.

I think he liked the fact that this put him in control even if the cable company decided to yank him around for a while. It also gave him ammo to use when confronting their support people.

I wound up setting up some wacky iptables magic so that traffic arriving on another port would be redirected to port 25. Then I put a huge note in his server's /etc/motd which included the ticket number where all of this was happening, and added a note to his account advising that a seriously nonstandard hack was in place.

It's been years, but I think this was because he was running some weird MTA which wouldn't just let you switch on port 587 or similar. In any case, the setting worked, and he was happy.

Thereafter, interactions with him were different. He'd ask for me, but even then, people weren't afraid of him any more. I guess he realized that we were on his side and at least some of us could deliver the kind of service he was paying for.

Side note: this is not the same guy who ran the web site we checked on a slow Saturday night in a previous post. This is actually a second person from the same part of the world who just happened to behave much the same way. Most techs really did not like dealing with either of them, but I managed to change that situation.

And hey, what's this? I just looked up this second guy's web site, and sure enough, he's also still hosted there. Good deal.