Writing

Software, technology, sysadmin war stories, and more. Feed
Monday, July 2, 2012

Hazing on my first night working tech support

On my first night as a web hosting support monkey, one of the guys decided to see how well I knew my stuff. One of our customers had a monitoring alert fire off. It was for SMTP, which meant something was generally wrong with their mail server and needed to be investigated.

As it turned out, the load average on that machine was rather high, and sendmail was doing its usual thing of refusing connections when that happens. I looked around and figured that their mysqld was burning all of that CPU time.

Upon mentioning this, this one particular tech said something like this:

Oh, customer X? Yeah, they're migrating next week. They know about mysql doing that on that old box. Just restart it and it should be fine. You know, just use killall.

killall is not my weapon of choice for such things, but I got the general sentiment. I found the proper init script magic and restarted it cleanly, and it went away. Some days later, they migrated, and that was that.

So where's the hazing? Easy. The machine in question was running Solaris. On Solaris, killall isn't like on Linux, where it'll fire off signals to matching tasks. Oh no.

On Solaris, killall... kills all (by default). Every pid you can signal gets a signal. This kills the box.

Fortunately, I knew better and yelled something back like "nice try" and went on with life. I had passed yet another test of my abilities.