Software, technology, sysadmin war stories, and more. Feed
Thursday, February 28, 2013

Welcome to Lake Tahoe! Now grab your pager.

Have you ever spent 5 hours on a bus to somewhere beautiful in the mountains, only to be told you were supposed to spend that day and maybe the next inside working on your computer? I have, and I hope nobody else ever has to experience this kind of treatment.

About four months into my pager monkey job, I was part of the on-call rotation along with everyone else. A typical day shift was 9 AM to 6 PM, night shift was 6 PM to 1 AM, and then there was time covered by Dublin from 1 AM to 9 AM -- all west coast times. The time zone difference meant Dublin's shift happened during their normal day.

Somehow, I wound up taking the AM shift for the week when the company had scheduled its annual ski trip to Lake Tahoe. When I found out about the situation, I shrugged it off and called it a fluke of bad timing. I didn't sign up for the trip, and fully intended to do my normal on-call shift either from the office or from home that week, depending on who was around. There'd be no point in being in the office if everyone else was gone.

The ski trip was to start on a Wednesday, very early in the morning. On Tuesday night around 5:30 PM, my manager got a hold of me. He really wanted me to come up there as part of a team effort (or something like that) and tried to convince me to go. I told him I had to be on call and that I hadn't signed up.

He said he would try to get the Dublin guys to cover, and fired off a mail to them. This was super short notice, and they were probably still asleep at that point, so we didn't expect it would work. I agreed that I would show up at the bus loading point at 4:30 the next morning and we would sync up. If things were going to be covered, then I'd get on the bus and would ride up there and have fun in the snow. Otherwise, I'd just wish them well, drive back home, and put in my day from there instead.

After my shift ended Tuesday night, I fought through the usual rush hour traffic and drove home. I packed quickly and then tried to get some sleep. It had been a miserable day on call and I felt like garbage. 4 AM came way too quickly, and I got up, got cleaned up, and drove to the office. There, I met my team at the bus, and things seemed fine. I climbed aboard and looked forward to not having to be oncall for a day or two. What luck!

The ride out of the Bay Area and up I-80 is a long one. I hadn't slept much the night before, and tried to sleep a little on the bus. I remember drifting in and out and having some short conversations with other folks sitting near me. At one point, my manager came up and said something to my seat mate about international dialing on his phone and how it wasn't working. I wasn't entirely "with it" at the moment so I didn't catch the whole thing.

One thing did stick with me, though, mostly because of the weird word he used. He said that when I got off the bus in Tahoe, I should get "ensconced", meaning "plugged in", and then tell him. I don't think I quite realized what that meant at the time, and even if I had, it's not like I could have done anything about it.

The bus pulled into Squaw Valley around 10:30. I grabbed my stuff and stepped off, then turned around to see this amazing sight:

Snow-covered slopes

I wasn't going to actually ski, but I was sure going to have some fun in all of that snow! That's what I thought, at least. Just then, my manager popped up and shattered that idea forever.

I was told the company had set up wireless access points in this one hotel right by the bus stop, and I should get online and "take over from Dublin". Yes, I was there, looking at the beautiful hills covered with snow, being hit in the face by the chilly air, and I was told I had to spend the day inside staring at a laptop and dealing with pages. I didn't quite realize what that meant at first and said something like "uh, okay". You know me, always there to help.

This is when the cold, lack of sleep, and hunger started hitting me. I hadn't had anything to eat yet that morning. I set off in search of food with a couple of friends from an affiliated team -- my actual teammates had split to hit the slopes. We walked into the little village and found that nothing would be open to serve food until 11 AM. Great. It's cold and there's no food to be had.

I decided to see what was up with my hotel accommodations. The company had basically booked every room in town for a three or four day span, and was rotating us through. A group would arrive on one day, spend the night, spend a chunk of the next day, and would leave that afternoon. Meanwhile, another group would arrive and overlap with them and replace them.

We walked over to my hotel which was on the other side of the little village. At my hotel, I was told that yes, I was on the list, but I couldn't access my room until 4 PM. So much for that. I couldn't get food, I couldn't flop down in a hotel room, and I was expected to start working as soon as possible. It just kept getting better and better.

We walked back towards town. By this point, it was just barely 11 AM, and this pizza place had opened. The three of us went in and had some food at last. It was maybe noon when we walked back to the first hotel by the bus stop where the company's wireless was set up. That's when I found out exactly what kind of accommodations we had.

The entirety of the wireless coverage was in this hotel lobby. It was basically four couches, a fireplace, and a chair or two, plus a WAP stuck on top of a tripod beaming signals into the room. Everyone there was just sitting there, head-down, looking at their computers, doing whatever nerdy stuff they usually did, and not saying anything. I sat down, got online, and edited the config file to put myself on call.

All the while, this feeling of dread had been growing inside. Here I was, stuck in that damned lobby for the next six hours while all kinds of stuff was happening just outside the door. I couldn't leave and experience any of it. I was tied down, surrounded by people, but effectively alone since they were all the stereotypical quiet, shy, and kind of distant Valley engineer type.

As the time ticked by, I had more and more trouble keeping it together. I couldn't exactly start sobbing right there with all of those people around me, but I couldn't leave, either. I asked someone if they knew when the bus ran and they said 8, 12, and 4. It was 1:30 by this point.

I started thinking about my options. I didn't have my car, so leaving under my own power was out, but I had a computer, Internet access, a credit card, and a cell phone. You can get a lot done with those four things, or even a subset of them. I started researching my options. There was a bus which went to the airport in Reno since a bunch of people from the east coast had flown through there. I actually went looking to see what sort of flight possibilities existed.

Yes, I was actually thinking about booking an immediate one-way flight from Reno to San Jose just to get out of that situation sooner. Now, the scheduling didn't pan out, but I was on the right track now: finding a way out of this hole they had put me in. I decided to take control of my own destiny for the first time that day.

I decided to just punt. I got a hold of the one teammate back in Mountain View who hadn't gone on the trip and was working a normal day in the office. I asked him if I could swap the on-call to him and said that I would owe him bigtime. He agreed, and so I made the change, forced it to commit to production, closed the lid on my laptop, and booked. I disappeared to a spot where nobody from the company would be: a bar and grill which (gasp) charged money for food.

I was seated at a table, and this was my view:

Snow globe

("Snow globe" was that year's theme, apparently.)

I spent some time there, and then slipped back to the bus stop area. When the 4 PM bus pulled up to take the previous day's group back to Mountain View, I just pretended to be part of that group and joined them. Then I just hid in the back and suffered another lengthy ride from the mountains to the office.

I finally got home around 8:30 PM. It had been a horrible day, and surprise surprise, I had to be on call the next morning.

The icing on this particular crap cake came a few days later. In a chat session with my manager, he actually tried to imply that I had screwed things up, and that it had been my responsibility to find coverage for my shifts, and that I hadn't adequately covered things. I reminded him that I had intended to work the entire week right here in town, and only gave in to his wacky idea at the end of the day on Tuesday because he had practically been pleading me to do it for "the sake of the team", or "to be a good sport", or whatever you want to call it. He had sent the mails to the Dublin guys looking for coverage.

As I said to someone at the time, you invite someone on a trip, you send out mails to the potential coverage candidates, and then when it goes south, you say you weren't going to take the responsibility? WTF?

Apparently, he had talked to Dublin while on the bus. That's what my hazy recollection of an international phone call was all about. They had agreed to take the coverage for part of the morning and then it was on me for the rest of the day. This meant things changed after I had gotten on board.

It was simple. If I had known I was going to be oncall from Tahoe, I would have never gotten on that bus. I would have just driven back home from the office, gotten some more sleep, and then done my 9-6 shift from here. Why else would I get on board?

When telling this story to a friend a few days later, this is how I summed up my time at the company so far: "I have done nothing useful in my tenure here except possibly fill in a gap in the oncall schedule". That wasn't entirely accurate, since I had actually been improving things here and there (when not being blocked, that is), but that's certainly how I felt.

I was just another body in a seat... with a pager attached.