Writing

Feed Software, technology, sysadmin war stories, and more.

Saturday, December 3, 2022

Short stories from outage reports

Not all of my stories are long. I have plenty of shorter ones.

...

Did you hear the one about the company that posted a bunch of videos to advertise something, and the backend screwed it up? Somehow, one of them got mixed up with someone else's video upload.

They were trying to advertise a sports league. They got something else entirely.

"... the video was switched to a dolphin swimming in a pool."

But it gets better: the customer asked the response team to not fix it, and instead to leave it alone because "engagement with their [ad] was going up".

This sounds hilarious, but what if it hadn't been another public video? Then it would have been a privacy disaster.

Crosslinked videos? That's bad news.

...

Or, how about the time a rare lightning storm rolled through, and a bolt hit some of the nearby electric infrastructure? It knocked half the campus offline, and everyone was forced to find other places to be, including the chefs who moved dinner to the one cafe building which still had power.

Someone put "zeus" in the root cause.

This one is actually funnier if you worked there, because there was a real service called zeus, and it caused all kinds of outages once upon a time.

...

Backhoes love to find fiber optic cables. Sysadmins know that the best thing to bring with you on a trip into the wild (in which you may get stranded somewhere) is a length of fiber. That way, you can bury it, and when the backhoe arrives to dig it up, you can hitch a ride back to civilization with the operator.

One fine day, a backhoe found a nice fat fiber run somewhere in the world. The updates from the scene were not encouraging.

"Cable is thirteen feet down and beside a creek. Water keeps filling up the space. Working to find a shallower access point. In the mean time, a larger backhoe has been requested. ETA 30 minutes."

You know the line from Jaws? We're gonna need a bigger boat? They're gonna need a bigger backhoe.