Writing

Feed Software, technology, sysadmin war stories, and more.

Saturday, February 29, 2020

No more pink mustache

As I write this, I'm in the final couple of days of what seems like the longest year in my career to date. While it was just a hair over 365 days, it seems like it went on forever. Living though this thing took a lot out of me, and choked off cycles I might use to do other, more useful, stuff.

It's amazing just how deep these "stacks" of blocked things can get.

18 years ago, I wrote some terrible software for maintaining my diary. Aside from a forced migration to C++ to use STL containers to get away from my degenerate hand-rolled data structures, it hadn't changed very much. I had wanted to improve it, but could never find the energy or focus to do it.

Part of the problem is that I don't want to write super-old-school C++ any more. That is, I want to be able to use a std::unique_ptr instead of my own hacky "scoped_ptr", and I also want to be able to use std::make_unique instead of whipping out the "new" everywhere. That is, I want a call to "new" to be the kind of thing you check for in a linter and reject.

Trouble is, I had an ancient machine in the flock. It didn't seem like it at the time, but my main serving system was magpie, a RHEL 6 box from 2015. It was just like the machines at work back then, and so it was nice and familiar. The problem is that it had an ancient gcc which only did C++98, never mind 03, and forget anything about 11 or 14.

I had tried to "side-load" a bunch of new stuff, but it always came back to bite me. You can't mix libraries across compilers in the world of C++, so it was all or nothing. I really wanted a fresh start that was actually using the default EVERYTHING, instead of having to own and maintain all of that crap myself.

To get away from it, I'd have to migrate to another box. That meant picking out new hardware at the hosting company. That was also a pain because their interface is wonky, and the prices hop around, and it seemed like it was actually going to get more expensive. There wasn't anything in the same range, at least, from what I could find given the limited amount of patience I had to spend on it.

So, for a long time, nothing happened. I thought about the need to do all of this stuff, but none of it made any progress.

I sat around. Time passed. I got older. My projects went nowhere.

Then, a couple of weeks ago, I put in my notice, and things started slowly opening up. I found myself able to sit down and figure out what hardware options the hosting company had, and even found something that was a slight discount on what I was already paying. Imagine that: a newer box and the monthly fee goes down!

I clicked the button, and a few hours after that, I had a blank box, just waiting to get the breath of life from me over the remote console. Yep, they somehow don't support either RHEL or CentOS 8, so I got to install it myself. It worked fine, and soon I had a regular box and the migration could begin.

A few days after that, everything had been moved over, the DNS had been flipped around, and the old box was wiped and scheduled for cancellation. All of my stuff is now being hosted on the new machine, and with any luck, nobody noticed. Say hello to snowgoose, the latest in my flock.

With that out of the way, my attention turned back to my source code. It started with a few fixes here and there, applying best practices about formatting and cleaning up lint in general. But then, something switched on and I really started going for it, removing explicit "::const_iterator" cruft everywhere and rewriting loops to use the more succinct "for (const auto& x : y) { ... }" notation.

One afternoon, something really kicked in, and I went after the diary code. I don't know exactly how this became so simple, but it was actually nowhere near as big a problem as I had originally figured it to be. Just under four hours later (not all spent at the computer), it was done. The beast was dead, and I had a new implementation that just worked.

It's amazing how much can be done when you're not sinking in quicksand.

Let's review what had to happen to get back to this point, in reverse order of what was blocked by what.

So, yep, I left a place that was broken at a scale that is hard to believe. It's a relatively small company that has the kind of broken that giant 100K+ employee places don't even manage to have, somehow.

It's not even that I just left them to rot, either. I spent the last five or so months trying to fix the cultural brain damage. I came to the decision to work on that after realizing that as long as it was that screwed up, none of the technical work would matter worth a damn.

Throwing down and putting it all on the line to fix the fundamental problems is what I was hired to do, and I did it. I threw down. I invested time, energy, tears, sleepless nights, and lots and lots of emotion. This is what you want from your "principal engineer" rank. Done properly, it's not just a level. It's a commitment.

There were lots of meetings with middle management, and then upper management... and yet, nothing improved. Nothing changed.

Put another way, I went to Batman to say "help, there's crime in our city", and Batman shrugged and went back to watching TV. That's pretty screwed up.

I have a bunch of stories to tell, and they're all kind of half-formed, bouncing around in my head, all trying to fit through the doorway to be told at the same time. I'm not sure how many of them will manage to make it into words and will be shared with the world just yet, so wait and see, I guess.

Oh, and remember: wages and working conditions! It's fair game.