Writing

Feed Software, technology, sysadmin war stories, and more.

Saturday, March 24, 2018

Heat loss from big corporate projects

Years ago, I wrote about something I had learned in "corporate America". It was called the "buckets of money" situation. It refers to how sufficiently large companies might move costs around. Team A might decide to stop providing some service, which saves them money, but then team B has to ramp up and deal with it, which then costs them more money. A might be doing better, but B is doing worse, and so it's a wash for the company itself. No real money has been saved.

That was five years ago. Since then, I've learned some more things. Specifically, not only do you have the aforementioned problem of just moving costs around and not necessarily getting a win, you also have a new problem: heat loss. This is where the very act of moving things around costs energy, and just like heat loss in the real world of science, it escapes from the system. This is something you can't get back once you spend it.

To bring back the old analogy, let's say instead of buckets of money, we have tanks of water. Both of them have 50 units (gallons, liters, whatever you like) of water in them at the moment. Together, they provide 100 units of service to the customer.

One day, someone at the water company decides they want to move it around. Maybe they want to make their tank weigh less. Who knows. But they adjust the balance and now their tank only holds 25 units, while the other one picks up the slack at 75 units. Together, they still hold the same 100 units for the customers, but what actually happened?

The process of changing the system to have a new balancing point took work. It involved investments of engineer time, design work, tech writers and documentation changes, and all kinds of other "meta" stuff. That is in addition to the actual cost of pumping the water from one tank to the other. This is something that would come up any time the system is rebalanced, and once spent, you can't get it back.

In the tech world, this might be what happens when the hardware provider says that you are now going to have half the disk space in next year's machines. Instead, they will have network storage available. This saves a lot of money on hardware, but now the software people have to adapt. They wind up designing some kind of more efficient scheme to fit onto the smaller machines, and in the end, the system runs, and outwardly, it looks the same.

However, a lot of engineers had to spend a lot of time to rebalance the system that way. Someone had to notice the problem and call time out, then someone had to analyze disk space usage to find out how bad it would be, and then come up with a way around it. Then other folks had to code it up, test it, roll it out, and switch everything else onto it. The time spent doing that was time not spent doing other things. That's time and energy the engineers and company as a whole never gets back.

This is why I warn people about a change to a system which is only supposed to move things around but which by itself does not introduce any particular advantages. If there's no obvious win from the proposed environment, then it's not actually a "wash" as some might say. It's really a net loss of energy and opportunities because of the work required to convert from the old system to the new.

There is one more insidious part of projects like these: they create work for people. Those people can work their tails off and deliver this stuff which does the same service outwardly but with a different balancing point. Nothing improves for the customers or even the system administrators. It might even get worse in some dimensions. But, it kept them busy, and let them show off, and they can take that all the way to the bank by way of a performance review system that optimizes for Shiny New Things.

Some would say this has already happened.