Swap, swap, swap, and bad places to work
It's been an interesting couple of days in the world of Linux VMs.
Earlier this week, a post to the linux-kernel mailing list talking about what happens when Linux is low on memory and doesn't have swap started making the rounds. As the author points out, things get... interesting.
I saw this, and started thinking of my own experiences in this realm, and then a second post appeared on the same topic, citing the first, and adding more color to the picture.
And then, someone submitted an eight year old post of mine to HN this afternoon, and that brought the whole thing back around.
So, yeah, back in 2007 at some big web company, the hosts started paging me, and I'd log in and they just ... felt weird. They felt like something I couldn't quite put my finger on at the time, but it was probably something I had picked up back in the '90s when messing with much smaller Linux boxes. Maybe I tried running without swap and paid the price. It's been so long I'm not even sure any more.
Then, closer to 2016 or so, I hit it again at some other big web company. There, I was trying to advocate for "tinyswap" -- not NO swap, but not massive multi-GB partitions, either. I met one of the memcg developers at the time who also thought it was a good idea. Sadly, I never saw any progress on the technique after moving teams and before leaving the company a few years later.
Now, here we are in 2019, and we have a fresh set of people still fighting over it, like it's some kind of brand new dilemma. It's not.
I stand by my original position: have some swap. Not a lot. Just a little. Linux boxes just plain act weirdly without it.
This is not permission to beat your machine silly in terms of memory allocation, either. I'll just steal something that I had queued up for another one of my "reliability list" posts, but which is too useful to leave out of this one.
Item: If you allocate all of the RAM on the machine, you have screwed the kernel out of buffer cache it sorely needs. Back off.
Put another way, disk I/O that isn't brutally slow costs memory. Network I/O costs memory. All kinds of stuff costs memory. It's not JUST the RSS of your process. Other stuff you do needs space to operate. If you try to fill a 2 GB box with 2 GB of data, something's going to have a bad day! You have to leave room for the actual system to run or it's going to grind to a stop.
So, yes, the original team I landed on had done a bad thing by trying to pack the machines to the gills with data. But, what was more likely to happen on a team where they didn't take me seriously? Me getting them to "be more efficient" in terms of memory utilization (while raising the reliability factor by not having it constantly choke on replication), or me bringing back swap to let it keep going the way it had been?
Would it change your mind if I said they were also doing the classic $thiscompany thing of "this one is deprecated and the new one (which isn't ready yet) will fix it"? Would you still think they'd take me seriously and "spend resources" on re-sharding everything to not use so much memory?
How about if I added this was the same developer who rejected a patch I made to fix another problem where it would barf on db files greater than 2.1 GB because their scp binary wasn't compiled with large file support?
Imagine it: it's your first time at a megacorp in Silicon Valley, you're already not sure you belong there at all, you're more than a little homesick, and to top it all off, tons of people are acting like complete assholes and hating on your attempts to just make things run properly. Could you really thrive in that environment? What makes you so sure?
Oh, you must be THE ONE.
For everyone else, you'd probably cry too. I sure did.
I don't work there any more. Obviously. And I'm happier for it.
Any lazy fool can deny a request and get you to "no". It takes actual effort to appreciate and recognize what they're trying to accomplish and try to help them get to a different "yes".