Writing

Feed Software, technology, sysadmin war stories, and more.

Friday, May 8, 2020

Responses to reader feedback

Ah yes, so yesterday's post about my stupid little RPC load tests blew up. There's been a lot of feedback to it and a couple of other related posts. I will attempt to answer or otherwise address some of the points.

...

Point: People want to know what language I wrote it in.

I'm not going to tell you, at least, not yet. It actually doesn't matter. I submit that you can write terrible services in any language, and you can probably write halfway decent services in any language. Where the rubber meets the road, however, is when you start squeezing that last 66% out of your hardware.

Yeah, that's right, I said it. Last 66%. It's that bad sometimes.

If you haven't had to work at a place that thinks that scaling up at 33% CPU utilization on a host is normal, then this might not make any sense. I have. It's terrible.

By not telling you what language it's in, I manage to starve out a whole side-shitshow of the "conversation" in which people have a full-on language war and forget about the fact that they still haven't managed to accomplish the same thing in their language of choice.

I bet it can be done in the language. Whether it can be done by the people who just bark all day and accomplish very little? That's another story entirely.

I will point out that much earlier in this process, a friend decided to write a similar server in a totally different language and also managed to totally destroy the numbers that other places take for granted. The language that friend used is also irrelevant here.

...

Point: People want to see the code.

Oh, I'm sure they do. Some people probably want it because it delivers. I'm not going to just give it away. Given that this kind of thing seems to be unreasonably rare in the world right now, I figure maybe there's money to be made here.

Just think of it. An unencumbered code base in which the only external dependency is protobuf, and last time I checked that's basically a 3-clause BSD so you can basically do whatever you want as long as you heed the documentation/copyright requirements.

Also, the protobuf part isn't exactly critical. You could replace it with a binary-compatible implementation and nobody would know the difference. The wire protocol is well-documented and is easy enough to hack up *by hand* if you just want to test things.

Has anyone tried making a second-source for gRPC? You know, speak the same language but not use that same implementation? What would that be like?

Imagine something that won't clone 50 different "sub modules" from random places on the Internet when you check it out. What a concept, right?

Also, as I already said, I'm not a part of that whole community thing for multiple reasons. I'm sure not going to hop into it with something like this.

...

Point: My single-threaded server was interesting.

It's not single-threaded, sorry. It uses one thread per connection as a deliberate design choice. I didn't feel like bothering with thread pools. Could I? Sure. Have I? No, or at least, not yet. If I need them, I'll add them. Right now, the simplicity is worth it.

...

Point: what about on-prem vs. dedicated?

I'm a veteran of the "dedicated server" market since I used to do tech support for that side of the world a long time ago. It was a natural thing to get a machine back when they were a $5/month employee special, and it was very easy to keep it around afterward.

I've been through several dedicated servers in that time. They just sit there and kind of do their thing. They're boring, just the way I want it. Any one of these boxes I've had over the years could run several business concepts far enough to find out if it's worth investing more in it. They usually run between $100 and $200/month.

To me, people saying "on prem" now means some closet in their office. Offices these days frequently have very nice connections, like gigabit or better, with decent transit to anywhere that matters. I came up in the days when having a dedicated T1 line (~1.5 Mbps) for Internet into your business was a big deal. I hosted stuff on those connections. They also worked fine.

Would I do that again? Maybe. It depends on how quickly I needed to get things happening, and whether I couldn't just glue something to the side of my existing server.

Everything comes with risks. Your office connection will have a bad day. Your servers will die. A squirrel will barbeque itself on your transformer and you'll lose power for the day. A dedicated provider will also have stupid things happen from time to time. Those machines also die now and then.

Different problems call for different solutions. Time is a factor, money is a factor, what resources you already possess is a factor, and the needed flexibility is a factor.

That spare box in the closet might just be enough to plant your flag. Just don't forget to re-assess the situation later on.

...

Point: what if I just do blocking I/O? (why use a condvar?)

I assume this means ditching the epoll stuff and just sitting there in read() or something like that. Besides the fact this is what I deliberately avoided in my design, there are other problems.

My "serviceworker" threads need to do some housekeeping from time to time. They might need to do an idle check in order to punt a client who hasn't spoken in a while. There might be other things to do. Also, I wanted the ability to clean up and shut down without having to do terrible things to get them "unstuck".

Generating a signal to knock it loose? Ugh. Cancelling a thread? Eww, no. Those are nasty compared to what I ended up doing.

I'm very happy with the situation which currently exists.

...

Point: what if I use a coroutine?

I like the fact that the only thing scheduling my workers is the kernel itself. You can't get it jammed up. If you call out to something that "blocks the worker", you aren't going to end up with a whole bunch of other stuff which also suffers right away.

It takes screwing up the whole machine in order to make everything stop. Until you get there, things just hop around, scheduling other places. It's also a nice slow descent rather than just driving off a cliff.

The same goes for me and having my own thread pool implementation. There would be the same concerns, plus the added worry of doing it correctly if I had to roll my own.

...

Point: you disappeared in 2013. are you still around?

This one is really confusing. It's like, yeah, I did stop writing every single day in 2013, because I got pulled into the big blue swamp in Menlo Park.

But even then, I still resurfaced from time to time when there was something I needed to get out: "don't kill -9 -1", "Apple's shipping a terrible git build you can't even turn off", that kind of thing.

Honestly, though, as a job ramps up, it takes more and more energy away from the pool that would otherwise be used to spend time writing here. That's how it goes. I sign up and I give it everything I have. Call it unhealthy, but it's the only way I can operate. No half-assing things.

How this person managed to read one post from 2013 and not see the rest of them from well past that point, I may never know.

...

Point: you're just saying "you're all so dumb for using Python".

No. That's not it at all.

Some people... are using Python... BADLY. Like, trained monkeys sitting in front of a bunch of dumb terminals wired up to Unix shell accounts running "nc -l (port)" in a loop could run the server better... that kind of badly.

Most people? You don't hear about them. It works well enough for them. They aren't using it badly.

Some people are also using PHP, C, C++, Perl, Ruby, Rust, Haskell, OCaml, Java, Clojure, BASIC, shell scripts, D, Nim, Kotlin, Julia, JavaScript, Scala and Crystal badly.

The rest? Aren't. You don't hear from them, either.

...

Point: our people can't handle anything else (different programming languages)

I've been told this a few times by relatively high-up people at different companies. This is really terrible! You honestly think you work with complete nincompoops who can't learn anything? If that's true, why do you keep them around?

I also posit: it's not true. The fact that they are people means they are clever tool-wielding primates. People, when you have faith in them, will generally come up with interesting solutions to the problems you put in front of them. Just look at what's been happening with this Twilight Zone lockdown craziness of the past couple of months.

Restaurants had tons of cleaning stuff and no sit-down traffic to use it on. People stuck at home needed cleaning stuff. Restaurants put two and two together and started selling directly to the public. Brilliant!

I posit that most people are capable of more than some of these detractors would think.

You know who I love working with? People who are new at this stuff, haven't seen too much yet, but are open to anything. Imagine a typical intern. They're still in school. They've done a few things in classes and maybe at one or two other internships. They know they have a long way to go.

I present a problem to them. They go "I don't know". I say "well, you know what, I don't either". Then we sit down and figure it out together. I love that! Those are some of the most fulfilling stories I can think of. I got to do this last summer and it was great.

You know who I don't like working with? The people who aren't willing to try something for a better outcome... or even ANY outcome. They'll dig in their heels and not try the thing I pitched, but they won't deliver on their so-called solution, either. You get the worst of both worlds.

It's easy to say "no" to everything. It takes actual effort to give an alternative. Don't be the one who always defaults to the laziest, cheapest, crappiest path of least resistance.

Trust in your employees and their ability to be thoughtful, growth-oriented human beings who are willing to grow and expand their horizons. If you don't, someone else will.

...

Point: most companies aren't (Google | Facebook | Instagram | ...)

Oh, don't I know it. The problem with this line is that it can be used well, and it can be used poorly, and I see it being used poorly more often than not.

But you know what, I've seen the innards of those big places and some smaller places - some directly, some via trusted insider reports. Guess what? Far, far smaller places are having miserable times with their "infra" because they thought they could keep getting away with it. After all, they aren't F, A, the other A, N, or G.

The thing is, if you keep thinking like that, you never will be.

Remember, the dirty "secret" of tech is that you have to ship whatever shit will stick to the wall and kinda-sorta work as soon as you can, or someone else will do it first and beat you to the punch. Then, if you stay in business long enough to survive those terrible do-whatever years, you then earn the opportunity to clean things up.

The problem is those places which never quite realize that it's time to shift gears and stop behaving like that tiny little shit-flinging company. They think what worked before will keep working now, and I'm sorry, that's frequently not the case.

...

Point: having all of those connections in one process is terrible, because what if it dies?

You're right, if it dies, then we have a problem. So let's make sure it doesn't die as much as possible. We'll check all of our return codes and not assume things work. We'll deal with whatever exception type stuff our language provides. We'll build a system that aborts individual requests or connections and doesn't crash the whole process.

That process, then, will probably stick around for a while.

But still, sure, okay, run a couple of them if you want. You could even run them on the same box if you're that paranoid about it. I wouldn't, but hey, you're the boss.

...

Point: what if it restarts? OMG TIME_WAIT and not being able to bind to the port again!

I'm not going to demonstrate it here right now (this post is more of an anthology device, and we're staying on the train), but trust me when I say I can run the server in a "while true; do thing; done" loop, ^C it, and it'll come right back up -- even if there are clients actively whacking away at it. It doesn't have to be a clean shutdown either. SIGKILL it or whatever you want, and it'll still work out fine.

Why? Because I did my homework and twiddled all of those knobs. It's not necessarily SO_REUSE* ... there are more. Really. Go look.

Why? Because while I was developing it, I was just using netcat as my "client" just to generate TCP connection activity, and having to wait for a minute or two for things to clear out after killing the server really sucked!

This is the kind of stuff that you discover pretty quickly if you're actually testing as you go, and don't just settle for "well, guess I'll go get a coffee", and decide to dig in and clean up your messes properly.

Also, if you have a reasonable system for discovering instances of your service, you shouldn't need to come up on the same port. It shouldn't matter where you are, as long as there are enough of you available within whatever locality is needed to meet your latency requirements.

...

Point: don't reinvent the wheel!!1!

Do you say the same thing to everything else on Hacker News? Because, honestly, there's very little new under the sun. Most of what you read about is the same thing from five years ago with another name slapped on it (and perhaps another layer of indirection/abstraction added).

I bet you don't. Ask yourself, then, why you said it to me.

...

Point: too much jargon.

I looked at the most recent post for some statistically improbable terms just to see how bad it might be. Let's just see.

I suspect I could post about baking chocolate chip cookies and get the same complaint from certain individuals. Baking soda! Unsalted butter! Cup measures! Golden brown! Kicking off!

Basically, there's no way for me to do it right, because it's done by me. The topic doesn't matter, the presentation doesn't matter, and the wording doesn't matter. I should sit down, shut up, and provide them technical output like a good little peon.

Not happening.

NEXT!

...

Point: (similar tales of woe about terrible systems boiling when you so much as look at them funny)

If not for this quarantine thing, I'd probably buy you a drink of choice for being a fellow member of the "keeps random crap running" club. Hang in there, and remember, there are places where it's not like that all the time. Bide your time and keep your options open.

...

Point: do you account for DDOS and network issues?

I haven't gone to any lengths from my perspective. There's nothing in the code (yet) which says "oh, I'm under attack, so maybe I should do something about it". I could, for example, wave off connections after certain operations reach a certain amount of latency, or limit connections per source address, or per user, or any other dimension that might be useful.

There is ONE limiter in there right now: I actively reject new connections once I hit 32000 serviceworker threads. It turns out that going much beyond that starts making life very interesting since your VSZ is, well, massive.

Why did I do the thread limiter? Well, early on, I was hammering the thing with connections, and noticed that it would fail to do the usual stuff to start a new thread when up around that size. Now, to be clear, the program did not die. It would just have a funny error when thread creation failed due to a lack of resources, like a mmap call behind the scenes failing to allocate more memory.

The rest of the program kept going, as you would expect. A memory allocation problem is no reason to crash the whole thing, after all.

In terms of "network issues", those would mostly be the concern of the client. The client is admittedly stupid right now. You give it a host and a port, it nails up a connection, and it's off to the races. It's not doing service discovery, channels, load balancing, or anything else of the sort. Why? Because I wanted to solve for the server side first.

The client could totally get to the point where it is told "go find me a LavaLamp", brings in a (limited) list of them, figures out who's close by and who isn't, then starts attempting connections and doing health checks to see who's actually available. Then it'll just balance the requests from the user onto those channels to those instances. If one dies, no big deal, it'll find another place to go. The user shouldn't even notice.

This is what you get for free when you work at Google with Stubby, or at Facebook with Service Router. It's also what you don't get when you work at a place that hasn't solved this problem yet.

Aside: so, yeah, if I want to go past 32K simultaneous connections, I'm going to have to do multiple clients per thread, or spawn another process, or something like that. 32000 threads at 8 MB + 4096 bytes each equals a LOT of address space, and it's not surprising that I'm hitting limits up that high.

I'm not worried. Not even a little.

...

Point: why epoll and not select()?

I wanted to use more than 1024 fds without going to lengths. I also wanted to learn how to use epoll correctly so that it'll be quite obvious the next time I see something using it poorly.

I know this automatically walls off non-Linux solutions. I'm okay with that for the moment. If it starts to matter at some point, then I'll pull a libevent and implement it multiple ways.

...

Point: this "benchmark" is synthetic and crap.

Well, yeah. Of course it is. It's a fake lava lamp.

But then I started hanging actual use cases off of it, and hey, good times. We haven't talked about those yet, but oh yes, I took this code and moved some older stuff onto it. These things actually do real work, like I/O from the disk - gasp! - and frobbing data structures which are behind mutexes. OMG, actual lock contention because of concurrent accesses!

It works great there too.

...

Point: this is the equivalent of driving bigger trucks across the bridge until they break.

Well, yeah, that's the whole point. I wanted to demonstrate how you eventually can't get any more throughput and it just starts slowing down. You want to find that knee in the graph and then stay below it.

Obviously.

Also, I've been using that Calvin and Hobbes comic to tell load testing people to stop killing my systems for a very long time now. One group made it the unofficial banner of their group.

It's a good one.

Go look.

More writing | Contact / send feedback