Feedback: environments, old BBSes, Honda's 2022 bug
Yep, it's time for another round of reader feedback.
An anonymous reader writes in response to the "process infection" story:
comment: Hi! Loved your recent post on environment inheritence, and the note about system's approach reminded me of something relevant that Pottering recently wrote aboit the on the Fedora mailing list (highlighted in LWN's article on Fedora and pkexec @ https://lwn.net/Articles/883547/
The thing with setuid/setgid is that the invoked privileged process inherits a lot of implicit state and context that people aren't really aware of or fully understand. i.e. it's not just env vars and argv[], it's cgroup memberships, audit fields, security contexts, open fds, child pids, parent pids, cpu masks, IO/CPU scheduling priorities, various prctl() settings, tty control, signal masks + handlers, … and so on. And it's not even clear what gets inherited as many of these process properties are added all the time.
If you do privileged execution of specific operations via IPC you get the guarantee that whatever is done, is done from a well-defined, pristine execution environment, without leaking context implicitly. The IPC message is the *full* vulnerable surface, and that's as minimal as it can get. And that's great.
[Yep, that's a nested blockquote. Because why not.]
Oh wow, thank you for mentioning that, unnamed reader! I definitely read that post on LWN at some point in the past couple of days when the pkexec thing started going around. It must have really worked its way into my head, and then eventually stirred up the memory of everything getting "nicely infected" at $OLD_JOB.
It's really a nasty situation if you think about it. There is a ridiculous amount of state out there, and nobody really knows what all of it is. We try to fool ourselves with these namespace hacks on Linux that some people call "containers", but they aren't the full story and probably never will be.
Every time I talk about trying to partition a machine at this level, I can hear my dad the retired mainframe programmer laughing at us all. Those machines knew a thing or two about actually dividing themselves into useful smaller pieces. Right now, on these dumb little (x86[_64]) boxes, we're still just playing with toys by comparison.
Imagine how nasty it is to have to fork + exec to start a subprocess just so you can run something and find out how it turns out. It runs afoul of multi-threaded locking and all kinds of other things that you aren't "allowed" to run post-fork and pre-exec, lest you trip over one of the many, many potholes and bear traps we've left behind.
Just think if you could say "something, please run X for me, and let me poke it with long sticks over stdin/stdout, and let me know when it finishes, and how it finished". Kind of like inetd, but for purely local invocations of various things, right?
At that point, would you even need to run everything as the same uid? I bet you wouldn't. You could wall off your web browser from your mail client from your music player from your USB-powered mug warmer controlled from... you get the idea.
Anyway, it just goes to show that sometimes it takes a good LWN post to trigger a memory to get me to post about something myself.
...
Regarding the SWC BBS picture, another reader had this to say:
That picture of the BBS that ran on all those PCs just reminded that in the same period, the well BBS ran 64 phone lines off of one sequent with 1.6 GB of RAM. Sure, it cost 25,000 1990 dollars, and they used it as their primary heat source for their office in Sausalito, but it strikes me as a much better setup than a bunch of PCs running DOS.
That brings back some memories. Now, I never called it, but even I had somehow managed to hear about it through the grapevine of what passed for a social environment back then in those dialup days. For those who haven't crossed paths with this thing, it stood for (stands for?) "Whole Earth 'Lectronic Link".
So at a glance, $25K of 1990 dollars sounds crazy high, right? You could buy a really nice car for that much money back then. For reference purposes, apparently that translates to about $53-54K here 32 years later.
Anyway, it sounds like a lot, but would it have been, really? All of those PCs in there must have cost a pretty penny too. Unless they were buying the absolute cheapest crap at wholesale prices, I have to imagine that having that many computers around would have cost a similar amount.
Maybe what happened is that it crept up on them. Maybe they had four or five, then five or ten... then a dozen... then two... then thirty... fifty... and the next thing you know, you are wrangling a "cluster" of MS-DOS machines. THE HORROR!
I'd have to dig around old magazines to get a feel for how much this stuff cost back in the day. I mean, I know what my family's machine cost, and it was not cheap. In the fall of 1989, we got a 386DX-25 with 4 MB of memory (expandable to 8), an 80 MB SCSI (!) drive, both sorts of high-density floppy (5.25", 3.5"), a VGA card and matching monitor, and a keyboard. It came out to an astonishing four thousand dollars... and 32 cents.
I actually kept the details from that original order form, so here's how it broke out.
- 386-25: $1649
- 80 MB (disk + SCSI controller): $519
- 1.44 MB FDD: $89 (assuming the other one was in the base price)
- VGA (assuming board + monitor): $569
- Math coprocessor (20 MHz): $479
- 4 MB memory: $399
That's $3704, and if you multiply that by 1.08, you get $4000.32 on the nose. I guess the local tax rate was 8% back then. (My dad picked it out, and he apparently spared no expense. Wild.)
Now, for a node of a BBS, you could skip the fancypants display, and go with the absolute cheapest POS you could get which would still let the machine boot. (It has to have SOME video card to placate the BIOS.) Grab some garbage CGA or MDA card from a machine that's going to be scrapped. It's not going to be hooked to a monitor for very long so who cares?
You wouldn't need either sort of floppy drive, so drop those from the bill of materials. (Yeah, the "BOM". You pick up some interesting terms when working at some of these places. Don't say this one out loud in an airport or on a plane, okay?)
I'd hope that you wouldn't need a math coprocessor to run a BBS, even if it had some nutty door games, so drop that right there.
If it was netbooting, then maybe you could do without a hard drive too? Those Netware-friendly NICs could have PROMs put on them to boot straight off the network, so no hard drive and no SCSI controller.
At this point you basically have a motherboard, a processor, some amount of memory, a really crappy video card, and a serial port, right? Maybe you could go really cheap on those? Maybe you don't need a 386, and can use a 286 or even an 8088. How much would it cost, anyway?
Here's the catch: to come in under $25K (the cost of the WELL mentioned above) with the 64 machines shown in the picture, each one would have to cost no more than $390 each. (390 x 64 ~= 25000). Could you get below that?
Then you find out that they either had twice as many machines, or were doubling up on some with multitasking. That either means now you have to make 128+ lines in that $25K (so about $195 per node), or you have to somehow get enough CPU + RAM to make the multitasking "work" on them.
All of this is just me guessing out loud, because feedback posts are entirely off the cuff with no real planning. This is as close as you'll get to me doing improv for you here, I guess.
I wonder how much money actually went into some of this stuff.
Oh, before I forget: that whole thing where it creeps up on you, and you go from having 5 of them, to 10, to 20, to 50...? Yeah, that's pretty much what happens when you let your software people ship whatever they want to "THE CLOUD" and let it just autoscale forever.
If you don't have someone like my friend the badass capacity engineer keeping people honest at your company, your laziest software people will bleed you to death, and the cloud vendors will love them for it.
...
Next up, a bit of older feedback that I'm only now getting around to, sorry about that. In response to the New Year's Day post about 2022 overflowing signed 32 bit integers, a reader says:
comment: This could be another instance:
Although, on page six someone quotes Honda as saying “We have escalated the NAVI Clock Issue to our Engineering Team and they have informed us that you will experience issue from Jan 2022 thru August 2022 and then it will auto-correct.”, which doesn’t fit. Others however say that Honda told them that a fix would be available sometime before August. Game of telephone and all that.
Fun times!
This one just blows my mind. I read through the posts and tried to make sense of it and tried to figure out just what they could have possibly screwed up to make this happen, and how the "auto-correct" could possibly happen as well. I still don't quite understand.
I'm hoping that someone will take this apart and will explain what's actually going on here. I'm sure it'll be another tale of awful, awful things about timekeeping that leads a bunch of people to drink (as usual). That's pretty much what you expect with that kind of story.
I hadn't heard about this one prior to this feedback, so thank you to that reader for mentioning it!