Writing

Feed Software, technology, sysadmin war stories, and more.

Tuesday, May 1, 2018

You don't need to be the thing to monitor the thing

Let's say you're a systems person, and you're taking a look at the stuff which runs on your machines. You see a bunch of things which don't need any sort of privileges: they don't bind to low-numbered ports, they don't mlock() things into memory, they don't create namespaces, or become other users, or any of that other fancy stuff. But, for some reason, they're running as root.

You figure this is a problem, so maybe it's time to kick these programs over to an unprivileged user. Maybe you choose "nobody". Well, as the old jwz line goes, "you now have two problems". Why? Well, that's a little harder.

The whole nobody schtick seems to persist thanks to NFS continuing to exist. It apparently started with the need to map disjoint sets of users onto something on distant machines. For instance, let's say you had "user1" on machine1, but not on machine2. Imagine you were in a world without Kerberos, without LDAP, without NIS/YP, and so on. Only local users in /etc/passwd existed.

What should the owner/group of an inode be if that user doesn't exist? That's where you get nobody from.

Somewhere along the line, people got the idea they should use this account to mean approximately "not root". This is yet another one of those situations where they mean well, but the implications can be troubling.

Here's why: let's say you punt all of your nasty little daemons into this swamp where they're all running as "nobody" (uid 65534, most likely). Someone breaks into one of them, and now they can do stuff on the machine as that uid. This includes messing with the other daemons running as that uid. Now, besides the fact you have unprivileged command execution happening, you also have cross-program exposures. It provides more "toeholds" for an attacker to use to get even more privileges.

What can you do, then? How about making a single uid for each program? That'll limit the amount of fun that can be had by gaining access to that account, if nothing else.

This then brings up yet another point: just how much stuff do you want to run as these "service user" accounts? Maybe you have a suite of software with a bunch of different moving pieces. One of them is the glorified database server. Another is an outboard monitoring system which does black-box scanning with TCP connections over loopback.

Do you suppose the monitor needs to run as the same user as the database? I'm guessing the answer is actually no. I made this mistake myself recently. I got the "not root" part right, I got the "not nobody" part right, but I still tripped over the "same user as the actual thing with all of the data in it" part.

As it turns out, the monitoring program doesn't need any sort of access to the actual database. Its whole purpose in life is to sit there and port-scan the range to see who or what is listening to make sure all is well (yes, really). Having it run as the same user as the db would mean that it could kill those processes, twiddle data on the disk, or any number of other terrible things.

Things which can lead to making this sort of mistake: making it difficult to obtain a "just me and nobody else" uid number on a fleet of machines. It can have a subconscious impact on the system design where you just casually veer into using the "dbuser" account without even thinking about making a "dbmon" account.

Things which could make it easier: having your program supervisor of choice (systemd?) provide this on the fly somehow, particularly if it doesn't matter *which* uid you get from run to run. Linux can do 32 bit uid/gid values, apparently, so why not grab a far-off range and just run things disjoint all over the place up there? One program, one uid/gid pair. They don't even need to be in the passwd or group files, although making some kind of NSS hack to resolve them would also be possible for people who REALLY care about that sort of thing in tools like ps and top.

Just food for thought.

...

Programming note: unlike the 2011-2013 span, I'm allowing myself to take vacations from writing this time out. There will be gaps in coverage now and then because the computer won't be coming with me.