Writing

Feed Software, technology, sysadmin war stories, and more.

Tuesday, May 29, 2012

Managed authentication for small businesses

Here's a problem I've heard from more than a few small and medium-sized businesses which have expanded to the point of having several Linux boxes. It generally crops up when someone new is hired and they need to have an account created on all of those machines. They show up for their first day of work and have nothing waiting for them. It can take months for them to finally get accounts and/or passwords for all of the machines.

A lot of time passes, and they think everything is done, but then something breaks one day, and they desperately need access to one particular machine. Unfortunately, that was the one machine which was overlooked during the manual creation of their various accounts, and the people who do have access are unavailable. Horrible hacks and workarounds must be arranged in order to figure it out, including getting one of the people with access on the phone and then logging in as them. Yes, this sort of thing really happens.

Obviously, some kind of centralized authentication system needs to be installed on these systems. Depending on who gets involved, all sorts of ideas might be put forward. Someone might suggest NIS+, but a quick look around at the state of documentation for running that on Linux should scare them away rather quickly.

So then there's Kerberos, and it sure seems a lot better. You need to make sure your DNS is sane (matching PTR and A records) and so are your clocks across all of your hosts. Then you get to install a bunch of new software everywhere - clients and servers - and configure it. If you're a small shop which hasn't set up some kind of centralized configuration management stuff yet, this happens manually on every system (buy a lot of bananas!), and has to happen every time another system goes online. Wonderful!

This also assumes that all of your machines are identical, and are running the same version of the same OS. If you're mixing and matching, and have Ubuntu and Red Hat and maybe even Slackware, life is three times as interesting, and that's just on your servers! You didn't forget about your clients, right? All of those machines, be they Linux, Macs, Windows, or that one Solaris box for that one guy who works in the basement, need to get "plugged in" to your authentication scheme.

All of this is certainly possible, but it starts building this massive structure which might not fly at a small operation. Something this big becomes a nontrivial consumer of time and resources.

While this has been going on, everyone has been logging into machines with ssh. Some of them use passwords, and some have key pairs. Someone gets a bright idea: rather than creating a new account for each new employee on all of our production machines, let's just have a bunch of role accounts! Then we can just add the public keys of new employees to the authorized_keys files for each role on every machine and save a lot of time!

Now they're heading in the right direction. With just a small change from where they were, they have now managed to eliminate half of the new employee workload by removing the need to create new accounts everywhere. After all, does everyone really need their own personal account on the production machines? I doubt it. Those machines are for your service, not for humans to use as personal shells.

Of course, now you have the problem of maintaining those authorized_keys files. Someone will probably write a breathtaking shell script which manages to iterate over all of the machines and runs a "ssh user@host 'echo public-key >> ~/.ssh/authorized_keys'" on each one, at least initially. That gets the whole "deployment" thing out of the way, and it gets new people onto those systems really quickly.

That strategy can work for a while. Eventually, it will need some help, though. People quit or otherwise no longer need access to some machines. A company might grow to where having everyone have the same access on every machine is unnecessary and perhaps is a little dangerous.

Around the same time, it becomes more and more important to keep track of all of your machines. Just having someone remember that there are 5 machines named after Disney princesses and another 6 which are named for Smurfs isn't good enough any more. Now you have several racks chock full of identical machines with no personalities and you need to keep track of them efficiently.

What to do, what to do.

This is where it gets interesting. Personally, I would adapt my sysadmin notebook to initialize a list of machines, then create a list of roles, and a list of users (actual people), and start building a matrix. These people are in this group, and this group owns these machines, and they want these roles on those machines, so take the SSH keys of those people and load them into the files on those role accounts. Phew!

I started thinking about this recently when two different friends related the same sort of problem in the space of a week. They need some infrastructure to manage this stuff and get it off their shoulders. What's more, neither one is really a programmer, so they need a ready solution and not a set of specs and a "good luck" from me.

At the moment, neither of their companies is ready to commit to such a project, so my friends are stuck maintaining their users/machines matrix the old fashioned way, by hand. I think this means both of their companies have a bus number of 1 for login-related systems as a result.

I know there have to be other companies out there who are suffering through the same problems. I would love to build this system for them to let them get back to their actual business and stop worrying about dumb systems administration problems which are common to everyone.

Are you in this situation? Do you want an escape? If so, I want to hear from you. You don't need to suffer forever.