Software, technology, sysadmin war stories, and more. Feed
Thursday, March 22, 2012

A half-baked ISP with entirely virtual users

Towards the end of the '90s, a couple of friends wanted to try building an ISP. They had gotten together with someone who had bought a fairly large BBS and all of its users and wanted to expand into TCP/IP access. I was tapped to figure out the user and account management, plus all of the usual services for them: RADIUS for logins, SMTP mail delivery, POP3 mail fetching, and basic FTP access for their web site space.

I came up with a clear goal early on: these users were not to have actual Unix accounts. There would be a couple of Linux boxes handling the load initially, but none of these users would be able to reach a shell (command prompt). I purposely designed it so they could not possibly gain a "foothold" on any of my server machines. Only the people who ran the place would be able to login in the usual ways (telnet and ssh).

This obviously meant some major changes were in store for those systems. I decided to handle it by coming up with my own scheme for verifying user details and then put some kind of interface between that and things which needed to authenticate users. Since all of the programs were C, my interface became a couple of functions which compiled into a new object, and then that would speak over the network to my server.

I knew that my backend would eventually need to have actual proper storage, like using a real database, but right now, time was of the essence. Today, you'd call this the "MVP" - minimum viable product. I wound up writing it to just use a tree of flat text files with some really simple format for e-mail address, name, password, and the other usual details. It would just sit there and wait for requests, then it would try to answer them as either true or false.

The requests it handled were fairly simple, too. One of them was "here's an identifier and a password, is it right?", and it would give an answer. This one was used by the RADIUS server. I had taken it and changed the way it authenticated users. Instead of having it do the usual crypting and shadow file checks, I just told it to call out to my auth function. My client code did the rest and returned the same sort of go/no-go answer to the regular RADIUS code. This same request was also used by the POP server which had a similar kind of hack applied.

Another request merely asked if a given e-mail address was valid. This was used during the RCPT stage of a SMTP transaction by sendmail, and was required so we could make it accept mail without having Unix accounts for these people, and without having to rig up all sorts of crazy hash hacks in sendmail.cf.

There was also a request where it would return the path to their FTP directory. The FTP daemon would use this. Once it had that data, it would chroot into that path and drop to an account which had permissions to read and write in there. This is back in the bad old days when FTP daemons still did dumb things like literally running /bin/ls, but the chroot broke that. I had to write an internal ls implementation to keep things running.

This was a whole mountain of ugly hacks. It created an authentication system nobody had ever seen before and effectively locally forked the code base of several fundamental services. But... it happened really quickly, and it meant that we could do anything we wanted just by adding more servers, since they didn't care where they were. Everything would just call back to the master and would keep on going.

I want to say that this thing went on to be huge, that all of the hacks were eventually rewritten to be proper code, and that I made a lot of money, but none of that ever happened.

Despite doing all of that, a bunch of fundamental things were dropped by other people. There was supposed to be a new circuit installed to deliver bandwidth, and it took forever to happen. When it did happen it was wonky and unreliable. Asking for them to fix it yielded nothing.

Worse, there was no terminal server. Oh, there had been plenty of promises and all of this, but the action was lacking. "Next week" would come and go, and then by the time anyone finally answered queries, the week which had been called "next week" was now "LAST week". Without a terminal server, we had no dialups. With no dialups, there could be no users, and without users, you can't make any money!

I got tired of running in place. I had delivered everything which had been on my to-do list, and now I was blocking on other people. After doing that for long enough, I gave up and left the project. I never bothered to find out what happened after that. I had other projects which needed me and where I didn't need to block on missing people.

I don't understand why this happens. It's almost like I'm running at one speed and the other person is at some completely different speed. After a while, the mismatch becomes too great, and I run out of stuff to do, and they haven't delivered much of anything. What's really crazy is when this happened at a place which was supposedly full of capable people. I've since learned most of that is just marketing.

One of these days, I will find someone who really wants to move at warp speed as well, and we will change the world.