Growing web applications from scratch
This post is actually a result of a request for notes on how to rig up a stateful web application by a friend of mine. Well, there are many ways to do that, and this is just one of them. I specifically refrain from describing this as the best, that all others need to just go away, and all other comparisons of that type.
So you want to create a web application. By that, I mean something which is somewhat dynamic, and has users poking at it and changing things. It probably remembers stuff for a non-trivial amount of time, like seconds or minutes. In other words, you're not just throwing static HTML at them, and you're actually accepting data back from them. You're probably serving that data back to them and a few other people, too.
You could try to make this happen by writing a bunch of CGI programs which wake up and talk to a database. One of them would probably be the reader, supplying JSON or maybe even fully populated web pages with that content included. Others would then need to be created to write back to this database. It's not much of a stretch, and it's not too scary. People have been doing this sort of thing for years.
The problem is that it's ridiculously heavyweight for something simple like a chat application, or a silly buzzword game for people watching their company's TGIF show. Also, any special logic you have for processing that data has to happen on the way to the database, or on the way back out. Now, remember that since you're a CGI program, you're going to start and stop for every hit. That's a lot of churn. Ouch.
Well, at this point, you could turn to something that keeps your program running. Maybe you could go suffer the kind of self-abuse which comes from FastCGI. Or you could try something like mod_perl or mod_python or go all the way to Django. Or you could go for something even more ShinyThing compliant and dive into Rails, which means now you've committed to Ruby.
What if you really like your usual way of writing daemons and other persistent standalone server processes and don't want to deal with someone's framework, let alone the language flavor of the month? Isn't there some way to keep all of this web cruft to a minimum and still get things done?
Then there's a server process written in your favorite language, just hanging out somewhere listening to a port, waiting for incoming connections. Your approach should be simple: accept a connection, parse a request, do something with it, and emit a response. You'll stay running the whole time, so if you want short-term state for something, just keep it in memory. My list of buzzwords works this way.
Finally, you have to connect those clients to your server. Your clients generate HTTP requests: GETs and POSTs. I imagine your server probably won't stoop to such craziness and will instead prefer something a little more sane. Try Facebook's Thrift or Google's Protocol Buffers. They're both open source. You can invent your own RPC mechanism or go find something which just needs a Stub. By the way, your users will never see this, so this can be just about anything.
Your server receives that request, chews on it, and generates a response if appropriate. Your client is probably going to expect JSON, so you could just emit JSON at your server and let your little CGI pipe just pass it back across as-is. Once you're done, close that socket and kick off the client, and go service some other client. That's it.
After getting the hang of this, new web services lose much of that feeling of staring up at a mountain, wondering how to get up there. The "how" part is gone, and now you just need to figure out a "what".