The legend of the headless login
I tend to write about a lot of random technical hacks and goofy problems I have encountered over the years. Sometimes, this is just so other people will realize that they aren't suffering alone. It's just my way of saying "it isn't just you, these systems are really ugly", and "I have also been there". Such tales can be useful when your boss doesn't believe that something can really break a certain way. My stories are yet another data point to say "actually, yes it can" and "sometimes you really have to do these horrible hacks".
In this case, I'm referring to a method of dealing with a server which needs to be reconfigured in the absence of a monitor or serial console. I've had this happen to me a couple of times when moving things around, and for whatever reason, a monitor was not available. Maybe it's an old box with only VGA out, all of your displays require something a bit newer, and you don't have any adapters. You get the idea.
The situation is simple enough: you and the box are at a new location and the network is up, but the box is configured with a static IP address for the old network so you can't get into it. I've handled this two ways in the past. Both of them tend to confuse onlookers who don't realize what's going on.
The first way is just to jump out to root on another machine on that same network and add an alias of the old network. You effectively pretend that your laptop (or whatever) is a member of the IP network which your orphaned server expects to see, then you just jump across. If you remember all the details, then you win.
I got to do this over the winter holidays with an old machine my parents had kept in a closet for the better part of a decade. Armed with just my laptop, I needed to remember the old network scheme and then passwords for that box to get in and do stuff. Happily, I did, and it worked out. There was nothing of consequence on the disk, so we just wiped it and sent it along to a new home.
But what happens if you can't go that way? Well, at that point, you dig around for a keyboard and try to do a blind login.
This is when you login at the console without being able to see what's going on. It usually includes a lot of guessing about the state of what's going on with a virtual console you can't actually check.
Typically, you start by pushing ALT-F1 for the first console. It should be on there already as a result of booting, but you do it anyway out of paranoia. Then you type in your username and hit ENTER. You hope for a small flicker from the disk activity LED to suggest that something just happened. Then you key in your password and hope for the best.
If you're lucky, you'll get a little more flickering from the disk and maybe even a chunk-kachunk or two as it logs you in. If you got this correctly, you're now sitting at a shell prompt. Now what?
Well, now you have to make sure you actually got to that shell prompt. I usually check this by carefully typing: e c h o (space) ^V ^G [ENTER].
If I get a beep from the PC squeaker, then it's highly likely that I logged in successfully and am sitting in my shell. Now it's time to su to get root powers and hopefully fix things. I do that and key in what I think is the root password.
The question is: how do I know if that worked? Well, now it's time to come up with some command which is easy to type which will only work if I made it over to uid 0. Just off the top of my head, I come up with this:
ls /root && echo ^G
This is quite a bit more typing, and now I must tread carefully. Pushing keys deliberately for too long might lead to repeated characters. Besides, if I'm running as root here, who knows what kind of badness might follow?
Anyway, if *that* works, then I get another beep.
It's a good bet that I'm in a root shell now, so now it's time to just type in some ifconfig commands and hope it works out. These can be verified by simply trying to connect. If it works, then it was probably right. If not, then something else is up. Knowing whether the machine had iptables rules and/or TCP wrappers can be helpful.
So finally, once I can ssh in, the rest is academic.
I've only had to do this a couple of times. I file this under "experience I wish I didn't have".