Software, technology, sysadmin war stories, and more. Feed
Saturday, April 7, 2012

"It works but I don't know why"

I got so many stories from working with the "network engineers" for so many years. There were so many bits and pieces of the infrastructure they were supposedly running that they really did not understand. Sometimes, you'd see a glimmer of hope, but it would rarely spread out into a full-blown beam of understanding.

At some point, one of those people decided he wanted ISDN into his house for more bandwidth and a permanently nailed-up dedicated connection. This wasn't a big deal since we had a couple of devices left over from a previous project and could recycle them trivially. So, he wound up with an Ascend Pipeline 50 at work and a Pipeline 75 at home.

Their local network was a /24 carved out of our bigger network allocation. All of their workstations and other things lived on it. We decided to carve a /28 or /29 subnet out of that for his house. This was no big deal. I jumped in and reconfigured things for his network instead of the old network and they just worked.

The thing he couldn't figure out was *how* it was working. He had at least one machine which was configured as if it had been plugged in at the office. It had a default gateway equal to the main router at work, which was standard operating procedure for that network. However, that machine was not at work, and it wasn't on the same network as that router... but it still worked.

Around this point, I got to explain the magic of proxy ARP. I had to switch it on the "work" ISDN router to make sure that other systems on that same network could get to the ones at his house. After all, we weren't about to go around adding more-specific routes for his subnet to every machine. Proxy ARP on that end made it look like they were sitting on the wire at the office.

I also had it running on his end. The Pipes knew enough about their respective networks that it could proxy for all of the hosts at the office. Now, all of his home machines should have been given a network configuration just for his subnet and a gateway of the P75 at his house. Somehow, he had neglected that detail, and they were configured for the larger (/24) network, with the big office router as the gateway. Because of the ARP proxying, those packets would find their way to the P75, would hop the ISDN to the P50, and would then be forwarded to the actual router. The hosts were none the wiser.

What is puzzling to me is how these things can remain a mystery. Network sniffing software is cheap and plentiful... at least, it is if you had a Linux box or similar. I don't think there was anything like Wireshark (or then, Ethereal) on Windows at the time, and NetXray or whatever was probably too expensive for him to have a copy at home. Also, Macs weren't yet on OS X at that point, so that wasn't an option, either.

Anyway, if he had fired up a sniffer, he would have seen his home machine put out an ARP request for the router at the office, and then *something* answered it. Further inspection of the hardware addresses would show that the masked stranger was in fact the Ethernet interface of his P75. The rest should just fall into place from there.

tcpdump -e can resolve a great many mysteries.