Hooking up a school to a T1 on the cheap
Working for a school district meant a lot of money-saving hacks would happen. One of them was those horrible non-BGP load balancing tricks I wrote about last fall. Another one was what we did to get the "alternative school" online, and then how their upgrades worked out.
This particular school was based out of a former carpet store, way out on a "pad site" in the middle of a parking lot. According to the telco, their demarc was on the side of a grocery store. Then the phone lines went under the parking lot, appeared at a burger stand, then ducked back under the ground to a photomat booth (remember them?) and after another conduit run, finally arrived at our school.
They would not guarantee anything more than analog/voice connectivity over those pairs. That meant absolutely no T1 service. Given that we didn't own the actual building, nobody wanted to actually invest any money in it by trenching in a new line from the curb. Somehow, I mentioned that we could try dropping a Linux box out there with a modem.
We decided to give it a shot. I built a box that would run pppd in a loop to keep dialing into my existing dialup pool, and installed dhcpd on it. This machine was going to do the whole hosting job for that school. They didn't even bother giving it a NT server like all of the other schools. Later, I wound up putting Samba on it, and it did all of this admirably. It just sat there and ran and ran.
About three years into this, someone finally decided to spring for an actual T1 line. We trenched out to the curb line, and the telco promised to put in a circuit by a given date. Now there was a new problem: they'd need something to actually drive that thing. We had a spare Paradyne DSU left over from an aborted project, but a new router would cost in the neighborhood of $2000.
Again, I piped up and mentioned that you could get something called a Sangoma WANPIPE for far less. It was basically a board that would speak PPP over V.35 to do the whole T1 thing. We could just drop it into their existing Linux box, hook it to the DSU, plug that into the new jack, and go on with life. It was viewed as something of an experiment, but again I got the go-ahead.
The card arrived well before the telco was anywhere near ready for us. I decided to get the kinks out of the system by testing it locally. I think the "network engineers" looked at me like I had a second head growing out of my neck when I proposed it. They couldn't figure out how I was going to bring up the link when the telco was clearly not ready.
It was stupidly simple. First, I dropped a new 3161 line card into our big DSU/CSU rack -- we had a spare one of those too -- and gave it a basic configuration. How did I figure that out? I read the manuals. Then I found the other end of the octopus cable over by where the rest of the ports plugged into a breakout panel of RJ45s from the DS3 mux we had. This is where the telco would eventually deliver the other end of that circuit.
I just took that slot's plug and hooked it to a regular RJ45 F-F coupler, and then crimped a straight-through cable of sufficient length to reach our patch panels. There, I plugged into a spare port which would "appear" by my desk.
Later, back at my desk, I crimped a second cable to go from the wall to the 3160 box, but this time I changed the pins around to effectively make a T1 crossover. It was easy enough given that I could find the pinout data with a quick web search. Then I just plugged that into the standalone DSU/CSU, then plugged its V.35 cable into the card which was temporarily inhabiting a local Linux box.
I gave the 3160 a configuration using the tried-and-true RTFM method, and pretty soon the physical part of the link came up. It was pretty neat, since now I could actually play with both ends of a circuit without a 15 minute drive in-between. Remote administration was not something the "network engineers" really got. What can I say. It gave me a better understanding of what all of those channel options really did. I doubt they had ever done anything of the sort.
So the next thing to do was to bring up the logical link. We already had a port on our huge Bay BCN router ready to go, so I dropped into the BCC ("blatant Cisco clone") CLI configuration tool on there and told it to create a line with PPP encoding. Then I did the same thing in the Sangoma stuff on the Linux box.
Nothing happened. Everything looked fine, but it just would not go. Somehow, I found out about the tcpdump-ish tool which came with the card that would dump raw frames from its network interface. I started that up and watched as the BCN started spraying some really crazy packets at me. They sure weren't PPP. After staring at it for a while, I eventually determined that despite the setting in BCC, it was actually using "Bay Standard" protocol on that line. Well gee, no wonder!
I made a quick trip to the machine the "network engineers" used to run Site Manager, the accursed GUI-ish thing which was used to control the Bay routers. There, I brought up the interface in it, and sure enough, it was not set to PPP. I deleted it and re-created it in Site Manager, and the link came up!
Again, looks of confusion and amazement. I simply did not understand why this seemed so special. Something isn't working, so you find out what it is doing, then analyze that to figure out what it thinks is going on. Then you smack it around until it thinks the right sort of thing is going on (that being PPP) and starts emitting proper traffic.
I guess the whole thing blew their minds: a T1 over a bunch of local copper, the crossover cable, running it on a Linux box, and being able to sniff the raw circuit. It seems that any sufficiently advanced technology really is indistinguishable from magic!
The telco eventually put our circuit in at both ends, and I took the proven-to-work equipment out to the site. There, I just had to put the configuration stuff on the Linux box, and not surprisingly, it Just Worked. I left the modem nailed up just in case. It turned out to be useful in a few occasions when the T1 went down for whatever reason.
This lasted another couple of years until it was bumped out by a "real" router of some flavor. It was the first production Linux box at that gig, and it just sat there and did its job quietly for five years. It was a simple screwdriver shop Pentium 150 with 16 MB of memory, a 2 GB IDE drive, and it managed to handle DHCP and file/print services for that school. It also backed itself up every week to a local tape drive.
If only all systems could be that reliable.
I never did figure out why I got a Bay standard link out of the BCC interface. I'm pretty sure I didn't do anything particularly stupid when configuring it, and that it did in fact say "PPP" in nice big letters. Considering how much of a hack job their CLI situation was, I guess I should be lucky I was able to even do that much. Earlier versions of their software didn't even let you do that.