What has two cable modems and a GPS receiver?
I used to tell a pseudo-joke that had the arrow of time reversed and conveniently avoided telling the truth. The context in which it was read (my office whiteboard at a certain company) told readers exactly what they needed to know. Here's how it went:
A: You don't want to know.
Q: What has two cable modems and a consumer GPS receiver?
Obviously, here, I need to fill in that missing context.
One day, I was trying to make sure a machine had a decent NTP reference so that we could do basic log entry association across a wide area. I think I was trying to capture kernel oopses from something really dumb involving quotas and certain kinds of RAID hardware.
This customer's machine was locked to the data center's NTP servers, so I decided to "ntptrace" it to see how it looked. Naturally, the first hop was the local machine. That's not surprising. Then the second hop was one of the local NTP servers (same machines as the DNS servers from the other story). That's when it got weird.
Hop #3 was some 24.x.x.x IP address with a PTR which confirmed it: it was some consumer-grade cable modem connection. It was running at stratum 2, and what came next was just ridiculous.
Hop #4 was another 24.x.x.x address on a consumer cable modem allocation. From all appearances, someone had taken one of those little receivers you use for hiking and was using it as a NTP reference.
So, you had hundreds or thousands of machines all relying on this wonky chain for time service. Instead of having proper relationships set up, whoever built this system apparently grabbed a pool server or two and just let it ride.
I mean, even my rinky-dink school district NTP configuration was better than that. I had two machines peered off each other, each with three distinct reference sources which were not shared by those machines. All of those sources had agreements which allowed us to use them for time service.
Given how stupidly simple it was to find and then establish this configuration, I can only wonder why they let it run like that. They probably would have been better off NOT providing the service so that individual customers would have to make their own arrangements. At least then some thought would go into them.