Uncle Andrew wants to tell you a festive story. The NTPmare shortly after Christmas.
A modest proposalNearly two years ago, on the afternoon of Monday 16th January, I received an interesting BitFolk support ticket from a non-customer. The sender identified themselves as a senior software engineer at NetThings UK Ltd.
Subject: Specific request for NTP on IP 85.119.80.232 Hi, This might sound odd but I need to setup an NTP server instance on IP address 85.119.80.232. wats 85.119.80.232 precious?85.119.80.232 is actually one of the IP addresses of one of BitFolk’s customer-facing NTP servers. It was also, until a few weeks before this email, part of the NTP Pool project .
“ Was ” being the important issue here. In late December of 2016 I had withdrawn BitFolk’s NTP servers from the public pool and firewalled them off to non-customers.
I’d done that because they were receiving an unusually large amount of traffic due to the Snapchat NTP bug . It wasn’t really causing any huge problems, but the number of traffic flows were pushing useful information out of Jump ‘s fixed-size netflow database and I didn’t want to deal with it over the holiday period, so this public service was withdrawn.
The Senior Software Engineer continued:
The NTP service was recently shutdown and I am interested to know if there is any possibility of starting it up again on the IP address mentioned. Either through the current holder of the IP address or through the migration of the current machine to another address to enable us to lease 85.119.80.232 .
Um…
I realise that this is a peculiar request but I can assure you it is genuine. That’s not gonna workObviously what with 85.119.80.232 currently being in use by all customers as a resolver and NTP server I wasn’t very interested in getting them all to change their configuration and then leasing it to NetThings UK Ltd.
What I did was remove the firewalling so that 85.119.80.232 still worked as an NTP server for NetThings UK Ltd until we worked out what could be done.
I then asked some pertinent questions so we could work out the scope of the service we’d need to provide. Questions such as:
How many clients do you have using this? Do you know their IP addresses? When do they need to use the NTP server and for how long? Can you make them use the pool properly (a vendor zone)? Down the rabbit holeThe answers to some of the above questions were quite disappointing.
It would be of some use for our manufacturing setup (where the RTCs are initially set) but unfortunately we also have a reasonably large field population (~500 units with weekly NTP calls) that use roaming GPRS SIMs. I don’t know if we can rely on the source IP of the APN for configuring the firewall in this case (I will check though). We are also unable to update the firmware remotely on these devices as they only have a 5MB per month data allowance. We are able to wirelessly update them locally but the timeline for this is months rather than weeks.Basically it seemed that NetThings UK Ltd made remote controlled thermostats and lighting controllers for large retail spaces etc. And their devices had one of BitFolk’s IP addresses burnt into them at the factory. And they could not be identified or remotely updated.
Oh, and whatever these devices were, without an external time source their clocks would start to noticeably drift within 2 weeks.
By the way, they solved their “burnt into it at the factory” problem by bringing up BitFolk’s IP address locally at their factory to set initial date/time.
I’ll admit, at this point I was slightly tempted to work out how to identify these devices and reply to them with completely the wrong times to see if I could get some retail parks to turn their lights on and off at strange times.
We are triggering ntp calls on a weekly cron with no client side load balancing. This would result in a flood of calls at the same time every Sunday evening at around 19:45.Yeah, they made every single one of their unidentifiable devices contact a hard coded IP address within a two minute window every Sunday night.
The Senior Software Engineer was initially very worried that they were the cause of the excess flows I had mentioned earlier, but I reassured them that it was definitely the Snapchat bug. In fact I never was able to detect their devices above background noise; it turns out that ~500 devices doing a single SNTP query is pretty light load. They’d been doing it for over 2 years before I received this email.
I did of course point out that they were lucky we caught this early because they could have ended up as the next Netgear vs . University of Wisconsin .
I am feeling really, really bad about this. I’m very, very sorry if we were the cause of your problems.Bless. I must point out that throughout all of this, their Senior Software Engineer was a pleasure to work with.
While NTP service is something BitFolk provides as a courtesy to customers, it’s not something that I wanted to sell as a service on its own. And after all, who would buy it, when the public pool exists? The correct thing for a corporate entity to do is support the pool with a vendor zone.
But NetThings UK Ltd were in a bind and not allowing them to use BitFolk’s NTP server was going to cause them great commercial harm. Potentially I could have asked for a lot of money at this point, but (no doubt to my detriment) that just felt wrong.
I proposed that initially they pay me for two hours of consultancy to cover work already done in dealing with their request and making the firewall changes.
I further proposed that I charged them one hour of consultancy per month for a period of 12 months, to cover continued operation of the NTP server. Of course, I do not spend an hour a month fiddling with NTP, but this unusual departure from my normal business had to come at some cost.
I was keen to point out that this wasn’t something I wanted to continue forever:
Finally, this is not a punitive charge. It seems likely that you are in a difficult position at the moment and there is the temptation to charge you as much as we can get away with (a lot more than 840 [+VAT per year], anyway), but this seems unfair to me. However, providing NTP service to third parties is not a business we want to be in so we would expect this to only last around 12 months. If you end up having to renew this service after 12 months then that would be an indication that we haven’t charged you enough and we will increase the price. Does this seem reasonable?NetThings UK Ltd happily agreed to this proposal on a quarterly basis.
Thanks again for the info and help. You have saved me a