NTP Pool Reaches 1000 Servers, Needs More

hgerstung writes "This weekend the NTP Pool Project reached the milestone of 1000 servers in the pool. That means that in less than two years the number of servers has doubled. This is happy news, but the 'time backbone' of the Internet, provided for free by volunteers operating NTP servers, requires still more servers in order to cope with the demand. Millions of users are synchronizing their PC's system clock from the pool and a number of popular Linux distributions are using the NTP pool servers as a time source in their default ntp configuration. If you have a static IP address and your PC is always connected to the Internet, please consider joining the pool. Bandwidth is not an issue and you will barely notice the extra load on your machine."

Google

This sounds like a job for Google.

Seriously. They are working to own every other bit of information. Why not "own" the method by which machines maintain time by throwing a thousand machines at it (an insignificant number compared to the 500k or more that make up their own server farm).

Re:Google

Well, unfortunately DynDNS will not work here, simply because the clients will resolve the IP address once at startup and then stick to it (it will not be re-resolved later). The NTP Project is working on that, but currently there is no chance to use DynDNS or even "pretty static" IPs ... Heiko

huh?

"Bandwidth is not an issue and you will barely notice the extra load on your machine."

If that is the case, why do they need more servers?

Re:huh?

There really should be an "Oops" button after you mod something; I've never done this myself but I've seen at least 2 or 3 of this type of message in the last few days.

Re:huh?

I've never done this myself but...

Well Doctor, I've got this friend who has a problem...

Re:huh?

No, the network time protocol accounts for latency and eliminates its influence almost completely as long as the latency is roughly symmetric, which it usually is for small packets.

Re:huh?

Yes. There are lots of time-sensitive tasks that require at least second-accuracy, some that require accuracy that's greater than that.

The first thing that comes to mind is remote logging. If I have several machines logging to some remote machine somewhere (as you should on any non-trivial system, to make a log falsification more difficult), it makes log analysis a lot easier if I know that the timestamps in the log are accurate and consistent across machines. Particularly if you ever have to dig through a break-in (or what you think might be a break-in), or just user stupidity, where you want to match actions taken on one machine to results on another.

At the very least, you want to make sure that all the clocks on the machines are accurate to at least the smallest interval of time that you might have two timestamps on the log apart by. Or if that's not possible, at least within a span so that the same human-initiated command will be discernible across the system at the same time in the logs.

Other things that involve remote data-collection have the same issue. At the very least, you need to have all your computers set so that they're accurate to some factor that's less than the time between data collections. While "data collection" sounds esoteric, it could be something as simple as sending emails from one computer to another, or combining two stacks of digital photos taken from some webcams (if they're portables, that's a separate ball of wax).

Now, do most of these things require all of the computers in your home network to be individually pinging a Level 2 timeserver? No. It would work just as well to have your gateway router get the time from a timeserver, and then offer NTP broadcasts to your network, so that everything could just synchronize itself. You'd have high precision local time, for synchronization, and reasonable accuracy time to a national standard. But that's beyond most users, so most OSes just have each workstation take care of things on its own.

Re:huh?

"Bandwidth is not an issue and you will barely notice the extra load on your machine."

If that is the case, why do they need more servers?

If I understand it right bandwidth isn't an issue because they can tailor how much of the pool load goes to your machine. When someone queries the pool their ntp client does a DNS query to pool.ntp.org. The pool's DNS server semi randomly returns the IP address of one of the volunteer servers in the pool. If you tell the pool operators that you have only a little bandwidth then the pool DNS server will only return your IP address say one tenth as often as it does the IPs for the high traffic servers. This allows you to decide how much load you're willing to bear. Even if the pool is overloaded, your machine doesn't have to be.

load

Bandwidth is not an issue and you will barely notice the extra load on your machine.

I think if their servers can't keep up, you *will* notice the load, at least until enough join.

Re:load

Their servers can keep up just fine, or at least the one I run can. My stats show 1GB per month traffic and the ntpd process taking about 1 minute/day of processor time. That has been relatively constant over the year or so the server has been in the pool.

I think this is just a case of more==better. A bigger pool means more people can use their local zone instead of the global zone, the whole system can handle more clients, less load on servers means even more may be willing to join, ...

Seriously, it's not that big a deal. Just thow your server into the pool and forget about it.

Re:load

Just thow your server into the pool and forget about it.

Isn't that a bit extreme? Should I maybe waterproof it first?

Re:load

Nah. I hear liquid cooling is great for overclocking.

Free GPS time equipment!

I must mention that right now by signing up for the pool now you also have a chance to get some really cool time keeping equipment [ntp.org]. :-)

Re:Free GPS time equipment!

Only if you already have a server in the NTP pool. And even then, they only get freebies to give away every few months. Then they decide from the applicants who gets them. So your chances are probably pretty slim. The long-timers will get them first.

So, what are they gonna say when overwhelmed?

"Pool's closed"?

I for one..

I for one.. *want* to be welcomed as a time-wielding overlord!

didnt they think of this?

Bandwidth is not an issue and you will barely notice the extra load on your machine. I think if their servers can't keep up, you *will* notice the load, at least until enough join.

do they have no way of routing/limiting traffic so that it isn't normally noticeable?

here's a question

Is having cable modem static enough? How often does IP change from, say, Comcast?

More NTP servers, Lower Quality?

I can understand the desire/need for NTP servers. The question for me becomes, does this reduce the quality of chips used in PCs? The chips that keep track of time don't have to be as accurate since, "hey, it can just sync up with NTP server." Once you let something simple like time slide, maybe they let other issues slide too because "Who is going to notice?"

Not so much the chips, but the timebase crystals..

The component that actually determines the stability and accuracy of the real-time clock in your PC is the timebase crystal, not the RTC chip itself.

Like every other component in mass-market electronic gear, it is chosen with minimum cost as the primary consideration. Such "value engineering" also has done away with the tiny trimmer capacitor that used to be present on most motherboards, which could be used (along with a frequency counter) to tweak the oscillator frequency for better accuracy.

For real accuracy, the timebase oscillator needs to be kept at a constant temperature, which isn't possible in a PC that gets turned on and off. Ideally, the crystal (or the entire oscillator circuit) is enclosed in a package equipped with a heater element and temperature sensor, and kept at a constant temperature. Such a circuit is called an OCXO, or Oven Compensated Crystal Oscillator, and is standard equipment on laboratory grade equipment like frequency counters and signal generators.

Better way To Do This

I think that a better method could be used to encourage diversity. They should take a page from the root DNS servers, or Akamai. Either use BGP anycast, which is what most of the root dns servers do now, which will probably never happen. Or, have a zone that network carriers should use on their local DNS servers, and by way of DNS lookups, encourage their customers to use. ntp.org has a default set of values for say time.overload.ntp.org that reflects the current pool. But I, as an ISP make my DNS servers directly answer queries for overload.ntp.org, and make entires such as:

time IN A 1.2.3.4
time IN A 1.2.3.5

where 1.2.3.4 and 1.2.3.5 are ntp servers on my local network. I don't allow people off my network to query my DNS servers for recursive queries, and the ntp.org DNS servers never tell anyone to use my name servers for this space anyways. This would mean that only my customers that use my DNS servers (about 99%) of them, would ever get answers for my time servers, and they would definitely be close.

And anyone whose network carrier doesn't bother to set this up, still gets generic answers from ntp.org. This works much better than just a big pool full of 1000 servers worldwide, even if you bother to use the country code dns regions, you still aren't always getting an ntp server anywhere near you.

Re:Better way To Do This

http://tf.nist.gov/service/time-servers.html [nist.gov]

All organizations interested in possibly hosting a NIST Internet Time Service server are invited to contact Time and Frequency Division Chief Thomas O'Brian for more information, including a description of the equipment that the organization must have available and a discussion of the other technical qualifications necessary to host a server: [e-mail] .

Re:Better way To Do This

You are absolutely correct that if network carriers provided NTP services properly on their nets, then the pool wouldn't be necessary. If you go through Usenet archives you can read the history and discussion behind the creation of the pool. Everyone realizes that the pool is an inferior solution that we are stuck with because the network access service providers won't do their job.

The next time I've got a free two hours for self-torture, I'll call Verizon Business customer support and ask them about NTP service. (It will take that long to be transfered to someone who understands the question.)

Re:Better way To Do This

A few thoughts...

Unlike a partnership with Akamai, there's no compelling monetary reason for an ISP to offer their own NTP server. Therefore, the easiest (least costly) solution -- at the ISP end -- is probably the most likely to win. Adding a line to dhcpd.conf is probably easier than configuring BIND to issue lies.

And while not everyone uses DHCP, they certainly have some mechanism for communicating things like DNS server addresses, default gateways, and so on. Using that same mechanism (be it DHCP, bootp, or snail mail) to inform the customer of the local NTP server seems trivial in every instance I can think of.

Clients that don't care will obviously ignore this data, but customers who do care can modify their client software accordingly.

Eventually (as in, within the MTBF of a Linksys router), if it ever gains any foothold, clients will use this data by default.

But I guess the most glaring problem to me is that, surprisingly often, the ISP's own DNS servers are slow and/or broken, and overridden. Much of Roadrunner's network is, for instance, assigned DNS servers which are so slow that when browsing the web, more time is spent on simple DNS lookups than on downloading and rendering content.

This, in turn, causes people like me to use a different DNS server on a different network. In my case, I use Level3's DNS at 4.2.2.1 because it is easy to remember and quite fast. Your suggestion ties together DNS and NTP inextricably, such that I'd also be using L3's NTP server by default, when all I really wanted was different DNS.

I don't want a solution to one network problem to have cascading effects on other network services. There's enough of that in the world already.

Remember, the whole point of this is to eliminate end-user manual NTP client configuration, and reduce network load, while offering the useful service of providing accurate time. And I can only hope that, after all of this, network-attached devices of all types will use this mechanism (whatever it is) to automatically derive time from a nearby NTP server.

Some of these devices will be reconfigurable to use whatever NTP server the user wants (certainly, my Linux box is), but hopefully some simpler devices will not be (think print server, networked DVR, WiFi LCD picture frame, or other minimally-configured box).

If a standard method for propogating NTP server names to end-users ever does get implemented, I shouldn't have to run a local copy of BIND and my own regimine of poison, just to allow independant settings for both DNS and NTP servers.

But that's all just my opinion. It is probably wrong. :)

GPS time with OpenBSD

If you grab a USB GPS receiver, I used a $60 BU-353 [google.com], you can have accurate time easily.

openbsd# dmesg | tail -3
uplcom0 at uhub0 port 2
uplcom0: Prolific Technology Inc. USB-Serial Controller, rev
1.10/3.00, addr 2
ucom0 at uplcom0
openbsd# nmeaattach cuaU0
openbsd# sysctl -a | grep hw.sensors
hw.sensors.nmea0.timedelta0=-328.10115 9 secs (GPS), OK, Tue May 15 19:48:46.898
openbsd# echo "sensor nmea0" > /etc/ntpd.conf
openbsd# echo "listen on *" >> /etc/ntpd.conf
openbsd# ntpd -ds
ntp engine ready
sensor nmea0 added
sensor nmea0: offset 328.097637
set local clock to Tue May 15 19:57:46 PDT 2007 (offset 328.097637s)
sensor nmea0: offset 0.020612
...

Re:GPS time with OpenBSD

Actually ... The USB latency can be pretty bad, so it's likely you'd get better time from a well-picked internet time server. You'd definitely get MUCH better time with a proper PPS (Pulse Per Second) time keeping GPS receiver or variations of that [meinberg.de].

Re:GPS time with OpenBSD

In addition to the latency of USB, the nmea output of a GPS unit may not be very accurate. Go for a GPS with pulse per second if you can find one for a reasonable price. A while back I was checking the chipset specs for the cheap GPS receivers to find one with a pulse per second output. I found some but I forgot which ones they were. Of course you would have to open the case and do a little soldering. I'm not sure how you would hook it up to your server once you got the pulse per second out. I think maybe to one of the pins on the serial port that would trigger an interrupt.

Under OpenBSD I've gotten much more stable timekeeping by recompiling the generic kernel with only one simple change. I set the processor type to 586 or 686 as the case may be. Specifically in the /usr/src/sys/arch/i386/conf/GENERIC file I removed "option I486_CPU" and "option I686_CPU" so that it would be correctly configured for my pentium 166 cpu. I think the pentium has some time keeping functions the 386 and 486 didn't have. Although I haven't found the parts of the kernel code where this change does its magic.

How about semi-dynamic IPs

If bandwidth requirements are low, I wouldn't mind joining the pool. But my ip is semi-dynamic: dynamically assigned, but rarely changes. I use DynDNS to get it.

A more practical solution...

Is to have DSL/cable modems provide the NTP service since they're facing the internet anyway.

atomic clock to PC connection?

Like a lot of guys here, we have an atomic self setting clock that works from radio broadcast. They are cheap now and work very well. What I am wondering is, do they make some sort of attachment clock, so it can set your computer's time that way? Like an atomic clock/usb cable connect thingee? Seems like if they did, we wouldn't need all these NTP servers, the government does the radio broadcasting and it is as accurate as it gets.

Storm

Have they asked the Storm network operators? I think they could donate a few thousand machines from around the globe pretty easily.

zero config and NTP?

This where a zero config version of NTP servers and client would be useful, to allow for the discovery of an NTP server on the local network, unless it already supports multicast discovery.

I am sure that there are many private networks where computers are still connecting to external time servers, when the could easily use a server on the local network.

how to get ntpd to stop listening on all interface

Anyone know how to get ntpd to stop listening on all interfaces? I have a host with several IPs and ntpd listens on all of them ... a little bit annoying.

# netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
udp 0 0 localhost:ntp *:*
udp 0 0 x1.example.com:ntp *:*
udp 0 0 x2.example.com:ntp *:*
udp 0 0 x3.example.com:ntp *:*
udp 0 0 x4.example.com:ntp *:*
udp 0 0 x5.example.com:ntp *:*
udp 0 0 x7.example.com:ntp *:*
udp 0 0 host.example.com:ntp *:*
udp 0 0 example.com:ntp *:*
udp 0 0 *:ntp *:*

Looks like my HTML-fu needs work too. Dunno how to stop having multiple whitespaces condensed to 1 space ... :(

My thought is

Some how this should all be merged into the bittorrent client.

not yet...

Shouldn't the "milestone" be 1024 servers?

Why not make it peer-to-peer

Why not design a time protocol that's peer-to-peer? Most people, like me and probably 99% of the people on the internet, can make due with time being within a few seconds (or frankly, within a couple of minutes) of accuracy, so if you only have a few nodes at the top that actually get the time from NTP and anyone who actually needs really accurate time using NTP, everyone else can share the time.

Seems to me that would get rid of the need for thousands of servers and would suit the needs of most users.

Re:Why not make it peer-to-peer

Because the number one rule of infrastructure is, "never trust the client." Peer to peer networks are full of malware/trojans/assholes, and generally far too easy to infiltrate with unwanteds.

And while I agree with your sentiment that I can live time being off by a little, I also run a lot of UNIX servers that use NFS heavily. I am far more concerned with all of my network machines agreeing on what time it is on my network, than being correct with the world. I sync two dedicated time servers to the ntp.org pools (soon to be three), and all my internal hosts sync to those two. Being synced with the world is very handy, and generally I would prefer it. But being in agreement with myself is non-negotiable, I just need it.

Windows XP's default time servers.

Is this why the default time.nist.gov and time.windows.com servers don't work sometimes?

Windows Time

For those interested, you can change your Windows time servers to NTP servers in the registry here: [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Cur rentVersion\DateTime\Servers]

Pool

I've been in the pool for years. It's a great service.

2 + 2 = 5

Bandwidth is not an issue and you will barely notice the extra load on your machine.

This doesn't add up. If it doesn't burden existing machines, then why do we need more of them?

security

I would join but only if there is an NTP server implemented in Java. I do have a few windows servers but I am not a professional admin and don't know much about ntp servers, so I don't trust any native program. If it would run on a Java virtual machine than it is OK.

Why I quit: ntpd sucks

Flamebait subject, but I kind of mean it.

I was in the pool for a while but quit because ntpd is wholly incapable of protecting itself. I ended up with about 50 abusers that polled for time once a second. I tried using the built-in filtering [isc.org] but it doesn't work, so ntpd was gleefully replying to each and every one of those requests.

Keep in mind that it has the logic to detect abusers - it just won't do anything about it. Well, it can be made to send a KOD (Kiss Of Death) packed that should make clients blacklist the server, but those same broken clients ignore KODs. I kid you not, the standard recommendation is to firewall them off [isc.org].

What? ntpd already knows its internal state, including a list of abusers. The code could be as simple as "def sendTimePacket(clientaddr): if clientaddr in blacklist then return; else sendpacket(clientaddr);", but they suppose that it's easier to write an external program to monitor that state, general firewall rules, connect to the firewall host, and insert them. No, really, that's not easier at all.

I like NTP and I liked the idea of helping serve it to the world, but until ISC decides to support at least basic anti-DOS functionality in ntpd, I won't be joining the pool again. And by "support", I mean at least lend moral support to people who would be willing to the work, instead of just telling them to alter their firewall.

1000 server are overkill

640 servers ought to be enough for anybody.

Inconsistency?

Does anyone else see an inconsistency in these two sentences?

This is happy news, but the 'time backbone' of the Internet, provided for free by volunteers operating NTP servers, requires still more servers in order to cope with the demand. Millions of users are synchronizing their PC's system clock from the pool and a number of popular Linux distributions are using the NTP pool servers as a time source in their default ntp configuration.

If you have a static IP address and your PC is always connected to the Internet, please consider joining the pool. Bandwidth is not an issue and you will barely notice the extra load on your machine."

They haven't got enough because of the amount of traffic but if you join the pool you will hardly notice the extra load?

Huh? Most home users should use their ISP.

This is what I do:

1) traceroute www.google.com

2) Pick the first router that belongs to my ISP.

3) Use it as my time server.

Multicast

If only the ISPs wouldn't block it...

Every machine in the world connecting via TCP to ask "what time is it?" sure doesn't sound very efficient does it?

Re:Unless netgear hears about you

Netgear products, built in permanent network offline crushes, for thousands of addresses. I even googled "Netgear will go and hard code your ip address into one of it's dumbass products". Nothing really came up.

Re:NTP Isn't Accurate

Hi AC,

The NTP Pool monitors the servers and only uses those with accurate time. A server drifting several seconds off would be taken out of the pool until it got fixed.

Also, the NTP daemons are Quite Good at ignoring the servers with Bad Time Keeping.

Using ntpd with the pool servers will give you much much much more accurate time than trying to set it manually after looking at a web page.

    - ask

Re:NTP Isn't Accurate

Please name one ntp server in the pool that it off by more than .5 seconds? The vast majority are accurate to under .1 seconds. I do not believe that the AC who said these aren't accurate understands how NTP works.

Re:NTP Isn't Accurate

You do realize all those times are in milliseconds [hp.com], right? So, the largest difference between your computer and one of the servers is 27 milliseconds, and the leading "-" in front of the hostname means it isn't even being used for synchronization. Also, either you didn't let it settle for a while, or your local computer clock is inaccurate, because you are still polling once a minute. A "healthy" computer clock will lower the poll frequency significantly if the local and estimated net clocks don't jitter much. I did have one machine with a clock that just sucked, so I had to make sure it was a client of another machine which could act as the timekeeper on my home network, and make sure it polled the timekeeper often.

Personally, I don't use the pool, and instead find some stable servers near to my ISP. But you really can't argue against the NTP pool as a default setup, since it works everywhere. So, if it bothers you, find some closer servers or convince your ISP to run a time server (many are already doing so). In both cities I've lived in, I was able to find an open stratum-1 server with a ~20ms delay (Thank you GPS).

Re:NTP Isn't Accurate

3 Minutes?!?

  I have my machines synced via ntp. ntpq reports than I'm no more than 3ms out of sync with a stratum 1 time server (9ms out of sync with UNSO) and that server is synced with GPS and USNO which as you said is never more than .0001ms out of sync with UTC.

    Eye-balling like you described I can verify that I am within 2000ms of http://time.gov/ [time.gov]. I think perhaps that that website may have had issue on the date you saw it being 3 minutes different than what NTP provided.

I'd show you the ntpq output but the lameness filters prevent it.

Re:NTP Isn't Accurate

I'm running Gentoo but I setup NTP and use pool.ntp.org and was curious about your post so I went to the site. Both my desktop and laptop are right on time.gov(to the second didn't test millisecond or anything) So maybe ubuntu uses a different time server but pool.ntp.org work 100%.

Re:non-root ntp server

Probably not. ntpd has to be able to set the system clock, and has to be able to listen on port 123.