Leap Second Bug Causes Crashes 230
An anonymous reader writes in with a Wired story about the problems caused by the leap second last night. "Reddit, Mozilla, and possibly many other web outfits experienced brief technical problems on Saturday evening, when software underpinning their online operations choked on the “leap second” that was added to the world’s atomic clocks. On Saturday, at midnight Greenwich Mean Time, as June turned into July, the Earth’s official time keepers held their clocks back by a single second in order to keep them in sync with the planet’s daily rotation, and according to reports from across the web, some of the net’s fundamental software platforms — including the Linux operating system and the Java application platform — were unable to cope with the extra second."
All of my servers were fine (Score:5, Insightful)
And I didn't do anything special, just kept their software up-to-date.
Re:All of my servers were fine (Score:5, Informative)
That can be hard for some people.
Re: (Score:3, Informative)
Agreed. Patches that aren't required to solve an ongoing incident impacting customer traffic require about 2 weeks advance notice to pass through change control, and that's if everything is perfect. A single error in a ticket can push that ticket out another week, and another, and so on.
Generally, we shoot for 3 weeks before we are allowed to install a patch. On average, it's about right.
Re:All of my servers were fine (Score:5, Informative)
the patch was posted back in March.
https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=6b43ae8a619d17c4935c3320d2ef9e92bdeed05d
You probably don't do much Java, then (Score:5, Informative)
As it turns out my biggest problems was customer-supplied software which uses their own java jre's. We install a jre by default and update it whenever possible, but some software (Adeptia, VLTrader, Alfresco) comes with their own ancient jre and scripts to call that over system-supplied java.
Not a single machine crashed (we are very explicitly in charge of what OS-version there's running) but a lot of java locked up and had to be restarted.
I can even see a small bump in the power-usage around two o' clock (0:00 GMT).
Re: (Score:2)
As it turns out my biggest problems was customer-supplied software which uses their own java jre's. We install a jre by default and update it whenever possible, but some software (Adeptia, VLTrader, Alfresco) comes with their own ancient jre and scripts to call that over system-supplied java.
Not a single machine crashed (we are very explicitly in charge of what OS-version there's running) but a lot of java locked up and had to be restarted.
So are you saying that, in addition to the Linux kernel glitch in question (which appears to cause some userland processes to spin [lkml.org]), there are purely-userland problems? Or, if you're running on a Linux that doesn't have John Stultz's fix, is it that some JREs are vulnerable to the Linux kernel glitch and others aren't?
Re:You probably don't do much Java, then (Score:5, Informative)
So are you saying that, in addition to the Linux kernel glitch in question (which appears to cause some userland processes to spin [lkml.org])
Actually, I'm not sure that's the case. John Stultz's mail from July 1, 2012 [lkml.org] speaks of a bug where clock_was_set() wasn't called after the leap second was added, and of a patch he was working on, so the bug in question might not have been fixed in March.
Re:You probably don't do much Java, then (Score:5, Funny)
I can even see a small bump in the power-usage around two o' clock (0:00 GMT).
Leap seconds contribute to global warming. We need to raise this at the next G8 summit.
Re: (Score:3)
Looked to me like it was only 64-bit Java, not 32-bit Java
Re: (Score:3, Informative)
What is this, 1990? All modern CPUs have protection against overheating and disabling that protection requires, at the very least, some crafty soldering or flashing a 3rd party BIOS. If you're capable enough to do that you're probably running some sci-fi prototype rig from the future using pressurized mercury phase transition cooling or something.
So no, I don't see how any properly set-up rig can make the CPU cook itself.
Re: (Score:2)
the patch was posted back in March.
https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=6b43ae8a619d17c4935c3320d2ef9e92bdeed05d
Or not [slashdot.org].
Re: (Score:3)
That can be hard for some people.
And also not necessary on Linux, with the exception of security updates. Even my machines with ancient images like Ubuntu 8 where completely unbothered. Probably you're ok with any Linux younger than 25 years. Most probably, Linux 2.0 would have been fine except for the security update question.
Re: (Score:3)
pesky software engineers, writing code for no reason.
Re: (Score:3)
No question there was a kernel bug, a race condition to be precise. Now fixed. The chance of hitting it is pretty small, but if you have enough servers or the right load, some of them will. Kernel hang was possible but livelock with high CPU was more likely. The workaround: set the time.
The chance of your home server hitting this was vanishingly small. You're more likely to get a power outage.
Re: (Score:3)
One of ours (running Java on Linux) started throwing out NTP alarms at 10 seconds after midnight, but it seems to have stayed up. However, the software on that particular system is especially vulnerable to leap second issues so we'd tested it pretty well beforehand.
Otherwise no-one has complained about any other systems going down so I presume they're OK.
Re:All of my servers were fine (Score:5, Insightful)
And I didn't do anything special, just kept their software up-to-date.
That's a nice ideal, but the reality is that many up-to-date "stable" distribution releases are still using kernels which are susceptible the leap second problem (and haven't had the patch back-ported to them). Ubuntu 8.04 LTS server is supposed to be supported until April 2013, and on my (updated!) system,
# uname -r
2.6.24-28-server
I like the idea of stable releases, but this is a glaring problem with the entire idea. Everyone extolls the wondrous virtues of package managers for Linux-based systems, but the dirty secret is that unless you stay bleeding-edge (which is usually the opposite of "server"), you'd better be happy with the 4-year old version of Apache, PHP, MySQL, and the Linux kernel you're running. Sure, it's possible to manually download and install packages from a newer release (assuming you can get past the dependency hell usually associated with it). Sure, it's possible to try and splice in (or "pin" packages using Debian parlance) from a newer repository. Sure, it's possible to install from source, compiling and installing everything by hand. But once you do any of these you've given up 90% of what makes the package manager useful and are just asking for dependency problems in the future.
And, all that aside, do you even know if the patch released to fix this problem is included in your distribution-released kernel? If you're not rolling your own kernel it can be nigh to impossible to know what's included and what's not -- in that case it doesn't even matter if it's up-to-date.
Re:All of my servers were fine (Score:4, Informative)
And, all that aside, do you even know if the patch released to fix this problem is included in your distribution-released kernel? If you're not rolling your own kernel it can be nigh to impossible to know what's included and what's not -- in that case it doesn't even matter if it's up-to-date.
Well you could read through the change log and release notes to find out.
Re: (Score:3)
That's a nice ideal, but the reality is that many up-to-date "stable" distribution releases are still using kernels which are susceptible the leap second problem (and haven't had the patch back-ported to them).
To which of the, apparently, two or more leap second problems are you referring? (The latest one, causing the bogus futex timeouts and subsequent CPU-eating spinfests, is, apparently, having a fix developed today, July 1, 2012, so getting that patched would be a little difficult - especially getting it patched before the leap second is introduced. :-))
Re: (Score:3)
Nice, huh? No wonder that 1/3 of the traffic on the internet is served from FreeBSD (data base on Netflix being a FreeBSD house and that they account for 1/3 of the internet's traffic).
But I thought that Netflix confirms BSD is dying?
Re: (Score:3)
NASA's Astronomy Picture of the Day, http://antwrp.gsfc.nasa.gov/apod/astropix.html [nasa.gov], has apparently been down all day; wonder if this is the cause.
Anyone heard from the Space Lab today?
Re: (Score:3)
Re:All of my servers were fine (Score:5, Informative)
Re: (Score:3)
NASA Goddard is near Baltimore. They lost power in the storm and are operating under "Code Red": http://www.nasa.gov/centers/goddard/ [nasa.gov]
Quite likely other misbehavior blamed on the leap second is actually the result of the storm (or like Pirate Bay, some unrelated crash).
Re:All of my servers were fine (Score:4, Interesting)
Our problem was with a third party monitoring solution - its daemon process brought every single one of our servers to a near halt by consuming all available cpu cycles at the stroke of gmt midnight.
The OS itself was fine.
This monitoring software is common enough that it likely was behind a lot of the issues seen around the 'net.
Re:All of my servers were fine (Score:5, Informative)
Our problem was with a third party monitoring solution - its daemon process brought every single one of our servers to a near halt by consuming all available cpu cycles at the stroke of gmt midnight.
The OS itself was fine.
Well, if you're talking a Linux kernel, the part of the OS that dealt with leap seconds was not OK [lkml.org], and was "not OK" in a fashion that could cause processes using futexes [kernel.org] to spin and consume all available CPU cycles when a leap second is introduced.
This monitoring software is common enough that it likely was behind a lot of the issues seen around the 'net.
...perhaps by virtue of either using futexes (in what I'm presuming is a legitimate fashion) or using something that uses futexes.
Re: (Score:2)
Interesting. I wonder what conditions had to have been met for a crash to happen, none of my servers had so much as a hick-up.
Re: (Score:5, Funny)
>hick-up.
The hick up watching the servers when the leap second came was you.
Re: (Score:3)
Configuration of the system to only accept 23:59:59 and not 23:59:60
Re: (Score:3)
Re: (Score:3)
If that actually happened, then they should have just made it do 23:59:59 twice instead of crashing all the computers. I would like somebody to give me a concrete reason why any computer system should actually crash because of a lost second.
If you send 23:59:59 twice, you have the same second in the system twice, which can potentially cause issues with logs. If everything is timestamped to the second/millisecond, how can you be sure an event happened in the first 23:59:59 second, or the second (or subsequent) 23:59:59 second?
Re: (Score:2, Informative)
The hard system lock bug due to a leap second was patched in 2.6.29, so either you've got some weird related bug, or something is very wrong.
Re: (Score:4, Interesting)
The hard system lock bug due to a leap second was patched in 2.6.29, so either you've got some weird related bug, or something is very wrong.
Well, the weird related bug would arguably count as something being wrong. Apparently there is a bug in the handling of the insertion of positive leap seconds that could cause weird behavior with [lkml.org]futexes [kernel.org], and that bug appears not to have been fixed until at least July 1, 2012 (I'm guessing John Stultz has worked up a patch [lkml.org]).
Re:I agree... (Score:4, Insightful)
When NTP knows that a leap second is to be added, it (on Linux at least) sets a flag in the kernel to say that at 23:59:59, please continue to 23:59:60 before going to 00:00:00. This is set by NTP anytime on the day that the leap second is due to be implemented, hence why a server running NTP on Linux would know that TODAY a leap second is due (cause they should always be posted at the 23:59:59 cross-over)
Linux (Score:4, Informative)
I'm a Linux admin at a fairly large hosting company. The only thing that I personally aware of happening this time around was that the time change triggered a bug in the OpenManage software on Dell servers causing it to use 100% CPU. The solution was to resync the time and restart OpenManage. It wasn't really a fault of Linux itself, but in OpenManage on Linux. Lots of datacenters use Dell hardware and I'm sure most use OpenManage, so I'm sure the problem was widespread.
Re:Linux (Score:5, Informative)
What you describe is a bug in the Linux kernel that causes problems for the Java VM that OpenManage uses.
It is not a bug in OpenManage at all.
Re: (Score:3)
It blew up Virtualbox for me as well. Guests were eating 100% CPU even though they were not aware of it, and after killing them the CPU load transferred to another Virtualbox service. Odd.
Reboot and it was working normally again.
Our Red Hat servers had no issues at all (Score:5, Insightful)
I'm uncertain why these reports keeps referring to some monolithic "Linux" that is supposed to have had issues - Red Hat's the biggest Linux vendor, and certainly their "Linux" handled it just fine.
What distros had issues?
Re:Our Red Hat servers had no issues at all (Score:5, Informative)
TFA mentioned that the RHE6 kernel had the bug, but not RHE5.
It appears also that system load was a big factor, so if your systems aren't busy on Saturday then they might not have crashed even if running an affected kernel.
Re: (Score:2)
TFA mentioned that the RHE6 kernel had the bug, but not RHE5. -- It appears also that system load was a big factor, so if your systems aren't busy on Saturday then they might not have crashed even if running an affected kernel.
Ah, ok - thanks, I managed to miss that. Most of our servers are still on RHEL 5 because of some odd issues we've experienced with LDAP under RHEL 6.
I've got a test/catch-all machine on RHEL 6, but that doesn't generally have to work very hard.
Re:Our Red Hat servers had no issues at all (Score:4, Informative)
Red Hat had a lot of issues.
https://access.redhat.com/knowledge/articles/15145
https://access.redhat.com/knowledge/solutions/154713
It depended entirely on your load. The buggy kernal code ran every 17 minutes for the 24hr period leading up to the leap-second insertion.
If you had enough load, your chance of dead-locking your system increased significantly.
Solution, strip the leap-second flag by manually setting your time.
Re: (Score:2)
I think my Debian stable box's latest rbot [ruby-rbot.org] build's launch_here.rb was acting weird from the leap bug because the CPU was going high even when idled. I rebooted after 55 days of uptime and it was fine.
Re:Gentoo (Score:2)
Clock: inserting leap second 23:59:60 UTC
No problem whatsoever on my Gentoo server, with a 3.3.1 hardened (Linux) kernel.
Re: (Score:3)
I've got a Slackware 12.0 box running 2.6.21.5 that crashed. Slackware 12.1 (2.6.24.5) and 12.2 (2.6.27.31) did not crash, but it sounds like these versions are vulnerable as well, I just got lucky.
Re:Our Red Hat servers had no issues at all (Score:5, Funny)
Windows?
Re:Our Red Hat servers had no issues at all (Score:5, Funny)
Sorry can't remember the name. It's the one that takes the credit for the work of others.
You must be talking about SCO, but if you're still running CND you should probably upgrade.
Re: (Score:3)
The bug is related to kernel version, IIRC (introduced somewhere in the 2.6 series, resolved in 3.2 or somesuch). So it depends what kernel the distros ran.
More like resolved yesterday [lkml.org] (today being July 2, 2012 where I'm typing this).
What about Windows and Mac? (Score:5, Interesting)
Re: (Score:2)
So far all I've heard about is affected Linux systems, did Windows and OS X just fine?
No problems on a couple of OS X machines that were on during the leap second - one running 10.7 Lion, the other 10.6 Snow Leopard (my laptop, which I was actively using).
Re: (Score:2)
well, none of my machines (all running Linux) were affected by the problem. I guess the bug only appeared in some systems.
Re: (Score:2)
Ditto. My Ubuntu server is still running fine (the only indication of the leap second is a message in dmesg output) and my Ubuntu laptop had no problems either.
Re: (Score:3)
And apparently neither did any desktop Linux systems.
Re: (Score:2, Funny)
"And apparently neither did any desktop Linux systems."
There are desktop Linux systems?
(ducks)
A.
Re: (Score:2)
Lots, on an absolute scale, but few relative to the number of Windows and OSX desktops. :(
Re: (Score:2)
My guess ist that Windows simply ignored it, so there never was a 61st second in a minute.
Beeing correct, on the other hand, might come as a surprise to more than one pieces of software.
Re:What about Windows and Mac? (Score:5, Informative)
My guess ist that Windows simply ignored it, so there never was a 61st second in a minute.
Well, if Microsoft's documentation of the SYSTEMTIME structure [microsoft.com] reflects the implementation, GetSystemTime() [microsoft.com], the claim in that man page^W^WMSDN page that "The system time is expressed in Coordinated Universal Time (UTC)" nonwithstanding, cannot acknowledge the existence of a 61st second in a minute ("The second. The valid values for this member are 0 through 59.", as the SYSTEMTIME page says).
But, just as on UN*X, you have "counter" and "human-style label" times (time_t, struct timeval, struct timespec are examples of the former, and a struct tm as returned by, for example, gmtime() is an example of the latter, on UN*X), with the Windows versions of those being SYSTEMTIME and FILETIME [microsoft.com] respectively. That page on FILETIME says nothing about leap seconds - does it just keep counting over a positive leap second or does it stop or what? And, if it doesn't just keep counting over a positive leap second, does it just freeze for a while second, or does it slow down over some period of time so that it eventually syncs up, or what?
As for NTP, Microsoft has a page on "How the Windows Time service treats a leap second" [microsoft.com], which says
(the author of which needs to be told what "inserted or deleted" implies - do they mean that, regardless of whether a leap second is inserted or deleted, the NTP client that is running Windows Time service is one second faster than the actual time?)
And then there's one more question: if there's anything in the NT kernel that deals with leap seconds, does any version have a glitch, as some versions of the Linux kernel do [kernel.org]?
If not, then many of the other problems might not exist on Windows. This email from John Stultz [lkml.org], the author of the fix linked to in the previous paragraph, seems to indicate that at least some of the problems, if not all of them, stem from a kernel bug, so it might be that Java and company might be Just Fine on systems that don't have a kernel glitch of that sort (so they might work fine on at least some non-Linux systems, as well as on Linux systems with the bug fixed).
Re: (Score:2)
As far as I can tell, all current operating systems handled it fine. It's applications that have problems, mainly server-type apps that actually use the clock for important things.
Linux being heavily affected is just a side-effect of most servers running Linux (although apparently some older versions don't handle leap seconds so cleanly - maybe that has something to do with it?).
Re:What about Windows and Mac? (Score:4, Interesting)
As far as I can tell, all current operating systems handled it fine. It's applications that have problems, mainly server-type apps that actually use the clock for important things.
Linux being heavily affected is just a side-effect of most servers running Linux (although apparently some older versions don't handle leap seconds so cleanly - maybe that has something to do with it?).
Yes, at least one of the problems appears to be a Linux kernel problem [lkml.org]. However, as that thread indicates, the consequence of this isn't a kernel crash; it causes futexes [kernel.org] to repeatedly time out (or, at least, causing futexes with timeouts to repeatedly time out). I'm guessing, perhaps incorrectly, that this might mean that code waiting for a futex gets a kernel wakeup due to a timeout, checks whether the condition being waited for has happened, discovers that it hasn't, sleeps in the futex again, gets a kernel wakeup due to a timeout, checks whether the condition being waited for has happened, discovers that it hasn't, sleeps in the futex again, lathers, rinses, repeats, so it makes no progress and chews up tons of CPU.
If so, then:
so Linux being heavily affected might also be a side-effect of, well, some versions of the Linux kernel having a bug that's triggered by leap seconds.
However, unless an application happens to use futexes in a fashion that trips over the bug, they won't be affected. It might be server applications that are most likely to do so, meaning that you might not see it on, say, a desktop or handheld Linux machine, or even on some servers.
Re: (Score:3)
So far all I've heard about is affected Linux systems, did Windows and OS X just fine?
The glitch mostly affected POSIX compliant operating systems as POSIX specifies a day as 86400.
So you're saying the glitch could affect OS X [opengroup.org] (or, at least, OS X Snow Leopard - although Leopard was also registered - but I'll bet Lion behaves, and Mountain Lion will behave, the same way)?
Re:What about Windows and Mac? (Score:4, Informative)
Also, look up the command: w32tm
FUD? (Score:2, Insightful)
Re: (Score:2)
It was a genuine bug but that doesn't make Linux or Java 'bad', all software has bugs. Good or Bad will depend on how many bugs, how often they bite and how badly.
The bug has already been fixed for months now, so the systems having trouble were the ones that weren't kept up to date.
Re:FUD? (Score:4, Informative)
The bug has already been fixed for months now
A bug might have been fixed for months now [kernel.org], but I don't think that's the bug here [lkml.org].
Re: (Score:3)
Now that Linux hit the same type of hurdle, we're all of a sudden being very nuanced about the definition of code quality? Typical.
Wow. You're still pissed over Azure failing, your Xbox disabling itself, your Zune crashing for a full day and your Outlook manhandling your appointments (on more than one occasion)?
Talk about carrying a grudge..
Re: (Score:2)
You'll need to talk with the people who made those comments. Of course, I did provide a few more criteria you might want to consider.
Re: (Score:2, Insightful)
I seem to recall Microsoft suffering from some leap year bugs. Boy, the Slashdot comments section lit up about how even a kindergarten programmer would catch the mistake and how it's indicative of the software quality coming out of Microsoft. Now that Linux hit the same type of hurdle, we're all of a sudden being very nuanced about the definition of code quality? Typical.
the difference being this bug was patched already it only affected systems the were not kept up to date. Microsoft however did not patch or fix their software until after the problem had already occurred. the fault lies not in Linux here it lays with system/server admins not updating their servers. where with Microsoft case it was their direct fault for not handeling a well known date issue. where with the Linux bug no one had heard of leap seconds until yesterday yet it did have a patch already out there n
Re: (Score:2)
the difference being this bug was patched already it only affected systems the were not kept up to date.
I would believe that except some of the recent Linux kernels did NOT properly handle the leap second. https://lkml.org/lkml/2012/7/1/19 [lkml.org] It was this improper handling of the time change associated with the leap second that sent some software into a tizzy, with the most common side effect being heavy CPU consumption. Some software seems to have have issues regardless of this bug as well.
I agree with the original statement, the if MS had done this the tone of this article would be different.
Re: (Score:3)
the difference being this bug was patched already it only affected systems the were not kept up to date.
A bug, perhaps [kernel.org]. This bug, perhaps not [lkml.org].
Re:FUD? (Score:5, Funny)
You don't need a leap second in order for that to happen. Firefox does that regularly.
Re: (Score:3)
Extremely weird (Score:5, Informative)
From my own machines and comparing notes with some other people (all in all, about 3k servers) the bug seems to affect machines randomly. Known facts:
There's a kernel patch that fixes the supposed issue: https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=6b43ae8a619d17c4935c3320d2ef9e92bdeed05d
Affects Debian stable a lot.
Affects Java and Virtualbox (starts using too much CPU).
Affected my browser (iceweasel on debian testing).
Affects SOME mysql installs (5.1 and 5.5, but not all, and of two identical installs one might be affected, the other not).
The fix has been posted at lot of places: /etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date; /etc/init.d/ntp start
(I'm all for switching unix time to a simple counter and leaving it to the calendar libs to put the leap seconds where necessary)
Re:Extremely weird (Score:5, Informative)
It's a race-condition, either crashing your ancient kernel or causing software using certain kernel-calls to effectively lock up. In both cases load seems to be a factor.
Over here the race-condition coincided with the actual leap-second and the start of the first batch of cronjobs at 02:00 local time.
(I'm all for switching unix time to a simple counter and leaving it to the calendar libs to put the leap seconds where necessary)
Bad idea. It would have prevented kernels affected by the race-condition from crashing, but would have meant most of your running software would have been either hit by this bug or would have been on the mercy of a 17 year old pimple-faced coder.
I think I prefer a crash over the mayhem caused by banking-software not handling a leap-second correctly. That could bankrupt whole countries.
Re: (Score:3)
From my own machines and comparing notes with some other people (all in all, about 3k servers) the bug seems to affect machines randomly. Known facts:
There's a kernel patch that fixes the supposed issue: https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=6b43ae8a619d17c4935c3320d2ef9e92bdeed05d [kernel.org]
I don't think that's the issue. The issue discussed in this lklm thread is a different issue with, presumably, a different John Stultz fix. [lkml.org]
The fix has been posted at lot of places: /etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date; /etc/init.d/ntp start
Presumably meaning "workaround" rather than "fix".
(I'm all for switching unix time to a simple counter and leaving it to the calendar libs to put the leap seconds where necessary)
Sounds good to me, but I thought that was a good idea back in the late '80's; the POSIX people thought otherwise, so....
At least as I read RFC 5905 [ietf.org], time stamps in NTP packets are essentially "simple counters", and count positive leap seconds and don't count seconds removed with negative leap seconds. I'm not sure what
Dreaded S60 bug... (Score:3)
It's like the Y2K bug, but every few years.
Re: (Score:2)
Hey, it's an excellent opportunity to drum up bogus consulting work!
"Are your old C programs able to handle a leap second! Think of how much money your company will lose when that one extra second of interest gets calculated on your bank accounts! You need me to check your code for you!"
"Thanks, see you around, for the next leap second!"
The IT industry definitely needs for leap seconds.
I always thought leap seconds were stupid (Score:3, Interesting)
Why not bundle them and apply them every 10 or 20 years?
And apparently I'm not alone:
http://en.wikipedia.org/wiki/Leap_second#Proposal_to_abolish_leap_seconds [wikipedia.org]
Hogwash, Astronomers can find coping mechanisms, it's either that or these ridiculous levels of stress for systems admins.
Re: (Score:3)
Hogwash, Astronomers can find coping mechanisms, it's either that or these ridiculous levels of stress for systems admins.
TAI [wikipedia.org] doesn't know about leapseconds, and it's the coping mechanism of choice for astronomy.
Re: (Score:2)
so there you go. there's no good reason for leap seconds. bundle them and apply them once a decade or more.
Re:I always thought leap seconds were stupid (Score:4, Interesting)
Re:I always thought leap seconds were stupid (Score:5, Insightful)
> Why not bundle them and apply them every 10 or 20 years?
The problem we have here is that leap seconds are rare. Things that are common are tested for, and quickly found if broken. Having something which only happens every 20 years is a recipee for disaster every 20 years.
My view is that NTP is at fault, because the 61th second is a brittle way to handle it. NTP should use the same method as google for smearing the leap second out over fx an hour: http://googleblog.blogspot.dk/2011/09/time-technology-and-leaping-seconds.html [blogspot.dk]
Only Linux affected? (Score:5, Interesting)
Re: (Score:3)
I'm managing a cluster of 2,400 nodes running FreeBSD, and AFAICS, none was tripped off by leap second NTP adjustments. On the other hand, 4 out of 180 Linux nodes crashed simultaneously at that very moment. All this is exceedingly weird, but may indeed point to a subtle bug in the Linux kernel (only?)
Could be [lkml.org], if "crashed" means "had some processes start spinning like mad". If it was a kernel-mode crash, that might be another bug.
Debian + Java = Issues (Score:3)
Google on how they fixed that.. (Score:4, Interesting)
Google official blog: "Time, technology and leaping seconds" (sept 2011)
http://googleblog.blogspot.in/2011/09/time-technology-and-leaping-seconds.html [blogspot.in]
I wonder if the leap second has anything to do with the labs Chubby paper / site currently being offline..
MySQL had issues only. (Score:2)
MySQL started spiking my CPU when the leap second hit. Only MySQL, and nothing else. It was odd.
Hmm, could this have been the cause of my issues? (Score:2)
I had a lot of programs (none Java-based though) taking up an inordinate amount of CPU, and high system CPU usage. Couldn't figure out the cause, and a reboot fixed it. In retrospect, I think it was around midnight UTC.
Please read this lkml thread before commenting (Score:4, Informative)
This linux-kernel mailing list thread [lkml.org] discusses a kernel bug that causes futexes [wikipedia.org] to repeatedly time out, so that code using them (which might include POSIX mutexes and condition variables, if that's what glibc uses for them on Linux) might spin.
That's not the kernel-leap-year-handling bug that was fixed back in March, so it's not as if a properly-patched kernel wouldn't get hit by this (unless you define "properly-patched" as "includes the patch John Stultz came up with on July 1, 2012").
So, yes, this particular bug is Linux-specific (i.e., there's a reason why it hit Linux servers), and might not be the fault of the userland code running atop it (so it might not, for example, be Java's fault).
What did the computers do with that extra second? (Score:3)
Data: She brought me closer to humanity than I ever thought possible, and for a time...I was tempted by her offer.
Jean-Luc Picard: How long a time?
Data: Zero point six eight seconds, sir. For an android, that is nearly an eternity.
We took the coward's way out... (Score:4, Informative)
I work at a fairly large international outfit, with data feeds coming and going to the far ends of the Earth. Everything we do is time-sensitive. Processing messages that depend on prior messages already being processed means we can't gracefully handle things coming in out of order.
We spent lots of time and money studying this problem, hired a high-priced consulting outfit to advise us and spun up lots of projects to mitigate the "risk" of the leap second. There were far too many meetings and conference calls with vendors, VARS and other people that wanted us to pay them for their time. What was determined was that we couldn't guarantee that nothing would crash or (gasp!) messages might be discarded or processed incorrectly, which was a risk we weren't willing to take. We run a full gamut of OSes, from HP/UX, Solaris, Linux, z/TPF, z/OS, DB2 etc etc.. You get the idea. Too many variables and too many systems to update and test with the limited funds and limited timeframe given.
In the end, we avoided the problem by shutting down all (and I do mean ALL) processing and flushing all the transactional systems to disk and suspending EVERYTHING from a minute before until a minute after the leap second. (Was that two minutes or two minutes PLUS one second? Clock math has always eluded me.) Shutting down all these interconnected systems in the correct order was a precision dance that, in the end, we didn't perform very well. Messages did end up being discarded. At precisely :20 seconds after the leap second, we began syncing all our systems with our internal NTP server and then at precisely one minute after, we slowly started everything back up. There were some systems that required a restart. We manually reprocessed those earlier discarded messages just as fast as our little fingers could type. In all it took us about 15 minutes to get everything spun back up, and all that time is getting charged to our SLA, which affects ALL our evaluations and year-end bonuses.
Lots of work was done, overtime was paid and buckets of money were given to lots of high-priced consultants and I personally will take a hit to my paycheck, all over ONE GODDAMNED SECOND.
Let's not do that again, okay?
Re:Linux kernel unable to cope? I think not. (Score:4, Informative)
There was a Linux kernel bug. See
http://news.ycombinator.com/item?id=4183122
http://marc.info/?l=linux-kernel&m=134110635328824&w=2
and
https://lkml.org/lkml/2012/6/30/122
Re: (Score:2)
yes, an old one that was patched before this became an issue. the issue if for un-updated/unpatched versions of Linux and shoddily written apps and java
Re:Linux kernel unable to cope? I think not. (Score:5, Interesting)
I run Arch Linux with kernel 3.4.4 and it went haywire. My machine was very heavily loaded at the time and when the leap second happened mysqld, firefox, and ksoftirq processes started consuming 100% CPU. The load factor was well over 10 and the machine was grinding along. It didn't actually fail but it was loaded down.
Even restarting the processes didn't fix it. The high load would go away once I stopped the processes but as soon as I started them again the load would come right back. I had Firefox open on a blank page not doing anything and it was slammed at 100% CPU and had a could ksoftirq tasks slammed at 100% CPU each too.
I had to reboot the machine to get it back to normal.
I have Ubuntu and Debian servers that for whatever reason did not add the leap second so they were fine. Their time was a second off today though (at least until ntp slowly corrected it or I manually intervened).
Re: (Score:2)
You would have been fine if you stopped ntpd before restarting the offending processes.
Re:Linux kernel unable to cope? I think not. (Score:5, Informative)
Restarting ntp wasn't enough for me, I had to reset the date with:
date -s "`date`"
Only one machine went haywire though.
Re: (Score:3)
Mods -- please mod this up to a thousand. kwardroid's fix fixed this for all the affected machines I've found so far.
Re: (Score:2)
Re: (Score:3, Interesting)
We will keep having these kinds of issues for as long as some people who fail to understand that time of day is an arbitrary number whose main utility lies in it being composed of predictable periods and divided into homogenous units. It should have no relation whatsoever to whatever time the sun happens to rise or set at any particular location and above all it should not be changed to accomodate fluctuations in the orbit of a rock circling an arbitrary star. Abominations like leap seconds or daylight savi
Re:Why now? (Score:5, Insightful)
and above all it should not be changed to accomodate fluctuations in the orbit of a rock circling an arbitrary star.
That is precisely the point of keeping track of the time of day, or day of the year.
time of day is an arbitrary number whose main utility lies in it being composed of predictable periods and divided into homogenous units.
You do not need a complex system like date time comprised of minutes hours, seconds, months, weeks, and years if you just want to measure time in a convient homogenous unit then define a time-zero, and just count milliseconds from that to whatever arbitrary distance into the past and future you want from that. Measure it kilo-seconds, mega-seconds, giga-seconds... etc.
The entire point of date/time is because we do in fact care a lot about how that "arbitrary counter" lines up with when we will be awake or asleep or eating at various points -- that's what makes it useful.
What we should have is what I've described above, time-zero and a counter. And translations from that to localized date time should be handled by a library.
Re:Why now? (Score:5, Funny)
What we should have is what I've described above, time-zero and a counter. And translations from that to localized date time should be handled by a library.
Which, sadly, POSIX doesn't let you have as "UNIX time" [opengroup.org]:
If there were a UN*X API to get a count of seconds since the Epoch (in addition to, or instead of, a call to get "seconds since the Epoch"), and a UN*X API to convert those to UTC and local time labels, that would get what you want. Modulo making it work with NTP, the former could be implemented with less difficulty than a call to get "seconds since the Epoch", and the latter is called "the Olson code complete with the leap seconds database".
However, that would then require some mechanism to allow code to schedule something to happen at a given UTC label; simply calculating the UNIX time for that UTC label, getting the current UNIX time, and scheduling it for then-now seconds in the future is insufficient, as the UNIX time for a given UTC label in the future might change if a leap second is scheduled between then and now. (Note that if you support scheduling something to happen at a given local civil time label would already require correction of that sort to handle DST rule changes.) This would also have to do something if you schedule an event for YYYY-DD-MM 23:59:59 and a negative leap second occurs so that there is no 23:59:59 on YYYY-DD-MM; "something" might be "let somebody know and ask them to correct it" or "do it at 00:00:00 on the next day", perhaps depending on the reason why it's scheduled.
Re: (Score:2)
Or maybe just pick a time standard that doesn't have leap seconds? There's at least 14 different time standards and I believe only UTC uses leap seconds. One or two even track the variations in the rotation of the earth itself (for astronomy stuff).
Re: (Score:3)
Considering leap-seconds happen every now and then, it seems odd that such fundamental things as Linux and Java can not handle it. AFAIK, it was just about for years ago since we last had a leap-second.
Perhaps the bug that was mentioned in the lkml thread that started with this message [lkml.org] was introduced less than four years ago, so the code in question had never gotten exposed to a leap second except perhaps in testing (I don't know how hard it is to reproduce it; John Stultz wasn't initially able to reproduce it in his testing [lkml.org], but eventually succeeded [lkml.org]).