As was reported by Wired.com, the interworkings between the Linux kernel, Java, and the GNU C library (glibc) failed to anticipate or accommodate for a one-second gap injected into UTC time by stepping backward one second at midnight on July 1, to keep time synchronized with the Earth’s rotation. Despite this being known beforehand and discussed at length in some circles, the bug quickly produced unexpected and devastating results to Internet and other computing infrastructure, including taking out much of Amazon’s server farm and services provided by many other well-known web enterprises.
Sadly, I wasn’t in the know about this until it was too late, and some of our servers at work got hit with it.
As surprisingly devastating a problem as this was, its fix was quite simple: stop the network time protocol (NTP) service, manually inject an accurate time value into `date`, and restart NTP:
service ntpd stop date `date +"%m%d%H%M%C%y.%S"` service ntpd start
It’s too bad that we couldn’t have used that one-second gap to jump back in time and prevent the problem from happening.