The July 1 Leap Second Bug

As was reported by Wired.com, the interworkings between the Linux kernel, Java, and the GNU C library (glibc) failed to anticipate or accommodate for a one-second gap injected into UTC time by stepping backward one second at midnight on July 1, to keep time synchronized with the Earth’s rotation. Despite this being known beforehand and discussed at length in some circles, the bug quickly produced unexpected and devastating results to Internet and other computing infrastructure, including taking out much of Amazon’s server farm and services provided by many other well-known web enterprises.

Image

Wired.com: ‘Leap Second’ Bug Wreaks Havoc Across Web

Sadly, I wasn’t in the know about this until it was too late, and some of our servers at work got hit with it.

As surprisingly devastating a problem as this was, its fix was quite simple: stop the network time protocol (NTP) service, manually inject an accurate time value into `date`, and restart NTP:

service ntpd stop
date `date +"%m%d%H%M%C%y.%S"`
service ntpd start

It’s too bad that we couldn’t have used that one-second gap to jump back in time and prevent the problem from happening.

Leave a comment