The One-Second Glitch That Crashed Reddit

Atomic clocks, the bedrock of modern timekeeping, tick with unwavering precision. Earth’s rotation, however, is anything but constant, slowing and speeding up unpredictably. To reconcile this mismatch, leap seconds have historically been inserted into Coordinated Universal Time (UTC), keeping it within 0.9 seconds of the planet's celestial rhythm.

Since their introduction in 1972, 27 positive leap seconds have been added. On these rare occasions, the clock progresses from 23:59:59 to an unprecedented 23:59:60, before finally rolling over to midnight. This extra second poses a unique challenge for computer systems.

Unix time, a fundamental measure, fundamentally assumes every day contains precisely 86,400 seconds, offering no native representation for 23:59:60. Operating systems like Linux traditionally address this by repeating a second, effectively stepping the clock back to play 23:59:59 twice.

This "non-monotonic" time adjustment, where wall-clock time momentarily appears to move backward, violates a core assumption across countless software applications. Such disruptions have repeatedly triggered critical failures, including the infamous Reddit outage in 2012, highlighting the inherent fragility of a system built on a second computers were never designed to love.

A prime example of this cascade occurred on June 30, 2012. That day, a leap second insertion triggered a critical bug in the Linux kernel. Rather than gracefully adjusting, the kernel’s hrtimer code, interacting with futex timers, caused processes to wake repeatedly and instantly. This drove CPU utilization to 100%, effectively deadlocking systems.

Reddit, a major online platform, bore the brunt of this subtle timekeeping adjustment. Its services became unavailable for approximately 30–40 minutes, demonstrating how a single, repeated second could ripple through production infrastructure and bring a widely used application to its knees.

This incident, and others like it, highlighted a fundamental engineering lesson: wall-clock time is not always monotonic. It can jump forward, step backward, or repeat, defying the implicit assumption in much software that time always progresses linearly.

For robust systems, developers must design logic for elapsed time and event ordering that does not blindly depend on time.Now(). Instead, they should account for potential discontinuities. This involves using monotonic clocks for duration measurements and carefully managing event sequences where time can appear to go backwards.

A different leap second, on New Year’s Day 2017, crippled Cloudflare’s DNS. Their Go code calculated DNS lookup durations by subtracting a start timestamp from an end timestamp. When the clock stepped back, the calculation yielded a negative duration, which a function expecting a non-negative argument then received, triggering a fatal panic.

This incident highlighted a fundamental flaw: many software systems assume time always moves forward, an assumption violated by the standard leap second adjustment. As Cloudflare explained, the "perceived resolution time will be negative," leading to unexpected crashes. For more details, see How and why the leap second affected Cloudflare DNS.

To circumvent such abrupt time jumps, major operators like Google and Meta employ a technique called leap smearing. Instead of inserting a full second at once, they subtly slow their system clocks over several hours, gradually distributing the extra second. This ensures time never appears to go backward to their internal systems.

Smearing avoids the sharp discontinuity that causes software to break, but it introduces a new challenge. Different organizations might use different smearing schedules, causing their systems to temporarily diverge on clock readings. This requires careful coordination of time sources to maintain consistency across distributed infrastructure.

International metrology bodies have now decided to eliminate the leap second, with the change slated to take full effect by or before 2035. This monumental shift allows a larger divergence between Coordinated Universal Time (UTC) and Earth’s rotational time, ending a half-century of these disruptive adjustments. The decision reflects growing consensus that the risks outweigh the benefits.

A critical driver for this change is the Earth’s recent, slightly accelerated rotation. This raised the unprecedented specter of a negative leap second, where one second would need to be removed from UTC. Such an adjustment, never before implemented, would pose even greater challenges for software that has historically struggled with adding a second.

For developers, this means the immediate threat of leap seconds will eventually recede, but the fundamental principles of robust timekeeping remain. Always use monotonic clocks for measuring durations and timeouts, which guarantee time only moves forward. Keep civil timestamps, often tied to UTC, distinct from elapsed-time measurements.

Finally, rigorously test all assumptions about clock behavior. Even without leap seconds, systems can still encounter:

• Time synchronization failures
• Network latency
• Manual clock adjustments

Careful design and validation remain paramount for reliable distributed systems.

What is a leap second?

A leap second is an extra second occasionally added to UTC to keep it roughly aligned with Earth’s irregular rotation.

Why can a leap second break software?

Some systems repeat or adjust a clock second, disrupting software that assumes time always moves forward or that every day has exactly 86,400 seconds.

What caused Cloudflare’s 2017 DNS issue?

A backward clock adjustment produced a negative duration in code that expected a nonnegative value, triggering a panic in a random-number function.

Should developers still use monotonic clocks?

Yes. Use monotonic clocks for measuring elapsed time; wall clocks can still be adjusted and are better suited to timestamps and calendar time.