Engineering Post-Mortem: When Distributed Locks Fail and How a Redis Expiry Oversight Caused Duplicate May Invoicing

Executive Overview

In modern distributed architectures, the race for execution efficiency often introduces subtle, high-impact failure modes. On June 2, software engineer Sergey Shinder and his infrastructure team learned this lesson the hard way. A routine end-of-month financial closing sequence—responsible for compiling and dispatching billing statements to hundreds of business clients—experienced a catastrophic systemic breakdown.

Due to a classic distributed systems anti-pattern, 212 business customers received their May invoices twice, while nine unfortunate clients received three identical billing statements. While no financial transactions were maliciously intercepted or altered, the incident triggered a wave of customer confusion, manual administrative remediation, and immediate structural refactoring of the platform’s concurrency control mechanisms.

At the heart of the failure was a reliance on time-bound Redis distributed locks without safety fencing tokens, coupled with an unanticipated surge in external API latency and payload density for the platform’s enterprise-tier accounts. The incident highlights a foundational truth in Site Reliability Engineering (SRE): time-based locks are not absolute guarantees of exclusivity; they are merely probabilistic leases. When execution time overruns lease duration, the distributed system inevitably enters a split-brain state, where multiple workers operate under the assumption that they hold exclusive authority over a shared resource.

This post-mortem details the chronological unfolding of the June 2 incident, analyzes the architectural vulnerabilities that permitted it to happen, examines the systemic metrics involved, reviews official statements and retrospective learnings, and outlines the robust, multi-layered defensive strategies implemented to ensure a recurrence is impossible.

Detailed Chronology: How the Invoicing Failure Unfolded

To understand how duplicate invoices found their way into customer inboxes, one must first examine the architecture of the billing pipeline. The platform’s invoicing engine operates on a distributed pool of eight concurrent worker processes. These workers pull customer account IDs from a central task queue, compile usage data, query external microservices for tax calculations, format the final document, persist it to the primary database, and dispatch an email notification.

The Standard Execution Model

Under normal operating conditions, the lifecycle of an invoice generation task proceeded as follows:

1. Queue Polling: A worker picks up a customer account from the processing queue.
2. Lock Acquisition: To prevent race conditions and ensure that no two workers process the same customer simultaneously, the worker issues a Redis command: SET key value NX EX 60. This creates a key unique to the customer, asserting a non-exclusive creation (NX) with a hard time-to-live (TTL) expiration of 60 seconds.
3. Task Processing: The worker compiles the invoice. Historically, this phase took an average of roughly four seconds.
4. Completion and Cleanup: Upon successfully writing the invoice to the database and sending the notification, the worker deletes the Redis lock key, freeing the customer record.

For four years, this simple architecture functioned without incident. Sixty seconds felt like an eternity for a task that consistently completed in four seconds. However, June brought anomalous operational pressures that exposed the fragility of this assumption.

The June 2 Anomaly

As the workers initiated the May billing run on the morning of June 2, two compounding variables collided:

• Payload Explosion: The platform’s largest enterprise customers exhibited unprecedented growth in usage metrics. Their billing records contained hundreds, and in some cases thousands, of individual usage line items.
• External API Degradation: The external tax calculation service—a mandatory third-party dependency invoked for every single usage line item—suffered severe latency spikes.

Consequently, the time required to generate invoices for top-tier accounts skyrocketed. Instead of the typical 4-second turnaround, processing these massive invoices began to take anywhere from two to five minutes.

The Cascading Failure Sequence

At 60 seconds into the processing of a massive enterprise invoice, the Redis lock silently expired.

Redis operates as a dumb data store in this context; it has no programmatic awareness of business logic, application threads, or whether the client that instantiated a key is still actively working. When the 60-second TTL elapsed, Redis efficiently purged the key from memory.

1. The Phantom Expiry (T+60s): Worker A was deep into compiling usage lines, communicating with the lagging tax service, completely unaware that its lease on the customer record had expired.
2. The Unauthorized Intrusion (T+61s): Worker B, polling the queue, encountered the now-unlocked customer record. Finding no active Redis lock, Worker B successfully executed its own SET NX command, acquired the lock, and initiated the exact same invoice generation sequence from scratch.
3. The First Duplicate (T+180s): Worker A finally completed its lengthy processing cycle. It successfully wrote the invoice, triggered the email dispatch, and deleted the Redis key. Crucially, the key it deleted at this stage no longer belonged to it; it belonged to Worker B.
4. The Triplication Cascade (T+240s): Because Worker A deleted the Redis key (which was actually holding Worker B’s active lease), the lock was prematurely destroyed. Worker C, also polling the queue, immediately grabbed the newly cleared lock and began processing the same invoice a third time.

By the time the dust settled, multiple workers had processed, persisted, and emailed overlapping iterations of the same billing statement to dozens of enterprise clients.

Supporting Context & Metrics: The Anatomy of a Distributed Lock Failure

The June 2 incident provides a textbook case study in the perils of distributed systems design, specifically regarding mutual exclusion and the fallacies of distributed computing.

The Fallacy of Time in Distributed Systems

In a single-threaded or localized multi-threaded environment, locks are bound to memory addresses and thread execution contexts. If a thread holds a lock, it retains it until explicitly released. In distributed systems, network partitions, garbage collection (GC) pauses, CPU starvation, and external API latency make deterministic timing impossible.

As Sergey Shinder noted in the post-mortem analysis:

"A lock with an expiry is really a lease. It ends at a time, whether the work is finished or not, and the holder cannot tell that it has ended. A slow call or a long garbage collection pause looks the same from inside: the worker carries on, holding something it lost minutes ago."

Key Metrics and Impact Scope

• Total Workers in Pool: 8 concurrent worker processes.
• Baseline Invoice Processing Duration: ~4 seconds.
• Peak Invoice Processing Duration (Enterprise Accounts): 2 to 5 minutes (120 to 300 seconds).
• Configured Lock TTL: 60 seconds.
• Severity Multiplier: The 60–second TTL was exceeded by up to 300%, guaranteeing that every large enterprise account experienced lease expiration mid-stream.
• Customer Impact:
  • 212 business customers received duplicate (2x) invoices.
  • 9 business customers received triplicate (3x) invoices.
• Remediation Speed: Credit notes were generated, verified, and dispatched to affected clients by the afternoon of the same business day.

Official Statements and Retrospective Learnings

The engineering team wasted no time shifting from incident containment to root-cause analysis and architectural hardening. Rather than viewing the event as a simple bug, leadership categorized it as a structural flaw in how concurrency safety was enforced.

The Pivot: Moving the Safety Guarantee Downstream

The core philosophical shift in the team’s post-mortem was moving the primary source of truth regarding uniqueness away from a transient, volatile cache (Redis) and embedding it permanently into the persistent data layer (the relational database).

Previously, Redis was tasked with preventing duplicate work from ever being initiated. Post-incident, Redis is treated strictly as an optimization layer—a mechanism to waste less work by preventing multiple workers from simultaneously hammering the database with identical heavy queries.

However, safety can no longer rely on Redis alone. If a lock expires, the system must fail safely at the database level rather than noisily in a customer’s email inbox.

Architectural Redesign

The engineering team implemented five distinct hardening measures to ensure this failure mode can never occur again:

1. Database-Level Unique Constraints:
   The billing database schema was updated to enforce a strict composite unique constraint on the combination of customer_id and billing_period. No matter how many workers attempt to write an invoice for the same customer and billing cycle, only the first transaction will succeed. Subsequent attempts will trigger a database constraint violation and abort gracefully.

2. Transactional Outbox Pattern:
   Email dispatches are no longer triggered opportunistically during worker execution. Instead, the system utilizes an outbox pattern. The email notification event is written to an outbox table within the exact same database transaction that saves the invoice. The email dispatch worker only fires after the database insert successfully commits, ensuring that duplicate invoice generations that hit the database constraint are quietly discarded before any communication reaches a client inbox.

3. Fencing Tokens:
   To prevent stale workers from executing writes even if they somehow bypass other checks, lock acquisition now incorporates a strictly monotonic fencing token—an integer that increments with every successful lock acquisition. The database validates this token on every write operation, rejecting any write accompanied by an obsolete token from an expired lease holder.

4. Safe Lock Release Scripts:
   Workers no longer blindly execute a DEL command on Redis keys. Instead, lock release is managed via an atomic Lua script that verifies the token value. A worker will only delete a key if the key’s current value still matches its own unique identifier, preventing a fast worker from accidentally destroying a slow worker’s newly acquired lock.

5. Dynamic Lease Renewal (Heartbeating):
   Workers engaged in long-running tasks now maintain an active heartbeat. Every 20 seconds, active workers ping Redis to extend their lease. If a lease renewal fails due to network partitions or worker degradation, the worker immediately aborts the job, preventing phantom processing.

6. Comprehensive Observability and Alerting:
   New monitoring metrics were established to trigger high-priority alerts whenever a Redis lock expires while the thread holding that lock is still actively executing. This provides early warning signs of performance degradation before customers are impacted.

Future Outlook: Building Resilient Distributed Workflows

The June 2 incident serves as a vital cautionary tale for engineering organizations scaling legacy workflows to meet enterprise demands. As software systems ingest larger payloads and rely more heavily on third-party microservices, execution times inevitably become non-deterministic.

By embracing the reality that distributed locks are fallible leases rather than absolute guarantees, Sergey Shinder and his team transformed a painful operational failure into an engineering masterclass in defensive system design. The integration of database-level unique constraints, transactional outboxes, fencing tokens, and automated lease renewals ensures that the platform’s billing infrastructure is now resilient against latency spikes, garbage collection pauses, and human oversight alike.

As distributed systems continue to grow in complexity, the lessons learned from this Redis timeout failure will undoubtedly serve as a benchmark for robust concurrency management across the broader software engineering community.