Post-mortem: GitHub MySQL replication lag

Incident Summary

Date: Composite incident (GitHub has documented multiple replication lag events; this is a representative analysis based on public disclosures) Duration: 4-8 hours of elevated replica lag, partial degradation continuing for 12+ hours Systems affected: MySQL read replicas serving GitHub.com, including issue trackers, pull request views, repository listings, and user profile data Impact: Read queries returned stale data. Users saw missing issues, outdated pull request statuses, and phantom "file not found" errors on recently created repos. Some reads fell back to primary, increasing primary load to dangerous levels. Root cause: A schema migration (ALTER TABLE or gh-ost operation) on a multi-billion-row table caused replica lag to grow from milliseconds to hours. Single-threaded replication on replicas meant large DDL operations blocked all subsequent replication events, creating a cascading stale-data problem across the platform.

GitHub runs one of the world's largest MySQL deployments. At this scale, operations that are routine for smaller databases become risky production events. Schema migrations sit at the intersection of every database risk factor: table locking, replication lag, disk I/O, and query plan invalidation. This incident illustrates what happens when those factors converge on a table with billions of rows and thousands of writes per second.

I think about this incident every time someone says "it's just an ALTER TABLE, we'll run it during low traffic." Schema migrations on large tables at GitHub's scale are not maintenance tasks. They are production deployments that deserve the same caution as a code release.

---

What Happened: The Timeline

The timeline reveals the fundamental problem with large DDL operations on replicated databases: the impact is not immediate, it is cumulative. Lag starts small and grows linearly. By the time it is noticeable to users, the backlog is already large enough that recovery takes hours even after the migration is paused.

The user-visible symptoms were varied and confusing. A developer opens a pull request at 11:00 AM. They see it in their dashboard (served from the primary via write-read-your-own-writes routing). Their teammate refreshes the PR list at 11:05 AM but the PR does not appear (served from a replica that is 40 minutes behind). The teammate thinks the developer has not opened the PR yet. Meanwhile, CI status checks that ran against a lagged replica returned outdated file contents for the diff. The symptoms looked like "GitHub is randomly broken" rather than "GitHub has replication lag," which made it harder for users and support to pinpoint the issue.

Another symptom: merge conflicts that should not exist. If a developer pushes a commit and immediately tries to merge a PR, the merge operation might read the base branch from a lagged replica that does not have the latest commit. The merge sees a conflict that would not exist if it read from the primary. This is one of those edge cases that only surfaces at scale with minutes of replication lag.

---

GitHub's MySQL Architecture

Before their migration to Vitess, GitHub operated one of the largest MySQL deployments in the world. Understanding the architecture explains why replication lag had such broad impact.

A few things stand out about this architecture:

Read-heavy workload. GitHub's traffic is overwhelmingly reads: viewing issues, browsing code, loading pull requests. The read-to-write ratio is roughly 10:1 or higher. This is why they use multiple read replicas. The primary cannot handle both writes and the full read load alone.

Asynchronous replication. MySQL replication is asynchronous by default. The primary writes to its binary log and returns success to the client immediately. Replicas pull from the binary log and apply changes at their own pace. This means replicas are always slightly behind the primary. Under normal conditions, the lag is sub-second and invisible to users. Under stress, it can grow to minutes or hours.

ProxySQL as the routing layer. ProxySQL sits between the application and MySQL, routing writes to the primary and reads to replicas. It monitors replica lag and can reroute reads to the primary when lag exceeds a configurable threshold. This is a safety mechanism, but it has a dangerous side effect: when replicas are lagged, the primary absorbs the full read load on top of its write load.

For interviews, this is a textbook read-replica architecture. If you are asked about database scaling, you should be able to draw this diagram from memory and explain the tradeoff: replicas give you read throughput, but replication lag means reads can be stale. The question is always "how stale is acceptable?"

---

Root Cause: Schema Migration on a Hot Table

The specific trigger was a schema migration on one of GitHub's largest and most actively written tables. Think of tables like issues, pull_requests, or repository_events, tables with billions of rows that receive thousands of writes per second.

What the migration looked like:

GitHub uses gh-ost (GitHub Online Schema Trickle) for online schema changes. gh-ost works by:

1. Creating a shadow copy of the table with the new schema
2. Copying existing rows from the original table to the shadow table in small batches
3. Tailing the binary log to capture any writes that happen during the copy
4. Applying those captured writes to the shadow table to keep it in sync
5. Performing an atomic table swap once the shadow table is fully caught up

On the primary, this is relatively graceful. gh-ost copies rows in small chunks, throttles based on server load, and does not lock the original table for the duration. Writes continue unimpeded. The primary barely notices.

The key difference: on the primary, gh-ost controls the pace. On replicas, there is no throttle. The replica processes the raw binlog events as fast as it can, but DDL events block the single applier thread.

The problem is on the replicas.

When gh-ost performs the row copy and the final table swap, those operations flow through the binary log to replicas as normal replication events. But the replicas do not have gh-ost's throttling logic. They receive the raw binlog events and must apply them through their SQL applier thread.

In MySQL versions prior to 5.7's parallel replication improvements (and even in early implementations of parallel replication), the SQL applier was effectively single-threaded for DDL operations. A large DDL operation would block the single applier thread, and all subsequent binlog events would queue behind it.

The cascade: