TiDB Architecture: MySQL Compatibility on a Distributed Layer

In the era of cloud‑native applications, the promise of “scale‑out MySQL” is louder than ever. Developers love MySQL’s familiar SQL dialect, its rich ecosystem of tools, and the low barrier to entry, yet they also need the horizontal elasticity that traditional relational databases struggle to provide. TiDB answers that call by re‑imagining MySQL on top of a truly distributed storage and compute fabric. It does so without sacrificing ACID guarantees, strong consistency, or the ability to run online schema changes while the database stays live.

Why does this matter beyond the data‑center? The same principles that keep a distributed SQL engine humming—self‑organization, fault tolerance, and graceful adaptation—are also at the heart of resilient ecosystems, from bee colonies orchestrating pollination to autonomous AI agents negotiating shared resources. By unpacking TiDB’s architecture—its TiKV storage layer, Placement Driver scheduling, and online schema change mechanisms—we can see how engineering for data can echo the engineering of nature and intelligent collectives.

In this pillar article we’ll dive deep into each component, illustrate real‑world numbers, and surface the concrete mechanisms that make TiDB a production‑grade, MySQL‑compatible database for the modern, distributed world.

1. TiDB at a Glance: MySQL Compatibility Meets Distributed Scale

TiDB is an open‑source, cloud‑native, NewSQL database built by PingCAP. Its core promise is MySQL 5.7 protocol compatibility with horizontal scalability and strong consistency. The architecture follows a separation of concerns:

Because the SQL layer speaks the same dialect as MySQL, existing applications, ORMs, and tools (e.g., MySQL Workbench, Liquibase, Flyway) can connect without code changes. However, TiDB does not ship with a monolithic binary; instead, it runs a fleet of stateless TiDB servers, a set of TiKV nodes, and a small quorum of Placement Driver (PD) instances.

Key numbers (as of the latest 7.5 release):

• Linear scalability: Adding a TiKV node can increase write throughput by ~30 % on a balanced cluster (benchmarked with YCSB at 10 M ops/s).
• Latency: Typical read latency stays under 5 ms for single‑row primary‑key lookups across a 5‑node cluster in a single AZ.
• Consistency: Guarantees strict serializability via Raft, comparable to the guarantees InnoDB provides on a single node.

The result is a platform where developers can write classic MySQL queries, while ops teams can grow the cluster to meet traffic spikes without sharding manually.

2. TiKV: The Distributed Key‑Value Engine Under the Hood

TiKV (pronounced “tee‑kv”) is the storage engine that powers TiDB’s durability and scalability. It is a Rust‑implemented, transactional key‑value store that follows the Google Percolator model—originally designed for Google Spanner. TiKV stores data as ordered key‑value pairs inside RocksDB instances, each running on a separate physical or virtual machine.

2.1 Region‑Based Sharding

TiKV automatically splits the keyspace into regions of roughly 96 MiB each (configurable). Each region is assigned a Raft group of three replicas (default replication factor = 3). When a region grows beyond the threshold, TiKV splits it into two new regions, each inheriting the same Raft group membership. This design yields several concrete benefits:

In practice, a 10‑TB TiDB cluster may contain ~100 000 regions, each independently scheduled.

2.2 Raft Consensus in TiKV

Every region runs its own Raft consensus group. The leader of a group handles all reads/writes for that region, while followers replicate logs. TiKV uses the Raft v2 implementation with the following guarantees:

• Leader election timeout: 1 s (configurable)
• Heartbeat interval: 200 ms
• Commit latency: ~2 × the network RTT + disk sync, typically 2–4 ms in a LAN.

Because each region is small, Raft’s log size remains manageable, preventing the “log bloat” problem that can plague monolithic Raft deployments.

2.3 MVCC and Transactional Guarantees

TiKV implements Multi‑Version Concurrency Control (MVCC) using timestamp‑based versioning. Each write gets a commit timestamp from the Timestamp Oracle (TSO) service provided by PD. Reads specify a read timestamp, ensuring they see a consistent snapshot. The process works like this:

1. Begin Transaction – TiDB obtains a startTS from PD.
2. Read – TiDB sends a Get request to TiKV with startTS; TiKV returns the latest version ≤ startTS.
3. Write – TiDB buffers mutations locally; on Commit, it obtains a commitTS from PD.
4. Two‑Phase Commit (2PC) – TiDB sends a pre‑write to each involved region (writes with lock and startTS). After all pre‑writes succeed, it sends a commit with commitTS.

The two‑phase commit guarantees serializable isolation across the cluster. In benchmark suites (e.g., TPC‑C), TiDB achieves ~10 k TPS on a 6‑node cluster with 5 ms latency, comparable to a high‑end single‑node MySQL with InnoDB.

2.4 Data Locality and Compression

TiKV stores keys in big‑endian order, enabling range scans to be performed efficiently. It also supports columnar compression (Snappy, Zstd) at the RocksDB level, achieving 1.8×–2.5× space savings for typical OLTP workloads. For example, a 1 TB MySQL InnoDB dataset compresses to ~440 GB in TiKV with Zstd level 3.

3. Placement Driver (PD): The Global Scheduler and Metadata Service

The Placement Driver is the brain of a TiDB cluster. It maintains cluster topology, metadata, and scheduling decisions for all TiKV regions. PD runs as a small, highly available quorum (usually 3 or 5 instances) using the same Raft algorithm as TiKV.

3.1 Timestamp Oracle (TSO)

Every transaction needs a globally unique, monotonically increasing timestamp. PD’s TSO service provides this by:

• Maintaining a logical clock (increments by 1 for each request).
• Optionally synchronizing with Physical Time (NTP) to bound the timestamp value (e.g., 48 bits for physical, 16 bits for logical).

The TSO can sustain >10 k requests/s per PD node with sub‑millisecond latency, which is crucial for high‑throughput workloads.

3.2 Region Scheduling Algorithms

PD continuously monitors region statistics (size, read/write QPS, hot keys) via heartbeat messages from TiKV. Its scheduler runs three main algorithms:

1. Balance Scheduler – Moves regions from overloaded stores to under‑utilized ones, targeting a store load variance < 15 %.
2. Hot Spot Scheduler – Detects “hot regions” (e.g., >10 k QPS) and replicates them to additional stores or splits them further.
3. Safety Scheduler – Ensures that no two replicas of the same region end up on the same physical host or rack, respecting topology constraints.

In a production deployment at PingCAP Cloud, PD’s hot‑spot scheduler reduced write latency spikes from 120 ms down to 8 ms by automatically splitting and relocating hot regions.

3.3 Store Management and Fault Detection

Each TiKV node sends a store heartbeat every 10 seconds, containing CPU, memory, disk usage, and latency metrics. PD flags a store as offline if three consecutive heartbeats are missed, then initiates region re‑replication to maintain the desired replica count. The automatic failover typically completes within 30 seconds, after which the cluster continues serving reads/writes without client‑side errors.

3.4 Configuration and Extensibility

PD exposes a RESTful API and a gRPC interface for custom schedulers. Organizations can plug in domain‑specific policies—for instance, a scheduler that keeps all regions containing “bee‑survey” data on nodes located in data centers powered by renewable energy. This extensibility mirrors the self‑governing capabilities seen in AI agent frameworks where policies can be dynamically injected.

4. Distributed Transaction Processing: The Percolator Model in Action

TiDB’s transaction engine draws heavily from Google’s Percolator design, which pioneered optimistic concurrency control combined with two‑phase commit across a distributed key‑value store.

4.1 Optimistic Concurrency Control (OCC)

When a transaction reads a row, TiDB records the read version (startTS). Writes are buffered locally until commit. At commit time, TiDB checks for write conflicts by ensuring that no other transaction has committed a later version of any key it touched. This check is performed during the pre‑write phase:

• TiKV places a lock on each key with the startTS.
• If another transaction already holds a lock with a higher startTS, the pre‑write fails, and TiDB aborts the transaction.

Because most OLTP workloads are write‑light relative to reads, OCC yields high concurrency with low lock contention.

4.2 Two‑Phase Commit (2PC) Flow

The 2PC process is orchestrated by the TiDB server but executed by TiKV nodes:

1. Pre‑write – TiDB sends a Prewrite RPC to each region, attaching the startTS and the mutations. TiKV writes a lock entry and a prepare record to its Raft log.
2. Commit – After all pre‑writes succeed, TiDB obtains a commitTS from PD and sends a Commit RPC to each region. TiKV then writes the final value with commitTS and removes the lock.
3. Rollback – If any pre‑write fails, TiDB issues a Rollback to all regions that succeeded, cleaning up locks.

The entire 2PC round‑trip typically takes 2–3 network hops (client → TiDB → TiKV leader). In a 3‑region transaction spanning three TiKV nodes, the commit latency averages 8 ms on a 10 Gbps LAN.

4.3 Distributed Deadlock Detection

TiDB implements a wait‑for graph in the PD leader to detect deadlocks across regions. If a cycle is found, the youngest transaction is aborted. Empirical data from the Alibaba Cloud TiDB deployment shows deadlock rates below 0.02 % even under 50 k concurrent transactions.

5. Online Schema Changes: Evolving the Database Without Downtime

One of TiDB’s most compelling features for production teams is its ability to perform online schema changes (OSC)—adding columns, modifying indexes, or even changing table engines—while the database remains fully operational.

5.1 The “DDL Job Queue”

All DDL statements are stored as jobs in the PD meta store. Each job has a state machine: Pending → Running → Done/Cancelled. The DDL worker in each TiDB server polls the job queue, picks up the next pending job, and coordinates the change across the cluster.

5.2 Chunk‑Based Backfilling

When adding a column with a default value, TiDB does not rewrite the entire table at once. Instead, it:

1. Adds a hidden column with a NULL default.
2. Backfills the column in chunks (e.g., 64 MiB per chunk) using background workers.
3. Updates the column metadata once all chunks are processed.

This approach limits the impact on CPU and I/O. In a benchmark with a 200 M‑row table (≈ 250 GB), adding a VARCHAR(64) column completed in 22 minutes, with average query latency staying under 6 ms.

5.3 Index Rebuilding

Creating a new secondary index triggers a distributed index build:

• TiDB scans each region’s primary key range.
• For each row, it extracts the indexed columns and writes an entry into the index table (a hidden KV table).
• The index is built in parallel across all TiKV nodes, leveraging the same Raft replication.

Because the index build uses snapshot reads at a fixed timestamp, it does not interfere with ongoing writes. In a real‑world scenario at Shopee, building a composite index on a 1.2 TB orders table took 45 minutes with negligible impact on the front‑end latency.

5.4 Schema Version Propagation

TiDB nodes maintain a schema version number stored in PD. When a DDL job reaches the Done state, PD increments the version. TiDB servers periodically fetch the latest version (default every 5 seconds). This ensures that all query planners instantly see the new schema without a restart.

5.5 Safety Checks and Rollbacks

If a DDL job encounters an error (e.g., insufficient disk space during backfill), TiDB automatically cancels the job and rolls back any partially applied changes. The rollback is also performed in chunks, preserving the same low‑impact guarantees.

6. High Availability, Fault Tolerance, and Raft Consensus Across the Cluster

TiDB’s resilience stems from the Raft consensus algorithm applied at two levels: region‑level (inside TiKV) and cluster‑level (PD). Understanding how these layers interact clarifies why TiDB can survive node, rack, or even data‑center failures.

6.1 Multi‑Region Replication

Each region’s Raft group maintains 3 replicas by default (configurable up to 5). The replication factor determines the fault tolerance:

If a TiKV node crashes, its regions’ leaders are automatically re‑elected on surviving replicas. The election timeout (1 s) ensures that leader change completes within 2–3 seconds.

6.2 Placement Constraints

PD can enforce topology constraints—for example, placing each replica on a distinct rack or availability zone. In a multi‑AZ deployment on AWS, a typical configuration spreads replicas across three AZs, guaranteeing that the loss of an entire AZ still leaves a quorum (2 out of 3) for each region.

6.3 PD High Availability

PD itself runs a Raft group of 3–5 nodes. The PD leader coordinates TSO and scheduling; if it fails, a new leader is elected within 500 ms. Because PD stores only metadata (not user data), the impact on query latency is minimal.

6.4 Disaster Recovery (DR)

TiDB supports cross‑region DR via TiDB Binlog (a change‑data‑capture pipeline). Binlog streams transaction logs to a remote TiDB cluster, which can be promoted to primary in a failover scenario. In PingCAP’s production tests, a 5‑GB/s binlog pipeline sustained a 1 TB/h data change rate with <2 seconds lag, enabling near‑real‑time DR.

7. Observability, Monitoring, and the TiDB Dashboard

Running a distributed system without visibility is akin to managing a bee hive in the dark. TiDB offers a rich observability stack that helps operators track performance, detect anomalies, and fine‑tune the cluster.

7.1 Metrics Exporter Stack

• Prometheus scrapes metrics from TiDB, TiKV, and PD exporters. Over 200 metrics are exposed, covering:
• Region size and hotness (tidb_region_size_bytes, tikv_region_write_bytes_total)
• Raft leader election latency (pd_raft_election_duration_seconds)
• Transaction latency (tidb_server_query_duration_seconds)

• Grafana dashboards (officially maintained) provide out‑of‑the‑box visualizations for cluster health, CPU/IO utilization, and DDL progress.

7.2 TiDB Dashboard

The TiDB Dashboard is a web UI built on top of the same metrics. It offers:

• Real‑time query profiling (similar to EXPLAIN ANALYZE).
• Hot region heatmaps that show which key ranges dominate I/O.
• DDL job inspector that visualizes backfill progress per region.

In a field study with a wildlife‑tracking platform, operators used the dashboard to spot a sudden spike in write latency caused by a single hot region storing GPS data; PD automatically split the region, and latency returned to baseline within minutes.

7.3 Tracing with OpenTelemetry

TiDB integrates with OpenTelemetry to trace SQL queries across the SQL, transaction, and storage layers. A typical trace for a SELECT shows:

1. SQL parsing (TiDB server)
2. Plan generation (TiDB optimizer)
3. Region lookup (PD)
4. Raft read (TiKV leader)
5. Result assembly (TiDB)

These traces help developers pinpoint bottlenecks, just as a beekeeper might trace pollen flow to identify a weak hive.

8. Real‑World Deployments: Lessons from Scale

8.1 PingCAP Cloud (SaaS)

• Cluster size: 30 TiKV nodes, 6 PD nodes across three AZs.
• Peak QPS: 120 k reads/s, 30 k writes/s.
• Latency: 99th‑percentile read latency 7 ms.
• Key achievement: Zero‑downtime schema migration of a 500 TB user‑profile table using TiDB’s OSC, completed in 3 hours.

8.2 Uber’s Real‑Time Dispatch

Uber replaced a sharded MySQL fleet with a TiDB cluster to power its surge‑pricing engine. The system processes ~2 M location updates per second. By leveraging TiKV’s region splitting, they kept write latency under 10 ms even during peak traffic (NYC rush hour).

8.3 Shopee’s Order Management

Shopee runs a TiDB cluster for order processing, handling ~15 M orders per day. They use TiDB’s global secondary indexes to support flexible search on order status and customer region. The online schema change feature allowed them to add a new promo_code column to the orders table without any service interruption.

8.4 Bee‑Survey Platform (Open‑Source)

A research project collecting pollination data from citizen scientists deployed TiDB on a Kubernetes cluster in a university data center. The platform needed to ingest 10 k sensor readings per second while supporting ad‑hoc analytics. TiDB’s MySQL compatibility let the team reuse existing Python ORM code, while the distributed storage ensured the data persisted even when the lab’s power was cycled.

These stories illustrate that TiDB’s architecture scales from small research clusters to global SaaS platforms, delivering consistent MySQL semantics throughout.

9. Parallels with Bee Colonies and Self‑Governing AI Agents

When we examine TiDB’s design, a striking analogy emerges with bee colonies:

Both systems thrive on local decisions that aggregate into global stability.