OceanBase at Didi: Large-Scale Operations Experience and New Feature Adoption

Author: Wu Qipeng, Head of Distributed Storage Operations at Didi

Didi Chuxing (hereafter “Didi”) is a one-stop, diversified mobility platform spanning ride-hailing, taxi, carpooling, designated driving, and more, serving 650 million customers worldwide. Since adopting OceanBase in 2024, Didi has rolled it out across many scenarios and replaced RocksDB and TokuDB, including its ride-hailing growth service, the core archive database of its middle platform, the designated-driving core archive database, EP, autonomous-vehicle services, and others. Taking core workloads such as the ride-hailing growth service and the archive database as examples, this article describes Didi’s database technology experience and its practice with new features.

Didi’s Database Use Cases and Technical Solutions

Scenario 1: The Core Archive Database

Didi’s archive database carries the high access volume of online workloads. It behaves more like an online cold store than a repository for archived cold data. Today the largest archive cluster holds 100TB, with QPS (queries per second) peaking at eight thousand (8,000/s). Because of its persistent, high-frequency access pattern, the traditional sharded-table model struggled to meet rapid scaling needs, so we concluded that migrating the core archive database to OceanBase was the critical path.

Status Quo: Cutting Costs with Large-Disk Hardware

Thanks to OceanBase’s advanced compression ratio and natively distributed architecture, in archive scenarios with 100+TB of data, storage costs dropped 20% compared to the TokuDB sharding architecture. In addition, because OceanBase can run on machines with larger disks, we saved even more on operating costs.

This raises a question: why can’t TokuDB use large-disk machines? There are two reasons:

• First, if TokuDB ran on large-disk machines, its instances would be huge, making backups and splits take too long and ultimately hurting service availability.
• Second, if you packed many small Toku instances onto a large-disk machine, a single-machine failure would multiply the blast radius.

By contrast, OceanBase’s fast scaling model based on Unit splitting can spread nodes within minutes and split Units within hours. This not only dramatically improves operational efficiency but also increases service stability.

So, how do we choose the right spec when using large-disk machines?

We evaluate how large a disk to use based on the single-machine recovery time, where differences in disk performance and network bandwidth are both factors that affect recovery time, so the exact disk size has to be tailored to the situation.

For example: using a 3TB MySQL service recovery time as the baseline, backup-and-failure recovery takes roughly 7–8 hours. When choosing large-disk machines for OceanBase, you can size the disk capacity against that same 7–8 hour standard.

Challenge: High Latency When Migrating Hundreds of TB

Because the archive database holds so much data, the business side worried that migrating from TokuDB to OceanBase carried stability risks around performance, data consistency verification, and migration efficiency, so we ran targeted tests and validation.

First, on performance: to keep response latency under control, we ran a canary test. Out of the thousands of tables in the upstream archive database, we migrated a small slice of data from each table—fast and cheap. But when the business side ran a traffic test, latency spiked by tens of times. Our SQL analysis revealed that, faced with thousands of tables, every first access to a SQL statement triggered hard parsing. How do we avoid this? After talking with the business side, our strategy was to merge the thousands of upstream tables into a handful of downstream tables; the business side only had to tweak a suffix to effectively reduce the number of hard parses. This greatly reduced response latency, keeping the 99th percentile within tens of milliseconds.

Second, migration efficiency is tied to data consistency verification. We used a 5TB test migration as a reference point to gauge the overall migration pace. During migration, however, we found data verification extremely time-consuming—verifying a few TB took several days, severely hurting migration efficiency. With help from the OceanBase community, we solved this by upgrading OMS and using its feature to filter columns during OMS data verification. In essence, this filters out large columns—and these large columns are usually non-core fields that have no impact on the business, yet excluding them cut verification time from days down to hours.

One more tip: during full and incremental migration with OMS, make heavy use of the OMS Diagnose feature. It can identify migration bottlenecks based on the current migration speed and, taking system capacity into account, suggest a reasonable migration plan—such as adjusting downstream concurrency or raising upstream concurrency—thereby improving migration efficiency.

Scenario 2: The Ride-Hailing Growth Service

Status Quo: Tens of Billions of Rows Running Efficiently

The ride-hailing growth workload is what we commonly call the feature store. The feature store maps tables to business lines, so each table has a huge number of columns and tens of billions of rows. With wide single rows and queries over specific ranges, peak online QPS reaches 25k/s. Because the business relies on the feature store for data aggregation and analysis, it requires the feature store’s response latency to stay within 80ms; therefore, when a single SQL statement runs longer than 200ms, the business triggers a circuit-breaker retry to avoid faults and larger negative impact.

Initially, we planned to use MySQL sharding to support the feature store’s requirements, but it never went into production. The reasons were:

• As mentioned above, this workload includes range queries across different dimensions, so we couldn’t split a single table.
• The feature store’s single rows are too wide, MySQL’s performance fell short, and validation failed.

Today, the feature store runs on OceanBase as follows:

1. OceanBase’s day-to-day response latency is around 30ms, meeting the business requirement.
2. Second-level DDL meets the need for rapid iteration, greatly improving iteration efficiency. With MySQL’s copy approach, a single table would take days to complete.
3. OceanBase’s partitioned tables and global indexes are extremely handy: you can partition by column to increase the concurrency of a single table and improve access efficiency for tens-of-billions-row tables; global indexes can satisfy range queries across different dimensions on a single table—such as querying by driver or by order. Not only are queries efficient, but operational complexity is also reduced.

Challenge: High-Concurrency Workloads

The feature store connects to a wide range of upstream business lines, each of which relies on it for data aggregation and analysis. When an upstream business line has an analysis request, the feature service breaks it into a model. For example, a single upstream query gets split by the feature service into multiple SQL statements that hit OceanBase concurrently—so the downstream OceanBase actually sees amplified traffic. The business also has a circuit-breaker retry mechanism beyond 200ms to guarantee its SLA to upstream.

In such a high-pressure scenario, risks are ever-present, for example:

• Sudden traffic surges or fluctuations can cause retry storms that hurt system stability.
• How to reasonably set throttling thresholds is closely tied to whether the system runs stably under high concurrency.
• According to OCP SQL diagnostics, the business has thousands of SQL templates, which can render throttling ineffective.

We took targeted measures to address these challenges.

1. Reworking the business retry logic.

Our approach was to talk with the business side and change the retry mechanism to a tiered one—from a fixed 200ms retry to a progressive 200ms, 400ms, 800ms.

For example, suppose a SQL statement normally runs in 100ms. When it times out at 200ms, something has clearly gone wrong: a network issue, a single-machine failure, or something else. Continuing to retry every 200ms is pointless and only adds extra load to the database. A tiered retry, by contrast, eases the pressure of retry storms.

2. Implementing throttling thresholds.

We parse SQL audit data during business peak hours to obtain concurrency and set reasonable thresholds. This also surfaces those ultra-high-concurrency SQL statements so we can guard against them in advance.

For SQL on high-concurrency business lines, what throttling threshold is appropriate?

Picking 5 or 10 off the top of your head has no basis. Instead, we parse SQL audit data during business peak hours to obtain concurrency. For instance, using a 30ms window—since the business’s daily average response latency is 30ms—we can determine how many concurrent calls a single SQL template has within that window. We choose to throttle based on the 90th-percentile concurrency: SQL exceeding the 90th percentile of concurrency prompts a notice to the business side to adjust and lower its concurrency.

3. Optimizing the massive SQL set.

After enabling throttling, we discovered a problem: the number of throttled templates reached the thousands, hurting throttling efficiency. Our SQL analysis found these throttled templates shared one thing in common—identical access conditions and identical accessed data, differing only in the order of the syntax. For OceanBase’s SQL throttling, too many SQL templates can make throttling ineffective. So we asked the business side to revise its logic and reduce the SQL templates from thousands to under 100, greatly lowering the risk of throttling failure.

4. Trying OceanBase 4.3.5 bp3.

The SQL-template-level throttling described above can only prevent a service from being brought down when a few problem SQL statements (slow queries, traffic surges, etc.) consume a large number of threads, eventually rendering the entire service unavailable.

For example: we have a 10C 50G UNIT tenant whose maximum single-machine concurrency can reach 40. When a single SQL statement’s concurrency is capped at 10, it can at most guard against fewer than 3 problem SQL statements. Once more than 3 such statements appear, the threads still get fully occupied, and overall service availability still can’t be guaranteed.

The new OceanBase 4.3.5 version perfectly supports database- and table-level throttling. This lets us set multi-tier throttling rules to effectively prevent problems at the SQL level or table level from taking down the entire service.

Building and Practicing a Database Operations System

1. Customized Monitoring and Alerting

First, hardware-aware threshold alerts. When a single OCP manages multiple OceanBase clusters, those clusters may include archive databases or high-concurrency databases with heavy traffic. Didi’s strategy is to use large-disk storage with few compute resources for the archive database and machines with more compute resources for the feature store.

This creates a problem: if both workloads share the same alert configuration, large-disk machines waste resources. We therefore need to configure different alerts for different hardware types to maximize resource utilization. Moreover, when swapping hardware, alerting strategies also need to be tailored to the new machine type.

We set the ob_server_sstable_percent_over_threshold alert threshold according to each physical machine’s configuration, avoiding false positives or missed alerts caused by a single uniform standard. For example, high-capacity archive machines are allowed a higher SSTable ratio, while high-concurrency feature-store machines get stricter thresholds, ensuring cluster stability.

Second, ZONE-level switch-isolation alerts. OceanBase implements multi-replica disaster recovery based on the Paxos protocol. When the majority of replicas fail, part of the business data or even an entire cluster can become unavailable. We therefore built a network-topology alert mechanism for UNITs deployed under the same TOR within different ZONEs, to prevent a single switch failure from invalidating part of the business data. This alert has become a core, high-priority item that must be responded to and handled immediately.

2. Rolling Out the Binlog Server 4.x High-Availability Version

In Didi’s business pipelines, whether for SQL or OceanBase, upstream data must be synced to downstream MQ or Kafka so that different business lines or scenarios can consume and analyze it. As a result, the binlog sync pipeline is a hard dependency for some workloads.

While rolling out the Binlog Server high-availability version, we validated its syntax compatibility, high availability, and data consistency—all met expectations. However, during repeated HA switchovers, all instances ended up switched onto a single machine. While this doesn’t affect the HA pipeline, it can cause resource imbalance; we hope future versions improve on this.

3. SOPs and Fire Drills

SOPs are something you build up gradually, layer by layer. They require the baptism of real incidents and, with the patient guidance of OceanBase’s official team, the accumulation of operations experience that is then distilled into SOPs. The benefits are obvious: they improve consistency, efficiency, and accuracy at work, and provide important support for the team’s operations. But SOPs alone aren’t enough to handle production risks—you also need fire drills to validate that the SOPs actually work. The two complement each other, continually improving the organization’s overall emergency response capability in the face of all kinds of incidents.

4. Setting Up an Operations Team

You might ask: is it really necessary to set up a dedicated operations team?

The point of an operations team is:

• Avoid single points of risk. Beyond 24-hour alerting, when facing a major incident we want to divide the work and stop the bleeding quickly.
• Help the team build technical reserves and better serve the business.
• Pool everyone’s wisdom and help OceanBase go further and fly higher at Didi.

Within the team, we do several things:

• Hold regular architecture analysis sessions—for example, kernel-technology deep dives, source-code analysis, and technical-solution sharing.
• Establish knowledge-transfer mechanisms—for example, documenting solutions and conducting incident postmortems so team members become part of the operations system and grow quickly.
• Run online incident-simulation drills. By using SOPs to simulate operations in a production-like environment and continually running fire drills, we can stay calm and resolve incidents in an orderly way when they happen.
• Participate in major change implementations.

Closing Thoughts: Our Expectations for the Database

Using OceanBase, we have two very direct takeaways.

1. OCP’s GUI-based tooling greatly lowers the difficulty of operations, turning previously complex tasks into simple, convenient ones. Even when managing hundreds of physical machines, you can handle it with ease. And through OCP’s APIs, you can quickly build some customized features.
2. The OceanBase database can meet multi-dimensional business needs, giving the business side the best of both worlds.

On database upgrades, though, we still have expectations. Currently OceanBase performs rolling upgrades on OB Servers and cannot keep individual OB Server versions inconsistent for the long term, which doesn’t meet our canary-operations standard for major operations. You can also upgrade via primary-standby databases or OMS sync pipelines—an approach that is robust, but if a cluster involves hundreds of machines, it wastes a lot of resources.

We therefore hope OceanBase will not only offer small-traffic canary upgrades but also support rollback, so that users can take appropriate measures to stop the bleeding if they hit online business incompatibilities or other unknown problems during a database upgrade.