From v3.1 to v4.3, OceanBase Steadily Powers Kuaishou's PB-Scale Core Business Scenarios

This article is excerpted from the e-book A Study of OceanBase Community Edition Use Cases in Pan-Internet Scenarios. Click the link to get the full content.

The Kuaishou app is one of China’s popular short-video and live-streaming applications. As an inclusive digital community, Kuaishou not only lets hundreds of millions of ordinary people record and share their lives, but also helps people discover what they need and put their talents to use. The content on the platform covers all aspects of life: users can record their daily moments through photos and short videos, and interact with fans in real time through live streaming, boosting users’ sense of happiness on a foundation of technological empowerment.

So how does an app like this—with daily visits exceeding tens of millions—handle user requests promptly and effectively when facing high-concurrency traffic? Our early approach was to configure multiple MySQL clusters on the backend to support high-traffic access, thereby solving large-data-volume storage and performance problems. But this traditional MySQL sharding approach had quite a few problems. After adopting OceanBase, the technical compatibility, operational convenience, data synchronization, business stability, and resource-scaling capabilities in our core business scenarios all improved enormously. At the same time, while using OceanBase, we continuously updated versions and accumulated hands-on experience with different versions.

I. From MySQL Sharding to an “Enhanced MySQL”

From its founding in 2011 to its IPO in 2021, Kuaishou’s daily active users surpassed hundreds of millions. Such massive traffic, on one hand, drove the rapid growth of businesses like live streaming and e-commerce; on the other hand, it placed unprecedented pressure on the underlying storage system. Although the traditional database sharding approach relieved this pressure to some extent and improved performance, the complexity and unsustainability of operations sowed even greater hidden dangers in the system.

Take the “order business” carried by the app as an example: when the total business data volume exceeded 150 TB, MySQL’s storage bottlenecks and performance shortcomings became increasingly obvious. To ease the impact of these problems on the business, we chose to address them with a sharding approach. However, continuous business growth kept increasing the number of shards in the underlying database, until the number of online MySQL shards reached 300+—which not only failed to fully solve the storage problem but also brought greater operational complexity. We had to keep modifying and adapting applications to deal with the difficulties that sharding brought.

A short-video app’s peak business QPS (queries per second) can reach over a million, with extremely high performance requirements. In this situation, a single cluster needs many MySQL nodes and still can’t guarantee that business requests return promptly during peak periods; and whether it’s the middleware or all the links downstream of the data, all of them need high-performance hardware to support the product-stability plan.

In addition, our TP business not only requires strong transactions and real-time read/write capability but also comes with AP needs. To ensure the system is stable and reliable, we needed to use a solution combining MySQL with ClickHouse, Elasticsearch, or Doris, and we might need to add more data replicas—undoubtedly bringing higher hardware costs.

We realized that a sharding approach can only relieve the problem as much as possible but cannot fundamentally solve it. We urgently needed a distributed-database solution that could meet business needs while offering high performance, flexible scaling, and lower operational complexity.

On the road of exploring distributed databases, we initially tried a certain brand’s distributed database. But in use we found it had problems in write performance, operations methods, and so on—for example, the operations platform was fairly simple and struggled to meet DBAs’ needs, and many kernel issues were hard to resolve. So we decided to try OceanBase.

Through comparative testing, we found that when a single table’s data volume exceeds 10 TB and keeps growing, performing a DDL operation on the certain brand’s distributed-database architecture is expected to take a week, whereas OceanBase needs less than a day. Because some businesses keep growing and need continuous table additions and source-data additions, DDL operations are numerous. That distributed database uses small partitions, and with large data volumes the number of regions increases significantly; once a crash, scaling, or node-replacement need arises, it’s very likely to affect business stability. By comparison, OceanBase uses large partitions, and the above business operations can be completed within hours, or at most days.

II. Benefits of Applying OceanBase in Core Business Scenarios

As of the end of 2023, Kuaishou’s short-video app had 8 OceanBase clusters, a machine scale of over 200 physical machines, and a data volume exceeding 800 TB, with the largest cluster holding over 400 TB. Initially we used OceanBase 3.1, and later upgraded to 4.3. All clusters provide online services, covering the core transaction-reconciliation system, the payment-gateway business system, and businesses that replace the MySQL primary to handle high-concurrency writes. After migrating to OceanBase 4.1, the clusters improved significantly in both business benefits and stability. Below, we use two core business scenarios as examples to introduce OceanBase’s actual results in production.

(1) Transaction-Reconciliation Scenario

As a short-video platform, e-commerce is one of Kuaishou’s most important business components. Normally, this business keeps daily traffic at a steady 80,000–90,000 QPS. During major live-streaming events, user traffic surges, and QPS quickly soars to ten or even a hundred times the usual level, reaching the million level; at this point, the data volume—even after compression—reaches over a hundred TB, which demands speed, stability, and resilience from the database.

• Millisecond-level latency. The business is extremely sensitive to latency and has very high TPS requirements; latency is usually required to be at the millisecond level. If a request can’t be completed within the required time, it affects the reconciliation results and causes data-inaccuracy problems.
• Stronger stability. If the database jitters, it causes a large number of reconciliation failures. In a transaction-reconciliation scenario, the database must not only stay stable over the long term at daily traffic peaks but also have no jitter during traffic surges.
• Strong resilience. When the data volume exceeds a hundred TB and a single replica of a single cluster reaches around 20 TB, as a single table’s data grows, system-resource consumption increases, which in turn affects the database’s response time. Therefore, with peak read/write requests reaching the million-QPS level, the database must be stable enough that response time is unaffected.

Before introducing OceanBase, both read and write operations in the transaction-reconciliation scenario were performed on MySQL. For the large-table problem, we used the traditional sharding approach—splitting large tables into multiple small ones and splitting the business read/write traffic across multiple MySQL instances. However, sharding has limitations in cross-database data consistency and cross-database transaction atomicity, and it can easily lead to data inconsistency in complex and exceptional situations, which in turn makes data-reconciliation results inaccurate. For example, refunds might go unrecorded or deduction amounts might be inaccurate, ultimately causing financial loss.

After solution research and selection, since OceanBase’s distributed architecture has inherent horizontal-scaling capability, when data volume keeps growing we only need to horizontally scale the cluster’s storage and compute to solve large-table querying and storage problems; moreover, with native distributed capability, it is better at handling distributed transactions. After adopting OceanBase, upstream business writes directly into the MySQL cluster, and each record, when written to an upstream MySQL shard, is synchronized in real time to OceanBase via Binlog. During data-reconciliation queries, the system runs the same query against both the upstream MySQL and the downstream OceanBase and compares the results, thereby guaranteeing the correctness of order status across the entire accounting system.

Figure 1 shows the “online performance of the transaction-reconciliation business.” Daily QPS is on the upper left, with a data figure of around 90,000; the upper right shows response time, with average latency under 10 ms and peak latency reaching 10,000 ms—this is because after the full compaction in the early hours each night, the business launches an extra dedicated thread to delete large amounts of historical data, causing latency to spike at that time, but the business side can accept this. The two curves at the bottom show write volume; the daily TPS counted by transaction is around 10,000, and the response time is 5–10 ms. OceanBase’s response time meets the business’s latency requirements, and system stability is guaranteed.

Figure 1: Online performance of the transaction-reconciliation business

(2) Payment Business Scenario

The payment business is the real-time business of e-commerce. On one hand, it serves merchants and customer service querying live-stream revenue; on the other hand, it involves payment-gateway-related aggregate queries. This business has three distinct characteristics.

(1) Large data volume. The payment business’s data volume is even larger than that of transaction reconciliation; a single cluster’s data volume can reach over a hundred TB, the largest cluster’s data volume has exceeded 400 TB, and a single table’s data volume reaches 10 TB or more.

(2) Complex aggregation. This business’s write traffic is far higher than its query traffic, and all backend queries must go through the payment gateway, involving aggregate queries over large amounts of data that may aggregate dozens or even over a hundred tables, so the business has high requirements for query performance.

(3) Frequent DDL. Because queries aren’t fixed, the business needs to add indexes frequently to improve speed. The previous approach was to synchronize data to an Elasticsearch cluster while writing it to the MySQL cluster, using Elasticsearch’s search capability to provide complex AP query analysis for the business. Although the new approach could meet business needs to some extent, it still had the following three problems.

• Insufficient data real-timeliness: data is written to MySQL and then synced to Elasticsearch, with poor timeliness, and data delays may occur due to MySQL’s continuous heavy writes.
• High cost: because of the added Elasticsearch cluster, the business has to consider not only MySQL’s cost but also Elasticsearch’s hardware and maintenance costs.
• More complex operations: it has to maintain both the MySQL cluster and the Elasticsearch cluster.

After introducing OceanBase, its online scalability easily solved the large-data-volume storage problem, and its HTAP capability provides real-time query analysis while guaranteeing data writes. In this business scenario, OceanBase’s complex-SQL analysis capability is no weaker than Elasticsearch’s. In addition, OceanBase’s online index-adding capability lets the business perform DDL changes at any time. After replacing the original MySQL+Elasticsearch solution with OceanBase, we not only eliminated the Elasticsearch service and hardware but also greatly reduced MySQL hardware costs—saving 50% of machine resources overall—while meeting the business’s query-performance requirements. In Figure 2, the left side is the MySQL+Elasticsearch solution and the right side is the OceanBase solution.

Figure 2: The MySQL+Elasticsearch and OceanBase solutions for data writing and query analysis

Figure 3 shows the online performance of the payment business, with write volume between 50,000 and 70,000 and queries under 10,000. The blue line indicates data deletion, and the green line is the response time of writes and reads.

Figure 3: Online performance of the payment business

Based on OCP, cluster scaling, monitoring, and alerting can be done conveniently, and it manages multiple OceanBase clusters. In addition, the business side loves using ODC, OceanBase’s query platform, to verify whether data was written successfully and whether the written results are correct. Because its interface operations meet daily database-access needs, with rapid version updates and prompt issue handling, hundreds of business users are using it.

III. Experience and Results of Upgrading from OceanBase 3.1 to 4.3

By September 2024, Kuaishou’s data volume rapidly grew to the PB level, and the number of OBServer nodes grew from the original 190-plus to nearly 300, with 9 OceanBase clusters deployed online in total. With data volume soaring, we began upgrading to OceanBase 4.x. Among them, clusters with larger data volumes (20 TB and above) have been upgraded to 4.2 or 4.3, while about half of the smaller clusters (under 10 TB) are still on version 3.x. So what performance improvements does version 4.x offer compared with 3.x?

From Figure 4, it seems the data in version 4.x didn’t grow. But if version 3.x had not been upgraded, with nearly a year of business development, the data volume would have grown by about 1.5 TB and the number of data nodes would have doubled. This demonstrates the magic of version upgrades: OceanBase 4.x can compress data more extremely than 3.x, saving storage space and thereby cutting storage and machine costs.

Figure 4: Comparison of the cluster lists for version 4.x and version 3.x

Previously, we used a certain distributed database to support the payment business. It used range partitioning, with each table auto-splitting; when write volume was large, it couldn’t utilize the performance of all machines, leading to poor performance under heavy traffic. If a traffic peak hit, the business had to throttle to guarantee the stability of underlying queries.

After introducing OceanBase 3.1, using hash partitioning, write performance improved greatly; DDL was faster, at least ensuring no throttling during business traffic peaks. After upgrading to OceanBase 4.3, the cost benefits improved further, and complex queries became faster—basically completing within 10 ms. Our payment business also has some AP query needs; when using OceanBase 3.1, there was only row-based storage and the business had to tolerate a certain query latency, whereas OceanBase 4.3’s hybrid row-column storage makes queries more real-time, with write latency under 1 ms.

Figure 5 shows the online performance of the payment business after upgrading to OceanBase 4.x.

Figure 5: Online performance of the payment business after upgrading to OceanBase 4.x

In addition, while using OceanBase 3.x, the business staff hoped to optimize imperfect features—for example, non-partitioned tables. When business volume is small, a single table is small and needs no partitioning; but as business volume grows, a single-partition table might max out a single CPU, or disk usage might fill up, causing a single replica to grow huge. At this point, OceanBase 3.x didn’t support converting a single partition into multiple partitions, while 4.x can do this. At the same time, the business staff hoped for faster database query speeds, and OceanBase 4.3’s hybrid row-column storage can greatly improve the performance of complex queries.

Besides solving business needs, an important factor in our OceanBase version upgrades is keeping up with version iterations to accumulate operations experience and avoid falling too far behind on the business version.

In short, after upgrading to OceanBase 4.x, Kuaishou has lower costs and faster queries. Take the payment gateway as an example: in version 3.1.x, this cluster had 65 nodes and was the largest cluster in the online environment. Before the upgrade, the data volume was 450 TB; after the upgrade, the machine scale shrank to 45, and the data volume was compressed to 330 TB. Machine costs dropped by 31%, and the data volume was compressed by about 27% compared with before.

In some TP+AP scenarios, we initially used OceanBase 3.1 to replace MySQL and meet the business’s HTAP needs. After upgrading to OceanBase 4.3, it became more stable, with higher performance and faster analysis. Also, in OceanBase 3.x, we needed to synchronize OceanBase data downstream to integrate with the big-data ecosystem, but this was inconvenient because Binlog wasn’t supported. In OceanBase 4.2, Binlog became compatible with MySQL Binlog, and this problem was solved.

At the same time, we also felt the iterative upgrades on OceanBase’s ecosystem-tools side. For example, before 2022, OCP, ODC, and others were prone to problems with upgrades and scaling; now we use a 12-machine OCP cluster to operate and manage 9 clusters with no scaling, monitoring, or alerting issues, and when OCP diagnoses a problem it can resolve it automatically without manual intervention.

IV. Six Major Benefits of Using OceanBase

From the two core business scenarios above, you can see that after using OceanBase, Kuaishou’s gains have been significant. In summary, they include the following six points.

(1) OceanBase is highly compatible with the MySQL engine, greatly lowering the barrier to development and use. Business staff can continue using OceanBase in the MySQL way without changing their habits. At the same time, because OceanBase is compatible with the MySQL protocol and syntax, data migration can be done smoothly, greatly reducing the cost of business migration and modification.

(2) Operations are more efficient and convenient; a single cluster can replace 300+ MySQL environments, significantly reducing operations-management costs and improving management efficiency.

(3) Data-synchronization performance improved; the response latency from upstream writes to the downstream OceanBase is smaller, data sync is faster, and latency was reduced by 3/4.

(4) OceanBase’s three-data-center-in-one-city deployment architecture achieves RPO=0 and RTO<8s disaster-recovery capability. Meanwhile, a read-only Zone can be added in another region to provide local-read service, improving query efficiency. Same-city disaster recovery and local reads provide dual assurance of stability and performance for the business.

(5) OceanBase has flexible resource-scaling capabilities, allowing linear expansion of compute and storage as the business grows, supporting massive data storage and computation and meeting future business-growth requirements.

(6) Compared with the traditional centralized database MySQL, OceanBase’s extreme compression at the storage layer effectively reduces the hardware cost of using databases—for example, it helped us save 50 machines.

V. Future Plans

In the future, we hope to continue deepening our community collaboration with OceanBase, unifying versions and leveraging OceanBase’s notable advantages in storage and performance in new versions. At the same time, building on OceanBase’s compatibility with the MySQL Binlog format, we’ll integrate with more downstream ecosystems and onboard more business. We will also participate in OceanBase code contributions and take part deeply in community co-building.

Finally, we’d like to recommend the WeChat account of Lao Ji, the head of OceanBase open source: “Lao Ji’s Tech Talk.” It continuously publishes all kinds of technical content related to #Database, #AI, and #Tech Architecture. If you’re interested, feel free to follow!

“Lao Ji’s Tech Talk” not only hopes to keep bringing you valuable technical sharing, but also hopes to contribute to the open-source community together with everyone. If you recognize the value of the OceanBase open-source community, light up a little star ✨! Every Star you give is the motivation behind our efforts.