vivo's Domestic Database Technology Reserve: Using OceanBase to Break Through the Storage and Performance Bottlenecks of Large-Scale Data

Author: Du Ting, Head of Storage Operations for vivo Internet

vivo is a technology company centered on smart terminals and intelligent services, serving 500 million+ users worldwide. The company is organized into multiple business lines, and over the past two years its internet business completed an upgrade of its underlying database solution to better support business growth. After adopting OceanBase, the business resolved the storage and performance bottlenecks of MySQL at large data scale: high-concurrency data update efficiency improved by 60%, complex query performance improved by 80%, and storage cost dropped by 50%.

The Technical Architecture That Once Kept Thousands of MySQL Clusters Running Stably Hits a Bottleneck

For over a decade, vivo had used MySQL as its underlying database. As shown in Figure 1, vivo’s MySQL clusters used a two-layer asynchronous replication architecture: the upper layer used a self-developed proxy service, Proxy, to provide capabilities such as encryption/decryption, auditing, and connection pooling, with fully automated operations management of Proxy through the operations platform — enabling health checks, automatic failover, automatic deployment, and more; the bottom-layer Agent handled failover, achieving multi-node high-availability deployment via Raft; and the Zookeeper in the middle synchronized MySQL metadata, ensuring data consistency across the MySQL clusters.

Figure 1: The two-layer asynchronous replication architecture of the MySQL clusters

This architecture and its management approach offered customizable high-availability failover and health-check capabilities, and had supported the stable operation of thousands of MySQL clusters. However, over the course of business growth, the solution gradually ran into bottlenecks in data storage and read/write performance.

Data storage: MySQL used a single-machine, multi-instance deployment architecture, with data stored on physical machines. Once the business grew to a certain scale, storage capacity was limited by single-machine disk, requiring sharding. But sharding scaled poorly, steadily driving up business, R&D, and operations costs. The total storage scale had already reached the PB level, facing high storage costs.

Read/write performance: vivo’s huge business scale involves a large number of complex queries, but MySQL’s complex query performance was low and could not meet business needs. Moreover, efficiency was low in large-batch, high-concurrency write scenarios, and replica lag was significant for upstream/downstream synchronization and offline queries, affecting the related businesses.

For the reasons above, vivo began researching database products that could solve its current problems. After research and comparison, it ultimately chose OceanBase — which offers horizontal scaling, resource isolation, low cost, strong disaster recovery, high MySQL compatibility, HTAP, and other features — and introduced it into the business environment.

Specifically, starting in the fourth quarter of 2023, the storage operations team kicked off systematic research, evaluating deployment architecture, features, performance metrics, application scenarios, success stories, security standards, and more. After the research was complete and following a department-level review, it formally decided to introduce OceanBase for platform construction (see Figure 2).

Figure 2: The OceanBase adoption process

In 2024, the project formally entered implementation:

• Completed standard-setting and carried out performance stress testing, high-availability verification, backup-and-recovery, and monitoring-and-alerting verification to ensure the conditions for going live were met;
• Built the supporting ecosystem in parallel, deploying core components such as OCP, OMS, and OBLoproxy, and integrating the downstream Binlog consumption pipeline to form a complete upstream-downstream ecosystem.

In July 2024, the first edge business completed its pilot onboarding. At the same time, the storage operations team gradually integrated OceanBase’s core capabilities (metadata management, data query, data change, cluster management) into the existing operations platform. By the end of 2024, over ten businesses and dozens of clusters had been put into production, with the scale continuing to expand.

Applying OceanBase Across Four Technical Scenarios, with Considerable Returns

Currently, in vivo’s internet business line, four database scenarios already use OceanBase, and the operations system has begun gradually migrating to OceanBase.

What Problems Did OceanBase Solve?

1. Data Archiving

Because MySQL’s existing data was huge and some businesses’ historical data no longer needed online access, cold-data archiving at this stage used the TokuDB compression solution, with a compression ratio of about 3–4x, mounted on large-capacity disk storage. As data kept growing and disk capacity ran short, the storage operations team had split the archive clusters multiple times, but the splitting operations had high operations cost and poor scalability. Introducing OceanBase — leveraging its data compression and distributed scaling capabilities — allowed independent clusters to be deployed at business granularity, achieving horizontal scaling and online compression. On one hand, this solved the problem of having to do MySQL-based sharding on the business side; on the other, it reduced business complexity and operations cost, and significantly reduced the storage footprint of historical data.

2. Horizontal Sharding

In the early days, the business team did not design sharding. Once the cluster grew and single-instance capacity became constrained, traditional MySQL required manual sharding — a large, lengthy effort. After introducing OceanBase, thanks to its native support for partitioned tables and automatic partitioning, data could be horizontally split automatically, reducing the investment in application refactoring and labor. And its multi-tenant resource isolation reduced the resource contention of the original single-machine, multi-instance architecture.

3. High Concurrency

For some extremely high-concurrency businesses, MySQL was prone to write lag and throughput drops, whereas OceanBase’s multi-node parallel processing can effectively spread out write pressure, guaranteeing low latency and high throughput.

4. HTAP Mixed Workloads

vivo’s internet business line has a large number of hybrid workloads that involve both transactions and real-time analytics. After introducing OceanBase, both TP and AP queries can be served, significantly simplifying the overall architecture.

Building an In-House Operations Platform for a Unified Operations System

In the early days of using OceanBase, vivo internet’s operations relied mainly on open-source platforms, such as OceanBase’s operations management platform OCP. OCP offers complete full-lifecycle data management — resource management, performance monitoring, cluster management, tenant management, backup and recovery — meeting basic operations needs. But vivo internet had its own in-house DaaS and PaaS platforms, so the storage operations team launched a plan to integrate OCP’s capabilities into the in-house platform, and has so far delivered some essential capabilities for both business R&D and DBA operations.

• For business R&D: implemented capabilities such as application detail queries, data queries, and data changes.
• For DBA operations: implemented management capabilities such as instance management, domain management, and account management.

As shown in Figure 3, the features with a blue background and white text are those already built; the features with a blue background and black text are under construction; and the features with a white background are planned for later. Through platform integration, a unified internal in-house operations platform will ultimately take shape, fully replacing the open-source platforms.

Figure 3: Operations platform build progress

What Returns Did OceanBase Bring?

High-Concurrency Scenarios: Over 60% Performance Improvement

One business had such large MySQL batch writes — 400,000 rows of data operations per second — that MySQL primary-replica lag stayed persistently high, posing a data-loss risk while also preventing the offline business from querying the data. After this business switched to OceanBase, the lag problem was resolved, achieving real-time data synchronization with the upstream and downstream ecosystem products. At the same time, thanks to the distributed architecture, the business’s write performance improved, and overall update performance improved by more than 60% (see Figure 4).

Figure 4: Performance improvement after a business switched to OceanBase

Large-Scale Data Change Efficiency Doubled, Complex Query Performance Up 80%

The business archive originally used the TokuDB storage engine, with data compressible 4x — which eased the physical-machine disk-capacity problem, but with 40 billion+ rows of data, it could not support DDL changes. After migrating to OceanBase, not only was the data storage compression ratio basically on par with TokuDB, but it could perform DDL on a large table with 40 billion+ rows in just 2 hours and 18 minutes — many times faster than TokuDB. OceanBase not only solved the problem that ultra-large-scale MySQL data could not be changed, but also enabled changes synchronized with the original business tables, thoroughly resolving the original challenge of online changes to large-scale archived data.

In addition, after large-scale data-query businesses switched to OceanBase, overall latency dropped by more than 30%, TP and AP coexistence for complex business queries was achieved, and performance improved by 80%.

Hardware, Operations, and Other Costs Dropped Significantly

Thanks to OceanBase’s high-compression characteristics, hundreds of TB of data were compressed by 60%, saving 50% on storage cost and significantly reducing hardware cost. Furthermore, compared with the operations effort spent on sharding, OceanBase’s native support for partitioned tables, automatic partitioning, and online scaling greatly freed up operations staff, substantially lowering both operations cost and business R&D development cost.

Building a Domestic Tech Stack Reserve and Improving the Operations Ecosystem

Going forward, vivo internet plans to roll out OceanBase along the following four lines and gradually improve its operations ecosystem and system.

1. Incorporate distributed relational databases into the tech stack for the long term.

The company’s existing database products already cover many types, such as MySQL, TiDB, Redis, Elasticsearch, and KV storage, but lack a distributed relational database proven at large scale. After evaluation, OceanBase can serve as a distributed-database complement for handling massive data and high-concurrency scenarios, and is included in the long-term product roadmap.

2. A reserve of domestic database technology.

Against a backdrop of growing uncertainty in international supply chains, open-source products like MySQL and Redis carry potential risk of being restricted. OceanBase is a domestically self-developed database and can serve as a key backup tech stack, with continued investment to refine its capabilities and safeguard business continuity.

3. Building the operations ecosystem.

Combining OceanBase’s ecosystem tooling, vivo will improve the upstream/downstream operations tooling to build a complete upstream-downstream ecosystem covering all stages — synchronization, backup, monitoring, auditing, data subscription, and so on — ensuring the business can onboard seamlessly.

4. Building the operations system.

vivo will continue building out the operations system, gradually integrating OceanBase’s metadata management, cluster management, monitoring and alerting, and automated operations into the existing unified operations platform, achieving consistency in style and experience with the established system.