70% Performance Boost — CR Vanguard Upgrades Its Core Database to OceanBase

This article is excerpted from the e-book “Case Studies of OceanBase Community Edition in Pan-Internet Scenarios”. Click the link to get the full version.

From Performance to Scalability: The Database Challenges Facing CR Vanguard

1. About CR Vanguard and Wanjia Digital

CR Vanguard is a leading retail chain under China Resources Group, with operations spanning mainland China and the Hong Kong market. Facing CR Vanguard’s many business needs and its interconnected operating environment, the group urgently needed to tighten the coupling between business lines so it could keep pace with the rapid growth of online, offline, logistics, finance, and other domains. Against this backdrop, CR Vanguard established its in-house IT subsidiary — Wanjia Digital Commercial Data Co., Ltd. The company focuses on retail: while serving CR Vanguard, it also operates on a market basis, providing end-to-end solutions and operations services for the core business systems of retailers and their ecosystems.

Figure 1: Overview of CR Vanguard Group

With the rapid advance of information technology and digital transformation, the database — as the cornerstone of data management and storage — plays an ever more critical role. CR Vanguard set out to provide an efficient, reliable, and secure data management solution for the enterprise through a digital database upgrade and the application of innovative, intelligent technologies. To that end, Wanjia Digital actively responded to the information-security strategic plans of the nation, the group, and CR Vanguard itself. By bringing in a self-developed database system, the company aimed to provide continuous support for key business operations, run them intelligently, and improve the operational efficiency of its business systems — thereby raising the quality of service to end consumers and creating an efficient “cost reduction — efficiency gain — risk compliance” cycle. This helps Wanjia adapt to a complex, ever-changing market and pursue sustainable business growth, giving the company an edge in fierce market competition.

2. The Status Quo of Traditional Databases and Their Pain Points

1. The Status Quo of Traditional Databases

Traditional database systems such as MySQL and Oracle have played an important role in data storage and processing. However, with the spread of the internet and mobile devices, data volumes have grown explosively. To cope with the surge in data, many enterprises adopt scaling architectures to improve and extend the performance and capacity of traditional databases. Among these, MySQL is a very popular choice for scale-out architectures. There are three common MySQL architectures. The first is the primary-replica replication architecture, which improves performance and capacity by replicating data to one or more replica servers.

Figure 2: MySQL primary-replica replication architecture

The second is the sharding architecture, which spreads data across multiple database instances to achieve horizontal scaling.

Figure 3: MySQL sharding architecture

The third is the read-write splitting architecture, which routes read and write operations to different database instances to improve concurrency.

Figure 4: MySQL read-write splitting architecture

MySQL scale-out architectures typically appear as the business expands. When the three architectures above can no longer keep the business stable, the performance problem can be addressed by combining sharding with read-write splitting.

Figure 5: MySQL scale-out architecture

It’s worth noting that as the cluster grows and the architecture’s complexity rises, the operations and development costs climb sharply, bringing a host of problems. A classic example is the “barrel effect,” where a single “short stave” drags down the stability of the entire system.

2. The Pain Points of Traditional Databases in CR Vanguard’s Business

CR Vanguard opened its first store in Hong Kong in 1984, and now has a 40-year history. Over the course of its retail business growth, the group’s systems evolved from a simple early inventory-and-sales system into a coordinated suite of systems today — logistics and supply chain, membership, online business, and more. But the parallel operation of old and new systems accumulated a wide range of issues, which, intertwined with the inherent limitations of traditional databases, can be grouped as follows.

(1) Performance bottlenecks: Traditional databases hit limits under heavy concurrency. For example, in a monitoring system with a few hundred hosts and 10,000–20,000 monitoring items, the backend database still has plenty of headroom. But when scaling up to tens of thousands of hosts and 500,000–1,000,000 monitoring items, MySQL exhibits heavy data lag — sometimes exceeding 30 minutes, at which point the monitoring data is no longer meaningful.

(2) Scalability limits: Traditional databases face certain scalability limits and struggle to meet ever-growing data demands. On the hardware side, constrained by CPU, memory, and storage, growing data volumes degrade database performance and increase response times. To keep the database healthy, we have to constantly monitor data volume and periodically purge data. This is in effect a compromise on database performance, because for the business the ideal state is keeping as much data queryable and available as possible. At the same time, as the system grows, code complexity increases, making maintenance harder. Expanding the system requires not only heavy capital investment but also substantial human resources, putting enormous cost pressure on the enterprise — a key contradiction that limits its development.

(3) Insufficient high availability: When faced with failures, traditional databases often struggle to guarantee high availability, affecting business continuity. CR Vanguard had made every preparation for cluster high availability, with traditional and newer designs such as primary-replica architecture, multiple replicas, sharding, and remote disaster recovery. But under extreme conditions, the RTO still ran from 10 minutes to half an hour. In some cases a human had to judge whether a failover was necessary; for database operations with higher risk levels and greater business importance, the whole team had to analyze and decide together.

(4) Multiple systems running in parallel: As CR Vanguard’s business evolved through different stages, its systems shifted from off-the-shelf packages to custom Java development, using a variety of databases such as IBM Informix, Oracle, and MySQL. The differences between databases were so large that different teams had to communicate frequently to define a unified interaction protocol and ensure smooth data flow — consuming a great deal of time and effort. Meanwhile, because the systems differed in technical architecture, data formats, and interface specifications, custom adapters and middleware had to be developed, raising integration difficulty. As business volume grew, resource consumption across business systems intensified, adding cost pressure; teams had to allocate hardware resources and network bandwidth sensibly, optimize system configurations, reduce resource contention, and improve overall performance.

(5) Degraded user experience: System users fall into internal and external groups, and there were three overarching experience pain points. First, different users have different needs in terms of features, interface, and operation, making it difficult to satisfy individual requirements. Second, users demand faster and faster response times, especially under high concurrency — but ensuring fast response is a challenge. Third, usability: complex features can make the system hard to operate, degrading the user experience.

(6) Maintenance difficulties: On one hand, traditional databases require heavy investment of people and resources to maintain and manage. On the other, because various issues crop up as data moves between systems, fault points are hard to pinpoint and require multi-step troubleshooting, lengthening repair times; any link in the chain can easily become a bottleneck that hurts overall performance, requiring resource configuration optimization. Operations staff use monitoring tools to locate problems, but certain monitoring blind spots require a more powerful operations team to perform real-time analysis — a real test of the monitoring team’s understanding of the business.

(7) Security concerns: Security is usually a key focus for traditional databases, requiring multiple measures to keep data safe. For backup and recovery, for instance, MySQL lacks an end-to-end solution, leading to incomplete backups, lost or corrupted backup files, and long recovery times. As another example, in a middleware-based MySQL architecture, auditing is difficult: tracking user access and data modifications or queries is tricky, and it’s usually hard to trace who originated a historical problem SQL. As mentioned above, the complex data paths also increase exposure to attacks.

Database Selection: Benchmarking MySQL Against OceanBase

Given all these pain points and challenges, the domestically developed databases that have grown popular in recent years came onto CR Vanguard’s radar. When choosing a database, we focused mainly on the following points.

• Meeting independent-R&D requirements: a fully self-developed domestic database with independent intellectual property rights, also meeting the compatibility requirements of our in-house systems.
• Compatibility: compatibility with existing systems (databases such as MySQL and Oracle, operating systems such as CentOS and RedHat), including protocols, data formats, and APIs.
• High availability: node-failure handling, disaster-recovery capability, and data scalability — node expansion, data partitioning, load balancing, and the like.
• Performance: read/write speed, concurrency handling, and data processing capability.
• Cost: migration cost, development cost, host and storage cost, and so on.
• Business coupling: coupling with various business workloads across scenarios, manifested in application adaptation and in SQL performance jitter across different scenarios.

Based on these principles, the Wanjia Digital technical team selected two domestic databases for benchmark and stress testing, observing how each performed in terms of performance, cost, and compatibility.

1. Benchmark Performance Comparison

Because the architectures of the databases under comparison differ, to ensure fairness we used total CPU and total memory as the baseline parameters rather than judging by host count. The host specification: 64 cores total CPU and 256 GB total memory. The results are shown in Figures 6 and 7.

Figure 6: Performance comparison between OceanBase and a certain distributed database

From the test results, compared with the other distributed database, OceanBase delivered QPS at roughly 200% or above across concurrency levels in the oltp_update_index scenario. In the oltp_read_only, oltp_read_write, oltp_update_non_index, and oltp_insert scenarios, OceanBase performed better, averaging a 40% QPS improvement across concurrency levels. In the oltp_point_select and oltp_write_only scenarios, the two databases traded the lead across concurrency levels, with roughly comparable overall performance.

Figure 7: Performance comparison details between OceanBase and a certain distributed database

2. Stress Test Comparison

The stress-test environment was the same as the benchmark environment. The results are shown in Figures 8 and 9.

Figure 8: Stress-test comparison results between OceanBase and a certain distributed database

From the business stress-test results, OceanBase performed better: compared with the other distributed database, its write QPS was 2x and its query QPS was 4x — while latency was only 1/4.

Figure 9: Stress-test comparison details between OceanBase and a certain distributed database

Across the various comparisons, OceanBase came out on top. It also makes the most of storage resources and reduces fragmented resources; compared with MySQL it can cut storage costs by about 60%, and by conservative estimates reduce overall costs by 30%. On the other dimensions — compatibility, high availability, and scalability — the two databases were fairly close, as shown in Figure 10.

Figure 10: Comparison of other items between OceanBase and a certain distributed database

After the comparison, CR Vanguard ultimately chose to replace its existing database with the natively distributed database OceanBase.

Adopting OceanBase: 70% Performance Boost and Significant Cost Benefits

1. The OceanBase Migration Process

In June 2022, Wanjia Digital brought in OceanBase version 3.x and began POC testing and core-system validation. By December 2022, we learned that the standalone-distributed integrated OceanBase 4.0 was about to be released, so we waited for the new version and promptly followed up with system migration testing. After several months of production practice, the results were highly satisfying. In 2024, we migrated a large batch of systems to OceanBase, making it our technical foundation.

Counting both production development of new projects and migration of legacy ones, Wanjia Digital now has fifty to sixty projects running on OceanBase, with more to be built on it in the future.

2. Lessons From the OceanBase Migration

In the early stage of adopting OceanBase, the Wanjia Digital technical team selected one of CR Vanguard’s core business systems as the target for database upgrade and transformation. We used OMS for the database migration, while the original MySQL cluster was a middleware-based, multi-instance sharded cluster.

OceanBase Migration Service (OMS) is a service provided by the OceanBase database that supports data exchange between homogeneous or heterogeneous data sources and OceanBase, with the ability to migrate existing data online and synchronize incremental data in real time. OMS offers a visual, centralized control platform: with simple configuration, you can migrate data in real time. OMS enables low-risk, low-cost, and high-efficiency real-time data migration and synchronization from homogeneous or heterogeneous databases into OceanBase. Its advantages include the following.

• Support for multiple data sources. OMS supports real-time data transfer between OceanBase and many kinds of data sources, such as MySQL and Kafka.
• Online, non-stop migration that’s transparent to business applications. Without stopping the service, you can seamlessly migrate data to OceanBase via OMS. After the application switches to OceanBase, all changes on the OceanBase database are synchronized in real time back to the pre-switch source database.
• High-performance, secure, and reliable data migration. OMS can replicate large volumes of data between heterogeneous IT infrastructures in near real time, with second-level latency. It can therefore be used for data migration, cross-city remote disaster recovery, emergency systems, real-time data synchronization, disaster tolerance, database upgrades and migrations, and more. OMS runs migration and synchronization tasks transparently and without interruption to business applications, while ensuring data integrity and transactional consistency. Full-migration performance can reach 100 MB/s and 200,000 TPS, and data-synchronization performance can reach 50,000 RPS. OMS also provides a highly available deployment architecture for stable, reliable transfer tasks.
• One-stop interaction supporting full-lifecycle data-migration management. From the management console UI, you can create, configure, and monitor migration tasks — interaction is simple.
• Real-time data synchronization that decouples the business. OMS supports real-time synchronization between OceanBase’s two tenant types and self-built Kafka or RocketMQ, and can be used for building real-time data warehouses, query offloading, reporting, and similar scenarios.
• Multiple data validations. OMS provides several data-consistency validation methods to guarantee data quality more comprehensively, with less time and higher efficiency. It also surfaces the differing rows and offers a quick path to correction.

We first conducted a migration assessment, evaluating the existing database’s performance, availability, and scalability and defining migration goals and a plan. Next, based on the assessment results, we drew up a detailed migration plan covering data backup, data conversion, node migration, and testing. Finally, after completing the migration and consolidation, we needed to monitor and maintain the new system over the long term to ensure it ran stably and met business needs.

1. Migration Assessment

This system used a middleware-based MySQL cluster with read-write splitting and sharding, as shown in Figure 11.

Figure 11: The original MySQL architecture of a certain system

The database used 5 instances, each with 10 shard databases, for 50 shard databases total; each instance had two replicas, merged via middleware into a single logical database with read-write splitting.

During assessment, we first estimated system performance — the actual production figure was 15 TB of data, with concurrency estimated at 3,000. High-frequency SQL was captured via backend monitoring as the Top 50. Next we assessed availability and scalability: the middleware-based MySQL architecture had already greatly improved scalability, and cluster capacity and compute could be expanded quickly by adding new MySQL clusters and middleware routing configuration — but a brief downtime was still required during cluster expansion. Third, we assessed post-migration data volume: about 6 TB after migration was expected, and OceanBase would need at least a 7 TB data disk to keep the data space healthy. Fourth, we loaded test data and ran high-frequency SQL stress tests to validate the database’s capacity. Fifth, we analyzed the system’s related business workloads, investigating each one in detail and validating them one by one.

Through this simulated assessment, we validated the feasibility of the new system and produced preliminary estimates of the OceanBase resources needed — CPU, memory, disk, and so on.

2. Migration Plan

For a 24/7 business in steady-state operation, the key difficulty was achieving a smooth, business-transparent migration. To that end, the Wanjia Digital technical team designed a clever step-by-step process to migrate the database in stages. Following a read-write-splitting strategy, we migrated read workloads first and write workloads later, ensuring the system transitioned to the new platform stably and smoothly — keeping users as unaware as possible.

Figure 12: The migration process of a certain system

Migrating a MySQL sharded cluster to OceanBase requires considering database consolidation — how to merge databases and tables is a migration challenge. Each large table had to be checked and validated, the uniqueness of every row confirmed, and an appropriate partition key chosen for the big tables to ensure optimal performance of the hot SQL. We also had to ensure historical data could be unloaded quickly, so that operational cleanup stayed simple and efficient. To that end, we analyzed and validated the database in detail and settled on a migration-and-transformation plan.

The primary key of this business’s large tables used the Snowflake algorithm, which only guarantees uniqueness within a single DB; across multiple DBs there is a tiny probability of primary-key collisions. For a small table, this can be fixed by querying and excluding primary keys; but for a large table with billions or tens of billions of rows, excluding by primary key is infeasible and would consume enormous resources. We therefore reworked the primary key: we abandoned the existing Snowflake-based key, added an auto-increment primary key, and set a range of starting values for the auto-increment chains across all DBs, as shown in Figure 13. This ensures the database’s primary keys won’t collide within a certain window — and within that window, we needed to consolidate and migrate as quickly as possible.

Figure 13: Migration method for large business tables

In the cutover plan, we modified the read and write applications to support dual data sources, set sensible rules, and migrated the business in batches throughout the process until migration was complete, as shown in Figure 14.

Figure 14: The migration cutover plan

The benefit of this approach is that it minimizes the migration risk for the entire business. The first step uses only a small fraction of traffic for migration testing; once confirmed problem-free, subsequent steps proceed, migrating read and write workloads one by one. Each migration step completes within 10 seconds, with minimal impact on the business.

3. Real-Time Stream Processing

In handling the data streams associated with the database business, Kafka plays a crucial role. Kafka supports many storage formats, among which Canal and Shareplex are widely used in the industry. These formats are broadly supported in OceanBase, making data flow more stable and reliable and greatly reducing migration development costs. OMS provides comprehensive support for these formats, smoothing the data-flow process so it’s no longer a thorny problem.

The Debezium format was the unified format Wanjia Digital adopted as it advanced its Flink ecosystem, but OMS V3 did not support it at the time. Retrofitting for it would have touched a great many upstream and downstream links and entailed a huge estimated effort. After discussion, the OceanBase-OMS development team carried out the corresponding development and adaptation for the Debezium format, ensuring our project proceeded smoothly. We’re grateful for the OceanBase technical team’s wholehearted support.

In the past, we captured cluster data changes in real time based on BinLog changes using kafka-connector. We had to listen to logs on each MySQL node, which was complex and hard to maintain. Task scheduling couldn’t guarantee real-time delivery, push latency was high, and under heavy load there were untimely pushes and poor reliability, as shown in Figure 15.

Figure 15: The original task scheduling model

After migrating to OceanBase, we adopted real-time stream processing based on OMS+Flink scheduling, replacing the high-latency MySQL+Kafka task-scheduling model, as shown in Figure 16.

Figure 16: Stream data processing based on OMS+Flink

OMS provides a visual, centralized control platform with UI-based operations, supports point-in-time synchronization, and has low maintenance cost. We use Flink streams to implement real-time data-processing logic, pushing processed data to the target system in real time via Flink’s StreamSink and TableSink, ensuring the target system can receive and process real-time data. Its checkpoint mechanism enables continuous task checking and recovery: by periodically checking checkpoint state during task execution, tasks can recover to a consistent state in the event of an exception.

The OMS+Flink solution keeps operations simple and data real-time: the entire data flow completes within 2 seconds, ensuring every data consumption is pushed to every user accurately, in real time, and reliably.

4. Migration and Consolidation Results

After thorough preparation and validation, we successfully migrated and consolidated one of Wanjia’s core systems onto the OceanBase database platform. Throughout the migration, users were unaware and the business systems ran stably. Production validation showed that OceanBase improved performance by about 70% over the original system and cut costs by about 50% — this migration-and-consolidation project was a complete success.

3. OceanBase Optimization Case Study

Leveraging OceanBase’s rich ecosystem, we also greatly simplified monitoring and operations — improving both the granularity of operations management and operational efficiency.

Take the performance tuning with OCP and ODC as an example.

1. The Problem Appears

Early one morning, business staff reported that after a program release, a newly added business requirement executed very slowly. The scenario was stable in the UAT environment, but after going live its efficiency dropped several-fold, causing business documents to pile up and unable to be processed in real time.

2. Problem Analysis

OCP: Using OCP’s SQL diagnostics, we found no obviously slow SQL in the TopSQL at that point in time. Talking to development, we learned this was a high-frequency SQL scenario, where even a few milliseconds of extra average response time affects the business; we quickly pinpointed the problem SQL and found no index-related issues. ODC: We ran the problem SQL in ODC and examined its actual execution plan, locating the issue — the SQL contained many RPC calls, as shown in Figure 17.

Figure 17: The ODC performance problem

3. Problem Resolution

Creating a table group eliminated the RPC calls. Figure 18 shows the basic info of the SQL execution plan after creating the table group — clearly, there are no more RPC calls.

Figure 18: How the ODC performance problem was resolved

4. The Benefits of the OceanBase Migration

We summarize the concrete benefits of the migration in the following five points.

• Cost savings: With high-compression storage technology, the original storage footprint shrank by about 60% after migration, hardware costs were cut by 50%, and overall business costs dropped by around 25%.
• Higher effective resource utilization: Aggregating multiple instances within a cluster, with multi-tenant resource isolation, reduces resource fragmentation and makes full use of resources.
• Improved business resilience and development efficiency: Optimizing the business architecture and unifying the tech stack lowered development difficulty, raised development efficiency, and strengthened business stability and scalability. Whereas the entire operations team used to be tied up shoring up the MySQL cluster, life is far easier now.
• Performance improvement: We removed the performance bottlenecks of the previous architecture, boosting system performance by 70% while also supporting real-time report queries — reducing data-pipeline development and maintenance work, with support for hybrid analytical scenarios.
• Higher operational efficiency: Platform-based database management supports white-screen DBA operations, improving operational efficiency and reducing the cost of building operations tooling and running operations.

Looking Ahead

In the future, the Wanjia Digital technical team will work to build a complete, standardized database system, strengthen team development, fully leverage its advantages, and optimize resource allocation and the monitoring-and-operations mechanism — pursuing cost reduction, efficiency gains, and sustainable business growth.

Finally, I’d like to recommend the WeChat official account “Lao Ji’s Tech Talk” run by Lao Ji, the head of OceanBase Open Source. It continuously publishes all kinds of technical content related to #Databases, #AI, and #Tech Architecture. If you’re interested, feel free to follow!

“Lao Ji’s Tech Talk” hopes not only to keep bringing you valuable technical content, but also to contribute to the open source community together with everyone. If you appreciate the OceanBase open source community, light up a little star ✨! Every Star you give is fuel for our efforts.