Migrating a Distributed Database to OceanBase — A Smooth Migration Built on NetEase Cloud Music's In-house CDC Service

Editor’s note: As a music product focused on discovery and sharing, NetEase Cloud Music led the industry from the “media-player era” into the “online community era,” reshaping the way people experience music in daily life. Behind such large-scale business data, what technical architecture is doing the heavy lifting? This article shares the optimization lessons NetEase Cloud Music learned while migrating its PB-scale sharded architecture to a native distributed database architecture.

Author: Lyu Yating, Senior Platform Development Engineer at NetEase Cloud Music

Migration Background and Core Challenges

The State of the Distributed Database DDB

The database most widely used at NetEase Cloud Music is its in-house distributed DDB. DDB is not a native distributed database; rather, it is a middleware built on top of MySQL storage nodes, with the following characteristics:

• Sharding middleware: two-level mapping with custom hashing. It uses MySQL servers as the underlying storage nodes and implements automatic routing.
• Standardization: SQL compatibility, globally unique auto-increment IDs, and compatibility with the MySQL wire protocol.
• Distributed transactions: data consistency guaranteed by the 2PC protocol.
• Elastic scaling: online data migration (full and incremental) via the NDC tool, with support for resumable transfers.
• High availability: common MySQL primary-replica replication for high availability and read-write splitting, with support for unitization.
• SQL statistics: support for SQL patterns, SQL frequency, slow SQL, and multi-dimensional QPS statistics.

As NetEase Cloud Music’s primary OLTP solution, DDB holds a large data volume. Its data-flow architecture is shown in Figure 1. As the figure shows, the DBI JAR package serves as the access interface providing services to upper-layer applications. The overall data-processing flow is as follows: when an upper-layer application issues SQL, DBI parses it to generate a syntax tree; the execution plan generated from the syntax tree is then dispatched by the executor directly to the underlying MySQL. Results from multiple MySQL nodes are aggregated at DBI for processing and finally returned to the upper-layer application.

Figure 1 DDB architecture

In addition to the DBI JAR package (similar to an SDK approach), we also offer a QS (Query Server) approach, which can be thought of as a proxy. QS is compatible with the MySQL protocol and can be deployed separately as a proxy, effectively acting as a MySQL instance. It supports both command-line and application-interface access, allowing applications to access QS just as they would access MySQL.

In Figure 1, users can drive DDL processing and metadata distribution and synchronization through the control flow. The Master node synchronizes metadata to the MetaStore (the metadata database) and then notifies every DBI application layer of metadata changes. While the DBI-and-application integration approach shortens the overall data path, it also introduces some operational issues. For example, when there are many DBIs and the underlying nodes undergo large-scale switchovers or changes, the overall operation is time-consuming; and if some DBIs fail to detect changes in time, the application’s reads or writes may be affected.

As a battle-tested database, DDB also has some pain points in practice.

• PB-scale storage with no efficient compression. The underlying MySQL nodes currently run version 5.7, which does not support efficient compression. Typically, compute resources are still abundant while storage hits a bottleneck first, leaving resource utilization extremely low.
• Complex and time-consuming scaling. DBAs spend more than 20% of their effort each year on scaling, which is very costly in terms of human resources.
• Minute-level failover. Although we have optimized failover many times, we can only reach minute-level granularity.
• Update operations cause replica lag, with a bottleneck in replication performance. Under MySQL primary-replica replication, large batches of updates expose a bottleneck in MySQL Binlog performance, leading to replication lag — unfriendly to data-sensitive workloads.
• Lack of continued maintenance. Because version iteration is very costly, it is difficult to perform subsequent version changes.

Based on these pain points, we decided to switch DDB to the native distributed database OceanBase.

Why Migrate from DDB to OceanBase?

OceanBase is a native distributed database (see Figure 2) built on a standalone-distributed integrated architecture. It has achieved key technical breakthroughs in elastic scaling, high availability, multi-active disaster recovery, storage engines, distributed transactions, HTAP, compatibility with multiple mainstream databases, and multi-tenancy, and it has been battle-tested in complex and demanding financial core business scenarios.

• Resource cost: in our tests, storage space can be reduced by at least 1/4.
• High availability: automatic failure recovery with RTO < 8 seconds.
• Efficient replication: efficient replication built on the Paxos protocol at the lower layer.
• Compatibility and stability: compatible with the MySQL protocol, with low migration cost.
• Operational cost: also thanks to MySQL protocol compatibility, operational cost is low.
• Community dividend: a healthy, highly active community ecosystem.

Figure 2 OceanBase product form

Based on a preliminary assessment of its product features and application cases, we believed OceanBase could solve the problems we faced with DDB.

Core Challenges in Migrating to OceanBase

However, since DDB and OceanBase are two database products developed by different companies with significantly different architectures, we encountered four challenges during migration.

The heterogeneous data synchronization challenge. The internal data-transfer tools of the two databases are mutually incompatible and cannot synchronize data directly. We therefore had to convert DDB’s sharding logic into OceanBase partitions while also guaranteeing cross-node consistency for distributed transactions.

The business-transparent migration challenge. The migration had to be transparent to the business. When DDL changes occur on the source side or MySQL nodes undergo primary-replica switchovers, online switching must not block the migration. The migration also placed high demands on the high availability of the migration components themselves, requiring zero-downtime upgrades — something OceanBase’s migration tool, OMS, does not yet support (support is planned for the future).

The reverse-migration efficiency challenge. Forward migration from DDB to OceanBase can reach speeds of up to GB/s, but reverse migration has low synchronization efficiency in large-table and high-traffic scenarios. It also needs to support fast comparison of TB-scale data.

The operational efficiency challenge. Thousands of synchronization tasks may be created during the entire migration. Relying on manual operations would be highly inefficient.

Migration Architecture and Adaptation Work

Because data migration involves two long-running synchronization tasks — forward and reverse synchronization — with strict requirements on link stability and low latency, we began a rigorous effort on migration architecture and adaptation.

The In-house CDC Service — NDC Architecture

First, our in-house CDC service, the NDC architecture, is similar to DTS. It is a service system that supports end-to-end full migration, incremental migration, and data subscription.

• NDC can be roughly divided into three parts: the source-side system, the NDC cluster, and the target-side system. The source side currently mainly supports NetEase’s internal OLTP-type databases and OceanBase; the target side supports OLTP and OLAP databases as well as message queues such as Kafka.
• NDC pulls full or incremental data from the source system, transforms and filters it, and writes it into the target system.
• NDC supports full and incremental synchronization/migration, and provides multiple verification modes — structure verification, full-data verification, and fast inconsistency re-checks — to ensure consistency between source and target. As shown in Figure 3, within NDC: the Dashboard is the application entry point; the Center handles metadata management; and the Engine nodes are stateless, responsible for pulling source-side information such as Binlog, parsing the data, and synchronizing it to the target database.

Figure 3 NDC architecture

The NDC Migration Plan

Migrating data with NDC also involves forward and reverse synchronization. Forward synchronization refers to DDB-to-OceanBase, and reverse synchronization refers to OceanBase-to-DDB. Because some rollback scenarios are involved, reverse synchronization must also be stable and safe.

1. Forward Synchronization

Forward synchronization mainly supports the following capabilities:

• Parallel Binlog parsing. Pulling and parsing in parallel on a per-MySQL basis speeds up parsing. The parallel pull-and-parse processes for each MySQL are arbitrary and stateless, so they can be distributed across multiple machines, greatly improving resource utilization.
• A pull-once, push-many model — parse once, use multiple times — to reduce intrusion on the source side. Establishing a MySQL Dump link on the upstream is perceptible to the source side; if a MySQL database or DDB table has many subscription needs, dozens of connections would have to be established on the source database. Whether on the primary or the replica, this causes impact and intrusion. The pull-once, push-many model therefore pulls and parses only once but allows reuse many times, reducing intrusion on the source side.
• Support for intra-table concurrent writes, with concurrent writes for different index data. Within a single MySQL, concurrency can be achieved across tables and indexes. The final per-table concurrency is therefore the number of MySQL instances over which the table is distributed multiplied by the concurrency set within each MySQL subtask, yielding very high actual concurrency. At the same time, resources on different machines can be used to run different MySQL-level tasks for the same table.
• Automatic response to upstream switchovers, with automatic detection of primary-replica switchovers and node changes. NDC can automatically detect upstream MySQL primary-replica switchovers and node changes without manual intervention.

Figure 4 Forward synchronization plan

2. Reverse Synchronization: Binlog or CDC?

Binlog mode was the synchronization mode we supported earliest — simple, easy to use, highly stable, and highly compatible. The overall flow is roughly as follows:

• The BC module pulls Clog and converts it into Binlog format, then writes it into Binlog files.
• MySQL Binlog ecosystem tools issue a Binlog subscription request to OBProxy, which forwards it to OBLogProxy.
• After receiving the Binlog subscription request, OBLogProxy starts the BD module, which reads the Binlog files and provides a subscription service — the Binlog Dump service.

In effect, to make things convenient for downstream consumers, we maintained compatibility with MySQL’s Binlog: the Clog produced by OceanBase is converted one extra layer into Binlog. But because the link is lengthened, efficiency is relatively low. Last year, the Binlog parsing mode ran at 25 MB/s, which was unacceptable to us. Later the OceanBase team optimized this figure to 140 MB/s, which has temporarily met business needs.

Figure 5 Binlog synchronization mode

Compared with Binlog mode, CDC mode is simpler because there is no need to convert Clog into Binlog; it pulls Clog directly using OBLogClient. The specific flow is as follows:

• After OBLogClient starts, it sends a message to OBLogProxy, specifying the user, databases and tables, incremental link position, and other information to subscribe to.
• Once OBLogProxy passes authentication, it starts the OBLogReader subprocess and passes the connection and related information to it.
• After OBLogReader starts, it extracts and parses Clog and sends it downstream in a specific data format to complete the data subscription.

Figure 6 CDC synchronization mode

OBCDC or OBLogProxy?

We also investigated OBCDC and OBLogProxy. Both belong to CDC mode, and OBCDC is the continuously iterated version of OBLogProxy.

• OBCDC: provides real-time incremental (transactional) data from the OceanBase database as a dynamic library. It requests the Clog (Redo) logs of each partition from the OceanBase database via RPC.
• OBLogProxy: a proxy for OBCDC. OBLogProxy calls OBCDC and acts as its consumer to obtain incremental transactional data from the OceanBase database.

Figure 7 Comparison of the OBCDC and OBLogProxy plans

Although CDC mode eliminates the extra cost of Binlog generation and parsing, neither form was an ideal solution because of considerations around technology stack, compatibility, and maintainability (OceanBase will no longer maintain CDC mode after version 4.2.0). We therefore did not ultimately adopt CDC mode.

We have now largely achieved business-transparent migration, zero data loss, GB/s forward synchronization efficiency, and 100+ MB/s reverse synchronization.

Figure 8 Forward and reverse migration flow

Adaptation and Optimization to Support Migration to OceanBase

During the migration, we completed adaptation and optimization in the following areas.

• Driver adaptation: the JDBC driver was downgraded to version 8.0.25.
• Syntax adaptation: removal of DML or syntax not yet supported by OceanBase (such as the insert ignore syntax).
• Binlog parsing adaptation: removal of the Checksum verification for RotateLogEvent.
• Dual-mode reverse synchronization: dual incremental modes (CDC and Binlog) and dual cache modes (in-memory queue and message queue) to handle different traffic scenarios, improving reverse synchronization speed.
• Metadata change awareness: improving NDC’s detection and response speed when the underlying MySQL undergoes a primary-replica switchover while DDB is the source side.

Going forward, we also plan to implement AI-driven migration — that is, intelligent operations for the NDC tool based on multi-agent collaboration over MCP — to reduce the operational cost of long-running synchronization tasks during migration. We are still in the early migration stage, and this work is in progress.

Current Progress and Future Plans

Currently, the various business lines at NetEase Cloud Music have gradually begun using NDC for validation, online synchronization, dual-running, and other forms of rollout. Some businesses have completed migration, and the surrounding infrastructure has begun to follow suit, verifying that the system is mature and usable. However, to ensure stable business switchover, our cutover cycle is relatively long, and there is still much work to do.

Next, we will enter the core pilot and large-scale migration phases, further refining the systematic build-out of OceanBase and its surrounding ecosystem — including ticketing, deployment, testing, tuning, and operations. Although we are still in the early stages, we have already received a great deal of help from the OceanBase community, and we are very grateful for OceanBase’s support.

Finally, we recommend the WeChat account of Lao Ji, the head of OceanBase open source — “Lao Ji’s Tech Talk” — which continuously publishes technical content related to #databases, #AI, and #technical architecture. Friends who are interested are welcome to follow!

“Lao Ji’s Tech Talk” hopes not only to keep bringing you valuable technical insights, but also to contribute to the open-source community together with everyone. If you appreciate the OceanBase open-source community, light up a little star ✨! Every Star you give is the motivation behind our efforts.