NetEase Games Brings OceanBase into Its DB SaaS: Storage Cost Cut by 60%, Backup and Recovery 3x Faster

Author: Tian Weifan, head of the relational-database operations team for NetEase Games’ SaaS services

1. The Architecture of the NetEase Games DB SaaS Platform

As one of China’s leading game-development companies, NetEase Games has always been at the forefront of independently developed online games. It offers a wide range of products and ancillary businesses, spanning hit titles such as Fantasy Westward Journey, A Chinese Odyssey, and Eggy Party, along with the game-trading marketplace “Cangbaoge” and a series of popular game services and ancillary product lines—each requiring different data-processing products to serve different business scenarios.

For these rich and diverse game and ancillary business scenarios, DB SaaS provides a one-stop database private-cloud service platform designed to meet every database need.

As shown in Figure 1, the DB SaaS service is divided mainly into three layers.

The first layer is the hardware service layer. It mainly provides self-built IDC data-center virtualization, public cloud (including AWS, Alibaba Cloud, GCP, and Microsoft Azure), and self-built private-cloud services, fully covering the needs of the underlying infrastructure.

The second layer is the database layer. On top of the hardware service layer, a powerful database service layer is built. This layer provides multiple types of database services—including document, in-memory, relational, KV, vector, and graph databases—fully meeting diverse data storage and processing needs.

The third layer is the database service-capability layer. This layer is divided into three aspects:

• First, database lifecycle management, which mainly provides full-lifecycle services for resource management, database architecture, and database instances.
• Second, data management services (DMS), providing secure, versatile, and convenient services for data query, analysis, and change.
• Third, data transfer services (DTS), including a variety of data-flow services such as data query, data rollback, table-and-index management, and business merging, splitting, and migration for users.

For the database service-capability layer, a comprehensive backup management system has also been built, primarily for game scenarios. The frequent updates in the gaming industry demand efficient, flexible backup solutions, so a series of powerful backup features has been integrated—routine backup, fast backup, incremental backup, schema-and-table backup, and backup inspection.

2. Game Business Scenario Characteristics and Database Selection Needs

(1) Game Business Characteristics and the Pain Points of Using MySQL

Take an ancillary game service as an example. In the early stages, the business used a single-instance primary-standby-replica architecture. As the number of connected games kept increasing, this architecture could no longer keep up with the growing volume of requests, so the database was sharded.

After sharding, a summary instance had to be introduced to handle large-scale aggregated queries. Initially this was handled by MySQL, but as the business kept growing, six major problems emerged:

1. High-concurrency response. At peak times, requests to the primary database approached 100,000 QPS, while the total QPS across all read replicas approached 1 million—too much for a single MySQL instance to bear.
2. Data synchronization lag. Read requests to the replicas demanded very low latency, but replicating and synchronizing data from multiple sources, on top of query pressure, caused lag.
3. Single-database storage pressure. A single node’s storage had already exceeded ten-plus TB.
4. Business isolation. A surge in traffic from one game’s event would affect the normal operation of other games.
5. The pain of DDL changes. Game server merges and splits required frequent, large-volume DDL changes, and the rapid iteration in a game’s early days also drove frequent DDL changes.
6. Difficult operations. Under traffic surges, the only relief was to add more instances; with large data volumes, scaling out replicas and performing backup and recovery consumed a great deal of time.

(2) Why Did NetEase Games Choose OceanBase?

Through in-depth discussions with the business teams, we identified the key characteristics the database had to have. OceanBase fit the bill perfectly:

1. Stability under high concurrency. The three-replica distributed architecture supports automatic failover, with RPO=0 and RTO<8s.
2. Transparent horizontal scaling. Smooth online scaling in and out, with automatic load balancing after scaling.
3. Resource isolation. Support for multi-tenancy, with CPU, memory, and IOPS isolation between tenants.
4. Timely data synchronization. Using the OMS migration tool, the data-sync pipeline has almost no lag.
5. HTAP capability. One system and one copy of data support HTAP scenarios.
6. Relatively low cost. The LSM-Tree storage engine saves 70%-90% of storage cost.
7. MySQL compatibility. Good MySQL compatibility, with no need to modify business code.

3. Solution and Testing: Multi-Layered Validation to Ensure Smooth Business Operation

(1) Test 1: Initial Baseline Testing

We used Sysbench to test mixed read-write, read-only, and write-only scenarios. In terms of OLTP performance, with small data volumes OceanBase 4.0 was almost on par with MySQL, and version 4.1 outperformed single-instance MySQL; with large data volumes (over 100 million rows), performance after scaling far exceeded single-instance MySQL.

In terms of storage compression, after exporting 5TB of data from upstream MySQL to OceanBase, the total across three replicas was only 2.1TB—700GB per replica—a data compression ratio of nearly 86%.

(2) Test 2: Dedicated Multi-Tenant Resource-Isolation Testing

We stress-tested two tenants simultaneously. Conclusions:

• Resource stability met expectations, with stable CPU and memory usage.
• Isolation met expectations, with no noticeable impact between tenants.

(3) Test 3: Compatibility and High-Concurrency Traffic Validation

We introduced a self-developed traffic-replay system (drcapture + drrecord) to perform compatibility validation and concurrent-traffic validation.

Compatibility testing was completed, with no compatibility issues whatsoever. Replaying at five times and six times the traffic scale, OceanBase still responded quickly with no anomalies.

(4) Test 4: Reliability Validation

We ran drills around four failure scenarios:

• Upstream machine crash → OMS sync lag stayed within tens of seconds
• OMS server failure → sync lag of roughly 30s-60s
• OceanBase node crash → business jitter kept under 10s
• Large-table DDL operation → skip DDL-change synchronization and execute it on OceanBase first

4. Sharing Experience from OceanBase Technical Practice

(1) OMS Data Synchronization Performance Tuning

When upstream write volume was high, synchronization often lagged. We traced it to high RPC latency in functions related to auto-increment sequences. The cause was that in Order mode, every request for the auto-increment sequence incurred RPC overhead.

Solutions:

• Option 1: Remove the auto-increment column attribute
• Option 2: Change Order to noorder mode, so each OBServer maintains its own attribute cache

(2) Queries During Synchronization Reading an Intermediate Transaction State

In a business scenario where a single transaction contained hundreds of DML operations, OMS split it by the default maxRecords=64, causing queries to read an intermediate state. The solution: adjust the parameter to 1024.

(3) Designing Partitioned Tables Sensibly

We reduced 512 hash partitions to 10-plus partitions, reducing RPC latency while still satisfying horizontal balance. For queries that do not use the partition key, we created global indexes.

(4) Designing Primary Keys or Unique Keys

Partitioned tables without a primary key or unique key caused duplicate data during synchronization; the fix was to add a primary key or unique key to the OceanBase tables.

5. Lower Cost, Higher Efficiency, Stable and Reliable: The Changes OceanBase Brought to NetEase Games

After bringing in OceanBase, we gained six benefits:

1. Query stability. Compared with MySQL, stability improved significantly, with almost no jitter.
2. Flexible scaling reduces high-concurrency pressure. After migrating MySQL read-only-replica QPS to OceanBase, the pressure dropped sharply.
3. Lower storage cost. Compared with a single MySQL replica, overall storage cost fell by more than 80%, and by another 30%+ after archiving compression.
4. Data timeliness effectively controlled. Peak lag was at most just 2s, fully meeting business needs.
5. Improved backup and recovery efficiency. Recovery efficiency improved by at least three times.
6. Simpler operations. Dynamic resource adjustment, GUI-based SQL throttling, and Paxos high availability are all transparent to applications.

In addition, we decided to integrate OceanBase’s ecosystem-tool capabilities into DB SaaS and build an OceanBase cloud platform, providing one-stop operations and management.

6. Summary and Outlook

Since bringing in OceanBase, NetEase Games has found the system very stable, with no performance jitter or synchronization-lag issues, effectively resolving the business pain points. Incorporating the OceanBase ecosystem tools into the DB SaaS platform has enriched its service capabilities.

Going forward, we plan to gradually reduce the number of MySQL read replicas and are considering migrating all business to OceanBase.