NetEase Personal Mail Upgrades Its Database to OceanBase: A Dual Breakthrough in Reliability and Stability

Preface

Born in 1997, NetEase Personal Mail has navigated more than two decades of internet waves. Backed by outstanding service and technical strength, it has grown into one of the most influential email brands in China and worldwide. NetEase operates six distinctive email domains—163, 126, yeah, vip163, vip126, and vip188—each precisely positioned for a different user group to meet diverse needs.

After years of accumulation and expansion, NetEase Personal Mail has built up a massive user base, with users spread across every corner of the globe. Its business scenarios are rich and complex, covering personal daily communication, business correspondence, marketing campaigns, information notifications, and many other domains. Under such a vast and complex business system, the demands placed on database performance, stability, and scalability are extremely high. This article shares the approach and hands-on engineering experience behind NetEase Personal Mail’s database upgrade—from a sharded database and MySQL to OceanBase.

1. Current State of the Data System

As mentioned above, NetEase Personal Mail comprises six core email domains—163, 126, yeah, vip163, vip126, and vip188—each of which has built its own independent data system. These systems support OLTP workloads such as email send/receive and user management, as well as OLAP scenarios such as log analysis and traffic statistics. Over years of rapid growth, the data scale has kept expanding. To fully tap the potential of technological evolution and build a more efficient data architecture, we actively explored innovative practices with distributed databases. The original system left room for improvement in the following areas.

1. The need to improve storage resource efficiency.

As the business data scale exceeded the TB level and kept growing rapidly, traditional databases faced new challenges in optimizing storage resources. Especially when implementing a high-availability architecture (such as primary-secondary replication), the ability to reuse resources needed further enhancement.

2. Building global dynamic resource scheduling.

The business characteristics of each email domain differ significantly, so resource allocation must balance secure isolation with elastic scheduling. How to flexibly allocate resources across domains and improve overall resource pool utilization became an important direction for the architecture upgrade.

3. Upgrading horizontal scalability.

When dealing with surging data volumes, the traditional architecture faced bottlenecks in the efficiency of single-machine vertical scaling. Migrating TB-scale tables required a long time window, calling for a more agile scale-out/scale-in mechanism to guarantee business continuity.

4. Evolving high availability and disaster recovery toward automation.

To meet higher-tier system reliability requirements, we needed to build second-level failover and eliminate the switchover risks caused by differences between primary and secondary resources. Automatic topology discovery and seamless connection migration also became key goals of the architectural evolution.

5. Building real-time analytics capabilities.

The current data analytics pipeline exhibited tiered timeliness, with some scenarios relying on T+1 data synchronization. Building a real-time data channel between the business database and the analytics database was critical to improving decision-making agility.

6. Upgrading operations automation.

Faced with the need to modify metadata on tables holding billions of rows, we needed to break through the efficiency bottleneck of DDL operations. At the same time, unified management of multiple database versions and the construction of automated change workflows were of great significance for improving operations agility.

2. Why OceanBase and a Comparative Analysis

(1) Research and Analysis of NetEase Personal Mail’s Selection Requirements

During the research and selection phase, we analyzed two distributed database solutions—OceanBase and a comparable database—and evaluated their performance across several key characteristics based on the business profile of NetEase Personal Mail. We ultimately chose OceanBase as the primary solution for the database upgrade. Below is an analysis, grounded in actual business needs, of why it met our selection expectations.

1. Data reliability and high-availability design.

• Strong data consistency: OceanBase is based on a distributed consensus protocol (such as Paxos), ensuring consistency across multiple data replicas. It effectively resolves the potential inconsistencies in traditional database primary-secondary synchronization, meeting the needs of scenarios that require high reliability and financial-grade data security.
• Intelligent disaster recovery: OceanBase provides an automatic failover mechanism that keeps the business continuously available in single-point failure scenarios, which is highly significant for supporting NetEase Mail’s complex, high-concurrency business scenarios.

2. Multi-tenant resource isolation.

• Fine-grained resource control: OceanBase supports tenant-level isolation of CPU, memory, and I/O, satisfying the security and independence requirements across multiple business domains.
• Elastic resource scheduling: It dynamically adjusts resource allocation according to business load, improving resource utilization and ensuring system stability—particularly well suited to the periodic traffic fluctuations of email services.

3. Optimized data storage efficiency.

• Intelligent compression engine: OceanBase’s storage architecture significantly reduces the storage space data consumes. Test results show that its compression efficiency outperforms traditional architectures, effectively cutting storage resource investment at the TB data scale.
• Cost reduction: Through compression technology and storage structure optimization, OceanBase effectively eases the resource cost pressure caused by growing business data volumes.

4. Scalability and elasticity.

• Online, transparent scaling: OceanBase supports fully online node addition/removal and data rebalancing, keeping migration operations transparent to the business and reducing the risk of downtime during scaling.
• Automatic load balancing: After a scale-out, data traffic is intelligently distributed to the new nodes, ensuring service quality and business continuity.

5. An integrated, intelligent operations platform.

• Out-of-the-box management: OceanBase provides the full-stack cluster management platform OCP, supporting visual operations for monitoring and alerting, backup and recovery, and deployment management.
• An upgraded operations paradigm: Complex manual operations can be standardized, helping improve NetEase Mail’s maintenance efficiency across multiple business scenarios.

6. Full compatibility with the MySQL ecosystem.

• Low migration cost: OceanBase is compatible with the MySQL protocol and syntax, so NetEase Mail’s business could complete a smooth migration without large-scale application rework.
• Seamless ecosystem integration: It natively supports the MySQL ecosystem toolchain (such as Flink CDC), allowing the existing toolchain to run seamlessly on the new platform and thereby improving the connectivity and timeliness of the analytics pipeline.

7. Technical ecosystem and service support.

• An active technical ecosystem: OceanBase enjoys rapid iteration driven jointly by the company and the open-source community, continuously strengthening its key technical capabilities and ensuring long-term technical support.
• Service assurance: The combination of open-source community responsiveness and enterprise-grade service provides more comprehensive support for mission-critical business.

(2) Sysbench Testing and a Comparison of the Two Database Products

To further validate the suitability of the database solution for the NetEase Personal Mail scenario, we ran performance tests on different versions of OceanBase and a comparable database, and conducted a comparative analysis alongside other database products in the industry. Some of the test conclusions are as follows:

• Test results show that in pure-insert and pure-query scenarios, OceanBase v4.3 outperforms the other tested solutions, making it suitable for AP workloads. In comprehensive insert/update/delete scenarios, OceanBase v4.2.5 performs more prominently, making it suitable for TP workloads.
• In multi-tenant scenarios, testing verified OceanBase’s resource isolation capability. During high-concurrency writes, we observed a correlation between disk IOPS limits and performance fluctuations.
• The test results show that OceanBase has a significant advantage in data compression.

The specific test procedures and environment configurations are described below.

1. Performance comparison.

We compared a comparable database v8.1, OceanBase v4.3, and OceanBase v4.2.5. The test environment was configured with three 48-core, 384 GB SSD machines, comparing their performance on select, insert, update index, and update non-index operations.

• select: Both databases peaked in the 1,200–1,500 thread range. Comparing QPS and TPS, OceanBase v4.3 was 1.3x OceanBase v4.2.5 and 1.7–1.9x the comparable database v8.1.
• insert: OceanBase peaked at 1,800 threads, while the comparable database peaked at 1,200 threads. Comparing QPS and TPS, OceanBase v4.3 was 1.3x OceanBase v4.2.5 and 2.27x the comparable database v8.1.
• update index: Both databases peaked at 2,100 threads. Comparing QPS/TPS for indexed-field updates, OceanBase v4.2.5 was 2.7x the comparable database v8.1 and 3x OceanBase v4.3.
• update non index: Both databases peaked at 2,100 threads. Comparing QPS/TPS for non-indexed updates, OceanBase v4.2.5 was 1.66x the comparable database v8.1 and 3.04x OceanBase v4.3.

2. Resource isolation.

To assess OceanBase’s resource isolation in multi-tenant scenarios and ensure tenants do not interfere with one another, we ran key comparison tests.

First, the single-tenant vs. dual-tenant stress test comparison is as follows.

• Test design: Within the same cluster, we ran two performance stress tests under equivalent conditions:
  • Scenario A: a single tenant (configuration: 12 cores, 40 GB)
  • Scenario B: two tenants (each configured with 12 cores, 40 GB)
• Validation goal: Observe whether, during the dual-tenant stress test, the performance of an individual tenant drops significantly due to resource contention compared with the single-tenant scenario, thereby judging the effectiveness of resource isolation.
• Conclusion: The comparison showed that under the same resource configuration, an individual tenant’s stress-test performance did not decline noticeably when multiple tenants ran in parallel. This demonstrates OceanBase’s effective isolation of compute and memory resources.

Second, the cluster resource-limit allocation stress test is as follows.

• Test design: All resources of a 3-node cluster were allocated to 3 business tenants. At the same time, we applied multi-scenario mixed read/write pressure with varying thread counts (24 to 2,400+) on these tenants.
• Validation goals:
  • Probe each tenant’s performance boundary (QPS/TPS) and resource saturation under resource-limit allocation.
  • Assess the stability of OBProxy under high load.
  • Monitor how the system tenant (sys) is affected under high load.
• Conclusions:
  • Tenant performance saturated. The QPS/TPS of all tenants stabilized once the concurrent thread count reached the 1,500–2,400 range, indicating the performance peak had been reached. Further increasing concurrency led to higher latency, meaning resources were already fully utilized.
  • OBProxy remained stable. OBProxy CPU usage grew smoothly as pressure increased (peaking at about 75%), and memory usage reached a high level shortly after startup and stayed steady, with no anomalies observed.
  • The sys tenant was undisturbed. The CPU and memory usage of the system tenant (sys) remained stable under global high pressure, with no noticeable fluctuation, proving the isolation guarantee for critical system services.

3. Data compression.

We again compared OceanBase with a comparable database. We loaded test data using Sysbench, with the following parameters:

--time=60 --threads=16 --report-interval=10 --db-driver=mysql --rand-type=uniform --tables=16 --table-size=10000000 oltp_common prepare

Product	At write completion	After 5 hours	After 24 hours	Data after stabilization
Comparable database v8.1.1	51.3G	51.3G	51.3G	51.3G
OceanBase v4.2.5	104.59G	77.46G	37.34G	37.34G

3. Technical Practice in Application Scenarios

(1) Migration Experience from Sharding to OceanBase

NetEase Personal Mail has now piloted OceanBase across several business lines. During the migration, we summarized a few lessons worth sharing.

1. Governing unique-key constraints when migrating in a distributed environment.

Background: In the MySQL-based sharding architecture, due to repeated historical migrations and scale-outs, some tables contained duplicate primary-key data across MySQL instances (for example, pid=1 existing simultaneously on different instances). The routing mechanism of the data interface accessed only the hash-matched instance, so the duplicate data stayed hidden for a long time—but it triggered unique-key conflicts when migrating to OceanBase.

Solution: First, run a distributed duplicate scan before migration:

SELECT pk FROM tb GROUP BY pk HAVING COUNT(*) > 1;

Then repair the data according to business rules:

• Keep the latest version of the data;
• Safely delete invalid redundant data;
• In special cases, restructure the primary key (for example, by adding a business prefix).

Outcome: A smooth migration into OceanBase’s strict-constraint environment, laying a foundation of data quality.

2. Multilingual character-set compatibility practice.

Background: The users of NetEase Personal Mail are spread across the globe, and the stored data spans many written languages. Some older programs still use GBK, whose character set supports only a limited set of languages such as Chinese, Japanese, and Korean. The legacy business system handled email addresses with the GBK character set; when storing characters outside GBK, such as Thai, MySQL 5.7 would silently truncate to the first valid byte, causing data loss.

With OceanBase, however, the character-set specification is strictly enforced, and it throws an error instead.

Solution: The application layer transcodes incompatible text, and we unified the storage on the UTF-8 character set. Outcome: This eliminated the character-set compatibility risk in the dual-write system and supports standardized storage of email addresses worldwide.

3. Strategies for partitioning and indexing massive tables.

When migrating large tables with heavy read/write traffic, we consider converting them into partitioned tables; reasonable partitioning and indexing can unlock the database’s maximum performance.

When planning partitions, we try to make high-frequency SQL go through the partition key, and for SQL that cannot use the partition key, we add indexes. OceanBase’s partitioned indexes are divided into local indexes and global indexes. According to stress-test results from other NetEase colleagues, global indexes degrade write performance, lowering the maximum QPS by roughly 20–50%. Therefore, in general we avoid global indexes whenever possible, or use them sparingly.

(2) OBKV Implementation Practice and Considerations

For KV storage, NetEase Personal Mail primarily uses Redis, and for some cold data, it also uses Tendis and Pika. We plan to explore OBKV-Redis as an alternative for a persistent cache database, to meet the business’s need for high compression and high availability in data storage. The reasons are:

• Redis protocol compatibility. It supports most of the Redis commands the business uses (cluster mode is not yet supported).
• Effective release of disk space. The underlying storage architecture of OBKV-Redis has strong compression, freeing up disk space and saving a great deal of memory.
• High availability. The connection method is the same as a single Redis node, and OBKV-Redis already provides multi-replica disaster recovery at the underlying layer.
• Integrated operations. It is managed uniformly by OceanBase’s operations platform, OCP.

To date, OBKV-Redis has been running stably in one business line for a while, with the key count reaching tens of billions and overall latency staying under 10 ms. During business peaks, when QPS reaches 20,000, the actual business latency still stays below 10 ms.

There are two areas for improvement. First, based on our observations, OBKV-Redis is not compatible with all Redis commands—traversal-type commands in particular are largely unsupported (such as keys and scan), and some partially supported commands may have issues; for example, running monitor may cause interruptions. Strict testing is required before launching it for a business. Second, OBKV-Redis monitoring depends on the OCP platform. Because OCP is in rapid iteration, its monitoring of QPS commands only covers the main ones, such as set and get… If the business runs setex, it cannot be monitored.

In addition, when the business writes string-type keys, the table row count corresponds one-to-one with the number of keys. But when writing hash keys, you will find the space consumed is very high, and the table row count reported by the OCP platform exceeds the number of keys.

When you inspect the table data via a MySQL connection, you will find that hash keys are amplified: the key name is padded to a fixed-length string, and each field occupies one row. As a result, the OCP platform ultimately counts the above key as 2 rows, so the key count cannot be fully referenced from OCP’s table statistics.

Based on the above experience, we believe OBKV-Redis is in a rapid-iteration phase, and every operation should be thoroughly tested and verified before applying it to online business.

4. Results of the Database Upgrade: Stable, Reliable, Cost-Effective

After this database upgrade, NetEase Personal Mail resolved some of the problems in its traditional architecture and achieved corresponding gains in performance optimization, cost control, and data reliability.

First, performance and stability broke through bottlenecks. Thanks to OceanBase’s standalone-distributed integrated architecture, QPS significantly surpassed single-instance MySQL in high-concurrency scenarios. Throughput improved substantially, and the system ran for several consecutive months without jitter.

Second, storage costs were optimized. OceanBase’s LSM-Tree-based storage architecture effectively reduces storage space requirements through compression. In actual business, after migrating 3.2 TB of data from the original MySQL primary-secondary two-replica setup to OceanBase’s three-replica setup, the total storage space dropped to 900 GB—a 72% storage cost saving, with a single-replica compression rate as high as 80%.

Third, real-time data processing was transformed. Thanks to OceanBase’s full compatibility with the MySQL ecosystem, the data analytics system made a leap from T+1 to real-time integration. Paired with the native HTAP features of version 4.3.5, analytical workloads can reuse the OLTP data replica when executing complex queries, compressing analytics latency to the sub-second level while preserving transaction processing performance.

Fourth, data reliability was upgraded. The multi-replica strong-consistency mechanism eliminates the risk of data inconsistency in the primary-secondary architecture. Failover and rapid recovery capabilities ensure service continuity, achieving a qualitative leap in disaster recovery over the traditional architecture.

Fifth, intelligent operations were realized. The OCP platform supports dynamic adjustment of tenant resources and online scaling, and combined with full-link tracing and diagnostics, it significantly improves operations agility.

Through this database upgrade, NetEase Personal Mail achieved initial results in system performance optimization and business capability enhancement. Going forward, we will continue exploring more technical directions—including the application of vector indexes and full-text indexes—and try to implement them in some analytical processing (AP) business scenarios, to further drive business growth.

Finally, we recommend the WeChat official account “Lao Ji’s Tech Talk,” run by Lao Ji, the OceanBase open-source lead. It continuously publishes a variety of technical content related to #Databases, #AI, and #Tech Architecture. Friends who are interested are welcome to follow!

“Lao Ji’s Tech Talk” not only hopes to keep bringing you valuable technical sharing, but also hopes to contribute to the open-source community together with everyone. If you appreciate the OceanBase open-source community, please light up a little star ✨! Every Star you give is the motivation behind our efforts.