Dmall and OceanBase: Achieving Full Inter-Tenant Resource Isolation and Low-Cost System Upgrades

This article is excerpted from the e-book “Case Studies of OceanBase Community Edition in Pan-Internet Scenarios”. Click the link to get the full version.

In today’s wave of digital transformation, enterprises face many challenges — and in retail SaaS scenarios in particular, the complexity and cost of data processing stand out. As a pioneer in retail digitalization, we are not only one of China’s top providers of holistic digital solutions but also a leader in the Asian market. We have over a hundred omnichannel systems spanning membership management, merchandise, marketing, O2O, POS, WMS logistics, AI clearance, AI shopping guides, and many other key links, providing all-round digital support for retail enterprises. Our customers span China, Southeast Asia, Europe, and beyond, including well-known online-celebrity merchants, leading national convenience-store chains, and joint-venture supermarket chains. Behind our rapid business growth, we also face challenges arising from the retail SaaS scenario and from the bottlenecks of our business systems.

This article shares our hands-on experience using OceanBase to upgrade our database architecture and simplify our tech stack — and thereby achieve cost reduction and efficiency gains — drawing on the challenges Dmall faced, the advantages of OceanBase and why we chose it, Dmall’s practice with OceanBase, and our resource-isolation practice under OceanBase’s multi-tenant architecture.

1. The Challenges Dmall Faced

1. High System Complexity

Dmall adopts a microservice architecture, with a great many business links across its end-to-end processes and a large overall application footprint. Correspondingly, the number of databases already exceeds 500. Moreover, as systems are continuously iterated and upgraded, the data scale keeps growing, making operations management ever harder. It’s like a vast, complex network in which every node carries critical business — pull one hair and the whole body moves — and operations staff must stay alert at all times to handle whatever problems may arise.

2. Fast Business Growth and Rising Demand for Horizontal Scaling

As the business flourished, we actively formulated an overseas strategy to expand into new markets abroad. However, the strict requirements of regional data-security laws meant we had to independently deploy an entirely new system to handle overseas business traffic. In the initial deployment phase, since it was hard to accurately estimate the eventual business scale and the rate of data growth, allocating database resources up front became a major challenge. To control costs, the common practice is to allocate fewer deployment resources at first. But before long, rapid business growth brings a surge in data, and how to scale quickly at that point becomes a thorny problem. It’s like discovering, on a high-speed train, that the track ahead needs an urgent widening — while time is very short.

3. Serving a Large Number of Merchants Within a Single Cluster

The SKU (stock-keeping unit) scale of convenience stores and supermarket chains varies widely, from a few thousand to tens of thousands. In this situation, it’s hard to deploy a separate system for each merchant. Our SaaS system therefore needs to support hundreds of small and medium merchant customers, with the data generated by all merchants sharing database resources at the underlying layer. It’s like a large warehouse in which different merchants’ goods must be stored and managed sensibly — preserving each one’s independence while making full use of the warehouse space to share resources efficiently.

4. High Resource Costs

Retail is an industry closely tied to people’s livelihoods, and its business systems must run around the clock. Whether it’s the procurement, sales, and logistics links of the supply chain or the e-commerce business, activity continues not only during the day but also at night and into the small hours. This causes a steady stream of data to flow into the database, making resource costs rise like an ever-climbing line with seemingly no limit. Dmall mainly uses six open-source databases; across the entire production environment, database instances already exceed ten thousand, and the data footprint approaches the 10 PB scale. Such a vast data volume and complex database environment undoubtedly pose enormous challenges for resource-cost control.

5. Persistently High Operations Costs

In using multiple databases, we ran into a series of thorny problems — including the difficulties brought by technical complexity, high operations costs, cumbersome management costs, steep learning costs, and complex delivery costs. Delivering a single private-deployment environment involves over a hundred systems, plus over a hundred database clusters and hundreds of database instances. It’s like constructing a grand building where every component must be carefully installed and tuned, and a problem in any one link can affect the whole system’s normal operation — so operations costs have remained persistently high.

2. The Advantages of OceanBase and Why We Chose It

1. The Advantages of a Distributed Database

To meet the challenges above, we began our database selection. Because a distributed database supports larger capacity, offers transparent scaling, provides financial-grade data security, and can improve development efficiency while lowering operations costs, it can better support business growth. We therefore firmly believe it is the future trend for databases, and this selection process evaluated only distributed-database products.

2. Business-Driven Selection Considerations

First, from a scalability standpoint, we faced many challenges. At present, the data volume of several single MySQL databases already exceeds 4 TB and is still growing fast. When we switched our largest MySQL database to a distributed database, its data volume had grown to 29 TB. Facing rapid data growth and MySQL’s capacity bottleneck, our DBAs (database administrators) were deeply concerned.

• On one hand, we kept urging the R&D team to clean up and archive data, but the results were often poor, because their main work is business-requirement iteration and they could hardly spare attention for data cleanup.
• On the other hand, we considered continuing to expand disk space. In a cloud environment, expanding space is relatively easy; the block storage offered by cloud vendors can reach 32 TB or more per disk. But with data continuing to grow, expanding disk space alone only defers the problem rather than solving it fundamentally, and we could face an even bigger predicament in the future.
• Beyond that, we could also choose a sharding scheme, but this is cumbersome and high-risk and would take months. Because such a scheme harms SQL capability, code changes are unavoidable.

We therefore hoped to use a distributed database’s transparent, scalable capabilities to smoothly support rapid business growth.

Second, from an operations-cost standpoint, while ensuring system stability, we aimed to reduce operations complexity. Take MySQL as an example: suppose a database is an “egg” and a MySQL instance is a “basket.” When deploying 1,000 databases, how should they be sensibly distributed across multiple MySQL instances? Which databases should sit in the same instance? Placing two resource-hungry, important databases in the same instance can lead to resource contention. Also, a payment-type database, though small in data volume, has very high business requirements and shouldn’t share an instance with others. Because of differences in business, priority, data-growth rate, and QPS (queries per second) requirements, DBAs frequently need to adjust the database layout. Even setting resource cost aside, the operations challenge alone is considerable. We hoped a distributed database could solve this and help DBAs automatically adjust the database layout (see Figure 1).

Figure 1: Using a distributed database to “auto-move the eggs”

Finally, from a high-availability standpoint, we hoped to guarantee the cluster’s high availability. When operating MySQL clusters, we usually use tools like MHA and Orchestrator for high availability, but they are all “bolt-on” forms that fundamentally can’t solve the split-brain problem caused by network partitions (see Figure 2). Because the database and the MHA component are two separate pieces of software, not in the same process, they lack consistent coordinated control. By comparison, a high-availability architecture like MySQL’s Group Replication is more reliable: like distributed databases such as OceanBase or TiDB, it’s based on the Paxos or Raft distributed-consensus protocol, and can achieve the high-availability goals of RPO=0 (zero data loss) and RTO<30s (recovery time objective under 30s).

Figure 2: Using a distributed database to achieve “financial-grade high availability”

3. Product-Selection Test Data

Based on the selection factors above, we chose the natively distributed database OceanBase, and further compared the QPS of OceanBase and MySQL on table read-write, table read-only, and table write-only, as well as the difference between them in storage cost. Table 1 lists the configuration for this product-selection test.

Table 1: Configuration for the product-selection test

Item	OceanBase	MySQL
Community version	v4.1.0	v5.7.16
Memory configuration	Tenant memory_size 16GB	innodb_buffer_pool_size 16GB
Per-machine configuration	32C RAID10 SSD	32C RAID10 SSD
Flush configuration	Default forced flush (no extra flush configuration parameters)	sync_binlog=1, innodb_flush_log_at_trx_commit=2
Concurrency	5, 10, 20, 30, 60, 120	5, 10, 20, 30, 60, 120
Test modes	read_write, read_only, write_only	read_write, read_only, write_only
Single test duration	300s, 18 tests total (concurrency × test mode)	300s, 18 tests total (concurrency × test mode)
Test method	Using obd test sysbench (built into obd), including the prepare, run, and cleanup steps	Using sysbench, including the prepare, run, and cleanup steps

With OceanBase and MySQL configured identically, comparing on a sysbench dataset of ten tables of 30 million records each, we found that when concurrency was below 20, MySQL performed slightly better than OceanBase. But for both QPS and latency, as concurrency increased, OceanBase’s performance gradually approached and could surpass MySQL’s. A comparison across OceanBase’s different configurations follows (see Figure 3).

Figure 3: Test result 1

For standalone deployment, whether you choose to access OBServer through OBProxy or connect to OBServer directly affects its read-write QPS on a sysbench dataset of ten tables of 30 million records each. The test results show that connecting to OBServer directly performs 30%–50% better than connecting through OBProxy. Therefore, for standalone deployment, we recommend connecting to OBServer directly to avoid the extra network overhead of accessing OBServer through OBProxy.

Performance also varies under different tenant memory configurations. The test results show that 32GB tenant memory improves performance by about 14% over 16GB tenant memory (see Figure 4).

Figure 4: Test result 2

4. OceanBase vs. MySQL Tablespace Comparison

In a production monitoring-snapshot scenario, we migrated 20 tables totaling 500 million rows from MySQL to OceanBase and found tablespace usage dropped by 60% (see Figure 5).

Figure 5: Tablespace comparison results

Based on the test data from the migration above, we ultimately decided to roll out OceanBase in production. In summary:

• In the production monitoring-snapshot scenario, comparing MySQL storage (single replica) with OceanBase storage (single replica), we found OceanBase’s data compression ratio reached 6:1, showing OceanBase’s excellent data-compression capability.
• In standalone deployment connected via OBProxy, when concurrency was low, OceanBase’s QPS and average latency were slightly worse than MySQL’s (though it’s worth noting that even OceanBase’s lowest QPS exceeded 10,000, and the higher the tenant memory, the better the QPS; the lowest average latency was 3 ms). However, as concurrency gradually rose, OceanBase’s performance improvements outpaced MySQL’s. Specifically, once concurrency exceeded 200, OceanBase’s performance approached or even surpassed MySQL’s.
• MySQL uses a single-layer architecture, while OceanBase uses a two-layer architecture (composed of OBProxy and OBServer). In standalone deployment, connecting to OBServer directly performs 30%–50% better than connecting through OBProxy. This is mainly because each additional architectural layer adds a corresponding network-level latency cost.

5. Why We Chose OceanBase

The reasons for choosing OceanBase are as follows.

(1) OceanBase is a standalone-distributed integrated database. This means it has significant advantages on two fronts: on one hand, its standalone performance can achieve the low latency and high performance of standalone MySQL, meeting the extreme-performance needs of the early business stage when data volume is small; on the other hand, its distributed nature lets us scale easily during periods of rapid business growth, with virtually no capacity limit. Together, these two traits address the performance and scalability needs across the entire business lifecycle — with no code changes and no downtime to migrate, while guaranteeing high performance.

(2) As a natively distributed database, OceanBase inherently provides automatic shard migration and load-balancing scalability, enabling transparent scaling without the business noticing. With extreme optimization based on the Paxos protocol, OceanBase 4.x further improves system availability, achieving RPO=0 and RTO<8s.

(3) OceanBase guarantees high database performance through an architecture that minimizes distributed overhead. Meanwhile, its high compression ratio can save more than 6x the cost compared with MySQL. In addition, OceanBase’s multi-tenant feature fits SaaS customers’ business scenarios perfectly, making resource isolation and scaling operations very easy.

On the data-processing latency and throughput of a distributed database, we need to be clear about the following: the latency of reading and writing data in memory is on the order of 0.1 μs, SSD latency is about 0.1 ms, intra-datacenter network latency is about 0.1 ms, and same-city inter-datacenter latency is about 3 ms. Memory read-write latency is 100–1,000x lower than that of SSD and network. In terms of throughput, memory read-write can reach 100 GB/s, while SSD is about 1–2 GB/s and a 10-gigabit network about 1.2 GB/s — memory’s throughput is roughly 100x that of SSD and network.

It’s worth noting that MySQL, as a standalone database, has its InnoDB and Server layers in the same process, so data interaction is highly efficient, giving it excellent performance and latency. For a distributed database, however, a compute-storage-separated architecture often brings network I/O overhead — a performance bottleneck by design. OceanBase’s distinctiveness lies in its architecture: it implements the SQL engine, storage engine, and transaction engine in a single process, with OBServer doing both compute and storage. The application connects to the OBServer cluster via OBProxy, and the OBServer nodes report data-routing information to OBProxy. After receiving an application-layer SQL, OBProxy forwards it directly, based on the routing information, to the most appropriate OBServer for execution (see Figure 6). If all the data is on the same OBServer node, then the SQL execution is single-machine — similar to MySQL — minimizing network I/O overhead.

Figure 6: OceanBase’s architecture

6. OceanBase’s Performance When Chosen

When data volumes are large or concurrency is higher, OceanBase clearly outperforms MySQL. To understand this, we studied OceanBase’s architecture in depth.

(1) OceanBase combines the low latency of a standalone database with the high throughput of a distributed one. In production business data, OceanBase’s share of single-machine transactions can reach over 80%. This is because the granularity of data sharding in OceanBase is a table or a partition. So when an update touches only a non-partitioned table, or a single partition of a partitioned table, the transaction is necessarily single-machine. Furthermore, if the multiple tables involved in a transaction all reside on the same machine, that transaction is also single-machine.

(2) Optimizing cross-machine joins via the Table group mechanism can turn some cross-machine transactions into single-machine ones. The partition-granularity design ensures most transactions are single-machine. After further optimizing high-frequency cross-machine joins via the Table group mechanism, the share of single-machine transactions can rise even higher. If more than 90% of all transactions in the entire business are single-machine, performance is naturally excellent.

(3) The OceanBase system distinguishes query priorities so that small queries run first. A large query can occupy at most 30% of the worker threads, a ratio adjustable via the large_query_worker_percentage configuration. When there are no small queries, a large query can use 100% of the worker threads. This mechanism is like the rules of the road on a highway. For example, if a highway has three lanes all carrying traffic, then it’s hard for a following car to overtake. But if a rule lets large vehicles (large queries) use only the rightmost lane while small cars (small queries) still have two lanes, this operating mechanism can effectively prevent slow SQL or large queries from blocking or dragging down the system.

These three points stem from OceanBase’s architectural design and from continuous optimization in practice — the keys to guaranteeing its high performance. So why can OceanBase deliver higher performance while also achieving lower cost?

7. OceanBase’s Cost When Chosen

OceanBase uses an LSM-Tree storage engine that supports both encoding compression and general compression, giving it an extremely high compression ratio. According to our test data (see Figure 7), OceanBase’s cluster storage footprint can save up to 75% compared with MySQL, achieving lower cost.

Figure 7: Test data for OceanBase vs. MySQL tablespace and total cluster storage

As Dmall’s service business grows rapidly, data volumes surge and node counts grow exponentially. In this situation, MySQL’s cost will soon exceed OceanBase’s. Test data from our real production environment shows OceanBase’s compression ratio reaches as high as 6x (see Figure 8). In the future, as business data keeps growing, OceanBase can continuously add new nodes, and its storage cost will grow far more slowly than MySQL’s.

Figure 8: Cost test data for OceanBase vs. MySQL

3. Dmall’s Practice with OceanBase

1. OceanBase’s Advantages in Dmall’s Business Scenarios

In summary, the OceanBase database has four main advantages.

(1) Lower storage cost. OceanBase achieves lower storage cost because it uses not only general-purpose compression algorithms but also advanced data-encoding techniques. From our hands-on experience, after migrating a MySQL database to OceanBase, the single-replica compression ratio can approach 90% — an outstanding result.

(2) More stable mixed deployment. OceanBase is more stable in mixed deployment because it can not only achieve inter-tenant resource isolation but also set further isolation within a tenant, effectively limiting I/O and CPU consumption at the user or database level — ensuring stable system operation.

(3) Both low latency and high concurrency. One important reason OceanBase can support both low latency and high concurrency is its configurable Primary Zone feature. By concentrating the leader replicas on a single OBServer node for reads and writes — similar to the read-write mode of standalone MySQL — it can deliver lower latency and better performance. And when high concurrency must be handled, OceanBase can flexibly configure replicas to load-balance across multiple Server nodes, supporting higher concurrency.

(4) More complete surrounding tools. We make broad use of the I/O and CPU resources of multiple servers, and the OCP management platform plays a crucial role here. OCP integrates rich features such as monitoring and alerting and backup and recovery, covering both common and complex needs comprehensively. During troubleshooting, its Top SQL, Slow SQL, and suspicious-SQL analysis are especially useful. In addition, the OMS data-synchronization feature is very powerful, supporting easy synchronization of data from many data sources into our OceanBase database.

2. Migration Practice from MySQL to OceanBase

To date, we have successfully deployed five business databases on OceanBase, totaling 20 TB of data. These businesses are critical to merchants — especially for large customers, whose warehouses and stores are tightly connected and require 24-hour nonstop operation, including the small hours. These are just some of the current use cases; in the future we’ll continue to expand OceanBase’s reach, migrating more business onto the OceanBase platform.

(1) Significant storage-cost benefits. When we migrated business from MySQL to OceanBase, the original 2.1 TB of single-replica data in MySQL shrank dramatically to 252 GB in OceanBase. This striking change owes to OceanBase’s general compression and data-encoding techniques, bringing the compression ratio close to an astonishing 90%. Considering that MySQL uses a one-leader-two-replica architecture while OceanBase uses a three-replica architecture, the combined math means OceanBase’s overall compression ratio delivered over 80% cost savings for us. This shows up not only in storage cost but also in significantly lower operations cost.

(2) More convenient operations. First, a distributed database’s scaling capability makes operations simpler than with a centralized database. For a DBA, maintaining and migrating a 20 TB database would require at least 40 MySQL instances of 1 TB each under MySQL, and the migration process would be lossy — interrupting all existing database connections and requiring at least one DBA to participate throughout. In OceanBase, however, because it supports dynamic scaling and lossless changes, we only need to expand the OBServer storage nodes and point the domain name to the new OBProxy to achieve a lossless migration. Moreover, OceanBase is based on the Paxos protocol, ensuring strong consistency and data safety. Second, with visual management tools, operations becomes more convenient. OCP provides visual management of backups and alerts, along with good observability — including Top SQL and Slow SQL monitoring — and natively supports a high-availability architecture. To ensure management independence and data clarity, we additionally built a metadata database separate from the business databases to support the OCP management platform.

Currently, the TPS in OceanBase has reached nearly 1,500, and the peak QPS approaches 3,000 — outstanding performance.

3. OceanBase’s Continued Evolution and Outlook

Since version 4.0, OceanBase has supported a standalone-distributed integrated architecture. Version 4.2 officially introduced OBKV Redis support, further enriching its capabilities. The latest 4.3 version adds support for a vectorized engine and columnar storage — and this engine is precisely a key building block for AI large models. OceanBase keeps evolving, in line with the vision of a future-ready database.

In the future, we’ll actively explore applying OceanBase in more business scenarios, including migrating the relational MySQL and the non-relational Redis databases to the OceanBase platform. Through this migration, we can effectively rein in the continuous growth of resource costs, optimize resource allocation, and thereby directly improve our profit margins.

4. Resource-Isolation Practice Under OceanBase’s Multi-Tenant Architecture

1. Inter-Tenant Resource Isolation

OceanBase’s multi-tenant architecture achieves physical-level isolation of resources such as CPU, memory, and I/O. This is crucial: it ensures that the business of different tenants won’t affect one another due to resource contention, avoiding the risk of one business’s problem spilling over to other tenants.

2. Fast Elastic Scaling of Tenants

In OceanBase, scaling a tenant up is extremely convenient. Take a tenant with 3 Zones, each Zone containing 2 machines (6 machines total), each machine holding 1 resource unit. To scale out horizontally, you only need to run a single SQL statement to easily add Zone 4 and Zone 5, increasing the resource units from 6 to 10. Likewise, scaling up vertically is simple: if the initial resource is 2C8G and the business grows, and you don’t want to add machines, you only need to upgrade the resource unit from 2C8G to 6C12G. These scaling operations are all dynamic and lossless, with no impact on the running business. For a DBA, the entire scaling process takes just one SQL statement, greatly reducing the workload. OceanBase’s multi-tenant capability therefore fits SaaS-scenario needs perfectly — saving cost while making later scaling convenient.

5. Summary

Facing the many challenges of the retail SaaS scenario, we chose the OceanBase database as our solution. With advantages such as its distributed architecture, high scalability, low operations cost, high availability, and excellent storage compression, OceanBase successfully helped us achieve full inter-tenant resource isolation and low-cost system upgrades. Through our migration practice from MySQL to OceanBase, we not only sharply reduced storage and operations costs but also improved system stability and performance. Meanwhile, our application of resource-isolation practice under OceanBase’s multi-tenant architecture provides strong support for our future development. As technology continues to advance, we’ll keep moving forward hand in hand with OceanBase, driving the digital transformation and upgrade of the new-retail industry.

Finally, I’d like to recommend the WeChat official account “Lao Ji’s Tech Talk” run by Lao Ji, the head of OceanBase Open Source. It continuously publishes all kinds of technical content related to #Databases, #AI, and #Tech Architecture. If you’re interested, feel free to follow!

“Lao Ji’s Tech Talk” hopes not only to keep bringing you valuable technical content, but also to contribute to the open source community together with everyone. If you appreciate the OceanBase open source community, light up a little star ✨! Every Star you give is fuel for our efforts.