The Architecture Design and Optimization of OceanBase's Standalone-Distributed Integrated Database
This article is excerpted from the e-book A Study of OceanBase Community Edition Applications in Pan-Internet Scenarios.
Overview
In OceanBase’s decade-plus of technical evolution, it has gone through three major architectural upgrades. The first upgrade was OceanBase versions 0.1–0.5 (2010–2015), when OceanBase achieved a quasi-distributed architecture through a single-write, multi-read design, comprising several different roles such as UpdateServer, ChunkServer, and MergeServer. The second upgrade was OceanBase versions 1.0–3.0 (2016–2022), when OceanBase became a peer-to-peer, fully distributed architecture in which all nodes could read and write, gradually supporting complete SQL functionality. The third upgrade was OceanBase 4.0, formally proposed in August 2022—the industry’s first standalone-distributed integrated database.
OceanBase 4.0 has a vivid metaphor: “small is big.” This is because the core of standalone-distributed integration is using a single system to achieve the transition from standalone to distributed, transparently to the user. Through the standalone-distributed integrated architecture, OceanBase meets users’ demand to scale their business from small to large: users don’t need to worry about choosing between a centralized or distributed technology path. They can start with a small-spec standalone deployment when the business volume is small, using a fully featured standalone deployment form, and then—as the business pressure changes—smoothly scale the database out from a single machine to multiple machines or even a massive distributed cluster, while also retaining the ability to smoothly scale back from many machines to a single machine. In other words, a single database meets the centralized and distributed architecture needs of a business as it grows from small to large—one system meeting every user’s full-lifecycle data storage and management needs.
This article briefly introduces, from a technical perspective, the design philosophy, technical optimizations, and business value of the standalone-distributed integrated architecture.
1. Three Architectural Iterations, From Distributed to Standalone-Distributed Integration
(1) The Technical Challenges of Going From Natively Distributed to Standalone-Distributed Integration
OceanBase began development in 2010, when it was divided into two layers: storage and compute. The upper layer was a stateless service layer providing SQL services, and the lower layer was a storage cluster composed of two types of servers. This architecture had a degree of scalability—read scalability in particular was strong—and since the SQL layer was stateless, it could scale freely. But the biggest problem with this architecture was single-point writes with multi-point reads, which made it impossible to scale under higher concurrency demands. At the same time, the split between the storage layer and the SQL layer made latency hard to control.
To solve these problems, OceanBase abandoned the earlier architecture and developed the OceanBase 1.0–3.0 architecture, in which every node could process SQL while also handling transactions and storing data. As shown in Figure 1, the vertical direction is the distributed scalable layer, where scalability is improved by continuously adding machines; the horizontal direction is the replication layer, which provides high-availability service capabilities.
Figure 1: OceanBase 1.0-3.0 architecture
Under this architecture, OceanBase became the only distributed database at the time to pass the TPC-C test, proving that the architecture’s scalability and concurrent processing capability could meet the needs of the vast majority of the world’s current online service systems.
But as business needs evolved, OceanBase, on its path toward becoming a general-purpose database, hoped to support smaller-scale applications as well. The sticking point was that under the 3.0 architecture, the number of transaction log streams was bound to the number of storage shards: the granularity of storage shards determined the granularity of transaction processing and high availability. This meant that as the number of storage shards increased, the number of log streams increased with them—and in smaller-scale business systems, this overhead became disproportionately large. Therefore, the number of storage shards needed to be decoupled from the transaction log streams, letting several storage shards share a single transaction log stream and the high-availability service it provides, achieving a balance between scalability and cost so as to better support small-scale applications.
In addition, considering the development patterns and lifecycle of a business, a database needs to be able to transition smoothly from a standalone mode that supports small-scale applications to a distributed mode that can handle massive amounts of data. The standalone-distributed integrated architecture was thus innovatively proposed, requiring it to combine the scalability of a distributed system with the functionality and standalone performance of a centralized database. Transactional ACID (Atomicity, Consistency, Isolation, Durability) is a fundamental requirement of a database, and the difficulty of a distributed database lies in how to guarantee transactional ACID in exceptional scenarios. At the core is how to achieve fault recovery based on the redo log, and how to achieve the atomicity of distributed transactions in exceptional scenarios based on the redo log.
To truly achieve integration, three key technical problems needed to be solved.
(1) Application transparency: going from a single machine to multiple machines should require no changes to the application, which requires the client to support dynamic routing technology so that when a partition migrates on the backend database, requests can be dynamically routed to the destination server. In addition, whether standalone or distributed, full SQL functionality needs to be supported.
(2) Standalone operations: a single machine has only one redo log, and the way a standalone transaction writes the redo log is similar to that of a classic standalone database. Classic standalone databases use a B+ tree storage engine; OceanBase’s technical innovation is to incorporate the data-blocking idea of B+ trees into an LSM-tree storage engine. On one hand, like an LSM tree, it has high compression capability and keeps hot data in memory to serve requests; on the other hand, by using a B+ tree-like data-blocking approach, it reduces the write amplification of the LSM tree. In OceanBase 4.1, even with strong synchronization across three machines, both standalone performance and storage cost are better than MySQL 8.0.
(3) Cross-machine operations: cross-machine operations are provided by the underlying distributed architecture, and the upper-layer SQL functionality is unaffected. If a transaction involves only one machine, it takes the standalone transaction path; if it involves multiple machines, distributed transactions are implemented via two-phase commit. In addition, performance is optimized as much as possible through techniques such as distribution, parallelism, and asynchronization.
(2) Feasibility Analysis: How to Eliminate the Overhead Brought by Distribution?
The first task of architecture design is feasibility analysis, the core of which is trade-offs. In designing OceanBase’s standalone-distributed integrated architecture, we made the following design assumption: although a distributed database can handle very large amounts of data, most operations are still standalone operations (>80%), and only a small fraction are cross-machine operations (<20%).
Based on OceanBase’s early experience being rolled out within Alibaba: in the pan-Internet industry, To-C online businesses can basically be made distributed by sharding on user ID (user_id), so the vast majority of subsequent operations are operations within a single user, with only very few cross-user operations. The financial industry is similar—most of the time you’re reading and writing your own account, and only a small fraction of the time are you doing cross-account operations like transfers. So the system’s optimization goal became: first, ensure that the 80% of standalone operations have no distribution-related overhead (in this article, “overhead” refers to extra cost), so these operations can be compared on the same starting line as a standalone database; and second, optimize the performance of the other 20% of cross-machine operations, pursuing the utmost performance wherever possible.
The distribution-related overhead of standalone operations comes mainly from two sources: high availability on one hand, and scalability on the other. Back in 2023, an Alipay Oracle DBA shared their experience: when Oracle turns on strong synchronization, performance drops by at least 30%. To achieve lossless disaster recovery, OceanBase’s underlying layer uses a Paxos-based strong synchronization scheme—that is, it achieves high standalone performance by changing the architecture.
Our approach is to make the database’s redo log submission asynchronous, so that the database’s internal worker threads don’t have to wait for the log submission to return a result. Even when the network and disk conditions are poor, the overhead brought by strong synchronization is relatively small. We used sysbench to benchmark the performance of OceanBase with strong synchronization across three machines, and the results show that Paxos strong synchronization costs OceanBase only about 8% in performance. This is entirely acceptable, and this loss can be made up through optimizations in other modules. The performance loss from scalability mainly comes from data sharding: each data shard needs to write a separate redo log. You can simply think of each data shard as a mini database—the more shards there are, the greater the shard-management-related distributed overhead on each machine.
The innovation of the standalone-distributed integrated architecture lies in the dynamic log stream. Each tenant on each machine has only one log stream, and all the data partitions of a tenant are dynamically bound to that log stream, thereby avoiding the overhead caused by a large number of log streams. Moreover, the binding of partitions to log streams is dynamic: when the system adds a new server, partitions can be dynamically unbound from the source log stream and rebound to the destination log stream, thereby achieving dynamic partition migration.
2. Technical Optimization: Reworking the Log Stream to Ensure Flexible Switching Between Standalone and Distributed, Along With Performance and Scalability
In traditional database solutions (see Figure 2), as the business grows, the selection path goes through a small-spec MySQL supporting small businesses, then a high-spec Oracle supporting medium businesses, then Oracle RAC to solve the basic scalability problem for large businesses, and possibly later an upgrade to a mainframe + DB2 solution to carry the core business.
Figure 2: Illustration of enterprise business scale and database selection strategy
This traditional path has several major flaws. First, there is a ceiling on scalability—it cannot scale flexibly and without limit as the business volume grows. Second, each upgrade and overhaul involves replacing a large amount of software and hardware, bringing incalculable costs. Finally, this upgrade route is a “one-way street”—it is hard to “turn back” in the face of the more winding business changes of real-world scenarios, and there is no way to avoid the excess cost investment.
OceanBase proposed the concept of “standalone-distributed integration” to explore using a single solution to meet the different challenges an enterprise faces at different stages of development and at different business volumes—letting the database grow alongside the business and meeting the enterprise’s need to “make one choice that supports the full lifecycle of business development.”
OceanBase’s “standalone-distributed integration” architecture has two meanings in terms of deployment form (see Figure 3). The first meaning is that the same OBServer can be deployed either standalone or distributed. The second meaning is that within an OceanBase distributed cluster, there can be standalone-form tenants. Furthermore, at both the tenant level and the cluster level, the standalone and distributed forms can be flexibly adjusted and switched, so the enterprise can choose the deployment mode best suited to its current needs.
Figure 3: Illustration of the OceanBase standalone-distributed integrated product form
In the initial stage, an enterprise can deploy a standalone OceanBase on small-spec machines. As the data scale grows, it can choose to replace it with a larger-spec single machine for vertical scaling, while deployment modes such as primary-standby and three replicas support the enterprise’s disaster recovery and high-availability needs. When the data scale grows further, the enterprise can choose horizontal scaling, expanding the cluster size to support large businesses.
“Standalone-distributed integration” doesn’t sound complicated—when a database can handle complex distributed transactions, standalone deployment hardly seems difficult. But that’s not the case. On the distributed architecture front, OceanBase’s years of rich experience in distributed TP, and its record-breaking TPC-C result, have already proven its ability to handle distributed transactions. So when facing the “standalone-distributed integration” proposition, the challenges mainly lie in the “standalone” and “integration” aspects.
Challenge 1: How to Improve the Database’s Performance in Small-Scale Standalone Mode So It Is on Par With a Standalone Database?
To achieve the atomicity and durability of distributed transactions, a distributed system’s log stream design incurs greater overhead than that of a standalone database, and its performance in supporting standalone transactions and small-spec distributed transactions is significantly lower than that of a standalone centralized database. For some enterprises, adopting a distributed system would actually cause database performance to drop, so they would rather choose to vertically upgrade the machine spec of their existing database at higher cost.
Specifically, the log stream is what guarantees a database’s transactional atomicity and durability. Building on the log stream, a distributed database introduces distributed transaction protocols such as two-phase commit to guarantee atomicity, and introduces election protocols such as Paxos to guarantee durability. These operations, added on top of those of a standalone database, also bring greater overhead. On the other hand, distributed writes are multi-point, so they produce multiple log streams. The number of log streams affects the number of groups participating in two-phase commit and Paxos elections, thereby affecting the overhead of distributed transactions.
In common distributed systems, the number of log streams corresponds to the number of data shards: the larger the data volume, the larger the number of partitions, and the greater the overhead of distributed transactions and the more pronounced the impact on performance. To make a distributed database’s standalone performance comparable to that of a standalone database, you need to reduce the number of log streams.
Facing this problem, OceanBase’s approach is to reduce the number of log streams to be related to the number of nodes. In standalone deployment mode, OceanBase is architecturally no different from a traditional standalone database, with only one log stream; when scaling out in a distributed manner, the number of log streams is related only to the number of nodes, significantly slowing the growth of the distributed cost.
By reworking the log stream, OceanBase achieves performance comparable to a standalone database in both standalone mode and small-scale distributed deployment mode. As shown in Figure 4, in a small-spec 4C16G scenario, the Sysbench test results show that OceanBase 4.0’s insert and update performance can reach twice that of MySQL 8.0, with several other metrics also on par with MySQL 8.0. At the same time, in standalone deployment mode, OceanBase’s storage cost is lower. Testing shows that in a TPC-H 100G scenario, OceanBase 4.0’s storage cost is only 1/4 that of MySQL.
Figure 4: 4C16G Sysbench performance test comparison, OceanBase Community Edition 4.0 vs. RDS for MySQL 8.0
Challenge 2: How to Flexibly Adjust and Switch Between Standalone and Distributed Modes Without Sacrificing Performance?
Having solved the performance problem in standalone and distributed deployment modes, the next problem the standalone-distributed integrated architecture needed to solve was how to ensure flexible horizontal scaling without sacrificing performance.
In addition to reducing overhead by reworking the log stream so that the number of log streams increases with the number of nodes during horizontal scaling (as mentioned earlier), OceanBase also introduced several methods that let users improve scaling performance manually or automatically.
As a distributed database, OceanBase supports partitioning to meet needs such as scalability and multi-point writes—that is, the data of a table is split into multiple partitions for storage. Gathering tables with the same partitioning scheme together forms a tablegroup. When the system performs load balancing, a tablegroup helps users gather related data onto the same machine—that is, the tables in the same tablegroup are bound to the same log stream. By setting up a tablegroup, users can avoid a large number of cross-machine operations, thereby effectively reducing the data-transfer overhead in join scenarios and improving performance. In the extreme case, if all the tables involved in a transaction are in one tablegroup, then it is a standalone transaction with no distributed overhead.
OceanBase also provides automated scheduling to help improve scaling performance. For example, OceanBase supports automatically aggregating RPCs, and supports scheduling the partitions of distributed transactions together through automatic load balancing to reduce distributed overhead.
Figure 5 shows how the tpmC value changes as the OceanBase cluster scales from 3 machines to 1,500 machines in a TPC-C test. The test results show that even with 10% distributed transactions, the OceanBase cluster’s performance still scales linearly with the cluster size.
Figure 5: TPC-C performance test, changes in OceanBase’s tpmC value across different node counts
Based on the above technical optimizations, OceanBase’s standalone-distributed integrated architecture, on one hand, has the high-performance, low-cost advantages of a standalone database, helping customers reduce cost and increase efficiency; on the other hand, it has the high-availability and scalability advantages of a distributed database, helping customers better unlock the value of their data. In addition, it has three characteristics: flexible, simple, and open.
1. Flexible.
Besides the flexible switching between standalone and distributed modes mentioned above, OceanBase’s built-in multi-tenant isolation architecture enables shared instances, thereby reducing cost as much as possible. As a user’s business volume grows, the standalone-distributed integrated architecture allows them to gradually expand from a small-spec instance to a distributed instance, scaling out and in flexibly.
2. Simple.
OceanBase 4.0 aims to reduce the complexity of database selection and operations, using a single database to meet the needs of different scenarios as much as possible. On one hand, the architectural differences between standalone and distributed can be unified through the dynamic log stream scheme mentioned above; on the other hand, a single system can handle multiple data models such as OLTP, real-time OLAP, Key-Value, JSON, and GIS, avoiding the need for users to adopt multiple systems due to insufficient infrastructure capabilities.
3. Open.
Early versions of OceanBase supported on-premises deployment, and it later gradually supported multi-cloud deployment and chose an open storage-compute separation scheme. Specifically, it discards the SQL-and-KV layering scheme: when the database reads and writes data blocks, it supports various local or remote distributed file systems—the database does the compute, and the file layer does the storage. The biggest difficulty of this approach is that it requires integrating high-availability and scalability handling into distributed transactions, which is complex to implement; the benefit is that it guarantees both performance and openness—the database’s internal SQL and KV modules no longer need RPC interaction, and it can adapt to various storage systems on multi-cloud platforms. In addition, OceanBase reduced its dependence on external file systems when designing its storage format, so users can deploy the database to different cloud platforms or even multiple clouds according to their actual business needs, while ensuring system stability and high performance.
3. One Choice Supports the Full Lifecycle of Business Development
For users and developers, the standalone-distributed integrated architecture seems to do everything—both standalone and distributed—so what exactly is the focus at this stage? In the short term, OceanBase’s standalone-distributed integration capability has value and significance for users and developers in the following two dimensions.
(1) Greatly lowering the barrier to distributed databases. Earlier NewSQL had poor standalone performance: the standalone sysbench performance of mainstream NewSQL systems in the industry, such as CockroachDB and YugabyteDB, was only 1/5 to 1/10 that of MySQL (data as of 2023). As standalone-distributed integrated databases gradually mature, this kind of NewSQL will be gradually replaced. At the same time, even in distributed deployment scenarios, compared with other layered distributed databases, OceanBase has a big performance advantage; with fine-grained control, once the proportion of standalone transactions increases, its efficiency can be on par with a standalone database. Many users have already replaced NewSQL systems with OceanBase to reduce cost and increase efficiency.
(2) Solving users’ need to scale from small to large. Most small and medium-sized enterprises have different requirements for database scale and performance at different stages of their business development. OceanBase can be deployed in standalone mode to provide capabilities on par with a standalone database, and at any moment, the administrator can change its form—from a standalone form to a distributed form, and from a distributed form back to a standalone form—switching dynamically. By comparison, even if the data volume at the current stage of the business isn’t large and a standalone database is enough to support it, choosing a standalone-distributed integrated database from the very start is clearly a better choice than modifying the application and switching databases after the business grows.
In summary, through native distribution, OceanBase guarantees the ACID of distributed transactions while meeting scalability needs, supporting enterprises in rapid iteration and upgrades transparently to the business. Through the standalone-distributed integrated architecture, it effectively supports the scale and demand changes of an enterprise’s full lifecycle, growing alongside the business and fundamentally solving the enterprise’s database application challenges.
“Lao Ji’s Tech Talk” hopes not only to keep bringing you valuable technical content, but also to contribute to the open source community together with all of you. If you appreciate the OceanBase open source community, give it a little star ✨! Every Star you give is the motivation behind our efforts.