OceanBase: Du Xiaoman Breaks Down Database Selection Across Five Technical Scenarios, Doubling Cost Savings, Performance, and Efficiency
This article is adapted from the June 21 session “OceanBase City Meetup · SQL Meets AI — Du Xiaoman × OceanBase Practice: A Unified Architecture Driving Breakthroughs in Both Efficiency and Cost“. Click the link to watch the video replay.
Du Xiaoman, formerly Baidu Finance. In April 2018, Baidu announced that its financial services business group had formally completed its spin-off financing agreement and begun operating independently. As a FinTech company, Du Xiaoman fully leverages Baidu’s AI strengths and technical capabilities, partnering with financial institutions to deliver better financial services through technology. This article describes the storage challenges that came with Du Xiaoman’s rapid business growth, along with its database solution.
Four Cost Advantages That Determined the Database Choice
The rapid expansion of Du Xiaoman’s financial business drove exponential growth in storage demand. At that point, the underlying database solution faced challenges including high-concurrency transactions from hundreds of millions of users, millisecond-level real-time risk-control decisions, high-throughput variable data writes, and extremely low-latency reporting and analytics. After an in-depth analysis by the big data team, we identified five problems with the storage architecture that made it difficult to meet these challenges and continue supporting the business.
First, Du Xiaoman had many homegrown technology stacks, such as the relational store DDBS, the KV store CKV, and the sparse massive-scale store Eggroll, among others. For frontline developers, the learning curve was steep—they had to integrate with and learn multiple usage patterns, leaving room to improve development and integration efficiency.
Second, resource costs were high: the host fleet was large, primary-secondary-standby deployments were redundant, and there were many small clusters, resulting in high storage costs, low CPU utilization, and resources that were hard to reuse.
Third, certain operations such as traffic switching and blocking, data rebalancing, and scaling were complex, opaque to the business, required coordination across multiple parties, took a long time to execute, and in some cases even needed custom solutions.
Fourth, there were availability risks: across all scenarios—process hangs, hardware failures, data center failures, regional disasters, and the like—none of the storage engines could reach a “four nines” (99.99%) SLA.
Fifth, the tooling ecosystem was lacking. Tool automation, and especially the degree of productization, was relatively low, leaving room to improve operational efficiency.
Given the business requirements and the difficulties of the existing database solution, we decided to choose a new product that could both meet business needs and unify all of our current technology stacks. So why did we ultimately choose OceanBase?
After an initial survey of several databases across migration cost, resource cost, learning cost, and operational cost, we found that OceanBase fit our selection requirements very well.
1. Low migration cost: compatible with the MySQL and Redis protocols, making migration smooth and seamless.
To unify the underlying architecture stack, the first requirement when replacing our database solution was that the migration cost could not be too high—otherwise we would face significant resistance from the business. Because OceanBase is compatible with the MySQL and Redis protocols, the entire migration was very smooth, with almost no rework, and we invested relatively little labor and time. It delivered positive returns to the business and thereby indirectly lowered the overall migration cost.
2. Low resource cost: an efficient execution engine, a TPC-C world record, and an ultra-high compression ratio.
OceanBase has a world-leading compute execution engine and once broke the TPC-C world record. Its extreme data compression ratio is also well known in the industry: compared with a traditional database like MySQL, it can save several times the storage space, dramatically lowering enterprise storage costs and delivering a significant improvement in resource utilization.
3. Low learning cost: a unified technology stack for relational, KV, vector, and more, with unified operations across multiple deployment architectures.
As mentioned above, our many technology stacks and components imposed a heavy learning burden on frontline developers and operations staff. OceanBase uses a single engine to handle the storage and computation of multiple data types—relational, KV, vector, and more—meeting the needs of all business scenarios, greatly reducing the learning cost and improving operational efficiency.
4. Low operational cost: data center and region-level disaster recovery, RTO < 8s, RPO = 0. Native distributed support, with scaling transparent to users.
OceanBase has a complete ecosystem of surrounding tools, including OCP (the cloud platform), OMS (data migration service), ODC (developer center), and OBD (installation and deployment tool), providing all-around support from installation and deployment, data migration, and development tools to operations management, monitoring, and alerting. On top of online scaling, the OCP cloud management platform enables very convenient, point-and-click scaling, greatly improving our operational capability, user experience, and work efficiency. In addition, OceanBase natively provides high availability. Our own testing showed that when a fault occurs, the business completes its switchover within 8 seconds with zero data loss, proving that OceanBase truly achieves RPO = 0 and RTO < 8s.
Five Scenarios Exploring the Feasibility of OceanBase in Production
How does OceanBase actually perform in real-world scenarios, and can it solve the data latency and query performance problems we faced before? The following five technical scenarios illustrate the answer.
Scenario 1: Low-Latency Queries Over Massive Data
In a query scenario where a single table holds about 1 billion rows of monthly data occupying roughly 30 GB of memory, queries typically touch only data within the month and the statements are complex. With our original MySQL solution, we had to split one table into many tables to achieve decent performance. But this made the business logic and usage logic more complex, failed to meet business needs, and incurred high storage costs—none of which aligned with the overall business goals.
After switching to OceanBase, in May 2025 a single table stored nearly 1 billion rows occupying 29.5 GB of memory. In the most response-intensive update scenario, it took 8 milliseconds—in line with our expectations—while select and insert scenarios took even less time. For overall business operation latency, the OceanBase solution kept it within 10 milliseconds, fully meeting business requirements.
In this scenario, our takeaway was: because OceanBase is a natively distributed architecture, we recommend using its partitioning capability for scenarios that would otherwise require manual table sharding, which lowers the rework cost for frontline staff and the cost of designing business logic. In addition, for complex query statements—for example, when an in clause contains too many values—you should bind the execution plan to avoid full table scans that lead to excessively long execution times.
Scenario 2: High-Performance Read/Write for Data Warehouse Processing
High-performance read/write for the real-time data warehouse was another business scenario that needed optimization, involving 100k writes + 360k reads, where queries spanned both TP point lookups and AP aggregation/join queries, and where existing data and incremental data lived in two different storage systems. Originally we used MySQL’s DDBS distributed storage for real-time read/write of incremental data, because accessing existing data created significant performance pressure when the data volume was large. For existing data, we used Eggroll, but since Eggroll is disk-based storage—essentially a simple RocksDB-based distributed architecture—its scaling cycle was relatively long.
The figure below shows our data warehouse processing pipeline. Before the overhaul, business data sources flowed through Kafka and Flink, read and wrote MySQL at the ODS layer, then read Eggroll, and went through multiple rounds of Kafka writes, with the pipeline running from the ODS layer to the DWD layer and then the DWS layer. Whoever developed this data pipeline had to be familiar with both the DDBS and Eggroll architectures and understand the entire chain from Kafka to Flink to Hive, involving multiple engines and pipeline processing stages, which made the delivery cycle very long.
After the overhaul, we switched DDBS and Eggroll to OceanBase and gained substantial benefits in overall architecture, delivery efficiency, scaling, and cost.
- Simplified architecture: instead of maintaining two storage engines, we now maintain a single interaction pipeline, and engineers writing this code only need to be familiar with one set of logic and interfaces to complete the entire pipeline.
- Double the delivery efficiency: beyond improved coding efficiency, the learning cost dropped sharply, and the development cycle shrank from 6 days to around 3 days—doubling delivery efficiency.
- Improved scaling efficiency: a scale-out that used to take days or even weeks can now be completed within hours after switching to OceanBase, which supports online scaling.
- Over 3x cost benefit: DDBS and Eggroll did not compress data well. After switching to OceanBase, the compression ratio reached 4:1, delivering a large cost benefit; combined with the performance gains, the overall cost benefit exceeded 3x.
Of course, during this overhaul we also accumulated some technical lessons worth sharing.
First, when using OceanBase, you should avoid cross-partition RPC communication as much as possible. If the number of RPC round trips is high, query performance degrades noticeably, so you need to design the business model as sensibly as possible—for example, including partition information in the SQL so it hits the local node, yielding the best possible performance.
Second, write conflicts on hot data. Because Flink synchronizes data in real time, if we update a variable, that variable is also read and written in real time, leading to write conflicts on hot data and performance degradation. We therefore recommend enabling the Early Lock Release feature for better overall performance.
Third, the impact of high-pressure writes of existing data on result point lookups. Because batch existing data must be loaded and updated daily, it contends for resources with normal business requests, lengthening their response times. Our solution was to apply resource isolation and control to the existing-data write tasks, further resolving the problem.
Going forward, we plan to use obbinlog in place of the existing Kafka data transmission pipeline, collaborate with the OceanBase community to optimize obbinlog’s performance, and build a monitoring pipeline based on Flink, OceanBase, and its tools—reducing the architectural complexity of the real-time data warehouse pipeline and further improving query and delivery efficiency.
In addition, we hope the latency of AP aggregation/join queries can be resolved, so in the future we will explore OceanBase’s hybrid row-column storage. We’ve already noted that OceanBase has released its first AP-oriented LTS version—V4.3.5—and we will continue to track how this version matures.
Scenario 3: High-Throughput Writes of Variable Data
Du Xiaoman has many scenarios where variables are processed on a T+1 basis; after processing, the variables must be loaded into storage to serve online business queries. Our original approach used HBase’s Bulkload capability to update the storage engine Eggroll. After Eggroll was replaced by OceanBase, we began using OceanBase’s OBLoader script.
We did run into some problems using the OBLoader approach. Because DGS is our internally developed object store with a unique protocol, it isn’t fully compatible with the OBLoader script—we first have to download data locally and then load it with the script, which is inefficient. But when updating T+1 data, we need to fully write all of a day’s data within that day; a single script loads too much data and the timeliness requirements are very high. When efficiency falls short, data that must be written within the day can’t get written, so the current approach can’t meet actual business needs.
For this, OceanBase officially recommended the bypass SDK approach, directly integrating with our homegrown DGS protocol to read remotely and load directly, avoiding an extra disk-write step. At the same time, for very large data volumes, it leverages K8s elasticity to dynamically launch multiple processes for sharded loading, improving overall throughput and ensuring the solution fits this scenario.
Scenario 4: Hybrid Queries Combining Scalar Filtering + Vector Search
In the AI wave, hybrid queries combining scalar filtering + vector search will become the norm. For this scenario, we currently rely on Elasticsearch, but we are actively exploring the possibility of switching to OceanBase and have begun performance testing.
In the performance tests, we used two test sets commonly used in the industry:
- Dataset: cohere 768 1M (768 dimensions, 1M vectors). With Limit 10, recall 0.9, the HNSW index type, and concurrency 50, in a baseline performance comparison OceanBase was 1.24x the performance of Elasticsearch.
- Dataset: cohere 1536 500K (1536 dimensions, 500K vectors). With Limit 10, recall 0.9, the HNSW index type, and concurrency 50, in a baseline performance comparison OceanBase was 1.04x the performance of Elasticsearch.
Based on the test results, OceanBase delivers a 10%–20% performance improvement in AI scenarios. We have already started a pilot with the business team and will further drive OceanBase’s adoption in AI scenarios.
Scenario 5: Extreme KV Read/Write Performance
In our KV scenario, the frontend faces 1.2M QPS of query pressure, which maps down to the underlying storage. We set up a three-node OceanBase cluster, and after testing, OceanBase delivered 600k QPS with an average latency around 1 millisecond, with persistent storage—fully meeting our business needs. OceanBase also persists existing data on disk, improving system stability and availability, with clear advantages over our original CKV solution. We therefore plan to officially roll out OceanBase in the KV scenario in the near future.
Summary and Outlook
For Du Xiaoman, adopting OceanBase isn’t just about replacing a traditional relational database to solve performance and cost problems. The greater significance is unifying many online storage technology stacks—relational, KV, AI vector, and more—achieving a double boost to both development and operations efficiency.
Looking further ahead, our internal tests indicate that OceanBase outperforms other mainstream vector databases, but these results have not yet been independently verified by a third party, such as the VectorDBBench third-party benchmark project. We hope OceanBase will pursue further validation, and we also look forward to deeper discussions with OceanBase’s experts, ultimately applying this in real business.
“Lao Ji’s Tech Talk” not only aims to keep bringing you valuable technical content, but also hopes to contribute to the open source community together with everyone. If you appreciate the OceanBase open source community, please light up a little star ✨! Every Star you give is what motivates us.