The Data Foundation for Zuoyebang's AI Business: OceanBase Vector Database Multi-cloud Deployment and Architecture Design Optimization

Author: Zhang Hengyan, Head of the DBA Team at Zuoyebang

Zuoyebang is a leading online education platform in China that deeply integrates technologies such as artificial intelligence and big data with teaching and learning, creating smart-education solutions that cover the full spectrum of teaching, learning, testing, evaluation, management, and research. At the technical foundation, Zuoyebang has continually sought databases that can power business growth. Since starting to evaluate OceanBase in 2022, it has adopted OceanBase at scale across many core workloads. This article details Zuoyebang’s database solutions and operations experience for its AI business and multi-cloud architecture.

AI Business Challenges and Solutions

As AI technology has exploded in popularity and deeply integrated into every industry, Zuoyebang has gradually introduced a variety of AI-driven features into its business scenarios, such as intelligent customer service, Q&A bots, and AI writing. These scenarios place higher demands on the RAG (Retrieval-Augmented Generation) architecture and vector knowledge base, involving key technical requirements like large-scale vector storage, high-concurrency retrieval performance, and low-latency real-time response. At the same time, given Zuoyebang’s existing multi-cloud architecture, we faced two main challenges when selecting and adapting the AI technology foundation.

Challenge 1: LLM content storage and moderation.

Have you ever felt this way? Once a business makes heavy use of large models, it rapidly consumes database storage space. Zuoyebang has many internal clusters, with daily data growth reaching the 10TB level, sharply increasing storage cost pressure. Meanwhile, in scenarios like AI writing and intelligent Q&A, the massive AI-generated content and multi-turn user conversations not only require hot and cold data to coexist but also need moderation. To relieve storage pressure, we usually clear out cold data—which, in a MySQL database, creates a lot of disk fragmentation.

Challenge 2: A surge in vector-database demand from RAG.

To ensure users get faster, more accurate search results when using the product, the business side wanted to add a vector database to support better product capabilities and user experience.

At first, the DBA team leaned toward purchasing cloud services to quickly meet the business needs. But the reality didn’t allow it. Zuoyebang’s application services use a multi-cloud architecture, and purchasing PaaS services on the cloud makes multi-cloud deployment difficult. Building a self-managed multi-cloud database service would cost the DBA team too much effort and money. Considering our cluster landscape—a few large clusters and many fragmented small clusters—and the fact that different businesses have widely varying scale requirements for the vector database, we needed to deploy a vector database that could be used flexibly according to business scale, to avoid wasting resources.

Against this backdrop, when choosing a vector-database solution we considered two options: Milvus and the OceanBase vector database. For the former, adding a new self-managed database type would require substantial build-out work, with high labor and time costs. Since other Zuoyebang workloads had already used OceanBase back in 2022, we didn’t need to rebuild the tech stack, so we chose OceanBase to support our AI business.

So how does OceanBase solve the above technical challenges?

First, in large-model scenarios, our tests showed that OceanBase uses about 1/6 the storage space of MySQL, enabling extreme compression of storage costs. Thanks to OceanBase’s distributed nature, a single cluster carries more than twice the data of MySQL. In addition, OceanBase’s merge mechanism can defragment, avoiding disk-fragmentation issues.

Second, for the vector-database need, we used the vector capabilities of OceanBase 4.3.5. OceanBase supports multi-cloud, so we didn’t need to rebuild anything and could quickly take on the AI business; its native multi-tenant architecture also significantly improves resource utilization. In mid-November, OceanBase released seekdb, a lightweight AI-native database that better handles the vector-data processing needs of fragmented small clusters.

Another point worth mentioning: the OceanBase community is active and offers stronger support than other open-source databases. When we rolled out OceanBase in our AI business, the Community Edition team helped us in many areas—such as query performance optimization, recall optimization, and SDK usage issues—significantly shortening the vector database’s time to launch.

Zuoyebang’s Multi-cloud Architecture Design and Optimization

As mentioned, Zuoyebang uses a multi-cloud architecture. When deploying OceanBase for the AI business, we had to consider several factors holistically:

• The problem of dedicated-line failures between multiple data centers and clouds;
• The business’s demand for cost reduction in a multi-cloud architecture;
• The integration of OceanBase with cloud-native services. Zuoyebang completed its cloud-native transformation early on, with the business largely deployed in K8s, while OceanBase is not recommended for deployment in K8s clusters.

Multi-cloud Architecture Selection

For the business tenant architecture, the candidate options are shown in the figure below. On the left is Option 1: deploy 3 ZONEs across three clouds. On the right is Option 2: build a multi-cloud cluster via cross-cloud primary-standby tenant replication.

We ultimately chose Option 2, because:

• The cross-cloud dedicated line takes about 5ms; Option 1 would have an obvious performance penalty, while Option 2 performs better.
• Primary-standby tenant switchover is controllable, so we can switch flexibly when a fault occurs.
• It handles cross-cloud dedicated-line failure scenarios. Suppose the dedicated lines among the three clouds fail and each cloud becomes an island—Option 2 handles this situation better.

Multi-cloud Architecture Design: Read-Write Splitting

Considering that the standby tenant’s data center can serve local reads—reducing cross-cloud bandwidth and latency—and that having the standby tenant take on some read traffic optimizes cost, we added a read-write splitting design on top of Option 2.

Through a self-developed proxy and ODP operations, we route business local reads—scheduling the read traffic in the standby tenant’s data center to the standby tenant—to achieve local reads. Because the OCP of the OceanBase 4.2.5 version we currently use isn’t capable of sensing the lag between primary and standby tenants, we made targeted modifications to automatically pull traffic off the standby tenant when it lags. This brings two benefits.

First, it improves resource utilization. Because the standby tenant has the same resource scale as the primary tenant, the business effectively has 50% capacity redundancy in the cold standby. By routing 50% of read traffic to the standby tenant via local reads, we fully utilize the standby tenant’s resources.

Second, it avoids consistency risk. Primary-standby tenant switchover often faces a problem: with no traffic on the standby tenant, there can be data-consistency or performance-load risks. Routing 50% of read traffic to the standby tenant therefore also makes the switchover more reliable.

For OCP’s deployment, we used a fairly traditional three-cloud deployment. This is because our performance requirements for OCP aren’t high and we can accept the cross-cloud performance penalty. At the same time, since the primary-standby tenant switchover logic depends on OCP, if OCP in turn depended on primary-standby tenant switchover, it would create a circular dependency. To avoid this, we needed to decouple the interdependencies among high-availability components.

Multi-cloud Architecture Design: Cloud-Native Proxy

Because OceanBase is not recommended for deployment in K8s, we added a cloud-native proxy layer between the client and the ODP operations platform. This is a lightweight proxy deployed inside the K8s cluster, serving two purposes:

• First, it doesn’t fully handle the MySQL or OceanBase protocol; instead it’s used for cloud-native service discovery and service observability.
• Second, when migrating from MySQL to OceanBase, the OceanBase username length may be constrained by frameworks, or special characters may cause compatibility issues; the cloud-native proxy can handle OceanBase authentication packets with no performance penalty. Users can connect to OceanBase using the same username and password as MySQL.

Besides the cloud-native proxy, we also strengthened ODP’s high-availability mechanism at the cloud-native layer. This is because ODP sometimes hits situations where L4 liveness checks pass but SQL can’t actually be returned; our approach is to add protocol-layer liveness checks to ensure ODP’s high-availability mechanism is fully reliable.

Multi-cloud Architecture Design: Tenant Isolation and Single-Cloud Closed Loops

To better troubleshoot problems, we set up tenant isolation and single-cloud closed loops in the access path. First, we reorganized the many-to-many relationship between ODP and tenants/clusters into a one-to-one or one-to-many model, because different tenants’ ODP and cloud-native proxies are deployed independently, avoiding mutual interference and lowering troubleshooting difficulty. Second, we set up a single-cloud closed loop from the client to ODP, so that only the path from ODP to OB Server crosses clouds, reducing cross-cloud latency and bandwidth usage.

A multi-cloud solution needs to be paired with regular drills, of which we usually run two types. One is routine primary-standby tenant switchover: we pick one of the business’s clusters and switch primary and standby tenants during off-peak hours, with the business side perceiving only a momentary blip. The other is a half-yearly drill that tests data-center outages by performing a disaster-recovery switchover of primary and standby tenants.

Business Application Scale and Large-Scale Operations Experience

OceanBase is now live across multiple core Zuoyebang workloads, with a total of 40+ clusters, 200+ tenants, and 20,000+ cores deployed—and still growing fast. It took three years to go from validation to large-scale OceanBase deployment.

In 2022, when OceanBase 4.0 was released, we noticed that its capabilities could solve our problems with multi-cloud deployment of distributed databases, so we engaged in deep discussions with the OceanBase community. From configuring OceanBase’s surrounding tools in 2023 to officially going live in our AI and overseas businesses in 2024, we ran comprehensive adaptation tests on OceanBase, covering compatibility, performance, stability, and more.

This year, our main work was accelerating the migration of core services to OceanBase. Because Zuoyebang’s resource usage approaches tens of thousands of cores, we needed to build matching large-scale operations capabilities to support the large-scale rollout of OceanBase for core workloads. Along the way, we accumulated a great deal of large-scale operations experience.

1. Integrate OCP into the Database Operations Platform

Our top recommendation is to integrate OceanBase’s operations platform, OCP, into the enterprise’s database operations platform. It delivers important value in many areas and lowers operational complexity.

First, OCP helps standardize configuration and speeds up problem discovery and resolution. Zuoyebang’s multi-cloud architecture is complex and includes many custom logical relationships; although these can’t be maintained inside OCP, OCP can be used to coordinate and manage these custom relationships, features, or tasks to enable rapid deployment. For example, users can deploy OBServer, ODP, and the self-developed proxy with one click. At the same time, because the resources used on each cloud—such as physical-machine specs—are inconsistent, OCP can also standardize tenant resource specs, effectively reducing the risk of tenant resource fragmentation.

Second, building custom operations features on top of OCP is very convenient. For example, sometimes we need to monitor certain special metrics; even though OCP’s monitoring and alerting features are already mature, they don’t include such a metric out of the box, so we need to build it ourselves. Or take batch switchover of primary-standby tenants: switching them one by one is too time-consuming, so we need to implement batch switchover in the operations platform. In these scenarios, building operations features on top of OCP is very convenient and gives us the capabilities we need.

Through the OCP operations system, Zuoyebang’s operations services have already achieved self-service for engineers, including cluster requests, approvals, cost allocation, horizontal/vertical resource scaling, as well as SQL querying and auditing—better supporting operations for large-scale workloads.

2. Use the Latest Stable Version of OceanBase

For those using OceanBase 4.X, we recommend upgrading directly to the latest stable version. We initially used OceanBase 3.x, later upgraded to 4.x, and went through V4.2.1.2 → V4.2.5.2 → V4.2.5.6, plus the latest V4.3.5 in our AI business. The lesson learned: newer versions offer the best performance, stability, and support for other features. For AP and vector capability needs, choose a version with stronger AP capability or stronger vector capability.

Besides choosing the right version, we also assign different clusters to different business types, effectively treating OceanBase clusters as resource pools:

• For core workloads, provide dedicated clusters to reduce interference between businesses;
• For non-core or small workloads, provide shared clusters to reduce machine costs and onboarding costs. We also provide a self-service migration path for the business so that, as a workload grows or becomes more important, it can smoothly migrate from a shared cluster to a dedicated one.

3. An Approach to Uneven Data Distribution

In some scenarios, we may encounter uneven data distribution. For example, one of Zuoyebang’s core clusters contains 4 tenants; during horizontal scaling, OCP reported that initiating a unit_num scaling operation for one user tenant succeeded. After all 4 tenants’ scaling tasks completed, we found that one node’s disk data kept growing until it filled up.

After investigation, we learned that when OCP considered the scaling task complete, the balancing algorithm had already laid out the logical distribution of the log streams, but the log-stream entities hadn’t yet reached their corresponding locations and still needed to migrate. As a result, we saw the corresponding disk data on that machine keep climbing. At the same time, because the V4.2.5.2 version we used then only considered partition count, the scaling process amplified data skew, ultimately filling up the disk space.

Our approach was to adjust the unit_group of the log streams one by one to resolve the full-disk issue, and to use the database platform to require that tenant scaling proceed to the next round only after the log-stream entities have finished migrating in the background.

For this kind of situation, we need to avoid imbalance caused by bad cases during log-stream splitting. OceanBase 4.2.5_bp4 fixed this issue, so you can upgrade the database to V4.2.5_bp4 or higher. We also recommend confirming that the load-balancing tasks for the Balance-related tables have completed, triggering a partition rebalance, and waiting for it to finish before adjusting other tenants’ unit_num.

4. Avoid a Tenant unit_num Scaling That Never Finishes

This is a common problem with older versions. You can check the transfer task history table; if it shows error code -7114, it means an active transaction during the transfer prevented the unit_num scaling from finishing. Solutions for this problem:

• For versions before V4.2.5_bp2, wait for active transactions to complete before performing the transfer, or have the transfer actively kill active transactions.

alter system set _enable_balance_kill_transaction = 'true' tenant='xxx';

• For V4.2.5_bp2 and V4.3.5 or higher, we recommend enabling the feature that lets transfers proceed without killing active transactions.

alter system set _enable_active_txn_transfer='true' tenant='xxx';

5. How to Resolve OMS Sync-Pipeline Lag

OMS is our commonly used data-migration tool. If a cluster carries very heavy traffic, the data-sync pipeline may lag. We hit this problem when using OMS to sync data from OceanBase to Kafka.

The main cause is that all of Zuoyebang’s core services must go through the big-data platform’s data collection; when upstream write pressure is too high, it causes downstream data-sync lag. Because Zuoyebang has a multi-cloud cluster architecture, the optimization is fairly complex, and specifically includes:

• Scheduling the store and incr-sync of the multi-cloud OMS to instances closer to the upstream and downstream.
• Controlling the write pressure of a single pipeline by splitting tenants, splitting tasks, and so on.
• Using the oms connector_utils.sh tool to analyze performance bottlenecks and modifying the relevant configuration per its suggestions:
  • Increase concurrency: sink.workerNum=64;
  • Increase memory: coordinator.connectorJvmParam 20-30gb;
  • Set source.useBetaListener=true to use LogMessage for faster parsing and fewer intermediate objects;
  • Set source.useSchemaCache=true to use the Schema cache and reduce intermediate objects.

Summary

Overall, the key turning points for OceanBase’s adoption in Zuoyebang’s AI business were twofold: on one hand, the large-model business brought exponential growth in stored data that the original MySQL could barely support; on the other, RAG services drove demand for a vector database. OceanBase’s functionality, performance, stability, and other test results all met our expectations, genuinely solving business pain points and fitting our multi-cloud architecture.

After adopting OceanBase, large-disk storage costs dropped 40%–50% compared to MySQL, and amid rapid business growth, we avoided frequent sharding. In addition, we delivered a standards-based multi-cloud vector-database solution that improved R&D efficiency—truly demonstrating what it means for database technology to power business growth.

Today Zuoyebang’s database SLA reaches 99.99%. With OceanBase’s operations tools, we’ve built automated operations and automated migration, our operations-assurance system keeps improving, and we’ve achieved self-service operations for engineers.