Escaping Three Major Dilemmas, Cultivating Five Key Scenarios | How TAL Education's AI Business Cut Costs by 86% with OceanBase

This article is excerpted from the e-book A Study of OceanBase Community Edition Use Cases in Pan-Internet Scenarios. Click the link to get the full content.

TAL Education Group focuses on education, offering both traditional quality-oriented education and digital education, with well-known brands including Xueersi Suyang, Xueersi Online School, and Bichi. The company’s business spans offline face-to-face courses, online courses, and after-school care services. We offer a range of humanities-and-arts courses and science-and-engineering courses aimed at promoting the all-round development of children. The company is currently making an active push into the smart-hardware business and has received a positive reception in the market.

At the same time, the company has invested heavily in the large-model field. In 2023, we released MathGPT, the country’s first large model in the mathematics domain; in 2024, we launched “Ask Anytime,” an app built on that large model. It not only gives direct answers but also breaks down the knowledge points behind the questions users ask, guiding them step by step to solve problems—with the focus on cultivating the user’s own problem-solving approach.

As a leading EdTech enterprise in China, with numerous business lines and massive amounts of data, our underlying data systems involve MySQL, Redis, MongoDB, Pika, RDS, PolarDB, CDB, TDSQL-C, and others, along with a multi-cloud architecture spanning Alibaba Cloud, Tencent Cloud, Baidu Cloud, and more. Pressured by the predicament of sharding and plagued by problems such as resource isolation and resource fragmentation, we added OceanBase to our tech stack to solve these issues. After piloting it, we not only improved availability and resource utilization, but also reduced operational complexity and gained additional cost benefits.

I. Three Pre-Selection Dilemmas and Four Selection Factors

For an enterprise with diversified business, choosing the right database is critical to the company’s growth. Because the group has a rich variety of business lines, each one chooses its database based on its own actual business scenarios, and the group’s database services are provided internally in a hybrid-cloud model. We have built a large number of database services in-house within our IDC, and we also use various database products provided by multiple cloud vendors, covering a fairly broad range of database types. After using so many database products, why would we still consider adopting OceanBase? There are three reasons.

First, our self-built services include a large number of MySQL instances. As a traditional standalone database, MySQL easily hits performance and capacity bottlenecks. At the same time, traditional middleware-based sharding solutions provide unfriendly support for distributed transactions and add complexity on the operations side.

Second, traditional databases are mainly deployed in a single-machine multi-instance fashion and lack resource-isolation capabilities, which sows hidden risks for the stable operation of online business. Moreover, because the control plane faces very scattered physical-machine resources, and resource allocation and reclamation only modify metadata, a large amount of resource fragmentation is inevitably produced.

Third, the current resource-deployment model lacks elastic-scaling capability. Resources are always allocated with a large amount of redundancy, leading to low overall utilization of database service resources and serious resource waste.

Given the dilemmas above, when selecting a new database we mainly focused on four capabilities.

(1) Architectural Design Determines the Database’s Performance and Disaster-Recovery Capability

Figure 1 is the architecture diagram of OceanBase 4.x, which has the following characteristics.

Figure 1: OceanBase 4.x architecture

(1) Multiple replicas: typically deployed as 3 Zones or 5 Zones, with each Zone composed of multiple server nodes (OBServers);

(2) Peer nodes: each node has its own SQL engine and storage engine, independently manages the data partitions it carries, communicates over TCP/IP, and serves collaboratively;

(3) No shared storage devices required: data is distributed across the nodes and does not rely on any device-level shared-storage technology, so no SAN network is needed;

(4) Partition-level availability: the partition is the basic unit of reliability and scalability, automatically achieving access routing, policy-driven load balancing, and autonomous fault recovery;

(5) High availability + strong consistency: an efficient, highly reliable engineering implementation of multiple replicas plus the Paxos distributed protocol, ensuring that data (logs) are durably persisted on a majority of nodes.

OceanBase v4.x’s standalone-distributed integrated architecture allows the majority of business requests to execute locally as single-machine transactions, avoiding the overhead of distributed transactions. Furthermore, if you set priorities for availability zones or Zones, you can even have all of a tenant’s requests turned into single-machine transactions executed locally. This performance advantage is very attractive for the business.

(2) Multi-Tenancy Capability for Resource Isolation and Flexible Configuration

As shown in Figure 2, OceanBase provides native multi-tenancy capabilities, supporting per-tenant configuration of data replica count, replica type, storage location, compute resources, and more. At the same time, OceanBase supports dynamic scaling of individual tenants, allowing online expansion and configuration adjustment. It provides automated operations within the cluster, guaranteeing complete resource isolation between tenants and relatively secure data isolation between tenants—helping us eliminate the risk of online resource mixing.

Figure 2: OceanBase provides native multi-tenancy capabilities

(3) Storage-Compression Technology for Additional Cost Benefits

OceanBase provides very advanced data-compression technology. Beyond solving performance and availability problems, it can also deliver substantial cost benefits, thanks mainly to the following three key technologies.

(1) Storage compression based on variable-length and fixed-length data: by using compression algorithms with a high compression ratio and fast decompression, the data compression multiple is increased and storage costs are reduced. Because of the structural characteristics of the LSM-Tree, a read-write separation design and row-level fine-grained record updates are adopted: changed data is kept in memory and written to disk in batches. This achieves the write performance of an in-memory database together with the storage cost of a disk-based database, while eliminating the random-write bottleneck and storage-fragmentation problems of the traditional B+Tree—giving higher write performance than the traditional approach of updating data blocks in real time.

(2) Storage compression based on data encoding: using a hybrid row-column storage format, disk data blocks are organized by column, and a self-developed encoding compression method (encoding) for hybrid row-column storage is applied. Using dictionary, delta, prefix, and other encoding algorithms across rows and columns, the data is encoded and compressed before the general-purpose compression algorithm runs, yielding an even higher compression ratio.

(3) Low-cost storage based on data-log separation: in the traditional Paxos protocol, the system needs three replicas (or five). OceanBase separates user data from log data—for example, log data uses three replicas (or five) under the Paxos protocol, while the user data itself can be stored using two replicas (or three or four). Under the same availability guarantee, data-log separation can save 20%–40% of user-data storage cost.

In addition, OceanBase has the following excellent characteristics: dynamic modification of the write memory, no modification of static data, support for batch writes of highly compressed data, strong data-consistency verification, and SSD-friendly elimination of random writes.

(4) Ecosystem Tools for Building a New Database-Operations System at Minimal Cost

Beyond its very powerful kernel capabilities, OceanBase also provides a rich set of ecosystem tools (see Figure 3) covering the full lifecycle of service operations, such as assessment and transformation, real-time migration, development management, production operations, replication and subscription, security control, and diagnostics and autonomy. With these powerful tools, we can build out our OceanBase operations system at very low cost.

Figure 3: OceanBase operations system

Among these, the tool we use most is OCP (OceanBase Cloud Platform). With OCP, most of our daily operations can be done through a graphical interface—creating clusters, adding availability zones, adding tenants, and so on—all completed simply by clicking buttons on OCP’s pages, a process that is both simple and reliable. OCP also provides monitoring and alerting, performance analysis, and data backup, turning OceanBase operations into a simple and pleasant task.

We also noticed that OceanBase provides good support for the MySQL ecosystem. In MySQL-compatibility mode, the Binlog Service can convert OceanBase’s logs into the MySQL Binlog format, achieving full compatibility with the MySQL Binlog protocol. The business can continue using data-synchronization tools from the MySQL ecosystem to complete data-subscription tasks, greatly reducing the cost of business modification.

II. Solving Database Pain Points Across Five Major Business Scenarios

From late 2022 to early 2023, TAL Education Group began building out its OceanBase infrastructure, after which the first business started its pilot run on OceanBase; the rollout situation is shown in Figure 4. OceanBase has now been running stably for more than two years, demonstrating extremely high stability with no online failures or issues during this period.

Figure 4: OceanBase’s application at TAL Education Group

Based on OceanBase’s stable performance, we plan to dig deeper into its unique and excellent features and migrate more business from other databases to OceanBase. Next, we’ll walk through OceanBase’s use across five business scenarios at TAL Education to explain TAL’s database-upgrade approach.

(1) Learning-Machine Scenario: Lower R&D Costs, Improved Stability

The “learning-machine business” is one of TAL Education Group’s more important database application scenarios at present. In just two years since its launch, the learning-machine business has experienced rapid user growth. However, unlike a typical internet business, almost all learning-machine users are paying users with a relatively high average order value. As a result, this business currently faces two major challenges: first, rapid data growth; second, extremely high requirements for data reliability.

For this important learning-machine scenario, when selecting a new database we mainly considered three points:

• First, since the business is in a phase of rapid growth, the database capacity must be able to scale horizontally—including both cluster throughput and data volume—and we also want performance not to degrade significantly as the data volume increases;
• Second, the new database must have strong disaster-recovery capabilities to ensure no data loss in extreme scenarios such as data-center failures;
• Third, because the learning-machine business not only sells hardware and courses but also involves a large amount of product operations, we have numerous big-data analytics needs, which require the database to be compatible with the MySQL binlog ecosystem.

Based on these three requirements, our research found that Alibaba Cloud’s PolarDB-X (see Figure 5; referred to below as “PolarDB”) and Ant Group’s self-developed OceanBase both met our needs—both are open-source products, both support distributed storage, and both have high data availability and reliability. To make a wiser choice, we conducted an internal feature comparison of the two products.

Figure 5: PolarDB-X architecture

The feature performance of the two databases is shown in Figure 6. Specifically:

• Both have an RTO (fault recovery time) of less than 8 seconds and can achieve zero data loss;
• Both use a multi-replica underlying architecture and ensure data consistency through the Paxos protocol, thereby avoiding split-brain problems;
• Since neither yet has Serverless technology, both somewhat lack automatic elastic scaling; both can achieve horizontal elastic scaling by adding nodes;
• Both support flow control and Kubernetes container orchestration;
• OceanBase’s native distributed architecture, thanks to its tenant features, has a prominent advantage in resource isolation;
• Cost is also a factor we care about, and OceanBase’s R&D and operations costs are both relatively lower.

Figure 6: Feature comparison of PolarDB-X and OceanBase

On the performance side, we compared MySQL, PolarDB, and OceanBase using a stress-test environment of 8 cores / 64 GB / 64 threads with three-data-center deployment; the results are shown in Figure 7.

Figure 7: Performance comparison of MySQL, PolarDB, and OceanBase

Ultimately, the main reason we chose OceanBase was its ecosystem. Although PolarDB’s database kernel is very powerful, its surrounding platform is relatively lacking. Currently, PolarDB mainly offers deployment methods such as a command-line interface and a Kubernetes Operator, which could place a significant strain on our R&D investment. Therefore, with comparable availability and kernel capability and similar performance, we leaned toward the option with the higher return on investment.

After choosing OceanBase, our next step was to design a high-availability plan for the learning-machine business, one that would fully leverage the primary-tenant feature provided in OceanBase 4.2.5. It’s worth noting that although the primary-tenant feature may have existed in earlier versions, 4.2.5 likely provides a business-consistency-oriented access method. This means that after a primary-standby switchover, the business side does not need to perceive any change, thereby ensuring the high availability of the business.

To ensure that the business can keep running even in extreme cases, we kept the original MySQL cluster and, through OMS’s reverse-synchronization mechanism, stream real-time data back into the OceanBase cluster, to guard against the case where both the primary and standby clusters become unusable during a failure. If a failure occurs, we can switch back to MySQL to keep serving the business. Although this architecture may not be optimal from a cost standpoint—and may even cost somewhat more than the original plan—given the business’s strict high-availability requirements, this investment is worthwhile. The architecture is shown in Figure 8.

Figure 8: Architecture of the primary, standby, and rollback clusters

In addition, our learning-machine business has a strong dependency on MySQL’s binlog. On one hand, the business relies on the real-time generation of binlog data to compute real-time reports, helping with scenarios such as letting users view their study reports; on the other hand, we also use this data to make operational decisions, such as recommending suitable courses to students or adjusting operational direction. Therefore, we need a data-warehouse-style solution to meet these needs.

To this end, we adopted the binlog service provided by OceanBase. As shown in Figure 9, this solution is fully compatible with the MySQL 5.7 binlog format and natively supports mature MySQL-ecosystem tools such as Canal and Flink. This way, we can use these tools to process and analyze binlog data, meeting the learning-machine business’s needs for real-time report computation and operational decision-making.

Figure 9: The binlog service solution

With this solution, our business switched from MySQL to OceanBase almost imperceptibly, meeting our data-subscription needs.

So what benefits has OceanBase brought us in this business?

Thanks to OceanBase’s partitioned-table capability, the business team no longer needs to worry about sharding logic, which has greatly saved R&D cost. In the past, every time we hit a capacity problem we had to split databases, but now we only need to keep adding OBServers to meet storage needs—saving roughly 10 person-days per operation. In addition, OceanBase’s cross-zone compute capability achieves an RTO under 8 seconds and an RPO of zero, which has greatly improved system stability.

Furthermore, OceanBase does a great job with large-scale binlog subscription for big data. It unifies and consolidates the binlog subscription entry point, enabling high availability between big data and the database.

However, in the course of using it, we also discovered some things to watch out for.

(1) When creating partitioned tables, we don’t recommend creating all future tables at once, because data has a hotspot problem that may cause some OceanBase servers to use far more storage than others (the latest version has improved data balancing, which we haven’t tried yet). To solve this, we can use OceanBase’s partition-management capabilities and create only one or two extra tables at a time.

(2) When deploying the OceanBase log process, it’s best to specify a storage directory for the binlog; otherwise it defaults to the root directory, which may cause the root directory to run out of space. Changing the directory afterward is fairly cumbersome and requires restarting the OceanBase log process, which may cause business losses. Also, OceanBase currently doesn’t support the flush log with vlog command, which means we can’t directly use the MySQL dump tool to pull a full data set and need to do some adaptation on the business side.

(3) When using OMS for partitioned-table and incremental synchronization, you need to set the corresponding parameters and adjust the JVM memory size; otherwise, full verification may fail or incremental synchronization may fail. As of February 2025, we had deployed five OceanBase clusters with 31 tenants running.

(2) Multi-data-center DR Scenario: Solving Split-Brain, Achieving Zero Data Loss

When using MySQL, we faced major disaster-recovery limitations. For example, when the primary data center failed, it led to a data split-brain. The process is shown in Figure 10: at time T1, if data center A is network-isolated, the high-availability component detects this and proactively switches the primary database to data center B. However, if data center A’s network later recovers, a double-write problem occurs, which from a whole-system perspective leads to a data split-brain. For our business, data being unavailable or lost might be barely acceptable, but the split-brain problem is the thorniest of all.

Figure 10: The process of a MySQL primary-data-center failure

Therefore, disaster recovery was a problem we urgently needed to solve.

Before introducing OceanBase, our approach was: when a data center failed, we took no automated loss-mitigation measures. Instead, we first waited for the business to switch traffic to data center B, then manually migrated the database to data center B. However, the problem with this approach was that the cluster’s failure-mitigation time was greatly prolonged, because it was a non-fully-automated process; and because data synchronization used a primary-replica approach with cross-data-center network latency, it could not guarantee against data loss. Although this approach could schedule business traffic from outside the data center, traffic such as scheduled tasks inside the data center could not be scheduled effectively, which could still lead to a data split-brain.

To address these unresolved pain points, OceanBase provides two disaster-recovery solutions.

The first solution is “three data centers in the same city” (see Figure 11). OceanBase’s data exists in multiple replicas, with consistency between replicas ensured by the Paxos protocol; simply deploying the cluster across three data centers solves data-center-level failures.

Figure 11: Three data centers in the same city

After any one data center fails, the majority replicas in the remaining data centers can continue to provide service. This way, the cluster can still serve externally, and the business only needs to switch traffic away from the failed data center to complete loss mitigation. With OceanBase’s three-data-centers-in-one-city capability, we can ensure the cluster’s fault-recovery time is under 8 seconds with zero data loss.

This solution does have one limitation, however: it places very high demands on the network quality among the three data centers. Since a business request may involve cross-data-center calls, this approach may not be suitable if the inter-data-center network is unstable or if the business is extremely sensitive to network latency.

The second disaster-recovery solution is called primary-standby tenants, shown in Figure 12. In short, two independent OceanBase clusters are deployed in data center 1 and data center 2 respectively. The tenant’s primary replica is created in data center 1, and a replica of that tenant is created in data center 2. OceanBase has a built-in component called the log transfer service that can synchronize changes from the primary replica to the standby replica in real time.

Figure 12: Primary-standby tenants

Under this disaster-recovery solution, data consistency between the primary and standby tenants is guaranteed. In terms of disaster-recovery scope, this solution is superior to the first because it can handle cluster-level failures. For example, if cluster A fails, we can seamlessly switch the business to cluster B to keep serving. However, this solution also has some shortcomings.

• The loss-mitigation process requires manual intervention. When data center A has a problem, we need to manually promote the standby tenant to the primary tenant—a non-automated process.
• It cannot fully solve the data split-brain problem mentioned earlier, but compared with MySQL’s synchronization performance and synchronization latency, the problem is greatly reduced.

Therefore, after a comprehensive consideration of inter-data-center network quality and latency, we ultimately decided to adopt the first solution.

Once the capacity plan was determined, the next step was to design a plan to migrate from MySQL to the OceanBase cluster; the overall architecture is shown in Figure 13. Thanks to OceanBase’s rich set of supporting tools—among which OMS provides bidirectional synchronization—the migration process was greatly simplified: we only needed to build a new OceanBase cluster across the three data centers, then use OMS for bidirectional data synchronization to complete the migration and setup of the entire cluster. The migration went very smoothly with minimal impact on the business.

Figure 13: Business migration architecture diagram

(3) AI Business Scenario: 86% Monthly Storage-Cost Savings

With the arrival of the AI wave, we face the challenge of storing massive amounts of multimodal data. TAL Education’s own investment in AI has increased significantly. To support each business line in quickly adopting AI capabilities, our department launched an AI foundational service aimed at accelerating the integration of business with AI technology and improving overall business efficiency.

However, as more services were onboarded and the business grew, we encountered a notable problem: the amount of data that needed to be retained increased sharply, and the data growth was extraordinarily large (see Figure 14). Specifically, our service’s data grows by nearly 10 TB per month. If we used traditional MySQL servers to carry this incremental data, a single standard server would struggle to even store one week’s worth. This not only limited data-storage capacity but also became an obstacle to ongoing business development and data utilization.

Figure 14: Data-growth trend

In addition, traditional database solutions require the business to perform complex sharding operations, which not only increases the difficulty of technical implementation but also significantly raises overall cost. Facing this pain point, we combined OceanBase’s massive-data storage capabilities with the cloud’s ultimate elasticity to explore a new solution.

We used OceanBase’s partitioned-table feature (see Figure 15) so that each OBServer carries part of a single table’s capacity, thereby spreading out the data-storage pressure. At the same time, OceanBase’s advanced compression technology effectively controls the rate of data growth, keeping data storage within a manageable range. Furthermore, by adding cloud ECS purchases and nodes—on the basis that OceanBase’s analytical tables can store incremental data—we purchase cloud ECS (Elastic Compute Service) every quarter and manually switch new partitions’ log traffic onto the new ECS to keep up with data growth.

Figure 15: OceanBase’s partitioned-table feature

But this solution faced two problems: first, uncontrollable cost—as data volume grew rapidly, we found ourselves needing to buy more and more ECS, possibly as many as three per month, making cost hard to control; second, data-migration challenges—although we added ECS to the nodes, the data migration could not be triggered automatically by OceanBase and had to be done during business off-peak hours, which posed a challenge to operations.

To solve the above problems, we devised the following improved plan.

First, buy small-spec ECS and attach cloud disks: we purchased three small-spec ECS instances and attached cloud disks to them. A single Alibaba Cloud disk can support up to 60 TB of storage, so we first purchase a 4 TB cloud disk each time.

Second, cloud-disk upgrades and OceanBase auto-detection: when the 4 TB cloud disk runs low on space, we directly upgrade the cloud disk’s capacity on the ECS; this operation requires no ECS restart and has no impact on the OceanBase service. After a period of time, OceanBase automatically detects the newly added disk space without any extra action on our part.

Then, simplified operations: operations work is now greatly simplified—we only need to expand the cloud disk on the corresponding ECS when a disk alert fires. This improved plan has brought us tremendous convenience.

The currently implemented solution has delivered significant benefits in several areas. On one hand, it requires no modification from the business side, allowing the business team to focus entirely on overall business development without being distracted by the details of data processing or storage solutions. This not only improves the efficiency of business development but also ensures that the business team can make full use of its resources and energy. On the other hand, the solution has successfully handled the current sharp growth in data volume. Through efficient storage and processing capabilities, it ensures stable data storage and management, meeting the business’s data-growth needs. These advantages enable us to better support business development and ensure the reliability and integrity of data.

In addition, by combining OceanBase with the cloud’s elasticity, TAL Education achieved a significant cost reduction. After the improvement, our new costs each month are only 14% of the original traditional solution—meaning we save 86% of the cost every month. This enables us to achieve higher data storage and processing capacity at a lower cost, thereby improving overall economic efficiency.

However, we also realize that the current maximum storage ceiling for a single instance is 650 TB, a limit we will inevitably hit sooner or later, so we’ve made future plans to address this challenge. We plan to migrate historical data to Object Storage Service (OSS) to break through the storage limit. OSS provides nearly unlimited storage space and can meet our long-term data-storage needs. At the same time, we’ll use OceanBase’s external-table capability to query and process data stored on OSS. This combination will ensure we can meet the entire business’s storage and data-processing needs while maintaining cost-effectiveness.

(4) Online-School Reports: 60% Better Performance Than MySQL

Currently, our online promotional campaigns are run through popular channels such as short videos. To operate these campaigns effectively, the business team needs to store relevant data—ad placements, revenue figures, and so on—in a database so that operators and ad buyers can view it in real time and adjust their placement strategies promptly. As a result, this platform has very high requirements for data-response speed.

Specifically, these report-related data tables fall into the OLAP category—mostly wide-table structures with data volumes in the tens of millions. The business team wants tens-of-millions-row queries against these wide tables to return results within seconds; otherwise the user experience would be severely affected. After receiving the requirements, we considered the columnar-storage capability provided in OceanBase 4.3, expecting it to deliver a significant boost to OLAP performance. So we migrated this tenant from OceanBase 4.2 to OceanBase 4.3 and converted all the tables the business uses into columnar format.

However, after the initial conversion, we found that relying on memory alone was still insufficient in some scenarios, so we did some additional performance tuning to ensure no performance regression in any business scenario. We selected some relatively complex business instances for testing, and the results showed that most OceanBase instances could return results in about one second—a 60% overall performance improvement compared with MySQL.

In the process of using memory, we also explored some effective approaches. These approaches not only improved performance but also optimized memory-usage efficiency. We think they’re worth sharing, as they may be enlightening for both teams and individuals.

When using columnar storage, we might initially encounter some uncertainty and challenges, especially regarding which kinds of queries are best suited to columnar storage. Through practice, we drew some conclusions. First, when a query’s filter conditions involve only a few columns (such as a single column or a handful of columns), columnar storage can perform well even without other indexes on the table. However, if the filter conditions involve many columns and there are no indexes, columnar storage may not perform well, because each column scan is relatively costly.

OceanBase’s official team also gave some advice. Due to optimization reasons, when a query’s aggregate functions include multi-column expression operations, columnar storage may not be well suited. This is because columnar storage currently hasn’t yet completed expression-vector optimization for this kind of aggregate computation, though it will be improved in later versions.

Another especially important point is that columnar storage is best suited to read-heavy, write-light scenarios. In the course of using it, we found that frequent data updates cause memory-side performance to drop and require timely data compaction to restore performance (see Figure 16).

Figure 16: Data compaction

In summary, using columnar tables requires meeting three criteria: the wider the table, the greater the advantage of columnar storage; the “where” condition should ideally filter on a single column to improve query efficiency; and the scenario should be “read-heavy, write-light.” For frequently updated scenarios, columnar storage may not perform well.

For the above optimization points, we offer the following suggestions:

• For OLAP scenarios, the business should specify the columnar flag when creating tables, or set the tenant parameter to column, so that subsequently created tables are all columnar tables—simplifying the table-creation process.
• If the business has batch data-import scenarios, run a compaction operation promptly after each import to avoid a substantial performance drop.
• OCP itself has a compaction-management storage policy, and we can adjust the relevant parameters to achieve automatic storage-performance optimization.

(5) Content Moderation: Ensuring Secure and Efficient Data Management

Our content-moderation service also stores the raw data of machine moderation and the moderation results in OceanBase, ensuring secure and efficient data management.

The content-moderation business mainly revolves around course teaching, during which there is a large amount of real-time interaction between teachers and students. Users may also interact with large models through our application, so interaction among these three parties is frequent. Under current policy requirements, all real-time interactions must be moderated. Therefore, our content-moderation service aims to comprehensively cover all of the group’s businesses, moderating all interaction information in real time, which poses many challenges for traditional databases:

• Because all of the group’s businesses require content moderation, and student class times are concentrated, the throughput requirements for the database cluster are extremely high.
• The business requires preserving all moderated raw content and results—both for regulatory compliance and possibly for subsequent manual review—so the amount of data to store is enormous.
• Machine-moderation data also needs to interface with manual review and the operations platform for operational work, which involves complex business-logic processing.

After we migrated to OceanBase, the above challenges were resolved, as shown in Figure 17.

• With its excellent high-scalability performance, OceanBase achieves on-demand scaling of both throughput and capacity, solving in one stroke the core difficulties the business encountered in database usage.
• OceanBase’s built-in parallel query and vectorized engine greatly improved query performance, delivering a smoother experience for users.

Figure 17: The situation after migrating to OceanBase

III. Business Exploration: Replacing More Databases with OceanBase

As OceanBase penetrated more of TAL Education’s business scenarios, we found there’s much more it can do—it can even do many of the jobs traditional databases handle, only better. Therefore, to further simplify the tech stack and improve system stability, we tried replacing more databases with OceanBase.

(1) Replacing ClickHouse with OceanBase Columnar Storage

The 4.3 version OceanBase released in 2024 features columnar storage and can replace our current ClickHouse service. The group currently uses ClickHouse only to store cleaned data from the log center; this data, after real-time aggregation, is used for monitoring dashboards and watching the boards during business peaks. To save labor costs, we chose to replace ClickHouse with OceanBase’s columnar-storage feature (see Figure 18).

Figure 18: Replacing ClickHouse with OceanBase’s columnar-storage feature

The reason for replacing ClickHouse is that, by comparison, OceanBase columnar storage has several significant advantages.

(1) OceanBase 4.3 introduced brand-new columnar storage and vectorization features, greatly boosting its AP capability; official data shows its performance is now on par with ClickHouse.

(2) OceanBase columnar storage is also compatible with row-based storage. This means that within the same cluster, you can have both columnar tables to meet AP business needs and row-based tables to support TP business, achieving flexible coverage of business scenarios.

(3) OceanBase columnar storage allows adding multiple kinds of indexes to columnar or row-based tables, such as row-based indexes or columnar indexes, enabling a single table to handle multiple business-access scenarios and improving the flexibility and efficiency of data access.

(4) OceanBase also supports advanced features such as materialized views and bypass import, which provide more possibilities for AP use scenarios in business development and further broaden its applicability.

(5) OceanBase’s support for complex queries is markedly better than ClickHouse’s; because ClickHouse excels at single-table operations, complex queries such as joins are often constrained by memory size, leaving its capabilities stretched thin.

(2) Replacing Pika with OBKV

Because the business scenario involves online teaching with frequent interaction between teachers and students—including teachers’ doodle/annotation information and students’ questions—all of this is stored as messages. Initially we chose to store the information in Redis, with the architecture shown in Figure 19, but rapid data growth drove costs too high to bear. So we switched to Pika as an alternative.

Figure 19: Message-storage architecture

However, in using Pika, we ran into several key problems:

• Pika occasionally exhibited performance jitter, causing brief service stalls;
• During database operations such as active switchover, a full synchronization is triggered. Given Pika’s huge data volume—reaching thousands of GB—a single full sync poses a fairly high risk to the business;
• The Pika version we use is fairly old, and although the new version 3.5 has been released, there’s no smooth upgrade path, which makes operating Pika quite a predicament for us.

While researching OBKV, we were surprised to find it was an excellent option for replacing Pika.

• OBKV shares the same underlying architecture as OceanBase, ensuring stable storage performance with an RTO under 8 seconds and an RPO of zero;
• OBKV launched a Redis mode that is fully compatible with our existing Redis protocol, meaning that migrating the business to OBKV requires no changes to the business layer;
• OBKV has an excellent data-compression ratio, which, given our large message volume, will significantly reduce costs.

The performance advantages are shown in Figure 20.

Figure 20: Performance comparison of OBKV and Pika

IV. Future Hopes

The above is some of TAL Education Group’s experience in upgrading its database solution. Although the benefits OceanBase has brought meet our expectations, in the course of using OceanBase we also hope it can become even more perfect.

First, there’s our need for OMS. As mentioned earlier, OceanBase has not only a powerful database kernel but also an excellent surrounding ecosystem of tools—especially OMS, which integrates data transfer, data subscription, data verification, and other functions into one comprehensive package. We hope that in the future OceanBase will continue to develop and refine OMS as an independent and core product, to better support our business. For example, in data synchronization—whether from MySQL to OceanBase or between other databases—OMS could help us solve the data-synchronization challenges we encounter in daily operations.

Second, at the kernel level, we hope OceanBase will add two key capabilities.

One is a storage-compute separation architecture. Since OceanBase currently uses a standalone-distributed integrated architecture—where CPU, memory, and storage are integrated together—we hope OceanBase can design an architecture that separates storage from compute. Looking at the distribution of our current online clusters, the OceanBase cluster’s storage resources are already about 70% used while CPU resources are only 10% utilized. This imbalance is often caused by disk-capacity limits, which prevents CPU resources from scaling effectively. A storage-compute separation architecture would greatly improve the flexibility of resource scheduling.

The other is using OceanBase to build a data lake. This need was raised by our big-data team, who hope to use OceanBase as a data-lake scenario to support external-table queries and federated queries.

Finally, we’d like to recommend the WeChat account of Lao Ji, the head of OceanBase open source: “Lao Ji’s Tech Talk.” It continuously publishes all kinds of technical content related to #Database, #AI, and #Tech Architecture. If you’re interested, feel free to follow!

“Lao Ji’s Tech Talk” not only hopes to keep bringing you valuable technical sharing, but also hopes to contribute to the open-source community together with everyone. If you recognize the value of the OceanBase open-source community, light up a little star ✨! Every Star you give is the motivation behind our efforts.