Weibo's Database Architecture Upgrade with OceanBase: Both Stable and Cost-Saving

Editor’s note: As one of China’s most influential social media platforms, Weibo (formerly Sina Weibo, stock code: 09898) has, since its launch in 2009, grown into a comprehensive social network combining content publishing, social interaction, and trending-topic dissemination. According to public data, Weibo’s monthly active users surpassed 583 million in 2024.

Faced with a huge user base and increasingly complex business scenarios, Weibo’s database platform confronted a key challenge: how to balance system stability and cost efficiency while ensuring the ability to handle massive data. After a rigorous technical evaluation, the team ultimately completed a strategic upgrade of its core database system. In this article, Yang Shanggang, head of Weibo’s database technology, takes a deep dive into this database evolution—one that bears on the experience of hundreds of millions of users—across dimensions ranging from technology selection and solution design to implementation in practice.

Weibo’s Database Platform Architecture and Current State

Weibo’s database platform is divided mainly into three parts: hosting common mainstream relational databases and NoSQL databases, and providing complete OLTP and OLAP solution support, driving the innovation and implementation of the company’s database technology.

In total, there are tens of thousands of database instances. The NoSQL databases are dominated by Redis, with daily access volume on the order of trillions and access peaks in the billions. Since 2011, we have made heavy use of Redis and developed several internal forks based on it, with persistently stored data reaching the PB scale.

The figure below shows Weibo’s database architecture. The resource layer includes early self-built physical machines, a private cloud, and a public cloud used to address elastic scaling of resources. The scheduling layer is responsible for resource management and scheduling, including an internally developed resource scheduling platform and open source Kubernetes. The service layer is mainly relational databases, including MySQL, PostgreSQL, ClickHouse, Milvus, OceanBase, MongoDB, and more. The wide variety of products currently leads to relatively high platform management costs, along with problems in stability and ease of use.

Because of the relatively weak economy in recent years, labor costs keep rising, while normal business growth, the continuous growth of storage resources, and architecture upgrades all require more manpower. So, to reduce costs and improve efficiency, we need a simpler, more efficient, lower-cost technical architecture.

Architecture Upgrade: Solving MySQL Pain Points First

Against a backdrop of cost control and refined risk management, we needed to support steady business growth with limited server resources while maximizing resource utilization and delivering efficiently. Among relational databases, MySQL is the most widely used and its problems are the most prominent, so in this architecture upgrade we prioritized solving MySQL’s pain points, such as:

• Lots of long-tail data. Data like Weibo comments grows on a massive scale every day and must be retained long-term, resulting in a great deal of long-tail data that takes up storage space.
• The unsustainability of sharding. When the spec a database instance needs exceeds the machine’s spec, you have to scale; usually we can only resort to a sharding solution, which meets business requirements for a while but brings higher management and business costs.
• High-availability and data safety problems. MySQL mainly synchronizes data via asynchronous replication; while it also supports semi-synchronous replication, that incurs some performance overhead. The MGR cluster solution supported in MySQL 5.7 is relatively complete, but its operational cost is relatively high. After a primary-secondary switchover, there’s a risk of data inconsistency.
• High storage cost. MySQL deployment mainly uses a one-primary-two-secondary architecture, equivalent to double backup. As shown in the figure below, from 2013 to 2020, the cost of HDDs and SSDs grew increasingly close, and HDDs’ advantage in storage cost didn’t change much.

To address MySQL’s pain points, we planned to optimize along several fronts.

1. Optimize storage cost by using different storage modes for different scenarios.

In 2016, to solve latency and performance problems, we migrated data from HDDs to QLC SSDs. However, as business scenarios changed, the cost advantage of QLC SSDs became less obvious; in some cases, using HDDs might be a better choice. In some scenarios, using QLC SSDs only reduced cost by about 10%–20%, whereas HDDs, thanks to their higher per-machine storage density, might reduce cost by a larger margin. So different storage modes are needed for different business scenarios.

2. Archive long-tail data to S3.

We plan to archive long-tail data to S3 and are currently researching a suitable solution. We know that the MariaDB plugin supports S3 storage; its essence is to achieve data synchronization through underlying S3, rather than traditional cluster data synchronization. This solution can significantly reduce storage cost—possibly even lower than using HDDs. But note that it places certain requirements on the business scenario: for example, tables with frequent data updates may be limited, and it’s better suited to storing data that rarely or essentially never updates. In addition, OceanBase also supports an S3 solution, whose feasibility we are researching.

3. Reduce cost through software and hardware compression.

Software compression and hardware compression are also areas of our overall testing. MySQL has supported compression since version 5.5, and versions 5.7 and 8.0 introduced new compression methods. However, MySQL’s built-in compression has relatively high performance overhead—for example, using MySQL compression can save about 40% of storage space, but performance may drop by about 66%. Since we learned that OceanBase has strong cost-reduction capabilities, we are considering using OceanBase to reduce hardware and storage costs while minimizing performance loss.

4. Resource pooling for elastic scaling.

Resource pooling is an important direction for solving elastic scaling, but its implementation requires strong R&D capabilities. We don’t currently have the conditions to implement resource pooling, but we can reference advanced industry solutions such as AWS or PolarDB, or leverage OceanBase’s resource pooling capability when introducing it. The main goal of resource pooling is to solve MySQL’s scaling difficulties, especially the challenges of data storage and scaling.

Through these various optimizations, we hope to effectively solve MySQL’s pain points and improve system performance and cost efficiency. While researching database solutions, we found that OceanBase fit our expectations well, so we began trying it in our business.

Replacing Sharding with Partitioned Tables: Both Stable and Cost-Saving

OceanBase is a fully self-developed, natively distributed database that supports smooth scaling and elastic resizing, is highly compatible with MySQL, can smoothly, quickly, and cheaply migrate applications and data, and has high availability (RPO = 0, RTO < 8s). It also has a very active open source community, so we began trying to introduce OceanBase in certain scenarios.

For example, one system had previously faced problems such as complex sharding splits, data inconsistency, wasted machine resources from many instances, and high operational cost. We tried replacing this system’s database with OceanBase, and both the initial testing and deployment went fairly smoothly, solving the problems the system had with MySQL.

The main benefits after adopting OceanBase were as follows:

1. We abandoned the original MySQL sharding strategy in favor of OceanBase partitioned tables. From the business’s perspective, it’s equivalent to a single large table, which simplified the business read/write logic and completely solved the data consistency problem.
2. OceanBase’s underlying storage uses general compression and data encoding compression, achieving a higher compression ratio than MySQL with no impact on performance. After migrating the system to OceanBase, storage usage dropped from 50TB to 27TB, saving 46% in cost.
3. The system has run stably for over half a year, proving OceanBase’s strong stability.
4. The OceanBase distributed database has high availability natively; once the cluster is deployed, there’s no need for a lot of extra work to maintain multiple data replicas, and the failure of a minority of nodes doesn’t affect business use. Unlike MySQL, there’s no need to introduce extra components to ensure cluster switchover, further reducing operational complexity and DBA operations costs.

Choose the Right Technical Strategy to Reduce Cost and Improve Efficiency

In the process of introducing OceanBase, we gained a lot of practical experience to share for reference.

Practice 1: Partitioned Table Design

Partitioned tables are a common optimization technique in database design, especially when handling large-scale data. For MySQL users, there are plenty of guidelines and docs on designing and using partitioned tables. However, early MySQL versions had some problems with partitioned tables, so in practice it was usually recommended to use sharded tables rather than partitioned tables. A single MySQL table is typically on the order of tens of millions of rows; beyond that, sharding is needed.

OceanBase’s partitioned-table design philosophy differs significantly from MySQL’s. It mainly achieves efficient management of large-scale data through partitioned-table design, with no upper limit on the data volume of a single table. For tables with large data volumes, partitioned tables are recommended, with each partition kept under 100GB (corresponding to MySQL’s 600GB–800GB). At the same time, the number of partitions is recommended to be between 2–3x the number of servers and 2x the number of logical CPU cores, to ensure balanced data distribution and avoid the data imbalance caused by having too few partitions.

OceanBase supports level-1 and level-2 partitioning. Level-1 partitioning is the common approach and includes common partition types such as HASH, KEY, LIST, RANGE, RANGE COLUMNS, and LIST COLUMNS; the specific choice should be based on the business scenario and data characteristics. For example, if the data has a time dimension, range partitioning may be more suitable; if the data needs to be partitioned by specific values, list partitioning may be the better choice.

Level-2 partitioning is equivalent to further splitting along a second dimension on top of level-1 partitioning. If you chose level-1 partitioning at the design stage and later need to extend the partitioning, directly modifying the partition key may be difficult; in that case, you can solve the problem by introducing level-2 partitioning. Level-2 partitioning must be designed carefully to ensure the partitioning is reasonable and extensible.

The purpose of creating partitions is to reduce the total amount of data read and written in specific SQL operations, thereby reducing response time while also improving scalability, manageability, and performance.

The main purpose of partitioned table design is to optimize data volume and response time and reduce the system’s burden. Through reasonable partitioning, you can effectively control concurrency and ensure a balance between data scale and performance. When designing partitioned tables, you should also note the following points.

1. Partition key selection. The partition key should be chosen carefully based on the business pattern (spreading out hot data, ease of maintaining historical data, the condition patterns of business SQL, partition pruning), trying to distribute data evenly and avoid hot spots.
2. Partition type selection. Choose a suitable partition type based on business needs, such as hash, range, or list partitioning.
3. The relationship between the partition key and the primary key. In OceanBase, the partition key must be a subset of the primary key—that is, the primary key must contain the partition key.
4. Range partitioning. Consider whether to use maxvalue; if you use maxvalue, you won’t be able to add partitions later.
5. The write amplification problem. To avoid write amplification, when choosing a table’s custom primary key, don’t use randomly generated values—try to keep them ordered, such as time-series increasing.
6. Consider partition pruning optimization. Partitioned table design is an important part of database optimization, especially when handling large-scale data. OceanBase’s partitioned table design is fairly flexible; in practice, you need to choose a suitable partitioning strategy based on business needs and data characteristics to optimize system performance and data management efficiency.

Practice 2: Statistics Gathering

Statistics gathering is an important part of database SQL optimization, especially when dealing with slow queries, which rely heavily on statistics. Whether MySQL or OceanBase, the query optimizer’s execution path depends on the accuracy of statistics. When there are many tables, the cost of gathering statistics is high and takes a long time. Among statistics, histograms are a fairly important part; gathering histograms involves complex computation, adding extra cost and time. Large partitioned tables by default gather statistics and histograms for level-2 partitions, level-1 partitions, and the whole table, at a cost of: 3 * {cost(full table scan) + cost(histogram)}.

There are five things to note when gathering statistics.

First, set a reasonable default gathering parallelism.

• The concurrency of statistics gathering needs to be controlled, because it has some impact on machine performance. We recommend gathering during off-peak hours (such as the early morning) to reduce the impact on the business.
• We recommend keeping parallelism within 8; when the business allows, you can increase concurrency to trade resources for statistics-gathering efficiency.
• Execute under the business tenant:

SQL>call dbms_stats.set_table_prefs('database_name', 'table_name', 'degree', '8’);

Second, configure the sampling ratio.

Configuring block sampling can reduce the amount of data to process during statistics gathering via block sampling, achieving the effect of fast gathering. But it sacrifices some accuracy of the statistics. You can set it as follows.

Enable block sampling:

SQL>call dbms_stats.set_table_prefs('databse_name','table_name','block_sample','True');

Configure the sampling ratio, which can be set according to the table’s order of magnitude; usually, sampling tens of millions of rows is enough to fully reflect a table’s data characteristics:

SQL> call dbms_stats.set_table_prefs('databse_name','table_name','estimate_percent','0.1');

Third, set the default histogram gathering method for columns, and consider setting columns with evenly distributed data to not gather histograms.

If all columns in the table have evenly distributed data, you can set all columns to not gather histograms as follows:

SQL>call dbms_stats.set_table_prefs('database_name', 'table_name', 'method_opt', 'for all columns size 1');

If only a very few columns in the table have unevenly distributed data and need histograms while the others don’t, you can set it as follows (c1, c2 gather histograms, c3 does not):

SQL>call dbms_stats.set_table_prefs('database_name', 'table_name', 'method_opt', 'for columns c1 size 254, c2 size 254, c3 size 1);

Skip large object fields; if the columns store large objects, gathering can be especially slow, so gather all columns except the large objects:

SQL>call dbms_stats.set_table_prefs('databse_name','table_name','method_opt','for columns col1,col2,col3,... size auto');

Fourth, the way statistics are gathered in different scenarios.

Use sampling for gathering statistics on large tables cautiously; when sampling is set for gathering large tables, the number of histogram samples also becomes very large, which can be counterproductive. Setting up sampling-based gathering is only suitable for gathering basic statistics without gathering histograms. Execute the following under the business tenant. Set all columns to not gather histograms:

call dbms_stats.set_table_prefs('database_name', 'table_name', 'method_opt', 'for all columns size 1');

Set the sampling ratio to 10%:

call dbms_stats.set_table_prefs('database_name', 'table_name', 'estimate_percent', '10');

Fifth, lock statistics.

Consider whether you can manually gather statistics on a large table and then lock the related statistics. Note that once a table’s statistics are locked, automatic gathering won’t update them—this suits scenarios where the data characteristics don’t change much and values aren’t sensitive. If you need to re-gather locked statistics, you must unlock them first.

Lock a table’s statistics:

call dbms_stats.lock_table_stats('database_name', 'table_name');

Unlock a table’s statistics:

call dbms_stats.unlock_table_stats('database_name', 'table_name');

Statistics gathering requires comprehensively weighing factors such as concurrency, sampling ratio, gathering method, and the table’s usage scenario, to ensure the optimizer can generate efficient query plans based on accurate statistics while avoiding excessive impact on system performance.

Exploring OceanBase Further in KV and Vector Scenarios

Because OceanBase has performed excellently in the business already in production, we plan to keep trying and exploring it in other business scenarios. Currently, our business data volume is large but the read/write volume is relatively low. In the future, we will try applying it to business scenarios with larger read/write volume and greater sensitivity to latency, to further validate OceanBase’s performance and suitability.

Scenario 1: KV Storage Optimization

Beyond TP scenarios, Weibo’s internal KV scenarios also face some challenges. Take Pika as an example—it currently has four problems.

1. Co-location affects Pika’s stability: Pika is co-located with MySQL, and there are many Pika instances. If scheduling is uneven, a single machine may end up with too many Pika instances, which affects stability, especially with hot data or unbalanced machine load.
2. Long-tail data: similar to MySQL, Pika, as a disk-based KV store, has long-tail data problems on some instances. This data must be stored long-term but can’t be cleaned up, putting significant pressure on storage management.
3. Scalability problems: Pika is essentially a standalone database, and its data scale and scalability face challenges; like MySQL, standalone scalability is one of the thornier current problems.
4. Cost optimization: cost is split into two parts—first, the migration cost of moving from Redis to Pika and so on, and second, the cost of introducing the new ARM platform.

Optimization measures break down into two dimensions. First, optimize scalability: for Pika’s scalability problems, we’ve researched related solutions and are currently considering management via Proxy + Pika. Second, adapt to the ARM architecture: we’ve already adapted Pika, the Redis service, and Memcached to the ARM architecture. Test results show that the ARM architecture significantly improves single-instance performance over an x86 architecture of the same spec, at lower cost. But the adaptation incurs some cost, especially on the ARM architecture, where compilation instructions and JCC (Just-In-Time Compiler) support differ from x86. In addition, MySQL may also be adapted to the ARM architecture later, but its adaptation may be harder than Redis’s because MySQL’s code is more complex.

Scenario 2: Vector Database Pain Points

In recent years, vector databases have developed rapidly and come in many varieties, such as the mainstream Milvus, Redis, Elasticsearch, PG Vector, and OceanBase—all of which support vector indexes and scalar indexes and are fairly full-featured. Currently, Weibo mainly uses Milvus and Elasticsearch in production, but their usage cost, storage cost, learning cost, and management complexity are all relatively high, and they also have performance and scaling bottlenecks when handling high-dimensional vector data. Based on the current state of Weibo’s vector database usage, we will evaluate our current technical architecture, refine our business application scenarios, and explore the implementation of OceanBase’s vector capabilities in internal business scenarios.

First, refine the business application scenarios. In the future, online business may continue to use a variety of databases, including MySQL, Elasticsearch, OceanBase, and more, and we will refine our choices based on business scenarios and data scale to optimize cost and performance.

Second, evaluate disk index upgrades, with Milvus versions continuously upgraded and optimized. For the vector databases currently in use, we need to keep upgrading to address their cost and performance problems.

Finally, test OceanBase’s vector capabilities. OceanBase has certain advantages in vector databases, manifested in:

• A rich variety of index types. It supports vector indexes, scalar indexes, semi-structured indexes (multi-value indexes, spatial indexes), full-text indexes, and more.
• Rich application integration. It uses SQL access, reusing clients in the MySQL ecosystem across various languages; it provides a Python API and an SDK similar to the Milvus SDK.
• Rich index creation syntax. Index types include HNSW, IVFFlat, and DiskANN (in processing); distance algorithms include L2, IP, COSINE, and Jaccard (in processing).
• Rich vector queries. It supports TOP N ranked similarity computation and hybrid queries of vectors and scalars.
• Convenient deployment and operations. The functionality is integrated into the database kernel, and you can use the same deployment, operations, monitoring, and data migration tools as the OceanBase Community Edition.

As AI technology develops, databases will integrate more deeply with AI, especially the combination of natural language and SQL, such as the SQL rewriting direction. At the same time, AI Agents will also become a direction for us to explore OceanBase’s vector capabilities—for example, database assistants and knowledge bases. We hope to use AI to make internal staff’s work more convenient, reduce operational complexity, and improve efficiency.

Other Scenarios

Besides planning to explore OceanBase in KV and vector scenarios, we will also carry out the following five plans during our use of OceanBase.

1. Integrate the OCP platform with our self-built operations management platform.

We will push forward integrating the OCP platform with our self-built operations management platform. If we are to use OceanBase at scale, we must effectively integrate it with our existing database management platform, thereby lowering usage cost and improving management efficiency.

2. Resource isolation.

Currently, our resource scheduling approaches include Kubernetes (K8s) and a homegrown scheduling system, which have certain limitations in achieving CPU and memory resource isolation. OceanBase’s multi-tenant architecture can achieve resource isolation better.

3. Automate data migration.

Data migration is the key step in migrating MySQL or other databases to OceanBase. Currently, data migration and validation are not yet fully automated and still need gradual refinement. We will work to increase the degree of automation in data migration, ensuring an efficient migration process and accurate data.

4. Cluster performance tuning.

As OceanBase’s use at Weibo expands, the cluster scale grows ever larger, and performance tuning will become ongoing work. We will gradually accumulate experience based on actual usage and optimize cluster performance to meet business needs.

5. Multi-availability-zone disaster recovery.

Disaster recovery is a key area for future development. As cloud services and self-built virtualized environments become more widespread, the deployment scale and density of MySQL or other databases keep increasing, and the need for disaster recovery and large-scale fault handling becomes ever more pressing. In the future, we plan to deploy OceanBase clusters across multiple data centers to improve disaster recovery. While this deployment may have some performance impact due to network issues, we will gradually evaluate and optimize it to ensure system high availability and stability.

Summary

After this successful database architecture upgrade, we continue to explore better architectures and more convenient development and operations approaches. As an increasingly popular database product, OceanBase has broad prospects in Weibo’s application scenarios, and we will actively explore more cost-reduction and efficiency strategies to meet future challenges. In this process, we will adhere to two principles:

• One size does not fit all. For internet companies, database selection is crucial and must comprehensively consider factors such as technical stability, staff skills, the operations system, and business scenarios.
• The one that fits you is the best one. We’ll keep an open mind, continue to follow OceanBase’s development, and strive to apply more product capabilities that fit our business scenarios, helping drive business growth.

Finally, I’d like to recommend the WeChat account “Lao Ji’s Tech Talk” run by Lao Ji, OceanBase’s open source lead. It continually publishes all kinds of technical content related to #databases, #AI, and #technical architecture. Anyone interested is welcome to follow!

“Lao Ji’s Tech Talk” not only aims to keep bringing you valuable technical content, but also hopes to contribute to the open source community together with everyone. If you appreciate the OceanBase open source community, please light up a little star ✨! Every Star you give is what motivates us.