How to Avoid Pointless Work — Qifu Technology Cuts Code-Refactoring Costs by 90% After Ditching Sharding with OceanBase

Author: Jia Jianlong, Head of Database at Qifu Technology

Qifu Technology, formerly known as #360 DigiTech, was founded in 2016 and went public on Nasdaq in 2018 (QFIN) and on the Hong Kong Stock Exchange in 2022 (03660). It was officially renamed “Qifu Technology” in February 2023. The company operates at a massive scale, serving 165 financial institutions, with 276 million registered users, 60.2 million credit-line users, and over RMB 6 billion in net operating profit in 2024. Swept up in the AI wave, the company launched a brand-new AI-centered strategy in 2025, committed to building an AI-driven fintech platform.

Qifu Technology uses a wide variety of databases, including MySQL, Redis, Pika, Elasticsearch, MongoDB, InfluxDB, PostgreSQL, Oracle, and TiDB. To simplify its technology stack and address the pain points of its existing databases, the Qifu Technology database team adopted OceanBase in 2024 to replace part of its MySQL and TiDB usage, achieving a substantial reduction in code-refactoring, storage, and hardware costs, along with technical upgrades in performance, stability, scalability, and more. This article shares Qifu’s database pain points across multiple scenarios, as well as its hands-on experience deploying OceanBase.

Traditional Databases vs. Distributed Databases

As the business kept expanding and data grew rapidly, the traditional databases Qifu Technology used gradually ran into many challenges in data storage, data processing, and data analysis, and their scalability and performance struggled to meet business needs.

First, traditional databases are constrained by the responsiveness of single-node storage and scaling, making it hard to meet storage needs when the business has a large data volume. Mainstream traditional databases today typically have a capacity of around 3 TB. When the data volume exceeds the database’s capacity, the usual approach is to split one into many or many into many, making the architecture more complex. For an initial split, the business may face significant refactoring. By contrast, a distributed database has no capacity limit, can in theory scale infinitely, and requires no business refactoring when scaling.

Second, traditional databases are constrained by single-machine resources. Heavy writes to a single table also hit a certain bottleneck, and beyond a certain threshold, serious latency problems arise. A distributed database can spread write pressure across different nodes through partitioning, expanding write capability and solving the single-machine write bottleneck.

In addition, scaling traditional open-source databases often requires adapting middleware and considerable DBA effort — including pre-work plan reviews and validation, the actual operation during the process, and post-work observation. A distributed database requires little manual effort during scaling and can automatically redistribute data.

Currently, some of Qifu Technology’s core systems still rely on #MySQL as their primary storage, but the traditional single-database architecture has gradually exposed a series of bottlenecks and risks. The typical problems encountered in practice are as follows.

1. MySQL single-database write pressure: certain consumer-behavior business scenarios place ever-increasing write pressure on the database, which MySQL cannot handle, keeping the load persistently high.
2. Sharding: when a business MySQL database needs to be split into shards, the code-refactoring cost is high.
3. Interface throttling: some backend scenarios require calling a large number of interfaces.
4. Primary-replica replication lag: when the business write volume is large, primary-replica replication lag is high.
5. Data-archiving needs: for certain compliance requirements, archived data must be stored long-term, with occasional manual query needs.
6. Reporting query performance: the performance of certain reporting queries cannot meet business needs.
7. Storage cost: data storage costs are high.

Hands-on Practice: From Small-Scenario Validation to 6 Clusters in Production

To break through existing bottlenecks, improve the overall performance and stability of the system, simplify the technology stack, and reduce the DBA’s learning and operational costs, the Qifu Technology database team decided to select a relatively complete database to replace its other databases.

For the distributed-database technology selection, the main considerations were: stability, performance, compatibility, scalability, ecosystem, community activity, support for database localization, and cost. After in-depth research and evaluation, Qifu Technology decided to adopt #OceanBase.

The Rollout Journey

As shown in Figure 1, Qifu Technology first encountered OceanBase in 2023 and, after evaluation, launched OceanBase rollout in 2024. After testing confirmed that OceanBase’s architecture design, features, and performance metrics all met requirements, real-environment validation began in two internal scenarios: one scenario replaced #TiDB with OceanBase to solve TiDB’s jitter and false alarms in business monitoring, and the other was an archiving business, where practice proved OceanBase can reach a compression ratio of about 80% at most.

Figure 1 Qifu Technology’s OceanBase rollout journey

After achieving notable results in the internal-promotion phase, the company first brought non-core businesses onto OceanBase. The operations work simultaneously completed internal adaptation; although full platformization was not yet achieved, all previously manual operations first had to be scripted and automated, and ticketing was also provided with integration capabilities in script form. At the same time, the downstream data warehouse needed to adjust its synchronization links: whereas it previously only had to pull MySQL Binlog, it now had to additionally extract incremental data from OceanBase.

In August, the database team selected a batch of scenarios that were related to real-time business and relatively important yet not order-critical — such as the procurement platform and behavior logs — as the second batch of migration and promotion targets. After the migration was complete, integration with the OceanBase operations platform was kicked off: for the situation where users frequently apply for instances and databases and the ticket volume is large, to avoid affecting other operations work, the database team moved all such requests into a ticket-based workflow for automated handling.

Stress-Test Data

Internally, Qifu Technology connects several hundred sets of MySQL used by various businesses, offering packages of different specifications for each business line to apply for on demand. To ensure OceanBase follows the same management model after going live, the database team defined standardized tenant packages in advance. The business side simply selects the desired package in the ticket system to automatically complete tenant creation, with no need to discuss the tenant’s resource details; if there are special needs beyond the packages, a separate application can be submitted.

The database team conducted performance stress tests on the resource specifications of each tenant package; detailed data is shown in Figure 2. Of course, this data is only used for internal selection and capacity planning and does not represent OceanBase’s peak performance.

Figure 2 Performance stress-test data for different tenant packages

Technical Architecture

Qifu Technology’s database architecture is divided into five layers overall (see Figure 3).

• Business service layer: all business services (including App, Web, and various interface services) access the database through a unified access layer.
• MySQL service layer: business traffic first reaches the LVS cluster, which provides a virtual VIP and forwards it to MySQL. MySQL uses a primary-replica architecture, with Orchestrator providing high availability. A single Orchestrator can monitor dozens of MySQL sets at the same time for automatic failover.
• DTS service layer: the data-archiving synchronization tool uses mysqldump or other data-migration tools, and the real-time data synchronization tool uses OMS.
• OceanBase cluster layer: to be compatible with the existing access mode and prevent OBProxy single points of failure, an IVS cluster is also placed in front of the OceanBase cluster layer.
• Data warehouse layer: the data warehouse obtains data via full-data pulls and Binlog subscription. It originally extracted data from MySQL via Binlog; after switching to OceanBase, an OceanBase Binlog cluster was deployed to ensure the downstream data-warehouse link runs normally.

Figure 3 Qifu Technology’s database architecture

In this architecture, the machines at the OceanBase cluster layer (for internal use) are primarily physical machines with NVMe SSD storage. As of August 2025, there were 6 OceanBase clusters internally, with 50+ nodes and a total storage capacity already reaching 30 TB+. According to business importance, we planned two OceanBase cluster sizes: small clusters and large clusters.

• Small clusters: with a business department as the cluster granularity. For example, product, risk control, operations, collections, and other businesses each deploy their own small cluster independently, ensuring data and resource isolation between departments.
• Large clusters: scenarios with low timeliness requirements and no strong business coupling — such as reporting, business monitoring, and archiving businesses — are consolidated into a single large cluster for centralized deployment, and this cluster can be scaled out to hundreds of nodes as needed.

Use Cases: Four Typical Technical Scenarios and Migration Experience

During the OceanBase rollout, the database team executed in batches according to the following four typical scenarios, all of which are now live.

1. Data archiving.

Leveraging OceanBase’s high compression, cold data in MySQL is periodically archived into OceanBase via OMS or other import tools, freeing up storage space and reducing storage costs.

2. HTAP scenario.

The backend business data store of the procurement platform involves both read-write operations for real-time business and scenarios needing analytical statistics. The platform originally used TiDB, but because TiDB’s stability and resource isolation could not meet the platform’s monthly-closing peak requirements, the entire procurement platform was migrated to OceanBase.

3. Offline data processing.

The offline data of the reporting business comes from multiple sources such as Excel and upstream MySQL, and a large amount of data lingers and is hard to clean up. There are also batch-job needs that require calling data interfaces from multiple business departments; the business logic is fairly complex and affects processing performance.

4. Write-only scenario.

User-behavior data is written into MySQL with a very large write volume; the business side performs no queries, with only a small amount of manual querying.

The following sections share Qifu’s migration experience in the above scenarios.

1. HTAP Scenario — Procurement Platform Migration

Refactoring Goals

• Performance and scalability: migrate from TiDB to OceanBase for better performance and scalability.
• Stability: TiDB 4.x frequently exhibited performance jitter and could not meet the procurement platform’s stability requirements for HTAP mixed workloads.
• Unified technology stack: OceanBase was being widely adopted internally, so the technology stack needed to be unified.
• Improved data-center disaster-recovery architecture: OceanBase’s financial-grade lossless disaster recovery and cross-data-center switchover capabilities could fill the gap in the existing data-center-level disaster recovery.

Refactoring Plan

• Partition large tables exceeding 10 million rows.
• Partition large tables by month.
• Migrate data from TiDB to OceanBase via OMS.
• Gradually switch business read-write traffic to OceanBase.
• Adapt the downstream big-data team’s platform.

Results

After the migration, overall storage cost dropped by 25%, stability exceeded expectations with zero jitter, and the execution time of statistical-type queries was shortened by 30%.

That said, we also encountered a few minor problems during the migration.

1. Data-consistency problem.

Analysis: unique-key constraints caused some data to be missing in OceanBase (due to inconsistent leading/trailing-space handling between OceanBase and TiDB).

Solution: the problem was confirmed through SQL verification. Combined with the business scenario, the unique-index field was changed to a composite field, and the business filtered out the data rows whose loss was acceptable.

2. Field default-value difference.

Analysis: decimal-type fields default to NULL in OceanBase but to 0.00 in TiDB.

Solution: if there is no business impact, it can be ignored; if there is an impact, the default value can be changed to 0.00.

3. Performance-tuning problem.

Analysis: an improperly set parallelism parameter affected query performance.

Solution: adjust the parallel_min_scan_time_threshold parameter (ms) to reach the optimum for both real-time business queries and AP queries.

4. Parameter-configuration problem.

Analysis: OLTP and OLAP scenarios have different parameter needs.

Solution: set the corresponding parameter configurations separately for different business scenarios — for example, using different parameter combinations for analytical queries and TP-type queries.

2. Write-Only Scenario — Migration of User-Behavior Data

The Original MySQL Environment

The user-behavior data scenario is a write-only scenario, with 1.6 TB of data space and a peak QPS reaching 20k+. The original MySQL used a one-primary, multiple-replica architecture deployed across data centers. When the write volume was large, I/O pressure was high and primary-replica lag was high; moreover, sharding required extensive code refactoring at too high a cost. As it happened, OceanBase was being promoted internally, so migrating the user-behavior data became one of the early use cases for promotion.

Migration Plan

In the early phase, OMS was used for full + incremental synchronization and data-consistency verification. After consistency verification was complete, the business stopped writing, then added OceanBase writes to achieve dual writes to both OceanBase and MySQL. After a month of continuous dual writes with no anomalies, the business confirmed that OceanBase met its needs, so MySQL was decommissioned and only OceanBase was written to. OceanBase has run smoothly ever since, and the business side has never reported any jitter.

It should be noted that the original MySQL schema did not account for distributed characteristics, so after migrating to OceanBase, large tables needed partitions added. As much as possible, choose the partition field according to the business scenario or partition by time. In addition, time-field partitioning may create hot nodes; in severe cases, consider using a second-level partition field to balance the hot spots.

3. SQL Problems Encountered

The database team once encountered two SQL problems. One was that adding a column via DDL could invalidate the SQL execution plan, causing a full table scan. The other was that the original bound execution plan for a query SQL became invalid, causing a sharp performance drop — what originally took only 3 seconds rose to 300 seconds after invalidation. For this, one temporary workaround is to manually perform a full-data major compaction before a DDL column-addition operation when an SQL execution plan invalidation problem appears. But in the long run, the OceanBase cluster needs to be upgraded from version 4.3.2 to 4.3.5, which resolves the issue completely.

At the root, the reason for the sharp SQL performance drop was that the userType table underwent join elimination by OceanBase’s query optimizer: due to the nature of an outer-join primary-key join, the optimizer considered the table redundant and eliminated it, so all hints involving the userType table became invalid. As a result, the query execution plan changed unexpectedly, causing a major performance drop.

For the performance problem caused by join elimination, we adopted three optimization plans.

Optimization plan 1: control the join order of the core tables.

• Use hint: /_+LEADING(a e) use_hash_/
• Effect: query time dropped from 300+ seconds to 48 seconds.

Optimization plan 2: specify the join method and index.

• Use hint: /_+leading(a e) use_nl(e) index(e idx_appl_no)_/
• Effect: query time dropped further to 8 seconds.

Optimization plan 3: prevent join elimination (the optimal plan).

• Use hint: /_+LEADING((a userType) e) index(e idx_appl_no) index(userType PRIMARY) NO_ELIMINATE_JOIN_/
• Key point: use NO_ELIMINATE_JOIN to stop the optimizer from eliminating the join with the userType table.
• Effect: query time dropped to 4 seconds — the best of all the plans.

Among these optimization plans, the core idea for solving the SQL performance problem is to use the NO_ELIMINATE_JOIN hint to force the original table-join relationship to be preserved, preventing the performance regression caused by the optimizer’s over-optimization, ultimately reducing query time from 300 seconds to 4 seconds.

4. Problems Encountered with the OMS Tool

Currently, the internal OMS cluster has 15+ nodes, involving 300+ synchronization tasks and 4000+ database tables. Inevitably, some problems arose. For example:

• Offline scenarios need to synchronize data from upstream MySQL, involving 100+ databases, with a large number of databases and tables, making task deployment cumbersome.
• OMS meta-information was originally stored in MySQL; as the task data volume grew larger and larger, MySQL’s storage came under significant pressure.
• Sharding DDL operations may cause schema inconsistency problems.

The solutions are also fairly simple:

• Through ticketing + interfaces + scripts, automatically create data sources, migrate automatically, and record the original data information.
• Change the OMS data-source storage from MySQL to OceanBase.
• After discussion with the business, adopt the sharding + view approach.

Value and Benefits: Cost Reduction, Efficiency Gains, and Zero Jitter

After adopting OceanBase, Qifu Technology gained benefits beyond expectations in both cost and technology.

Cost Benefits — Multi-faceted Cost Reduction

Storage cost: OceanBase’s high compression ratio directly reduces storage cost. After introducing OceanBase for the archiving business, the cluster’s data volume dropped from 10 TB to 5 TB, halving storage cost.

Hardware cost: after replacing TiDB with OceanBase, the number of machines was also reduced, with the overall server count reduced by 30+.

Operational cost: OceanBase’s online scaling is more convenient and faster than other databases, greatly saving operational labor cost. OceanBase also provides a variety of tools, such as the white-screen management tool OCP, the installation and deployment tool OBD, the data-migration service OMS, and the diagnostic tool obdiag — highly automated with little manual intervention. It is worth mentioning that the RTO triggered by a single node is extremely short, greatly lowering the difficulty of DBA operations.

Code-refactoring cost: thanks to OceanBase’s unique native distributed-database characteristics, the code-refactoring cost previously caused by sharding dropped by 90%.

Technical Benefits — Stronger Business Support

Performance improvement: as a native distributed database, OceanBase can process complex SQL in parallel across multiple nodes, greatly shortening response time and significantly improving performance. As a result, compared with before:

• A single machine can carry more traffic.
• A sudden surge in business traffic does not cause large performance fluctuations, handling traffic spikes well.
• For offline-query batch jobs, the overall task execution time was shortened by 40%.
• Traffic for the user-behavior-logging business increased by 30%, and jitter was eliminated.

Stability improvement: OceanBase offers financial-grade high availability, forging stability as a product advantage. OceanBase has many core technologies for failure detection. Compared with our previous Orchestrator + MySQL primary-replica architecture, it not only has stronger disaster-recovery capabilities — lowering the failure rate and improving system availability — but its operations have almost no impact on the business. After going live with OceanBase, business stability improved significantly, able to carry more business traffic without jitter and effectively shortening the execution time of business tasks.

Scalability: OceanBase supports online scaling, and with white-screen management tools such as OCP, scaling can be performed via a white-screen interface. For scenarios that require capacity management, elastic scaling is very convenient and effectively enhances concurrent-processing capability.

Future Plans: Broad Adoption and a Unified Technology Stack

In the future, Qifu Technology will use OceanBase more deeply and sustainably across four dimensions: technology evolution, business support, team building, and tooling platforms.

Business support: going forward, Qifu Technology will adopt OceanBase as the storage solution in more scenarios. As the business uses OceanBase, the database team will help resolve the problems the business encounters and continuously improve the experience.

Technology evolution: the database team will investigate OceanBase’s vector and KV features to meet the growing need for vector databases. Currently, OceanBase’s application within Qifu Technology is steadily expanding; from an operations perspective, using a unified technology stack is the best solution as long as technical requirements are met.

Team building: Qifu Technology will train OceanBase operations talent and build the related knowledge system. On the one hand, internal groups study OceanBase and regularly hold OceanBase-centered technical sharing sessions within the company. On the other hand, during the learning process, the company accumulates internal knowledge, promotes the technology, and builds up operations experience around OceanBase.

Tooling platforms: Qifu Technology plans to integrate OceanBase operations tools with the internal management platform to increase the automation of OceanBase application lifecycle management. At the same time, it will refine operations procedures and improve operational efficiency — for example, with regular inspections and task consolidation for OMS synchronization tasks.