How Database Technology and Solutions Support the Data Needs of the Pan-Internet Industry
The content of this article is drawn from the e-book A Study of OceanBase Community Edition Use Cases in Pan-Internet Scenarios. Get the full version here: https://open.oceanbase.com/learning
Author: Mei Qing, architect at Zhejiang Yunqu Technology, with 18 years of experience in database operations and architecture. He currently focuses on third-party operations services for databases, including domestic distributed databases.
Driven by the digital economy, data has become a strategic national factor of production and is reshaping the model of industrial productivity. Digital technologies centered on the internet, artificial intelligence, and big data are driving the rapid growth of pan-internet industries such as e-commerce, social networking, fintech, and smart manufacturing (hereafter the “pan-internet industry”). However, the exponential growth of data scale leaves enterprises facing systemic challenges in storage, computing, security, and data governance. To meet these challenges, the focus of database technology in the pan-internet industry has shifted toward distributed databases, which have evolved into a variety of technical architectures.
Different scenarios in the pan-internet industry have different characteristics, and businesses place different demands on the database; these varied scenarios have given rise to different solutions.
(1) Sharding to Address Compute and Storage Bottlenecks
When the internet industry first took off, the growth of data scale pushed traditional centralized databases to their limits in both performance and storage capacity, and scaling hardware vertically was costly with diminishing returns. Leading internet companies were the first to switch to open-source databases (such as MySQL and PostgreSQL). Since a single database has limited compute and storage capacity, they layered a distributed database middleware on top of individual databases. This middleware applied sharding technology (commonly known as “splitting databases and tables”), producing the earliest distributed relational databases. This approach was relatively easy to understand and adopt, and was widely used in the early days.
Foreign companies, led by Google, encountered similar business challenges first and successively released the Spanner and F1 products. Inspired by these technologies and shaped by the needs of their own business scenarios, domestic companies produced two categories of solutions. Neither relies on traditional databases; they were initially called NewSQL and later came to be known as native distributed databases.
The first separates compute from storage. The storage layer uses a Key-Value structure, and when the compute layer parses SQL and fetches data, it maps to KV objects in the storage layer. The compute layer is stateless, making it easy to scale in and out, while the storage layer leverages multi-replica data and replication technology to achieve high availability and online elasticity. Large business tables are split at the storage layer into smaller units distributed across the nodes.
The second does not separate compute from storage. Large business tables are split into smaller partitions using partitioning technology. Likewise, each partition has multiple replicas, with high-availability and data replication technology providing online elasticity.
(2) LSM-Tree Structure with Tiered Data Compression to Reduce Storage Costs
The read-write model of traditional relational databases (such as Oracle and MySQL) is the B-Tree model, whose read-write design is fairly balanced. Such databases do not enable data compression, because doing so would cause a noticeable drop in performance. As a result, in pan-internet scenarios the storage cost of these databases can be very high. The read-write model of NewSQL databases is mostly the LSM-Tree model, which turns all random writes into sequential writes—particularly well suited for use with solid-state storage (SSD). LSM-Tree data is stored in tiers on disk, with different compression algorithms at different tiers, balancing performance and storage cost. This is currently the mainstream approach in the pan-internet industry’s cost-reduction and efficiency-improvement strategies.
(3) Multi-Replica Data Combined with Distributed Consensus Protocols for High Availability
As the pan-internet industry’s use of distributed databases grows ever larger, so too does the high-availability challenge. Localized failures are inevitable, and the requirement is that faults recover quickly and automatically without losing any data. Traditional database techniques such as asynchronous and semi-synchronous primary-standby replication cannot strictly guarantee this. Distributed databases primarily use distributed consensus protocols—represented by Paxos and Raft—to achieve reliable multi-replica failover and data replication. Within the same city, this high-availability capability can deliver a failure-recovery RTO at the second level and an RPO of 0 (zero data loss).
The Raft protocol is a simplified version of Paxos; the two differ in performance and stability under high concurrency. They also have different requirements for network bandwidth and stability. The Paxos model is more complex, which gives it somewhat better overall behavior in high-concurrency, high-throughput scenarios.
As the pan-internet industry expands the scope of database high availability from a single data center to three data centers in the same city, or to a three-data-center-across-two-regions setup, the number of data replicas may grow from 3 to 5, increasing overall storage cost. To cope with the rising pressure of storage and network costs, additional replica types such as log replicas and arbitration replicas have been developed. These make it possible to achieve the original high-availability capabilities with just 2 or 4 data replicas plus one arbitration replica.
(4) Hybrid Row-Column Storage to Serve Both OLTP and OLAP Workloads
As the data scale of the pan-internet industry swells, the early approach was to use separate database products supporting row storage and column storage to serve OLTP and OLAP scenarios respectively. This introduced extensive data synchronization pipelines and additional data-replica redundancy, making the overall cost very high as well. Native distributed databases leverage their multi-replica capability to support hybrid row-column storage within a single replica, or to deploy different replicas as row store or column store. This reduces external data synchronization schemes and lowers the average number of data replicas. In addition, the SQL engine selects row or column storage based on the characteristics of the business SQL and the data volume, improving overall average business performance. This design also avoids data silos.
The above summarizes the architectural characteristics of distributed databases in the pan-internet industry for coping with the challenges of large data volumes and high concurrency. Different databases implement different technologies for sharding, replica types, replica count and synchronization, data storage models and compression, distributed transactions, and more. All can meet business needs to varying degrees; the differences may lie in the data migration experience, the database operations experience, and performance and stability under peak load—areas that still need to be explored in real-world work.