Data-Driven Innovation and Transformation in the Pan-Internet Industry, and New Opportunities for Databases

This article is excerpted from the e-book A Study of OceanBase Community Edition Use Cases in Pan-Internet Scenarios. To get the full version, click “Read the Original” at the end of the article.

Author: Liu Huayang, currently a database architect at a SaaS company in the pan-internet industry. He has 20 years of experience in databases: spanning traditional industries to the pan-internet industry, and covering large state-owned enterprises, well-known joint-stock companies, and foreign-funded financial institutions; from on-premises databases to cloud-native databases. He excels at comprehensively analyzing specific business needs and other factors to select the right database product and reduce a company’s overall database costs.

Introduction

In recent years, the internet industry has been undergoing unprecedented change. As cloud computing, IoT, AI, and other technologies have matured and been adopted, the internet industry is no longer confined to technology-based online services and products such as e-commerce and digital payments. Instead, it is rapidly and deeply integrating with the real economy, offline products, and the customer experience, forming a broader pan-internet industry (hereafter the “pan-internet industry”) and creating a new “Internet of Everything” business landscape.

In this new Internet of Everything landscape, market demand is more diverse, personalized, and intelligent than before. This places higher demands—such as stability, scalability, and real-time processing capability—on the underlying data storage and management infrastructure: the database management system. To the pan-internet industry, the database is what coal, water, and electricity are to daily life—it safeguards the most basic lifeline of the business.

1. Data-Driven Innovation and Transformation in the Pan-Internet Industry

In the popular imagination, the “internet” means new technology and gives the impression of “burning through cash.” In reality, today’s pan-internet industry is becoming increasingly “practical.” Through happy marriages with traditional industries, many offline brick-and-mortar stores use mini-programs, livestreaming, and other technology channels to combine customer traffic with service, achieving a virtuous cycle of business operations by acquiring customers online and delivering the experience offline. For example, Pop Mart’s offline “trendy toy blind-box vending machines,” combined with its online “blind-box draw” mini-program, create an integrated online-retail model.

According to data from the National Bureau of Statistics, China’s national online retail sales reached 15.4 trillion yuan in 2023, up 11.0% year over year. Of this, online retail sales of physical goods reached 13.0 trillion yuan, up 8.4%, accounting for 27.6% of total retail sales of consumer goods.

Online retail is essentially the “Internet+” that everyone talked about ten years ago—that is, industries of all kinds connecting to the internet to launch online models such as online retail and online education. Beyond online retail, the pan-internet industry also includes the industrial and manufacturing sectors, whose transformation is reflected in the digital transformation that has been booming in recent years: factories connect to digital platforms that unify the management of production equipment, production processes, delivery coordination, intelligent control, and operational decision-making within a single system. Through a “data + models + applications” approach, they optimize the enterprise’s traditional production, operations, and service models.

2. New Opportunities the Pan-Internet Industry Brings to Database Technology

In line with the consistent demands of enterprise development, requirements for databases boil down to four areas: maximizing cost-effectiveness, maximizing efficiency, security and compliance, and flexible deployment.

(1) Strengthening Data Processing and Comprehensive Analytics Capabilities

As the data scale of the pan-internet industry intensifies, future database systems must possess HTAP capabilities and the ability to handle multiple data formats. The traditional dual-track architecture of a transaction processing system plus a data warehouse involves complex synchronization data pipelines; maintaining data consistency across different systems is difficult, increasing both architectural complexity and operational cost.

Key technical capabilities include:

1. HTAP-native engine — A unified storage and compute architecture in which the same data replica supports both OLTP and OLAP.
2. Vectorization and in-memory computing optimization — Millisecond-level transaction response and high-throughput analytical queries.
3. Relational + semi-structured multi-model integration — Unified SQL access and seamless cross-model queries.
4. Extended models such as time-series and vector — Multi-model data processing for “one database, many uses.”

(2) More Stringent High-Concurrency and Low-Latency Transaction Requirements

For businesses that are both complex and real-time, you need to support millions of concurrent accesses while guaranteeing data consistency and availability. Specifically:

1. Ultra-high-concurrency transaction engine — A distributed lock-free architecture and MVCC.
2. Low-latency response — Low-latency feedback between primary and secondary nodes within 100 milliseconds.
3. Tunable transaction isolation and consistency — Multiple isolation levels, with dynamic switching between strong consistency and performance.

(3) Providing Lower-Cost Cloud Deployment Capabilities

More and more enterprises require flexible database deployment across public cloud, private cloud, and hybrid cloud environments:

1. Multi-cloud deployment and cross-cloud data consistency
2. Reduced dependence on cloud vendors and optimized resource utilization
3. Cross-region, cross-data-center hybrid cloud deployment

(4) Refocusing Operations on the Business and Deepening AIOps

1. Protocol compatibility and seamless migration — Native compatibility with MySQL, PostgreSQL, and Oracle.
2. Zero-barrier operational experience — A visual console and a SQL IDE.
3. AI-driven, fully automated operations — Intelligent alerting, self-healing of faults, and automatic index optimization.
4. Multi-language SDKs and APIs — Python, Java, Go, Node.js, and more.
5. Visual operations and monitoring dashboards — Remote management, metric drill-down, and custom alerts.

(5) Security, Compliance, and Database Security Technology Support

1. Fine-grained access control — RBAC and data isolation.
2. Data access auditing and logging — Tracking access to core sensitive data.
3. Region-specific data protection — Meeting compliance requirements across different regions.
4. Anomaly detection and protection powered by AI — Real-time monitoring and intrusion detection.

Conclusion

Driven by data, market demand in the pan-internet industry is becoming ever more diverse. Databases must make greater breakthroughs in data analytics, concurrent processing, cost control, ease of use, hybrid cloud, appliance form factors, and security and compliance. Only by riding the tide of the times, following market developments, and continuously pursuing technical innovation and ecosystem collaboration can databases truly empower enterprises to achieve sustained growth in a rapidly changing market.