NetEase Open-Sources lb-driver: Driver-Layer Load Balancing for OceanBase OBProxy

Posted on In Views: Word count in article: 822 Reading time ≈ 3 mins.
After migrating its core business from the DDB sharding middleware to OceanBase, NetEase Yunxin built and open-sourced lb-driver to eliminate the link latency and operational pain caused by SLB load balancing. It implements Statement-granularity OBProxy load balancing, health checks, and dynamic scaling directly in the JDBC driver layer.

Preface

In enterprise-grade distributed database deployments, OceanBase has become the distributed database of choice for core workloads in finance, social, e-commerce, and more, thanks to its high availability, high throughput, and horizontal elastic scaling. As part of its architecture upgrade, NetEase Yunxin also chose OceanBase as the database for its core business, replacing the in-house sharding middleware DDB.

A typical OceanBase architecture relies on an OBProxy proxy cluster to accept front-end connections, route SQL requests, and manage partition reads and writes. In traditional integration designs, applications usually rely on SLB / NLB / Layer-4 load balancing as the traffic entry point for the OBProxy cluster. After switching from DDB to OceanBase, we noticed that the RT of some business SQL had increased. On analysis, we concluded that besides the added latency from OceanBase’s Paxos strong consistency, the extra hop through the load balancer on the network path was also non-negligible.

Pain Points

The mainstream OceanBase integration architecture: Application Service → Database Connection Pool → SLB Load Balancing → OBProxy Cluster → OBServer Cluster

This standard architecture generally works fine, but under high-concurrency, high-traffic scenarios it has some clear problems:

  • Redundant link, higher latency. The extra SLB forwarding adds a network hop, increasing the RT of a single SQL request (typically around 0.2ms); under high-concurrency peak traffic, the latency amplification becomes even more pronounced.
  • Coarse load-balancing granularity. SLB is a standard Layer-4 load balancer; it isn’t aware of the business protocol and can only balance load per connection, which can lead to uneven OBProxy load.
  • Performance bottleneck. All database traffic converges on the SLB cluster, so during promotions or peak traffic it’s prone to connection saturation and bandwidth bottlenecks.
  • Less-than-smooth scaling operations. Database connections are generally long-lived, so bringing OBProxy nodes online/offline or updating them takes a long cycle, and sometimes requires applications to coordinate restarts.
  • Limited fault detection. Layer-4 load balancers like SLB only support TCP port-connectivity health checks, and can’t detect anomalies such as a port that’s reachable but a process that’s hung and unable to handle SQL.

Client-Side Load Balancing (lb-driver)

Inspired by DDB-lbd

In fact, the DDB sharding middleware we originally used already supported lbd, with the architecture: Application Service → Database Connection Pool → LBD → QS Cluster → MySQL Cluster. As you can see, DDB and OceanBase share some architectural similarities. However, ddb-lbd’s implementation is coupled to DDB, so it couldn’t be reused directly for OceanBase. We therefore decided to draw on the design ideas of ddb-lbd, take the best and improve the rest, and re-implement a more general lb-driver tailored to OceanBase workloads.

Load-Balancing Principle

Under the Layer-4 SLB model, the load-balancing strategy is obviously connection-oriented. But once we push the strategy down into the driver layer, we can do much more. In lb-driver’s implementation, we balance load at JDBC’s Statement granularity — you can think of it as analogous to how nginx, in a Layer-7 proxy scenario, distributes requests at the granularity of individual HTTP requests.

Health Checks and Automatic Eviction of Faulty Nodes

lb-driver automatically probes all OBProxy nodes, periodically sending a select 1 business heartbeat to determine whether a node is healthy. It also monitors the execution of business SQL: whenever a specific anomaly occurs (such as a connection-establishment timeout, or OceanBase error codes in the -9000 to -8000 range), it immediately marks the connection as unhealthy and, once the error count reaches a threshold, blocks the affected node outright.

Fast Node Scaling

You can configure the full list of OBProxy nodes directly in the address string, or fetch the OBProxy node list dynamically via a standalone config-server service. config-server can use etcd or nacos for dynamic configuration, allowing the OBProxy cluster to scale up or down dynamically without restarting the application.

Non-Intrusive to the Business

lb-driver depends only on slf4j and mysql-connector-java. You simply add the lb-driver dependency to your project and replace com.mysql.jdbc.Driver with com.netease.nim.lbd.LBDriver — no business-code changes required.

Overall Architecture

lb-driver comprises the following components: a separately deployed config-server, the SqlProxyProvider (which provides the OBProxy node list), the ConnectionManager (which manages the connection lifecycle and the statement-level load-balancing strategy), and two scheduled tasks, balance and detect (for connection-level load balancing and health checking).

Production Practice

Currently, all of NetEase Yunxin’s online OceanBase clusters have migrated to the lb-driver model, and other departments are gradually migrating to the lbd model as well.

Outlook

lb-driver is now open-sourced on GitHub: https://github.com/netease-im/lb-driver (stars and forks welcome, and feel free to open issues with any questions). We are beneficiaries of the OceanBase Community Edition, and as lb-driver is part of the OceanBase open-source ecosystem, we hope to give back to the open-source community and build together with everyone.