Migrating from HBase to OceanBase: The Ultimate Solution for Real-Time Data Writes with Flink

Posted on In Views: Word count in article: 771 Reading time ≈ 3 mins.
This article introduces a real-time data-write solution based on Flink and OceanBase OBKV-HBase—fully compatible with the HBase API, supporting buffered batch writes and unified stream-batch processing. It provides examples of table creation, Flink SQL mapping, and real-time Kafka writes, along with performance tuning and troubleshooting practices.

Are you frustrated by HBase’s high operations costs?

Are you looking for the best target storage solution for real-time data streams?

Flink + OceanBase OBKV-HBase may be the answer you’ve been waiting for!

Why Should You Care About This Solution?

Scenario 1: The “Last Mile” of HBase Migration

If your team is considering migrating from HBase to OceanBase but worries: how can the existing Flink real-time data streams migrate seamlessly? How can HBase API compatibility be guaranteed? Can throughput, latency, and stability after migration meet production requirements?

This solution is made exactly for you! OceanBase OBKV-HBase is fully compatible with the HBase API and, paired with the Flink connector, lets you migrate smoothly.

Scenario 2: “Data Ingestion” in Real-Time Data Warehouse Construction

If you’re building a real-time data warehouse and need to write the real-time data streams from Kafka into wide-table storage, support high-concurrency writes while meeting near-real-time query and analysis needs, and guarantee data consistency and reliability.

The Flink + OceanBase combination can perfectly solve your pain points! Flink handles stream processing, and OceanBase handles online storage and querying.

Scenario 3: Data Storage in AI/LLM Applications

In the AI era, OceanBase’s HBase mode naturally supports flexible table structures, and paired with Flink’s real-time processing capabilities, you can easily build the data foundation for AI applications.

Core Advantages: Why Choose OceanBase OBKV-HBase?

🚀 HBase API Compatibility

For developers familiar with HBase, the migration cost is nearly zero! Just modify the connection configuration to switch over seamlessly.

⚡ High-Performance Real-Time Writes

Supports buffered batch writes: high-throughput data import for batch scenarios, and millisecond-level latency for real-time scenarios.

🔄 Unified Stream-Batch Processing

Supports both Flink’s streaming writes and batch writes—one system meets multiple needs.

🛡️ Enterprise-Grade Reliability

A robust fault-tolerance mechanism, OceanBase’s native high availability, and support for multi-replica high availability and strong consistency.

Quick Start

OBKV-HBase is OceanBase’s wide-table database compatible with the HBase interface. The Flink OBKV HBase connector (flink-connector-obkv-hbase) is implemented on obkv-hbase-client-java and supports writing data in real time to OceanBase’s HBase-mode tables via Flink SQL.

Step 1: Create an HBase Table

In OceanBase, HBase tables are mapped via a naming convention: each column family corresponds to one physical table, in the format hbase_table_name$family_name.

1
2
3
4
5
6
7
CREATE TABLE `user_info$basic` (
  `K` varbinary(1024) NOT NULL,    -- rowkey
  `Q` varbinary(256)  NOT NULL,    -- qualifier
  `T` bigint(20)      NOT NULL,    -- timestamp
  `V` varbinary(1024) DEFAULT NULL, -- value
  PRIMARY KEY (`K`, `Q`, `T`)
);
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
CREATE TABLE user_info_sink (
  rowkey STRING,
  basic ROW<name STRING, age INT>,
  contact ROW<email STRING, phone STRING>,
  PRIMARY KEY (rowkey) NOT ENFORCED
) WITH (
  'connector' = 'obkv-hbase',
  'odp-mode' = 'true',
  'odp-ip' = '127.0.0.1',
  'odp-port' = '2885',
  'schema-name' = 'your_database',
  'table-name' = 'user_info',
  'username' = 'user@tenant#cluster',
  'password' = 'your_password'
);

Steps 5-6: Write Test Data and Verify

1
2
3
INSERT INTO user_info_sink VALUES
  ('user001', ROW('Alice', 25), ROW('alice@example.com', '13800138000')),
  ('user002', ROW('Bob', 30), ROW('bob@example.com', '13900139000'));

Configuration Parameters

Required Parameters

  • connector: fixed value obkv-hbase
  • username: format user@tenant#cluster
  • password: user password
  • schema-name: database name
  • table-name: HBase table name

Connection Modes

Config URL mode (direct connection): requires url, sys.username, sys.password
ODP mode (proxy): set odp-mode=true, odp-ip, odp-port

Performance Tuning

  • sync-write: whether to write synchronously (default false, buffer recommended)
  • buffer-flush.buffer-size: buffer size (default 1000 rows)
  • buffer-flush.interval: flush interval (default 1s)
  • max-retries: maximum number of retries (default 3)

Usage Example

Real-Time Writes from Kafka

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
CREATE TABLE kafka_source (
  user_id STRING, user_name STRING, age INT,
  email STRING, phone STRING, event_time TIMESTAMP(3),
  WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
) WITH (
  'connector' = 'kafka',
  'topic' = 'user-events',
  'properties.bootstrap.servers' = 'localhost:9092',
  'format' = 'json'
);

INSERT INTO user_hbase_sink
SELECT user_id,
  ROW(user_name, age) AS profile,
  ROW(email, phone) AS contact
FROM kafka_source;

Best Practices

Performance Optimization

  • High-throughput batch: buffer-size=5000, interval=5s
  • Low-latency real-time: buffer-size=500, interval=500ms
  • Ultra-low latency: sync-write=true

Troubleshooting

  • Connection failure: check the network, verify the username and password, confirm the ODP configuration
  • Write failure: check the Flink logs, verify the table structure, confirm permissions
  • Slow performance: increase the buffer, raise parallelism, check cluster load

Reference Information

Thanks to Sun Chaoyang and Zhao Mingyuan from the ecosystem team for their professional guidance.