OceanBase Offline Installation

Posted on In , Views: Word count in article: 5.7k Reading time ≈ 21 mins.
A guide to the offline installation of an OceanBase distributed three-replica cluster, covering passwordless SSH, disk planning, deploying OBServer/OBProxy with OBD, and tenant verification end to end.

Summary

I’m sharing the offline installation document I wrote back in August. I may publish a series of usage documents afterward.

This article aims to run OceanBase for production workloads and deploys the distributed version. If you just want to test and try out OceanBase quickly, please refer to https://open.oceanbase.com/quickStart
​

Installation Flow

​OceanBase Offline Installation

Preparation Before Installation

Before installation, all operations are performed as root. During the installation process, you can use the corresponding regular user.

Terminology

  • OBD: OceanBase Deployer, the OceanBase deployment tool
  • Control machine: the machine running the OBD installation package
  • OBServer: the OceanBase database process/service running on each physical machine where OceanBase is installed
  • OBProxy: OceanBase Proxy, OceanBase’s high-performance reverse proxy server. It offers features such as preventing connection drops, masking back-end anomalies (crashes, upgrades, network jitter), MySQL protocol compatibility, strong validation, hot upgrade support, and multi-cluster support.
    ​

Deployment Model

This example uses the classic three-replica deployment model with 4 machines:

  • 1 machine deploys OBProxy – it’s recommended to deploy the client application together with OBProxy to reduce the initial network latency.
  • A 1-1-1 deployment of a 3-replica OceanBase cluster, where each zone represents one replica. In this example, a zone contains only one machine. In production, the three zones are often deployed in a “two regions, three centers” model: three data centers with three replicas, one replica per data center, with two of the data centers located close to each other.

​

Software and Hardware Requirements

Item Description
OS Red Hat Enterprise Linux Server 7.x (kernel Linux 3.10.0 or above)
CentOS Linux 7.x (kernel Linux 3.10.0 or above)
Anolis OS 8.x (kernel Linux 3.10.0 or above)
CPU Enterprise users: minimum 16 cores, 32 cores or above recommended
Personal testing: minimum 2 cores, 8 cores or above recommended
Memory Enterprise applications: minimum 64G, 256G or above recommended
Personal testing: minimum 8G, 64G or above recommended
Disk type SSD recommended
Disk space 4 times the memory size or more
File system ext4 or xfs; use xfs when the data volume exceeds 16TB
NIC Gigabit interconnect or above

Set Up Passwordless SSH Login

Before installation, you need to configure the environment on each machine. These settings must all be done as the superuser. It is recommended to set up trusted login (i.e., passwordless login) from the control machine to the OBServer and OBProxy machines. For how to set up passwordless SSH login, see https://open.oceanbase.com/docs/community/oceanbase-database/V3.1.0/optional-set-password-free-ssh-logon
​

Two scripts are recommended to make it easy to run commands and copy files in batches across the cluster.

Batch copy files; you can replace the host list with your own actual machine list.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#/usr/bin/bash


hosts=(
    "ob001"
    "ob002"
    "ob003"
    "obdriver"
    )
    for host in "${hosts[@]}"
    do

        echo "begin to scp " $@ " on " $host
        scp -r $1 $host:$2
    done

Batch run commands; you can replace the host list with your own actual machine list.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
/usr/bin/bash



hosts=(
    "ob001"
    "ob002"
    "ob003"
    "obdriver"
    )
    for host in "${hosts[@]}"
    do
        echo "begin to run " $@ " on " $host
        ssh $host $@
    done

Create the Operating User

For personal testing, you can use the root account directly. For enterprise users, it’s recommended to create a regular user to avoid security impact on the system. In this example, admin is used as a demonstration; enterprise users can use whatever account they typically use as needed.
​

1
2
3
useradd -U admin -d /home/admin -s /bin/bash
mkdir -p /home/admin
sudo chown -R admin:admin /home/admin

Set the password

1
passwd admin

​

Set sudo privileges

vi /etc/sudoers # add a line for oceanbase

1
2
3
4

# Add the following lines
# %wheel        ALL=(ALL)       NOPASSWD: ALL
admin       ALL=(ALL)       NOPASSWD: ALL

Disk Planning

The OceanBase database server relies on 3 directories. For personal testing, you can put all data on a single disk, but for enterprise users you must mount 3 separate disks: the data disk, the transaction log disk, and the OBServer installation disk. When a machine doesn’t have 3 disks, or when using a RAID array, you need to partition the disk or the logical volume of the disk array into 3 partitions. For partition sizes, refer to the notes below:

  • Data disk

    • The configuration parameter is data_dir. Plan the data disk well according to business needs. The data disk holds the baseline data; physically there is only one baseline data file, block_file, under the installation directory store/sstable. It is created in one shot when the OBServer process starts, with its size pre-allocated based on the startup parameter datafile_disk_percentage (default 95%), and it cannot be resized after creation. OceanBase scales out and in by adding and removing machines; single-node disk-level scale-up and scale-down are not currently supported.

  • Transaction log disk

    • The configuration parameter is redo_dir. The recommended size is 3 to 4 times the OBServer memory or more. The transaction log disk contains multiple fixed-size small files, located in the installation directory store/{clog,ilog,slog}, automatically created and cleaned on demand. When the disk fills to 80%, self-cleanup logic is triggered, but only on the premise that the in-memory data corresponding to this log data has already been merged into the baseline data, allowing it to be deleted. For the same amount of data, the transaction log size is roughly three times the size of the in-memory data. So the upper limit of space required by the transaction log disk is proportional to the total business data between two merge operations. The empirical formula is: transaction log file size = 3 to 4 times the upper limit of incremental data in memory.

  • OBServer installation disk

    • The configuration parameter is home_path. The recommended size is 200G or above (to keep 7 days or more of logs). The OceanBase rpm package installation directory is under /home/admin/oceanbase, where the baseline data file and transaction log file are linked via symbolic links to the two separate disks described above. There is also another continuously growing file, the OB runtime log, under the installation directory log. The OB process itself cannot self-delete the runtime logs; a scheduled task or operations script is needed to handle the deletion logic.

After partitioning the disks, you can check with the df -h command. The result is as follows:

​OceanBase Offline Installation

In this example: /data is the data disk, 1TB in size; /redo stores the redo logs; /home/admin/oceanbase stores the OceanBase binary and runtime logs.
​

Check directory permissions

1
2
3
4
5
6
7
8
ls –al # run this command
drwxr-xr-x 2 admin admin 4096 Feb 9 18:43 
drwxr-xr-x 2 admin admin 4096 Feb 9 18:43 log1

If the admin user lacks permission, run the following commands as root 
chown -R admin:admin /data
chown -R admin:admin /redo
chown -R admin:admin /home/admin

​

Pre-check

For enterprise users, it’s recommended that the hardware configuration and software configuration (operating system, OS kernel, glibc, python, and other packages) of all machines running OBServer be consistent, and that the OBProxy machines and OBServer machines have consistent software configuration (operating system, OS kernel, glibc, python, and other packages).

Check the Operating System

The currently supported operating systems are:
​

Red Hat Enterprise Linux Server 7.x (kernel Linux 3.10.0 or above)
CentOS Linux 7.x (kernel Linux 3.10.0 or above)
​

  1. Log in to the server as the root user
  2. Check the OS version

RedHat7 displays as follows:

1
2
[root@redhat-04 /root]#cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Maipo)

CentOS7 displays as follows:

1
2
[root@centos-01 /root]#cat /etc/redhat-release 
CentOS Linux release 7.2.1511 (Core)

On an Anolis system:

1
2
3
4
5
6
7
8
9
10
[root@anolis ~]# cat /etc/os-release
NAME="Anolis OS"
VERSION="8.2"
ID="anolis"
ID_LIKE="rhel fedora centos"
VERSION_ID="8.2"
PLATFORM_ID="platform:an8"
PRETTY_NAME="Anolis OS 8.2"
ANSI_COLOR="0;31"
HOME_URL="https://openanolis.org/"

​

Other systems, such as Debian9, display as follows:

1
2
3
4
5
6
7
8
9
10
root@ob001:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

On an Ubuntu system:

1
2
3
4
5
6
7
8
9
10
11
12
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

For some systems, such as Ubuntu/Debian, you need to install yum:

1
2
3
4
5
6
7
sudo apt-get update
sudo apt-get install build-essential 
sudo apt-get install yum -y
sudo apt install yum-utils -y
sudo ln -s /bin/bash /bin/sh
apt-get install alien -y
apt-get install rpm
  1. Check the kernel version. The OS is required to be 3.10.0 or above.
1
2
[root@centos-01 /root]#uname -r 
3.10.0-327.el7.x86_64

Check Memory

1
free -g

Enterprise applications: minimum 64G, 256G or above recommended.

If free -g

​OceanBase Offline Installation

shows that the memory in the free column is less than the memory_limit configured in the configuration file, you need to clear the cache or modify the memory_limit configuration so that memory_limit is smaller than the value in the free column. The cache-clearing operation is as follows:

1
# echo 3 > /proc/sys/vm/drop_caches

​

Check Disks

​

Make sure that the disks corresponding to data_dir, redo_dir, home_path in the configuration file have been mounted, that the directories corresponding to data_dir and redo_dir are empty, and that the disk usage of the directory corresponding to data_dir is below 4%.
​OceanBase Offline Installation

Check the NIC Name

​OceanBase Offline Installation
In the configuration file there is a configuration item “devname” that needs to specify the NIC. When starting the OBServer service, you need to specify the NIC with the “-i” parameter. A server may have multiple NICs and multiple IPs, and OBServers communicate with each other relying on the specified NIC and IP. You can use the ifconfig command to view the NIC name (you need to install the net-tools dependency package first); just make sure a valid NIC exists.

In this example:
​OceanBase Offline Installation

Configure limits.conf

ulimit is used to limit the resources used by processes started by the shell. For personal testing you can skip this, but enterprise users must set it.

There are two ways to modify resource limits: one is to specify them at the session level at startup, and the other is to modify the /etc/security/limits.conf configuration file, which takes effect globally.

The limits involved by the OBServer process include the maximum thread stack space size (stack), the maximum number of file handles (open files), and the core file size (core file size).

As shown below, when starting the OBServer process, set the maximum stack space size to unlimited, the maximum number of file handles to 655350, and the core file size to unlimited, all at the session level.

1
2
3
4
5
6
7
8
9
10
11
$vi /etc/security/limits.conf add
root soft nofile 655350
root hard nofile 655350
* soft nofile 655350
* hard nofile 655350
* soft stack 20480
* hard stack 20480
* soft nproc 655360
* hard nproc 655360
* soft core unlimited
* hard core unlimited

Exit the current session and log in again.
​

Check whether the configuration has taken effect

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
ulimit -a
# Run this command; the resource limit details are as follows (blocks, -c) 0
core file size
data seg size scheduling priority file size
pending signals max locked memory
(kbytes, -d) unlimited (-e) 0
(blocks, -f) unlimited (-i) 772861
(kbytes, -l) 64
max memory size
open files
pipe size
POSIX message queues real-time priority
stack size
cpu time
max user processes virtual memory file locks
(kbytes, -m) unlimited (-n) 1024
(512 bytes, -p) 8
(bytes, -q) 819200
(-r) 0 (kbytes, -s) 8192
(seconds, -t) unlimited (-u) 655360
(kbytes, -v) unlimited (-x) unlimited

​

Configure the “sysctl.conf” File

To ensure OceanBase runs properly, please modify the “/etc/sysctl.conf” configuration on all physical machines before installing OceanBase (to improve Linux system performance).

Some parameters have already been set by the operating system in advance.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# for oceanbase
## Modify the kernel asynchronous I/O limit
fs.aio-max-nr=1048576

## Network optimization
net.core.somaxconn = 2048
net.core.netdev_max_backlog = 10000 
net.core.rmem_default = 16777216 
net.core.wmem_default = 16777216 
net.core.rmem_max = 16777216 
net.core.wmem_max = 16777216

net.ipv4.ip_local_port_range = 3500 65535 
net.ipv4.ip_forward = 0 
net.ipv4.conf.default.rp_filter = 1 
net.ipv4.conf.default.accept_source_route = 0 
net.ipv4.tcp_syncookies = 0 
net.ipv4.tcp_rmem = 4096 87380 16777216 
net.ipv4.tcp_wmem = 4096 65536 16777216 
net.ipv4.tcp_max_syn_backlog = 16384 
net.ipv4.tcp_fin_timeout = 15 
net.ipv4.tcp_max_syn_backlog = 16384 
net.ipv4.tcp_tw_reuse = 1 
net.ipv4.tcp_tw_recycle = 1 
net.ipv4.tcp_slow_start_after_idle=0

vm.swappiness = 0
vm.min_free_kbytes = 2097152

# This is the data directory of oceanbase
kernel.core_pattern = /data/core-%e-%p-%t

Here, in “kernel.core_pattern = /data/core-%e-%p-%t”, /data is the data directory of OceanBase. In addition, for personal testing you can also set only “fs.aio-max-nr=1048576”.
​

1
2
# Make the configuration take effect
sysctl -p

​

Disable the Firewall and SELinux

Personal testing can skip this, but it’s recommended for enterprise users.

Disable firewalld

1
2
3
4
# Run these 3 commands in order 
systemctl disable firewalld 
systemctl stop firewalld
systemctl status firewalld

Disable SELinux

vi /etc/selinux/linux

1
2

SELINUX=disabled

Run this command

1
2
3
setenforce 0
# Check that the configuration has taken effect
cat /etc/selinux/config

Set Up Clock Synchronization

You can skip setting up clock synchronization in any of the following situations:

  1. The NTP clock is already in sync
  2. Deployed as a standalone version
  3. Personal testing

​

The time on each server in an OceanBase cluster must stay consistent; otherwise, the OceanBase cluster will fail to start and faults will occur at runtime. For enterprise users, clock synchronization is extremely important. A deviation of under 50ms between the physical machine and the time server can be considered in sync, and the maximum tolerated deviation cannot exceed 200ms. When it exceeds 200ms, a no-leader situation occurs; after restoring clock synchronization, restarting the observer can recover the state.

Check Clock Synchronization

1
sudo clockdiff  $IP

Configure Clock Synchronization

https://open.oceanbase.com/docs/community/oceanbase-database/V3.1.0/optional-configuring-clock-sources
​

Installation

Installation Package Components

Download all installation packages from https://open.oceanbase.com/softwareCenter/community. The package versions shown in this article may already be outdated, so please download the latest version of the packages from the open-source OceanBase official site.
​OceanBase Offline Installation
If your machine can access the public network and add a third-party YUM software source, you can run the following commands to install OBD using OceanBase’s official software source:

1
2
3
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://mirrors.aliyun.com/oceanbase/OceanBase.repo
sudo yum install -y ob-deploy

​

scp all the software packages to the control machine.

​

Install OBD

Currently using the root user; this operation is performed only on the control machine.

Online Installation

1
yum install -y ob-deploy

Local Installation

CentOS or RedHat

1
yum install ob-deploy-1.1.0-1.el7.x86_64.rpm

Ubuntu/Debian

1
alien -i ob-deploy-1.1.0-1.el7.x86_64.rpm

Install OBLibs

Currently using the root user; this needs to be run on every machine.

Online Installation

1
yum install -y oceanbase-ce-libs

Local Installation

First copy oceanbase-ce-libs-3.1.0-3.el7.x86_64.rpm to each machine.
​

CentOS or RedHat or Anolis

1
yum install oceanbase-ce-libs-3.1.0-3.el7.x86_64.rpm

Ubuntu/Debian

1
alien -i oceanbase-ce-libs-3.1.0-3.el7.x86_64.rpm

Install OBServer & OBProxy

Switch to the admin user.
​

Add the OceanBase Database Offline Software Packages to the Local Mirror

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
admin@obdriver:/data/rpm$ obd mirror clone *.rpm
name: libobclient
version: 2.0.0
release:2.el7
arch: x86_64
md5: f73cae67e2ff5be0682ac2803aba33a7ed26430e
add libobclient-2.0.0-2.el7.x86_64.rpm to local mirror
name: obclient
version: 2.0.0
release:2.el7
arch: x86_64
md5: 1d2c3ee31f40b9d2fbf97f653f549d896b7e7060
add obclient-2.0.0-2.el7.x86_64.rpm to local mirror
name: ob-deploy
version: 1.1.0
release:1.el7
arch: x86_64
md5: c01dbbebc7f44b700833ce6846df09f20033675c
add ob-deploy-1.1.0-1.el7.x86_64.rpm to local mirror
name: obproxy
version: 3.1.0
release:1.el7
arch: x86_64
md5: 0b17cf0459a3b53c5a2febb6572894d183154c64
add obproxy-3.1.0-1.el7.x86_64.rpm to local mirror
name: oceanbase-ce
version: 3.1.0
release:3.el7
arch: x86_64
md5: b73bcd531bdf3f087391991b290ff2cbcdaa0dc9
add oceanbase-ce-3.1.0-3.el7.x86_64.rpm to local mirror
name: oceanbase-ce-libs
version: 3.1.0
release:3.el7
arch: x86_64
md5: 528144ec7ff0194a8b326491a396b8f5c87b1eaa
add oceanbase-ce-libs-3.1.0-3.el7.x86_64.rpm to local mirror


1
2
3
4
5
6
7
8
9
10
11
12
13
admin@obdriver:~$ obd mirror list local
+-------------------------------------------------------------------------------------------+
|                                     local Package List                                    |
+-------------------+---------+---------+--------+------------------------------------------+
| name              | version | release | arch   | md5                                      |
+-------------------+---------+---------+--------+------------------------------------------+
| libobclient       | 2.0.0   | 2.el7   | x86_64 | f73cae67e2ff5be0682ac2803aba33a7ed26430e |
| obclient          | 2.0.0   | 2.el7   | x86_64 | 1d2c3ee31f40b9d2fbf97f653f549d896b7e7060 |
| ob-deploy         | 1.1.0   | 1.el7   | x86_64 | c01dbbebc7f44b700833ce6846df09f20033675c |
| obproxy           | 3.1.0   | 1.el7   | x86_64 | 0b17cf0459a3b53c5a2febb6572894d183154c64 |
| oceanbase-ce      | 3.1.0   | 3.el7   | x86_64 | b73bcd531bdf3f087391991b290ff2cbcdaa0dc9 |
| oceanbase-ce-libs | 3.1.0   | 3.el7   | x86_64 | 528144ec7ff0194a8b326491a396b8f5c87b1eaa |
+-------------------+---------+---------+--------+------------------------------------------+

Download the Configuration Files

Download all the configuration files from https://github.com/oceanbase/obdeploy/tree/master/example/autodeploy
There are several configuration files:
​

​

In this example, we use the distributed example, and we scp the distributed configuration file to the control machine.

Modify the Configuration File

In this example, distributed-with-obproxy-example.yaml is used.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
## Only need to configure when remote login is required
# user:
#   username: your username
#   password: your password if need
#   key_file: your ssh-key file path if need
#   port: your ssh port, default 22
#   timeout: ssh connection timeout (second), default 30
oceanbase-ce:
  servers:
    - name: z1
      # Please don't use hostname, only IP can be supported
      ip: 192.168.1.2
    - name: z2
      ip: 192.168.1.3
    - name: z3
      ip: 192.168.1.4
  global:
    # The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field.
    home_path: /root/observer
    # The directory for data storage. The default value is $home_path/store.
    # data_dir: /data
    # The directory for clog, ilog, and slog. The default value is the same as the data_dir value.
    # redo_dir: /redo
    # External port for OceanBase Database. The default value is 2881.
    # mysql_port: 2881
    # Internal port for OceanBase Database. The default value is 2882.
    # rpc_port: 2882
    # Defines the zone for an observer. The default value is zone1.
    # zone: zone1
    # The maximum running memory for an observer. When ignored, autodeploy calculates this value based on the current server available resource.
    # memory_limit: 58G
    # The percentage of the maximum available memory to the total memory. This value takes effect only when memory_limit is 0. The default value is 80.
    # memory_limit_percentage: 80 
    # The reserved system memory. system_memory is reserved for general tenants. The default value is 30G. Autodeploy calculates this value based on the current server available resource.
    # system_memory: 22G
    # The size of a data file. When ignored, autodeploy calculates this value based on the current server available resource.
    # datafile_size: 200G
    # The percentage of the data_dir space to the total disk space. This value takes effect only when datafile_size is 0. The default value is 90.
    # datafile_disk_percentage: 90
    # System log level. The default value is INFO.
    # syslog_level: INFO
    # Print system logs whose levels are higher than WARNING to a separate log file. The default value is true. The default value for autodeploy mode is false.
    # enable_syslog_wf: false
    # Enable auto system log recycling or not. The default value is false. The default value for autodeploy mode is on.
    # enable_syslog_recycle: true
    # The maximum number of reserved log files before enabling auto recycling. When set to 0, no logs are deleted. The default value for autodeploy mode is 4.
    # max_syslog_file_count: 4
    # Cluster name for OceanBase Database. The default value is obcluster. When you deploy OceanBase Database and obproxy, this value must be the same as the cluster_name for obproxy.
    # appname: obcluster
    # Password for root. The default value is empty.
    # root_password:
    # Password for proxyro. proxyro_password must be the same as observer_sys_password. The default value is empty.
    # proxyro_password:
  z1:
    zone: zone1
  z2:
    zone: zone2
  z3:
    zone: zone3
obproxy:
  servers:
    - 192.168.1.5
  global:
    # The working directory for obproxy. Obproxy is started under this directory. This is a required field.
    home_path: /root/obproxy
    # External port. The default value is 2883.
    # listen_port: 2883
    # The Prometheus port. The default value is 2884.
    # prometheus_listen_port: 2884
    # rs_list is the root server list for observers. The default root server is the first server in the zone.
    # The format for rs_list is observer_ip:observer_mysql_port;observer_ip:observer_mysql_port.
    # Ignore this value in autodeploy mode.
    # rs_list: 127.0.0.1:2881
    # Cluster name for the proxy OceanBase Database. The default value is obcluster. This value must be set to the same with the appname for OceanBase Database.
    # cluster_name: obcluster
    # Password for obproxy system tenant. The default value is empty.
    # obproxy_sys_password:
    # Password for proxyro. proxyro_password must be the same with proxyro_password. The default value is empty.
    # observer_sys_password:
1
2
3
4
5
6
7
## Only need to configure when remote login is required
# user:
#   username: your username
#   password: your password if need
#   key_file: your ssh-key file path if need
#   port: your ssh port, default 22
#   timeout: ssh connection timeout (second), default 30

Modify the username and password.
​

Usually these few variables need to be set by hand: each machine’s ip, home_path, data_dir, and redo_dir. In this example, they are changed to /home/admin/oceanbase/ob, /data/ob, and /redo/ob respectively, corresponding to the disks mounted earlier.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
oceanbase-ce:
  servers:
    - name: z1
      # Please don't use hostname, only IP can be supported
      ip: 172.30.62.200
    - name: z2
      ip: 172.30.62.201
    - name: z3
      ip: 172.30.62.202
  global:
    # The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field.
    home_path: /home/admin/oceanbase/ob
    # The directory for data storage. The default value is $home_path/store.
    data_dir: /data/ob
    # The directory for clog, ilog, and slog. The default value is the same as the data_dir value.
    redo_dir: /redo/ob

Configure the proxy, modifying the ip and home_path.

1
2
3
4
5
6
obproxy:
  servers:
    - 172.30.62.203
  global:
    # The working directory for obproxy. Obproxy is started under this directory. This is a required field.
    home_path: /home/admin/oceanbase

Additionally, here’s a recommended site https://www.bejson.com/validators/yaml_editor/ that can run YAML validation on the configuration file. Quite often a configuration file has one extra space or one space too few, which is extremely hard to spot.
​

​

Start the Installation

For offline installation, there’s one extra step. When you can’t reach the server, you need to delete the remote repo configuration to avoid wasting time on connecting to the remote repo.

1
rm -fr ~/.obd/mirror/remote/*.repo

Start the installation

1
admin@obdriver:~$ obd cluster autodeploy obtest -c distributed-with-obproxy-example.yaml

Check whether the installation succeeded

1
2
3
4
5
6
7
8
admin@obdriver:~$ obd cluster list
+------------------------------------------------------------+
|                        Cluster List                        |
+--------+---------------------------------+-----------------+
| Name   | Configuration Path              | Status (Cached) |
+--------+---------------------------------+-----------------+
| obtest | /home/admin/.obd/cluster/obtest | running         |
+--------+---------------------------------+-----------------+
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
admin@obdriver:~$ obd cluster display obtest
Get local repositories and plugins ok
Open ssh connection ok
Cluster status check ok
Connect to observer ok
Wait for observer init ok
+-------------------------------------------------+
|                     observer                    |
+---------------+---------+------+-------+--------+
| ip            | version | port | zone  | status |
+---------------+---------+------+-------+--------+
| 172.30.62.200 | 3.1.0   | 2881 | zone1 | active |
| 172.30.62.201 | 3.1.0   | 2881 | zone2 | active |
| 172.30.62.202 | 3.1.0   | 2881 | zone3 | active |
+---------------+---------+------+-------+--------+

Connect to obproxy ok
+-------------------------------------------------+
|                     obproxy                     |
+---------------+------+-----------------+--------+
| ip            | port | prometheus_port | status |
+---------------+------+-----------------+--------+
| 172.30.62.203 | 2883 | 2884            | active |
+---------------+------+-----------------+--------+

Modify the Configuration

The OceanBase database has hundreds of configuration items, and some of them are coupled. Before you’re familiar with the OceanBase database, it’s not recommended to modify the settings in the example configuration file. This example is meant to illustrate how to modify a configuration and make it take effect.
​

For an introduction to all parameters, please refer to

1
https://github.com/oceanbase/obdeploy/blob/master/plugins/oceanbase/3.1.0/parameter.yaml
1
2
3
4
5
6
7
# Use the edit-config command to enter edit mode and modify the cluster configuration
obd cluster edit-config lo
# Change sys_bkgd_migration_retry_num to 5
# Note that the minimum value of sys_bkgd_migration_retry_num is 3
# After saving and exiting, OBD will tell you how to make this change take effect
# This configuration item only needs a reload to take effect
obd cluster reload lo

Verification

Install obclient

obclient is usually installed on the control machine; you need to switch to the root account.

Online Installation

1
2
yum install -y libobclient
yum install -y obclient

Local Installation

​

CentOS or RedHat or Anolis

1
2
yum install libobclient-2.0.0-2.el7.x86_64.rpm
yum install obclient-2.0.0-2.el7.x86_64.rpm

Ubuntu/Debian

1
2
alien -i libobclient-2.0.0-2.el7.x86_64.rpm
alien -i obclient-2.0.0-2.el7.x86_64.rpm

On Debian, you need to add the path to the system environment variable

1
export PATH=/app/mariadb/bin:$PATH

​

On Ubuntu, you need to add the path to the system environment variable

1
export PATH=/u01/obclient/bin:$PATH

​

​

Install the MySQL Development Packages

If you need to run programs like sysbench or tpch, you need to install the MySQL development packages.

​

CentOS or RedHat or Anolis

1
2
3
yum install mariadb
yum install mariadb-libs
yum install mariadb-devel

Ubuntu

1
apt-get install mariadb-server

Debian

1
apt-get install mysql-server mysql-client libmariadbd18 libmariadbd-dev

Check Tenants

To use OceanBase, you need to create a tenant; real user applications must run under a tenant.
There are 2 ways to create a tenant:
You can use obd to create a tenant. When you create a tenant with obd, it allocates all the resources.

1
obd cluster tenant create ${cluster_name} -n ${tenant_name}

​

To create a tenant, please refer to
https://open.oceanbase.com/docs/community/oceanbase-database/V3.1.0/create-a-user-tenant
​

In this example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
admin@obdriver:~$ mysql -h${obproxy_ip} -P${obproxy_port} -uroot
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.6.25 OceanBase 3.1.0 (r3-b20901e8c84d3ea774beeaca963c67d7802e4b4e) (Built Aug 10 2021 07:51:04)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]> use oceanbase;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MySQL [oceanbase]> select * from gv$tenant;
+-----------+-------------+-------------------+-------------------+----------------+---------------+-----------+---------------------------------------------+
| tenant_id | tenant_name | zone_list         | primary_zone      | collation_type | info          | read_only | locality                                    |
+-----------+-------------+-------------------+-------------------+----------------+---------------+-----------+---------------------------------------------+
|         1 | sys         | zone1;zone2;zone3 | zone1;zone2,zone3 |              0 | system tenant |         0 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
|      1001 | mytest      | zone1;zone2;zone3 | RANDOM            |              0 |               |         0 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
+-----------+-------------+-------------------+-------------------+----------------+---------------+-----------+---------------------------------------------+
2 rows in set (0.00 sec)

MySQL [(none)]>

​

Best Practices for Enterprise Users

Disks

The OB runtime logs, transaction logs, and data files must be kept separate. If there aren’t 3 disks available, you can partition one disk into 3 partitions.
​

Clock Dependency

The time on each server in an OceanBase cluster must stay consistent; otherwise, the OceanBase cluster will fail to start and faults will occur at runtime. For enterprise users, clock synchronization is extremely important. A deviation of under 50ms between the physical machine and the time server can be considered in sync, and the maximum tolerated deviation cannot exceed 200ms. When it exceeds 200ms, a no-leader situation occurs; after restoring clock synchronization, restarting the observer can recover the state.
​

Network Latency

The network latency between servers cannot exceed 200ms; otherwise, synchronization will lag severely and elections may be affected.
NIC settings:
It’s recommended to configure two 10-gigabit NICs in bond mode named bond0, using either mode1 or mode4. mode4 is recommended; if using mode4, the switch needs to be configured with 802.3ad. For NIC names, it’s recommended to use eth0 and eth1. It’s recommended to use the network service rather than NetworkManager.

Parameter Settings

  • When the system write TPS is too high and exceeds the system’s capacity, to prevent the system from becoming unresponsive
1
alter system set writing_throttling_trigger_percentage=75 tenant=all(or a specific tenant name);
  • Unless the business application is configured with reconnection retries, it’s recommended to disable rotating merge, since switching leaders cannot guarantee transactions won’t be killed
1
ALTER SYSTEM SET enable_merge_by_turn = 'False';
  • Memory settings
    • The tenant’s CPU-to-memory ratio is recommended to be no lower than 1:4; otherwise OOM is likely;
    • The minimum memory spec for a regular tenant is tentatively set to 5G or above;
    • When a tenant’s memory is too small, it’s recommended to increase ob_sql_work_area_percentage (default 5%); for tenants with less than 10G of memory, configuring around 20% is recommended;
    • Partition count limit: it’s recommended not to exceed 100,000 partitions per machine. In addition, the partition count is limited by memory; each replica reserves 168KB of memory, so 10,000 replicas require at least 1.68G of reserved memory. In other words, a 1G tenant can create at most around 6k partitions, so you need to set the tenant memory based on the planned number of partitions; also per-machine.
    • Physical memory usage limit, default 80, the percentage of memstore memory available. It’s recommended to set it to 90 for servers with 256G of memory or more, and keep the default 50 for less than 256G.
1
ALTER SYSTEM SET memstore_limit_percentage = '90';

​

  • Slow query threshold adjustment: trace_log_slow_query_watermark defaults to 100ms and can be adjusted based on business characteristics. If the threshold is set too small, printing a large number of trace logs will affect performance. The MySQL default is 1s. The large query time is 10s.
1
2
3

ALTER SYSTEM SET trace_log_slow_query_watermark = '1s';
ALTER SYSTEM SET large_query_threshold = '10s';

​

  • CPU concurrency adjustment
1
2
3
4
5
-- CPU concurrency parameter, recommended to set to 4, and 2 for arm systems
ALTER SYSTEM SET cpu_quota_concurrency = '4';

-- Resource soft-load switch, controls the resource balancing watermark, default 50%, meaning unit balancing happens when CPU/memory usage exceeds 50%. For production, it's recommended to set it to 100, achieving the effect of manually controlling unit distribution
ALTER SYSTEM SET resource_soft_limit = '100';
  • Minor freeze / merge related
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
-- Configure 50 minor freezes
ALTER SYSTEM SET minor_freeze_times = 50;

-- Minor-freeze trigger watermark percentage. Recommended to set to 70 for 256G or more, and 60 for less than 256G
ALTER SYSTEM SET freeze_trigger_percentage = '60';

-- Data copy concurrency of 100
ALTER SYSTEM SET data_copy_concurrency = 100;
-- Data copy-out concurrency on the server of 10
ALTER SYSTEM SET server_data_copy_out_concurrency = 10;
-- Data copy-in concurrency on the server of 10
ALTER SYSTEM SET server_data_copy_in_concurrency = 10;

-- Minor-freeze warm-up time, default 30s. Setting it delays the time the minor freeze is released; change it to 0s
ALTER SYSTEM SET minor_warm_up_duration_time = '0s';
-- Configure the chunk memory size (recommended to keep the default value 0 and let ob allocate it itself)
ALTER SYSTEM SET memory_chunk_cache_size = 0;


-- Maximum number of kept versions, affects available disk space, default 2. To keep one more version of data on the data disk, change it to 1
ALTER SYSTEM SET max_kept_major_version_number = '1';
ALTER SYSTEM SET max_stale_time_for_weak_consistency = '2h';

​

  • Transaction related
1
2
3
4
5
ALTER SYSTEM SET clog_sync_time_warn_threshold = '1s';
ALTER SYSTEM SET trx_try_wait_lock_timeout = '0ms';(the default is already 0ms, no need to modify)

-- It's recommended to disable one-phase commit; this parameter's default value is false
ALTER SYSTEM SET enable_one_phase_commit='False';
  • Partition migration speed control. If the cluster load is very low, you can speed up partition migration by increasing the number of concurrent tasks; increase the migration concurrency
1
2
3
alter system set data_copy_concurrency=40;
alter system set server_data_copy_out_concurrency=20;
alter system set server_data_copy_in_concurrency=20;
  • Compression related
1
2
-- (the default is already zstd_1.0, no need to modify), but the system supports multiple compression algorithms
ALTER SYSTEM SET default_compress_func = 'zstd_1.0';

​

  • Cache refresh related
1
ALTER SYSTEM SET autoinc_cache_refresh_interval = '43200s';
  • Prepared statement: server-side ps is controlled by the _ob_enable_prepared_statement switch. Except for objdbc and oci users who can use server-side ps following the instructions in the documentation, it’s not recommended in other cases;
1
2
-- Prepared Statement parameter, recommended to set to 0 for setups not building connections with Java
ALTER SYSTEM SET _ob_enable_prepared_statement = 0;
  • System related
1
2
3
4
5
ALTER SYSTEM SET server_permanent_offline_time = '7200s';


-- (5M recommended for public cloud, 5M recommended for external environments, otherwise the default 30M is recommended)
ALTER SYSTEM SET syslog_io_bandwidth_limit = '5M';
  • Cluster upgrade strategy: when performing a version upgrade or temporarily taking machines online/offline, you can first perform a minor freeze, which can reduce the recovery time when starting the observer
  • Best strategy for bulk-importing large amounts of data: if the cluster is multi-tenant and a tenant needs to bulk-import data, to avoid affecting other tenants, you can restore the following two parameters after the import is done:
      1. Adjust cpu_quota_concurrency = 1 to prevent CPU contention between tenants
    • Enable multi-round minor freezes to reduce merge triggers, which can improve import speed

​

  • Tenant primary_zone configuration
    • Set primary_zone to a specific zone. Applicable scenario: the business uses a single table, zone_name1 is in the same data center as the application, and zone_name2 and zone_name3 serve as follower replicas with no business traffic normally. The specific zone order should follow the data center priority and be configured according to the application’s and ob’s data center setup.
1
ALTER TENANT SET PRIMARY_ZONE = 'zone_name1;zone_name2,zone_name3';
  • Spread primary_zone across all full-featured zones. Usage scenario: the business uses partitioned tables, all replicas in the cluster are in the same data center, or the network latency between zones’ data centers is within 1ms, and all replicas are needed
1
ALTER TENANT SET PRIMARY_ZONE = 'zone_name1,zone_name2,zone_name3';

Tenant Settings

  • Concurrency settings
1
2
3
4
5
6
7
8
9
10
11
12
13
-- Maximum concurrency, default 32. For businesses with large queries, it's recommended to set it to 128
SET GLOBAL ob_max_parallel_degree = 128;


/*
parallel_max_servers is recommended to be set to 10 times the resource unit CPU count allocated to the test tenant
For example, if the unit configuration used by the test tenant is: create resource unit $unit_name max_cpu 26
Then set this value to 260
parallel_server_target is recommended to be set to parallel_max_servers * number of machines * 0.8
So the value would be 260*3*0.8=624
*/
set global parallel_max_servers=260;
set global parallel_servers_target=624;
  • Recycle bin settings
1
2
3
4
5
-- Recycle bin parameter. In scenarios with very frequent DDL execution, this must be disabled to avoid abnormal tenant performance caused by too much DDL execution
SET GLOBAL recyclebin = 0;

-- Truncate rollback parameter. In scenarios with very frequent truncate execution, this must be disabled
SET GLOBAL ob_enable_truncate_flashback = 0;
  • Client command length: the length of commands the OB client can send is limited by the tenant system variable max_allowed_packet (default 4M); you can increase it as appropriate;

obproxy Configuration

  • obproxy liveness probing
1
2
3
4
5
alter proxyconfig set sock_option_flag_out = 2;  --  2 stands for keepalive
alter proxyconfig set server_tcp_keepidle = 5;  --  idle time before starting keepalive probing, 5 seconds.
alter proxyconfig set server_tcp_keepintvl = 5;  -- interval between two keepalive probe packets, 5 seconds
alter proxyconfig set server_tcp_keepcnt = 2;  --  maximum number of keepalive packets to send, 2. At most 5+5*2=15 seconds to detect a dead_socket.
alter proxyconfig set server_tcp_user_timeout = 5;  --  timeout for waiting for the TCP layer's ACK confirmation message, 5 seconds.