Integrating OceanBase Monitoring with Prometheus/Grafana

Summary

This article introduces how to integrate OceanBase monitoring with Prometheus and Grafana.

​

​

Installation Flow

The general process breaks down into three main steps:

1. Install OceanBase and OBAgent
2. Install Prometheus and Grafana
3. Configure Prometheus and Grafana

Install OceanBase and OBAgent

For how to install OceanBase, refer to the previous article OceanBase Offline Installation. This section focuses on how to install OBAgent; you can also refer to the document Use OBD to Deploy OBAgent.

OBAgent is a monitoring data collection framework. OBAgent supports both push and pull data collection modes, which can satisfy different application scenarios. The plugins OBAgent supports by default include host data collection, collection of OceanBase database metrics, monitoring data label processing, and an HTTP service for the Prometheus protocol. To make OBAgent support collection from other data sources or customize the data processing flow, you only need to develop the corresponding plugin.

For the OBAgent configuration, the OBAgent settings are added on top of the original configuration. For details, refer to distributed-with-obproxy-and-obagent-example:

obagent:
  depends:
    - oceanbase-ce
  # The list of servers to be monitored. This list is consistent with the servers in oceanbase-ce. 
  servers:
    - name: server1
      # Please don't use hostname, only IP is supported.
      ip: 172.19.33.2
    - name: server2
      ip: 172.19.33.3
    - name: server3
      ip: 172.19.33.4
  # Set dependent components for the component.
  # When the associated configurations are not done, OBD will automatically get the these configurations from the dependent components.
  depends:
    - oceanbase-ce
  global:
    # The working directory for obagent. obagent is started under this directory. This is a required field.
    home_path: /root/observer
    skip_proxy_sys_private_check: true

A few special notes:

1. The name “oceanbase-ce” under depends must match the cluster name in the configuration file.
2. The configuration under servers must be exactly the same as the servers configuration in the “oceanbase-ce” section of the configuration file.
3. Remember home_path; you’ll need this path later.

After the installation is complete, you can run “obd cluster display” and see that OBAgent has already started.

obd cluster display obtest
Get local repositories and plugins ok
Open ssh connection ok
Cluster status check ok
Connect to observer ok
Wait for observer init ok
+-------------------------------------------------+
|                     observer                    |
+---------------+---------+------+-------+--------+
| ip            | version | port | zone  | status |
+---------------+---------+------+-------+--------+
| 172.30.62.210 | 3.1.1   | 2881 | zone1 | active |
| 172.30.62.211 | 3.1.1   | 2881 | zone2 | active |
| 172.30.62.212 | 3.1.1   | 2881 | zone3 | active |
+---------------+---------+------+-------+--------+

Connect to obproxy ok
+-------------------------------------------------+
|                     obproxy                     |
+---------------+------+-----------------+--------+
| ip            | port | prometheus_port | status |
+---------------+------+-----------------+--------+
| 172.30.62.213 | 2883 | 2884            | active |
+---------------+------+-----------------+--------+
+---------------------------------------------------+
|                      obagent                      |
+---------------+-------------+------------+--------+
| ip            | server_port | pprof_port | status |
+---------------+-------------+------------+--------+
| 172.30.62.210 | 8088        | 8089       | active |
| 172.30.62.211 | 8088        | 8089       | active |
| 172.30.62.212 | 8088        | 8089       | active |
+---------------+-------------+------------+--------+

Install Prometheus and Grafana

Pick a machine to install Prometheus and Grafana. Try not to use one of the OBServer machines. In this example, Prometheus and Grafana are deployed on the OBProxy machine.

1. Download Prometheus and Alertmanager from https://prometheus.io/download/. This chapter will not cover how to use Alertmanager.
2. Download Grafana from https://grafana.com/grafana/download?pg=get&plcmt=selfmanaged-box1-cta1
3. Copy the Prometheus and Grafana archives to the OBProxy machine.
4. Extract Prometheus and Grafana.

# tar -xzf prometheus-2.31.0.linux-amd64.tar.gz
# tar -xzf grafana-enterprise-8.2.3.linux-amd64.tar.gz

Configure Prometheus and Grafana

Configure Prometheus

1. Copy the Prometheus configuration file from the OBAgent machine into the Prometheus installation directory.

# cd prometheus-2.31.0.linux-amd64
# mv prometheus.yml prometheus.yml.old
# scp -r observer001:/root/observer/conf/prometheus_config/* . 

Notes:

1. observer001 is one of the machines where OBAgent is installed.
2. /root/observer is the home_path configured earlier for OBAgent in the configuration file.
3. A few files are copied over from observer001: prometheus.yaml and rules. rules stores the scrape rules, and prometheus is the file that configures Prometheus.

Start Prometheus:

nohup ./prometheus --config.file=./prometheus.yaml >> run.log 2>&1 &

Check run.log to view the runtime logs. Under normal conditions,

# curl http://localhost:9090/metrics

returns a large amount of data.

Configure Grafana

# cd grafana-8.2.3/
# nohup bin/grafana-server > run.log 2>&1 &
# ps -ef|grep grafana

You can use either run.log or ps -ef|grep grafana to confirm that Grafana is working properly.

Open the Grafana page. On first login, enter admin/admin, then set the administrator password, and then add a data source.

​

After going into Add data source, select Prometheus, then configure Prometheus, with the key setting being the URL.

​

Import the configuration items.

​

OceanBase has prepared 15215 and 15216 in advance: one for monitoring OceanBase, and one for monitoring the host.
After loading the templates, you’ll see the two preset dashboards in the dashboard list.

​

Congratulations, you have finished integrating OceanBase monitoring with Prometheus and Grafana.