OceanBase Developer Handbook, Part 5: How to Debug OceanBase

Abstract

The OceanBase Developer Handbook mainly guides developers on how to participate in OceanBase development, clearing obstacles you may encounter while preparing to contribute. This section covers the following articles, with more to be added in the future. For now, the OceanBase source code references the Open-Source Database OceanBase Source Code Walkthrough series on the OceanBase open-source official site:

1. How to compile the OceanBase source code
2. How to set up an IDE development environment
3. How to become an OceanBase Contributor
4. How to edit the OceanBase documentation
5. How to debug OceanBase
6. How to run tests
7. How to fix bugs

This article introduces how to debug OceanBase. We recommend several approaches:

1. Remotely debug OceanBase with VS Code
2. Debug OceanBase locally with gdb
3. Debug OceanBase locally with CLion in a Linux environment

Steps

Preparation

One important step in debugging OceanBase is obtaining OceanBase’s startup parameters. Each machine has its own hardware configuration, which leads to different startup parameters. The approach, however, is basically the same.

1. Install and deploy an environment using OBD (https://github.com/oceanbase/obdeploy).

1. For a standalone deployment in a networked environment, refer to the document https://open.oceanbase.com/quickStart
2. For a distributed environment or an offline deployment, refer to the document https://open.oceanbase.com/docs/community/oceanbase-database/V3.1.1/deploy-the-distributed-oceanbase-cluster

2. After successfully deploying the environment, compile the debug build of OceanBase. Refer to the earlier document How to Compile the OceanBase Source Code.
3. Capture OceanBase’s startup parameters (via ‘ps -ef|grep observer’).
4. (Optional) In a distributed environment, replace the observer installed and deployed by OBD with your compiled observer binary.

In my standalone test environment, after installing and deploying OceanBase with OBD, my OceanBase startup parameters are as follows:

observer -r xxx.xxx.xxx.xxx:2882:2881 -o __min_full_resource_pool_memory=268435456,enable_syslog_recycle=True,enable_syslog_wf=True,max_syslog_file_count=4,memory_limit=69G,system_memory=27G,cpu_count=19,datafile_size=1029G,clog_disk_utilization_threshold=95,clog_disk_usage_limit_percentage=98 -z zone1 -p 2881 -P 2882 -n obcluster -c 1 -d /home/xxxxxxx/observer/store -i em1 -l INFO

Debugging With VS Code

1. Set up the remote connection environment.

1.1 Establish trusted login from the development machine to the test machine. Refer to the document https://open.oceanbase.com/docs/community/oceanbase-database/V3.1.1/optional-set-password-free-ssh-logon
1.2 Set up the VS Code remote debug environment with "Remote-SSH: Connect to Host...". Refer to the article https://blog.csdn.net/zbbzb/article/details/102957076/ to configure it.

2. Connect to the remote machine over remote SSH.

Ctrl + p
Select Remote-SSH: Connect to Host

3. Open the corresponding source code directory.

4. Refer to the earlier article How to Compile the OceanBase Source Code.

5. Set the debug startup parameters via the menu “Run” –> “Add Configuration”. If you’ve set them up before, modify the startup parameters via the menu “Run” –> “Open Configurations”.

My configuration is as follows:

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
    
        {
            "name": "observer",
            "type": "cppdbg",
            "request": "launch",
            "program": "${OB_SRC_DIR}/build_debug/src/observer/observer",
            "args": ["-r", "${IP}:2882:2881", "-o", "__min_full_resource_pool_memory=268435456,enable_syslog_recycle=True,enable_syslog_wf=True,max_syslog_file_count=4,memory_limit=69G,system_memory=27G,cpu_count=19,datafile_size=1029G,clog_disk_utilization_threshold=95,clog_disk_usage_limit_percentage=98", "-z", "zone1", "-p", "2881", "-P", "2882", "-n", "obcluster", "-c", 1, "-d", "/home/XXX/observer/store", "-i", "em1", "-l", "INFO"],
            "stopAtEntry": true,
            "cwd": "${OB_SRC_DIR}",
            "environment": [],
            "externalConsole": false,
            "MIMode": "gdb",
            "setupCommands": [
                {
                    "description": "Enable pretty-printing for gdb",
                    "text": "-enable-pretty-printing",
                    "ignoreFailures": true
                }
            ]
        }
    ]
}

Note: the arg parameters here come from the startup parameters obtained during the preparation in step one. Each machine has its own configuration. My parameters are as follows, where:

1. ${OB_SRC_DIR} is the source code directory, and ${IP} is the observer's bound IP.
2. You need to set "cwd" to ${OB_SRC_DIR}.
3. It's recommended to set "stopAtEntry" to true.
4. In the args parameters, the directory set by -d, "/home/xxxxx/observer/store", needs to be set to the real value.
5. In the args parameters, the device name set by -i, "em1", is the device name corresponding to the IP.

6. Start debugging by clicking the menu “Run” –> “Start Debugging”.

Debugging Locally With gdb Directly

1. Log in to the remote machine and enter the ${OB_SRC_DIR} source code directory.

2. Refer to the earlier article How to Compile the OceanBase Source Code.

3. Edit .gdbinit in your home directory and add the following line, replacing ${OB_SRC_DIR} with the root directory of the OB source code:

add-auto-load-safe-path ${OB_SRC_DIR}/.gdbinit

4. vi ${OB_SRC_DIR}/.gdbinit

file build_debug/src/observer/observer
set args "-r", "XXX.XXX.XXX.XXX:2882:2881", "-o", "__min_full_resource_pool_memory=268435456,enable_syslog_recycle=True,enable_syslog_wf=True,max_syslog_file_count=4,memory_limit=69G,system_memory=27G,cpu_count=19,datafile_size=1029G,clog_disk_utilization_threshold=95,clog_disk_usage_limit_percentage=98", "-z", "zone1", "-p", "2881", "-P", "2882", "-n", "obcluster", "-c", 1, "-d", "/home/longda/observer/store", "-i", "em1", "-l", "INFO"
b main
r

Note: the args parameters here come from the startup parameters obtained during the preparation in step one. Each machine has its own configuration. My parameters are as follows, where:

1. ${OB_SRC_DIR} is the source code directory, and ${IP} is the observer's bound IP.
2. You need to set "cwd" to ${OB_SRC_DIR}.
3. The current working directory must be ${OB_SRC_DIR}.
4. In the args parameters, the directory set by -d, "/home/xxxxx/observer/store", needs to be set to the real value.
5. In the args parameters, the device name set by -i, "em1", is the device name corresponding to the IP.

5. We recommend using TUI. TUI is gdb’s built-in graphical interface, which is fairly intuitive. Here’s a quick note on how to switch to it and the common commands.

1. gdb -tui + (executable program) enters the tui graphical interface directly.

2. After entering gdb, use the command focus to enter the tui graphical interface, or use the shortcut Ctrl+x+a (note the key order; mnemonic: x = focus, a = another).

3. In tui, use the same shortcut Ctrl+x+a to return to gdb's native interface.

4. In gdb, ↑ and ↓ switch between the previous and next command, but in tui they only control the code view. To switch commands, use Ctrl+n (mnemonic: next) and Ctrl+p (mnemonic: previous), which are actually gdb's native shortcuts.

6. In the source code directory, type gdb to start the gdb debug session.

gdb 

Debugging Locally With CLion

CLion makes reading source code very convenient: symbol navigation works smoothly, and it natively supports code formatting via clang-format. However, I haven’t tried CLion remote debugging—only local CLion debugging. That said, if you want to debug OceanBase locally with CLion, your development machine must run Linux.

CLion is the most comfortable way to debug, but also the most complex, with very demanding requirements.

1. Refer to the earlier article How to Compile the OceanBase Source Code.
2. Configure CLion’s CMake.

Refer to the image for the detailed steps. Note that:

"Build Directory" needs to be set to "build_debug".
"CMake options" needs to be set to "${OB_SRC_DIR} -DCMAKE_BUILD_TYPE=Debug", where ${OB_SRC_DIR} needs to be changed to the real full directory path.

3. Wait a few minutes for CMake generation to finish, then click the menu “Run” –> “Edit Configurations”. You can also, as in the image below, select the build target observer.

4. Click the menu “Build” –> “Build observer” to compile observer.

5. Modify the startup parameters by clicking the menu “Run” –> “Edit Configurations”. After the dialog appears:

On my machine, “Program Arguments” is:

-r ${ip}:2882:2881 -o __min_full_resource_pool_memory=268435456,enable_syslog_recycle=True,enable_syslog_wf=True,    max_syslog_file_count=4,memory_limit=8G,system_memory=4G,cpu_count=16,datafile_size=44G,clog_disk_utilization_threshold=95,clog_disk_usage_limit_percentage=98 -z zone1 -p 2881 -P 2882 -n obcluster -c 1 -d ${data_dir} -i ${devname} -l INFO

${ip}: the local machine’s IP.

${data_dir}: the data directory.

${devname}: the network interface name corresponding to the IP, usually eth0 or lo.

“Working Directory” must be ${OB_SRC_DIR}.

6. Open the file src/observer/main.cpp and set a breakpoint at the main function.

7. Start debugging by clicking the menu “Run” –> “Debug Observer”.