LLM vs. Small Models: The Right Way to Build an AI Agent for Domestic Database Operations

Author: Sun Peng, R&D Engineer at Dayan (Beijing) Technology Co., Ltd.

A Bold Claim: General-Purpose, Full-Scale LLMs Are “Not Suitable” for Powering Intelligent Diagnosis and Operations of Domestic Databases

Traditional database operations have long faced three core challenges:

• Explosive growth in data volume: As the data volume of modern applications grows rapidly, the number of database instances increases accordingly, and monitoring metrics become more complex. Faced with a massive number of database instances, manual operations and diagnosis become increasingly strained.
• Over-reliance on experience: The diversity of database types makes it hard to quickly spread and pass on expert experience, and pinpointing a fault takes tens of minutes on average.
• Limitations of traditional techniques: Static optimizers based on rules (RBO) or cost (CBO) struggle to adapt to complex, changing query scenarios, and their ability to handle unstructured data is relatively weak.

With the application of AI LLMs, these problems now have entirely new solutions. Through their dynamic optimization capabilities, AI LLMs break through the limitations of traditional static optimizers and can generate efficient execution plans in real time. Their natural-language interaction greatly lowers the technical barrier for complex queries, and their multimodal analysis capabilities allow heterogeneous data such as logs and performance metrics to be processed in a unified way. These advances have driven a qualitative leap in database operations—from passive response to proactive defense, and from relying on experience to relying on intelligent decision-making.

However, in the process of using LLMs to power database operations, two problems emerged:

1. General-purpose LLMs have insufficient knowledge of domestic databases and are hard to use in production.

When using LLMs to diagnose problems on traditional databases (such as Oracle and MySQL), the results are usually quite good. However, when the same techniques are applied to domestic databases, the diagnostic results often fall short. This is mainly because current general-purpose LLMs were exposed to relatively little knowledge about domestic databases during training.

In addition, out of concern for enterprise data security and compliance, operational data in production environments cannot be uploaded to external networks, which means it cannot serve as a real-time reference for a “full-scale” LLM.

2. The observability and accuracy of database operational data are insufficient.

Even when using a fully capable “full-scale” LLM, the probability of hallucination remains high. For example, when analyzing a database load issue, if only coarse-grained information such as the total data volume is provided, the LLM often struggles to accurately reconstruct the load trend within a specific time window.

Exploring the Feasibility of Replacing Full-Scale LLMs with Small Models

When putting AI capabilities into production, many enterprises face a practical challenge: limited by the difficulty of obtaining high-end GPUs or the high cost of compute, they cannot deploy a fully capable “full-scale” LLM on their intranet. For reasons of cost and deployment constraints, enterprises often can only choose a more economical, privately deployed small-model approach.

So, can we deploy a lower-cost small model in a resource-constrained production environment while giving it capabilities close to those of an LLM?

This idea faces two key challenges:

• First, how to make a small model reproduce the powerful reasoning and generalization abilities of an LLM;
• Second, how to make up for the LLM’s insufficient coverage of domestic database knowledge.

Going back to the landmark 2020 paper Language Models are Few-Shot Learners, it proposed an “in-context learning” mechanism, showing that an LLM can achieve online learning by feeding in contextual information, without updating its parameters.

Inspired by this, when using a small model to analyze a specific problem, we tried feeding the model the relevant background knowledge, metric data, real-time runtime state, and a carefully designed prompt all together, using the context to enhance its reasoning ability. After multiple rounds of testing and validation, the small model’s answer accuracy is now close to that of a “full-scale” LLM.

(1) High-Quality Data Is the Core Foundation for Putting Small Models into Production

Applying a small model to intelligent database operations typically requires meeting three key prerequisites:

First, strong system observability. This includes providing rich and accurate runtime metrics and statistics, such as system performance metrics, wait-event analysis, complete logging and TRACE data, as well as macro-level AWR reports and micro-level ASH information.

Second, an accumulated, high-quality body of operational knowledge. This body of knowledge spans two dimensions:

• Operational theory mainly comes from structured knowledge resources such as authoritative original-vendor documentation and third-party technical books;
• Operational practice covers unstructured or semi-structured experiential assets such as expert experience summaries and a library of user fault cases.

Compared with theoretical knowledge, an LLM more easily understands and absorbs operational experience drawn from real-world scenarios. Therefore, during the training and inference of small models, injecting experiential data from real business scenarios is especially important.

Third, reliance on powerful reasoning-model capabilities. When high-quality data is combined with strong reasoning ability, even in a resource-constrained, privately deployed environment, a small model can achieve accuracy and stability close to those of an LLM.

(2) Intelligent Metric Processing to Build Strong Observability for the Database

Intelligent metric processing can build strong observability for the database. First, data is collected from various IT operations targets such as databases and middleware—for example, runtime data and log data—along with processed data obtained from the data middle platform, yielding a metric set. Next, secondary processing is performed on the raw values to derive statistical values such as the incremental difference over a time window, the average, and the per-occurrence average. Then, further processing on top of these statistical values yields related values such as the mean, stability, trend assessment, and risk assessment.

At the same time, a knowledge graph must be built. Because building an operational knowledge graph is the foundation of digitalization capability, an initial operational knowledge graph is formed through knowledge organization, and the knowledge graph is continually refined and enriched based on real application cases, so its analytical ability keeps improving.

(3) Architecture Design for Intelligent Database Operations and Diagnosis

The intelligent database diagnosis and analysis flow mainly includes the following key steps:

1. Data collection and processing — Collect and process key performance metrics from the OceanBase database to build comprehensive, fine-grained observability
2. Data storage — Store the collected and processed metric data uniformly in a data warehouse
3. Fault-model triggering and analysis — When a particular fault model is triggered, the system automatically launches the anomaly-detection AI Agent:
   • It extracts background knowledge related to the fault model from the knowledge graph
   • It simultaneously retrieves the database’s current real-time runtime metrics from the data warehouse
   • It combines this with a preset prompt template and feeds the consolidated information into the LLM
   • The LLM then retrieves relevant cases, expert experience, and scenario-specific information from the vector database, and finally outputs a clear, accurate diagnostic analysis report

So far, we have successfully built and deployed three types of AI Agents:

First, the alert-analysis Agent:

• When the database generates an alert, the Agent can automatically call the LLM for in-depth analysis
• After the analysis, it generates a structured diagnostic report and pushes it to the relevant personnel via email and other channels

Second, the SQL-optimization Agent:

• It can automatically identify Top SQL in the database (such as frequently executed or resource-intensive statements)
• It uses the small model to intelligently optimize these SQL statements, generating optimization suggestions and a diagnostic report

Third, the inspection Agent:

• Users can set a specific time window as needed, and the inspection Agent performs a comprehensive check of the database’s health during that window
• The Agent generates a detailed health report based on the inspection results

The two scenarios below illustrate how these Agents work and how effective they are:

1. A small model analyzing a lock-conflict scenario.

First, OceanBase’s blocking and blocked data for the lock conflict must be sent to the small model along with the background knowledge. The small model can then use the blocking data and background knowledge to find each root blocker, suggest terminating the relevant sessions to resolve the blocking, and at the same time provide root-cause analysis and diagnostic recommendations.

2. A small model performing SQL optimization.

When the small model optimizes SQL performance, it first deeply analyzes the SQL execution plan, identifies the most significant performance bottleneck, and gives the rationale based on the specific execution path.

For example, if the execution plan shows a partitioned table used as the driving table with a Nested Loop join, and that join contains two levels of looping, this means every record in the partitioned table triggers a full scan or lookup of the driven table, causing the number of data accesses to grow exponentially. Optimization suggestions include changing the Nested Loop to a Hash Join or Merge Join; considering building an index on the relevant columns of the driven table; or adjusting the partitioning strategy to reduce the scan range.

Practice Shows: AI Agents’ Diagnostic Reasoning Is Already Combat-Ready

To validate the diagnostic reasoning ability of AI Agents, we tested four typical fault models in a lab environment, including the Oracle database’s log-sync latency anomaly and hot-block contention problems, as well as the OceanBase database’s excessive blocking sessions and excessive active sessions problems.

The test results are encouraging: whether using a full-scale LLM or a privately deployed small model, the analysis accuracy reached over 90%, surpassing the accuracy that current in-house professional operations staff can achieve even with expert tools.

In the lab environment, regardless of whether a full-scale LLM or a privately deployed small model was used for analysis—including analyzing SQL, optimizing SQL, and analyzing alerts—the time taken was generally within 3 to 5 minutes (the Tongyi Qianwen Qwen 3 series of models took about two minutes to analyze). By contrast, traditional expert analysis—collecting metric data and analyzing root causes along the diagnostic path—takes around 3 hours, and may even take half a day or a full day. In comparison, AI-based intelligent diagnosis is far more efficient than operations experts, delivering a more than 50x improvement.

This article is based on the OceanBase “Data✖️AI” Hackathon. Check out more outstanding work here: https://open.oceanbase.com/ai-hackathon