AI Application Development: An Innovative Unstructured Document Analysis System That Avoids the Limitations of Traditional Approaches

The Challenges of Storing and Analyzing Unstructured Data in the Big Data Era

In the big data era, the demand for storing unstructured data keeps growing. According to an IDC report, unstructured data already accounts for more than 92.9% of all existing data, and new data storage paradigms have emerged to meet this need.

Storing and analyzing unstructured documents requires the attribute extraction capabilities of large models. In real-world business scenarios, enterprises accumulate vast amounts of unstructured data in many different formats and with complex content—product development documents, logs, reports, contracts, medical records, and more. Such data is difficult to store and retrieve directly. Using large models to perform attribute extraction—structuring the key information so it can be stored and retrieved efficiently—has become an urgent need for today’s enterprise data management and intelligent operations. For example, the typical scenarios mentioned above—operations logs, product development documents, résumés, medical records, financial statements—are all unstructured documents that urgently require structured extraction and analysis.

Today, the ability to store and analyze unstructured documents has become an essential foundation for AI applications in enterprise data intelligence scenarios, directly affecting the effectiveness and efficiency of downstream applications such as intelligent search, automated reporting, and risk monitoring. However, storing and analyzing unstructured documents faces four challenges:

1. Difficult to organize and store. Diverse document formats, blurry chunk boundaries, and difficult metadata management make organization and storage hard.
2. Must accommodate many query needs. A single filter may simultaneously involve sparse vector matching, dense vector matching, and scalar filtering—satisfying these varied hybrid search needs is difficult.
3. Hard to analyze efficiently. Low latency (LLM operators): complex queries may involve LLM-based semantic operators, and it is difficult to guarantee both efficiency and latency optimization.
4. Difficult to scale horizontally and update incrementally. The number of documents grows dynamically, making incremental index maintenance hard.

Faced with these challenges, existing data storage approaches each have strengths in their own application scenarios, but they also have shortcomings and struggle to support the capabilities required by an intelligent unstructured document analysis system.

• Traditional relational databases: used for storing and managing structured data, suitable for scenarios with stable data structures and clear relationships such as business forms, fixed transaction processing, and electronic invoices. Their advantage is strong ACID transaction support—mature and stable—but they offer very weak support for unstructured data.
• Data lakes: used for turning large-scale raw data into assets, storing raw unstructured data. Low storage cost, slow query speed, few analytical features.
• Data warehouses: used for core business decision support based on preprocessed, normalized data, storing processed (semi-)structured data. Higher storage cost, slower query speed, more analytical features.
• Vector databases: used for real-time AI inference scenarios, storing vector data. High storage cost, fast query speed, and only support vector similarity queries.

As we can see, to meet the challenges of storing and analyzing unstructured data, a storage approach needs to have at least the following characteristics:

• Strong ACID transaction support;
• High-performance SQL queries and analytics;
• Support for unstructured data;
• High scalability and flexibility;
• Low storage cost;
• Comprehensive indexing and exact queries;
• Rich AI ecosystem integration;

Building an Intelligent Unstructured Document Analysis System on OceanBase—QUEST

Few storage solutions on the market today can effectively meet the challenges of storing and analyzing unstructured data, and OceanBase is one of them. We therefore built an AI database application—QUEST—on OceanBase to help enterprises automatically parse, structure, and analyze unstructured data, improving data utilization and solving the problem of information silos.

(1) The Limitations of Traditional Approaches

Take product development documents as an example. Enterprise data lakes often accumulate large quantities of long, unstructured documents. In theory, these documents can be extracted into an “attribute-document” two-dimensional table, where each row corresponds to a document and each column to an attribute. In practice, however, the following challenges arise:

• Massive data, focused queries: each user query typically concerns only a tiny subset of the document collection, and the attributes of interest usually appear in only a few text chunks of a document. This is a “needle in the sea”-style requirement.
• Dynamically changing attribute needs: users’ query attributes are hard to fully predefine in advance, and they keep changing as the business evolves.

Therefore, how to efficiently handle the dynamic filtering of attributes and documents and perform fast online extraction has become a technical bottleneck that traditional systems struggle to overcome.

Mainstream attribute extraction systems today typically take a full-extraction approach: whenever a new query attribute is encountered, the system performs a comprehensive attribute extraction on every single document in the collection. This approach brings the following limitations:

• High LLM token cost: large-scale full extraction consumes an enormous amount of LLM token resources, keeping computation and operating costs persistently high.
• Inefficient filtering: user queries often concern only a small number of documents, but full extraction cannot fully leverage the focusing effect of a filter (such as a SQL WHERE clause), wasting a lot of compute on irrelevant documents.
• Insufficient ability to handle dynamic queries: users’ attribute and analysis needs change frequently, and traditional full extraction struggles to respond in time. Every time a new attribute is added, the entire document collection must be scanned again, which is extremely inefficient.

(2) The Limitations of Recent Research

In recent years, systems such as DocETL and Evaporate have attempted to improve on traditional approaches, proposing the “filter first, then extract attributes” optimization path and workflow design methodology, aiming to reduce ineffective extraction operations.

Compared with traditional databases, the filter operator in unstructured document attribute extraction systems differs significantly in both implementation and cost.

Specifically, in an unstructured document attribute extraction system, executing a filter often depends on extracting attribute values as well. In other words, the system must first extract the values of the relevant attributes from each document before it can evaluate the relational algebra of a SQL query.

Therefore, for complex SQL statements—especially those with multiple filter conditions—if a new attribute involved in a filter is not found in the existing cache, the system must additionally perform a large amount of online extraction. This “extract in order to filter” process greatly increases LLM consumption and overall processing cost. In addition, a join operation can also be viewed as a form of filter; its process likewise involves attribute extraction for the columns participating in the join, and the join operation itself can filter out some columns from a table. Recent research has not done enough on the coordinated ordering and optimization of key operators such as filter and join, so it still falls short when facing complex SQL queries with multiple filters.

(3) QUEST’s Innovative Optimizations

To address the above problems, the QUEST system makes comprehensive optimizations, including the following four points.

• Sampling-based selectivity estimation: extraction is performed on sampled documents to obtain a “sample table,” which can be used to estimate the selectivity of any filter, enabling a smarter filtering strategy.
• LLM token cost prediction: by gathering statistics on the length of attribute-related text chunks obtained through RAG (Retrieval-Augmented Generation), the system accurately predicts LLM processing cost, and combines this with selectivity metrics to reasonably order filter priorities.
• Coordinated filter-join ordering optimization: QUEST converts a join operation into an IN operation and orders it together with the filters on the joined table, thereby maximizing the use of filtering to prioritize excluding irrelevant (non-hot) documents and greatly reducing ineffective extraction operations.
• Precise context localization powered by RAG: QUEST uses RAG technology to precisely locate the context in which the attribute to be extracted resides, so there is no need to pass the entire document to the LLM. This substantially shortens the context length, which not only improves extraction accuracy but also further lowers cost.

(4) The Design and Implementation of the QUEST System

The prototype design of OceanBase for QUEST comes from a paper we published at a top-tier international conference, which received a “strong accept” rating at SIGMOD 2025. Let’s first look at the overall framework of the system.

As the figure above shows, OceanBase mainly participates in the secondary index construction and retrieval processes in steps one and two—the four circles marked in the figure. The first step of the overall workflow is to build document-level and chunk-level indexes offline: each original document is chunked, and the chunks are then stored in the corresponding database tables.

The second step is online SQL query, which comprises five processes in total:

1. Sample some documents. The LLM extracts the attributes appearing in the user query Q from them to generate a sample table S, used for query enhancement and optimization.
2. Document retrieval. Q document topic + document index: filter out irrelevant documents.
3. Chunk retrieval. (Q attribute name, S-Evidence) + chunk index: obtain the chunks relevant to the attributes.
4. Based on the selectivity of the attributes in the sample table S and the total token count of the attribute-related chunks, the system generates an optimized execution plan.
5. The system processes the documents one by one according to the plan, calling the LLM to extract from the relevant text chunks the attributes appearing in Q, and caches the results.

In short, QUEST converts a query over unstructured documents into a task of extracting and optimizing the query-relevant attributes.

Below are the operational details of how we use OceanBase in the processing flow. First, we need to store the chunk-level secondary index of the documents in OceanBase’s two-dimensional tables. For the document table, we store the document ID, document content, document summary, document summary embedding vector, and document title; for the chunk table, we store the chunk ID, chunk content, chunk embedding vector, and the document ID the chunk belongs to.

After the tables are created, we need to build the document index and the chunk index separately.

• Document index construction: the documents table.
  • Build a dense vector index on summary_embedding (HNSW semantic similarity).
  • Build a sparse vector index on content (BM25 keyword matching).
  • Build scalar indexes on metadata such as title.
• Chunk index construction: the text_chunks table.
  • Build a dense vector index on embedding (HNSW semantic similarity).
  • Build a sparse vector index on content (BM25 keyword matching).
  • Build scalar indexes on metadata such as doc_id.

Let’s use a simple example. Suppose the query is SELECT age, name, team FROM NBAPlayer WHERE age>30. To process this query, the system must first extract the age attribute in the filter WHERE age>30 from the documents, and then further extract the name and team attributes from the matching documents. So how do we use OceanBase to find the age-related text chunks in each document and feed them to the LLM for extraction?

For the document table, perform a full-text search using the word “NBA,” then run a semantic similarity search using the alpha vector obtained by embedding the table name NBAPlayer, thereby returning documents whose topic is NBAPlayer and filtering out irrelevant ones.

For the chunk table, perform a scalar search constrained by doc_id, run a full-text search using the words “age” and “birthdate,” then run a semantic similarity search using the attribute name age and the Evidence description attached to age, thereby extracting the age-related text chunks from a specific document.

This functionality is one of the core operations of the QUEST system, and OceanBase helps us implement it in an elegant way. The experimental results of the QUEST system:

To test QUEST’s effectiveness, we ran experiments on several datasets including WikiText, SWDE, and LCR—where LCR is a legal dataset and SWDE is a film-and-TV-related dataset—and compared it against other leading unstructured document analysis systems in the field. We found that QUEST clearly outperforms the others in accuracy, cost, and latency.

Why Can OceanBase Support AI Applications with Higher Accuracy, Lower Cost, and Shorter Latency?

So why is OceanBase able to meet the challenges of storing and analyzing unstructured documents and help us build a QUEST system with higher accuracy, lower cost, and shorter latency?

1. An integrated database for AI.

Compared with traditional single-paradigm storage, OceanBase is an integrated database that brings together the advantages of different storage paradigms. It can better meet the challenges of storing and analyzing unstructured documents than the standalone vector databases on the market.

2. Outstanding vector search performance.

For AI application development, OceanBase delivers outstanding vector search performance. The figure below shows the results of a vector database query benchmark, measured live at the OceanBase 2025 Developer Conference. As the figure shows, the number of vector queries OceanBase executes per second is far higher than that of other mainstream vector databases.

3. Multi-modal integrated hybrid search: dense vector + sparse vector + scalar.

OceanBase supports multi-modal integrated hybrid search, allowing users to use all three filter types—dense vector, sparse vector, and scalar—within a single query statement.

4. SQL-Python development interfaces.

OceanBase vector search provides flexible access interfaces. It supports not only SQL access via clients in various languages using the MySQL protocol, but also SDK access (for example, Python), delivering a simple and efficient AI application development experience.

5. Integration with mainstream AI frameworks.

OceanBase integrates with mainstream AI frameworks and services such as LangChain, LlamaIndex, Dify, and Fast-GPT, and is compatible with the standard MCP protocol. It can provide developers with a solid platform and framework across a variety of AI application development scenarios, streamlining the development workflow.

6. Strong open source community support.

The OceanBase community has an active collaborative atmosphere and strong official support. Community members actively participate in OceanBase SIGs (Special Interest Groups) across different sub-fields to collaborate on technology, even holding weekly meetings to discuss development needs and plans—fostering exchange among members and improving development efficiency.

Summary and Outlook

The QUEST system is especially well suited to complex scenarios such as enterprise-scale data lakes, knowledge management, compliance document spot checks, organizing product R&D materials, and contract review, helping users complete online attribute extraction and analysis at low cost, with high efficiency, and intelligently—under the dynamic need to “care about only a small number of documents and attributes.”

Going forward, we will add support in QUEST for querying multi-modal documents that contain images, which requires us to store images in OceanBase. Thanks to OceanBase’s excellent multi-modal integrated interface design, we expect the design for storing and indexing images to be quite similar to the design for text—fully demonstrating OceanBase’s advantages in AI application development.

From our experiments, developing AI applications on OceanBase has three main advantages: support for the integrated database paradigm, native support for distributed scaling, and a strong product ecosystem. In addition, OceanBase can perform queries and storage operations with integrated compute and storage, so developers don’t need to write an extra query engine—greatly accelerating our application development.

As the data foundation of the AI era, OceanBase provides high-quality, scalable, and easy-to-query data sources for RAG and Agent applications. We believe OceanBase will deliver even greater value in the AI application field in the future.

💌

Lao Ji’s Tech Talk hopes not only to keep bringing you valuable technical content, but also to contribute to the open source community together with all of you. If you appreciate the OceanBase open source community, give it a little star ✨! Every Star you give is the motivation behind our efforts~💕

https://github.com/oceanbase/oceanbase