Suanzhi Future Builds an OceanBase Hybrid Search Foundation for Life-Science AI Data Synthesis

Suanzhi Future (算秩未来) is building a retrieval foundation for life-science data used in AI training-corpus synthesis. The target workload must support exact identifier lookup, fuzzy discovery, and eventually semantic similarity. A gene name may be incomplete or misspelled, a sequence fragment may only match approximately, and a researcher may need related sequences without knowing an exact identifier.

The proposed answer is an OceanBase-based hybrid-search foundation that brings business data, large-scale metadata, full-text indexes, and planned vector retrieval into one enterprise data architecture.

Scale and starting requirements

Suanzhi Future adopted OceanBase for life-science large-model training-corpus synthesis. Its algorithm and platform teams wanted to move file-based source data into a database so that the training pipeline could be automated. The database needed unique-ID lookup first, with regular-expression matching, full-text search, and vector retrieval planned as the system evolves.

The numbers are large:

• Approximately 20 TB of raw biological files across 14,081 source files
• Approximately 3.1 billion records modeled into 9 business tables
• Molecules, DNA, RNA, genes, proteins, and central-dogma relationships from authorities such as NCBI

The core objective is to turn file archives into governed, indexable, and traceable data assets that AI pipelines can use directly.

OceanBase data foundation for life-science records

Raw documents and scientific sequences do not naturally fit a single relational row. The team models source material as queryable entities, separating document metadata, scientific attributes, and sequence fragments. Vector representations are a planned extension of this model.

The ingestion pipeline moves from raw files to governed, searchable records:

JSON source files → field governance → parent-child document model → structured and full-text indexes → planned vector indexes

From JSON ingestion to governed fields

Source files arrive as JSON with identifiers, types, lengths, sequence payloads, and rich extra metadata such as annotations, references, and species information. The first step is field governance: promote high-value search fields—such as cid, gene names, locus tags, and taxonomy tags—into typed columns while preserving the original source context in JSON.

This governance step turns a file archive into a queryable OceanBase dataset instead of a collection of opaque blobs.

Parent-child modeling for arrays, metadata, and long sequences

Scientific records rarely fit one flat row. Arrays, one-to-many relationships, and oversized sequences belong in child tables linked to a parent record. The parent holds the core object; children carry repeating attributes, annotation lists, and sequence fragments. The team decomposed nested JSON into three parent-child table groups.

The schema spans molecules, DNA, genes, RNA, proteins, and central-dogma relationships. Each domain uses explicit primary keys, composite indexes, and partition keys to support efficient exact filters at this data scale.

Partitioning strategy: the data is not time-partitioned because these assets are long-lived reference records. Instead, tables use HASH partitioning on CID + ID with 32 partitions per table to distribute globally unique access keys.

Building structured, full-text, and vector indexes

Once entities are modeled, OceanBase provides the index types needed for the retrieval design:

• primary and composite indexes for identifier and attribute filters;
• full-text indexes for gene names, locus tags, and annotation text;
• vector indexes for planned embedding-based similarity over sequence representations.

Original JSON is retained so every hit can be traced back to source context for model training.

Three retrieval layers: L1, L2, and L3

Hybrid search is designed as a coordinated, three-layer recall model:

Layer	Mode	What it does	Example
L1	Exact lookup	Matches identifiers and structured filters with precise conditions	Look up a compound by CID or InChI; fetch a protein by NCBI or UniProt ID
L2	Full-text search	Finds text with keywords, aliases, or partial matches—even with typos	Search gene names or JSON tags for brca or related annotations
L3	Vector search (planned)	Will rank items by similarity in embedding space	Find sequences similar to a reference, even when names differ

The intended query path can chain the layers: narrow results with L1 structured conditions, broaden discovery with L2 full-text search, then rank related items through L3 vectors. Keeping these capabilities in one OceanBase foundation is intended to simplify operations, consistency, and query orchestration.

Handling oversized sequences: the 500 MB large-object limit

A single gene sequence may be hundreds of megabytes or even several gigabytes. The team must split oversized sequences because of the approximately 500 MB limit for an OceanBase large object (LOB).

Instead of storing one oversized blob, the team chunks a sequence into smaller fragments. Each fragment retains an identifier and sequence number so the complete sequence can be reconstructed on read. Applications query metadata and fragments through the parent-child model without needing to handle the underlying fragment count.

This design preserves traceability: every fragment still links back to its source context and metadata while keeping storage and retrieval within platform limits.

Current and planned OceanBase hybrid-search queries

The current system supports:

• L1 exact queries for identifiers, attributes, and structured filters;
• L2 fuzzy lookup through full-text search on gene names and scientific annotations;

The unified chain:

structured filtering → full-text retrieval → planned vector similarity → AI data synthesis

Suanzhi Future can use governed results from the current exact and full-text paths in downstream model-training and synthesis workflows. The target architecture places the data lifecycle—from ingestion and chunking through multi-mode retrieval—within one OceanBase cluster, reducing the need to synchronize multiple systems as the vector stage is completed.

Container and Operator deployment

Suanzhi Future runs its data stack entirely on Kubernetes. MySQL and PostgreSQL use Operators; OceanBase is deployed through OceanBase Operator; Kafka uses Strimzi; and Redis uses Helm. Traffic enters through OBProxy in front of a three-replica (1:1:1) OceanBase cluster.

Early rollout surfaced operational friction the team hopes OceanBase will address:

• Dashboard zone expansion timeouts left Pods healthy but OBZone/OBServer states inconsistent; recovery required manual alter system add server and OBResourceRescue.
• Zone scaling exposed only nodeSelector, not pod affinity controls.
• Node or zone shrink operations could leave the state machine inconsistent.
• Topology, task progress, failures, and recovery guidance were not unified in one view.

The wish list: one-click Operator-driven scale-out with pre-checks, affinity, progress tracking, and rollback runbooks—alongside stronger hybrid-search and GraphRAG capabilities on the same platform.

Platform outcomes

• Approximately 20 TB of multi-source scientific data can become queryable rather than file-bound.
• Approximately 3.1 billion records can be managed under a governed schema rather than as scattered source files.
• Current multi-path retrieval spans IDs, fields, and full text; vector recall is the next stage.
• Shorter pipelines can reduce cross-system synchronization and operational overhead.
• Traceable results link each hit back to its source context.
• A staged evolution path moves from foundational retrieval toward AI-ready applications.

Phase 1 (ingestion and base queries) and phase 2 (fuzzy and full-text recall) are complete. Phases 3 and 4—vector recall and orchestrated AI context—remain in progress. The target end state is a unified data foundation with composable search that returns high-quality context for AI data synthesis.