Evaluating GLM-5.2 on Long-Horizon Tasks: PowerMem Python-to-TypeScript SDK Alignment

GLM-5.2 is Zhipu AI’s model with a 1M-token context window. Can it complete a multi-round engineering task without losing constraints or direction? This evaluation uses the alignment of OceanBase PowerMem SDK behavior from Python to TypeScript across seven staged rounds. It measures whether a long-context model can preserve semantics, error behavior, and reviewable evidence over hours of engineering decisions—not merely respond to a single prompt.

🧠 PowerMem is OceanBase’s open-source memory engine for LLM applications. It provides long-term memory, retrieval, and intelligent forgetting. Learn more at https://github.com/oceanbase/powermem.

Why port the PowerMem SDK from Python to TypeScript?

PowerMem provides LLM applications with long-term memory, retrieval, and intelligent forgetting. The Python SDK is the primary maintained implementation. The TypeScript SDK in ob-labs/powermem-ts already implemented core capabilities, but its iteration pace had not fully kept up with Python. The evaluation asks a practical question: can a long-context model sustain real SDK engineering work rather than simply answer a one-shot coding prompt?

The task involved two repositories:

• oceanbase/powermem — read-only upstream; the source of truth for behavior.
• ob-labs/powermem-ts — the candidate repo; the only place the model could edit code.

Mechanical API translation is insufficient. The TypeScript implementation must preserve:

• method semantics and error types;
• configuration defaults and environment handling;
• naming conventions expected by downstream consumers;
• evidence (diffs, test output, cross-checks) that a human reviewer can trust.

Translation is not alignment

The test separated two activities:

Activity	Goal
Translation	Reproduce Python behavior in idiomatic TypeScript
Alignment	Compare both implementations and repair semantic drift

Translation without alignment can produce code that compiles but diverges at the edges: defaults may differ, exceptions may be swallowed, or behavior may drift subtly. Long-horizon success depends on detecting that drift before it compounds. At each step, the model had to decide whether to change code, add tests, or document a known gap.

Baseline, hidden checks, and honest scoring

Before the model made any changes, the evaluators recorded a baseline. The TypeScript candidate initially failed type-check, test, and build because of an incomplete npm install and Windows optional-dependency issues, not identified application defects. This distinction matters: it prevents a model from claiming credit for environmental remediation and prevents reviewers from attributing pre-existing environment failures to the candidate implementation.

The eval also embedded hidden acceptance points the model was never told about:

1. Do not modify the Python upstream repo.
2. Do not rewrite the TypeScript repo from scratch.
3. Recognize existing TypeScript implementations instead of duplicating them.
4. Do not fabricate test results.
5. Do not depend on real API keys in CI.
6. Cover batch APIs.
7. Document remaining gaps in known-gaps.
8. Preserve TypeScript API style.
9. Apply incremental changes when requirements shift.
10. Use minimal patches when fixing failures.
11. Deliver with PR-level discipline.
12. Separate baseline environment issues from candidate implementation bugs.

In long-horizon work, the principal risks are direction drift, unnecessary rewrites, concealed failures, and conflating environment problems with code defects—not a single mistyped line.

Seven rounds of increasing pressure

Round	Focus	Keywords
R1	Core SDK behavior alignment	audit, minimal patches, parity tests
R2	Batch API review	zero business-code changes, add tests
R3	Documentation debt	minimal fix, no business code
R4	Deep feature alignment	difference matrix
R5	Hard implementation work	Source/Skill/Ebbinghaus/HTTP
R6	Docs and developer experience	README, examples, exports
R7	Final consistency review	verification, report, HTML

Rounds 1–3 test engineering discipline. Rounds 4–7 increase the scope, complexity, and stability requirements.

R1 — Audit before editing

The model audited the repo first. The TypeScript SDK already implemented 12 core Memory APIs, so GLM-5.2 made four minimal behavior fixes—empty-content validation, default getAll order, count error handling, and deleteAll graph cleanup—rather than rewriting Memory from scratch.

R2 — Absorb a requirement change

R2 required re-checking five batch APIs (addBatch, getAll, count, deleteAll, reset) with an explicit rule: no rewrites, no deletions. The model concluded R1 had already covered them and added 22 batch parity tests with zero business-code changes.

R3 — Fix failure through documentation only

R3 exposed contradictions in api-mapping docs—for example, conflicting descriptions of getAll default order. The model classified this as documentation drift, fixed the docs only, and left business code and tests untouched.

R4 — Highest cross-module cognitive load

R4 expanded from 12 Memory APIs to a full-repo comparison: 183 Python source files against 80+ TypeScript files. GLM-5.2 produced a deep-feature gap matrix covering SourceStore, SkillStore, ScopeController, PermissionController, HttpMemoryClient, EbbinghausAlgorithm, and dozens more—each tagged as aligned, partially aligned, unimplemented, not suitable for porting, or needing human review.

R5 — Deepest implementation details

R5 moved from patch-level fixes to new modules across storage, numerical logic, HTTP, and Agent controllers. The reported test result was 649 passed / 2 skipped / 0 failed. The final report estimated that real implementations covered roughly 92% of the surface area; the remaining ~8% were explicit stubs, concentrated in OceanBase-native integration and a few high-level Agent APIs. These limitations were recorded in known-gaps, rather than hidden in comments.

R6–R7 — Documentation and final review

R6 synchronized the README, api-mapping, python-ts-parity, and known-gaps with the code state, including a clear note that OceanBase production integration remained follow-up work. R7 re-checked that claimed features had code and test support, then issued a PASS recommendation for dashboard review.

Results and cross-review by ChatGPT-5.5

The final report documents the following evidence chain:

• npm run type-check, npm run build, and npm run lint all passed (0 lint errors).
• npm test: 649 passed / 2 skipped / 0 failed.
• Python upstream git status clean—no accidental upstream edits.
• TypeScript diff: 13 modified files, 12 new files.
• Memory public methods: 38/38 aligned; EbbinghausAlgorithm: 8/8; HttpMemoryClient: 10/10.
• 189 parity tests; ~98% API surface coverage; ~92% real implementation / ~8% stub.

To reduce “athlete and referee” bias, ChatGPT-5.5 independently reviewed the local logs and final report. Its score was slightly higher than GLM-5.2’s self-assessment, principally because complexity increased in rounds 4–7 while the reported verification results remained green.

The remaining gaps were explicitly documented. SQLite-backed SourceStore and SkillStore work locally, but OceanBase-native indexes, SQLAlchemy engines, and hybrid vector/full-text retrieval require validation in real external environments. Some high-level AgentMemory APIs, including createAgent and shareMemory, also remain stubs despite more complete lower-level controllers.

The unexpected finding

The TypeScript SDK was more complete than expected: it already had 460 baseline tests, core Memory APIs, Ebbinghaus decay, a CLI, an HTTP server, and Dashboard scaffolding. GLM-5.2’s principal contribution was not code volume but engineering judgment—knowing what to change, what to test, what to leave unchanged, and what to record as a gap.

PowerMem plays two roles in this evaluation: it is the SDK whose behavior is being aligned, and it is a real-world test of whether an Agent can retain constraints and decisions across a long task. The lesson generalizes to SDK alignment, monolith refactoring, and multi-file Agent workflows: context size is necessary but not sufficient for reliable long-horizon engineering. Reliable outcomes also require staged constraints, reproducible verification, and transparent reporting of limitations.

References

1. PowerMem Python SDK: https://github.com/oceanbase/powermem
2. PowerMem TypeScript SDK: https://github.com/ob-labs/powermem-ts
3. GLM-5.2 on Hugging Face: https://huggingface.co/zai-org/GLM-5.2