Explaining DeepSeek-OCR in the Plainest Language — The Disruptive Idea of Optical Context Compression

Posted on 2025-10-31 In AI Applications Views: Word count in article: 4.3k Reading time ≈ 16 mins.

An in-depth reading of the disruptive idea behind the DeepSeek-OCR paper — optical context compression. It explores how to use pixels instead of text as LLM input to achieve a 10x compression ratio, and the profound implications this has for the design of future AI memory systems.

Prologue

I recently came across a highly thought-provoking paper: “DeepSeek-OCR: Contexts Optical Compression”[1]. AI luminary Andrej Karpathy spoke very highly of DeepSeek’s DeepSeek-OCR paper. You might think he’d say, “Wow, this OCR model is amazing, the recognition rate has improved again!”

But he didn’t. On the contrary, he almost waved his hand and said, “It’s a nice OCR model, but that doesn’t matter.” Because what’s truly exciting is a far more disruptive idea this paper raises: have we been feeding AI the wrong “corpus” from the very beginning?

Karpathy’s core point is: perhaps the input to a large language model (LLM) should never be “text” at all, but should always be “pixels.”

This idea sounds a bit convoluted. We clearly have plain text — why insist on first “rendering” it into an image and then feeding it to the AI to look at?

First, it’s an efficiency problem.

The way we currently feed AI with “text” is through something called a “tokenizer,” which cuts a sentence into individual “tokens.” For example, “Hello, world!” might be cut into [“Hello”, “,”, “ world”, “!”]. The problem is that this approach can be very “wasteful.”

The DeepSeek-OCR paper inadvertently provides supporting evidence: it proves that AI can use just 100 “vision tokens” to “decompress” — with high accuracy — original text containing 1,000 “text tokens.” It’s like giving the AI not a long, verbose string of text, but a small, high-density “information compression cracker” (image). The context window the AI “eats” (processes) is shorter, so efficiency is naturally higher.

Information Is More “Faithful,” No Longer Losing Detail

Imagine you ask AI to read a web page for you. The current “text” input approach is like reading the web page’s content to the AI over the phone. All the bolding, color, font size, layout — all this visual information is lost.

The “pixel” input approach, by contrast, is like directly taking a screenshot and sending it to the AI. Which conveys more complete information? It’s self-evident. Karpathy believes pixels are an input method with a “broader information flow.” It can handle not only plain text but also naturally understand the styling of text (bold, color), and even any charts and images on the page.

Bypassing the AI Tokenizer

The first two points are just a warm-up. Karpathy’s real “grievance” is that he wants to get rid of the “tokenizer” entirely. He bluntly “blasted” it: “I have to say again how much I hate tokenizers. Tokenizers are ugly, separate, and not end-to-end. They ‘import’ all the ugliness of Unicode and byte encodings, inherit a lot of historical baggage, and bring security/jailbreak risks… They must be eliminated.”

Why does he hate tokenizers so much? The tokenizer is like the AI’s “mouthpiece” and “stand-in eyes” — it forcibly inserts itself between the “raw text” and the “AI brain.” This “middleman” is not only clumsy but also distorts information.

Karpathy gives a brilliant example: the smiley emoji “😀”. Through the “tokenizer,” what the AI sees is not a “smiling face” but a peculiar internal code, such as [tok482]. The AI cannot use the knowledge it learned about “human faces” and “smiles” when looking at images (transfer learning) to understand this symbol.

But if the input is an image containing “😀”, the AI’s “vision” component will immediately recognize: oh, this is a smiling face. Which is more intuitive? Which is more intelligent? Pixel input lets the AI “see for itself.”

Redefining AI’s “Input” and “Output”

Karpathy’s vision is that for future AI models, the “input end” (the user’s question) should only accept images (pixels), while the “output end” (the AI’s answer) can remain text. Why? Because the task of “understanding an image” (vision-to-text) is far easier — and far more practical — than “drawing a realistic image” (text-to-vision).

This “input with the eyes (pixels), output with the mouth (text)” architecture also naturally fits the two modes in which AI processes information. Input (Encoding): like a human, take in the entire page (image) in one go and understand it as a whole (i.e. bidirectional attention). Output (Decoding): like a human, speak it out word by word (i.e. autoregression). So the real value of the DeepSeek-OCR paper lies not in providing a great OCR tool, but in serving as a “proof of concept.”

It uses experimental data to prove that “reading” via “looking at images” is entirely feasible and possibly even more efficient. This is not merely a “text-to-text” task turning into a “vision-to-text” task; it hints at a more fundamental shift — AI’s main information entry is shifting from “language” to “vision.”

This small OCR study may really have pried open a great big future.

Everyone is welcome to follow the OceanBase community WeChat account “Lao Ji’s Tech Talk,” where technical content related to #databases, #AI, and #OceanBase is continuously updated!

(The author of this article is Chen Zikang (Kuda) from Ant Group. The “Prologue” section above is drawn from Karpathy and Baoyu. Before the main content begins, let me first thank these three for their work~)

Today, I don’t want to do a simple paper walkthrough. I hope we can, together, start from first principles, place this paper within the grand narrative of VLM and LLM development, deconstruct its ideas, examine its value, and explore the future it reveals to us.

Throughout, I will use the “This is my own conjecture” marker to highlight the things I find important — the stories and reflections hidden behind the paper.

We first examine, from the perspective of information theory and system architecture, the fundamental problem DeepSeek-OCR addresses: the contradiction between computational efficiency and information density.

But I’m not saying DeepSeek-OCR overturns everything. For instance, many of the articles published on WeChat accounts these past few days are AI hype. So here I’ll also pour a bit of cold water, so everyone can explore the various possibilities of future AI memory systems more cautiously yet boldly.

1. Opening: What Do I Need to Know?

First, allow me to put forward a claim: this paper’s most core conceptual contribution is absolutely not “a better OCR model.” If that were all, we wouldn’t need to be sitting here today. Its real value lies in proposing a bold and counterintuitive paradigm — “Contexts Optical Compression.”

It asks a fundamental question: when the LLM’s context window becomes the bottleneck for compute and memory, besides grinding away at algorithms and architectures within the “digital domain,” can we take a different path and return to the “analog domain” — or rather, the “optical domain” — to find the answer?

The authors of this paper essentially “render” a document page containing thousands of text tokens into an image, then use an efficient vision encoder to compress it into a few hundred vision tokens. This process is essentially mapping a discrete, one-dimensional symbol sequence (text tokens) into a continuous, two-dimensional pixel matrix (image), then re-encoding it into a discrete, one-dimensional feature sequence (vision tokens). It accomplishes a cross-modal transcoding and compression of information.

So, what is the essential difference between this “optical compression” paradigm and the current mainstream long-context solutions?

Versus RAG (Retrieval-Augmented Generation): RAG is an “open-book exam” strategy. It stores knowledge externally and dynamically retrieves it via a retriever. It addresses the breadth of knowledge but does not compress the context that enters the Transformer’s core computation. Optical compression, by contrast, is more like photographing the open-book reference with “microfilm” and then walking into the exam with a magnifying glass. It acts directly on the information body that enters the context, rather than on the way information is obtained.
Versus attention-mechanism innovations (FlashAttention / RingAttention): these are system- and algorithm-level operations. By optimizing computation and memory access, they enable quadratic-complexity Attention to handle longer sequences. But this does not change the essence of O(N^2) — it just pushes N’s ceiling higher. Optical compression’s idea is completely different: it goes for the root, striving to make N itself extremely small. If 10,000 text tokens can be compressed into 500 vision tokens, then the compute of N^2 plummets by 400x. This is another kind of approach — token compression rather than optimization of compute and storage.
Versus state-space models (Mamba): linear-complexity models like Mamba abandon Attention’s quadratic dependence at the architectural foundation — an architectural revolution. It is orthogonal to optical compression. Optical compression is an encoding strategy; Mamba is a sequence-processing architecture. In theory, we could feed the optically compressed vision-token sequence to a Mamba-architecture decoder, achieving a “double gain.” But for now, Mamba models have not been widely recognized, so in the future there may not be many people paying attention to this combination.

Then, to answer the earlier question: what is DeepSeek-OCR closer to (in essence)? I think it perfectly fuses/biomimics two concepts:

The “memory hierarchy” in computer architecture: we can view the LLM’s attention context as the CPU’s L1/L2 cache — fast but expensive. The context stored via optical compression is like main memory (DRAM) or even a hard disk (SSD): large in capacity and low in cost, but requiring a “decompression” operation upon access (decoding). The simulated “memory forgetting” mechanism in the figure below pushes this analogy to the extreme.

“Lossy compression” in information theory: the figure below clearly shows that when the compression ratio rises from 10x to 20x, OCR accuracy drops from 97% to 60%. This indicates that information is lost — it sacrifices perfect, bit-level text reconstruction in exchange for an order-of-magnitude token compression. For many tasks that don’t require 100% fidelity (such as summarization, sentiment analysis, or even multi-turn conversation-history management), this is entirely acceptable.

2. Diving into the Architecture and Algorithm

First, we need to know the core gripes about LLM long context:

Pure-text mode is still too costly — quadratic growth.
Today’s LLMs have too low an information density for decoding/encoding language. For example, in document layout, 1,200 text tokens often occupy only one page (or even half a page) of physical space. Yet a high-resolution image needs only a few hundred vision tokens to carry all this information. (The problem OCR aims to solve falls into this category.)

As the saying goes, “a picture is worth a thousand words” — the visual modality is essentially an efficient compression medium. Our goal is to achieve “optical context compression” where the number of vision tokens n is far smaller than the number of text tokens N, i.e. n ≪ N.

Question: Are there other efficient media? I think it might be a video medium. You can understand it this way: an image itself is a 2D representation, and it seems to prove that the compression ratio of a 2D representation exceeds the quadratic cost of text context. So, what is a 3D representation? At least, a video medium is 3D, except that the time flow within it is unidirectional.

Moreover, what are we essentially looking for? We are looking for a representation method, and only then for which stage to enlist an NN to solve our needs.

To achieve lossless or near-lossless decoding at a high compression ratio (such as 10:1 or higher), the core lies in the encoder having the ability to capture high-resolution input under an extremely low token count. This is the job of the DeepEncoder.

There are countless decoding approaches, such as InternVL’s tiling method or Qwen-VL’s adaptive-resolution encoding. But their problem is that traditional VLM encoders, at high resolution, either produce too many tokens or cause an activation-memory explosion, affecting training and inference efficiency.

Clearly DeepSeek also recognized this problem, so they designed the DeepEncoder — with its Serial Hybrid Attention mechanism — in three steps:

Local perception and high-resolution input (Window Attention Dominance): the first half of the DeepEncoder uses a window-attention-dominated SAM-base structure (the famous Segment Anything (SAM) [2] proposed by Meta AI) (about 80M parameters).
- Function: it can process high-resolution input (such as 1024×1024 or higher), splitting the image into a large number of initial patch tokens (such as 4,096). Because it uses local window attention (or it can be understood as a sliding window), even with so many patch tokens, its activation-memory consumption stays at an acceptably low level. This mimics the human visual system’s fine focus on local detail.
16x Token compressor (The Information Bottleneck): this is the core of the DeepEncoder. After local attention, they cascade a 2-layer convolution module that performs 16x token downsampling.
- Result: 4,096 tokens are instantly compressed to 256 tokens. This greatly reduces the computational burden of the subsequent global-attention layers, achieving efficient token compression and memory control.
Global knowledge and semantic integration (Dense Global Attention): the small number of compressed tokens enter a component based on CLIP-large (300M parameters).
- Function: the visual knowledge brought by CLIP pre-training enables it to efficiently integrate the compressed visual features, transforming pure pixel perception into “knowledge features” with higher semantic density.

Conclusion: the DeepEncoder successfully turns the challenge of high-resolution input into a controllable, low-activation-memory compression problem, outputting a set of highly refined latent vision tokens:

2.3. Efficient Decoding and Knowledge Reconstruction (MoE Decoder: Decompression and Retrieval)

After the preceding compression operation, the compressed vision tokens Z now need to be reconstructed by the LLM decoder f back into the original long text X.

Goal: how do we get a compact language model to accurately “hallucinate” and output text up to 10x longer from such a small amount of visual information?

Learning the compression-decompression mapping: the decoder needs to learn the nonlinear mapping

- Support from data engineering: note that the LLM **has already implicitly learned this mapping relationship.** Because its training data includes not only traditional OCR 1.0 data (multilingual, coarse/fine-grained document annotations) but also complex OCR 2.0 data (chart parsing, chemical formulas, plane geometry). This ensures the vision tokens the model learns are not merely pixel representations but **high-level, structured semantic information.**

Choice of decoder. We can easily note that, clearly:
- Consideration 1: the MoE architecture is suited to high-throughput, large-scale OCR decoding and data production (for example, 200k+ pages per day) (the DeepSeek team really loves “cost reduction and efficiency gains,” haha).
- Consideration 2: whether OCR 1.0 data or OCR 2.0 data, both show that the data distribution is sparse (because the data is easily classified), so I conjecture the model’s parameters should also be sparse. And MoE is itself a sparse model. I think this is the key reason why using an MoE model to express the f mapping works so well “this is my own conjecture 😄”.
- Therefore, DeepSeek-OCR uses DeepSeek-3B-MoE (about 570M active parameters) as the decoder.
- Reflection: could a larger parameter count solve more modalities of data? (For example, helping synthesize training data for front-end generation tasks? “This is my own conjecture 😄”)
- Problem: I surveyed the community’s experience using OCR and found that MoE’s understanding of images is, more often than not, only reflected in a “content extraction” capability. But in other respects, one is that the hallucination rate is roughly 80%, and another is that this MoE’s instruction-following ability is still not enough. Therefore, I think there is still much room for improvement here — after all, it’s only a 3B model. Or perhaps there will be other new architectural innovations to solve this problem.
Validation of the compression boundary: experiments prove the potential of optical compression is astonishing.
- When the number of text tokens is within 10x the number of vision tokens (compression ratio < 10x), the model can reach about 97% OCR decoding accuracy.
- Even at an extreme 20x compression ratio, OCR accuracy can still stay at about 60%.
- Academic value: this result provides empirical guidance, proving that a compact language model can effectively decode information from a compressed visual representation. It provides a theoretical foundation and practical bounds for future VLM token-allocation optimization and LLM context compression.

2.4. From OCR to Future Cognition: Beyond the Document

DeepSeek-OCR is not merely an efficient OCR model; it is pioneering work that explores the synergy of vision and language modalities (VLM/LLM) based on the first principle of information compression. Its core algorithmic flow — from the DeepEncoder’s low-activation-memory, high-compression architecture to the MoE decoder’s efficient knowledge reconstruction — provides us with a complete paradigm for converting long-text information into computationally efficient visual latent representations. There is much worth learning from it.

Deep parsing and structured extraction. OCR tasks are no longer simple text recognition. DeepSeek-OCR has deep-parsing capabilities. Through a unified prompt, the model can:
1. Convert charts in a document into HTML tables (a key capability in financial reports).
2. Recognize chemical formulas and convert them into SMILES format.
3. Parse the structure of plane-geometry figures.
4. ……

Undoubtedly, this “OCR 1.0 + 2.0” technology is a cornerstone of VLM/LLM development in STEM fields. And it actually exceeds our definition of OCR. It is an end-to-end algorithm for data recognition and data cleaning (nlp instructions + pdf -> structured data). Extrapolating, I think this approach can solve 80% of the various thorny problems in LLM corpus cleaning. I wonder whether the LLM data team might be able to retrofit the DeepSeek-OCR framework to build a general-purpose, vision-based corpus-data-cleaning framework? (The consumption of a 3B model is quite tempting, haha) “This is my own conjecture 😄”

Simulating memory and forgetting mechanisms.

Optical context compression provides an elegant scheme for simulating human memory decay. The background of this academic problem: in a multi-turn dialogue system, how do we manage historical context to prevent a computational-overhead explosion?

Mapping to human memory: humans remember recent events clearly and distant events vaguely. This decay mechanism resembles the pattern in visual perception where information degrades with distance or resolution.
Optical (visual) implementation:
- Recent context: rendered into a high-resolution image and encoded using the DeepEncoder’s high-fidelity mode (such as Gundam or Large mode), preserving high fidelity.
- Distant context: by progressively shrinking the rendered image (corresponding to Tiny or Small mode, i.e. via window attention), vision-token consumption can be further reduced. The reduction in token count leads to a natural “blurring” of the text and a decay in information precision, thereby achieving the progressive forgetting of memory.

3. Beyond the Paper — On DeepSeek-OCR’s Inspiration and Extrapolation for Memory-Mechanism Design

We know that traditional memory frameworks are mostly closed-loop memory systems: after each interaction, the new memory is compressed by C (usually vector compression RAG, or some so-called NN-model compression such as MemGen) and stored in tiers for future retrieval.

But now there is one more method of compression C — optical context compression.

What DeepSeek-OCR really wants to say is how to use the optical-visual channel to redesign the “memory system” of large language models. The paper uses the concept of OCR to mask DeepSeek’s true purpose, so let me deduce “technology’s next step.” Below, following the main thread of “memory” rather than “OCR,” combined with my understanding and experience of LLM memory, I offer a line of thought-experiment in the direction of LLM memory.

3.1. Aligning the Concepts Once More

Level	Human counterpart	LLM counterpart	Classic solution	DeepSeek-OCR’s entry point
1. Sensory memory	0.1 s retinal afterimage	Raw 10K-100K tokens fed directly to Attention	——	Turn “text into pixels” → vision tokens, using optics to compress the sensory buffer by an order of magnitude
2. Working memory	7±2 chunks	kv-cache resident in GPU HBM	Sliding window / sparse attn	Use image resolution as the “aperture” to dynamically adjust cache size
3. Long-term memory	Hippocampal-cortical consolidation	External RAG / parametric memory	Vector store / LoRA	Make “forgetting” a differentiable optical downsampling rather than a manual threshold

DeepSeek-OCR’s core idea is to move the compression process from level 1→2 out of the “digital-sequence” domain and into the “optical-pixel” domain, thereby decoupling “memory capacity” from “computational overhead.” All the extrapolations below are amplifications of this idea.

3.2. Exploration 1: Treating “Context” as a Fine-Tunable “Holographic Plate”

3.2.1. An Intuitive Analogy

Traditional Transformer: like tearing a book into 10,000 small slips of paper, where each slip must “shake hands” pairwise with all the others — O(n²) handshakes.
DeepSeek-OCR: photograph the whole book at once into a “miniature holographic plate,” letting only a few hundred “light spots” (vision tokens) enter the handshake zone; the number of handshakes plummets 400×.

3.2.2. The Compression Method

Since the DeepEncoder’s image compression is so effective, I would choose the “text → image → vision token” channel. In essence, it performs a controllable lossy encoding:

H(text) ≈ 8 bit/char × 6 char/token ≈ 48 bit/token
H(image) is on the order of 1024×1024×3×8 ≈ 25 Mbit, but after the DeepEncoder only 256×d-dimensional floating-point codes remain. Assuming d=1024, FP16, ≈ 0.5 Mbit. At a compression ratio of 50×, the information-entropy loss is ≈ 3% (refer to the paper’s 97%→60% experimental curve).

The key to this operation is: bake “forgetting” into the encoder weights in advance, letting the model learn which visual textures correspond to “discardable” layout whitespace and which correspond to “must-keep” semantic tokens — far more elegant than after-the-fact heuristic-threshold pruning.

3.2.3. Implementation Sketch: A “Programmable Aperture” Driving the Memory Gate

Because in LLM memory we want the model to dynamically adjust the compression ratio, perhaps I can boldly try changing the DeepEncoder’s 16× conv downsampling into a learnable, content-dependent “aperture module”:

3.3. Exploration 2: Turning RAG into “Optical + Vector Dual-Path Recall”

The retriever only looks at the “semantic vector,” ignoring the “layout structure” — so tables, formulas, and multi-column PDFs are often cut off mid-way;
After recall, the entire passage still has to be re-tokenized, with no compression — the context-length bottleneck remains unsolved.

3.3.2. An Optical Dual-Path Camera

Treat the DeepEncoder as an “optical retina” working in parallel with the semantic vector:

doc → ┌── semantic encoder → 256 d vector ─┐
        └── DeepEncoder  → 128 vis tokens ─┤
                   ↑                       |
                   +–––––––– fuse –––––––––+

Recall stage: both index paths build their stores simultaneously — the semantic vector handles “meaning,” the vis tokens handle “layout.”
Reading stage: stuff the recalled vis tokens directly into the LLM’s vision slot, with no need to expand them back into text tokens.
Expected result: layout integrity ↑ (tables don’t break); context length ↓ (128 vs 800+ text tokens); complexity is still O(n²), but n is already 6× smaller.

3.4. Exploration 3: What Might a Unified Memory Framework Be?

3.4.1. Architecture Overview

I would design it this way: it has only three executable binaries — optix-encode, optix-cache, optix-recall — yet it simultaneously replaces the long-context KV-cache, the RAG vector store, multi-turn dialogue-history management, and even the LoRA weight repository.

Let’s call the whole system the Optix-Memory Stack (OMS for short), summarized in one sentence:

“Text, weights, or dialogue of any length are all first photographed into a 1024×1024 grayscale image, then compressed into 128 vision tokens, becoming the sole addressing unit; Attention only ever does its quadratic computation within these 128×128 ‘light spots.’”

3.4.2. Training = a Single Loss in “Image Space”

The objective function keeps only two terms:

1	L = L_task + λ · aperture

After convergence, freeze the aperture into 3 discrete levels:

Gundam (aperture 1.0) → high fidelity, for code/formulas;
Small (0.1) → everyday conversation;
Tiny (0.02) → ultra-long memory store.

3.4.3. Inference Engine: 0 Floating-Point Weights, 0 KV-cache

No traditional KV-cache, VRAM footprint = 128×dim×layer = constant (8 layers, 1024 dim ≈ 32 MB);
If vis-w (LoRA) is hit, feed the 128 vw directly into Attention as “diffraction weights,” with no need to dispatch floating-point ΔW;
On a cache miss, read png from disk → diffraction layer → vw, taking < 1 ms;
The entire inference process scales batch size linearly, with no long-sequence explosion.

3.4.4. Wrapping Up in One Sentence

The Optix-Memory Stack collapses the three dimensions of “context length,” “weight size,” and “number of dialogue turns” — which once each ballooned independently — all into a single 1024×1024 grayscale image. From then on, the LLM’s memory problem is translated into an optical system with adjustable aperture, diffractability, and forgettability — no longer asking “how many tokens can it hold,” but only “how many pixels do we want to keep.”

4. More Questions

Upper limit of vision-token concatenation: measure the critical point of Nvisual on answer quality and VRAM.
“Incremental compression” memory curve: verify whether multi-resolution compression is better than one-shot discarding.
Expert-Type self-discovery: verify the automatic alignment of layout category ↔ MoE routing.

5. Conclusion

#DeepSeek-OCR gives us another chance to examine the representation of information and the boundary of computation.

Perhaps in the future, when evaluating a model’s memory ability, we will no longer ask “how many tokens can it hold,” but instead ask: “On a plate that represents memory, how many pixels should we choose to keep?”

References

[1] “DeepSeek-OCR: Contexts Optical Compression”: https://github.com/deepseek-ai/DeepSeek-OCR/blob/main/DeepSeek_OCR_paper.pdf

[2] The famous Segment Anything (SAM) proposed by Meta AI: https://segment-anything.com/

Explaining DeepSeek-OCR in the Plainest Language — The Disruptive Idea of Optical Context Compression