布吉岛
LLMKV CacheDeepSeekLarge ModelsInference

A Hundred Messages In, How Does the Model Still Remember the First?

4 min read

Starting from how a large model generates text, this explains what KV Cache is, why it is expensive, and how DeepSeek compressed that memory to one eighth, from V2's MLA to V4.1's CED.

On September 10, DeepSeek launched V4.1-Flash. Its official page shows a comparison chart saying the new model’s cache takes only 890 bytes per token — one quarter of the previous-generation V4-Flash and one four-hundred-and-thirty-seventh of V1 from 2024.

The same document also says that for long conversations and long documents, a cache hit costs $0.003 per million tokens. Getting the price down to that figure is the same thing as that 890 bytes.

This cache is where the model remembers context. How it came about, why it used to be so expensive, and how DeepSeek made it smaller step by step — let me start from the beginning.

While Answering, the Model Keeps Looking Back

A large model generates text one word at a time. You feed it a passage; it first chops it into tokens — roughly words or characters — then predicts what the next token is, appends it, predicts the next one, and so on.

When predicting each new token, it looks back at all the tokens before it. This looking back is the core of the Transformer architecture, and it’s called attention.

Each token produces three things in this process: query, key, and value. Think of it as looking up reference material: the key is what clue a token can offer, the value is the content it actually holds, and the query is what the current token is looking for. The current token takes its query, compares it against the key of each preceding token, computes how close or far they are, and then weights and retrieves the corresponding values by that closeness.

The keys and values of the preceding tokens are used in this round and will be used again in the next. Generating the 100th token requires looking back at the first 99; generating the 101st requires looking back at the first 100. Recomputing them every time is too wasteful, so the model stores the keys and values it has computed and fetches them directly next time. What’s stored is the KV Cache.

One Layer Isn’t Enough — Many Are Stacked

That attention routine is what a single layer in the model does. The model is many layers stacked, one on top of the next.

Two main things happen in a layer. Each position first does attention, pulling information relevant to itself back out of the context. Once pulled back, it’s handed to the feed-forward network for some in-place processing.

Doing each of these once is one layer. What you get afterward is a representation processed once, not yet the final answer. To get the model to fully understand a passage, you have to let it look and think through several rounds — the number of layers is the number of rounds.

The rounds also divide the labor. Use the model’s intermediate output to guess the next word, and you’ll find the guesses from the lower layers are all over the place, getting more accurate the higher you go. Probe what each layer has learned and the pattern is much the same: the bottom layers focus on words, grammar, and local collocations; higher up they start handling reference and sentence meaning; only further up do abstract concepts and reasoning emerge. The same sentence is processed into different forms at different layers.

So could you go wide instead of deep? Width can pack in more parameters, but it can’t make up for depth. The chains of logic and multi-step computations in language are mostly nested structures, one layer wrapping another. There’s a class of functions that a shallow-but-wide model can only fit if it grows absurdly large, while a deep model can handle with a few layers. So depth can’t be removed.

Between the layers runs a trunk called the residual. The result after a layer’s processing is added back to the original and passed on. You can think of it as a draft that keeps getting amended, where later layers don’t tear up and rewrite everything before them.

The cost comes with it. The more layers, the more parameters and VRAM, and the computation for generating each token multiplies by the number of layers. For the KV Cache mentioned earlier, each layer stores its own copy, so more layers means a bigger bill.

What the Feed-Forward Network Does

I’ve mentioned the feed-forward network several times; let me add a couple of sentences here.

Attention gathers information scattered through the context; the feed-forward network processes that information and fills in the knowledge learned during training. Each token goes its own way, using the same set of weights.

Sixty to seventy percent of the model’s parameters are in this part, and much of the common sense and facts it remembers are stored here.

It’s a different matter from the KV Cache. The KV Cache stores content computed on the fly for this conversation and is discarded when the chat ends. The knowledge in the feed-forward network is fixed at training time and doesn’t occupy that cache.

It’s now popular to split a large feed-forward network into many small experts, with each token picking only a few to pass through. In DeepSeek V4.1-Flash, each layer has 1 shared expert plus 384 routed experts, and a token activates only 6 of them.

Why This Memory Has Always Been Expensive

As mentioned, the model is built layer upon layer. Every layer does that looking-back routine. Each layer has its own weights, the same sentence comes out processed differently after different layers, and the keys and values each layer computes differ, so each must be stored separately.

With many layers, the same context gets stored in many copies. The longer the context, the more the copies times the length adds up.

At million-token context, this cache can reach hundreds of GB, which no single GPU can hold. Anyone who wants long context must first solve where this memory goes.

DeepSeek’s Two Earlier Steps

DeepSeek has been working on this memory since V2.

V2 was released in May 2024 and introduced an attention structure called MLA. Previously each layer stored the keys and values themselves; MLA switched to a low-dimensional vector, so a token keeps only about 512 dimensions per layer and expands it back when needed. The official claim at the time was that the KV Cache shrank by more than 90% compared with its own 67B dense model.

V3.2 took a different direction. It added sparse attention: each query no longer computes against every preceding token, but scores them first and then looks at a small, most-relevant batch. What it saves is mainly computation and the running cost of long context.

By V4, the model family split into two lines, V4-Pro and V4-Flash. V4-Flash’s backbone is a 284B-parameter MoE that activates only 13B at a time, supports million-token context, and keeps the cache-hit price very low. That cheapness rests on a premise: its KV Cache is already fairly small, about 3.5KB per token.

V4.1 Turns the Memory from Per-Layer Copies into a Building-Wide Shared One

V4.1-Flash was released on September 10: a 552B-parameter MoE with a larger backbone than V4-Flash but fewer parameters activated each time. It compressed the KV Cache to a quarter again.

The approach has several parts, each corresponding to the bill above.

Structurally it made a split, officially called Causal Encoder-Decoder. The 40 layers are cut in half: the first 20 act as the encoder, the last 20 as the decoder. The global KV the decoder needs is projected directly from the hidden state of the encoder’s last layer, no longer generated by each decoder layer on its own. The per-layer copies mentioned earlier become one shared copy for the whole building. As a side effect, the model activates only 8B parameters when reading input and 16B when writing output.

It also reuses across layers, officially called CSA2. It assigns each attention layer one of three modes: compute fully, recompute the index, or reuse another layer’s index. The main KV and the keys used for the index can be shared across layers, and later layers only search within the candidate range given by the previous layer, so the index overhead no longer grows with context length.

On precision, the main KV is stored in 4-bit floats, so the same content occupies only a quarter of the original bit width.

The sliding-window part isn’t written to disk. One block of attention only handles the most recent window, and this part’s KV is not persisted to disk; when needed, you can restore it by replaying the most recent n_win tokens. Disk usage thus drops to one eighth of the previous generation.

Stacking these parts together, the global cache per token drops from about 3.5KB to 890 bytes.

The Savings Are Visible

After V4.1-Flash launched, cache-hit input costs $0.003 per million tokens, with output at $0.60 during off-peak hours. Against V4-Pro’s $0.022 and $1.98, that’s an order of magnitude apart. The concurrency cap also rose from 500 to 2,500.

These numbers matter most directly to people running agents. While working, an agent repeatedly feeds the same context to the model; the higher the cache-hit ratio, the cheaper the bill. With a KV Cache four times smaller, the same VRAM holds four times as much content, and the hit rate follows.

Back to That 890 Bytes

What does 890 bytes mean? A phone photo easily runs three or four megabytes, and an episode of a TV series is a few gigabytes. The model spends only 890 bytes to remember a token — at the cost of remembering a compressed summary rather than the original words.

Compression itself has always been doable; the hard part is not losing capability after compressing, which is why taking up more space used to be preferable. That V4.1 can compress to an eighth and still gain points comes from redesigning the structure and retraining from scratch. The hard part is the training, not that 890 number.

References

Comments