How DeepSeek V4 Uses Less Memory: KV Cache Explained

Long context is expensive for one reason above all others: the key-value (KV) cache. Every token a transformer has already read leaves behind a stored key and value for each layer, and every new token has to look at all of them. At a million tokens, that stored state, not the model weights, is often what decides how many conversations a GPU can serve at once. Matthew Berman's video "How DeepSeek Uses Less Memory", embedded above, walks through how DeepSeek attacks that problem in its V4 family, and this article digs into the underlying engineering using DeepSeek's technical report and the vLLM team's implementation write-up.

The short version is that DeepSeek V4 stops treating the KV cache as one uniform, token-proportional block. It mixes two compression schemes across its layers, keeps a small uncompressed sliding window for local detail, stores keys and values as shared vectors, and drops numeric precision where it can. According to vLLM's analysis, one sequence at a 1M-token context needs about 9.62 GiB of KV cache in bf16, roughly 8.7 times smaller than the 83.9 GiB estimated for a 61-layer DeepSeek V3.2-style stack. Storing the attention cache in fp8 and the indexer cache in fp4 cuts that by roughly another factor of two.

Key points:

• c4a layers compress about 4x and use sparse top-512 selection; c128a layers compress about 128x and use dense attention over the result.
• A 128-token uncompressed sliding window keeps local detail intact.
• vLLM reports about 9.62 GiB of bf16 KV cache per 1M-token sequence, versus about 83.9 GiB for a V3.2-style stack.
• fp8 attention and fp4 indexer caches cut memory by roughly another 2x.
• Compression is lossy, so independent long-context testing matters.

When a language model generates text, it computes a query for the new token and compares it with the keys of every earlier token to decide what to attend to. Recomputing those keys and values from scratch at every step would be hopelessly slow, so inference engines cache them. The cache grows linearly with context length, and it is multiplied by the number of layers. Weights are a fixed cost shared by every user on a GPU. The KV cache is a per-request cost that grows with every token.

That is why long-context pricing, concurrency limits and "context rot" complaints often trace back to memory rather than raw compute. A model with a 1M-token window that needs tens of gigabytes per conversation can serve only a handful of simultaneous long sessions on a single accelerator, which pushes providers to charge more or cap usage. Anything that shrinks the cache directly raises the number of users a deployment can handle, and it makes huge contexts commercially realistic rather than a demo feature.

DeepSeek has attacked this problem for several model generations. Multi-head Latent Attention (MLA), introduced earlier in the DeepSeek line, already compressed keys and values into a smaller latent representation. DeepSeek Sparse Attention (DSA) then reduced how many cached entries each query actually reads. V4 builds on both ideas and adds sequence-level compression, so the cache shrinks along the token axis as well as the feature axis. For wider context on the family, see our DeepSeek V4 everything we know explainer and the DeepSeek V4 GA review.

To judge the saving you need the starting point. The vLLM team estimates a presumed 1M-context sequence on a V3.2-style stack of 61 layers at 1,408 bytes per token per layer, which works out at 1408 B × 1M × 61 ≈ 83.9 GiB per sequence. Each layer stores a main MLA entry (576 values in bf16) plus a small indexer entry (128 values in bf16) that supports sparse selection. Every layer is a full-attention layer whose cache grows with the whole sequence.

Nobody serves one sequence per GPU, so the practical effect is brutal: at that footprint, a node with hundreds of gigabytes of high-bandwidth memory holds only a few long conversations once the weights are loaded. The left-hand panel of the hero figure above shows this uniform design, with every layer storing the same token-proportional blocks.

DeepSeek V4 replaces that uniform stack with a hybrid. According to the vLLM write-up, the model has 61 layers split between two compression types: 30 layers of c4a and 31 layers of c128a. In DeepSeek's own terminology these correspond to Compressed Sparse Attention (CSA) with a milder compression rate and Heavily Compressed Attention (HCA) with an aggressive one, and the technical report covers both.

c4a (the milder scheme). Each compressed token is a weighted sum of 8 uncompressed tokens, with a stride of 4, so the cache is about a quarter of its original length. Because there are still many compressed entries, DeepSeek Sparse Attention picks the top-k of them for each query. The default k is 512.

c128a (the aggressive scheme). Each compressed token is a weighted sum of 128 tokens, with a stride of 128. At 1M context there are at most about 8,000 compressed entries, few enough that the layer can use full attention over them, which vLLM frames as top-k = 8192.

The sliding window. Compression blurs fine detail, and the most recent tokens usually matter most. So every layer also keeps a 128-token window of uncompressed keys and values, which preserves local context near the compression boundary.

Shared key and value. Keys and values share vectors, saving about 2x memory. Because positional information is applied through rotary embeddings, V4 applies an inverse RoPE to the attention output to keep positions correct after the sharing.

The combination is clever because the two layer types do different jobs. The c4a layers retain finer-grained, selectively accessed memory. The c128a layers give the model a very cheap global summary it can attend to densely. Together with the local window, each layer sees recent text exactly, distant text approximately, and the most relevant distant text in detail.

The most useful official-style visual comes from the vLLM team, who built the serving implementation. Their animation of the c4a pattern shows how queries relate to compressed entries and to the local window.

Memory figures vary by source because each compares a different baseline, so it is worth being precise about what each claim means.

• vLLM (bf16 cache): about 9.62 GiB per sequence at 1M context, about 8.7x smaller than the 83.9 GiB V3.2-style estimate.
• Mixed precision: using bf16 for rotary dimensions and fp8 for the rest reduces the cache by nearly half again. In DeepSeek's report the 1M-context cache comes to about 2% of a bf16 GQA8 baseline.
• Relative to V3.2: reporting around the release says V4 Pro uses about 10% of V3.2's KV cache at 1M tokens, and V4 Flash about 7%. NVIDIA's framing is a 90% reduction in KV cache memory burden.

These are consistent rather than contradictory: a 10x reduction against V3.2 is a 90% saving, and the 2% figure is against a different, much larger baseline (grouped-query attention in bf16). The honest takeaway is "roughly an order of magnitude less cache than the previous generation", not any single decimal-place figure. One third-party page describes a "V4.1 Flash" with an 890-byte-per-token footprint and FP4 caching; we could not corroborate that elsewhere, so we do not rely on it.

Compression reduces how many entries are stored. Quantisation reduces how many bits each entry takes. According to vLLM's description, the attention cache uses fp8 and the indexer cache uses fp4, which together cut the KV cache by roughly another 2x compared with the bf16 estimate. Prefill uses a bf16 KV cache, while decode uses partly token-wise fp8.

The indexer is the component that scores which compressed entries each query should read. It tolerates lower precision because it only has to rank candidates, not reproduce values exactly, so fp4 there is a sensible place to save bits. The values the model actually attends over need more care, hence fp8, with bf16 retained for the rotary-position dimensions where precision errors would corrupt positional information.

A cache with several layer types has a practical problem. Full-attention-style blocks grow with the sequence, while sliding-window and compressor-state blocks stay a constant size per request, independent of sequence length. The hero diagram shows this as two groups feeding a single hybrid KV cache manager, which maintains a shared block pool for both kinds. In vLLM's account, the per-layer figures at 1M context are roughly 320 MiB for each c4a layer and about 8 MiB for each c128a layer, which is why the aggressive layers are nearly free by comparison.

Engineering the kernels matters as much as the architecture. vLLM's decode-path diagram below shows the c4a operator graph, with kernel fusions and multi-stream partitioning that run the indexer on a separate stream from the main attention path. A memory saving only becomes a throughput gain if the extra bookkeeping of compression, top-k selection and mixed precision does not eat the benefit, and that is what the fused kernels are for.

Cheaper long context. If a provider can hold roughly ten times as many long conversations per GPU, long-context requests can be priced much closer to short ones. That is consistent with DeepSeek's aggressive API pricing across the V4 line, which we cover in DeepSeek V4.1 Flash 1017 and DeepSeek V4 Pro.

Agents that remember more. Coding and research agents accumulate enormous transcripts of tool calls. A smaller cache makes it feasible to keep that history in context rather than summarising it away. This sits alongside DeepSeek's other memory work, such as the conditional-memory approach in our Engram architecture guide, and tooling like the DeepSeek Harness.

Local deployment. The cache is only half of the memory story: the weights of these models remain large, so a smaller KV cache helps most when you are serving many users or very long prompts, not when the weights alone already fill your machine.

Compression is lossy. Averaging 8 or 128 tokens into one entry discards detail by construction. The sliding window and top-k selection are there to limit the damage, but retrieval of a single exact fact from the middle of a very long context is where such schemes are most likely to be tested. Independent long-context evaluations matter more than the memory figures.

Numbers depend on baselines. As above, a "90% reduction" and "2% of baseline" can both be true. Check what is being compared before repeating a headline.

Engineering complexity. A hybrid cache with several block types needs specialised serving support. vLLM has built it for V4, but other engines may lag.

Secondary sources. Part of this article relies on vLLM's write-up and press coverage of the technical report. We have not independently measured memory use.

Other labs attack the same bottleneck differently. Grouped-query attention shares keys and values across heads. Sliding-window attention caps how much history each layer reads. State-space and linear-attention hybrids replace some attention layers with fixed-size recurrent state. DeepSeek's approach is distinctive in compressing along the sequence axis while still allowing sparse access to the compressed entries. For how rival open models compete on long-context agentic work, see our Kimi K3 launch coverage.

DeepSeek V4's memory story is a systems story. The headline model quality gets the attention, but the reason a million-token context is affordable is a layered set of choices: two compression rates, a local window, shared keys and values, and low-precision storage. The reported result is roughly an order of magnitude less KV cache than its predecessor. Watch independent long-context tests, and watch how quickly other inference engines adopt similar hybrid cache managers.