KV-Cache Optimization: Reducing Memory for Long Context

Published: July 2026 | Reading time: 10 minutes | Topic: AI Infrastructure

---

The Memory Wall

As context windows grow from 4K to 32K, 128K, and even 1M tokens, the KV-cache (Key-Value cache) becomes the dominant memory bottleneck in transformer inference. For a single 70B parameter model with a 128K context window, the KV-cache alone can exceed 32 GB of GPU memory — more than the model weights themselves.

This post breaks down the key techniques for taming the KV-cache, from attention variants to compression strategies.

What Is the KV-Cache?

During autoregressive generation, each new token must attend to all previous tokens. Without caching, this means recomputing attention for the entire history at every step — O(n²) total work.

The KV-cache stores the Key and Value vectors from previous tokens so each new decode step only needs to:

1. Compute K, V for the new token
2. Append them to the cache
3. Run attention over the cached K, V

Memory per token ≈ 2 × num_layers × num_heads × head_dim × bytes_per_element

For LLaMA 3 70B (80 layers, 64 heads, 128 dim, FP2): ~5 MB per token

Technique 1: Multi-Query Attention (MQA)

Standard Multi-Head Attention (MHA) stores separate K, V projections for every attention head. MQA shares a single K, V across all heads.

# MHA: K, V shape = [batch, num_heads, seq_len, head_dim]
# MQA: K, V shape = [batch, 1, seq_len, head_dim]  (shared)

# Memory savings: num_heads × less KV storage
# LLaMA 3 70B: 64x reduction in KV-cache size

Tradeoff: Slight quality degradation on some benchmarks, but often negligible in practice. Used in GPT-4 and many proprietary models.

Technique 2: Grouped-Query Attention (GQA)

GQA is the sweet spot between MHA and MQA. Instead of one KV for all heads or separate KVs per head, groups of heads share a single KV projection.

# GQA: K, V shape = [batch, num_kv_groups, seq_len, head_dim]
# LLaMA 3 70B uses 8 KV groups for 64 query heads → 8x KV reduction

num_key_value_groups = num_attention_heads // num_kv_heads
# e.g., 64 heads / 8 KV heads = 8 groups

This is the default in LLaMA 3, Mistral, and most modern open models. Quality is virtually identical to MHA while providing significant memory savings.

Technique 3: Sliding Window Attention

Instead of caching all previous tokens, only keep the most recent W tokens (window size). Older tokens are simply dropped from the cache.

This limits memory to O(W) per request instead of O(sequence_length). Models like Mistral’s 32K context use this, and the recent Mixtral paper showed that sliding windows can be combined with full attention at key layers for a quality/performance tradeoff.

# Sliding window of 4096 tokens
# KV-cache max size: 4096 × 5 MB/token = 20 GB → 20 MB
# (for a 70B model with GQA)

Technique 4: KV-Cache Quantization

Store the KV-cache in lower precision without significantly affecting output quality:

• FP16 → FP8: 50% memory reduction, minimal quality loss
• FP16 → INT8: 50% memory reduction, slight quality impact
• FP16 → INT4: 75% memory reduction, more noticeable degradation

vLLM supports FP8 KV-cache natively since v0.5.0, and the quality impact is typically <0.5% on standard benchmarks.

Technique 5: Cross-Layer Attention (CLA)

Adjacent layers in transformers often have very similar attention patterns. CLA shares KV projections across 2-4 consecutive layers, reducing the number of unique KV-caches.

# Standard: 80 layers → 80 unique KV-caches
# CLA (share every 2 layers): 40 unique KV-caches → 50% savings
# CLA (share every 4 layers): 20 unique KV-caches → 75% savings

The MiniCPM3 and MobileLLM papers demonstrated this effectively for edge deployment.

Technique 6: Token Pruning and Eviction

Not all tokens are equally important. Techniques like H₂O (Heavy-HitterOracle) identify and evict tokens with low attention scores:

1. Track cumulative attention score for each cached token
2. When cache is full, evict tokens with lowest scores
3. Protected tokens: recent tokens + high-attention „heavy hitters“

This can reduce cache size by 50-80% with minimal quality impact on most tasks.

Comparison Table

Technique	Memory Savings	Quality Impact	Complexity
Multi-Query Attention (MQA)	Very high (10-50x)	Small	Model architecture change
Grouped-Query Attention (GQA)	High (4-16x)	Negligible	Model architecture change
Sliding Window	Configurable	Task-dependent	Serving-side
FP8 Quantization	50%	Minimal	Serving config
INT4 Quantization	75%	Moderate	Serving config
Cross-Layer Attention	50-75%	Small	Model architecture change
Token Pruning (H₂O)	50-80%	Task-dependent	Serving-side

Production Recommendations

For most production deployments in 2026, we recommend the layered approach:

1. Start with a GQA model (most modern open models use GQA by default)
2. Enable FP8 KV-cache quantization in vLLM (one config line)
3. Use prefix caching to share KV-cache across requests with common system prompts
4. Add sliding window if you don’t need perfect recall beyond your window size
5. Consider speculative decoding for workloads dominated by long generation

Conclusion

KV-cache optimization is the single most impactful lever for scaling LLM inference. A combination of GQA, FP8 quantization, and prefix caching can reduce effective memory per request by 5-10x, enabling larger batch sizes, longer contexts, and lower costs.

The key is to match the optimization to your workload: agentic systems benefit most from prefix caching, chat applications from sliding window, and high-throughput APIs from quantization + continuous batching.

---

Part of our AI Infrastructure series. Also read: AI Inference Optimization: From Batching to Continuous Batching.