AI Inference Optimization: From Batching to Continuous Batching

Published: July 2026 | Reading time: 12 minutes | Topic: AI Infrastructure

---

Introduction

Deploying large language models in production is a fundamentally different challenge from training them. While training gets most of the attention, inference optimization is where the real engineering battle is won or lost. The difference between a well-optimized inference stack and a naive one can be 10x in throughput, 5x in latency, and 8x in cost.

In this guide, we’ll walk through the key techniques that power modern LLM serving — from basic static batching to the cutting-edge continuous batching systems behind platforms like vLLM, TensorRT-LLM, and SGLang.

The Problem: Why Inference Is Hard

Transformer inference has two distinct phases with very different computational characteristics:

• Prefill (encode): Process the entire input prompt in parallel. Compute-heavy, benefits massively from GPU compute.
• Decode (generate): Generate one token at a time. Memory-bandwidth-bound, each step reads the entire KV-cache.

This asymmetry means that a naive implementation either wastes GPU compute (during decode) or runs out of memory (with large KV-caches). The goal of inference optimization is to keep the GPU saturated during both phases.

Level 1: Static Batching

The simplest optimization: group multiple requests together and process them as a single batch.

# Naive approach: one request at a time
for request in requests:
    output = model.generate(request)  # GPU underutilized!

# Static batching: process N requests together
batched_output = model.generate(requests)  # Better GPU utilization

Problem: All requests in a batch must finish before new requests can start. If one request generates 2000 tokens and others finish in 50, the GPU sits idle waiting for the long one. This is called the „convoy effect.“

Level 2: Dynamic Batching (Triton-style)

NVIDIA’s Triton Inference Server introduced dynamic batching: requests are grouped on-the-fly, and the server waits a configurable amount of time for more requests to arrive before executing.

# Triton dynamic batching config
max_batch_size: 32
batching {
  max_queue_delay_microseconds: 100000  # Wait 100ms for more requests
}

This improves latency for bursty traffic but still suffers from the convoy effect within each batch.

Level 3: Continuous Batching (The Game Changer)

Continuous batching, first introduced in the Orca paper (2022) and popularized by vLLM (2023), changes the fundamental unit of scheduling:

Key insight: Instead of batching at the request level, batch at the iteration level. After each decode step, insert new requests and remove completed ones.

# Continuous batching pseudocode
batch = initial_requests
while batch.has_active_requests():
    # Process one decode step for ALL active requests
    batch = model.decode_step(batch)
    
    # Remove completed requests
    batch.remove_finished()
    
    # Insert new waiting requests
    batch.add_new(waiting_queue)
    
    # Handle preemption if memory is tight
    if memory_pressure():
        batch.preempt_lowest_priority()

Result: Throughput improvements of 2-8x over static batching because short requests don’t wait for long ones, and the GPU is never idle when there’s work to do.

PagedAttention: The Memory Breakthrough

Continuous batching requires solving a memory problem: each request’s KV-cache grows dynamically, leading to fragmentation. vLLM’s solution is PagedAttention, inspired by virtual memory paging in operating systems.

Instead of allocating contiguous memory for each request’s KV-cache, PagedAttention:

1. Divides the KV-cache into fixed-size pages (typically 16 tokens per page)
2. Maps pages tables per request (like virtual memory page tables)
3. Allows the same page to be shared across requests with common prefixes

# Without PagedAttention: 60-80% memory wasted to fragmentation
# With PagedAttention: <4% memory waste

# Example: 100 requests, avg 512 tokens
Naive KV-cache:   100 × 2048 × 2 × 40 × 256 × 2 bytes = ~8.4 GB (worst case)
PagedAttention:   ~4.2 GB (actual needed) + ~0.17 GB overhead

Level 4: Advanced Techniques (2025-2026)

The state of the art keeps evolving. Here are the techniques powering production systems in 2026:

1. Chunked Prefill

Split the prefill phase into smaller chunks interleaved with decode steps. This prevents long prompts from blocking short-generation requests, improving tail latency significantly.

2. Speculative Decoding

Use a smaller „draft“ model to generate candidate tokens, then verify them in parallel with the larger model. When the draft is correct (often 60-80% of tokens), you get 2-3x decode speedup for free.

# Speculative decoding with Eagle-3
# Small model drafts 5 tokens → Large model verifies all 5 in parallel
# If tokens 1-3 match: save 4 decode steps
# If mismatch at token 2: fallback, accept only token 1

3. KV-Cache Quantization

Store the KV-cache in FP8 or INT4 instead of FP16, cutting memory usage by 50-75%. vLLM’s FP8 KV-cache support is now production-ready with minimal quality degradation.

4. Prefix Caching

Cache the KV-cache for common prompt prefixes (system prompts, few-shot examples). When a new request shares the same prefix, skip the prefill entirely. This is especially powerful for agentic workloads that use large, consistent system prompts.

5. Disaggregated Serving

Separate prefill and decode onto different GPU pools. Prefill runs on compute-optimized GPUs (H100), decode on memory-bandwidth-optimized ones (L40S). This allows independent scaling of each phase.

Benchmark Comparison

Technique	Throughput (tok/s)	P99 Latency (ms)	Memory Efficiency
Static batching	1x baseline	High (convoy)	Poor (fragmentation)
Dynamic batching	1.5-2x	Medium	Poor
Continuous batching	3-5x	Low	Good (pages)
+ Chunked prefill	3.5-5.5x	Very low	Good
+ Speculative decode	5-8x	Very low	Good
+ KV-cache FP8	5-8x	Very low	Excellent

Choosing Your Stack

For production deployments in 2026, here’s our recommendation:

• Easy start: vLLM with continuous batching + PagedAttention (works out of the box)
• Maximum performance: TensorRT-LLM with speculative decoding + FP8 quantization
• Multi-model serving: SGLang with radix attention for shared prefixes
• Cloud managed: Anyscale Endpoints, Together AI, or Fireworks (they handle optimization for you)

Conclusion

Inference optimization is no longer optional — it’s a core competency for anyone deploying AI at production scale. The techniques covered here (continuous batching, PagedAttention, chunked prefill, speculative decoding, KV-cache quantization) can collectively deliver 5-8x throughput improvement and 50-75% memory reduction over naive implementations.

Start with vLLM’s defaults, measure your bottlenecks, and layer in advanced techniques as needed. The golden rule: measure first, optimize what matters.

---

Looking for more deep-dives on AI infrastructure? Check out our Infrastructure and Optimization topic pages.