LLM Serving at Scale: Architecture Patterns for Production

Published: July 2026 | Reading time: 14 minutes | Topic: AI Infrastructure

---

Introduction

Serving a single LLM is straightforward. Serving one reliably to millions of users at sub-second latency is one of the hardest distributed systems problems in modern software engineering. This post covers the architecture patterns that power production LLM services in 2026.

The Anatomy of a Production LLM Stack

A mature LLM serving architecture has five key layers:

1. Load Balancer / API Gateway: Rate limiting, auth, request routing
2. Request Scheduler: Queuing, prioritization, batching decisions
3. Prefill Cluster: GPU pool optimized for compute-heavy prompt processing
4. Decode Cluster: GPU pool optimized for memory-bandwidth-heavy token generation
5. KV-Cache Store: Distributed memory/cache for cross-request sharing

Pattern 1: Replicated Monolith

The simplest production pattern: each GPU instance holds a full model copy and handles requests end-to-end.

Client → Load Balancer → [GPU Node 1: full model]
                         [GPU Node 2: full model]
                         [GPU Node 3: full model]

When to use: Small models (<13B) that fit on a single GPU, low-to-medium traffic, rapid prototyping.

Pros: Simple, no inter-GPU communication, easy to scale horizontally.

Cons: Wasted duplication (each node stores full model), memory-bounded by single GPU.

Pattern 2: Tensor Parallelism Across Nodes

Split a single model layer across multiple GPUs. Each GPU holds a slice of each weight matrix, and communication (all-reduce) happens at each layer boundary.

# Tensor parallel degree = 4
# Each layer's weight matrix split column-wise across 4 GPUs
# After each layer: all-reduce to synchronize activations

GPU 0: W[:, 0:dim/4]    GPU 2: W[:, dim/2:3*dim/4]
GPU 1: W[:, dim/4:dim/2] GPU 3: W[:, 3*dim/4:dim]
       ↓ all-reduce ↓
    Full activation output

When to use: Models too large for a single GPU (70B+ on 16GB GPUs), latency-sensitive workloads.

Tradeoff: High communication overhead (NVLink helps enormously). Very inefficient for decode (small batch = lots of comms, little compute).

Pattern 3: Pipeline Parallelism

Split the model vertically: different GPUs handle different layers. Each GPU processes all tokens through its assigned layers, then passes activations to the next.

# Pipeline parallel degree = 4
GPU 0: Layers 0-19    GPU 1: Layers 20-39
GPU 2: Layers 40-59   GPU 3: Layers 60-79

# Challenge: "pipeline bubbles" — idle time while waiting for previous stage
# Solution: micro-batching (split a batch into micro-batches to fill the pipeline)

When to use: Very large models (405B+), inter-node communication with lower bandwidth than NVLink.

Pattern 4: Disaggregated Prefill-Decode

This is the most important pattern of 2025-2026. Since prefill and decode have opposite computational profiles, they should run on differently optimized hardware.

# Prefill nodes: H100 GPUs (compute-optimized)
Prefill Scheduler → [H100 × N] → produces KV-cache

# KV-cache transfer via RDMA/NVLink
KV-Cache Transfer ─────────────────────→

# Decode nodes: L40S or A10G (cost-optimized, high memory bandwidth)
Decode Scheduler → [L40S × M] → generates tokens

Key insight from the Splitwise and DistServe papers: separating prefill and decode can improve throughput by 2-4x because each phase can be independently optimized.

Pattern 5: Expert Parallelism (for MoE)

Mixture-of-Experts models (DeepSeek, Mixtral, GPT-4) use a different parallelism strategy: each expert lives on a different GPU, and a router sends tokens to the right expert.

# DeepSeek-V3: 256 experts, 8 active per token
# Expert parallelism: each GPU hosts several experts
# All-to-all communication after router selects experts

Token Batch → Router (top-k=8) → Expert GPU 0, 3, 7, 12, 15, 22, 31, 44
                                              ↓
                                    All-to-Aall Communication
                                              ↓
                                    Weighted Combine → Output

Request Scheduling: The Hidden Complexity

A production scheduler must balance five competing objectives:

• Throughput: Maximize tokens/second across all GPUs
• Latency: Meet P99 latency SLAs (often <100ms for first token)
• Fairness: Prevent any single user from monopolizing resources
• Priority: VIP/production traffic gets precedence over experiments
• Preemption: Know when to pause low-priority requests for high-priority ones

vLLM’s default scheduler uses a FCFS (first-come-first-served) policy with preemption. More sophisticated systems like Sarathi-Serve implement chunked prefill scheduling that reduces prefill-decode interference.

Autoscaling LLM Services

Autoscaling LLM serving is fundamentally different from web services:

# Bad metric: CPU/GPU utilization (decode is memory-bound, GPU "util" misleads)
# Good metrics:
#   - Time to First Token (TTFT) P95
#   - Time Per Output Token (TPOT) P95
#   - KV-cache memory utilization
#   - Queue depth / waiting requests

# Scaling rules:
#   TTFT P95 > 200ms → scale up prefill nodes
#   TPOT P95 > 50ms → scale up decode nodes
#   KV-cache > 80% → scale up total GPU pool
#   Queue depth = 0, GPU idle > 30% → scale down

Multi-Region Deployment

For global LLM services, deploy model replicas in multiple regions:

• Data sovereignty: Some jurisdictions require data to stay local (EU AI Act, GDPR)
• Latency: Users get lower TTFT from nearby regions
• Failover: If one region goes down, traffic routes to the next closest

The KV-cache makes failover tricky: in-flight requests lose their cache state when a node fails. Solutions include replicating KV-cache across racks and using prefill-agnostic serving architectures.

Recommended Production Stack (2026)

Component	Recommended Tooling
Serving engine	vLLM or TensorRT-LLM
Orchestration	Kubernetes + KServe or NVIDIA Triton
Load balancing	Envoy or NGINX with custom LLM-aware routing
Monitoring	Prometheus + Grafana (track TTFT, TPOT, cache hit rate)
Autoscaling	Custom KEDA scaler on TTFT/TPOT metrics
Model registry	HuggingFace Hub or internal S3 + metadata
A/B testing	Route fraction of traffic to canary model versions

Conclusion

Production LLM serving in 2026 requires careful co-design of hardware topology, parallelism strategy, request scheduling, and autoscaling. The disaggregated prefill-decode pattern is becoming the gold standard for high-throughput deployments, while tensor parallelism remains essential for single-request latency on massive models.

Start simple (replicated monolith on vLLM), profile your bottlenecks, and evolve your architecture as scale demands. The best architecture is the one that meets your SLA at the lowest cost.

---

Part of our AI Infrastructure series. Also read: AI Inference Optimization and KV-Cache Optimization.