vLLM's approach exists because it optimizes for throughput under load, not startup time: PagedAttention: Memory-efficient attention that handles long sequences without OOM Continuous Batching: Dynamically adds new requests to in-flight batches, keeping GPUs saturated CUDA Graphs: Eliminates per-kern

„vLLM Startup Deep Dive: Why Loading a Model Takes 7 Minutes (and Why Ollama Is Instant)“

Q: Why Ollama Is Instant

Ollama (using llama.cpp) loads models in ~5-15 seconds. The architectural differences are fundamental: 1. Model Format: GGUF vs Safetensors GGUF is a single binary file designed for fast mmap() loading. The OS maps the file directly into memory without reading it through CPU buffers. Tensors are fau

Q: Docker Configuration Notes

Both containers use the same image (nvcr.io/nvidia/vllm:26.05-py3) with different Cmd arguments: # Censored vllm serve /models/Qwen3.6-35B-A3B-FP8 --served-model-name Qwen3.6-35B-FP8 --max-model-len 262144 --gpu-memory-utilization 0.85 --reasoning-parser qwen3 --default-chat-template-kwargs '{"

Management Summary

When switching models on our NVIDIA GB10 inference server, vLLM takes ~7 minutes to become ready. This feels absurd when the model is already downloaded to disk. Meanwhile, Ollama loads models in seconds. This post explains exactly what happens during those 7 minutes, why it’s not a bug, and why the two systems are architecturally incompatible approaches to the same problem.

Key takeaways:
– The bottleneck is CPU-side tensor deserialization (not disk I/O, not decompression)
– FP8 models do use less GPU memory — they stay FP8 in memory, no decompression to BF16
– vLLM trades startup time for much higher serving throughput (continuous batching, PagedAttention, CUDA graphs)
– Ollama trades serving throughput for instant startup (lazy mmap, pre-compiled kernels, no CUDA graphs)
– Both approaches are correct — they optimize for different use cases

The Setup

We run two vLLM containers on an NVIDIA DGX Spark (GB10):

Container	Model	Format	Disk Size	GPU Memory
`vllm_qwen`	Qwen3.6-35B-A3B (censored)	FP8	35 GB	34 GB
`vllm_uncensored`	Qwen3.6-35B-A3B (HauhauCS)	BF16	65 GB	65 GB

Both use the same Docker image (nvcr.io/nvidia/vllm:26.05-py3) and listen on port 8000. A switch script stops one container before starting the other. The models live on the GX10’s NVMe SSD and are mounted read-only into the containers.

What Happens During Those 7 Minutes

The startup pipeline has four distinct phases. Here’s the actual timing from our logs:

Phase 1: Weight Loading (~6 minutes)

Loading safetensors checkpoint shards: 100% | 42/42 [06:23<00:00, 9.12s/it]
Loading weights took 383.81 seconds
Model loading took 34.23 GiB memory and 388 seconds

The censored model has 42 shards of ~800MB each. Each shard takes ~9 seconds to process. The uncensored model has only 17 shards (but ~4GB each) and loads in ~5.5 minutes despite being nearly 2× larger.

The bottleneck is CPU-side deserialization, not disk I/O. Here’s what happens per shard:

Read from NVMe: ~800MB at ~3.5 GB/s = ~0.2 seconds. Negligible.
Parse safetensors header: Each shard has a JSON header describing tensor names, dtypes, and byte offsets. Fast.
Deserialize tensors: The CPU must validate shapes, convert byte offsets into properly-shaped PyTorch tensors, and allocate memory for each one. A single shard contains hundreds of individual tensors (weight matrices, bias vectors, etc.). This is pure CPU work on the GB10’s ARM cores.
Copy to GPU memory: Over NVLink-C2C, this is fast.

The GB10’s ARM CPU (Cortex-X925) is capable but not desktop-x86-fast for this type of sequential metadata processing. The 9 seconds per shard is the CPU parsing and allocating, not the disk or the GPU.

Why fewer, larger shards are faster: The uncensored model (17 × 4GB) loads faster than the censored model (42 × 800MB) because there are fewer open/parse/close cycles. Each shard has fixed overhead regardless of size.

Phase 2: torch.compile (~30 seconds on cached runs)

torch.compile took 117.72 s in total   (first run)
torch.compile took  28.02 s in total   (cached run)

PyTorch’s torch.compile traces the model’s computation graph and produces optimized, hardware-specific CUDA kernels. On first run, it must autotune — testing multiple kernel implementations per operation and benchmarking each. This takes ~118 seconds.

On subsequent starts, the compiled kernels are cached at /root/.cache/vllm/torch_compile_cache/ inside the container. Loading from cache takes ~28 seconds. This is why the second startup is faster — but only if you restart the same container (not recreate it).

Phase 3: CUDA Graph Capture (~28 seconds)

Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100% | 51/51
Capturing CUDA graphs (decode, FULL): 100% | 35/35
Graph capturing finished in 19 secs

CUDA graphs record sequences of GPU operations into a replayable „movie.“ Instead of submitting thousands of small GPU commands through the CPU (each with ~5μs launch overhead), vLLM records them once and replays the entire batch with one CPU call. This dramatically reduces CPU overhead during inference.

vLLM pre-captures graphs for 51 different batch sizes (1, 2, 4, 8, …, 512) so it has an optimized graph ready for any request size. This takes ~28 seconds and cannot be cached — the graphs depend on runtime memory addresses.

Phase 4: Auto-tuning (~5 seconds)

[AutoTuner]: Tuning trtllm::fused_moe::gemm1: 100% | 10/10
[AutoTuner]: Tuning trtllm::fused_moe::gemm2: 100% | 10/10

FlashInfer tests multiple implementations of fused MoE (Mixture of Experts) operations and picks the fastest for this specific GPU. Results are cached in ~/.cache/flashinfer/ and load instantly on subsequent starts.

Total Timeline

Phase	Time	Cacheable?
Weight loading	~390s	No
torch.compile	118s → 28s	Yes (inside container)
CUDA graph capture	~28s	No
Auto-tuning	~5s	Yes
Total	~7 min	~4 min cached

FP8 vs BF16: Does FP8 Actually Save Memory?

Yes. This was a point of confusion worth clarifying with hard numbers.

The censored model’s weights are stored on disk as FP8 (8-bit floating point). When loaded, they stay FP8 in GPU memory. The GPU’s Tensor Cores operate directly on FP8 data — no decompression to BF16 occurs.

Evidence from the logs:

dtype=torch.bfloat16, quantization=fp8

This means: compute dtype is BF16 for non-quantized operations (LayerNorm, etc.), but the weight storage format is FP8. A 35B parameter model at FP8 = ~35GB. At BF16 it would be ~70GB. The FP8 model uses half the memory.

	Disk	GPU Memory	Shards
Censored (FP8)	35 GB	34 GB	42 × 800MB
Uncensored (BF16)	65 GB	65 GB	17 × 4GB

FP8 is a quality tradeoff — 8-bit vs 16-bit means slightly less precision per weight. For most inference tasks the difference is negligible, and Blackwell’s FP8 Tensor Cores are designed to work natively with this format.

Why Ollama Is Instant

Ollama (using llama.cpp) loads models in ~5-15 seconds. The architectural differences are fundamental:

1. Model Format: GGUF vs Safetensors

GGUF is a single binary file designed for fast mmap() loading. The OS maps the file directly into memory without reading it through CPU buffers. Tensors are faulted in on-demand when the GPU first accesses them.

Safetensors stores tensors as raw byte ranges with a JSON header per shard. Each tensor must be individually parsed and copied by the CPU.

2. Lazy Loading vs Eager Loading

Ollama mmap’s the GGUF file and lets the OS page-fault weights in during the first forward pass. „Loading“ is nearly instant — just setting up page tables.

vLLM loads all weights eagerly (before serving) because it needs to pre-allocate the KV cache and CUDA graphs based on available memory.

3. Pre-compiled Kernels vs JIT Compilation

llama.cpp uses hand-written C++/CUDA kernels compiled at build time. Zero startup compilation.

vLLM uses torch.compile to JIT-compile kernels at runtime. 28-118 seconds depending on cache state.

4. No CUDA Graphs

llama.cpp doesn’t use CUDA graphs. No capture step needed.

vLLM captures ~86 CUDA graphs at startup. ~28 seconds.

Comparison Table

Operation	Ollama/llama.cpp	vLLM
Model format	GGUF (mmap)	Safetensors (parse each shard)
Weight loading	~5s (lazy mmap)	~6 min (eager CPU deserialization)
Kernel compilation	Pre-compiled C++	torch.compile (28-118s)
CUDA graphs	Not used	Capture (28s)
Total startup	~5-15 seconds	~7-8 minutes
Serving throughput	Lower	Much higher

The Tradeoff

vLLM’s approach exists because it optimizes for throughput under load, not startup time:

PagedAttention: Memory-efficient attention that handles long sequences without OOM
Continuous Batching: Dynamically adds new requests to in-flight batches, keeping GPUs saturated
CUDA Graphs: Eliminates per-kernel CPU launch overhead
FP8 Native Compute: Blackwell Tensor Cores operate directly on FP8 data

Ollama optimizes for fast startup and simplicity. It’s ideal for development, testing, and single-user scenarios. vLLM is designed for production serving with concurrent requests.

Both are correct. They solve different problems.

Can We Make vLLM Start Faster?

The only big win would be eliminating Phase 1 (weight loading). Two theoretical approaches:

1. Keep the container running. Don’t stop/start — just leave the model loaded. This is the simplest solution. The tradeoff: you can’t switch models without the 7-minute wait.

2. GPU memory checkpointing. Save the GPU memory state to disk and restore it on startup. This would require something like CRIU (Checkpoint/Restore in Userspace) for CUDA processes, which is experimental and not production-ready.

3. Pre-load into CPU memory. Load weights into CPU RAM on boot, then only do the GPU memcpy on container start. vLLM doesn’t support this mode today.

For now, the practical answer is: minimize container restarts. When you must switch models, expect 7 minutes. The torch.compile cache already helps on subsequent starts of the same container (28s instead of 118s for that phase).

Docker Configuration Notes

Both containers use the same image (nvcr.io/nvidia/vllm:26.05-py3) with different Cmd arguments:

# Censored
vllm serve /models/Qwen3.6-35B-A3B-FP8 
  --served-model-name Qwen3.6-35B-FP8 
  --max-model-len 262144 
  --gpu-memory-utilization 0.85 
  --reasoning-parser qwen3 
  --default-chat-template-kwargs '{"enable_thinking": false}' 
  --enable-auto-tool-choice --tool-call-parser qwen3_coder 
  --enable-prefix-caching

# Uncensored
vllm serve /models/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-bf16 
  --served-model-name Qwen3.6-35B-Uncensored-FP8 
  --max-model-len 262144 
  --gpu-memory-utilization 0.85 
  --reasoning-parser qwen3 
  --enable-auto-tool-choice --tool-call-parser qwen3_coder 
  --enable-prefix-caching

Key differences: the uncensored container omits --default-chat-template-kwargs (thinking is enabled), and the served model name differs. Both bind to host port 8000, so only one can run at a time.

Running vLLM on NVIDIA GBX Spark (GB10) with Qwen3.6-35B. Two containers, one port, seven minutes of patience.