Inference optimization

TL;DR

• Two latency metrics define LLM UX: TTFT (Time to First Token, target under 500ms) and TBT (Time Between Tokens, target under 50ms for smooth streaming).
• KV caching stores attention key-value tensors so each new token only computes against cached context, not the full sequence from scratch.
• Continuous batching increases GPU utilization 20-30x over naive static batching by inserting new requests as old ones finish.
• vLLM's PagedAttention manages KV cache memory like an OS manages virtual memory pages, eliminating 60-80% memory waste from pre-allocation.
• Speculative decoding uses a small draft model to propose tokens verified in parallel by the large model, achieving 2-3x decode speedup.
• KV caching and continuous batching are non-negotiable for production. Everything else is optimization on top.

---

The problem it solves

Your chatbot works great in development. Single-user latency is acceptable, the model fits in VRAM, and responses stream smoothly. Then you launch to 1,000 concurrent users and everything breaks.

A 70B model generates tokens one at a time. Each token requires a full forward pass through all 80 layers, reading tens of gigabytes of weight data from GPU memory. If you serve requests sequentially, the GPU spends most of its time waiting for memory reads, not computing. Utilization sits at 5-10%. You're paying $3/hour for an A100 that's 90% idle.

I've seen teams hit this wall within days of launching an LLM-powered product. The math is brutal: a 70B model doing naive single-request inference produces maybe 30 tokens/second. At 50 tokens per response, that's 0.6 requests per second. For 100 concurrent users generating simultaneously, you'd need roughly 170 GPUs. At $3/hour per GPU, that's $370,000/month.

The root cause isn't compute speed. Modern GPUs are enormously powerful. The problem is utilization: naive inference wastes most of that power on memory stalls and idle cycles. Inference optimization is the set of techniques that close this gap.

---

What is it?

Inference optimization is the collection of techniques that reduce latency, increase throughput, and lower cost when serving LLMs in production. The techniques compose: a modern serving stack like vLLM applies KV caching, continuous batching, PagedAttention, and prefix caching simultaneously. Understanding each one separately is the prerequisite to reasoning about tradeoffs.

Think of it like a restaurant kitchen. A raw LLM is a chef who cooks one dish start-to-finish before taking the next order (sequential inference). Inference optimization is everything that turns that into a real kitchen: prep stations (KV cache), taking new orders as plates go out (continuous batching), efficient shelf space for ingredients (PagedAttention), and a sous chef who preps dishes the head chef just has to approve (speculative decoding).

Two metrics anchor every optimization decision. If an interviewer asks about LLM serving, start with these.

---

How it works

Latency metrics: TTFT and TBT

Time to First Token (TTFT) measures how long after sending a request the user sees the first token appear. This is the "thinking" delay. For interactive chat, TTFT above 2 seconds feels broken. Competitive production targets are under 500ms.

Time Between Tokens (TBT) is the interval between each streamed token. Human reading speed is roughly 250 words per minute. At 1.3 tokens per word, that's about 5.4 tokens/second, or ~185ms between tokens. Anything under 50ms TBT looks instantaneous to users. Above 100ms, streaming starts to feel choppy.

These metrics create different optimization pressures. TTFT is dominated by the prefill phase (processing the entire input prompt at once). TBT is dominated by the decode phase (generating one token at a time). Some optimizations help one metric but hurt the other.

For your interview: TTFT and TBT are the first thing you name when discussing LLM serving. They show you understand the user experience side, not just the infrastructure side.

KV cache

LLMs generate tokens autoregressively: each new token depends on all previous tokens. The transformer's attention mechanism computes query, key, and value vectors for the current token, then attends over the keys and values of all prior tokens. Without caching, you'd recompute K and V for tokens 1 through N-1 every time you generate token N.

KV cache stores the key and value tensors from every prior attention computation. When generating token N, the model reads cached K/V for tokens 1 through N-1 and only computes new K/V for token N. The computation per decode step drops from processing the full sequence to processing a single token.

The tradeoff is memory. For a 70B model with 80 layers, each layer stores K and V tensors sized proportionally to the sequence length and hidden dimension. A single 128K-context request can consume 50-100GB of KV cache. This is why KV cache management is the central challenge of LLM serving, and why PagedAttention was such a breakthrough.

Continuous batching

Static batching groups multiple requests together and processes them as a single batch. Every request in the batch must wait for the longest one to finish before any results are returned. If request A generates 10 tokens and request B generates 500, request A's response is held until B completes.

Continuous batching (also called iteration-level batching) solves this by checking for completed requests at every decode step. When a request emits its end-of-sequence token, it exits the batch immediately, and a waiting request takes its slot.

The throughput increase is dramatic. Static batching on a 70B model might achieve 200-400 tokens/second total. With continuous batching, the same GPU can push 2,000-4,000 tokens/second because slots never sit idle waiting for the longest request.

I've seen production systems go from needing 40 GPUs to needing 4 just by switching to continuous batching. It's the single highest-impact optimization after KV caching itself.

PagedAttention

Even with KV caching and continuous batching, there's a hidden waste problem: memory fragmentation. Traditional KV cache implementations pre-allocate a contiguous memory block for the maximum possible sequence length per request. If you set max_seq_len to 4096 but most responses are 200 tokens, you're wasting 95% of the allocated KV cache memory.

PagedAttention (introduced by vLLM) borrows the virtual memory concept from operating systems. Instead of pre-allocating contiguous blocks, it divides KV cache into small fixed-size pages (typically 16 tokens each). Pages are allocated on demand as the sequence grows, and freed immediately when the request finishes.

This eliminates two problems at once. Memory fragmentation disappears because pages don't need to be contiguous. Memory waste drops because you only allocate what's actually used. The vLLM paper showed that PagedAttention reduces KV cache memory waste from 60-80% to near zero, effectively allowing 2-4x more concurrent requests on the same hardware.

Speculative decoding

The decode phase is memory-bandwidth-bound: the GPU reads the entire model for each token but only does a tiny amount of compute. Speculative decoding exploits this by using a small, fast draft model (e.g., a 1B model) to generate several candidate tokens quickly. Then the large target model verifies all candidates in a single forward pass.

Here's the key insight: verification is parallelizable. The target model can check 5-10 draft tokens in roughly the same time it takes to generate one token, because the bottleneck is reading the model weights (which happens once regardless of how many tokens you verify). If the draft model's predictions match, you've generated 5-10 tokens for the cost of one target model forward pass plus one cheap draft model pass.

In practice, the draft model agrees with the target model 70-90% of the time for typical text. When it disagrees, you fall back to the target model's token and continue. The net speedup is typically 2-3x on long outputs, with zero quality loss because the final token distribution is mathematically identical to the target model alone.

The catch: speculative decoding helps decode latency (TBT) but doesn't help prefill (TTFT). It also adds complexity and requires maintaining two models in memory. For short outputs (under 20 tokens), the overhead may not be worth it.

Prefix caching

Most production LLM deployments use a system prompt: instructions, persona, tool definitions, few-shot examples. This prompt is identical across all requests. Without prefix caching, every request re-processes the system prompt from scratch during prefill.

Prefix caching stores the KV cache state for shared prompt prefixes. When a new request arrives with the same prefix, the system loads the cached KV state and only processes the user-specific portion. For a 2,000-token system prompt with a 200-token user query, this skips 90% of the prefill work.