Introduction
Getting a large language model to produce the correct output is no longer the biggest challenge for most AI teams. The real difficulty is making that output fast enough, affordable enough, and reliable enough for real-world applications.
A model may perform well in a controlled environment but struggle once request volume increases. Latency can rise as queues grow, while costs can quickly escalate when applications process larger batches, longer prompts, and increasingly complex workloads.
Inference optimization addresses this gap. The goal is to serve the same model to more users, faster and at lower cost, without changing the model’s training. These optimizations range from scheduling and memory-management techniques that require no changes to the model itself to architectural improvements that are incorporated directly into the model.
To understand why these techniques work, it helps first to understand what happens during LLM inference.
Understanding the Two Phases of LLM Inference
Every forward pass through a decoder-only LLM consists of two distinct phases: prefill and decode. Although they are part of the same inference process, they have very different performance characteristics.
The first phase is prefill. When a request arrives, the model processes the entire input sequence to calculate the intermediate key and value tensors needed for generation. Because all input tokens are already available, the model can process them in parallel. This makes prefill highly efficient at using GPU compute resources and means the phase is generally considered compute-bound.
The second phase is decode. Once the first output token has been generated, the model produces additional tokens one at a time. Each new token depends on the tokens that came before it, preventing the model from generating a single sequence fully in parallel.
This changes the performance bottleneck. During decode, the GPU often spends more time moving model data between memory and compute units than performing mathematical operations. As a result, GPU memory bandwidth becomes a critical constraint during generation.
This distinction explains why different inference metrics tell different stories. Time-to-first-token (TTFT) primarily reflects prefill performance, while tokens per second after the first token provides a better indication of decode performance. Improving one does not necessarily improve the other.
Understanding prefill and decode also provides the foundation for understanding the KV cache, batching, attention optimization, and other techniques used to accelerate inference.
KV Caching and the Memory Challenge
One of the most important optimizations for the decode phase is KV caching. Instead of recalculating the key and value tensors for every previous token each time the model generates a new token, the system stores these tensors in GPU memory and reuses them.
This approach trades memory consumption for computation. Rather than repeatedly performing the same calculations, the model can retrieve information that has already been computed.
The problem is that KV cache requirements grow with both sequence length and batch size. A 7-billion-parameter model operating at 16-bit precision can require several gigabytes of memory for its KV cache at moderate sequence lengths. As context windows become longer and more requests are processed simultaneously, memory can quickly become the primary constraint on concurrent serving capacity.
Traditional KV cache management can introduce another problem: memory fragmentation. If the system reserves a large contiguous memory region for every request based on the maximum possible sequence length, much of that memory may remain unused.
Attention addresses this problem by applying the concept of virtual memory paging to the KV cache. Instead of requiring one large contiguous allocation for every request, the cache is divided into smaller fixed-size blocks. These blocks can be allocated as needed and stored non-contiguously in memory.
As generation progresses, new blocks are added only when they are required. This significantly reduces wasted memory and allows serving systems to support larger batches and higher throughput on the same hardware. The technique is implemented by popular inference frameworks such as vLLM.
Prefix caching takes caching a step further. When multiple requests share the same beginning portion of a prompt, the KV cache associated with that shared prefix can be computed once and reused.
This can be especially useful in applications with long system prompts, shared instructions, or retrieval-augmented generation pipelines where many requests process information using the same context. Instead of repeatedly performing the same prefill work, the system can reuse previously computed states.
Developers looking to explore this area further can also refer to KV Caching in LLMs: A Guide for Developers and The Complete Guide to Inference Caching in LLMs.
Batching for Better GPU Utilization
Running one inference request at a time can leave significant GPU capacity unused. Model weights must be loaded into memory regardless of how many requests are being processed, so handling multiple sequences simultaneously can spread that cost across a larger workload.
The simplest approach is static batching, where the system waits for a fixed group of requests and processes them together. While this can improve GPU utilization, it introduces an important inefficiency: different requests typically require different numbers of output tokens.
If one request finishes quickly while another continues generating, the shorter request may effectively occupy a slot without doing useful work. The entire batch is constrained by its longest-running sequence.
Dynamic batching improves this approach by collecting requests that arrive within a short time window. The system can begin processing once it reaches a maximum batch size or a predefined timeout. This provides a balance between utilization and responsiveness. Shorter timeouts generally reduce queuing latency but may result in smaller batches, while longer timeouts can produce larger batches at the cost of additional waiting time.
However, dynamic batching still has a limitation because requests that enter the same batch generally continue progressing together.
Continuous batching, also known as in-flight batching, addresses this problem by allowing requests to enter and leave the batch independently. When one sequence finishes generating, it can immediately be removed and replaced with another waiting request instead of forcing the entire batch to wait.
This keeps the GPU busy even when requests have widely different output lengths. Continuous batching has therefore become a standard scheduling strategy in production inference systems such as vLLM and TensorRT-LLM.
For a deeper comparison, see Static vs. Dynamic vs. Continuous Batching in LLM Inference.
Optimizing the Attention Mechanism
Attention is one of the most computationally important components of a transformer, and several optimization techniques have been developed to reduce its memory and compute requirements.
Traditional multi-head attention (MHA) maintains separate key and value heads for each attention head. Multi-query attention (MQA) takes a different approach by allowing multiple query heads to share a single set of key and value heads.
The mathematical structure of the attention computation changes, but the major inference advantage comes from reducing the amount of key and value data that needs to be loaded from memory during decoding. Since decode is heavily constrained by memory bandwidth, reducing memory traffic can have a meaningful effect on performance.
The tradeoff is that MQA can slightly reduce model quality, and the model needs to be trained or fine-tuned with the architecture in place to fully benefit from it.
Grouped-query attention (GQA) provides a middle ground between MHA and MQA. Instead of giving every query head its own key and value heads or having all query heads share one pair, GQA divides the key and value heads into groups. Multiple query heads can then share each group.
This approach retains many of the memory-efficiency benefits of MQA while preserving more of the representational capacity associated with MHA.
Flash Attention uses a different strategy. Rather than changing the attention architecture, it changes how attention is calculated and how data moves through GPU memory.
Traditional attention implementations can repeatedly write intermediate results to slower GPU global memory. FlashAttention uses techniques such as tiling and operation fusion to keep more intermediate data in faster on-chip memory, significantly reducing memory traffic while maintaining the same mathematical result.
Because Flash Attention can be used without retraining the model, it is particularly attractive as a practical inference optimization.
For a visual explanation of these techniques, A Visual Guide to Attention Variants in Modern LLMs provides a useful overview.
Model Compression for Smaller and Faster Models
While inference optimization can make serving a model more efficient, model compression takes a different approach by reducing the amount of data that needs to be processed in the first place.
Quantization reduces the numerical precision used to represent model weights and, in some cases, activations. A model using 16-bit floating-point weights requires approximately two bytes per parameter. Reducing the representation to 8-bit precision cuts that requirement roughly in half, while 4-bit precision reduces it further.
For a 7-billion-parameter model, the difference between FP16 and INT4 can mean approximately 14 GB versus 3.5 GB of memory for the weights alone. That difference can determine whether a model fits on a particular GPU or whether a more expensive hardware configuration is required.
Techniques such as GPTQ and AWQ have made low-bit quantization increasingly practical, allowing large models to operate at lower precision while maintaining much of their original quality.
Sparsity provides another form of compression. Many trained neural networks contain weights that contribute relatively little to the final output. By pruning selected weights and setting them to zero, models can potentially use specialized sparse representations and computation.
Modern NVIDIA GPUs also provide hardware support for structured sparsity. For example, Ampere and later architectures support 2:4 structured sparsity, where two values in every group of four are zero. Under compatible conditions, this can provide substantial performance improvements without requiring the same level of software optimization associated with more general sparse computation.
Knowledge distillation takes another route by training a smaller student model to reproduce the behavior of a larger teacher model. Rather than simply removing information from the original model, the student learns from the teacher’s output distributions as part of the training process.
This can produce a smaller model that retains much of the behavior of its larger counterpart. For applications where latency and deployment cost are critical, distillation can be particularly useful because it creates a model specifically designed for the target deployment environment.
Speculative Decoding for Latency-Sensitive Applications
Autoregressive generation presents a fundamental limitation: each generated token depends on the token that came before it. This makes it difficult to parallelize generation within a single sequence.
Speculative decoding works around this limitation by using two models. A small, fast draft model generates several candidate tokens, while the larger target model evaluates those candidates in parallel.
The target model determines which proposed tokens it would have generated itself. Matching tokens are accepted, while generation resumes from the point where the draft and target models disagree.
The key advantage is that the target model can verify multiple candidate tokens in a single forward pass. When the smaller model has a high level of agreement with the larger model, several tokens can effectively be generated for the cost of a single target-model verification step.
Importantly, speculative decoding can preserve the exact output distribution of the target model when implemented correctly. Its biggest advantages are generally seen in latency-sensitive, single-request scenarios such as interactive applications, where traditional batching may not provide enough benefit.
In workloads that are already heavily batch-oriented and keeping the GPU fully utilized, the relative benefit can be smaller.
Scaling LLM Inference with Parallelism
As models grow larger, a single GPU may no longer have enough memory or compute capacity to handle them efficiently. Parallelism allows inference workloads to be distributed across multiple GPUs.
Tensor parallelism divides individual model layers across multiple devices. Weight matrices are partitioned so that each GPU performs a portion of the computation, after which the devices synchronize their results.
This approach can reduce the amount of model weight memory required on each GPU while distributing computation across the available hardware. It is particularly useful when the model cannot fit comfortably on a single device or when lower inference latency is required.
Pipeline parallelism takes a different approach by dividing the model vertically. Consecutive groups of layers are assigned to different GPUs, and activations move from one stage to the next.
The main challenge is the presence of pipeline bubbles. During certain periods, some GPUs may remain idle while waiting for work to arrive from another stage. Microbatching can reduce this inefficiency by keeping multiple batches in flight at the same time.
Pipeline parallelism is particularly relevant when serving very large models across multiple devices or when the workload provides enough concurrency to keep the pipeline occupied.
Another emerging approach is prefill-decode disaggregation. Instead of running both phases on the same pool of hardware, the system separates prefill and decode workloads and routes them to different groups of machines.
The reason is straightforward: prefill is primarily compute-intensive, while decode is heavily influenced by memory bandwidth. These workloads therefore have different hardware requirements.
Separating them allows infrastructure teams to optimize each stage independently. Long prefill operations can be handled by compute-optimized hardware without consuming capacity needed for ongoing decode workloads. As context windows continue to expand, this architectural approach is becoming increasingly relevant for large-scale inference systems.
Putting It All Together
LLM inference optimization is not a single technique. It is a collection of strategies designed to address different bottlenecks throughout the inference pipeline.
The first step is understanding whether the workload is limited by prefill or decode. From there, teams can select techniques that target the specific constraint. KV caching and PagedAttention address memory efficiency, while prefix caching can eliminate repeated computation. Continuous batching improves GPU utilization across variable workloads, while attention optimizations such as GQA and FlashAttention reduce memory traffic and computation.
Model compression techniques such as quantization, sparsity, and knowledge distillation reduce the resources required to run models. Speculative decoding can reduce latency in interactive workloads, while tensor and pipeline parallelism allow increasingly large models to run across multiple GPUs. Prefill-decode disaggregation goes further by separating fundamentally different workloads so each can be optimized independently.
The most effective inference systems typically combine several of these techniques rather than relying on a single optimization. The right combination depends on the model, hardware, workload characteristics, context length, concurrency, latency requirements, and cost targets.
As LLM applications continue to move from experimentation into production, inference optimization will become increasingly important. The ability to make models faster, more efficient, and more economical can be just as important as the capabilities of the models themselves.