Why production LLMs run out of memory before compute
One of the first surprises engineers encounter in production large language model (LLM) deployments is that GPUs often run out of memory before they run out of compute. When latency begins to increase, the first assumption is often that the GPUs have reached their limits. In some cases, they have. But in many cases, the system has simply run out of memory.
This is one of the more counterintuitive aspects of serving LLMs. During training, compute dominates the discussion. During inference, memory often becomes the limiting resource long before the GPUs are fully utilized.
That distinction matters because it changes where engineers should focus their optimization efforts. Adding more GPUs may increase available compute, but if memory remains the constraint, those additional resources often deliver far less improvement than expected.
Understanding why requires looking beyond the model itself and examining what happens during inference.
KV cache changes the economics of inference
Inference is fundamentally different from training. During autoregressive generation, the model doesn't recompute every previous token each time it generates a new one. Instead, it stores the intermediate key and value tensors in memory so they can be reused throughout the conversation.
That KV cache is what makes modern LLM inference practical. It is also one of the largest consumers of GPU memory. The scale of the problem becomes apparent quickly. For Llama 3 70B running in FP16, the KV cache consumes approximately 320 KB per token. A single 4,096-token request requires roughly 1.3 GB of GPU memory for the KV cache alone. As concurrent requests increase, memory consumption grows linearly – even though the model weights remain unchanged.
Unlike the model weights, which are fixed regardless of the number of users, the KV cache grows dynamically with every active request. Longer context windows require more memory. More concurrent users require more memory. As both increase, memory consumption grows even though the model never changes.
Consider a 70B parameter model serving requests with a 4,096-token context window. The model weights stay exactly the same whether one user is connected or one hundred. The KV cache does not. Every active request creates additional state that must remain resident in GPU memory throughout generation.
Production systems don't simply serve models. They serve models and the growing memory footprint those models create. Unlike model weights, the KV cache continues growing as context length and concurrency increase.
Why memory becomes the first production constraint
Memory pressure rarely appears during a proof of concept (POC). A handful of concurrent users seldom expose it. Production does.
As request volume increases, engineers often notice an unexpected pattern. GPU utilization stays lower than expected while concurrency plateaus and latency begins increasing. The instinct is to assume more compute is required.
More often, memory is the limiting factor. Traditional memory allocation makes this problem worse. Many inference systems reserve enough KV-cache memory for the maximum possible sequence length at the beginning of every request. If a request reserves memory for thousands of tokens but completes after only a few hundred, much of that memory remains allocated even though it is no longer being used efficiently.
The result is fragmentation. It's similar to disk fragmentation in a traditional operating system. Memory exists, but it can't be used efficiently because it has been allocated poorly.
Memory appears to be full even though much of it contains unused capacity. That limits concurrency, reduces effective batch sizes, and leaves expensive GPU compute waiting for memory rather than useful work. The bottleneck isn't computation. It's allocation.
Better allocation creates more capacity
One of the most important advances in production inference has been rethinking how the KV cache is managed. Instead of reserving one large contiguous block of memory for every request, modern serving systems increasingly divide the KV cache into smaller blocks that are allocated only when needed.
PagedAttention, popularized by vLLM, approaches the problem differently. Rather than allocating memory for the largest possible request, it allocates memory incrementally as sequences grow. When generation completes, those blocks become available for new requests without leaving large, fragmented regions behind. The impact extends beyond memory efficiency: Better memory utilization supports more concurrent requests; more concurrent requests enable larger and more efficient batches; and larger batches improve GPU utilization. The optimization begins with memory, but the benefits extend across the entire inference system. That's an important theme throughout production engineering. Optimizations rarely operate in isolation. That's because production inference is a systems engineering problem rather than a sequence of independent optimizations.
“Sometimes the fastest token to process is the one you never have to process again.”
Not every token needs to be processed again
Memory management isn't only about allocation. It's also about avoiding unnecessary work. Many production applications repeatedly process the same information: system prompts, few-shot examples, shared retrieval context, and system instructions. Without memory-aware caching, every request repeats the same prefill computation even though much of the input is identical.
Prefix caching approaches the problem differently. Instead of recomputing those shared prefixes, the serving system reuses the corresponding KV-cache entries across requests. Less computation is required. Time to first token improves. GPU resources are spent generating new information rather than reproducing work that has already been completed.
As context windows continue to grow, eliminating redundant computation becomes increasingly important. Sometimes the fastest token to process is the one you never have to process again.
Compression is another engineering decision
Memory can also be optimized by reducing the size of the KV cache. KV-cache quantization stores cached tensors using lower-precision formats such as INT8 or FP8, reducing the amount of GPU memory required for each active request. That additional capacity can often be converted directly into higher concurrency.
Like every optimization discussed throughout this series, though, it involves tradeoffs. Lower precision reduces memory consumption but may affect generation quality depending on the model and workload. The acceptable tradeoff depends on the workload. Some applications can tolerate lower precision with little measurable impact. Others cannot.
There isn't a universally correct answer.
The objective is to understand where memory is constraining the system and choose the optimization strategy that best fits the workload.
Memory management is a systems problem
It's tempting to think about PagedAttention, prefix caching, and KV-cache quantization as independent techniques. They aren't.
Each addresses a different aspect of the same engineering challenge.
- PagedAttention improves allocation.
- Prefix caching reduces redundant computation.
- KV-cache quantization increases effective capacity.
Together, they allow the inference system to support more users, improve GPU utilization, and reduce infrastructure cost without changing the underlying model. That's why memory management has become one of the highest-leverage engineering disciplines in production AI.
Organizations often assume they need more compute. More often, they need to manage memory more intelligently.
Looking ahead
Serving architecture establishes how an inference system is organized. Memory management determines how efficiently that system uses one of its most valuable resources. The next layer focuses on something different.
Efficient memory management creates capacity. The next engineering problem is deciding how to use that capacity. The next paper in this five-part Engineering the Harness white paper series examines batching as a scheduling problem: how continuous batching, chunked prefill, token-level scheduling, and admission control keep GPUs productive while maintaining application latency SLOs. Read “Batching strategies: Throughput vs. latency in practice.”