Why production LLM systems are really scheduling systems
In this paper: Batching turns available GPU capacity into useful work by negotiating throughput and latency against application SLOs.
Production inference is fundamentally a scheduling problem. Once serving architecture and memory have been addressed, the next question isn’t whether the GPU can perform the work; it's how the work should be scheduled.
That decision begins with understanding the workload: how requests arrive, how long they run, which are latency-sensitive, and which can tolerate delay.
This is where many production deployments encounter their next engineering challenge. Serving architecture establishes how an inference system is organized. Memory management determines how efficiently it uses GPU memory. Batching answers a different question:
How do you keep expensive GPUs doing useful work while still meeting latency service-level objectives (SLOs)?
Production workloads introduce competing objectives that don't exist during experimentation. Production LLM systems serve requests that arrive unpredictably, vary dramatically in length, and compete for the same hardware resources. The engineering challenge isn’t batching requests together; it's deciding how work should be scheduled as those requests enter and leave the system.
EXL Engineering Field Guide: Engineering the Harness Series
Moving an LLM from proof of concept to production changes the engineering problem. Model accuracy still matters, but production success increasingly depends on the model’s serving harness: the architecture, memory management, scheduling, and optimization decisions that determine latency, throughput, GPU utilization, scalability, and cost.
This five-part field guide examines those engineering layers as one interconnected system and discusses the tradeoffs engineering teams must make to build LLM applications that perform reliably and economically at scale.
Explore the field guide
Moving an LLM from proof of concept to production changes the engineering problem. Model accuracy still matters, but production success increasingly depends on the model’s serving harness: the architecture, memory management, scheduling, and optimization decisions that determine latency, throughput, GPU utilization, scalability, and cost.
This five-part field guide examines those engineering layers as one interconnected system and discusses the tradeoffs engineering teams must make to build LLM applications that perform reliably and economically at scale.
Explore the field guide
Production traffic is inherently uneven
Most proof-of-concept environments present an idealized view of inference. Requests arrive at predictable intervals. Prompt lengths are similar. Response lengths are relatively consistent.
Production looks nothing like that. Prompt lengths, output lengths, arrival patterns, and latency expectations differ. But production schedulers have to account for all of them simultaneously. That heterogeneity is what transforms batching from a simple grouping exercise into a scheduling problem.
One user may ask a simple question that requires only a few tokens. Another may submit a lengthy document that requires a large context window and an extended response. Some requests are interactive and latency-sensitive. Others are background jobs that can tolerate additional delay. Treating all of those requests as if they were identical, creates unnecessary inefficiencies. That is why batching strategies that work well during experimentation often become liabilities in production.
Static batching breaks under real workloads
Static batching is straightforward: Collect a group of requests, execute them together, and wait until the batch finishes before beginning the next one. That approach works well for many traditional AI workloads because inputs generally require a similar amount of computation.
Autoregressive generation behaves differently. Each request generates tokens at its own pace. Some complete almost immediately. Others continue generating for hundreds or even thousands of decoding iterations. The result is that the shortest requests spend much of their time waiting for the longest one to finish.
This is the convoy effect. The GPU is effectively synchronized to the slowest request in the batch. The GPU is occupied, but completed or short-running requests cannot release their resources until the longest request in the batch finishes.
As production workloads become more diverse, those idle cycles accumulate. Throughput falls, latency increases, and expensive GPU resources spend more time waiting than performing useful work.
Bigger batches are not always better
The natural response is often to increase batch size. Larger batches generally improve GPU utilization, but they also introduce a tradeoff. Every request must wait until enough additional work arrives to fill the batch, increasing queueing delay before execution even begins.
Once execution starts, larger batches can also increase Time Per Output Token (TPOT). A configuration that delivers acceptable TPOT at one batch size may exceed the application's latency SLO as the batch grows. Higher GPU utilization provides little benefit if response times no longer meet application requirements.
This is the central balancing act of production inference. Higher throughput often comes at the cost of higher latency. The engineering challenge is determining where that operating point belongs.
An internal document-processing pipeline may prioritize maximum throughput because latency is relatively unimportant. A customer-facing assistant may willingly sacrifice some throughput to preserve fast responses. Neither choice is universally correct. The workload determines the objective. The batching strategy should support it.
Continuous batching changes the scheduling model
Modern serving systems approach the problem differently. Rather than waiting for an entire batch to complete, continuous batching schedules work at each decode iteration. Finished requests leave immediately. New requests enter immediately. The scheduler is no longer managing batches. It is managing decoding iterations.
Requests enter the batch as capacity becomes available. Completed requests leave immediately. New requests replace them without waiting for the entire batch to finish. This seemingly simple change fundamentally alters GPU utilization.
Instead of waiting for the longest request to complete, the scheduler continuously replenishes the batch with new work. GPU utilization becomes constrained primarily by queue depth rather than by the slowest request in the system. The scheduler is no longer waiting for an entire batch to complete. It's continuously looking for the next unit of useful work.
The objective shifts from building larger batches to keeping the GPU continuously productive.
Long prompts introduce another challenge
The prefill phase processes the entire prompt in a single forward pass, while the decode phase generates one token at a time. These phases place very different demands on the hardware. Prefill is compute-bound. Decode is memory-bandwidth-bound. Long prompts can monopolize GPU compute, delaying interactive requests that are already in the decode phase.
Chunked prefill addresses this by breaking large prompts into smaller segments that can be interleaved with ongoing decode operations. Rather than allowing one long prefill operation to dominate the GPU, the resource scheduler alternates between prompt processing and token generation, allowing engineers to balance Time to First Token (TTFT) for new requests against Time Per Output Token (TPOT) for active ones.
The result is a more balanced system.
Time to First Token (TTFT) for new requests remains reasonable while Time Per Output Token (TPOT) for active requests stays within acceptable service-level objectives. Once again, the objective is balance – not maximization.
Pull quote
“The objective isn't maximum throughput. It's the highest sustainable throughput that still satisfies the application's latency SLOs.”
Scheduling becomes policy
As production systems mature, batching becomes less about grouping requests and more about deciding which work should execute next.
- Interactive applications
- Background batch jobs
- High-priority users
- Low-priority workloads
GPU capacity ≠ Latency headroom
Each may require different treatment. Some workloads don't require interactive latency at all. If a business process can tolerate scheduled execution, maximizing throughput may be the better engineering decision than minimizing response time.
Modern serving systems increasingly make resource scheduling decisions at the token level rather than at the request level. GPU time becomes a shared resource allocated according to policy instead of simple arrival order.
That policy also extends to admission control. Available GPU capacity is not the same thing as available latency headroom.
Accepting every incoming request may maximize utilization for a short period, but it can also push latency beyond acceptable SLOs for every user in the system. Well-designed serving systems recognize that preserving predictable latency is often more valuable than accepting one additional request. In production, saying "not yet" is frequently better than saying "yes" to everyone.
Throughput and latency are negotiated – not maximized
One theme runs consistently through every batching strategy. There is no universally optimal configuration. The appropriate batch size, scheduling policy, and admission thresholds all depend on the workload and the service-level objectives the application must satisfy. There is no universally optimal configuration because every production deployment is balancing a different set of constraints.
A nightly batch pipeline should not be engineered like an interactive coding assistant. Likewise, a customer support chatbot should not inherit the scheduling strategy of an offline document-processing system simply because both happen to use the same foundation model.
The model may remain unchanged. The serving behavior should not. The objective isn't maximum throughput. It's the highest sustainable throughput that still satisfies the application's latency SLOs.
Looking ahead
Serving architecture organizes the inference system. Memory management ensures that GPU memory is used efficiently. Batching strategies determine how available resources are converted into useful work.
Serving architecture organizes the system. Memory management creates usable capacity. Batching determines how effectively that capacity is converted into useful work.
The final paper in the five-part Engineering the Harness white paper series examines inference optimization: how techniques such as quantization, speculative decoding, Flash Attention, and distillation can remove unnecessary work and improve production economics once the underlying serving system is sound. Read “Inference optimization: What it takes to go from pilot to production.