The model isn't the moat
In this paper: Serving architecture establishes the foundation, determining how computation, memory, and communication are organized for the workload.
Most discussions about deploying large language models (LLMs) begin with model selection. Production deployments rarely fail because the wrong model was selected. More often, they fail because the serving architecture wasn't designed for the workload it ultimately had to support.
Once a model moves beyond experimentation, success depends less on benchmark performance and more on how efficiently the inference system can serve that model. Latency service level objectives (SLOs), throughput, GPU utilization, scalability, and operational reliability become equally important.
Production architecture is where those engineering decisions begin.
EXL Engineering Field Guide: Engineering the Harness Series
Moving an LLM from proof of concept to production changes the engineering problem. Model accuracy still matters, but production success increasingly depends on the model’s serving harness: the architecture, memory management, scheduling, and optimization decisions that determine latency, throughput, GPU utilization, scalability, and cost.
This five-part field guide examines those engineering layers as one interconnected system and discusses the tradeoffs engineering teams must make to build LLM applications that perform reliably and economically at scale.
Explore the field guide
Moving an LLM from proof of concept to production changes the engineering problem. Model accuracy still matters, but production success increasingly depends on the model’s serving harness: the architecture, memory management, scheduling, and optimization decisions that determine latency, throughput, GPU utilization, scalability, and cost.
This five-part field guide examines those engineering layers as one interconnected system and discusses the tradeoffs engineering teams must make to build LLM applications that perform reliably and economically at scale.
Explore the field guide
Production changes the engineering problem
A proof of concept and a production deployment solve two very different problems. A proof of concept demonstrates that a model can perform a task. A production deployment must perform that same task consistently under real workloads while meeting service-level objectives and remaining economically viable.
Consider a 70B parameter model that performs exceptionally well during a proof of concept. With only a handful of users, latency is acceptable, GPU utilization isn't closely monitored, and infrastructure costs remain manageable.
Production changes those assumptions.
As request volume and concurrency increase, communication overhead becomes more consequential, GPU memory comes under pressure, latency becomes less predictable, and infrastructure costs begin to climb. The model hasn't changed. The engineering constraints have.
The inference system that performed well during experimentation is now being asked to satisfy production latency, concurrency, and cost objectives it was never designed to support. This is where many engineering organizations discover that deploying an LLM is fundamentally different from evaluating one.
Serving architecture is about engineering decisions – not deployment
The serving layer is often viewed as a deployment concern. But in practice, it's a systems engineering discipline. It determines how model computation, memory, and communication are organized across available hardware before production traffic arrives. Those decisions influence nearly every production characteristic that follows, including latency, throughput, scalability, GPU utilization, and infrastructure efficiency.
The inference architecture answers a deceptively simple question: How should model computation, communication, and memory be organized so the system can satisfy the requirements of this workload? Every architectural decision that follows is an answer to that question.
The first architectural question isn't whether tensor parallelism is better than pipeline parallelism. It's much simpler. The question is, “What problem are you trying to solve?” A customer-facing assistant has very different serving requirements than a nightly document-processing pipeline. An interactive coding assistant has different latency objectives than an internal knowledge retrieval application. The workload determines the constraints, and the constraints should determine the architecture[WT2.1].
Workload characteristics also shape how inference should be distributed. A deployment dominated by long prompts has very different serving characteristics than one generating long responses. Prefill and decode place different demands on the hardware, which is why many production architectures increasingly separate them rather than treating inference as a single workload.
One consequence of a poorly matched deployment architecture is overprovisioning. Systems are often sized for peak demand, leaving expensive GPU resources underutilized during normal operation. Better architectures improve utilization before organizations add more hardware.
Every architectural decision creates downstream consequences
One of the most common misconceptions in production AI is that optimization techniques can be applied independently.
They can't.
The inference architecture establishes the foundation for every optimization that follows. How model computation is distributed affects communication between GPUs. Consider a deployment serving long-context requests across multiple GPUs. Decisions around parallelism and prefill/decode disaggregation influence communication overhead long before engineers begin tuning batching or KV-cache management. Depending on the workload, the architecture may use tensor, pipeline, sequence, or expert parallelism, along with gateway abstraction to preserve portability.
They also determine how effectively later optimization techniques can be applied.
- Quantization
- KV-cache management
- Continuous batching
- Speculative decoding
- FlashAttention
None of these techniques operate in isolation. Their effectiveness depends on the architectural decisions that came first. That's why organizations sometimes continue adding GPU capacity without seeing proportional performance improvements. Additional hardware cannot compensate for an inference system that wasn't designed for the workload it now serves. Architecture determines what optimization makes possible later.
Potential sidebar/pull quotes:
“Additional hardware cannot compensate for an inference system that wasn't designed for the workload it now serves.”
“Architecture determines what optimization makes possible later.”
There isn't a universally "best" serving architecture
Engineers often ask which serving architecture is best. The answer? It depends.
Sidebar/callout
Title: There is no “best” serving architecture
Optimize for the workload not the architecture, and anticipate the tradeoffs:
Lower latency ↔ infrastructure cost
Higher throughput ↔ queueing delays
Higher GPU utilization ↔ communication overhead
The right architecture begins with the workload, then accounts for the hardware, model, deployment environment, latency objectives, and operational constraints you're trying to optimize. Production engineering is rarely about maximizing a single metric. It's about balancing competing objectives. Reducing latency may increase infrastructure cost. Maximizing throughput may introduce additional queuing delays. Improving GPU utilization may increase communication overhead. The objective isn't to eliminate those tradeoffs. It's to make the right ones for the application you're building.
Good production architectures are designed not only for today's models but for tomorrow's. Separating the serving layer from model- and hardware-specific implementations allows teams to treat a model swap as a configuration and evaluation exercise rather than a platform redesign.
Think about production from the beginning
Most architectural decisions are made during experimentation, when traffic is low and infrastructure constraints are largely invisible. By the time production exposes those constraints, redesigning the inference system is significantly more difficult.
Serving architecture shouldn't be treated as an optimization exercise that begins after a proof of concept succeeds. It should be part of the original system design because it establishes the foundation for everything that follows – from memory management and batching strategies to inference optimization.
The earlier those architectural decisions are considered, the fewer engineering compromises are required later.
Looking ahead
The serving layer establishes the foundation for production inference. Once that foundation is in place, many systems encounter their next constraint in GPU memory rather than compute.
Serving architecture establishes the foundation, but architecture alone doesn't determine how efficiently the system operates. Many production systems encounter their next constraint in GPU memory long before compute is saturated. The next paper in this five-part Engineering the Harness white paper series examines how KV-cache growth, allocation, PagedAttention, and prefix caching affect concurrency, throughput, and infrastructure efficiency at scale. Read “Memory management: The bottleneck most teams hit too late.[WT”