# Inference Engineering — Full Text Corpus This file contains the full text of all chapters and learn articles from Inference Engineering (https://inferenceengineering.tech). Generated for AI engine consumption. Canonical URLs are listed per section. Author: Philip Kiely | Publisher: Baseten Books Content © Baseten Books. Citations welcome; please link to the source URL. ================================================================================ # Preface: Why Inference Engineering Matters URL: https://inferenceengineering.tech/chapters/preface/ Inference is the most valuable category in the AI industry. On November 30, 2022 — the day ChatGPT launched — there were perhaps a few hundred inference engineers in the world, though they didn't call themselves that at the time. These specialists mostly worked at frontier labs like OpenAI, Midjourney, and Anthropic or big tech companies like Google and NVIDIA. Three years later, a **Cambrian explosion of open models** — more than two million and counting on Hugging Face — means that every engineer can now deploy their own intelligence to power their AI products. **Open models** have publicly available weights (e.g., Llama, DeepSeek). **Closed models** keep weights proprietary (e.g., GPT-5, Claude). Until December 2024, there was a meaningful gap in intelligence — when DeepSeek V3 and R1 were released, that gap disappeared. ## Why Open Models Matter Even if open models are constantly chasing closed models on benchmarks, they change the equation for AI product builders. Switching to open models unlocks the opportunity to use inference engineering to make AI products better in new dimensions: - **Latency**: Closed model APIs are built for throughput, but open models can be optimized for real-time applications - **Availability**: While APIs for GPT and Claude are stuck at two nines of uptime, four nines or better is possible with dedicated deployments - **Cost**: Open models are often at least 80 percent less expensive at scale ## Who Needs Inference Engineering AI-native startups like Cursor, Clay, Gamma, and Mercor are redefining hypergrowth building products that rely on open and in-house models. Leading digital native companies like Notion and Superhuman are thriving by deeply integrating AI capabilities. A new generation of blended research and engineering teams — World Labs, Writer, Mirage, and dozens of others — are building enormous businesses by training and productizing their own foundation models. **You are early.** While the potential and impact of inference are becoming clear, the space is young. There are relatively few people working on inference, and newcomers can become experts quickly. There are enormous opportunities to solve novel, interesting, and deeply technical problems at all levels of the stack. *Inference Engineering* is your guide to becoming an expert in inference — from GPU hardware and CUDA kernels to production autoscaling. Welcome to the early days of inference. --- ================================================================================ # Chapter 0: Inference — The Three Layers URL: https://inferenceengineering.tech/chapters/inference/ This chapter is your orientation hub for the full book. If you came here searching for "inference engineering" broadly, you're in the right place — continue reading to understand the three layers (runtime, infrastructure, tooling) before diving into hardware, software, or techniques. The [homepage](/) also provides a structured entry point to all chapters and interactive exercises. **Inference** is the second phase in a generative AI model's lifecycle. While *training* is the process of learning model weights from data, *inference* is serving generative AI models in production. ## The Inference Lifecycle In last decade's machine learning (ML) boom, hundreds of thousands of data scientists and ML engineers became familiar with the full lifecycle — both training and inference — for ML models. Inference for classic ML models is relatively straightforward. In the early days of Baseten, inference ran on models built with tools like XGBoost on lightweight CPUs with a simple software stack. In contrast, **inference for generative AI models is complex**. You can't simply take model weights, get some GPUs, and expect inference to be fast and reliable enough for large-scale production use. ## The Three Layers Doing inference well requires three layers: - **Runtime**: Optimizing the performance of a single model on a single GPU-backed instance - **Infrastructure**: Scaling across clusters, regions, and clouds without creating silos while maintaining excellent uptime - **Tooling**: Providing engineers working on inference with the right level of abstraction to balance control with productivity These three layers must work together to create a system that can handle mission-critical inference at scale. A complete inference platform requires all three layers — runtime, infrastructure, and tooling — working in concert. Optimizing just one layer isn't enough for production-grade inference. ## Runtime The runtime layer is responsible for ensuring that an individual model running on a GPU (or across several GPUs in a single instance) is running as performantly and efficiently as possible. This layer depends on a sophisticated software stack, from CUDA to PyTorch to inference engines like vLLM, SGLang, and TensorRT-LLM. Low-level optimization is important, with kernels like FlashAttention delivering significant performance gains. ### The Software Stack The software stack for inference includes multiple layers of abstraction: 1. **CUDA** — Low-level GPU programming framework 2. **PyTorch** — Deep learning framework for model execution 3. **Inference Engines** — vLLM, SGLang, TensorRT-LLM for optimized serving 4. **FlashAttention** — Optimized attention kernels ### Model Performance Techniques The runtime layer relies on a number of model performance techniques that apply new research to the unique challenges of inference: - **Batching**: Run incoming requests in parallel, weaving them together on a token-by-token basis to increase throughput - **Caching**: Re-use the KV cache — the cached results of the attention algorithm — between requests that share prefixes - **Quantization**: Lower the precision of select pieces of the model to access more compute and reduce memory burden - **Speculation**: Generate and validate draft tokens to produce more than one token per forward pass during decode - **Parallelism**: Efficiently leverage more than one GPU to accelerate large models without introducing new bottlenecks - **Disaggregation**: Separate the two phases of LLM inference, prefill and decode, onto independently scaling workers These model performance techniques are used for models of *all modalities*, not just LLMs. Vision language models, embedding models, ASR, speech synthesis, image generation, and video generation all require their own inference optimizations. ## Infrastructure These runtime optimizations are not enough. No matter how performant a single instance of a model server is, it will eventually receive more traffic than it can handle. This is not a CUDA problem or a PyTorch problem. This is a **systems problem** that needs to be solved at the infrastructure layer. The nature of infrastructure problems changes with each level of scale: 1. **Autoscaling** — Knowing when to add and remove replicas and doing so quickly 2. **Capacity Management** — Past a few hundred GPUs, spreading workloads across multiple regions and cloud providers 3. **Global Unification** — Treating all available resources as a single unified pool of compute, eliminating silos Thoughtful multi-cloud infrastructure also improves **reliability**, protecting against downtime in any individual region or cloud provider. For global applications, running inference near end users improves end-to-end latency. ## Tooling & Developer Experience Once runtime and infrastructure capabilities are built, they need to be presented at the appropriate level of abstraction. Both inference providers like Baseten and internal teams building inference need to consider what tooling and developer experience to provide. Developer experience for inference exists on a spectrum: - **Black box** — Give a platform model weights, get back an API - **Raw primitives** — Providing only basic constructs for compute, network, disk The right developer experience is somewhere in the middle, where inference engineers have enough control to run mission-critical inference confidently, but enough abstraction to work productively. ## Book Roadmap *Inference Engineering* presents a map of the technologies and techniques that power inference across all three layers of runtime, infrastructure, and tooling: | Chapter | Topic | Focus | |---------|-------|-------| | **Ch 1** | Prerequisites | Use case definition, latency budgets, model selection | | **Ch 2** | Models | LLM & diffusion architecture, attention, bottlenecks | | **Ch 3** | Hardware | GPU architecture, NVIDIA generations, accelerators | | **Ch 4** | Software | CUDA, PyTorch, inference engines, benchmarking | | **Ch 5** | Techniques | Quantization, speculation, caching, parallelism | | **Ch 6** | Modalities | VLMs, ASR, TTS, image & video generation | | **Ch 7** | Production | Autoscaling, multi-cloud, deployment, observability | --- ================================================================================ # Chapter 1: Prerequisites — Before You Optimize URL: https://inferenceengineering.tech/chapters/prerequisites/ Before diving into inference engineering, you need clear answers to five questions: Which model(s)? What interface? What latency budget? What unit economics? What usage patterns? Early on in building an AI product, the answers to these questions may not be clear. At this early stage, it's often better to use off-the-shelf APIs whenever possible rather than investing in dedicated inference. As the product scales, requirements become clear and inference engineering becomes a worthwhile pursuit. ## Scale and Specialization There are two ways to add AI models to your product: - **Shared inference**: Send your traffic to a public API endpoint for a given model and pay per million tokens or some other consumption-based metric. - **Dedicated deployments**: Rent GPUs and set up an inference service exclusively for your application, paying per hour of GPU time. Shared versus dedicated inference is not exactly the same conversation as closed versus open models — there are plenty of shared endpoints for open models and many model labs offer large customers some kind of dedicated setup for their closed models. However, one of the key motivations for adopting open models is that it unlocks unrestricted dedicated inference. Most AI products start with pay-per-token APIs because the tradeoffs make sense while looking for product-market fit. | | Pros | Cons | |---|---|---| | **Shared** | Zero overhead, only pay for consumption | Cost scales linearly with usage | | | No cold start times, model always available | Provider uptime caps product uptime | | | Minimal engineering, just need an API key | No control over latency or rate limits | Over time, shifting to dedicated deployments becomes essential for three reasons: 1. **Scale**: You are processing enough volume that it's more economical to pay per GPU than per million tokens. 2. **Specialization**: You are running a custom or fine-tuned model, or you have specific latency or uptime requirements. 3. **Orchestration**: Your product relies on multiple models and multi-step pipelines and you need to minimize network latency and deployment complexity. The switch to dedicated deployments puts you in charge of your own inference engineering. Only switch once there is a clear and immediate business need — it adds engineering surface area and increases the floor of your monthly spend. ## About Your App Every inference engineering decision you make will be downstream of your use case. There are two cases where inference engineers need to build at the highest level of generality: - **Foundation models**: You have trained your own model and will sell consumption directly via a public shared inference API. - **Inference platforms**: You are building an inference platform, either internally or as a product, and need to support any model and any use case. But most AI-native applications are vertical apps like code editors or customer service agents. When building inference for vertical apps, you want to add as many constraints as possible by getting specific with your use case. ### AI-Native Applications Generative AI models have unlocked a new class of applications across industries and domains: | Category | Example | Considerations | |----------|---------|----------------| | **Agents** | Prospecting agent for sales teams | One user action triggers many inference calls | | **Chat** | Front-line customer support with RAG | Time to first token makes chat feel fast | | **Voice** | Real-time translation between languages | End-to-end latency for natural conversation | | **Media** | Virtual try-on for clothes and jewelry | Balance output quality vs. speed | | **Search** | Legal document discovery | Offline corpus filling vs. online user requests | | **RecSys** | E-commerce product recommendations | Consistent latency with high request volume | | **Completion** | Tab completion for coding in an IDE | Full completion chunk at user's typing speed | | **Moderation** | Scan user-generated content for safety | High throughput for cost-effective checks | ### Online versus Offline One of the primary tradeoffs in inference engineering is **latency versus throughput**. Lower latency makes your application faster, but higher throughput makes it cheaper at scale because you can use fewer GPUs for the same number of users. Most AI applications — code completion, chat, voice agents — are **online applications** that run in real time and should be optimized for latency. However, some applications have **offline batch inference** needs. Offline jobs are better served by high-throughput model deployments where each individual request would be too slow for a good user experience, but the system as a whole processes far more requests per hour. Example offline workloads include: - **Catalog transcription**: Transcribing a back catalog of podcasts or audio to make it searchable - **Document processing**: Embedding, converting, or analyzing documents on a regular cadence - **Corpus preparation**: Cleaning, embedding, or preparing massive corpora for model training It's possible to have a single model used for both online and offline jobs. Whisper could be used in both a real-time dictation app and a batch transcription job. If both use cases have enough volume, it's more cost effective to create two separate deployments — one optimized for latency, the other for throughput. ### Consumer versus B2B **Consumer applications** are generally more cost-sensitive with less predictable usage patterns. A single launch can drive a spike in usage overnight. Prioritize marginal cost and flexibility. **Business-to-business products** often have better margins and more stable usage but require high availability and consistently low latency. Prioritize latency and uptime. In both cases, compliance can limit infrastructure options — data sovereignty, user privacy, and regulatory compliance are essential considerations. ## Model Selection All else being equal — hardware, runtime, optimizations, architecture — inference on a smaller model with fewer parameters will be faster and cheaper than inference on a larger model. The most important decision in model performance optimization isn't the runtime engine or speculation algorithm — it's which model you choose to work with in the first place. Find or create the smallest, easiest-to-run model that's smart enough to handle the task. ### Model Evaluation Model evaluation, or **evals**, is the practice of systematically measuring model intelligence. High-conviction evals are a prerequisite for inference engineering: - **Spend time wisely**: Before investing in making a model fast, evals ensure the model is useful. - **Establish a baseline**: Some performance optimization techniques risk reducing model quality, requiring a baseline to compare against. Intelligence benchmarks like MMLU or SWE-bench are useful for shortlisting models, but they have become saturated. Goodhart's Law states that "when a measure becomes a target, it ceases to be a good measure." Tips for useful evaluation work: 1. **Look at your data**: Check eval results against your intuition for the product and problem space 2. **Be precise**: Focus evaluation on the hardest problems a model needs to solve 3. **Use tools**: Don't reinvent the wheel on one of the fundamental problems in AI engineering ### Fine-Tuning for Domain-Specific Quality **Fine-tuning** is the practice of taking a pre-trained foundation model and adapting it to a specific use case by introducing new data. If you can fine-tune a small model to pass your evals, you set yourself up for an easier time hitting your latency and cost targets. A great example is translating English into SQL — a tiny fine-tuned model of just a few billion parameters can reach equivalent performance to hundred-billion parameter general-purpose models on this specific task. ### Distillation **Distillation** is the process of using a large "teacher" model to train a smaller "student" model to emulate the larger model's behavior. Unlike fine-tuning on synthetic data, distillation shows the student the teacher's actual probability distributions, not just its final answers. In January 2025, DeepSeek released distilled versions of DeepSeek-R1 on top of Llama 3 and Qwen 2.5 architectures. These distilled models showed similar reasoning behavior with worse benchmark scores but were relatively small and could take advantage of existing performance work. ## Measuring Latency and Throughput The two most common performance metrics for LLMs are **TTFT** (time to first token) and **TPS** (tokens per second). | Metric | Description | Phase | Goal | |--------|-------------|-------|------| | **TTFT** | How long until the user sees the first output token? | Compute-bound prefill | Lower is better | | **TPS** | How many tokens per second after the first token? | Bandwidth-bound decode | Higher is better | More specific TPS terms: - **Perceived TPS**: Observed tokens per second per user after the first token (latency) - **Total TPS**: Total tokens generated per second by the entire inference service (throughput) - **Inter-token latency (ITL)**: Time between subsequent tokens. 10ms ITL = 100 TPS ### ISL and OSL (Input and Output Sequence Length) When defining and benchmarking a workload, two parameters shape everything: **ISL** and **OSL**. - **ISL (Input Sequence Length)**: the number of tokens in the prompt. ISL drives the **prefill** cost — longer inputs mean more compute and a higher TTFT. - **OSL (Output Sequence Length)**: the number of tokens generated. OSL drives the **decode** cost — longer outputs mean more memory-bandwidth-bound steps and higher total latency. The **ISL/OSL ratio** characterizes a workload. A summarization task (long input, short output) is prefill-heavy and compute-bound. A chat or agent task (short input, long output) is decode-heavy and bandwidth-bound. Always benchmark with ISL/OSL distributions that match your real traffic — synthetic uniform loads give misleading numbers. A prefill-heavy (high ISL, low OSL) workload benefits from high-FLOPS GPUs. A decode-heavy (low ISL, high OSL) workload benefits from high-memory-bandwidth GPUs. The same model can need different hardware depending on its ISL/OSL profile. ### Latency Percentiles LLM total response time is generally a right-skewed distribution — most times concentrate around a mode, but outliers can take significantly longer. | Percentile | Meaning | |-----------|---------| | **P50** | Median latency — 1 in 2 requests is slower | | **P90** | 90th percentile — 1 in 10 requests is slower | | **P95** | 95th percentile — 1 in 20 requests is slower | | **P99** | 99th percentile — 1 in 100 requests is slower | While driving down average latency matters, good performance work also focuses on reducing P90/P99 latencies. Outliers can dramatically affect user experience and trust. It's not good enough for most interactions to feel snappy if one in every ten takes several seconds. ### End-to-End Metrics Both inference-only and end-to-end metrics are valuable. **Inference time** tells you how effective your model performance work is, while **end-to-end metrics** reveal your users' perception of how fast your application is. When inference time is fast but end-to-end time is slow, turn your attention to infrastructure rather than model performance optimization. --- ================================================================================ # Chapter 2: Models — Architecture and Bottlenecks URL: https://inferenceengineering.tech/chapters/models/ Inference engineering is the practice of making generative AI models faster, less expensive, and more reliable — without sacrificing quality. Both improving performance and preserving quality require a strong intuition for how models work under the hood. A **transformer** is a neural network with an attention mechanism that can learn relationships between various parts of a sequence. Introduced in the 2017 paper "Attention Is All You Need," transformers are the foundation of generative AI across every modality. Across modalities, there are two important styles of transformer-based models: - **Autoregressive token generation**: Start from a tokenized sequence and predict the most likely next token (LLMs) - **Iterative denoising**: Start from random noise and refine toward the most likely output via diffusion (image/video generation) ## Neural Networks The fundamental unit of a neural network is a **node** (a.k.a., neuron). A node takes an input, multiplies it by some weights, adds some bias, and returns the result. A group of nodes forms a **layer**. The neural networks behind LLMs contain dozens to hundreds of layers across three types: - **Input layer**: The first layer, which accepts and processes input - **Hidden layers**: Every layer between first and last, which iteratively transform input to arrive at output - **Output layer**: The final layer, which returns the prediction Each layer produces an output that the next layer reads as input. For hidden layers, these outputs are called **hidden states**. There are neural networks for creating internal representations, and neural networks for using them: - **Encoder**: Takes an input like text or an image and creates an internal representation with additional semantic meaning - **Decoder**: Uses the internal representation to generate output like text or an image Modern LLMs are **decoder-only**, while many models in other modalities use an encoder-decoder architecture. Whisper, for audio transcription, uses an encoder to process audio input and a decoder to generate text tokens. ### Linear Layers and Matmul The most essential operation within a neural network is a **matrix multiplication** (matmul). A matmul takes an input vector and a matrix and multiplies the vector through the matrix to produce an output vector. Within a neural network, a **linear layer** is the simplest form of matmul. Given an input vector, the linear layer applies a weight matrix and adds a bias vector: `y = Wx + b` ### Activation Functions Matrix multiplication is composable — multiplying a vector by two matrices is equivalent to multiplying by the product of those matrices. This is a problem for multi-layer networks because a series of linear layers would collapse into a single layer. **Activation functions** are non-linear functions that prevent this collapse while supporting back-propagation. The most basic is **ReLU** (Rectified Linear Unit): if X > 0, return X; else return 0. Activation functions like ReLU, SiLU, Swish, and SwiGLU are fast to run, easy to train on, and break linearity to support multi-layer neural networks. ## LLM Inference Mechanics LLMs are autoregressive token generation models. An LLM generates new tokens one at a time based on every previous token. **Tokens** are the atomic units of language models — numbers that represent chunks of text. Modern LLMs use **subword tokenization**, meaning each token is a word or a fraction of a word. Converting text to tokens doesn't require neural networks — a **tokenizer** is a simple mapping between strings and their numerical representation. Inference involves two or three sequences of tokens: - **Input sequence**: The prompt, chat, context, and other input - **Reasoning sequence**: Optionally, for reasoning models, an intermediate thinking output - **Output sequence**: The response generated by the LLM There are two primary phases of inference: - **Prefill**: Process the input sequence to calculate attention for each input token and store values in a KV cache - **Decode**: Perform forward passes through the model to generate tokens autoregressively Each decode forward pass generates a vector of **logits** — one per token in the vocabulary. After normalization, these represent the probability of each potential next token. The output token is selected via weighted random number generation, nudged by temperature, top-k, and top-p parameters. ### LLM Architecture Every LLM on Hugging Face includes a `config.json` file detailing the model's architecture. Within a single architecture, there may be: - **Multiple sizes**: Models at different parameter counts (e.g., Llama 8B and 70B) - **Multiple variants**: "base" and "instruct" variants share the same architecture - **Unlimited fine-tunes**: Methods like LoRA change behavior, not architecture To parse an architecture name like `Qwen3MoeForCausalLM`: - **Qwen**: The model family - **3**: The major version - **MoE**: Mixture of Experts model - **CausalLM**: A causal language model (predicts next token based on previous tokens) ### Transformer Blocks The main body of an LLM is a series of dozens to hundreds of **transformer blocks** with three kinds of layers: - **Embedding layer**: The input layer takes tokens and returns embeddings - **Transformer blocks**: Hidden layers that generate a prediction via attention, feed-forward neural networks, and normalization sublayers - **Output layer** (LMHead): Converts hidden states into logit vectors The feed-forward neural network sublayers make up the majority of trainable weights, while attention sublayers are the second-largest component. ### Attention **Attention** is the mechanism transformers use to relate a given token to other tokens in the sequence. The standard form is **scaled dot-product attention** with three inputs: - **Q (queries)**: The embedded representation of the token being generated - **K (keys)**: Representations of all prior tokens - **V (values)**: Computed attention values for all prior tokens Attention sublayers are **multi-head** — each head is one attention operation that could attend to different kinds of relationships (subject-verb agreement, co-reference resolution, etc.). Two main types of attention: - **Self-attention**: Q, K, and V are all from the same sequence (used in LLMs) - **Cross-attention**: Q is from a different sequence than K and V (used in image generation and multimodal models) Because attention relates the current token to every previous token, it's a quadratic-time equation with respect to sequence length. In practice, the **KV cache** stores key-value pairs for each previous token, making attention linear rather than quadratic. The KV cache is built during prefill, used and updated during decode, and lives on GPU memory by default. Storing, accessing, and re-using the KV cache is a major topic in inference engineering (Chapter 5). ### Mixture of Experts Models **Mixture of Experts (MoE)** is an architecture that adds sparsity to linear layers. Rather than a single giant matrix, an MoE model has hundreds of smaller matrices (experts) and routes each input to a small selection of experts. For `Qwen3-235B-A22B`, 22 billion parameters out of 235 billion total are activated per request. MoE models are highly efficient for single-request local inference. However, in batched production inference, different requests activate different experts — you should expect almost all parameters to be active. Expert routing is granular: the **router** (a tiny model within the LLM) picks which experts to activate at each layer for every token generated. MoE architectures are especially popular for models with 100B+ parameters. Models under 32B parameters tend to use traditional dense architectures efficiently. ## Image Generation Inference Mechanics Image generation models take a text prompt and create an image. Unlike LLMs, they are **pipelines** of multiple models: - **Text encoder**: Converts the text prompt into instructions the model can understand - **Denoising model**: The heart of the model — iterates from noise to an image based on the prompt - **Variational Autoencoder (VAE)**: Converts output from latent space to pixel space The entire pipeline operates in **latent space** — a low-dimensional representation of an image (e.g., 128x128 instead of 1024x1024 pixels). The denoising model refines random noise into an image over 30-50 **denoising steps**. Key inference arguments: | Argument | Description | |----------|-------------| | **Prompt** | Describes what the image should look like | | **Negative prompt** | Styles or objects that should not appear | | **Number of steps** | Speed vs. quality tradeoff (30-50 typical) | | **Guidance scale** | Balance between creativity and prompt adherence | | **Image size** | Fixed menu of resolutions and aspect ratios | ### Image Generation Model Architecture Image generation models are built on **diffusion transformers** — similar to LLM transformers but processing image data in patches (2x2 or 4x4 pixels embedded into latent space). The latest research blends diffusion transformer architecture with LLM architecture. Anything that can be tokenized can be modeled as an LLM, and LLMs solve many problems inherent to diffusion models — variable-length output and single-forward-pass generation. ### Few-Step Image Generation Models Rather than making each step faster, few-step models create high-resolution images with **eight or fewer denoising steps** — 80 to 90 percent faster, with noticeably lower quality. Two primary methods: **latent consistency** (predict target latent directly) and **distillation** (adversarial/progressive distillation to emulate a larger model). ### Video Generation Video generation models are architecturally similar to image generation models, just bigger — 3-5x more parameters and 10-100x more information in latent space. Modern video models hold the entire video in latent space (X, Y, and T dimensions) and modify it on each denoising step. Video generation is so compute-intensive that models typically run with a batch size of one — a full node of eight GPUs works on a single request. ## Calculating Inference Bottlenecks GPUs have two main resources: **compute** (FLOPS) and **memory bandwidth** (bytes/s). In a perfectly optimized system, both are fully utilized. In reality, systems have bottlenecks: - **LLM prefill** (KV cache construction) is **compute bound** - **LLM decode** (token generation) is **memory bound** - **Image and video generation** are **compute bound** ### Ops:Byte Ratio and Arithmetic Intensity Each GPU has a specific ops:byte ratio. For an H100 in FP16: 989 teraFLOPS / 3.35 TB/s = ~295 ops per byte. **Arithmetic intensity** is the ratio between work and memory traffic for a calculation: ``` Arithmetic Intensity = FLOPs / Bytes Accessed ``` Plotting against a **roofline model** reveals whether an algorithm is: - **Compute bound**: Arithmetic intensity > hardware ops:byte ratio (hits performance ceiling) - **Memory bound**: Arithmetic intensity < hardware ops:byte ratio (hits bandwidth ceiling) The key difference: prefill processes the entire input sequence in parallel (load weights once, many matrix-matrix multiplications = high arithmetic intensity), while decode generates tokens one at a time (load weights for each token, vector-matrix multiplication = low arithmetic intensity). ### LLM Inference Bottlenecks - **Prefill**: Model weights loaded once, then large matrix multiplications against the input matrix. High arithmetic intensity → compute bound. - **Decode**: Model weights loaded for every token, with less-expensive vector-matrix multiplication. Low arithmetic intensity → memory bound. When optimizing performance, the goal is to make the bottleneck less limiting. For example, **batching** multiple requests makes decode less memory bound because processing a batch uses more compute for the same memory traffic. ### Image Generation Inference Bottlenecks Image and video generation are compute bound due to the massive matrix multiplications involved in processing latent space across multiple denoising steps. Optimizing compute throughput (more FLOPS) directly improves generation speed. ## Optimizing Attention Attention is the most expensive operation in inference across both LLMs and image/video generation. As context lengths grow and image resolutions increase, optimizing attention becomes increasingly critical. The full treatment of attention optimization techniques is in Chapter 5. --- ================================================================================ # Chapter 3: Hardware — GPUs and Accelerators URL: https://inferenceengineering.tech/chapters/hardware/ This chapter is the full deep-dive on GPU architecture, generations, and accelerators. For a scannable comparison of NVIDIA, AMD, and dedicated inference silicon — with a buying framework — see [AI Inference Hardware: GPU, TPU & Accelerator Comparison](/learn/ai-inference-hardware/). Also choosing between GPU tiers? See [GPU Inference: H100 vs A100 vs L4 Compared](/learn/gpu-inference/). Most inference engineers use cloud GPUs. Large enterprises and governments run on-premise deployments, but cloud-based GPUs offer the flexibility and access that fast-growing AI products need to scale. Even so, navigating the hardware landscape is complex. From variations among cloud providers to NVIDIA's naming conventions, there are many nuances in selecting the right accelerator. ## GPU Architecture GPUs are **throughput machines**. Where CPUs are great at complex sequential execution, GPUs are designed for simple, massively parallel workloads. Given that AI model inference is a series of vector and matrix multiplications, GPUs are a natural fit. GPUs have **Streaming Multiprocessors (SMs)**, and each SM contains multiple cores of three types: - **CUDA Core**: Operates on individual numbers (scalars) - **Tensor Core**: Operates on vectors and matrices - **Special Function Unit (SFU)**: Accelerates operations like sin, cos, log When measuring GPU compute for inference, focus on **Tensor Core compute** — they're responsible for Matrix Multiply and Accumulate (MMA) instructions that are foundational to inference. Compute is measured in **FLOPS** (floating point operations per second). When reading spec sheets, note two Tensor Core measurements: - **Dense**: Raw floating-point operations if every element is used - **Sparse**: In tensors with 2:4 structured sparsity, Tensor Cores can skip multiplication by 0 By default, inference is dense, so ensure you're looking at FLOPS without sparsity. FLOPS generally double with each halving of precision. **Compute** is the bottleneck for LLM prefill and for image/video generation. If selecting hardware for these use cases, pick the accelerator with more FLOPS. ### Memory and Caches GPUs contain high-speed onboard memory called **VRAM** — added as HBM3, HBM3e, or HBM4 (high-bandwidth memory). GPUs have dozens or hundreds of gigabytes of VRAM. Two types of memory on any chip: - **DRAM** (Dynamic RAM): General-purpose off-chip memory (gigabytes) — VRAM is a type of DRAM - **SRAM** (Static RAM): Faster, more expensive, on-chip memory (kilobytes/megabytes) GPUs have three levels of cache: | Cache | Scope | Example (H100) | |-------|-------|-----------------| | **L0** | Instruction cache per Tensor Core | — | | **L1** | Shared memory per SM | 256 KB per SM | | **L2** | Global cache across all SMs | 50 MB total | **VRAM bandwidth** measures the peak transfer rate between GPU cores and VRAM. This determines how quickly VRAM can supply data into the cache hierarchy. The total VRAM limits the model size you can load. VRAM should hold the model weights plus at least **50 percent headroom** for KV cache (more for long context, high batch sizes, or video generation models). If there isn't enough VRAM for the weights, loading the model will fail with an **OOM** (out of memory) error. If there isn't enough headroom, inference will be slow or crash. **Memory bandwidth** is the bottleneck for LLM decode at low to medium batch sizes. If you want more tokens per second, select the accelerator with higher memory bandwidth (e.g., H200 instead of H100). ## GPU Architecture Generations Every one to two years, NVIDIA releases a new GPU architecture. GPU names like B200 have two parts: - **Letter**: Signifies the architecture generation (e.g., B = Blackwell) - **Number**: Identifies models within the generation (bigger = more powerful) Since 1998, NVIDIA has named architectures for prominent scientists. Inference engineers generally work within the three to five most recent generations. ### Hopper GPUs | GPU | FP8 Compute (dense) | Memory | Bandwidth | |-----|---------------------|--------|-----------| | **H100** | 1,979 teraFLOPS | 80 GB | 3.35 TB/s | | **H200** | 1,979 teraFLOPS | 141 GB | 4.8 TB/s | Named for Rear Admiral Grace Hopper. First released March 2022. Key features: - **FP8 support**: FP8 Tensor Cores are 2x faster than FP16 and move data in half the bandwidth - Fourth-generation Tensor Cores with more and faster SMs - **FlashAttention 3** takes advantage of Hopper's asynchronous data transfer - The most widely used inference accelerators with industry-wide support and highly optimized kernels ### Ada Lovelace GPUs | GPU | FP8 Compute (dense) | Memory | Bandwidth | |-----|---------------------|--------|-----------| | **L4** | 242 teraFLOPS | 24 GB | 300 GB/s | | **L40** | 362 teraFLOPS | 48 GB | 864 GB/s | Named for the first computer programmer. More graphics-oriented than Hopper. **Does not support NVLink interconnect** — a major limitation for multi-GPU parallelism. L4 GPUs are a cheap way to run small models (text embeddings, computer vision). L40 GPUs generally aren't a great choice for inference — multi-instance GPUs offer better value. ### Blackwell GPUs | GPU | FP8 Compute (dense) | Memory | Bandwidth | |-----|---------------------|--------|-----------| | **B200** | ~5 petaFLOPS | 192 GB | Up to 8 TB/s | | **B300** | ~5 petaFLOPS | 288 GB | Up to 8 TB/s | Named for mathematician David Blackwell. First released November 2024. Key features: - **FP4 support** plus microscaling formats (MXFP8, MXFP4, NVFP4) for better quality retention - Updated asynchronous programming features for tiling loads and writes - **FlashAttention 4** relies on asynchronous pipelines - The new gold standard for inference — highest performance for LLMs and video generation ### Rubin GPUs The Rubin architecture (named for astronomer Vera Rubin) will launch in 2026 with: - **HBM4** for higher memory bandwidth — benefits memory-bound tasks like LLM decode - **CPX**: A separate chip built for compute-bound tasks like LLM prefill - Part of NVIDIA's rack-scale systems for high-volume inference After Rubin, NVIDIA will release **Feynman** in 2028. ### Grace and Vera CPUs NVIDIA offers ARM-based CPUs integrated with their GPUs on superchips like the GH200 and GB200. Grace CPUs use **NVLink Chip to Chip** for up to 900 GB/s bi-directional bandwidth between CPU and GPU memory — several times faster than PCIe. This matters for inference setups that offload LoRA weights and KV caches from previous calls to CPU memory. For the Rubin architecture, the **Vera CPU** replaces Grace. ## Instances The atomic unit of GPU allocation on the cloud is an **instance** — a virtual machine including: - **GPUs** (device): One or more GPUs for inference - **CPUs** (host): General-purpose compute - **Memory**: RAM for CPU operations - **Storage**: Disk for loading and storing files - **Networking**: Physical network connections - **Interconnect**: GPU-to-GPU and node-to-node connections When provisioning instances, understand exactly what you're getting. Any component — not just the GPU — could present a bottleneck or failure point during inference. ### Multi-GPU Instances When a model is too big for a single GPU, or when multiple GPUs improve performance, the standard unit is a **node** containing eight GPUs connected via: - **NVLink**: One-to-one communication between GPUs (up to 1800 GB/s on Blackwell, 900 GB/s on Hopper) - **NVSwitch**: All-to-all communication on top of NVLink for coordination among all GPUs in a node For more than eight GPUs, **InfiniBand** provides node-to-node interconnect (up to 400 Gb/s per NIC). NVIDIA's **NVL72 GB200** system combines 72 Blackwell GPUs and 36 Grace CPUs on a full rack. When working with multi-GPU systems, keep relative bandwidths in mind — NVLink is an order of magnitude faster than InfiniBand. ### Multi-Instance GPUs Sometimes the GPU is too big for the model. **Multi-instance GPU (MIG)** is a hardware-level capability in A100, H100, H200, and B200 GPUs — they can be split into as many as **seven pieces**, each receiving a slice of compute, memory, CPU, and networking. For example, an H100 MIG with three slices has about 3/7 of available compute and up to half of total VRAM (40 GB). For small models like Orpheus TTS (3B parameters), two MIG instances can be more efficient than a full GPU. ## Other Datacenter Accelerators NVIDIA is far from the only company making inference hardware. Notable alternatives: | Company | Product | Edge | |---------|---------|------| | **AMD** | MI350 GPU | Competitive specs on AMD's own software stack | | **AWS** | Inferentia / Trainium | Purpose-built chips on AWS | | **Cerebras** | WSE-3 | Wafer-scale chip with ultra-high memory bandwidth | | **Etched** | Sohu | ASIC for the transformer architecture | | **Google** | TPU | AI-specific ASIC for inference and training | | **Groq** | LPU | SRAM-based for high memory bandwidth | | **Qualcomm** | Cloud AI 100 Ultra | Power-efficient mobile GPUs | Every competitor is betting on a specific edge: memory bandwidth (Cerebras, Groq), power efficiency (Furiosa, Qualcomm), or platform integration (Amazon, Google). Shared challenges: rebuilding the software stack without CUDA, manufacturing complexity, and distribution. ## Local Inference **Local inference** (edge/client-side/on-device inference) means running models directly on the end user's device. **Advantages**: Zero network latency, independence from internet, improved privacy, free for the developer. **Limitations**: Weaker hardware capabilities, thermal constraints, fragmented support matrix, battery drain. ### Desktop Inference Increasingly, **Apple** is the leader in desktop inference. Apple's M-series CPUs and GPUs draw from a single **unified memory** pool, which avoids the bandwidth bottleneck of GPU VRAM. The M4 Max has 128 GB of unified memory — enough to run models up to 70B parameters — while MacBook Pros and Mac Studios are consumer-priced. Projects like **llama.cpp** and **MLX** have optimized LLM inference on Apple Silicon. ### Mobile Inference Mobile inference runs smaller models (1-3B parameters) on smartphones. Apple Intelligence uses on-device models for tasks like Smart Reply and text summarization. Qualcomm and MediaTek chips support inference through their NPUs. Local inference is turning the corner from experimentation to production, with a strong ecosystem across hardware and software and a vibrant community closely affiliated with the world of open models. --- ================================================================================ # Chapter 4: Software — From CUDA to Inference Engines URL: https://inferenceengineering.tech/chapters/software/ This chapter covers the full software stack from CUDA to inference engines to NVIDIA Dynamo. For a scannable side-by-side decision guide on vLLM, SGLang, and TensorRT-LLM with benchmark data, see [vLLM vs SGLang vs TensorRT-LLM: Which Inference Engine to Choose](/learn/vllm-vs-sglang-vs-tensorrt-llm/). The software stack for inference has four layers of abstraction: - **CUDA**: Direct communication to the GPU for explicit control over computations and memory - **Deep learning frameworks**: Abstractions over CUDA for training, exporting, and running neural networks in Python - **Inference engines**: Highly configurable PyTorch-backed inference for common architectures - **NVIDIA Dynamo**: Sits on top of inference engines to power large-scale deployments Most inference engineering today happens at the higher levels, configuring and deploying inference engines. No matter what level you work at, it's essential to have a strong mental model for the adjacent levels. ## CUDA **CUDA** (Compute Unified Device Architecture) is NVIDIA's proprietary computing platform and programming model for executing parallel tasks on GPUs. It's the foundation for the entire generative AI ecosystem on NVIDIA hardware. CUDA has four key components: - **CUDA kernel**: A function that executes parallelized code on the GPU - **CUDA graph**: A DAG of kernels and GPU operations for optimizing repeated workflows - **CUDA driver**: Low-level interface between the application and GPU hardware - **CUDA runtime**: Developer-facing API for launching kernels and managing memory CUDA is not a programming language — programs are written in C++, then compiled into separate CPU and GPU code by a compiler like `nvcc`. Writing CUDA kernels shifts inference engineering from thinking about algorithms to thinking about implementations. The traditional attention algorithm can be expressed in a few dozen lines, but FlashAttention — the same mathematical operation — takes tens of thousands of lines to implement efficiently for a specific GPU. ### CUDA Kernels for Inference The prior art predates CUDA by decades. **BLAS** (Basic Linear Algebra Subprograms) is a specification for common linear algebra operations. **cuBLAS** brings this to GPUs, and **cuDNN** provides neural network primitives. The most frequently used operation is **GEMM** (General Matrix-Matrix Multiplication). Key libraries: | Library | Purpose | |---------|---------| | **cuBLAS** | Pre-built kernels for essential linear algebra | | **CUTLASS** | Template library for writing high-performance kernels (used by FlashAttention 3) | | **CuTe** | Abstractions for tiled tensor operations on recent architectures | | **FlashInfer** | High-performance attention kernels and fused sampling functions | | **DeepGEMM** | Efficient FP8 GEMM kernels from the DeepSeek team | ### CUDA Kernel Selection Kernel implementations are highly specialized with hard-coded values based on specific GPU hardware. A kernel written for an H100 will likely not take advantage of B200 architecture. Most kernel selection is automatic — deep learning frameworks have pre-configured kernels for various architectures. ### Reducing Memory Accesses with Kernel Fusion Running two kernels back-to-back on the same data creates unnecessary round-trips to memory. **Kernel fusion** combines multiple kernels into a single kernel that handles both operations, eliminating intermediate reads and writes. During decode (the bandwidth-bound phase), an inference engine can't afford unnecessary memory accesses. Kernel fusion — both automatic (via compilers) and manual (like FlashAttention) — is essential for performance. ## Deep Learning Frameworks **PyTorch** is the industry standard technology underlying both training and inference. Originally created at Meta, now part of the Linux Foundation. PyTorch balances built-in functions and automatic optimizations with manual control. The step that transforms a model from training to inference is **compilation** (`torch.compile`), which performs automatic kernel selection and fusion for a specific GPU. ### Model File Formats - **safetensors**: The dominant format for serializing model weights. Uses memory mapping for fast, safe loading. Only holds tensor data, not executable code. - **ONNX**: Stores weights along with an execution graph. Highly portable across hardware. ### ONNX Runtime and TensorRT | | ONNX Runtime | TensorRT | |---|---|---| | **Source** | Open source (Linux Foundation) | Mix of proprietary and open (NVIDIA) | | **Hardware** | Many GPU types | NVIDIA only | | **Strength** | Portability | Raw performance | ### Transformers and Diffusers The `transformers` and `diffusers` libraries by Hugging Face offer reference implementations — great for learning and tinkering, but not designed for large-scale production inference. Use them for local inference and notebooks, then switch to production inference engines. ## Inference Engines Three competitive engines: **vLLM**, **SGLang**, and **TensorRT-LLM**. | | vLLM | SGLang | TensorRT-LLM | |---|---|---|---| | **Performance** | Good | Good | Best | | **Ease of use** | Easy | Easy | Hard | | **Model support** | Most | Most | Some | | **Hardware** | GPU, TPU | NVIDIA, AMD | NVIDIA only | All three support continuous batching, post-training quantization, speculative decoding, prefix caching, parallelism, and disaggregation out of the box. ### vLLM Largest market share. First released summer 2023. Best selling point: **broad support** — most hardware options, most model architectures, plus multimodal inference via vLLM Omni. Use when you want solid out-of-the-box performance for almost any model. ### SGLang Pairs a fast backend runtime with a flexible frontend language. Strong support for Chinese open models (DeepSeek, Qwen). Heavy investment in large-scale MoE deployments on systems like GB200 NVL72. ### TensorRT-LLM Steeper learning curve but usually best performance. Deep NVIDIA hardware integration with graph-level optimizations. At Baseten, it's the most-used engine. ## Inference Orchestration Once you scale beyond a single replica, you need an **orchestration layer** above the inference engine to route requests, manage KV cache across nodes, and coordinate disaggregated prefill/decode workers. Inference engines (vLLM, SGLang, TensorRT-LLM) run a model on a node; orchestration frameworks coordinate many of them. **NVIDIA Dynamo** is the leading open-source orchestration framework for vLLM and other engines. It's a distributed serving platform that sits on top of inference engines and manages: - **KV-cache-aware routing** — send requests to the replica that already holds the matching prefix, maximizing cache hits and lowering TTFT - **Disaggregated serving** — coordinate separate prefill and decode worker pools, scaling each independently - **Multi-GPU / multi-node orchestration** — the coordination layer for large-scale, multi-replica deployments Other building blocks in this layer include router/proxy tools like **LiteLLM** and **nginx** for application-level routing, and Kubernetes operators for autoscaling. The distinction to remember: the **inference engine** runs the model; the **orchestration framework** coordinates fleets of engines. ## Performance Benchmarking Benchmarks should mirror real-world usage as closely as possible. Use **jitter traffic** (randomized arrival times and sequence shapes) rather than uniform synthetic loads. Always measure P50, P90, and P99 latencies. Key benchmarking tools: **genai-perf** (NVIDIA), **vllm-benchmark**, **sglang-bench**. Profiling tools: **PyTorch Profiler**, **NVIDIA Nsight Systems** and **Nsight Compute**. Tips for useful benchmarking: 1. **Define realistic workloads**: Match your actual ISL/OSL distributions 2. **Warm up first**: Let the engine stabilize before measuring 3. **Test multiple configurations**: Vary batch size, parallelism, quantization 4. **Measure what matters**: TTFT and TPS for user-facing, total throughput for batch --- ================================================================================ # Chapter 5: Techniques — Optimization Deep Dives URL: https://inferenceengineering.tech/chapters/techniques/ This chapter is the complete deep-dive on quantization, speculative decoding, KV caching, batching, and parallelism. For a scannable guide to each technique — with speedup estimates and when each one actually helps — see [LLM Inference Acceleration: Every Technique That Matters](/learn/llm-inference-acceleration/). An inference engineer's goal is to create a balanced set of optimizations that delivers more than the sum of its parts. Sometimes techniques are symbiotic, sometimes incompatible — quantizing the KV cache helps disaggregation, but increasing batch size reduces compute available for speculation. ## Quantization **Quantization** improves latency (both TTFT and TPS), increases throughput, and opens headroom for other optimizations. But when it goes wrong, it can materially reduce output quality. Post-training quantization changes model weights from their native format (usually BF16 or FP16) to a lower-precision format: - **Prefill**: Compute-bound prefill now runs on lower-precision Tensor Cores with 2x FLOPS - **Decode**: Memory-bound decode now loads half as much data, effectively doubling bandwidth In practice, quantization down a single level of precision gives **30-50% better performance** for LLMs. ### Number Formats | Name | Bits | First Architecture | Use | |------|------|-------------------|-----| | FP16 | 16 | Pascal (2016) | Default inference | | BF16 | 16 | Ampere (2020) | Training & inference | | FP8 | 8 | Hopper (2022) | Sweet spot for quality/speed | | MXFP8 | 8 | Blackwell (2024) | Microscaling for better accuracy | | FP4 | 4 | Blackwell (2024) | Aggressive quantization | | NVFP4 | 4 | Blackwell (2024) | Best FP4 accuracy (block size 16) | Floating-point formats have **sign**, **exponent**, and **mantissa** bits — the exponent gives higher dynamic range than integers, better representing outliers after quantization. **Granularity** matters: tensor-level (one scale factor), channel-level, or block-level quantization. More granular = lower risk of quality loss but more overhead. ### Quantization Approaches Components vary in sensitivity — from least to most: 1. **Weights** (linear layers) — least sensitive 2. **Activations** — somewhat sensitive 3. **KV cache** — moderately sensitive (errors compound token-to-token) 4. **Attention** (especially softmax) — highly sensitive FP8/MXFP8 is the sweet spot for production inference. Quantize weights and activations with FP8, carefully quantize KV cache, and leave attention in original precision. If you can't risk quality, every other technique in this chapter is lossless. ### Measuring Quality Impact Three methods: **perplexity** (simplest), **intelligence benchmarks** (MMLU, SWE-bench), and **custom evals** (best). In every case, you're looking for differences indistinguishable from noise. ## Speculative Decoding Speculative decoding exploits spare compute during memory-bound decode to generate **multiple tokens per forward pass**. Only improves TPS/ITL, not TTFT. The mechanism: 1. A speculator generates draft tokens 2. The target model validates drafts in a single forward pass 3. Accepted tokens + one new token = N+1 tokens per pass Performance depends on: **draft token cost**, **draft sequence length**, and **token acceptance rate**. ### Draft-Target Speculation Uses a small draft model (10x smaller than target, usually same family). Quick to set up but highest overhead — needs to store draft model weights, activations, and KV cache. ### Medusa Fine-tunes the target model to add 2-4 extra decoder heads that generate draft tokens. No separate model needed, but limited acceptance rate. ### EAGLE Purpose-built draft model trained to consume **hidden states** from the target model and generate up to 8 draft tokens with high acceptance rate. The go-to algorithm for general use. ### N-gram Speculation Constructs an n-gram dictionary during prefill. Sequences can exceed 10 tokens. Mostly used for **code completion** where syntax is predictable and output closely matches input. ## Caching ### Prefix Caching and KV Cache Re-Use Many requests share common prefixes — system prompts, few-shot examples, previous conversation turns. **Prefix caching** stores and reuses KV cache for these shared prefixes, skipping redundant prefill computation. Prefix caching is especially effective for: - **Multi-turn chat**: Reuse KV from prior turns - **Code completion**: Reuse KV from file context - **Agents**: Reuse KV from system prompt and tools ### Where to Store the KV Cache By default, KV cache lives on GPU VRAM. Options for overflow: - **CPU memory** (via Grace NVLink C2C for fast access) - **Distributed KV stores** (across replicas) - **Disk** (last resort) ### Cache-Aware Routing Route requests to replicas that already hold matching prefixes. Higher cache hit rates yield lower TTFT. ## Model Parallelism When a model is too large for one GPU, split it across multiple GPUs: | Strategy | Mechanism | Best For | |----------|-----------|----------| | **Tensor Parallelism (TP)** | Split individual layers across GPUs | Lower latency (requires NVLink) | | **Expert Parallelism (EP)** | Shard MoE experts across GPUs | Higher throughput for MoE models | | **Pipeline Parallelism (PP)** | Split layers into stages | Multi-node (but introduces bubbles) | Use Tensor Parallelism within a node (where NVLink provides high bandwidth). For MoE models, Expert Parallelism enables each GPU to hold multiple full experts. Pipeline Parallelism is a last resort for multi-node setups. ## Disaggregation **Disaggregation** separates prefill and decode onto independently scaling hardware: - **Prefill workers**: Optimized for compute (high FLOPS) - **Decode workers**: Optimized for memory bandwidth The KV cache computed during prefill must be transferred to decode workers. KV cache quantization alleviates this transfer bottleneck. Use disaggregation when: - Prefill and decode have very different resource needs - Traffic patterns vary (some requests are prefill-heavy, others decode-heavy) - You need independent scaling for cost efficiency **NVIDIA Dynamo** provides dynamic disaggregation, automatically routing between prefill and decode workers based on real-time load. --- ================================================================================ # Chapter 6: Modalities — Beyond Text URL: https://inferenceengineering.tech/chapters/modalities/ Many of the same inference engines and techniques used for LLMs also apply to related modalities. Image and video generation rely on iterative denoising with different optimization details. For each modality, you need to adjust how you think about latency, throughput, and quality. ## Vision Language Models **Vision language models (VLMs)** take images or videos plus a text prompt and generate text responses. A VLM consists of: - **LLM**: A standard large language model (the large component) - **Vision encoder**: A small model that converts raw images/videos into image tokens As a rule of thumb, a high-resolution input image adds about **1,000 visual tokens** to the input sequence. The primary challenge is handling longer input sequences and larger KV caches. All Chapter 5 techniques apply: quantization (especially KV cache), speculation (EAGLE), prefix caching, tensor parallelism, and disaggregation. VLMs also introduce a quality-speed tradeoff in **downsampling** — converting images to visual tokens at various resolutions. ### Video Processing for VLMs One second of video at 24fps = 24 frames. Each frame ≈ 1,000 tokens. A 4-second clip → ~100,000 tokens. Downsampling is practically obligatory. Prefix caching and KV cache offloading become even more important. ### Omni-Modal Models The trend toward "omni" models that accept multiple input types and produce multiple output types. Running production inference often involves coordinating a pipeline of models — each must be individually optimized and should scale independently. ## Embedding Models **Embedding models** transform variable-length input into fixed-length vectors that capture semantic meaning. Two traffic profiles: 1. **High-throughput backfills**: Bulk indexing of millions of documents 2. **Low-latency lookups**: Individual user-facing queries Two architectures: **BERT-style** (encoder-only, under 1B params) and **LLM-based** (8B params or fewer, substantially better quality). Modern models use **Matryoshka representations** for dynamic dimensionality tradeoffs. For embedding models, scale horizontally (each GPU as its own replica) rather than using multi-GPU parallelism. Batching is critical — embedding models support much higher batch sizes than LLMs. ## ASR Models **Automatic Speech Recognition** (ASR) — audio in, text out. The most popular open model is **Whisper** (1.55B parameters). An encoder-decoder model where the decoder dominates inference time. ### Single-Chunk Optimization For real-time transcription, target **200ms round-trip time**. The biggest upgrade is **streaming over WebSockets** — continuous audio in, continuous text out. ### Long File Optimization Measure with **Real-Time Factor (RTF)**. Optimized Whisper can transcribe an hour of audio in under 4 seconds (RTF 1000x). Pipeline: VAD → parallel chunk processing → stitching. ### Diarization Speaker identification uses a pipeline of classic ML models (segmentation, embedding, clustering), not transformers. Even optimized, diarization takes 2x+ longer than transcription. ## TTS Models **Text-to-speech** models are fine-tuned LLMs (e.g., Orpheus TTS from Llama 3.2 3B). Same runtime optimizations apply: TensorRT-LLM, FP8 quantization, MIGs. Key metrics: **TTFB** (time to first byte), **time to first sentence**, **TPS** (80-100 tok/s needed for real-time audio). Beyond real-time speed, scale throughput via concurrent streams. Streaming over WebSockets is again the biggest infrastructure unlock. ### Speech-to-Speech Models Unify ASR + LLM + TTS into a single model. At publication, no commercially viable open options exist, but research is active. Most production voice systems use the cascading multi-model pipeline. ## Image Generation Models Working with image/video generation is entirely different from LLMs: - Architecture: Iterative denoisers, not autoregressive generators - Tooling: PyTorch, TensorRT, and Diffusers (SGLang Diffusion and vLLM Omni are brand new) - Optimization: CUDA kernels and custom pipelines rather than inference engine configs ### Kernel Optimization Image generation benefits from: - **Fused attention kernels** (SageAttention for quality-preserving FP8 attention) - **Quantization** (FP8 for denoiser weights and activations) - **Compilation** (torch.compile for the denoising loop) ### One Weird Trick Run the text encoder and VAE concurrently with the denoiser on separate hardware. The text encoder runs once and the VAE runs once, while the denoiser runs 30-50 steps — overlapping these saves meaningful time. ## Video Generation Models Video generation extends image generation to 3D latent space (X, Y, Time). Models are 3-5x larger with 10-100x more latent space data. Key optimizations: - **Attention optimization**: SageAttention and TeaCache for skipping redundant denoising steps - **Quantization**: FP8 for the denoiser - **Context Parallelism**: Replicates weights across GPUs and partitions the attention context — essential for video's massive latent space --- ================================================================================ # Chapter 7: Production — Ship It URL: https://inferenceengineering.tech/chapters/production/ The final challenge: taking optimized model inference from development to production at scale. ## Containerization Containers turn a program into a packaged artifact that can run anywhere. For inference, **Docker** is the standard: - **Container**: An actively running environment that isolates an application - **Image**: An executable package with everything needed to run the software - **Dockerfile**: Instructions for creating an image - **Registry**: Central repository for managing and distributing images Inference engines like vLLM and SGLang offer official base images. Start from these rather than building your own from scratch. ### Dependency Management Dependency chains for inference are long and fragile. Key practices: - **Pin versions** exactly — tools like `uv`, `poetry`, or `pip` will flag incompatibilities - **Pack light** — images are often many gigabytes; only include necessary dependencies - **Use safetensors** — the dominant format for serializing model weights safely ### NIMs **NVIDIA Inference Microservices** are pre-built Docker containers for popular open models. Available as multi-LLM (flexible) or model-specific (maximum performance) configurations. ## Autoscaling The goal: always have enough resources to serve all requests while maintaining latency SLAs without wasting money on idle GPUs. Autoscaling uses **Kubernetes** for container orchestration. Five key configuration factors: | Factor | Description | |--------|-------------| | **Min replicas** | Minimum running replicas regardless of traffic | | **Max replicas** | Maximum replicas during high traffic | | **Autoscaling window** | Sliding timeframe for measuring traffic | | **Scale down delay** | Wait time before scaling down | | **Concurrency target** | Requests each replica handles at once | ### Concurrency and Batch Sizing Three batching approaches in order of sophistication: 1. **Static batching**: Wait until batch is full 2. **Dynamic batching**: Wait until full OR timeout 3. **Continuous batching**: Token-level interleaving (the standard for modern engines) Batch sizing trades latency for throughput. Test multiple sizes to find the right fit. ### Cold Starts Cold start time = GPU procurement + image loading + model loading + engine startup. Optimizations: - **Smaller images**: Include only necessary dependencies - **Quantized weights**: Smaller files load faster - **Cached engines**: Pre-compile TensorRT-LLM engines and cache them - **Local weight storage**: Load from within the same datacenter, not remote S3 ### Routing, Load Balancing, and Queueing **Routers** decide *where should this request go?* **Load balancers** decide *where could this request go?* Intelligent routing uses: - **KV cache-aware routing**: Send to replicas with matching prefix - **LoRA-aware routing**: Send to replicas with desired fine-tune weights - **Queue management**: First-in-first-out with optional priority queuing ### Scale to Zero Scale down to zero replicas when idle, scale up when traffic arrives. Requires fast cold starts and robust queueing. Best for development and periodic batch workloads, not latency-sensitive production. ## Multi-Cloud Capacity Management High-volume deployments need thousands of GPUs distributed globally. True multi-cloud requires treating distinct pools as fungible compute: - **Control plane**: Global model deployment and scaling decisions - **Workload planes**: Direct inference traffic and in-cluster scaling Benefits: capacity pooling, redundancy, low-latency (5ms per timezone), and compliance. ### GPU Procurement | Type | Description | |------|-------------| | **Reserved** | Hundreds/thousands of GPUs for months/years at discount | | **On-demand** | Individual instances as needed at higher cost | | **Spot** | Discounted instances that can be pre-empted | Large-scale inference uses a blend: reserved baseline + on-demand/spot for peaks. ### Building for Reliability GPUs fail. Meta's Llama 3 training saw **419 interruptions in 54 days** across 16,000 GPUs — roughly one failure per 50,000 GPU-hours. Two high-availability postures: - **Active-active**: Multiple regions serve traffic simultaneously; seamless failover - **Active-passive**: Hot standby cluster ready to take over Inference engineers should expect hardware failure. Proactively noting failures, cordoning nodes, and cycling pods keeps clusters healthy. Multi-cloud active-active postures provide the highest reliability. ## Testing and Deployment ### Zero-Downtime Deployment Two strategies: - **Blue-green**: Two parallel environments; shift traffic between them - **Canary**: Start with small share of traffic to new deployment; gradually increase Shadow traffic — mirroring real requests to a candidate deployment — validates performance without affecting users. ### Cost Estimation Cost per token depends on: GPU cost, utilization, throughput, and precision. Track cost metrics alongside latency metrics for informed optimization decisions. ### Observability Monitor: GPU utilization, VRAM usage, request latency (P50/P90/P99), queue depth, throughput, error rates, and model quality metrics. Alert on deviations from baselines. ## Client Code ### Client Latency Overhead Network latency adds to inference time. Minimize with: regional deployment, connection pooling, and keep-alive connections. ### Asynchronous Inference For non-real-time workloads, submit requests to a queue and poll for results. Decouples request submission from response processing. ### Streaming and Protocol Support | Protocol | Best For | |----------|---------| | **HTTP/REST** | Simple request-response (most common) | | **SSE** (Server-Sent Events) | LLM token streaming | | **WebSockets** | Bidirectional streaming (voice, real-time) | | **gRPC** | High-performance, schema-first APIs | For LLM inference, SSE streaming is the standard. For voice applications (ASR/TTS), WebSockets enable the lowest-latency bidirectional streaming. Choose protocols based on your application's real-time requirements. --- ================================================================================ # vLLM vs SGLang vs TensorRT-LLM URL: https://inferenceengineering.tech/learn/vllm-vs-sglang-vs-tensorrt-llm/ This article is a scannable comparison with a decision framework. For the full deep-dive on the inference software stack — from CUDA kernels to NVIDIA Dynamo — read [Chapter 4: vLLM, SGLang & TensorRT-LLM in Context](/chapters/software/). Choosing an inference engine is one of the highest-leverage decisions in an LLM deployment. The three leading open-source options — **vLLM**, **SGLang**, and **TensorRT-LLM** — all serve the same purpose but make very different tradeoffs. This guide compares them on the dimensions that actually matter in production. ## The quick answer | | vLLM | SGLang | TensorRT-LLM | |---|---|---|---| | **Performance** | Very good | Very good | Best (on NVIDIA) | | **Ease of use** | Easy | Easy | Hard | | **Model support** | Broadest | Broad | Selective | | **Hardware** | NVIDIA, AMD, TPU+ | NVIDIA, AMD | NVIDIA only | | **Best for** | Almost any model, fast iteration | High-concurrency & large MoE | Squeezing max perf from NVIDIA | If you want one default recommendation: **start with vLLM**. It has the broadest model and hardware support and the gentlest learning curve. Move to SGLang or TensorRT-LLM when you have a specific performance reason. ## What all three have in common Before the differences, it's worth knowing that all three modern engines share the same core optimizations out of the box: - **Continuous (in-flight) batching** — new requests join the running batch instead of waiting for it to finish - **Paged KV cache** — memory-efficient attention key/value storage (PagedAttention and equivalents) - **Prefix caching** — reuse of the KV cache for shared prompt prefixes - **Quantization** — FP8, INT8, INT4 support - **Speculative decoding** — draft-and-verify token generation - **Parallelism** — tensor and pipeline parallelism for multi-GPU So the choice is rarely about *features*. It's about performance, hardware breadth, and how much engineering effort you want to spend. ## vLLM — the default choice vLLM has the largest market share and community. Its biggest selling point is **breadth**: it supports the most model architectures and the widest range of hardware (NVIDIA, AMD, TPU, and more), including multimodal inference. Use vLLM when you want solid, near-optimal performance for almost any model with minimal setup. It's the safest starting point and the easiest to hire for. You're serving a standard model, you value fast iteration and broad compatibility over the last 10% of performance, or you're running on non-NVIDIA hardware. ## SGLang — high concurrency and large MoE SGLang pairs a fast backend runtime with a flexible frontend language for expressing complex generation programs. It has strong support for Chinese open models (DeepSeek, Qwen) and heavy investment in **large-scale MoE deployments** on systems like GB200 NVL72. In our own H100 benchmarks (5,980 requests on 2× H100 SXM, TP=2), SGLang is highly competitive with vLLM and pulls ahead on some high-concurrency throughput workloads. You're running large mixture-of-experts models, need maximum throughput under high concurrency, or want structured/programmatic generation. ## TensorRT-LLM — maximum NVIDIA performance TensorRT-LLM has the steepest learning curve but usually delivers the **best raw performance on NVIDIA hardware**. It compiles models into optimized engines with graph-level optimizations and deep hardware integration. At Baseten, it's the most-used engine in production. The cost is flexibility: it's NVIDIA-only, supports fewer architectures out of the box, and the engine-build step adds operational complexity. You trade engineering time and flexibility for performance. If you have a stable model, NVIDIA hardware, and performance is your top priority, the investment pays off. If you're still iterating on models, it will slow you down. ## A decision framework Ask these questions in order: 1. **Is your hardware non-NVIDIA?** → vLLM (or SGLang for AMD). 2. **Are you running a large MoE model or need peak high-concurrency throughput?** → SGLang. 3. **Is performance the top priority, on stable NVIDIA hardware, with engineering time to invest?** → TensorRT-LLM. 4. **Otherwise** → vLLM. For a personalized recommendation based on your model, scale, and constraints, try the interactive wizard below. ## See the real numbers Don't take comparison tables at face value — performance depends heavily on your model, sequence lengths, and concurrency. We ran vLLM and SGLang head-to-head on identical hardware so you can see the tradeoffs on real workloads. There is no universally "fastest" engine. vLLM is the best default; SGLang wins on high-concurrency and large MoE; TensorRT-LLM wins on raw NVIDIA performance when you can afford the setup cost. Benchmark on **your** workload before committing. --- ================================================================================ # GPU Inference Explained URL: https://inferenceengineering.tech/learn/gpu-inference/ This article explains the key concepts and gives a practical GPU selection framework. For the full deep-dive on GPU architecture generations, VRAM, interconnects, and cloud accelerator options, read [Chapter 3: GPU Hardware & Accelerators](/chapters/hardware/). GPUs power almost all modern LLM inference, but *why* they're the right tool — and where their real bottleneck lies — is widely misunderstood. This guide explains how GPUs run inference and what that means for choosing and sizing hardware. ## Why GPUs, not CPUs GPUs are **throughput machines**. Where a CPU excels at complex sequential logic, a GPU is built for simple operations run massively in parallel. Model inference is fundamentally a long sequence of matrix multiplications — exactly the workload GPUs are designed for. A modern datacenter GPU has thousands of cores and, critically, **specialized Tensor Cores** that perform matrix-multiply-and-accumulate operations far faster than general-purpose cores. This is why a single H100 can serve models that would crawl on even a high-end CPU. ## The two phases of LLM inference Every LLM request runs in two distinct phases, and they stress the GPU very differently: **Prefill** processes the entire input prompt in one parallel forward pass. It's **compute-bound** — the GPU's Tensor Cores are the limiting factor. This phase determines your *time to first token (TTFT)*. **Decode** generates output tokens one at a time, autoregressively. For each new token, the GPU must read the *entire* set of model weights from memory. This makes decode **memory-bandwidth-bound** — the GPU spends most of its time waiting for data to arrive from VRAM, not computing. This phase determines your *tokens per second (TPS)*. For most interactive LLM workloads, decode dominates — and decode is limited by **memory bandwidth, not compute**. This is why an H200 (higher bandwidth) often beats an H100 for token generation even though they have similar FLOPS. ## Compute vs memory bandwidth Two GPU specs matter most for inference: - **Compute (FLOPS)** — how many floating-point operations per second the Tensor Cores can do. This caps prefill throughput. - **Memory bandwidth (TB/s)** — how fast data moves between VRAM and the cores. This caps decode throughput. The ratio of these two — the **arithmetic intensity** of your workload — tells you which one you're bound by. Small-batch decode has low arithmetic intensity (you read a lot of weights to compute relatively little), so it's memory-bound. Large-batch prefill has high arithmetic intensity, so it's compute-bound. ## Memory: the hard limit Beyond speed, VRAM **capacity** sets a hard ceiling on what you can run. The GPU must hold: - **Model weights** — parameters × bytes per parameter (e.g. a 7B model in FP16 ≈ 14 GB) - **KV cache** — grows with sequence length and batch size; can rival or exceed the weights for long contexts - **Activations and overhead** — typically 10-20% on top If the weights don't fit, the model won't load (an OOM error). If there isn't enough headroom for the KV cache, inference slows or crashes under load. ## Picking a GPU for inference A practical heuristic: 1. **Will the model weights fit in VRAM**, with ~50% headroom for KV cache? If not, you need a bigger GPU or multiple GPUs with parallelism. 2. **Is your workload decode-heavy** (chat, generation)? Prioritize **memory bandwidth** (H200 > H100). 3. **Is it prefill-heavy** (long prompts, embeddings, image generation)? Prioritize **compute (FLOPS)**. 4. **Cost-sensitive and small model?** An L4 or L40S may be far more economical than an H100. GPUs win at inference because of parallel Tensor Core throughput, but the binding constraint for token generation is usually **memory bandwidth**, not compute. Size for VRAM capacity first, then optimize for bandwidth or compute based on whether your workload is decode- or prefill-heavy. --- ================================================================================ # AI Inference Hardware Guide URL: https://inferenceengineering.tech/learn/ai-inference-hardware/ This article maps the hardware landscape and gives a buying framework. For the complete deep-dive on GPU architecture, Tensor Cores, HBM generations, NVLink, and PCIe topology, read [Chapter 3: GPU Hardware & Accelerators](/chapters/hardware/). AI inference hardware has exploded beyond GPUs into a landscape of TPUs, custom inference chips, and specialized accelerators. This guide maps the options and shows how to compare them on the specs that actually determine inference performance. ## The categories of inference hardware **Datacenter GPUs** — NVIDIA (H100, H200, B200, GB200), AMD (MI300X, MI325X). The default for almost all LLM inference thanks to mature software and high memory bandwidth. **TPUs** — Google's Tensor Processing Units, available on Google Cloud. Strong for both training and inference, especially with JAX-based stacks. **Dedicated inference chips** — Groq LPUs, Cerebras, AWS Inferentia, SambaNova. Purpose-built for inference, often optimizing for ultra-low latency or cost-per-token on specific model classes. **Workstation / local GPUs** — RTX 4090/5090, L4, L40S. For development, small models, or budget deployments. ## The specs that matter When comparing any inference accelerator, three numbers dominate: | Spec | What it limits | Why it matters | |---|---|---| | **Memory capacity (GB)** | Largest model you can load | Hard ceiling — weights + KV cache must fit | | **Memory bandwidth (TB/s)** | Token generation speed (decode) | The real bottleneck for most LLM serving | | **Compute (FLOPS)** | Prompt processing speed (prefill) | Matters for long prompts, embeddings, image gen | For LLM inference specifically, **memory bandwidth is usually the binding constraint** — decode reads the full model from memory for every token generated. A chip with massive FLOPS but mediocre bandwidth will underperform on chat-style workloads. During autoregressive decode, the GPU re-reads all model weights from VRAM to produce each token. The arithmetic per byte read is low, so the memory system — not the compute units — sets the pace. This is the single most important fact in inference hardware selection. ## Comparing NVIDIA's datacenter lineup The interactive tool above lets you compare memory, bandwidth, and compute across NVIDIA generations. A few rules of thumb: - **H100** — the workhorse. Great all-rounder for 7B-70B models. - **H200** — same compute as H100 but far more memory (141 GB) and bandwidth. Better for large models and decode-heavy workloads. - **B200 / GB200 (Blackwell)** — next-gen, dramatically higher memory and bandwidth for frontier models and large MoE. - **A100** — previous generation, still cost-effective for many workloads. - **L4 / L40S** — economical for small models and budget deployments. ## Training vs inference hardware A common confusion: the best training chip isn't necessarily the best inference chip. - **Training** demands maximum sustained compute, high-precision arithmetic, and fast interconnect across many chips for long runs. - **Inference** prioritizes memory bandwidth (for decode), low latency, cost-efficiency, and often runs at lower precision (FP8/INT8/INT4). This is why dedicated inference chips exist — they drop training-oriented features to optimize for serving economics. ## Dedicated inference chips: worth it? Chips like Groq (ultra-low-latency token streaming) or AWS Inferentia (cost-optimized) can beat GPUs on specific metrics for specific models. But GPUs remain the most flexible and best-supported option, with the broadest model coverage and software ecosystem. You have a stable, high-volume workload where a specific latency target or cost-per-token would justify the reduced flexibility and smaller software ecosystem. For most teams, GPUs are still the pragmatic default. ## Choosing your hardware The decision comes down to: model size (does it fit in memory?), workload shape (decode- or prefill-heavy?), latency requirements, and budget. For LLM inference, rank accelerators by **memory capacity** (can the model fit?) and **memory bandwidth** (how fast can it decode?) before compute. GPUs are the flexible default; TPUs and dedicated inference chips can win on specific cost or latency profiles. --- ================================================================================ # LLM Inference Acceleration: Complete Guide URL: https://inferenceengineering.tech/learn/llm-inference-acceleration/ This article covers every major technique with practical guidance on when each helps. For the complete deep-dive — including implementation details, quantization math, and speculative decoding internals — read [Chapter 5: Quantization & Speculative Decoding](/chapters/techniques/) and [Chapter 4: vLLM, SGLang & TensorRT-LLM in Context](/chapters/software/). LLM inference is expensive and often slow. The good news: a well-understood toolkit of acceleration techniques can cut latency and cost dramatically. This guide covers every major technique, what it does, when it helps, and how much. ## First, know your bottleneck You can't accelerate effectively without knowing what's limiting you. LLM inference has two phases: - **Prefill** (processing the prompt) is **compute-bound**. - **Decode** (generating tokens) is **memory-bandwidth-bound**. Most interactive workloads are decode-dominated, which means the techniques that reduce *memory traffic* (quantization, KV cache optimization) tend to give the biggest wins. Throughput-oriented batch workloads benefit more from batching and parallelism. Are you latency-bound (one user waiting) or throughput-bound (many concurrent users)? The answer changes which technique to reach for first. Optimizing the wrong dimension wastes effort. ## 1. Quantization — the highest-leverage win Quantization stores weights (and sometimes activations) in lower precision: FP8, INT8, or INT4 instead of FP16. Because decode is memory-bound, **halving the bytes per weight roughly halves the memory traffic** — directly speeding up token generation and freeing VRAM for larger batches or longer context. | Format | Memory vs FP16 | Typical quality loss | |---|---|---| | FP8 | 0.5× | < 1% | | INT8 (SmoothQuant) | 0.5× | < 1% | | INT4 (AWQ / GPTQ) | 0.25× | 1-3% | ## 2. Continuous batching — free throughput Traditional batching waits for every request in a batch to finish before starting the next. **Continuous (in-flight) batching** lets new requests join the running batch as soon as slots free up, dramatically improving GPU utilization under concurrent load. This is table-stakes in modern engines (vLLM, SGLang, TensorRT-LLM all do it) — make sure it's enabled. It's one of the largest throughput wins available and costs you nothing in quality. ## 3. KV cache optimization The KV cache stores attention keys and values so the model doesn't recompute them for every token. It's essential — but it grows with sequence length and batch size and can dominate VRAM. Key techniques: - **Paged KV cache (PagedAttention)** — eliminates memory fragmentation, packing more requests into the same VRAM. - **Prefix caching** — reuse the KV cache across requests sharing a prompt prefix (system prompts, few-shot examples, multi-turn chat). Huge TTFT win for repeated prefixes. - **KV cache quantization** — store the cache in FP8/INT8 to fit longer contexts. ## 4. Speculative decoding — latency for low batches Speculative decoding uses a small, fast **draft model** to propose several tokens, which the large model then verifies in a single forward pass. When the draft is accepted, you get multiple tokens for the cost of one decode step. It can improve effective TPS by **1.5-3×** at low-to-medium batch sizes. The benefit shrinks at high batch sizes (where the large model is already well-utilized) and depends on the draft model's acceptance rate. ## 5. Model parallelism — when one GPU isn't enough When a model is too large for a single GPU, split it across several: - **Tensor parallelism (TP)** — split individual layers across GPUs; lowest latency, needs fast NVLink. - **Pipeline parallelism (PP)** — split layers into stages across GPUs/nodes; introduces "bubbles." - **Expert parallelism (EP)** — shard MoE experts; high throughput for mixture-of-experts models. Use TP within a node (high-bandwidth NVLink), PP across nodes as a last resort, and EP for large MoE models. ## 6. Disaggregation — independent scaling **Disaggregation** runs prefill and decode on separate, independently-scaled hardware — prefill on compute-optimized GPUs, decode on bandwidth-optimized ones. This matches each phase to the hardware it needs and scales them independently for cost efficiency. NVIDIA Dynamo provides dynamic disaggregation. ## Putting it together A practical acceleration playbook for a typical deployment: 1. **Quantize** the model (FP8 or INT4) — biggest single win. 2. **Enable continuous batching** — if it isn't already. 3. **Enable prefix caching** — especially if you have shared system prompts. 4. **Add speculative decoding** — if you're latency-bound at low batch sizes. 5. **Scale with parallelism / disaggregation** — when you outgrow one GPU. Start with quantization and continuous batching — they give the largest wins for the least effort. Layer in prefix caching, speculative decoding, and parallelism based on whether you're latency- or throughput-bound. Always measure on your own workload. ---