# Inference Engineering

> The definitive interactive guide to AI model inference — GPU hardware, inference engines (vLLM, SGLang, TensorRT-LLM), quantization, KV caching, and production autoscaling. Written by Philip Kiely, published by Baseten Books.

## About

Inference Engineering is an interactive technical guide covering everything needed to deploy and optimize AI models in production. The site combines reading chapters, interactive calculators, quizzes, and guided learning paths.

- [Inference Engineering Home](https://inferenceengineering.tech/): Full interactive guide to AI inference engineering
- [Learning Paths](https://inferenceengineering.tech/paths/): Structured learning paths for different engineer types
- [Benchmarks](https://inferenceengineering.tech/benchmarks/): Real H100 benchmark data comparing inference engines
- [Interactive Exercises](https://inferenceengineering.tech/exercises/): Calculators and simulators for inference engineering

## Chapters

- [Preface: Why Inference Engineering Matters](https://inferenceengineering.tech/chapters/preface/): The explosive growth of open models and why inference engineering is the most important skill in AI.
- [Chapter 0: Inference — The Three Layers of Inference](https://inferenceengineering.tech/chapters/inference/): Introduces the three layers of inference: runtime, infrastructure, and tooling. A map of the entire book.
- [Chapter 1: Prerequisites — Before You Optimize](https://inferenceengineering.tech/chapters/prerequisites/): Use case definition, latency and cost budgeting, model selection and evaluation, and fine-tuning for quality.
- [Chapter 2: Models — Architecture and Bottlenecks](https://inferenceengineering.tech/chapters/models/): Technical architecture of LLMs and diffusion models — transformers, attention, MoE, and inference bottlenecks.
- [Chapter 3: Hardware — GPUs and Accelerators](https://inferenceengineering.tech/chapters/hardware/): GPU architecture, compute and memory, NVIDIA generations (Hopper, Blackwell, Rubin), instances, and alternatives.
- [Chapter 4: Software — From CUDA to Inference Engines](https://inferenceengineering.tech/chapters/software/): CUDA kernels, PyTorch, model formats, inference engines (vLLM, SGLang, TensorRT-LLM), NVIDIA Dynamo, and benchmarking.
- [Chapter 5: Techniques — Optimization Deep Dives](https://inferenceengineering.tech/chapters/techniques/): Quantization, speculative decoding, KV cache re-use, model parallelism, and disaggregation in practice.
- [Chapter 6: Modalities — Beyond Text](https://inferenceengineering.tech/chapters/modalities/): Vision language models, embeddings, ASR, TTS, image generation, and video generation inference.
- [Chapter 7: Production — Ship It](https://inferenceengineering.tech/chapters/production/): Containerization, autoscaling, multi-cloud, deployment, observability, and client code for production inference.

## Learn Articles

- [vLLM vs SGLang vs TensorRT-LLM](https://inferenceengineering.tech/learn/vllm-vs-sglang-vs-tensorrt-llm/): The three leading open-source inference engines, compared on performance, ease of use, and hardware support — with real benchmark data and a decision framework.
- [GPU Inference Explained](https://inferenceengineering.tech/learn/gpu-inference/): How GPUs actually run model inference — compute vs memory bandwidth, prefill vs decode, and why the bottleneck is almost never what you think.
- [AI Inference Hardware Guide](https://inferenceengineering.tech/learn/ai-inference-hardware/): The landscape of AI inference hardware — GPUs, TPUs, and dedicated inference chips — and how to compare them on the specs that actually matter.
- [LLM Inference Acceleration: Complete Guide](https://inferenceengineering.tech/learn/llm-inference-acceleration/): The complete toolkit for making LLM inference faster and cheaper — quantization, speculative decoding, KV caching, batching, and parallelism — and when each one actually helps.

## Interactive Tools

- [VRAM Calculator](https://inferenceengineering.tech/exercises/vram-calculator/): Calculate memory requirements for model inference: weights, KV cache, activations, and overhead vs GPU capacity.
- [Arithmetic Intensity Calculator](https://inferenceengineering.tech/exercises/arithmetic-intensity/): Compute the ops:byte ratio and determine whether your workload is compute-bound or memory-bound with a roofline plot.
- [KV Cache Sizing Calculator](https://inferenceengineering.tech/exercises/kv-cache-sizing/): Calculate KV cache memory for different models, sequence lengths, batch sizes, and precisions.
- [Quantization Quality Estimator](https://inferenceengineering.tech/exercises/quantization-estimator/): Explore precision tradeoffs: memory savings, speedup estimates, and quality risk across quantization strategies.
- [Speculative Decoding Simulator](https://inferenceengineering.tech/exercises/speculative-decoding/): Simulate draft-verify cycles: adjust draft length, acceptance rate, and overhead to see effective TPS improvement.
- [GPU Selection Advisor](https://inferenceengineering.tech/exercises/gpu-advisor/): Answer questions about your model, priorities, and budget to get ranked GPU recommendations with reasoning.
- [Model-to-Hardware Recommender](https://inferenceengineering.tech/exercises/hardware-recommender/): Input your model and performance requirements to get ranked GPU configuration recommendations with reasoning.

## Key Resources

- [Benchmarks](https://inferenceengineering.tech/benchmarks/): Reproducible H100 benchmark data comparing vLLM, SGLang, and TensorRT-LLM across real workloads.
- [Learning Paths](https://inferenceengineering.tech/paths/): Structured learning paths: Getting Started, Infrastructure Architect, and Performance Optimizer tracks.
- [Cheat Sheets](https://inferenceengineering.tech/cheat-sheets/): Quick-reference guides for inference engineering concepts, formulas, and decision frameworks.
- [Guides](https://inferenceengineering.tech/guides/): Interactive wizards for choosing serving frameworks, hardware, and optimization strategies.

## Note
Content © Baseten Books. Citations welcome; please link to the source URL.