NVIDIA Dynamo: The Orchestration Layer That Turns vLLM Into a Cluster

SGLang: Structured Generation for LLMs • Self-Hosted LLM 2026: Ollama vs vLLM vs LocalAI

What Is NVIDIA Dynamo? #

Dynamo is NVIDIA’s open-source, datacenter-scale inference stack — but the framing that matters most is what it isn’t: it doesn’t replace SGLang, TensorRT-LLM, or vLLM. It’s the orchestration layer above them, turning a cluster of GPUs running any of those engines into a coordinated multi-node inference system, with disaggregated serving, intelligent routing, multi-tier KV caching, and automatic scaling working together.

🔗 GitHub: https://github.com/ai-dynamo/dynamo 📖 Docs: docs.nvidia.com/dynamo

Apache-2.0 licensed, built in Rust for performance, Python for extensibility, at 7,600+ GitHub stars with 160+ community contributors, and a commit from August 1, 2026 — the day before this article.

When to Use It (and When Not To) #

The README is upfront about the boundary:

• You’re serving LLMs across multiple GPUs or nodes and need to coordinate them
• You want KV-aware routing to avoid redundant prefill computation
• You need to independently scale prefill and decode (disaggregated serving)
• You want automatic scaling that meets latency SLAs at minimum total cost of ownership
• You need fast cold-starts when spinning up new replicas

“If you’re running a single model on a single GPU, your inference engine alone is probably sufficient.” — that’s not a hedge, it’s the project telling you when not to add the complexity.

Backend Support #

Published Results (Each Sourced, Not Just Asserted) #

Each of these is attributed to a specific benchmark or source rather than presented as an unsourced headline number — worth checking the linked methodology before treating any single figure as your expected result.

Core Capabilities #

New in 1.0 #

• Zero-config deploy (DGDR, beta) — specify model, hardware, and SLA in one YAML; AIConfigurator profiles the workload and Planner optimizes topology automatically
• Agentic inference — per-request hints for priority, expected output length, speculative prefill, plus LangChain and NeMo Agent Toolkit integrations
• Multimodal E/P/D — disaggregated encode/prefill/decode with embedding cache, 30% faster TTFT on image workloads
• Video generation — native FastVideo + SGLang Diffusion support, real-time 1080p on a single B200
• K8s Inference Gateway plugin — KV-aware routing inside the standard Kubernetes gateway

Quick Start #

Container (fastest) #

docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.3.0

# Inside the container — start frontend and worker
python3 -m dynamo.frontend --http-port 8000 --discovery-backend file > /dev/null 2>&1 &
python3 -m dynamo.sglang --model-path Qwen/Qwen3-0.6B --discovery-backend file &

curl -s localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-0.6B",
  "messages": [{"role": "user", "content": "Hello!"}],
  "max_tokens": 100
}' | jq

tensorrtllm-runtime:1.3.0 and vllm-runtime:1.3.0 containers are available the same way.

PyPI install #

uv pip install --prerelease=allow "ai-dynamo[sglang]"   # or [vllm]

Kubernetes (recommended for production) #

# Zero-config deploy: specify model + SLA, Dynamo handles the rest
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: my-model
spec:
  model: Qwen/Qwen3-0.6B
  backend: vllm
  sla:
    ttft: 200.0   # ms
    itl: 20.0     # ms
  autoApply: true

Pre-built recipes exist for common models (Llama-3-70B on vLLM, DeepSeek-R1 on SGLang disaggregated, Qwen3-32B-FP8 on TensorRT-LLM), plus cloud-specific guides for AWS EKS, Google GKE, Azure AKS, and Amazon ECS.

Two Request-Routing Topologies #

Both expose an OpenAI-compatible API and support the same backends, disaggregated serving, and KV-aware routing — the choice is about where routing decisions live in your infrastructure, not a feature trade-off.

Use Cases #

1. Scaling Past a Single Node #

Once one GPU (or one node) can’t serve your traffic, Dynamo coordinates multiple nodes as one inference system instead of you hand-rolling load balancing across independent engine instances.

2. Cutting Redundant Prefill Compute #

KV-aware routing avoids recomputing prefill for requests that hit workers already holding relevant KV cache — directly behind the published 2× TTFT improvement.

3. Meeting Latency SLAs Without Overprovisioning #

The Planner’s SLA-driven autoscaling targets a specific TTFT/ITL budget rather than scaling on raw CPU/GPU utilization, aimed at the 80% fewer SLA breaches at lower TCO result.

4. Serving Reasoning, Multimodal, or Video Workloads at Scale #

Native support for disaggregated multimodal encode/prefill/decode and video generation (FastVideo, SGLang Diffusion) extends beyond text-only LLM serving.

Related Repositories #

Related Articles #

• SGLang: Structured Generation for LLMs — one of the three inference engines Dynamo orchestrates rather than replaces
• Self-Hosted LLM 2026: Ollama vs vLLM vs LocalAI — for single-node self-hosting, the scale below where Dynamo’s orchestration layer becomes relevant

Conclusion #

NVIDIA Dynamo targets a specific, well-defined gap: individual inference engines are excellent at serving one model efficiently on one GPU or node, but coordinating dozens of GPUs — disaggregating prefill from decode, routing around redundant KV computation, autoscaling to an SLA rather than a CPU graph — is a separate, harder problem most engines don’t solve themselves. Apache-2.0 licensed, backed by NVIDIA, and unusually disciplined about sourcing its own performance claims, it’s a serious option once you’ve outgrown “one engine, one GPU” — and the README itself tells you plainly when you haven’t.

Best for: Teams running LLM inference across multiple GPUs or nodes who need disaggregated serving, KV-aware routing, or SLA-driven autoscaling on top of vLLM, SGLang, or TensorRT-LLM — not single-GPU deployments.

GitHub: https://github.com/ai-dynamo/dynamo

Last updated: 2026-08-03

References & Sources #

• NVIDIA Dynamo on GitHub
• Dynamo documentation
• Dynamo Architecture Deep Dive
• Dynamo Feature Matrix