NVIDIA Dynamo

NVIDIA Dynamo is a framework for serving a model from many workers: an OpenAI-compatible HTTP frontend, a router that sends each request to the worker most likely to hold its prompt’s KV cache already, prefill/decode disaggregation, and a Kubernetes operator. Inference engines plug into it as workers.

Started with dynamo as its first argument, the LayerScale image runs the engine as a Dynamo worker: the same engine, behind Dynamo’s backend interface. Dynamo’s frontend takes the HTTP request, applies the chat template, tokenizes, and handles stop strings, tool-call parsing and reasoning parsing. The worker receives token ids and streams token ids back.

Opt-in

The worker ships in the same layerscale/layerscale image and runs only when the container’s first argument is dynamo; every argument after it goes to the worker. Without that argument, the image runs layerscale exactly as before, with no new flags. Without Dynamo, nothing in this page applies.

Behind Dynamo, the HTTP API is Dynamo’s frontend, not LayerScale’s server. The sessions API and the other LayerScale routes are not available through it, and authentication belongs in front of the frontend.

Sessions are not available through Dynamo: the frontend does not forward x-layerscale-session-id, so every request reaches the worker as a stateless turn. A session turn needs layerscale serve.

The worker links Dynamo 1.5.0. Run it with the frontend of the same release: pip install ai-dynamo==1.5.0, or the nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.5.0 image.

Running locally

Dynamo needs no etcd or NATS on one machine: workers and the frontend find each other through a directory (DYN_DISCOVERY_BACKEND=file), and requests travel over TCP.

# The worker. DYN_SYSTEM_PORT turns on its /live, /health and /metrics.
docker run -d --gpus all --network host --stop-timeout 30 \
  -e LAYERSCALE_LICENSE_KEY=LSK-... \
  -e DYN_DISCOVERY_BACKEND=file -e DYN_FILE_KV=/kv -v /tmp/dynamo-kv:/kv \
  -e DYN_SYSTEM_PORT=9090 \
  -v /models:/models:ro \
  layerscale/layerscale:latest \
  dynamo --model /models/Qwen3-8B --served-model-name Qwen/Qwen3-8B

# The frontend, on port 8000, with the KV-aware router, sharing the discovery directory.
pip install ai-dynamo==1.5.0
DYN_DISCOVERY_BACKEND=file DYN_FILE_KV=/tmp/dynamo-kv \
  python -m dynamo.frontend --http-port 8000 --router-mode kv

curl http://127.0.0.1:8000/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3-8B","messages":[{"role":"user","content":"hi"}],"max_tokens":64}'

A worker takes longer than Docker’s default 10 seconds to stop: it first leaves discovery and waits out Dynamo’s 5-second grace period, then frees its GPUs. --stop-timeout 30 (or docker stop -t 30) lets it exit cleanly instead of being killed, which would leave its entry behind in the discovery directory. Kubernetes’ default 30-second grace period is enough.

The container runs as root, so the files it writes in the discovery directory are root’s. Create the directory first and make it writable for the frontend’s user (mkdir -p /tmp/dynamo-kv && chmod a+rwX /tmp/dynamo-kv), or run the frontend as root.

The worker registers with Dynamo only after the weights are loaded and the warm-up has run. Until then the frontend does not list the model, and /live and /health answer 503.

Dynamo reads the tokenizer and chat template from --model: the directory, or for a Hugging Face repo id, the Hugging Face cache the engine downloads it into. Pass --served-model-name to choose the name clients send. Without it, the name is the model’s own.

Flags

docker run --rm -e LAYERSCALE_LICENSE_KEY=LSK-... layerscale/layerscale:latest dynamo --help prints both groups (the image checks the license before it starts anything, help included).

Engine options. Every option in Configuration works the same way here: --model, --license-key, --ctx, --max-num-seqs, --gpu-memory-utilization, --max-total-tokens, --chunked-prefill-size, --kv-cache-dtype, --device, --tp, --no-prefix-cache and --speculative-algorithm. --api-key is accepted and does nothing, because the worker serves no HTTP API.

Worker options. Dynamo’s own. Each one can also be set through its environment variable, which is how the Kubernetes operator passes them.

Flag	Environment	Default	Meaning
`--served-model-name <NAME>`		the model’s name	The model name clients ask for.
`--disaggregation-mode <M>`	`DYN_DISAGGREGATION_MODE`	`agg`	`agg` runs whole requests. `prefill` and `decode` split them; see Disaggregated serving. `encode` is refused, because LayerScale has no separate encoder.
`--namespace <NS>`	`DYN_NAMESPACE`	`dynamo`	Dynamo namespace. The frontend serves the workers in its own namespace.
`--component <NAME>`	`DYN_COMPONENT`	`backend`	Component name within the namespace.
`--endpoint <NAME>`	`DYN_ENDPOINT`	`generate`	Endpoint name.
`--endpoint-types <LIST>`	`DYN_ENDPOINT_TYPES`	`chat,completions`	Which frontend routes serve this model.
`--custom-jinja-template <PATH>`	`DYN_CUSTOM_JINJA_TEMPLATE`	none	A chat template to use instead of the model’s.
`--dyn-tool-call-parser <NAME>`	`DYN_TOOL_CALL_PARSER`	none	The frontend’s tool-call parser for this model.
`--dyn-reasoning-parser <NAME>`	`DYN_REASONING_PARSER`	none	The frontend’s reasoning parser for this model.
`--exclude-tools-when-tool-choice-none <B>`	`DYN_EXCLUDE_TOOLS_WHEN_TOOL_CHOICE_NONE`	`true`	Leave tools out of the chat template when `tool_choice` is `none`.
`--enable-rl`	`DYN_ENABLE_RL`	off	Dynamo’s reinforcement-learning control endpoint.

Dynamo’s runtime reads its environment directly:

Variable	Meaning
`DYN_DISCOVERY_BACKEND`	`etcd` (default), `kubernetes`, `file` or `mem`.
`DYN_FILE_KV`	The directory for `file` discovery.
`ETCD_ENDPOINTS`, `NATS_SERVER`	Where etcd and NATS are, when used.
`DYN_REQUEST_PLANE`	`tcp` (default) or `nats`.
`DYN_SYSTEM_PORT`	The port for `/live`, `/health` and `/metrics`. Off unless set.
`POD_IP`	The address a prefill worker gives decode workers to pull KV from. Without it, the address of the default route.

What a request may ask for

The frontend turns an OpenAI request into token ids and sampling options. The worker applies max_tokens, temperature, top_p, seed, ignore_eos, and logprobs with up to 20 top_logprobs. A request with no max_tokens runs until end of sequence or the context limit.

The frontend applies stop strings and stop token ids. The worker refuses the following with a 400 instead of ignoring them: n or best_of above 1, beam search, top_k, min_p, frequency, presence or repetition penalties, guided decoding, min_tokens, max_thinking_tokens, prompt logprobs, image input and prompt embeddings.

A streamed response without logprobs arrives as the engine produces it, step by step. With logprobs, the tokens and their logprobs all arrive in the final chunk; since the frontend sees no tokens before then, the worker also stops such a request on its stop strings, and the frontend still trims the text and sets the finish reason.

Metrics and health

Dynamo owns the worker’s HTTP surface, on DYN_SYSTEM_PORT:

/live and /health answer 503 until the engine is loaded and warmed up, then 200. A worker whose engine stops exits with a non-zero status, so its supervisor restarts it.
/metrics has Dynamo’s dynamo_component_* families and, after them, every layerscale_* family that layerscale serve exports, under the same names and labels. See the API reference. Requests served through Dynamo count in the request, token and latency families as they would on the HTTP API.
The worker also reports its KV pool to Dynamo, for rank 0: dynamo_component_total_blocks, dynamo_component_gpu_cache_usage_percent and dynamo_component_kv_cache_hit_rate. The router uses the same numbers as a load signal.

Engine log lines are the same JSON objects as from layerscale serve, mixed with Dynamo’s own log lines.

KV-aware routing

With --router-mode kv on the frontend (DYN_ROUTER_MODE=kv), the router sends each request to the worker that already holds the most of its prompt in its prefix cache. Each worker tells the router what it holds. A page is reported when it enters the prefix cache as its request finishes, and again as removed when the pool evicts it.

Dynamo routes in blocks of 16 tokens, which is LayerScale’s KV page size, so a router block is one page. The worker registers that size, along with its context length, pool size and batch limits, when it joins.

The router starts empty for a worker that restarts: the worker tells it to forget what it held before. Pages written during the warm-up are not reported. With --no-prefix-cache the worker keeps no prefix cache, so it reports nothing and the router places its requests by load alone.

Disaggregated serving

Prefill and decode can run on separate workers, so a long prompt never stalls the decode steps of other requests:

worker() {  # worker <gpu> <system port> <mode>
  docker run -d --gpus device=$1 --network host --stop-timeout 30 \
    -e LAYERSCALE_LICENSE_KEY=LSK-... \
    -e DYN_DISCOVERY_BACKEND=file -e DYN_FILE_KV=/kv -v /tmp/dynamo-kv:/kv \
    -e DYN_SYSTEM_PORT=$2 \
    -v /models:/models:ro \
    layerscale/layerscale:latest \
    dynamo --model /models/Qwen3-8B --served-model-name Qwen/Qwen3-8B --disaggregation-mode $3
}
worker 0 9091 prefill
worker 1 9092 decode
DYN_DISCOVERY_BACKEND=file DYN_FILE_KV=/tmp/dynamo-kv \
  python -m dynamo.frontend --http-port 8000 --router-mode kv

On one machine, give each worker its own GPU, as above.

How a request runs:

The frontend sends the request to a prefill worker. The worker computes the prompt’s KV, copies it to host memory, frees its GPU pages, and answers with where to fetch the copy. It keeps the copy until a decode worker fetches it, or its decode request is cancelled first, for at most 60 seconds.
The frontend sends the request on to a decode worker. The decode worker pulls the KV over a TCP connection straight from the prefill worker, writes it into its own cache, and generates. It recomputes the prompt’s last token, so it samples the first output token itself.
A health-check request, or a request the router marks to run whole, runs start to finish on whichever worker gets it.

Under load:

A prefill worker refuses new requests as overloaded (the frontend answers 503) while the KV waiting in its host memory has reached its cap: its own KV pool’s size, and at most half the host memory free when it started. That only happens when decode workers are not fetching; prompts queued on the prefill worker do not count.
A decode worker without free pages for the KV runs the whole prompt itself instead of failing the request.
A transfer connection is closed after 60 seconds.
usage.prompt_tokens_details.cached_tokens reports the prompt tokens a worker did not compute: prefix-cache hits on an aggregated worker, the transferred KV on a decode worker.

Requirements:

Both workers load the same model with the same --tp and the same --kv-cache-dtype. The decode worker checks the KV’s layout, which covers the model’s configuration, the page geometry, the cache precision and an fp8 cache’s scales, and refuses a mismatch before allocating anything.
Models with recurrent layers beside attention, such as Gated DeltaNet hybrids, keep state outside the paged KV cache. That state cannot be moved, so a worker in prefill or decode mode refuses to start with such a model. Serve those models aggregated.
Decode workers must reach prefill workers over TCP, on a port the prefill worker picks when it starts. Allow pod-to-pod traffic between them.
The transfer is neither authenticated nor encrypted. A transfer id is random, 64 bits long, works once and expires after 60 seconds. Keep the workers on a private network.

The KV travels through host memory on both sides (GPU to host, TCP, host to GPU), so a transfer costs time in proportion to the prompt’s KV size.

Kubernetes

With the Dynamo operator installed, deploy with a DynamoGraphDeployment. Two manifests are in the repository:

dynamo/deploy/agg.yaml: a frontend with the KV-aware router and one aggregated worker.
dynamo/deploy/disagg.yaml: a frontend, a prefill worker and a decode worker.

kubectl create secret generic layerscale-license --from-literal=license-key=LSK-...
# Set the worker image to yours, put the model on the model-cache PVC, then:
kubectl apply -f dynamo/deploy/agg.yaml
kubectl port-forward svc/layerscale-agg-frontend 8000:8000

What the manifests do and why:

No backendFramework. The operator recognises the engines that ship with Dynamo by their launch command. For any other command it applies no engine-specific changes, which is what this worker needs. It still adds the discovery settings, DYN_NAMESPACE, DYN_COMPONENT, DYN_SYSTEM_PORT=9090 and the probes on /live and /health, and the worker reads all of them.
runtimeVersionOverride: "1.5.0" on every worker. The operator reads the Dynamo version from the image tag, and a LayerScale image tag is not one. The override declares the Dynamo release the worker links. Without it, the operator refuses an image tag that is not a version, or misreads one that is (a 0.1.0 tag would turn off the health checks and tighten the liveness probe).
The image is the LayerScale image, and the command is /app/layerscale-loader, its entrypoint. Set the command: without one, the operator starts the container with a shell. The args start with dynamo, which selects the worker, followed by the worker’s flags. Name the container main.
The license key comes from the layerscale-license Secret, as LAYERSCALE_LICENSE_KEY.
The model is a directory on a PersistentVolumeClaim named model-cache, mounted at /models. To load a Hugging Face repo id instead, pass it as --model, mount the claim at /root/.cache/huggingface so the download is kept, and set HF_TOKEN from a Secret for a gated repo.
nvidia.com/gpu limits set how many GPUs a worker sees. --tp defaults to all of them. One worker is one pod: tensor parallelism stays within the pod, and multi-node workers are not supported.
POD_IP on the prefill worker, from the pod’s status, is the address decode workers pull KV from.

Scale with replicas on each component. The operator’s startup probe gives a worker two hours to download, load and warm up; a startupProbe on the main container replaces it.

Profiling and SLA planning

Dynamo’s DynamoGraphDeploymentRequest profiles a model, picks a configuration for a latency target, and generates the deployment. Its backend field accepts only the engines bundled with Dynamo, and its profiler sizes deployments from performance models of those engines. LayerScale is not one of them, and that list cannot be extended without changing Dynamo itself. Dynamo’s SLA planner depends on the same profiling data.

Deploy LayerScale with a DynamoGraphDeployment, as above, and size replicas and the prefill/decode split from your own measurements.

Known limits

Sessions. The sessions API is not available through Dynamo. Chat turns could be pinned to a session’s worker, but its data plane (pushed data, flash queries, the event streams) has no route through Dynamo’s frontend, so the worker serves stateless requests only. Use layerscale serve for sessions.
KV block manager. Dynamo’s KVBM, which offloads KV blocks to host memory or disk and brings them back, is not integrated. The worker’s KV cache is LayerScale’s own paged pool on the GPU, with its own prefix cache.