Deployment

The image

LayerScale ships as one Docker image on Docker Hub:

• layerscale/layerscale:latest
• layerscale/layerscale:latest-cuda

The two tags are the same image: the engine, the optional NVIDIA Dynamo worker, and the CUDA runtime libraries they link against.

The image exposes port 8080, takes the license key from LAYERSCALE_LICENSE_KEY (or --license-key), and runs serve 0.0.0.0:8080 --model /models unless given other arguments. Given a Hugging Face repo id as --model, the engine downloads it into /root/.cache/huggingface inside the container; mount that path from the host so the download is kept, and pass -e HF_TOKEN=hf_... for a gated model. See Models.

GPU requirements

The CUDA image carries native kernels for NVIDIA Hopper (compute capability 9.0) and the Blackwell generation (10.0, 10.3, 11.0, 12.0 and 12.1), plus forward-compatible PTX for newer parts. Hopper is the floor: the kernels rely on its shared memory size and asynchronous barriers, so earlier generations (Ampere, Ada and older) are not supported and fail with CUDA’s no kernel image is available for execution on the device.

For NVIDIA GPUs the host needs a driver that supports CUDA 13.3, and Docker with the NVIDIA Container Toolkit so --gpus all works. Nothing else comes from the host; the runtime libraries are inside the image.

Running

docker run -d --name layerscale --gpus all -p 8080:8080 \
  -e LAYERSCALE_LICENSE_KEY=LSK-... \
  -v /models:/models:ro \
  layerscale/layerscale:latest \
  serve 0.0.0.0:8080 --api-key --model /models/Devstral-Small-2507

Pass engine flags after the image name; see Configuration. --api-key makes clients send the license key as a bearer token; without it the API is open to anyone who can reach the port. The engine binds only after the weights are loaded and the warm-up has run, so a port that refuses connections for a while after start is expected. Watch docker logs -f layerscale for the ready line.

Multi-GPU

--tp N splits the model across GPUs 0 to N-1. Without it the engine uses every GPU visible in the container, so restrict either with --gpus '"device=0,1"' or with --tp. A free license permits one GPU, so tensor parallelism needs a Pro license: on a multi-GPU host, pass --tp 1 or the engine refuses to start.

# 119 GB of fp8 weights across two 80 GB cards. The checkpoint declares a
# million-token context, so pin --ctx to what the pool can hold.
docker run -d --gpus all -p 8080:8080 -e LAYERSCALE_LICENSE_KEY=LSK-... \
  -v /models:/models:ro \
  layerscale/layerscale:latest \
  serve 0.0.0.0:8080 --api-key --model /models/Mistral-Small-4-119B-2603 --tp 2 --ctx 16384

The GPUs must be able to read each other’s memory (NVLink or PCIe peer access), and the model’s heads, KV heads, intermediate size and vocabulary must divide by N.

Memory

The engine budgets --gpu-memory-utilization (default 0.9) of each card’s total memory for weights plus KV cache, and sizes the KV pool from what is left after the weights. Lower it when other processes share the card. --max-total-tokens pins the pool to an exact token count instead. --ctx bounds each sequence; a free license caps it at 32,768 tokens.

Health and readiness

GET /health (or /healthz) needs no key. It answers 200 with "status": "ok" while the engine thread is stepping or idle, and 503 with "status": "unavailable" if the engine has stopped or stalled. Use it for both liveness and readiness probes; before the engine is ready the port is not open at all. A fault the engine cannot recover from (at --tp above 1, a GPU that never arrives at a step) fails the requests in flight, turns /health to 503, closes the port a moment later and ends the process with a non-zero exit status within about a second, so run it under a supervisor that restarts it (restart: unless-stopped, a Kubernetes Deployment).

readinessProbe:
  httpGet:
    path: /health
    port: 8080
  periodSeconds: 5
livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 600
  periodSeconds: 15

Set the liveness delay to cover the model’s load and warm-up time.

Metrics

GET /metrics returns Prometheus text. Under --api-key it requires the key, so give the scraper a bearer token:

scrape_configs:
  - job_name: layerscale
    authorization:
      type: Bearer
      credentials: LSK-...
    static_configs:
      - targets: ["layerscale:8080"]

The metric names are listed in the API reference.

Logs

Every line on stdout is one JSON object with ts, level and event. There is a request line per API call with route, status, token counts, time to first token and total time, so a log pipeline can build latency dashboards without scraping anything else.

Shutdown

SIGTERM or SIGINT stops accepting connections, lets in-flight requests finish for up to 30 seconds, then exits. docker stop sends SIGTERM; give it a stop timeout of at least 30 seconds.

Behind a reverse proxy

• --api-key (the license key as a bearer token) is the only authentication, and it is off by default. Put the server behind TLS before exposing it, and do not bind serve to a public interface without a proxy in front.
• Streaming responses are Server-Sent Events. Turn off response buffering for the /v1/ paths (proxy_buffering off in nginx) or tokens arrive in bursts.
• Long generations exceed default upstream timeouts. Raise the proxy’s read timeout well past the longest expected response; the session event stream stays open indefinitely and sends a heartbeat every 15 seconds.
• /v1/sessions/{id}/ws needs the Upgrade and Connection headers forwarded.
• The engine expects a request to arrive in full within 60 seconds and accepts bodies up to 32 MiB; align the proxy’s limits.
• Health checks from the proxy or an orchestrator can hit /health without a key. Everything else forwards the client’s Authorization header unchanged.

An nginx location that covers all of the above:

location / {
    proxy_pass http://127.0.0.1:8080;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_buffering off;
    proxy_read_timeout 3600s;
    client_max_body_size 32m;
}

$connection_upgrade is the usual map $http_upgrade $connection_upgrade { default upgrade; '' close; } in the http block.

Behind NVIDIA Dynamo

To serve from many workers behind NVIDIA Dynamo’s frontend and KV-aware router, with prefill and decode on separate GPUs if wanted, start the same image with dynamo as its first argument (docker run ... layerscale/layerscale:latest dynamo --model /models/...). It then runs as a Dynamo worker instead of serving its own API. See NVIDIA Dynamo for the worker’s flags, discovery, the frontend and Kubernetes.