Configuration

LayerScale is configured entirely on the command line. There is no configuration file. In the Docker image, everything after the image name is passed to the engine; the default is serve 0.0.0.0:8080 --model /models.

layerscale [options] serve [ADDR]
layerscale [options] generate <REQUEST>
layerscale [options] cli-interactive

--help prints the same reference. Flags may come before or after the command.

Commands

Command	What it does
serve [ADDR]	Start the HTTP API on ADDR. Default 127.0.0.1:8080. Use 0.0.0.0:8080 inside a container.
generate <REQUEST>	Run one chat completion and print it. REQUEST is the JSON body of /v1/chat/completions (messages, temperature, max_tokens and so on).
cli-interactive	Chat with the model at a prompt. /max and /temp set sampling; /help lists the commands.

Every command needs --model and a license key.

Options

The flags use the names common among inference servers (--kv-cache-dtype, --device, --gpu-memory-utilization), so an existing launcher carries over with few changes.

Flag	Default	Meaning
--license-key <KEY>	LAYERSCALE_LICENSE_KEY	The LSK-... license key. Required, as the flag or the environment variable.
--api-key	off	Require the license key as the API bearer token (Authorization: Bearer <LSK-...> or x-api-key) on every route except /health, /healthz and OPTIONS. Without it the API is open to anyone who can reach the address.
--model <DIR|ORG/NAME>	none, required	Model directory with config.json, tokenizer.json (or tekken.json) and either model.safetensors or model.safetensors.index.json plus its shards, or a Hugging Face repo id, loaded from the local Hugging Face cache (HF_HUB_CACHE, $HF_HOME/hub, ~/.cache/huggingface/hub) and downloaded into it first when it is not there. A gated repo needs HF_TOKEN; see Models.
--ctx <N>	the model’s max_position_embeddings, clamped to what the KV pool holds for one sequence	Context length in tokens per sequence. The clamp is logged as ctx_defaulted; the license may cap it further.
--max-num-seqs <N>	128, or fewer when the KV pool cannot back 1024 tokens each (4 on the CPU)	Maximum sequences batched together.
--gpu-memory-utilization <F>	0.9	Fraction of each GPU’s total memory for weights plus KV cache. The denominator is the card’s total memory, not what is free when the engine starts.
--max-total-tokens <N>	derived from the memory budget	Exact KV pool size in tokens across all sequences, overriding --gpu-memory-utilization. A smaller pool saves memory and may preempt sequences.
--chunked-prefill-size <N>	2048	The most of one request’s prompt prefilled in one forward step. A burst of prompts packs several chunks into one step: a dense model’s step is the chunk, a routed model’s up to four chunks (chunk × experts / top_k, capped at 8192; 4096 with INT4 experts, whose decode rows ride beside the chunks). Larger prefills a long prompt faster and makes every decode row wait longer behind it.
--kv-cache-dtype <T>	auto	auto, bfloat16, fp8 or fp8_e4m3. auto stores the model’s dtype unless the checkpoint calibrated an fp8 KV cache, which is then stored in fp8 with its own scales. fp8 halves the pool, so the same card holds twice the context; it needs head_dim 128 and a checkpoint with no sliding-window layers. fp8_e5m2 is recognised and refused.
--device <NAME>	the GPU the build has	auto, cpu, cuda or rocm. Asking for a backend the build lacks is an error, not a fallback.
--tp <N>	every GPU the host shows	Split one copy of the model across GPUs 0..N-1. 1 disables tensor parallelism. Above 1 needs a Pro license. See Models.
--no-prefix-cache	off, so caching is on	Do not share KV pages between requests with a common prefix.
--speculative-algorithm <A>	ngram	ngram or none; any other value is a startup error. Prompt-lookup speculative decoding: a greedy decode drafts the continuation of text already seen in the context and confirms it in one forward pass. none is a bit-exact decode.

Notes

Memory. By default the engine takes --gpu-memory-utilization of the scarcest card’s total memory, subtracts the weights, and gives the rest to the KV pool. --max-num-seqs then defaults to 128, or fewer when the pool cannot back 1024 tokens per sequence. If the pool cannot hold one full context the engine refuses to start; raise --gpu-memory-utilization or --max-total-tokens, or lower --ctx.

Context. A request whose prompt reaches the context limit is refused with a 400 that names the limit. A free license clamps the context at 32,768 tokens whatever --ctx says.

Tensor parallelism. Without --tp the engine uses every GPU it can see. A license that permits fewer GPUs than the host shows refuses to start until --tp is passed at or under the cap. A free license permits one GPU, so tensor parallelism needs a Pro license; on a multi-GPU host pass --tp 1.

License caps. The license log line at startup prints the caps in force (max_sessions, max_ctx, max_gpus; null means unlimited). The free tier caps are one live session, a 32,768-token context and one GPU, so tensor parallelism needs a Pro license; see layerscale.ai/pricing.

Speculative decoding. The engine runs with --speculative-algorithm ngram unless told otherwise. Pass none when outputs must be reproducible bit for bit across runs, for example when comparing engines.

Examples

# Serve on all interfaces with a 16k context on two GPUs
layerscale --model /models/Mistral-Small-4-119B-2603 --ctx 16384 --tp 2 serve 0.0.0.0:8080

# One completion, greedy
layerscale --model /models/Devstral-Small-2507 generate \
  '{"messages":[{"role":"user","content":"hi"}],"temperature":0,"max_tokens":64}'

# The same server in Docker
docker run --gpus all -p 8080:8080 -e LAYERSCALE_LICENSE_KEY=LSK-... \
  -v /models:/models:ro \
  layerscale/layerscale:latest \
  serve 0.0.0.0:8080 --model /models/Mistral-Small-4-119B-2603 --ctx 16384 --tp 2

Logs

Every log line is one JSON object on stdout with ts, level and event fields. Startup emits engine, license, loading_model, model_loaded, engine_config and ready, plus ctx_defaulted when the checkpoint’s context ceiling was larger than the pool. Every API call emits a request line with the route, status, token counts, time to first token and total time.