Configuration

LayerScale is configured entirely on the command line. There is no configuration file. In the Docker image, everything after the image name is passed to the engine; the default is serve 0.0.0.0:8080 --model /models.

layerscale [options] serve [ADDR]
layerscale [options] generate <REQUEST>
layerscale [options] cli-interactive

--help prints the same reference. Flags may come before or after the command.

Commands

Command	What it does
serve [ADDR]	Start the HTTP API on ADDR. Default 127.0.0.1:8080. Use 0.0.0.0:8080 inside a container.
generate <REQUEST>	Run one chat completion and print it. REQUEST is the JSON body of /v1/chat/completions (messages, temperature, max_tokens and so on).
cli-interactive	Chat with the model at a prompt. /max and /temp set sampling; /help lists the commands.

Every command needs --model and a license key.

Options

Flag	Default	Meaning
--license-key <KEY>	LAYERSCALE_LICENSE_KEY	The LSK-... license key. Required, as the flag or the environment variable. The same key is the API bearer token.
--model <DIR>	none, required	Model directory with config.json, tokenizer.json (or tekken.json) and either model.safetensors or model.safetensors.index.json plus its shards.
--ctx <N>	the model’s max_position_embeddings	Context length in tokens per sequence. The license may cap it.
--seqs <N>	as many full contexts as the KV pool holds, up to 256 (4 on the CPU)	Maximum sequences batched together.
--gpu-memory-utilization <F>	0.9	Fraction of each GPU’s total memory for weights plus KV cache. The denominator is the card’s total memory, not what is free when the engine starts.
--max-total-tokens <N>	derived from the memory budget	Exact KV pool size in tokens across all sequences, overriding --gpu-memory-utilization. A smaller pool saves memory and may preempt sequences.
--chunked-prefill-size <N>	1024	Maximum tokens batched into one forward step, and therefore the prefill chunk size.
--kv-cache-dtype <T>	auto	auto, bfloat16, fp8 or fp8_e4m3. auto stores the model’s dtype unless the checkpoint calibrated an fp8 KV cache, which is then stored in fp8 with its own scales. fp8 halves the pool, so the same card holds twice the context; it needs head_dim 128. fp8_e5m2 is recognised and refused.
--device <NAME>	the GPU the build has	auto, cpu, cuda or rocm. Asking for a backend the build lacks is an error, not a fallback.
--tp <N>	every GPU the host shows	Split one copy of the model across GPUs 0..N-1. 1 disables tensor parallelism. See Models.
--no-prefix-cache	off, so caching is on	Do not share KV pages between requests with a common prefix.
--speculative-algorithm <A>	default	default or none. Prompt-lookup speculative decoding: a greedy decode drafts the continuation of text already seen in the context and confirms it in one forward pass. none is a bit-exact decode.

Notes

Memory. By default the engine takes --gpu-memory-utilization of the scarcest card’s total memory, subtracts the weights, and gives the rest to the KV pool. --seqs then defaults to how many full contexts fit, up to 256. If the pool cannot hold one full context the engine refuses to start; raise --gpu-memory-utilization or --max-total-tokens, or lower --ctx.

Context. A request whose prompt reaches the context limit is refused with a 400 that names the limit. A free license clamps the context at 32,768 tokens whatever --ctx says.

Tensor parallelism. Without --tp the engine uses every GPU it can see. A license that permits fewer GPUs than the host shows refuses to start until --tp is passed at or under the cap.

License caps. The license log line at startup prints the caps in force (max_sessions, max_ctx, max_gpus; null means unlimited). Every feature is available on both tiers; see layerscale.ai/pricing.

Speculative decoding. The engine runs with --speculative-algorithm default unless told otherwise. Pass none when outputs must be reproducible bit for bit across runs, for example when comparing engines.

Examples

# Serve on all interfaces with a 16k context on two GPUs
layerscale --model /models/Mistral-Small-4-119B-2603 --ctx 16384 --tp 2 serve 0.0.0.0:8080

# One completion, greedy
layerscale --model /models/Devstral-Small-2507 generate \
  '{"messages":[{"role":"user","content":"hi"}],"temperature":0,"max_tokens":64}'

# The same server in Docker
docker run --gpus all -p 8080:8080 -e LAYERSCALE_LICENSE_KEY=LSK-... \
  -v /models:/models:ro \
  layerscale/layerscale:latest \
  serve 0.0.0.0:8080 --model /models/Mistral-Small-4-119B-2603 --ctx 16384 --tp 2

Logs

Every log line is one JSON object on stdout with ts, level and event fields. Startup emits engine, license, loading_model, model_loaded, engine_config and ready. Every API call emits a request line with the route, status, token counts, time to first token and total time.