LayerScale
LayerScale is an inference server for open-weight language models. It is one binary, shipped as a Docker image, that loads a Hugging Face checkpoint from a local directory and serves it over HTTP on NVIDIA Hopper and Blackwell GPUs.
What it serves
POST /v1/chat/completions, the OpenAI chat API, with SSE streaming and tool calling.POST /v1/messages, the Anthropic Messages API, with streaming and tool use.- Stateful sessions: a conversation whose KV cache stays on the GPU between requests, so a turn costs only the tokens it adds.
- Background data ingestion into a session, flash queries the engine keeps answered as data arrives, and an event stream over SSE or WebSocket.
- Prometheus metrics and health probes.
Every request parameter is either implemented or refused by name with a 400. Nothing is silently dropped.
What you need
- An NVIDIA GPU of the Hopper or Blackwell generation and a driver that supports CUDA 13.3.
- Docker with the NVIDIA Container Toolkit.
- A model directory of Hugging Face safetensors. The engine does not download models.
- A license key from layerscale.ai/get-license. The free key runs one GPU, one session and a 32,768-token context.
Contents
| Page | What it covers |
|---|---|
| Getting started | Key, docker run, first request |
| Configuration | Every command line flag and its default |
| Models | Directory layout, quantized checkpoints, KV cache precision, tensor parallelism |
| API reference | Authentication, errors, every stateless route |
| Sessions | The stateful API: turns, data push, flash queries, events, WebSocket |
| Deployment | Image tags, GPU requirements, multi-GPU, probes, reverse proxies |
| Troubleshooting | Startup and request errors and what they mean |
| Pricing | Free and Pro license tiers |
Performance figures are published at layerscale.ai/benchmarks.
Client libraries
- Python:
pip install layerscale - TypeScript:
npm install @layerscale/layerscale - OpenCode plugin:
opencode-plugin-layerscale
Both clients cover every route above, sessions and the WebSocket included. The OpenAI and Anthropic SDKs work against the two compatible routes: set the base URL to the server and the API key to the license key.