API reference

The server speaks HTTP/1.1 with keep-alive, chunked request bodies and Server-Sent Events for streaming. The base URL is wherever serve bound, http://127.0.0.1:8080 by default.

Routes

Method	Path	Purpose
POST	/v1/chat/completions	OpenAI chat completion, streaming and tool calling
POST	/v1/messages	Anthropic message, streaming and tool use
GET	/v1/models	The loaded model
GET	/v1/models/{id}	The loaded model, 404 for any other id
GET	/health, /healthz	Readiness, no key needed
GET	/metrics	Prometheus text
OPTIONS	any	CORS preflight, no key needed
POST, GET	/v1/sessions	Create and list sessions
GET, DELETE	/v1/sessions/{id}	Inspect and delete a session
POST	/v1/sessions/{id}/generate	Ask a question over a session’s context
POST	/v1/sessions/{id}/push	Queue data for background ingestion
POST, GET	/v1/sessions/{id}/flash	Register and list flash queries
DELETE	/v1/sessions/{id}/flash/{fid}	Remove a flash query
GET	/v1/sessions/{id}/events	The session’s event stream (SSE)
GET	/v1/sessions/{id}/ws	The session’s event stream and push channel (WebSocket)

The session routes are described on the Sessions page. A trailing slash is ignored. A known path with the wrong method answers 405; an unknown path answers 404.

Authentication

By default the API is open: no route needs a key. Started with --api-key, every route except OPTIONS, /health and /healthz requires the license key the server was started with:

Authorization: Bearer LSK-...

x-api-key: LSK-... is accepted too, which is what Anthropic SDKs send. Dashes and whitespace inside the key are ignored. A missing or wrong key answers 401 with missing or invalid license key; send it as 'Authorization: Bearer <LSK-...>', in the envelope of the route (Anthropic’s for /v1/messages, OpenAI’s elsewhere). A WebSocket upgrade without the key is refused with a plain HTTP 401 before the handshake. Without --api-key any Authorization or x-api-key header is ignored.

Errors

OpenAI routes, the session routes included:

{"error": {"message": "'top_k' is not supported", "type": "invalid_request_error"}}

Status	type
401	authentication_error
402, 403	permission_error
400, 404, 405, 408, 409, 413, 431	invalid_request_error
429	rate_limit_error
5xx	server_error

/v1/messages uses Anthropic’s envelope:

{"type": "error", "error": {"type": "invalid_request_error", "message": "'max_tokens' is required"}}

with authentication_error (401), permission_error (403), not_found_error (404), rate_limit_error (429), invalid_request_error (other 4xx), overloaded_error (529) and api_error (5xx).

On a streaming response the status line has already gone out, so an engine error arrives as an event: an OpenAI error object as a data: frame followed by data: [DONE], or an Anthropic event: error frame.

Request parameters the engine cannot honour are refused by name with a 400. The exceptions are model, which is accepted and ignored on every route, and metadata on /v1/messages.

POST /v1/chat/completions

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Authorization: Bearer $LAYERSCALE_LICENSE_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "system", "content": "Be brief."},
      {"role": "user", "content": "Why is the sky blue?"}
    ],
    "max_tokens": 200,
    "temperature": 0.2
  }'

Request

Field	Default	Notes
messages	required	Roles system, user, assistant, tool (ipython is accepted as tool). content is a string or an array of {"type": "text", "text": ...} parts; other part types are refused.
max_tokens or max_completion_tokens	none	Left out, the reply runs until it stops or the context is full (finish_reason: "length").
temperature	0.6	0 is greedy.
top_p	0.9
seed	none
stop	none	A string, or an array of up to 4 strings of at most 64 bytes each. Excluded from the returned text.
stream	false
stream_options	none	{"include_usage": true} appends a final usage chunk.
tools	none	{"type": "function", "function": {"name", "description", "parameters"}}.
tool_choice	auto	auto, none, required, or {"type": "function", "function": {"name": ...}}. required and a named function put the call’s opening in the prompt (Qwen3 and Mistral templates; refused by name on the others) and run the turn without thinking: enable_thinking: true or a reasoning_effort of medium and up beside one is refused.
parallel_tool_calls	true	false: at most one tool call this turn. The turn ends where the call does, one decode step sooner.
reasoning_effort	none	none, minimal, low (no thinking) or medium, high, xhigh (thinking), for templates that render a reasoning setting.
chat_template_kwargs	none	date_string sets the template’s date; enable_thinking (a bool) turns a reasoning checkpoint’s thinking block on or off (on Qwen3 false closes an empty think block in the generation prompt, as the reference does), and an explicit reasoning_effort wins over it. add_generation_prompt and tools_in_user_message are accepted only as true. Other keys are refused.
ignore_eos	false	Keep generating past end of sequence, to exactly max_tokens.
logprobs	false	Report every generated token’s logprob in choices[0].logprobs (see below).
top_logprobs	none	0 to 20: also list that many of each position’s most probable tokens. Needs logprobs: true.

Assistant messages may carry tool_calls and reasoning_content; tool messages carry tool_call_id. Refused by name: n or best_of other than 1, top_logprobs above 20 or without logprobs: true, logit_bias, top_k, min_p, repetition_penalty, a nonzero presence_penalty or frequency_penalty, a response_format other than text, and the deprecated functions and function_call.

The x-layerscale-session-id header runs the turn against a session. A session_id body field is refused with 400.

Response

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "created": 1757000000,
  "model": "devstral-small-2507",
  "choices": [{
    "index": 0,
    "message": {"role": "assistant", "content": "..."},
    "logprobs": null,
    "finish_reason": "stop"
  }],
  "usage": {"prompt_tokens": 31, "completion_tokens": 57, "total_tokens": 88}
}

finish_reason is stop, length or tool_calls. logprobs is null unless the request asked for it; then it is {"content": [...]} with one entry per completion token, in order:

{"token": " Paris", "logprob": -0.0123, "bytes": [32, 80, 97, 114, 105, 115],
 "top_logprobs": [{"token": " Paris", "logprob": -0.0123, "bytes": [32, 80, 97, 114, 105, 115]}, ...]}

The logprobs are the model’s raw log-softmax, before temperature and top_p, so they are the same numbers at any temperature. top_logprobs is sorted most probable first and is empty when the request did not ask for alternatives. token is the token’s bytes read as UTF-8, with U+FFFD for a byte-fallback piece of a character; bytes are the token’s own. Tokens a tool call’s scaffolding forced into the reply were not sampled, and report logprob 0 and no alternatives. A request asking for logprobs decodes in the same steps as any other, at any --tp; its cost is the copy of its rows’ numbers back to the host. On a session turn, usage.prompt_tokens_details.cached_tokens reports the prompt tokens served from held KV; it is present only when nonzero.

Streaming

With "stream": true the response is text/event-stream. Each data: frame is a chat.completion.chunk: first a delta of {"role": "assistant"}, then content deltas, then a final chunk carrying finish_reason, then data: [DONE]. With logprobs, every entry arrives on that final chunk as choices[0].logprobs.content (the engine streams text, not token ids); OpenAI’s SDKs accumulate the list across chunks, so the result is the same list a non-streaming reply carries. With include_usage, every chunk carries "usage": null and one more chunk with empty choices and the totals precedes [DONE]. No bytes are sent while the prompt is being prefilled.

Tool calling

Offer tools and read finish_reason: "tool_calls":

"message": {
  "role": "assistant",
  "content": null,
  "tool_calls": [{
    "id": "call_...",
    "type": "function",
    "function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"}
  }]
}

arguments is a JSON string. Text ahead of the first call is returned as content; with no text it is null, and text between or after calls is not returned. Send the result back as a tool message with the same tool_call_id. When streaming, each call arrives as two chunks: one with id, type, the function name and empty arguments, then one with the complete arguments string. They are sent the moment the model closes the call (its closing marker on Qwen3 and Qwen3.5, the end of its arguments object on Mistral), before the finish chunk, so a reply with several calls streams each as it closes; a call is never streamed as content. Llama’s bare JSON call goes out when the reply ends, and so does gpt-oss’s, whose <|call|> ends the reply; a gpt-oss reply that does not call streams as it is written.

With tool_choice required or a named function the call’s opening goes into the prompt: Qwen3’s <tool_call> and its JSON up to the name (required) or through "arguments": (a named function), Mistral’s [TOOL_CALLS] (and the name and [ARGS] for a named function). The model’s first token is already the call’s, and those tokens count as prompt tokens, not completion tokens. The reply is the call, finish_reason: "tool_calls" (OpenAI reports stop); a forced call the model botches comes back as content beginning with the opening, finish_reason: "stop", and counts in tool_calls_malformed_total. required with one tool on offer is that tool, named. A named tool that is not in tools, or tools empty, is refused.

With parallel_tool_calls: false the turn ends where its first call does (its closing marker, or the end of its arguments object for a format with none): the step that would sample what follows the call never runs, and the reply carries that one call. Where the format gives the engine no such point (gpt-oss, Qwen3.5, Llama 4) the reply is cut to the first call instead. On a session the call’s last token, and on Qwen3 the closer forced after it, have no KV yet; the returning turn prefills them with the tool result.

POST /v1/messages

The Anthropic Messages API over the same engine.

curl http://127.0.0.1:8080/v1/messages \
  -H "x-api-key: $LAYERSCALE_LICENSE_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "system": "Be brief.",
    "messages": [{"role": "user", "content": "Why is the sky blue?"}],
    "max_tokens": 200
  }'

Request

Field	Default	Notes
messages	required	Roles user and assistant. content is a string or an array of blocks: text and tool_result in user messages; text, tool_use and thinking in assistant messages.
system	none	A string or an array of text blocks.
max_tokens	required
temperature	0.6
top_p	0.9
seed	none
stop_sequences	none	An array of strings.
stream	false
tools	none	{"name", "description", "input_schema"}.
tool_choice	{"type": "auto"}	auto, none, any, or {"type": "tool", "name": ...}. any and a named tool open the call in the prompt, as required and a named function do on /v1/chat/completions, and stop_reason is tool_use. "disable_parallel_tool_use": true beside the type asks for at most one tool call: the turn ends where the call does, one decode step sooner.
ignore_eos	false
metadata		Accepted and ignored.

top_k and thinking are refused. The session header works as on /v1/chat/completions.

Response

{
  "id": "msg_...",
  "type": "message",
  "role": "assistant",
  "model": "devstral-small-2507",
  "content": [{"type": "text", "text": "..."}],
  "stop_reason": "end_turn",
  "stop_sequence": null,
  "usage": {"input_tokens": 31, "output_tokens": 57}
}

stop_reason is end_turn, max_tokens, stop_sequence or tool_use. A tool call is a tool_use block with id, name and input (the arguments as the model wrote them when they are valid JSON, so a 19-digit id stays exact; normalized when only a lenient read of them succeeds; the raw string if they were not JSON). On a session turn, usage.cache_read_input_tokens reports the tokens served from held KV.

Streaming

Named SSE events in Anthropic’s order: message_start, content_block_start, content_block_delta (text_delta for text, input_json_delta for tool arguments), content_block_stop, message_delta with the stop reason and usage, message_stop. Text comes only ahead of the calls, in block 0, which stops before the first tool_use block starts; each tool_use block is its start, one input_json_delta with the whole arguments, and its stop, sent as the call closes. An engine error mid-stream is an error event.

GET /v1/models

curl http://127.0.0.1:8080/v1/models -H "Authorization: Bearer $LAYERSCALE_LICENSE_KEY"

{"object": "list", "data": [{"id": "devstral-small-2507", "object": "model", "created": 1757000000, "owned_by": "local"}]}

GET /v1/models/{id} returns the same entry when id matches the model name (case-insensitive) and 404 otherwise. There is exactly one model per server.

GET /health

No key needed. 200 while the engine is stepping or idle, 503 when the engine thread has stopped or stalled. /healthz is an alias. The body is the same either way; status is ok or unavailable:

{
  "status": "ok",
  "uptime_ms": 120345,
  "queued": 0,
  "running": 2,
  "kv_pages_total": 4096,
  "kv_pages_free": 3900,
  "connections": 3,
  "steps": 15020,
  "preemptions": 0,
  "requests_started": 41,
  "requests_ok": 39,
  "requests_failed": 0,
  "requests_cancelled": 0,
  "prompt_tokens": 12034,
  "generated_tokens": 8871,
  "sessions": 1,
  "session_turns": 6,
  "prefix_cached_tokens": 9021,
  "session_cached_tokens": 5040,
  "session_ingested_tokens": 1200,
  "session_flash_hits": 3,
  "session_ready_exits": 1,
  "tool_calls_malformed": 0,
  "tool_calls_recovered": 0,
  "tool_calls_jump_forward": 12,
  "tool_calls_forced_tokens": 138,
  "session_call_turns_exact": 4,
  "session_call_turns_diverged": 0
}

Every field after uptime_ms mirrors a /metrics counter (without the _total suffix) or gauge of the same name. step_tokens, the three speculative_* counters and connections_rejected are on /metrics only, as are the histograms.

GET /metrics

Prometheus text; the key is required. Every metric is prefixed layerscale_, and every sample carries the same two constant labels: model_name (the model directory’s name, lowercased) and backend (cuda, rocm or cpu).

Counters: requests_started_total, requests_ok_total, requests_failed_total, requests_cancelled_total, prompt_tokens_total, generated_tokens_total, steps_total, step_tokens_total, speculative_accepted_total, speculative_drafts_total, speculative_draft_tokens_total, preemptions_total, connections_rejected_total, session_turns_total, prefix_cached_tokens_total, session_cached_tokens_total, session_ingested_tokens_total, session_flash_hits_total, session_ready_exits_total, tool_calls_malformed_total, tool_calls_recovered_total, tool_calls_jump_forward_total, tool_calls_forced_tokens_total (the tokens those runs carried), session_call_turns_exact_total (session turns returning from a tool call whose prompt kept the whole previous turn, so the call re-rendered exactly), session_call_turns_diverged_total (those whose prompt diverged inside the call, so it was prefilled again).

Gauges: queued, running, kv_pages_total, kv_pages_free, connections, sessions, ready.

curl http://127.0.0.1:8080/metrics -H "Authorization: Bearer $LAYERSCALE_LICENSE_KEY"

Histograms

Metric	Buckets	One observation per
time_to_first_token_seconds	1 ms to 10 s	model-served request, from its first generated token
inter_token_latency_seconds	10 ms to 2.5 s	model-served request of two or more tokens
e2e_request_latency_seconds	0.3 s to 8 min	model-served request
request_decode_time_seconds	0.3 s to 8 min	model-served request of one or more tokens
request_prompt_tokens	1 to 50,000 tokens	model-served request
request_generation_tokens	1 to 50,000 tokens	model-served request
iteration_tokens	1 to 16,384 tokens	forward step

Each exposes _bucket{le="..."} (cumulative, +Inf last), _sum and _count. The latency buckets and sums are seconds; the token ones are raw counts. A request that errored or was cancelled is counted by how it ended and never timed, and so is one the session caches answered without the model (the flash and speculative finish reasons, session_flash_hits_total and session_ready_exits_total). TTFT is taken at the first generated token whether or not it has streamed out yet, so a reply held back whole for tool-call detection is timed from the token.

Labelled counters

requests_finished_total{finish_reason} is the exclusive breakdown of concluded requests: every one lands in exactly one of stop, length, tool_calls, cancelled, error, flash (answered from a session’s flash cache), speculative (the ready-position exit) and other.

request_errors_total{error_kind} counts the requests answered with an error status, by class — never by the caller’s message, so the label set is bounded:

error_kind	Status
validation	400, and any other 4xx not named here
auth	401, 403
not_found	404, 405
conflict	409
timeout	408, 504
overload	429, 503
internal	5xx other than 503 and 504

Reading them

• inter_token_latency_seconds is a per-request mean, (e2e - ttft) / (completion_tokens - 1), observed once when the request completes. Its p99 is therefore a tail of request means, not of tokens.
• requests_ok_total includes cancelled requests: a cancelled request finishes with what it had. So requests_cancelled_total is a subset of it, ok + failed is every concluded request, and requests_finished_total{finish_reason} is the exclusive split.
• Queue time and prefill time are not exposed separately. The engine records no admission timestamp, and adding one would put a clock read in the step loop. time_to_first_token_seconds covers queueing and prefill together; request_decode_time_seconds covers the rest of the request.
• Speculation has two rates: acceptance is speculative_accepted_total / speculative_draft_tokens_total, and how often speculation fires at all is speculative_drafts_total / steps_total.

layerscale_build_info

Always 1; everything is in its labels, which are the two constant labels followed by:

Label	Value
version	The crate version
git_sha	The commit, twelve hex digits: LS_GIT_SHA from the build environment (CI passes the pipeline’s), else what git said about HEAD when the build script last ran, else unknown
kv_cache_dtype	bfloat16 or fp8_e4m3
ctx	Context length in tokens
max_seqs	Sequences in the running batch
kv_pages, page_size	The KV pool’s size and page
chunked_prefill_size, step_tokens	The scheduler’s prefill chunk and per-step token budget
tp	Tensor-parallel GPUs
speculation, prefix_cache	true or false

Join on it to break a dashboard down by build or by configuration.

CORS

Every response carries Access-Control-Allow-Origin: *. A preflight OPTIONS answers 204 on any path, allowing GET, POST and OPTIONS with the Content-Type, Authorization and x-layerscale-session-id headers, and needs no key. DELETE and the x-api-key header are not in that list, so the session delete routes and key-as-x-api-key are for same-origin or non-browser callers.

Limits

Limit	Value	When exceeded
Request body	32 MiB	413
Request headers	64 KiB	431
Time for a request to arrive in full	60 s	408
Request bodies buffered across all connections	256 MiB	503
Socket read or write	30 s	connection closed
stop sequences	4, each at most 64 bytes	400
WebSocket frame	16 MiB	connection closed

There is no fixed connection cap; each connection is a thread and the host’s limits apply. A prompt that reaches the context length is refused with prompt has N tokens but the context limit is M (raise --ctx). A non-streaming request whose client disconnects is cancelled; a streaming one stops at the first failed write.