API reference

The server speaks HTTP/1.1 with keep-alive, chunked request bodies and Server-Sent Events for streaming. The base URL is wherever serve bound, http://127.0.0.1:8080 by default.

Routes

Method	Path	Purpose
POST	/v1/chat/completions	OpenAI chat completion, streaming and tool calling
POST	/v1/messages	Anthropic message, streaming and tool use
GET	/v1/models	The loaded model
GET	/v1/models/{id}	The loaded model, 404 for any other id
GET	/health, /healthz	Readiness, no key needed
GET	/metrics	Prometheus text
OPTIONS	any	CORS preflight, no key needed
POST, GET	/v1/sessions	Create and list sessions
GET, DELETE	/v1/sessions/{id}	Inspect and delete a session
POST	/v1/sessions/{id}/generate	Ask a question over a session’s context
POST	/v1/sessions/{id}/push	Queue data for background ingestion
POST, GET	/v1/sessions/{id}/flash	Register and list flash queries
DELETE	/v1/sessions/{id}/flash/{fid}	Remove a flash query
GET	/v1/sessions/{id}/events	The session’s event stream (SSE)
GET	/v1/sessions/{id}/ws	The session’s event stream and push channel (WebSocket)

The session routes are described on the Sessions page. A trailing slash is ignored. A known path with the wrong method answers 405; an unknown path answers 404.

Authentication

Every route except OPTIONS, /health and /healthz requires the license key the server was started with:

Authorization: Bearer LSK-...

x-api-key: LSK-... is accepted too, which is what Anthropic SDKs send. Dashes and whitespace inside the key are ignored. A missing or wrong key answers 401 with missing or invalid license key; send it as 'Authorization: Bearer <LSK-...>', in the envelope of the route (Anthropic’s for /v1/messages, OpenAI’s elsewhere). A WebSocket upgrade without the key is refused with a plain HTTP 401 before the handshake.

Errors

OpenAI routes, the session routes included:

{"error": {"message": "'top_k' is not supported", "type": "invalid_request_error"}}

Status	type
401	authentication_error
402, 403	permission_error
400, 404, 405, 408, 409, 413, 431	invalid_request_error
429	rate_limit_error
5xx	server_error

/v1/messages uses Anthropic’s envelope:

{"type": "error", "error": {"type": "invalid_request_error", "message": "'max_tokens' is required"}}

with authentication_error (401), permission_error (403), not_found_error (404), rate_limit_error (429), invalid_request_error (other 4xx), overloaded_error (529) and api_error (5xx).

On a streaming response the status line has already gone out, so an engine error arrives as an event: an OpenAI error object as a data: frame followed by data: [DONE], or an Anthropic event: error frame.

Request parameters the engine cannot honour are refused by name with a 400. The exceptions are model, which is accepted and ignored on every route, and metadata on /v1/messages.

POST /v1/chat/completions

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Authorization: Bearer $LAYERSCALE_LICENSE_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "system", "content": "Be brief."},
      {"role": "user", "content": "Why is the sky blue?"}
    ],
    "max_tokens": 200,
    "temperature": 0.2
  }'

Request

Field	Default	Notes
messages	required	Roles system, user, assistant, tool (ipython is accepted as tool). content is a string or an array of {"type": "text", "text": ...} parts; other part types are refused.
max_tokens or max_completion_tokens	512
temperature	0.6	0 is greedy.
top_p	0.9
seed	none
stop	none	A string, or an array of up to 4 strings of at most 64 bytes each. Excluded from the returned text.
stream	false
stream_options	none	{"include_usage": true} appends a final usage chunk.
tools	none	{"type": "function", "function": {"name", "description", "parameters"}}.
tool_choice	auto	auto or none. required and a named function are refused.
reasoning_effort	none	none, minimal, low (no thinking) or medium, high, xhigh (thinking), for templates that render a reasoning setting.
chat_template_kwargs	none	date_string sets the template’s date. Other keys are refused.
session_id	none	Run the turn against a session.
ignore_eos	false	Keep generating past end of sequence, to exactly max_tokens.

Assistant messages may carry tool_calls and reasoning_content; tool messages carry tool_call_id. Refused by name: n or best_of other than 1, logprobs, top_logprobs, logit_bias, top_k, min_p, repetition_penalty, a nonzero presence_penalty or frequency_penalty, a response_format other than text, and the deprecated functions and function_call.

Response

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "created": 1757000000,
  "model": "devstral-small-2507",
  "choices": [{
    "index": 0,
    "message": {"role": "assistant", "content": "..."},
    "finish_reason": "stop"
  }],
  "usage": {"prompt_tokens": 31, "completion_tokens": 57, "total_tokens": 88}
}

finish_reason is stop, length or tool_calls. On a session turn, usage.prompt_tokens_details.cached_tokens reports the prompt tokens served from held KV; it is present only when nonzero.

Streaming

With "stream": true the response is text/event-stream. Each data: frame is a chat.completion.chunk: first a delta of {"role": "assistant"}, then content deltas, then a final chunk carrying finish_reason, then data: [DONE]. With include_usage, every chunk carries "usage": null and one more chunk with empty choices and the totals precedes [DONE]. No bytes are sent while the prompt is being prefilled.

Tool calling

Offer tools and read finish_reason: "tool_calls":

"message": {
  "role": "assistant",
  "content": null,
  "tool_calls": [{
    "id": "call_...",
    "type": "function",
    "function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"}
  }]
}

arguments is a JSON string. Text beside the calls is returned as content; with no text it is null. Send the result back as a tool message with the same tool_call_id. When streaming, each call arrives as two chunks: one with id, type, the function name and empty arguments, then one with the complete arguments string.

POST /v1/messages

The Anthropic Messages API over the same engine.

curl http://127.0.0.1:8080/v1/messages \
  -H "x-api-key: $LAYERSCALE_LICENSE_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "system": "Be brief.",
    "messages": [{"role": "user", "content": "Why is the sky blue?"}],
    "max_tokens": 200
  }'

Request

Field	Default	Notes
messages	required	Roles user and assistant. content is a string or an array of blocks: text and tool_result in user messages; text, tool_use and thinking in assistant messages.
system	none	A string or an array of text blocks.
max_tokens	required
temperature	0.6
top_p	0.9
seed	none
stop_sequences	none	An array of strings.
stream	false
tools	none	{"name", "description", "input_schema"}.
tool_choice	{"type": "auto"}	auto or none.
session_id	none	Run the turn against a session.
ignore_eos	false
metadata		Accepted and ignored.

top_k and thinking are refused.

Response

{
  "id": "msg_...",
  "type": "message",
  "role": "assistant",
  "model": "devstral-small-2507",
  "content": [{"type": "text", "text": "..."}],
  "stop_reason": "end_turn",
  "stop_sequence": null,
  "usage": {"input_tokens": 31, "output_tokens": 57}
}

stop_reason is end_turn, max_tokens, stop_sequence or tool_use. A tool call is a tool_use block with id, name and input (the parsed arguments, or the raw string if they were not valid JSON). On a session turn, usage.cache_read_input_tokens reports the tokens served from held KV.

Streaming

Named SSE events in Anthropic’s order: message_start, content_block_start, content_block_delta (text_delta for text, input_json_delta for tool arguments), content_block_stop, message_delta with the stop reason and usage, message_stop. An engine error mid-stream is an error event.

GET /v1/models

curl http://127.0.0.1:8080/v1/models -H "Authorization: Bearer $LAYERSCALE_LICENSE_KEY"

{"object": "list", "data": [{"id": "devstral-small-2507", "object": "model", "created": 1757000000, "owned_by": "local"}]}

GET /v1/models/{id} returns the same entry when id matches the model name (case-insensitive) and 404 otherwise. There is exactly one model per server.

GET /health

No key needed. 200 while the engine is stepping or idle, 503 when the engine thread has stopped or stalled. /healthz is an alias. The body is the same either way; status is ok or unavailable:

{
  "status": "ok",
  "uptime_ms": 120345,
  "queued": 0,
  "running": 2,
  "kv_pages_total": 4096,
  "kv_pages_free": 3900,
  "connections": 3,
  "steps": 15020,
  "preemptions": 0,
  "requests_started": 41,
  "requests_ok": 39,
  "requests_failed": 0,
  "requests_cancelled": 0,
  "prompt_tokens": 12034,
  "generated_tokens": 8871,
  "sessions": 1,
  "session_turns": 6
}

The remaining fields mirror the counters /metrics exports.

GET /metrics

Prometheus text; the key is required. Every metric is prefixed layerscale_.

Counters: requests_started_total, requests_ok_total, requests_failed_total, requests_cancelled_total, prompt_tokens_total, generated_tokens_total, steps_total, step_tokens_total, speculative_accepted_total, preemptions_total, connections_rejected_total, session_turns_total, prefix_cached_tokens_total, session_cached_tokens_total, session_ingested_tokens_total, session_flash_hits_total, session_ready_exits_total, tool_calls_malformed_total, tool_calls_recovered_total, tool_calls_jump_forward_total.

Gauges: queued, running, kv_pages_total, kv_pages_free, connections, sessions, ready.

curl http://127.0.0.1:8080/metrics -H "Authorization: Bearer $LAYERSCALE_LICENSE_KEY"

CORS

Every response carries Access-Control-Allow-Origin: *. A preflight OPTIONS answers 204, allowing GET, POST and OPTIONS with the Content-Type and Authorization headers, and needs no key.

Limits

Limit	Value	When exceeded
Request body	32 MiB	413
Request headers	64 KiB	431
Time for a request to arrive in full	60 s	408
Request bodies buffered across all connections	256 MiB	503
Socket read or write	30 s	connection closed
stop sequences	4, each at most 64 bytes	400
WebSocket frame	16 MiB	connection closed

There is no fixed connection cap; each connection is a thread and the host’s limits apply. A prompt that reaches the context length is refused with prompt has N tokens but the context limit is M (raise --ctx). A non-streaming request whose client disconnects is cancelled; a streaming one stops at the first failed write.