Sessions

A session is a named context whose KV cache stays on the GPU between requests. A chat turn against a session pays prefill only for the tokens it adds; the rest is resumed from held KV. On top of that, data can be pushed into a session and ingested in the background, and flash queries are standing questions the engine re-answers after every data batch, so reading one costs no GPU time.

Sessions hold KV until deleted. Creation is explicit: a turn against an unknown id is a 404, never a new session. One request runs per session at a time; a second concurrent request is refused with 409.

A free license permits one live session. The engine’s own ceiling is 256.

Create, list, inspect, delete

# Create, with a generated id
curl -X POST http://127.0.0.1:8080/v1/sessions \
  -H "Authorization: Bearer $LAYERSCALE_LICENSE_KEY"

# Create with a chosen id and a sliding data window
curl -X POST http://127.0.0.1:8080/v1/sessions \
  -H "Authorization: Bearer $LAYERSCALE_LICENSE_KEY" \
  -H "Content-Type: application/json" \
  -d '{"id": "ticker-feed", "window": 8192, "pre_decode": true}'

Body fields, all optional:

Field	Meaning
id	1 to 128 characters of A-Z a-z 0-9 _ -. Default: a generated sess_.... 409 if it already exists.
window	Token cap on the ingested-data region. When new data does not fit, the oldest whole entries are evicted. A whole number below the context minus the query reserve. Without it the region is append-only, and data that no longer fits is dropped and counted.
pre_decode	After every ingested batch, pre-decode the query entry and cache the logits at the ready position, for the fast_answer path of /generate. Default false.

The response, and the shape of every session object:

{
  "id": "ticker-feed",
  "object": "session",
  "created": 1757000000,
  "tokens": 0,
  "kv_tokens": 0,
  "busy": false,
  "turns": 0,
  "data_end": 0,
  "data_version": 0,
  "pending": 0,
  "pre_decode": true,
  "ready": false,
  "evicted": 0,
  "window": 8192
}

tokens is the session’s whole token sequence; kv_tokens how many of them have KV; data_end where the durable context (system prompt, conversation, ingested data) ends; data_version increments once per ingested batch; pending counts pushed entries not yet ingested; evicted counts entries the window evicted. frozen_end, ready_token and ready_gap appear once they have values.

curl http://127.0.0.1:8080/v1/sessions -H "Authorization: Bearer $LAYERSCALE_LICENSE_KEY"
curl http://127.0.0.1:8080/v1/sessions/ticker-feed -H "Authorization: Bearer $LAYERSCALE_LICENSE_KEY"
curl -X DELETE http://127.0.0.1:8080/v1/sessions/ticker-feed -H "Authorization: Bearer $LAYERSCALE_LICENSE_KEY"

The list is {"object": "list", "data": [...]} in creation order. Delete answers {"id": ..., "object": "session", "deleted": true} and releases the KV; it answers 409 while a foreground request is in flight.

Errors: 404 no such session '...', 409 session '...' has a request in flight, 403 the free license permits at most 1 live session(s); delete one first.

Chat turns on a session

Add session_id to a /v1/chat/completions or /v1/messages body. Send the whole conversation as usual; the engine finds the longest prefix already held and prefills only what is new. The response is the ordinary one, plus usage.prompt_tokens_details.cached_tokens (OpenAI) or usage.cache_read_input_tokens (Anthropic) saying how many prompt tokens came from held KV.

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Authorization: Bearer $LAYERSCALE_LICENSE_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "ticker-feed",
    "messages": [{"role": "user", "content": "Summarise the feed so far."}],
    "max_tokens": 200
  }'

Tool calling works the same way. When a turn ends in a tool call, the engine pre-decodes the tool-result framing while the client runs the tool, so the next turn starts from it.

An edited history still works: the engine encodes the whole prompt and resumes from wherever the tokens diverge from what the session holds.

Push data

curl -X POST http://127.0.0.1:8080/v1/sessions/ticker-feed/push \
  -H "Authorization: Bearer $LAYERSCALE_LICENSE_KEY" \
  -H "Content-Type: application/json" \
  -d '{"data": ["AAPL 231.10 +0.4%", "MSFT 415.22 -0.1%"]}'

data is a string or a non-empty array of strings. The acknowledgement is immediate; ingestion happens in the background and bumps data_version per batch:

{"pushed": 2, "dropped": 0, "pending": 2, "pending_bytes": 36, "total_pushed": 2, "total_dropped": 0, "data_version": 0}

A session queues at most 4,096 pending entries or 8 MiB; past that the oldest unprocessed entry is dropped and counted in dropped. The producer never blocks. A foreground turn takes precedence over ingestion at the next step boundary.

Ask over the session

POST /v1/sessions/{id}/generate asks a question over the session’s durable context: the system prompt, the conversation and the ingested data. The question and its answer stay ephemeral. They do not become part of the session, so many questions can run over the same state.

curl -X POST http://127.0.0.1:8080/v1/sessions/ticker-feed/generate \
  -H "Authorization: Bearer $LAYERSCALE_LICENSE_KEY" \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Which symbol moved most?", "max_tokens": 32}'

Field	Default	Notes
prompt	required	The question.
max_tokens, temperature, top_p, seed, stop, ignore_eos, stream	as for chat completions
fast_answer	none	An array of words. With pre_decode on, if the model was already confident at the ready position and its first token starts one of these words, that word is returned with no GPU work.
gap_threshold	2.0	The logit gap the ready position must clear for fast_answer to fire.

{
  "id": "genr_...",
  "object": "session.generation",
  "session_id": "ticker-feed",
  "text": "AAPL, up 0.4%.",
  "finish_reason": "stop",
  "data_version": 1,
  "usage": {"prompt_tokens": 14, "completion_tokens": 7, "total_tokens": 21}
}

A prompt that exactly matches a registered flash query whose answer is current is served from the cache with "flash": true, flash_id, confidence and zero usage. A fast_answer hit carries "speculative": true and logit_gap.

With "stream": true, frames are data: {"text": "..."} per piece, then one data: frame with "done": true and the fields above without text, then data: [DONE].

Flash queries

A flash query is a question registered on a session. The engine re-evaluates it in the background after every ingested batch and keeps the latest answer. A /generate whose prompt matches it exactly is answered from the cache, and the event stream pushes every new answer as it lands.

# Register
curl -X POST http://127.0.0.1:8080/v1/sessions/ticker-feed/flash \
  -H "Authorization: Bearer $LAYERSCALE_LICENSE_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "Which symbol moved most?", "max_tokens": 16}'

# List
curl http://127.0.0.1:8080/v1/sessions/ticker-feed/flash \
  -H "Authorization: Bearer $LAYERSCALE_LICENSE_KEY"

# Remove
curl -X DELETE http://127.0.0.1:8080/v1/sessions/ticker-feed/flash/1 \
  -H "Authorization: Bearer $LAYERSCALE_LICENSE_KEY"

query is required; max_tokens defaults to 32 and is clamped to 1..256. At most 20 flash queries per session (400 past that); a duplicate query is a 409. The entry:

{
  "id": 1,
  "object": "session.flash_query",
  "query": "Which symbol moved most?",
  "max_tokens": 16,
  "tokens": 9,
  "fresh": true,
  "value": "AAPL, up 0.4%.",
  "data_version": 1,
  "confidence": 0.82,
  "evaluated_at": 1757000010
}

fresh is true when value was computed against the current data_version. value, data_version, confidence and evaluated_at are absent until the first evaluation. The list response is {"object": "list", "data": [...], "data_version": N}; delete answers {"id", "object", "deleted": true, "remaining"}.

Event stream (SSE)

GET /v1/sessions/{id}/events is a long-lived text/event-stream:

curl -N http://127.0.0.1:8080/v1/sessions/ticker-feed/events \
  -H "Authorization: Bearer $LAYERSCALE_LICENSE_KEY"

Event	Data	When
connected	{"session_id", "data_version", "flash_queries"}	On connect
flash_ready	{"id", "query", "value", "data_version", "confidence"}	On connect, a replay of every cached answer; then whenever an answer changes
data_updated	{"data_version", "tokens", "pending"}	Whenever a batch is ingested

A : heartbeat comment goes out after 15 seconds of silence. The stream ends when the session is deleted or the server shuts down. A subscriber that falls more than 64 events behind is dropped; reconnecting replays the current answers.

WebSocket

GET /v1/sessions/{id}/ws with the standard upgrade headers opens a socket that carries the same events out and push in. Send the license key as Authorization: Bearer ... on the handshake; without it the upgrade is refused with a plain HTTP 401. Every message is one JSON text frame of the form {"type": ..., "data": {...}}.

Server to client:

type	data
connected	{"session_id", "data_version", "flash_queries"}
flash_ready	as the SSE event, replayed on connect and then live
data_updated	as the SSE event
push_ack	{"pushed", "dropped", "pending", "total_pushed", "total_dropped", "data_version"}
pong	{}
error	{"message", "code": 400}

Client to server:

type	Message
push	{"type": "push", "data": "one entry"} or {"type": "push", "data": ["entry", ...]}
ping	{"type": "ping"}
close	{"type": "close"}

The server pings every 15 seconds; a client that answers with pongs, as every WebSocket library does, keeps the socket’s read timeout from firing. Frames are limited to 16 MiB and must not be fragmented.