Models

The model directory

--model takes a local directory in the Hugging Face layout, or a Hugging Face repo id (see Downloading from Hugging Face). A directory is read as it is:

hf download mistralai/Devstral-Small-2507 --exclude 'consolidated*' --local-dir /models/Devstral-Small-2507

Mistral’s repositories ship a second copy of the weights in their own consolidated*.safetensors format beside the Hugging Face shards. The engine reads only the Hugging Face index, so excluding that copy halves the download.

Downloading from Hugging Face

--model also takes a Hugging Face repo id, such as --model mistralai/Devstral-Small-2507. A local directory of the same name takes precedence. The engine looks for the repo in the local Hugging Face cache first, where hf download without --local-dir also puts it. The cache is found as huggingface_hub finds it: HF_HUB_CACHE, then $HF_HOME/hub, then ~/.cache/huggingface/hub. A complete copy there (config.json and every weight shard) is loaded at the snapshot refs/main points at, without the network, and served under the repo’s name.

A repo that is not there, or only partly there, is downloaded at main into that cache before the load, in the layout huggingface_hub uses, so hf and the engine share one copy. Only what the engine reads is fetched: the top-level config, tokenizer and template files and the safetensors checkpoint. Other frameworks’ weights (*.bin, *.pth, *.gguf, original/) and Mistral’s second consolidated* copy are skipped. Eight files transfer at once. Progress is logged as JSON lines: download at the start, download_progress every ten seconds with the percentage, gigabytes done and rate, download_done at the end. An interrupted download resumes where it stopped on the next start.

The transfer runs through the system’s curl, which the Docker image includes. Settings are read as huggingface_hub reads them:

Variable	Effect
HF_TOKEN	Access token for gated and private repos. HUGGING_FACE_HUB_TOKEN is also read, and without either, the token file hf auth login saves ($HF_HOME/token, or HF_TOKEN_PATH).
HF_HUB_CACHE, HF_HOME	Where the cache is.
HF_HUB_OFFLINE=1	Never download; a repo that is not cached is refused.
HF_ENDPOINT	A Hub mirror instead of https://huggingface.co.

A gated model (Llama, Gemma and others whose license you accept on the model’s page) is served only to an account that has accepted its license, through that account’s access token. Accept the license on the model’s Hugging Face page, create a read token at huggingface.co/settings/tokens, and pass it as HF_TOKEN. In Docker, pass it with -e HF_TOKEN=hf_..., and mount the cache (-v ~/.cache/huggingface:/root/.cache/huggingface) so the download outlives the container. A container that downloads into its own filesystem logs a download_cache warning saying so.

File	Required	Used for
config.json	yes	Architecture and hyperparameters. The architecture comes from architectures[0], never from weight shapes.
model.safetensors, or model.safetensors.index.json plus the shards it names	yes	Weights. Shards must be plain files in the same directory.
tokenizer.json or tekken.json	one of them	The tokenizer. tekken.json is what Mistral ships for Devstral and its relatives.
tokenizer_config.json or chat_template.jinja	no	Read once to pick the chat template variant the checkpoint was trained on. The engine renders its own port of the template; the Jinja is not executed.
hf_quant_config.json	no	NVIDIA ModelOpt quantization scheme, when the checkpoint keeps it beside config.json.

The model’s id on the API (GET /v1/models, the model field of every response) is the directory’s name in lower case.

Precision and quantized checkpoints

Unquantized checkpoints run in bfloat16. Quantized checkpoints are read from quantization_config in config.json, or from hf_quant_config.json for ModelOpt:

quant_method	Accepted
fp8	Per-tensor or blockwise (weight_block_size) weights, static or dynamic activation scales
compressed-tensors	llm-compressor exports; actorder=group is refused
gptq, awq	4-bit only; AWQ must be the gemm version, and a desc_act GPTQ export is refused when its g_idx actually permutes
modelopt	quant_algo of FP8 or NVFP4; a kv_cache_quant_algo of FP8 calibrates the KV cache
mxfp4	MXFP4 weights

Any other method fails at load with config.json quantization_config: quant_method '...' is not implemented, which names the methods this build reads.

A compressed-tensors W4A16 export — int4 weights in groups of 128 along the input dimension, symmetric, with bf16 scales and no zero points, which is how llm-compressor writes pack-quantized — is a resident format: the routed and shared experts stay int4 on the card and are dequantized inside the kernel, rather than being expanded to bfloat16 at load. That is what makes a 60 GB Llama 4 Scout export serve on one 94 GB card. Everything the export’s ignore list names — attention, the router, the embedding and the LM head — stays bfloat16, and so does the KV cache unless --kv-cache-dtype says otherwise.

KV cache precision

--kv-cache-dtype auto stores the cache in the model’s dtype, unless the checkpoint calibrated an fp8 KV cache (a ModelOpt kv_cache_quant_algo, or a compressed-tensors kv_cache_scheme), in which case it is stored in fp8 with the checkpoint’s own scales. Force it with bfloat16 or fp8. fp8 halves the pool, so the same card holds twice the context. It requires head_dim 128, is refused on a checkpoint with sliding-window layers (those attention kernels are built over a bf16 cache only), and is a memory decision rather than a speed one.

Context length

--ctx defaults to the checkpoint’s max_position_embeddings, clamped to what the KV pool this run gets can hold for one sequence; a checkpoint without one must be started with --ctx. The clamp is what makes a checkpoint whose ceiling is architectural rather than practical start at all — Llama 4 Scout says 10,485,760, which would size the rope tables, the pinned page metadata and the pool at figures no card has — and it is logged as ctx_defaulted with the two numbers. --ctx overrides it in either direction; the engine then refuses to start unless the KV pool holds at least one full context. A free license clamps the context at 32,768 tokens.

Tensor parallelism

--tp N splits one copy of the model across GPUs 0..N-1. Without the flag the engine uses every GPU the host shows. Requirements:

• Attention heads, KV heads, the intermediate size and the vocabulary must divide by N.
• Every GPU must be able to read every other’s memory (NVLink or PCIe peer access).
• The license must permit N GPUs. A free key permits one, so tensor parallelism needs a Pro license; on a multi-GPU host pass --tp 1.