Models

The model directory

--model takes a local directory in the Hugging Face layout. The engine reads it directly and never downloads anything, so fetch the checkpoint first:

hf download mistralai/Devstral-Small-2507 --exclude 'consolidated*' --local-dir /models/Devstral-Small-2507

Mistral’s repositories ship a second copy of the weights in their own consolidated*.safetensors format beside the Hugging Face shards. The engine reads only the Hugging Face index, so excluding that copy halves the download.

File	Required	Used for
config.json	yes	Architecture and hyperparameters. The architecture comes from architectures[0], never from weight shapes.
model.safetensors, or model.safetensors.index.json plus the shards it names	yes	Weights. Shards must be plain files in the same directory.
tokenizer.json or tekken.json	one of them	The tokenizer. tekken.json is what Mistral ships for Devstral and its relatives.
tokenizer_config.json or chat_template.jinja	no	Read once to pick the chat template variant the checkpoint was trained on. The engine renders its own port of the template; the Jinja is not executed.
hf_quant_config.json	no	NVIDIA ModelOpt quantization scheme, when the checkpoint keeps it beside config.json.

The model’s id on the API (GET /v1/models, the model field of every response) is the directory’s name in lower case.

Precision and quantized checkpoints

Unquantized checkpoints run in bfloat16. Quantized checkpoints are read from quantization_config in config.json, or from hf_quant_config.json for ModelOpt:

quant_method	Accepted
fp8	Per-tensor or blockwise (weight_block_size) weights, static or dynamic activation scales
compressed-tensors	llm-compressor exports; actorder=group is refused
gptq, awq	4-bit only; GPTQ act-order (desc_act) is refused
modelopt	quant_algo of FP8 or NVFP4; a kv_cache_quant_algo of FP8 calibrates the KV cache
mxfp4	MXFP4 weights

Any other method fails at load with quant_method '...' is not implemented.

KV cache precision

--kv-cache-dtype auto stores the cache in the model’s dtype, unless the checkpoint calibrated an fp8 KV cache (a ModelOpt kv_cache_quant_algo, or a compressed-tensors kv_cache_scheme), in which case it is stored in fp8 with the checkpoint’s own scales. Force it with bfloat16 or fp8. fp8 halves the pool, so the same card holds twice the context. It requires head_dim 128 and is a memory decision rather than a speed one.

Context length

--ctx defaults to the checkpoint’s max_position_embeddings; a checkpoint without one must be started with --ctx. The engine refuses to start unless the KV pool holds at least one full context. A free license clamps the context at 32,768 tokens.

Tensor parallelism

--tp N splits one copy of the model across GPUs 0..N-1. Without the flag the engine uses every GPU the host shows. Requirements:

• Attention heads, KV heads, the intermediate size and the vocabulary must divide by N.
• Every GPU must be able to read every other’s memory (NVLink or PCIe peer access).
• The license must permit N GPUs. A free key permits one, so on a multi-GPU host pass --tp 1.