Skip to main content

When a run fails

A run can stop before results appear, or complete with numbers that look wrong. Most causes are configuration, not platform faults, and the logs tell you which.

Common causes​

SymptomLikely causeFix
Out of memory (OOM)The model does not fit the GPU at the chosen precision, or gpu_memory_utilization / max_num_batched_tokens is set too high.Use a smaller quantization, add GPUs by raising tensor parallelism, or pick a server with more GPU memory.
Model server never becomes readyBad engine args: a max_model_len the model does not support, or a tensor-parallel size that does not divide the GPU count.Start from a verified recipe. See Choosing engine arguments.
Framework version not supported on the hardwareThe selected version does not match the GPU or driver.Pick a different version from the dropdown.
Gated model will not loadThe Hugging Face token is missing.Add it in the Profile HF-tokens panel (API only in v4.0).
Server offlineThe host missed its heartbeat.Pick another server or wait for the heartbeat to resume.
All-zero latencies or implausible token countsThe framework did not actually serve the requests.Open the logs below and read the Framework stream.

Read the logs​

Every job records its logs, and the fastest way to see why a run failed is to read them.

Open the Logs drawer from Monitoring: on an execution row or a workload row, click View logs. The drawer lists each job and gives three streams per job:

StreamWhat it holds
FrameworkThe model server's own output (vLLM, SGLang, TensorRT-LLM). OOM errors, rejected flags, and startup failures appear here.
ToolThe benchmark client's output: request pacing, per-request errors, and timeouts.
AgentThe worker agent's lifecycle log: model-server launch, readiness polling, and teardown.
tip

For a failed run, start with the Framework stream. An OOM or an invalid engine argument shows up there first.

If a run completes but a results may be unreliable badge appears, the data is still on the Reporting page. Treat it with extra skepticism, and read the logs before trusting the numbers.