When a run fails
A run can stop before results appear, or complete with numbers that look wrong. Most causes are configuration, not platform faults, and the logs tell you which.
Common causes
| Symptom | Likely cause | Fix |
|---|---|---|
| Out of memory (OOM) | The model does not fit the GPU at the chosen precision, or gpu_memory_utilization / max_num_batched_tokens is set too high. | Use a smaller quantization, add GPUs by raising tensor parallelism, or pick a server with more GPU memory. |
| Model server never becomes ready | Bad engine args: a max_model_len the model does not support, or a tensor-parallel size that does not divide the GPU count. | Start from a verified recipe. See Choosing engine arguments. |
| Framework version not supported on the hardware | The selected version does not match the GPU or driver. | Pick a different version from the dropdown. |
| Gated model will not load | The Hugging Face token is missing. | Add it in the Profile HF-tokens panel (API only in v4.0). |
| Server offline | The host missed its heartbeat. | Pick another server or wait for the heartbeat to resume. |
| All-zero latencies or implausible token counts | The framework did not actually serve the requests. | Open the logs below and read the Framework stream. |
Read the logs
Every job records its logs, and the fastest way to see why a run failed is to read them.
Open the Logs drawer from Monitoring: on an execution row or a workload row, click View logs. The drawer lists each job and gives three streams per job:
| Stream | What it holds |
|---|---|
| Framework | The model server's own output (vLLM, SGLang, TensorRT-LLM). OOM errors, rejected flags, and startup failures appear here. |
| Tool | The benchmark client's output: request pacing, per-request errors, and timeouts. |
| Agent | The worker agent's lifecycle log: model-server launch, readiness polling, and teardown. |
tip
For a failed run, start with the Framework stream. An OOM or an invalid engine argument shows up there first.
If a run completes but a results may be unreliable badge appears, the data is still on the Reporting page. Treat it with extra skepticism, and read the logs before trusting the numbers.