Troubleshooting Agentic Bench
Find the symptom, read the cause, and apply the fix. Messages in quotation marks are the exact text the product shows.
While you configure the run
| Symptom | Cause | Fix |
|---|---|---|
| Save & Run is unavailable, with "An active plan is required to start a run." | Your organization has no active plan. | Choose a plan on the plans page, then return to the project. |
| The workload card never shows Ready to run. | A required field is empty or invalid. | Open each step without a green check mark. The invalid field shows its message. |
| "Enter a value for tool_call_parser. The run cannot be saved without it." | The model is not in the verified list, so it has no tool call parser. | Enter the parser the model needs under Engine args. See Use a model that is not in the list. |
| A typed task set is not accepted. | The reference is not name@revision, or the revision is latest, head, main, or master. | Enter an exact revision, for example terminal-bench/terminal-bench-2-1@6. |
| Split prefill and decode across machines cannot be turned on. | A requirement is not met. The reason is shown under the switch. | See Requirements. The usual fix is to set tensor parallel size equal to GPUs per node and pipeline parallel size to 1. |
| "This cluster group has fewer than two machines. A prefill and decode split needs at least two." | The cluster group has one machine. | Create a cluster group with two or more machines. |
| "Distributed KV runs are not included in your current plan." | A split needs a plan that includes Distributed KV runs. | Upgrade the plan, or run without a split. |
| "Choose CPUs for at least one machine, or switch to Control or Unpinned." | Pinned is selected, but no CPUs are entered. | Enter CPUs for at least one machine, or choose another mode. |
| "CPUs not reported yet" beside a machine. | The machine's agent has not reported its CPU layout. | Update and restart the Metrum agent on that machine, then reload the page. Pinned mode needs the layout. |
| A CPU list is refused because the machine keeps those CPUs for metrics collection. | Each machine keeps one or two physical cores for its telemetry collectors. | Choose CPUs from the list the message gives. |
| "Pipeline parallel size (N) needs more nodes than this group has (M)." | Pipeline parallel size is larger than the number of machines. | Lower it, or add machines to the group. |
| A machine is missing from the Machines list when you create a cluster group. | The machine is not onboarded to your organization, or its agent is not running. | Onboard the machine and start its agent. |
When a run starts
| Symptom | Cause | Fix |
|---|---|---|
| The run waits, and a notice says how many runs are ahead. | Another run is using the cluster group, or the previous run's cluster is still being torn down. | Wait. The run starts on its own when the group is free. |
Kubernetes cluster fails on chronyc waitsync. | The machine's clock did not synchronize to within 0.5 seconds, or chrony could not be installed. | Make sure the machine reaches an NTP server and chrony runs, then run again. |
| Kubernetes cluster fails with "Another Kubernetes node agent runs on this machine." | The machine already runs another Kubernetes cluster, such as kubeadm, rke2, MicroK8s or kind. Metrum does not stop or remove it. | Remove that cluster from the machine, or choose another machine. |
| Kubernetes cluster fails with "nvidia-container-runtime is not on the agent's PATH." | The NVIDIA Container Toolkit is missing, or installed where the agent does not look. | Install the NVIDIA Container Toolkit, or add its directory to the agent's PATH, then run again. |
| A level is refused with "This level's ... sandboxes need about ... GB of disk." | A machine has less free disk than the level's sandboxes need, measured on the run's earlier levels. | Free disk on that machine, or lower the agent count of the level. |
| A machine "has not reported its GPUs." | The machine's agent has not registered its hardware. | Start the Metrum agent on the machine so it registers its GPUs, then run again. |
| The run is refused because of the build variant. | The pinned build variant is for another GPU vendor than your machines. | Set Build variant to Automatic. |
| Model loading fails with a model access message. | The model is gated or private, and your saved Hugging Face token has not accepted its license. A mistyped model ID gives the same answer from Hugging Face. | Check the model ID. Accept the model's license on Hugging Face with the account whose token you saved, or save a token that has access. |
| Model loading takes a long time. | The model weights are downloading. | Wait. Large models take a long time to download the first time on each machine. |
| Kubernetes cluster takes several minutes. | The cluster forms for the first time on this cluster group. | Wait. This is expected on first use. |
| A job stops with "METRUM_AGENTIC_BENCH_TELEMETRY_GATEWAY_URL is required". | The machine was installed with the organization install command before release 4.2.2, and its agent settings lack the telemetry address. | Follow the fix in the 4.2.2 release notes, or reinstall the agent with a current install command. |
During the validation trial
| Symptom | Cause | Fix |
|---|---|---|
| The validation trial fails, and every level is cancelled. | A platform failure: the sandbox did not start, the model endpoint did not answer, or the grader could not run. | Read the reason on the stage tracker, fix it, and run again. |
| The validation trial fails, but the levels still run. | The agent finished and failed the task, or the trial timed out. The platform worked. | No fix is needed to continue. If every trial fails, check the model and its tool call parser. |
| Every trial fails at once with a model request error. | vLLM rejects the agent's requests. The usual cause is a missing or wrong tool call parser, or enable_auto_tool_choice removed. | Restore enable_auto_tool_choice = true, and enter the right tool_call_parser for the model. |
| Every trial fails because a task is not found. | A typed task code does not exist in the dataset. | Check the exact task directory name in the dataset. |
During the levels
| Symptom | Cause | Fix |
|---|---|---|
| Many trials end as timeouts, while finished trials mostly pass. | Agents wait longer for the model at high levels, so tasks need more than their own time limit. | Raise the Agent time multiplier, for example to 2. |
| A level shows Timed out or is stopped as stalled. | The level ran past its level time limit. | Raise the multiplier if trials were still progressing, or lower the level. Check the model server's queue in the report. |
| A level ran below its target count. | Some sandboxes failed to start, or the wait for them timed out. | Check host memory and disk. Lower Sandbox memory, add machines, or lower the level. |
| A level did not start. | The sandboxes could not each reserve the minimum memory the agent needs. | Lower the level, free memory on the machines, or add machines. |
| A Pinned or Control run fails before a level. | The check before each level found the CPU mode was not in effect on a machine. | Read the machine and the check result on the stage tracker. Update the agent on that machine, then run again. |
| Enterprise cases fail after a long wait for the model. | A model call that receives no output for 1,400 seconds fails. The level has more agents than the model server can serve. | Lower the level, or add GPUs. |
| Time to first token is very high on a split run. | KV moves over TCP, or the decode prompt cache is off. | Check the transport on the run. Use machines with active RoCE v2 ports for RDMA, and keep Prompt cache on decode machines on. |
| The level shows Completed although some tasks failed. | The level ran every planned trial. Failed tasks are a quality result, not a run failure. | No fix. Read the pass rate and the failure reasons. |
In the report
| Symptom | Cause | Fix |
|---|---|---|
| No level qualifies. | Every level missed the pass rate or the GPU busy ratio. | Read Qualification for the gate each level missed. A low GPU busy ratio at low levels is normal: lower the threshold, or test higher levels. |
| Low levels miss the GPU busy gate, and high levels qualify. | A few agents cannot keep the GPUs busy. | Expected. Size hardware from the levels that qualify. |
| A metric shows Not captured, Not available on this hardware, or No samples recorded. | The source did not report it, or the hardware does not expose it. | No fix is needed. The value is unknown, not zero. |
| Whole-level goodput is much lower than goodput to p95 finish. | One or a few very long trials held the level open. | Read goodput to p95 finish, and look at the longest trial in Inside one level. |
| Two runs of the same workload give different results. | A setting differs, such as the task set revision, the version, the multiplier, the transport, or the CPU mode. | Compare Test conditions of both runs. See Comparing runs. |
Frequently asked questions
Do I need to install Kubernetes? No. Agentic Bench forms a Kubernetes cluster over your cluster group's machines when the run starts, and tears it down at the end. Each machine needs only the Metrum agent.
How long does a run take?
Setup takes from a few minutes to about half an hour, mostly for the cluster and the model download. Each level then takes about as long as its slowest few trials. Short tasks such as fix-git finish a 64-agent level in a few minutes on a capable GPU. The level time limits in Step 5 are ceilings, not estimates.
Can I stop a run that is in progress? The web app has no stop button for an Agentic Bench run. A level that runs too long stops on its own at its level time limit. To stop a run sooner, ask your Metrum AI Bench administrator.
What does a run install on my machines, and what stays after it? A run installs k3s and the model serving stack, and removes them when it ends. The model weights stay in a cache on each machine for later runs. See How Agentic Bench works.
Which ports and outside sites do my machines need? See Network access.
Can I run two workloads on the same cluster group at once? No. Runs on one cluster group queue, and each run gets the whole group.
Can I bring my own agent? Not from the web app. Coding Agent runs use the Deep Agents agent, and Enterprise Agent runs use the Metrum enterprise agent.
Can I use tasks other than Terminal-Bench?
Yes, for the Coding Agent. Type any Harbor dataset reference as name@revision in Task set, then type its task names in Task codes.
Can I compare a Coding Agent result with an Enterprise Agent result? No. They measure different work and are never compared.
Why does my model need a tool call parser? The agent works by calling tools. vLLM needs to know how the model writes tool calls to turn them into structured calls. Without the parser, vLLM rejects the agent's first request.
Why must tensor parallel size equal GPUs per node for a split? A split runs one engine per machine, and that engine must use every GPU the run uses on the machine. Otherwise the split would use less hardware than the run without a split it is compared with.
Does a failed validation trial always stop the run? No. Only a platform failure stops it. A task the agent failed still lets the levels run.
What does a level that missed a gate tell me? What happened at that load. Size hardware only from levels that qualify.
Known limitations
- Only vLLM has been validated. SGLang and TensorRT-LLM appear in the framework list.
- Every machine in a run must have the same GPU vendor. Intel accelerators and Google TPU are not supported.
- A NUMA node cannot be pinned as a unit.
- A prefill and decode split needs tensor parallel size equal to GPUs per node, and pipeline parallel size 1.
- An Enterprise model call with no output for 1,400 seconds fails.
- On AMD GPUs, the GPU busy figure counts queued work. Read it with memory activity and power.
Get help
When you contact Metrum support, include the project name, the run's start time, the failed stage or level, and the message shown. The Workload Configuration panel on the monitoring page lists every setting of the run.