Monitor a run and read the report
This page explains what you see while an Agentic Bench run is in progress, and how to read the report when it finishes.
Monitor a run
Open the run from Monitoring. The page shows, from top to bottom:
| Element | What it shows |
|---|---|
| Workload Configuration | Every setting the run uses: task set, tasks, model, version, cluster group, parallelism, sandbox CPU mode, prefill and decode split, levels, and SLA. |
| Status banner | What the run is doing now, in plain words, for example "Downloading model weights and loading them onto the GPUs". |
| Queue notice | When the run waits for its cluster group: how many runs are ahead. |
| Stage tracker | Each stage of the run, with its state, start time, duration, and the reason for a failure. |
| Rungs table | The validation trial and every concurrency level, with status, task progress, start time, and duration. |
| Telemetry badge | The health of each telemetry source. It opens the per-source detail when a source is delayed or failing. |
Run stages
| Stage | What happens | Typical time |
|---|---|---|
| Host checks | Checks free disk, memory, GPU memory, network ports, clock synchronization, and the container runtime on each machine. | Seconds |
| Benchmark runtime | Prepares the Agentic Bench runtime, the task dataset, and the task images. | Minutes the first time, faster after |
| Kubernetes cluster | Forms the Kubernetes cluster over the group's machines, one machine included, and installs what the model server needs. | A few minutes |
| Model deployment | Deploys the model server onto the selected machines. | Under a minute |
| Pods starting | Schedules the model server and pulls its container images. | Minutes |
| Model loading | Downloads the model weights and loads them onto the GPUs. | Depends on model size and network speed |
| Serving endpoint | Routes traffic to the model server and waits for it to answer. | Under a minute |
| Validation trial | Runs one trial with one agent to prove the sandbox, the agent, and the model endpoint work together. | About as long as one task |
| Measurement | Runs each concurrency level and records its results. | See the level durations below |
| Teardown | Stops the model server and releases the machines. | A few minutes |
Each stage shows Pending, Running, Passed, Failed, or Skipped. A failed stage shows the reason the machine reported.
How a level runs
- The level creates one sandbox per agent and waits until every sandbox is ready.
- All agents start at the same moment. The release time is recorded, and throughput is measured from it.
- Each agent works a task. When it finishes, it takes the next trial until the level's trials are done.
- A grader scores each finished trial with the task's own tests.
- The level's telemetry and results are stored, and the next level starts.
If some sandboxes fail to start, or the wait for them times out, the level runs below its target count, and the report marks it. An Enterprise level in that state fails its fleet validity gate.
Example level durations from Metrum validation runs of the fix-git task:
| Hardware | Level | Level time | Median trial |
|---|---|---|---|
| 2 × RTX PRO 6000, one GPU per machine | 64 agents | 4 min | 2.5 min |
| 2 × RTX PRO 6000, one GPU per machine | 256 agents | 16 min | 5.4 min |
| 2 × L40S machines, split over TCP | 256 agents | 55 min | 41 min |
| 2 × L40S machines, split over TCP | 600 agents | 144 min | 105 min |
Level status in the Rungs table
| Status | Meaning |
|---|---|
| Pending, Queued | The level has not started. |
| Running | The level's sandboxes or agents are running. |
| Completed | Every planned trial ran and finished. This includes a level where some tasks failed their tests. |
| Partial | The level stopped with trials unfinished. |
| Failed | The level ran no trial, or the platform failed before it measured anything. |
| Cancelled | The level was cancelled, for example because the validation trial failed for a platform reason. |
| Timed out | The level ran past its level time limit and was stopped as stalled. |
Read the report
The report has six sections. Use the section links at the top to move between them.
| Section | What it answers |
|---|---|
| Verdict | What is the highest agent count that met the SLA? |
| Scaling | How do pass rate, goodput, throughput, agent time, and GPU headroom change from level to level? |
| Qualification | Which levels qualify, and which gate did each other level miss? |
| Every curve | All charts across all levels. |
| Inside one level | What happened inside one level: telemetry, harness timing, shell time, and a trace of one attempt. |
| Test conditions | The hardware on every machine, the SLA, and every recorded factor of the run. |
Verdict
The verdict names the highest concurrency level that is a valid operating point. A level is a valid operating point when:
- its pass rate is at least the minimum pass rate, and
- its GPU busy ratio is at least the minimum GPU busy ratio, and
- for an Enterprise run, every enterprise gate passes.
When a higher level missed a gate, the verdict also names the first such level. That level is where the hardware stopped meeting your SLA.
When no level qualifies, the verdict shows the finished level with the most verified goodput and marks it as not qualified.
The verdict section also holds a level replay: a control that plays the headline level back over time, so you can see agents start, work, and finish. Read the Qualification section to see which gate each level missed.
Qualification
Each level carries one label:
| Label | Meaning |
|---|---|
| Qualified | The level met every gate. You can size hardware from it. |
| Missed a gate | The level finished but missed at least one gate. The label names the gate and the measured value. |
| Not gated | The level could not be judged: it is still running, it did not finish, it did not run, or no attempt finished. |
Size hardware only from a Qualified level. A level that missed a gate tells you what happened at that load, not what the hardware can sustain.
Key metrics
| Metric | What it means |
|---|---|
| Pass rate | Passed trials divided by graded trials. Trials the platform lost, such as a sandbox that never started or a grader that could not run, are left out of the denominator and counted separately. |
| Verified goodput | Passed tasks per hour across all agents. It counts only tasks the grader confirmed. |
| Verified goodput (to p95 finish) | Passed tasks per hour up to the moment 95% of the level's trials had finished. One very long trial cannot halve the figure. The verdict ranks levels by this figure when it is available. |
| Agent time, p50 to p99 | Time from the agent's first model call to its last action. It leaves out sandbox start, setup, grading, and teardown. |
| Attempt time | The whole trial, including sandbox setup, grading, and cleanup. The wait before the agent's first model call is left out, because the harness starts agents one after another. |
| Model throughput | Output tokens per second across the whole model server. The report also shows prompt tokens per second, because agents send far more prompt tokens than they generate. |
| GPU busy ratio | The share of time the GPUs were doing compute work, measured from the GPU's own activity counters: SM active on NVIDIA, graphics activity on AMD. It is not the coarser utilization percentage a driver shows. On a prefill and decode split, it covers the decode GPUs only. |
| Tokens per GPU-busy-second | Prompt plus output tokens divided by GPU busy time. |
| Sandbox CPU per GPU busy | Agent sandbox CPU time divided by GPU busy time. A high value means the agents' CPU work, not the model, limits the level. |
| Accelerator headroom | How busy the GPUs were at each level, against the minimum GPU busy ratio, and the most KV cache memory they used. Below the floor, the GPUs had spare capacity. |
On AMD GPUs, graphics activity counts any time the GPU had work queued. A level can clear the GPU busy floor while doing little arithmetic. On AMD, read GPU busy together with memory activity and power before you conclude the GPUs are saturated.
| Metric | What it means |
|---|---|
| Peak requests waiting | The most model requests waiting in the model server's queue. A rising queue means the model server is the limit. |
| First-attempt success | The share of tasks that passed on their first attempt. |
| Agent deadline rate | The share of trials that finished within your agent deadline. Shown only when you set a deadline. |
| Serving goodput | The share of model requests within your TTFT and inter-token latency ceilings. Shown only when you set a ceiling. |
| Power per agent | The power of every machine, added up, divided by the number of agents at the level. |
| Cost per completed task | Hardware cost per passed task, computed from hourly host, accelerator, and storage cost rates recorded for the run. The web app has no field for these rates, so the figure is usually blank. It is never shown as zero. |
A metric the run could not capture shows Not captured, Not available on this hardware, or No samples recorded. It never shows as zero.
Find the bottleneck
| What you see at the first level that missed a gate | Likely limit | What to try |
|---|---|---|
| Peak requests waiting rises, GPU busy ratio is high, agent time grows | The model server's GPUs | More GPUs or replicas, a smaller or FP8 model, a prefill and decode split. |
| GPU busy ratio is low, sandbox CPU per GPU busy is high, host CPU is near 100% | Host CPU | More machines, more CPU per sandbox, or Pinned sandbox CPUs to separate agents from the model server. |
| Timeouts rise while the pass rate of finished trials stays high | Time limits | A higher agent time multiplier. |
| Sandboxes fail to start, fleet validity fails | Host memory or disk | Lower the level, add machines, or lower sandbox memory. |
| Time to first token rises sharply on a split run | KV transfer | Check the transport. TCP is much slower than RDMA. |
Inside one level
Pick a level with the level chips, then a view. The arrow keys move between levels. Each chip shows the level's qualification label, and the headline level is marked.
| View | What it shows |
|---|---|
| Summary | What the level counted: how many trials were requested, attempted, passed, and unfinished, with gauge readings. A histogram of trial durations by outcome. Why attempts did not pass. For an Enterprise run, Enterprise evidence: the gates, quality scores, shared services, and agent behavior. |
| System telemetry | Charts over the level's time window: CPU, memory, pressure, disk, network, GPU activity, power, clocks, temperature, model server queue, time to first token, and KV cache use. |
| Harness and shell | Where each trial's time went: harness wait, model calls, tools, gaps between steps, and the time after the last action. Also the time spent in each shell command, summed over every agent. |
| Trace | One trial on a timeline: agent steps, model calls, tool calls, and shell commands. Choose the median, fastest, slowest, or first failed trial. |
Test conditions
The last section lists everything the run was measured under, so a result can be reproduced and compared:
| Group | Examples |
|---|---|
| Hardware on every machine | Chassis, CPU, GPU, GPU driver, memory, operating system, kernel. A value that differs between machines is flagged. |
| SLA | The agent time multiplier, thresholds, deadline, output token cap, reasoning effort, and latency ceilings. |
| Agent | The agent framework and version, the arrival pattern, sandbox isolation, and timeouts. |
| Workload | The task set and revision, the tasks, and each task image's content digest. |
| Environment | The serving topology, replica count, tensor parallel size, KV cache size, prompt caching, the exact vLLM launch command, the Harbor version, the KV transport on a split, and the sandbox CPU mode. |
| Reporting | The thresholds and settings the qualification used. |
Why attempts did not pass
The Why attempts did not pass card groups failed trials by reason, with the message the trials recorded and how many trials share it.
| Reason class | Meaning |
|---|---|
| Task tests did not pass | The agent finished, and the task's own tests failed. This is a quality result, not a platform failure. |
| Agent | The agent's own loop or code failed. |
| Tool | A tool the agent called failed. |
| Model | The model endpoint or a model request failed. |
| Sandbox | The task sandbox failed. |
| Evaluator | The grader failed, independent of the task's quality. |
| Timeout | The trial ran past a time limit. |
| Infrastructure | A machine, container, network, or storage failure. Not counted against the pass rate. |
| Unknown | The error could not be classified safely. |
How a trial ended is recorded separately: graded by the evaluator, timed out, stopped by a guard, reached the agent's turn limit, a runner error, interrupted, or a setup timeout. A setup timeout means the sandbox or agent never finished setup, so the agent never ran. It counts as an infrastructure failure, not a task failure.
Export the data
| Export | What you get | Where |
|---|---|---|
| Download Excel report | One workbook per run: a contents sheet, every table, and every metric time series, grouped by category, with column names and units. | The report header, and the Cmd+K command palette. |
| Export all data | Every row the database holds for the run, as Parquet files in one ZIP, with a manifest of row counts and checksums. The export runs on the server, so you can leave the page. The file stays available for 7 days. | The report header, and the Cmd+K command palette. |
Keyboard and command palette
Press Ctrl+K, or Cmd+K on a Mac, to open the command palette. Type "Agentic Bench" to list its actions:
| Where | Actions |
|---|---|
| Launcher | Choose the dataset, the sandbox CPU mode, and the build variant. Turn the prefill and decode split and the decode prompt cache on or off. |
| Run progress | Show or hide the telemetry sources. |
| Report | Move to the next, previous, or headline level. Open a level's summary, telemetry, harness, or trace view. Trace the median, fastest, slowest, or first failed trial. Go to the verdict, scaling, qualification, or test conditions. Show every curve. Play or pause the level replay. Download the Excel report, or export all run data. |