Skip to main content

Monitor a run and read the report

This page explains what you see while an Agentic Bench run is in progress, and how to read the report when it finishes.

Monitor a run​

Open the run from Monitoring. The page shows, from top to bottom:

ElementWhat it shows
Workload ConfigurationEvery setting the run uses: task set, tasks, model, version, cluster group, parallelism, sandbox CPU mode, prefill and decode split, levels, and SLA.
Status bannerWhat the run is doing now, in plain words, for example "Downloading model weights and loading them onto the GPUs".
Queue noticeWhen the run waits for its cluster group: how many runs are ahead.
Stage trackerEach stage of the run, with its state, start time, duration, and the reason for a failure.
Rungs tableThe validation trial and every concurrency level, with status, task progress, start time, and duration.
Telemetry badgeThe health of each telemetry source. It opens the per-source detail when a source is delayed or failing.

Run stages​

StageWhat happensTypical time
Host checksChecks free disk, memory, GPU memory, network ports, clock synchronization, and the container runtime on each machine.Seconds
Benchmark runtimePrepares the Agentic Bench runtime, the task dataset, and the task images.Minutes the first time, faster after
Kubernetes clusterForms the Kubernetes cluster over the group's machines, one machine included, and installs what the model server needs.A few minutes
Model deploymentDeploys the model server onto the selected machines.Under a minute
Pods startingSchedules the model server and pulls its container images.Minutes
Model loadingDownloads the model weights and loads them onto the GPUs.Depends on model size and network speed
Serving endpointRoutes traffic to the model server and waits for it to answer.Under a minute
Validation trialRuns one trial with one agent to prove the sandbox, the agent, and the model endpoint work together.About as long as one task
MeasurementRuns each concurrency level and records its results.See the level durations below
TeardownStops the model server and releases the machines.A few minutes

Each stage shows Pending, Running, Passed, Failed, or Skipped. A failed stage shows the reason the machine reported.

How a level runs​

  1. The level creates one sandbox per agent and waits until every sandbox is ready.
  2. All agents start at the same moment. The release time is recorded, and throughput is measured from it.
  3. Each agent works a task. When it finishes, it takes the next trial until the level's trials are done.
  4. A grader scores each finished trial with the task's own tests.
  5. The level's telemetry and results are stored, and the next level starts.

If some sandboxes fail to start, or the wait for them times out, the level runs below its target count, and the report marks it. An Enterprise level in that state fails its fleet validity gate.

Example level durations from Metrum validation runs of the fix-git task:

HardwareLevelLevel timeMedian trial
2 × RTX PRO 6000, one GPU per machine64 agents4 min2.5 min
2 × RTX PRO 6000, one GPU per machine256 agents16 min5.4 min
2 × L40S machines, split over TCP256 agents55 min41 min
2 × L40S machines, split over TCP600 agents144 min105 min

Level status in the Rungs table​

StatusMeaning
Pending, QueuedThe level has not started.
RunningThe level's sandboxes or agents are running.
CompletedEvery planned trial ran and finished. This includes a level where some tasks failed their tests.
PartialThe level stopped with trials unfinished.
FailedThe level ran no trial, or the platform failed before it measured anything.
CancelledThe level was cancelled, for example because the validation trial failed for a platform reason.
Timed outThe level ran past its level time limit and was stopped as stalled.

Read the report​

The report has six sections. Use the section links at the top to move between them.

SectionWhat it answers
VerdictWhat is the highest agent count that met the SLA?
ScalingHow do pass rate, goodput, throughput, agent time, and GPU headroom change from level to level?
QualificationWhich levels qualify, and which gate did each other level miss?
Every curveAll charts across all levels.
Inside one levelWhat happened inside one level: telemetry, harness timing, shell time, and a trace of one attempt.
Test conditionsThe hardware on every machine, the SLA, and every recorded factor of the run.

Verdict​

The verdict names the highest concurrency level that is a valid operating point. A level is a valid operating point when:

  • its pass rate is at least the minimum pass rate, and
  • its GPU busy ratio is at least the minimum GPU busy ratio, and
  • for an Enterprise run, every enterprise gate passes.

When a higher level missed a gate, the verdict also names the first such level. That level is where the hardware stopped meeting your SLA.

When no level qualifies, the verdict shows the finished level with the most verified goodput and marks it as not qualified.

The verdict section also holds a level replay: a control that plays the headline level back over time, so you can see agents start, work, and finish. Read the Qualification section to see which gate each level missed.

Qualification​

Each level carries one label:

LabelMeaning
QualifiedThe level met every gate. You can size hardware from it.
Missed a gateThe level finished but missed at least one gate. The label names the gate and the measured value.
Not gatedThe level could not be judged: it is still running, it did not finish, it did not run, or no attempt finished.

Size hardware only from a Qualified level. A level that missed a gate tells you what happened at that load, not what the hardware can sustain.

Key metrics​

MetricWhat it means
Pass ratePassed trials divided by graded trials. Trials the platform lost, such as a sandbox that never started or a grader that could not run, are left out of the denominator and counted separately.
Verified goodputPassed tasks per hour across all agents. It counts only tasks the grader confirmed.
Verified goodput (to p95 finish)Passed tasks per hour up to the moment 95% of the level's trials had finished. One very long trial cannot halve the figure. The verdict ranks levels by this figure when it is available.
Agent time, p50 to p99Time from the agent's first model call to its last action. It leaves out sandbox start, setup, grading, and teardown.
Attempt timeThe whole trial, including sandbox setup, grading, and cleanup. The wait before the agent's first model call is left out, because the harness starts agents one after another.
Model throughputOutput tokens per second across the whole model server. The report also shows prompt tokens per second, because agents send far more prompt tokens than they generate.
GPU busy ratioThe share of time the GPUs were doing compute work, measured from the GPU's own activity counters: SM active on NVIDIA, graphics activity on AMD. It is not the coarser utilization percentage a driver shows. On a prefill and decode split, it covers the decode GPUs only.
Tokens per GPU-busy-secondPrompt plus output tokens divided by GPU busy time.
Sandbox CPU per GPU busyAgent sandbox CPU time divided by GPU busy time. A high value means the agents' CPU work, not the model, limits the level.
Accelerator headroomHow busy the GPUs were at each level, against the minimum GPU busy ratio, and the most KV cache memory they used. Below the floor, the GPUs had spare capacity.
note

On AMD GPUs, graphics activity counts any time the GPU had work queued. A level can clear the GPU busy floor while doing little arithmetic. On AMD, read GPU busy together with memory activity and power before you conclude the GPUs are saturated.

MetricWhat it means
Peak requests waitingThe most model requests waiting in the model server's queue. A rising queue means the model server is the limit.
First-attempt successThe share of tasks that passed on their first attempt.
Agent deadline rateThe share of trials that finished within your agent deadline. Shown only when you set a deadline.
Serving goodputThe share of model requests within your TTFT and inter-token latency ceilings. Shown only when you set a ceiling.
Power per agentThe power of every machine, added up, divided by the number of agents at the level.
Cost per completed taskHardware cost per passed task, computed from hourly host, accelerator, and storage cost rates recorded for the run. The web app has no field for these rates, so the figure is usually blank. It is never shown as zero.

A metric the run could not capture shows Not captured, Not available on this hardware, or No samples recorded. It never shows as zero.

Find the bottleneck​

What you see at the first level that missed a gateLikely limitWhat to try
Peak requests waiting rises, GPU busy ratio is high, agent time growsThe model server's GPUsMore GPUs or replicas, a smaller or FP8 model, a prefill and decode split.
GPU busy ratio is low, sandbox CPU per GPU busy is high, host CPU is near 100%Host CPUMore machines, more CPU per sandbox, or Pinned sandbox CPUs to separate agents from the model server.
Timeouts rise while the pass rate of finished trials stays highTime limitsA higher agent time multiplier.
Sandboxes fail to start, fleet validity failsHost memory or diskLower the level, add machines, or lower sandbox memory.
Time to first token rises sharply on a split runKV transferCheck the transport. TCP is much slower than RDMA.

Inside one level​

Pick a level with the level chips, then a view. The arrow keys move between levels. Each chip shows the level's qualification label, and the headline level is marked.

ViewWhat it shows
SummaryWhat the level counted: how many trials were requested, attempted, passed, and unfinished, with gauge readings. A histogram of trial durations by outcome. Why attempts did not pass. For an Enterprise run, Enterprise evidence: the gates, quality scores, shared services, and agent behavior.
System telemetryCharts over the level's time window: CPU, memory, pressure, disk, network, GPU activity, power, clocks, temperature, model server queue, time to first token, and KV cache use.
Harness and shellWhere each trial's time went: harness wait, model calls, tools, gaps between steps, and the time after the last action. Also the time spent in each shell command, summed over every agent.
TraceOne trial on a timeline: agent steps, model calls, tool calls, and shell commands. Choose the median, fastest, slowest, or first failed trial.

Test conditions​

The last section lists everything the run was measured under, so a result can be reproduced and compared:

GroupExamples
Hardware on every machineChassis, CPU, GPU, GPU driver, memory, operating system, kernel. A value that differs between machines is flagged.
SLAThe agent time multiplier, thresholds, deadline, output token cap, reasoning effort, and latency ceilings.
AgentThe agent framework and version, the arrival pattern, sandbox isolation, and timeouts.
WorkloadThe task set and revision, the tasks, and each task image's content digest.
EnvironmentThe serving topology, replica count, tensor parallel size, KV cache size, prompt caching, the exact vLLM launch command, the Harbor version, the KV transport on a split, and the sandbox CPU mode.
ReportingThe thresholds and settings the qualification used.

Why attempts did not pass​

The Why attempts did not pass card groups failed trials by reason, with the message the trials recorded and how many trials share it.

Reason classMeaning
Task tests did not passThe agent finished, and the task's own tests failed. This is a quality result, not a platform failure.
AgentThe agent's own loop or code failed.
ToolA tool the agent called failed.
ModelThe model endpoint or a model request failed.
SandboxThe task sandbox failed.
EvaluatorThe grader failed, independent of the task's quality.
TimeoutThe trial ran past a time limit.
InfrastructureA machine, container, network, or storage failure. Not counted against the pass rate.
UnknownThe error could not be classified safely.

How a trial ended is recorded separately: graded by the evaluator, timed out, stopped by a guard, reached the agent's turn limit, a runner error, interrupted, or a setup timeout. A setup timeout means the sandbox or agent never finished setup, so the agent never ran. It counts as an infrastructure failure, not a task failure.

Export the data​

ExportWhat you getWhere
Download Excel reportOne workbook per run: a contents sheet, every table, and every metric time series, grouped by category, with column names and units.The report header, and the Cmd+K command palette.
Export all dataEvery row the database holds for the run, as Parquet files in one ZIP, with a manifest of row counts and checksums. The export runs on the server, so you can leave the page. The file stays available for 7 days.The report header, and the Cmd+K command palette.

Keyboard and command palette​

Press Ctrl+K, or Cmd+K on a Mac, to open the command palette. Type "Agentic Bench" to list its actions:

WhereActions
LauncherChoose the dataset, the sandbox CPU mode, and the build variant. Turn the prefill and decode split and the decode prompt cache on or off.
Run progressShow or hide the telemetry sources.
ReportMove to the next, previous, or headline level. Open a level's summary, telemetry, harness, or trace view. Trace the median, fastest, slowest, or first failed trial. Go to the verdict, scaling, qualification, or test conditions. Show every curve. Play or pause the level replay. Download the Excel report, or export all run data.

What's next​