Skip to main content

Agentic Bench methodology

Agentic Bench measures how many concurrent AI agents a given set of machines can sustain while the agents still finish their work correctly. This page explains how that number is produced, what it depends on, and which conditions change it.

For how to set up a run in the web app, see Configure an Agentic Bench run.

Principles​

  1. Measured, not extrapolated. Every number comes from the hardware under test, running the model you selected. Nothing is estimated from published benchmarks or vendor data sheets.
  2. Only verified work counts. A task counts as passed only when an independent grader confirms the result. The agent cannot grade its own work. Failed, unfinished, or timed-out work never counts as capacity.
  3. Every condition is recorded. The hardware, software versions, settings, and serving layout of each level are stored with its results, so a result can be reproduced and compared fairly.
  4. Unknown is never zero. A value the run could not measure is shown as not captured or not available, never as zero.
  5. Platform failures are kept apart from task failures. A sandbox that never started is a platform failure, not a wrong answer by the agent.

What is under test​

Agentic Bench tests the whole system an agent runs on, not the model alone.

PartRole in the measurement
The GPUs and the model serverServe every agent's model calls. Shared by all agents in a level.
The modelPlans, chooses tools, and writes answers. Its quality decides the pass rate.
The agentRuns the loop: model call, tool call, observe, repeat. Coding Agent runs use the open-source Deep Agents agent. Enterprise Agent runs use the Metrum enterprise agent.
The sandboxesOne isolated container per agent, where the agent's tools and commands run. They use the machines' CPUs, memory, and disk.
The host machinesCPUs, memory, disk, and network shared by the model server, the sandboxes, and any shared services.
The network between machinesCarries model traffic between machines and, on a prefill and decode split, the KV cache.

The software is built from open-source components:

  • Kubernetes joins the machines into one cluster for the run.
  • llm-d and vLLM serve the model.
  • Harbor creates the sandboxes and grades the trials.
  • Deep Agents is the coding agent.
  • Coding tasks come from Terminal-Bench.

For what each component does and where it runs, see How Agentic Bench works.

How load is applied​

A fixed ladder of concurrency levels​

You choose the exact concurrency levels to test, for example 1, 8, 32, and 64. Each level is one step of the ladder, also called a rung. The levels run in increasing order.

  • Every level you set runs. The run does not stop early when a level misses the SLA, and it does not choose levels for you.
  • A level that fails does not stop the levels after it.
  • The report decides afterwards which levels qualify.

Closed-loop agents​

Concurrency is the number of agents working at the same moment. It is not a request rate. Each agent runs one trial at a time. When an agent finishes a trial, it starts the next one, until the level's trials are done. The load the model server sees therefore depends on how fast the agents work, the same way it does in production.

Trials per level​

A level runs at least one trial per agent, and every task the same number of times:

Trials at a level = tasks × the larger of (trials per task) and (level ÷ tasks, rounded up)

For example, 3 tasks with 3 trials per task run 9 trials at 1 and 8 agents, 33 trials at 32 agents, and 66 at 64 agents. At the 1-agent level, the 9 trials run one after another.

Start barrier​

At the start of each level, every sandbox is created and the level waits until all of them are ready. Then all agents start at the same moment. Throughput is measured from that moment, so sandbox start-up time does not dilute it. If some sandboxes fail to start, or the wait times out, the level runs below its target count, and the report records why.

The validation trial​

Before the first level, one trial of the first task runs with one agent, on the same model server, agent, and sandbox setup as the sweep.

Validation outcomeEffect
PassedThe levels run.
Graded as failed, or timed outThe levels still run. The platform worked, and a model that fails one task can still be measured.
A platform failure, such as a sandbox that did not start or a model endpoint that did not answerEvery level is cancelled, so no time is spent on a sweep that cannot produce a result.

Time limits​

Each task has its own time limits from its dataset definition: setup, agent work, and grading.

  • The agent time multiplier multiplies the agent limit only. Its default is 1, and it can be set from 1 to 576. Many concurrent agents share one model server, so each agent gets fewer tokens per second, and a task can need more time than its own limit.
  • Each phase gets a fixed safety allowance of 35 minutes above the task's own limit.
  • A trial that passes a limit is stopped and recorded as a timeout.
  • Each level has a level limit: every trial at its own ceiling, scheduled across the level's agents, plus a setup allowance. It is never below 2 hours and never above 7 days. A level that passes its limit is stopped as stalled.
  • A task that is not in Metrum's catalog for the dataset is budgeted as the slowest known task, never as a short one.

How work is graded​

  • Coding Agent. Each Terminal-Bench task ships its own tests. After the agent finishes, Harbor runs those tests in the task's sandbox. The tests decide pass or fail. The output of every test is stored with the trial, so a failure can be explained from the report.
  • Enterprise Agent. Each case has its own grader that scores the agent's output against known answers. Contract review is scored on clause categories and quoted spans. Redaction is scored on unmasked personal data. Email triage is scored on verdicts and cited evidence.
  • Partial credit. A grader may award a score between 0 and 1. The pass rate counts passes. The task completeness score averages the scores, so it can differ from the pass rate.

What the pass rate counts​

Pass rate = passed trials ÷ graded trials

Graded trials leave out trials the platform lost: a sandbox that never started, a setup that timed out before the agent ran, or a grader that could not run. Those are counted separately as infrastructure failures, so a platform problem never lowers the agent's score. The report shows the counts beside the rate, for example 242/598, so you can see how many trials were graded.

Every failed trial carries a reason class: task tests did not pass, agent, tool, model, sandbox, evaluator, timeout, infrastructure, or unknown. A trial stopped by a runaway guard is never counted the same as a wrong answer.

How a level qualifies​

A level is a valid operating point when it meets every gate:

GateCoding Agent defaultEnterprise Agent default
Minimum pass rate85%75%
Minimum GPU busy ratio60%40%
Enterprise integrity, completion, and quality gatesNot appliedAll must pass

You can set both thresholds for each run. The GPU busy ratio gate makes sure a level that passes did so while it actually used the hardware. A level that passes every task while the GPUs sit mostly idle says little about capacity.

The verdict is the highest level that qualifies. When a higher level missed a gate, the first such level is named as the point where the hardware stopped meeting the SLA. When no level qualifies, the report shows the finished level with the most verified goodput and marks it as not qualified.

What is measured​

Measurement windows​

FigureWindow
Throughput and goodputFrom the start barrier to the end of the level.
Goodput to p95 finishFrom the start barrier until 95% of the level's trials have finished. One very long trial cannot dominate it.
Agent timeFrom the agent's first model call to its last action. Sandbox setup, the wait before the first model call, grading, and cleanup are left out. This is the time the agent deadline is measured against.
GPU busyThe level's window, without the start-up time before the first model call.

Resource telemetry​

Each machine reports telemetry throughout the run. Telemetry and grading are kept apart: telemetry never decides whether a task passed.

SourceWhat it covers
HostCPU use, run queue, CPU and memory pressure, memory, disk input and output, network traffic and retransmits, power from the server's management controller.
GPUCompute activity, tensor and memory activity, memory use, power, clocks, temperature, power and thermal throttling, PCIe and GPU-to-GPU traffic. NVIDIA and AMD GPUs each use their vendor's counters.
Model serverRequests running and waiting, time to first token, inter-token latency, request latency and queue time, prompt cache hits, KV cache use, preemptions, batch size, prompt and output tokens.
SandboxesCPU, memory, CPU throttling, and process counts for every sandbox.
Agent tracesEvery model call, tool call, and shell command of every trial, with its duration, on one timeline per trial.
Network fabricRoCE, InfiniBand, and network card counters per machine, and KV transfer time and bandwidth on a split.

Each machine keeps one physical core, or two on a machine with 64 or more CPUs, for the telemetry collectors. A full level can therefore never starve the collectors. Machine clocks must be synchronized to within 1 millisecond, so timings from different machines line up.

Resource classes​

CPU and memory figures are reported per class, so infrastructure overhead never inflates or hides what an agent costs.

ClassCovers
AgentThe agent's own reasoning loop.
TaskThe sandbox's tool work: shell commands, file edits, builds.
Model servingThe model server and its GPUs, shared by every agent.
HarnessThe run controller and the orchestration around it.

Sizing ratios​

RatioMeaning
Tokens per GPU-busy-secondPrompt plus output tokens divided by GPU busy time. Model throughput normalized by how busy the GPUs actually were.
Sandbox CPU per GPU busyAgent sandbox CPU core-seconds divided by GPU busy seconds. A high value means the agents' CPU work, not the model, limits the level.
CPU, GPU, and energy per passed taskThe cost of one verified task in CPU time, GPU busy time, and energy.
Cost per completed task(Host, accelerator, and storage cost per hour) × level hours ÷ passed tasks. Shown only when all three rates are set. Never shown as zero.

Model serving​

  • Replicas. The model server runs (machines ÷ pipeline parallel size) × (GPUs per node ÷ tensor parallel size, rounded down) copies of the model, and spreads requests across them.
  • Routing. The router sends each model call to the replica with the shortest request queue and the most free KV cache memory, with equal weight. It does not route by prompt cache content. In Metrum testing, routing by prompt cache sent every request to one replica while the other stayed idle.
  • Tool calling. The agent sends every request with automatic tool choice. vLLM must start with automatic tool choice on and a tool call parser that matches the model, or it rejects the first request. Verified models carry these settings in their profile.
  • The prompt cache between levels. In a Coding Agent run, the model server stays up across levels. Its prompt cache can therefore already hold shared prefixes, such as the agent's system prompt, from earlier levels. In an Enterprise Agent run, the model server and the shared services restart before every level. Each level starts with an empty cache that only its own agents share.
  • Precision and settings are recorded. The framework version, build, engine settings, and any automatic fallback, such as a KV cache precision change, are recorded on every level.

Prefill and decode split​

With a split, prefill machines compute each prompt's KV cache and send it to decode machines, which generate the answer. The split uses vLLM's NIXL connector for both GPU vendors.

  • Each machine runs exactly one engine. Pipeline parallel size must be 1, and tensor parallel size must equal the GPUs used on each machine. This keeps a split comparable with a run without a split on the same GPUs.
  • At least one machine prefills and at least one decodes.
  • KV moves over RDMA when every machine has an active RoCE v2 port, and over TCP otherwise. Each level records which.
  • The GPU busy gate and tokens per GPU-busy-second use the decode machines' GPUs only.
  • Every request is split. Each level records its layout, for example 1P1D, and split levels are never averaged with levels without a split.

Sandbox resources​

Every sandbox at every level gets the same limits, so levels stay comparable.

ResourceRule
CPUEach sandbox reserves one tenth of a core and may use up to its CPU limit, 1 core by default. The kernel shares the cores among the sandboxes on a machine, so CPU contention at high levels is part of the measurement.
MemoryEach sandbox may use up to its memory limit, 4 GiB by default. Before each level, each sandbox reserves an equal share of the free memory below that limit. A level whose sandboxes cannot each reserve the minimum the agent needs does not start.
DiskThe scheduler reserves disk per sandbox, 256 MiB by default up to 768 sandboxes and less above that. The level's disk check uses the disk the sandboxes actually wrote.
PlacementSandboxes spread across every machine in the cluster, so a level draws on the whole cluster's capacity.
CPU setUnpinned uses every CPU except the telemetry cores. Control and Pinned use a pinned runtime, with all CPUs or with the CPUs you choose. A check on every machine before each level confirms the mode is in effect, and the run fails if it is not.

Comparing runs​

Two results are comparable only when these are the same:

Must matchWhy
Task set and its revisionTerminal-Bench 2.0 and 2.1 have different images, limits, and tests for the same task code.
Task codes and trials per taskDifferent tasks have different difficulty and length.
Model, framework version, and buildEach changes speed and quality.
Output token cap and reasoning effortBoth change how the agent behaves and how long it works.
Agent time multiplierIt changes which slow trials count as timeouts.
Parallelism and replica countThey change how much capacity the model server has.
Prefill and decode split, and its transportTCP and RDMA results are not comparable.
Sandbox CPU mode and sandbox limitsThey change CPU contention and memory pressure.

Compare levels at the same concurrency, and size hardware only from levels that qualify.

Edge cases and limitations​

Configuration​

  • A split needs tensor parallel size equal to GPUs per node, and pipeline parallel size 1. Any other layout is refused. A split engine that used fewer GPUs than the baseline would compare unlike hardware.
  • A split needs two or more machines of one GPU vendor, and a plan that includes Distributed KV runs.
  • GPUs per node is capped by the smallest machine in the cluster group. Extra GPUs on larger machines stay unused.
  • GPUs that do not divide by the tensor parallel size stay idle. 3 GPUs at tensor parallel size 2 serve one copy and leave one GPU unused.
  • The number of machines must divide evenly by the pipeline parallel size.
  • Only vLLM is validated. SGLang and TensorRT-LLM appear in the framework list but have not been validated with Agentic Bench.
  • An unverified model needs its tool call parser. A wrong parser saves without error and fails the validation trial, because the agent cannot read the model's tool calls.
  • A typed dataset reference needs an exact revision. latest, head, main, and master are refused. Its tasks are checked only when the run starts.
  • NUMA nodes cannot be pinned as a unit. Memory placement on a NUMA node has not been verified, so the NUMA buttons fill in the node's CPUs instead.

Measurement​

  • The validation trial gates on the platform, not on quality. A validation trial the agent fails still lets the sweep run.
  • Every level runs. A run with levels far above capacity spends time on levels that cannot qualify. Each such level stops at its level time limit at the latest.
  • The 1-agent level runs its trials one after another, so it can take longer than higher levels.
  • One long trial can hold a level open. Whole-level goodput then reads low. Goodput to p95 finish corrects for it, and the verdict uses it when it is available.
  • A high pass rate on few graded trials is weak evidence. Check the graded count beside the rate. A level with many infrastructure failures has a small denominator.
  • GPU busy on AMD counts queued work. Graphics activity can clear the floor while the GPU does little arithmetic. Read it with memory activity and power.
  • The coding prompt cache carries over between levels. Later levels of a Coding Agent run may start with shared prefixes already cached. Enterprise levels always start cold.
  • TCP splits are much slower than RDMA splits. Raise the agent time multiplier for a TCP split, and compare it only with other TCP runs.

Enterprise​

  • Contract Review sends about 122,000 tokens per agent. The model needs a context of at least 131,072 tokens, and KV cache memory limits the agent count.
  • A model call with no output for 1,400 seconds fails. At levels far above what the model server can serve, queued calls reach that limit and the cases fail.
  • Email Threat Analysis runs its search index, embedder, and reranker on host CPU, beside the sandboxes.

Glossary​

See the Agentic Bench glossary.