Skip to main content

Agentic Bench overview

Agentic Bench measures how many AI agents your hardware can run at the same time while the agents still finish their work correctly.

Each agent is a real, tool-using AI agent. It plans, calls tools, edits files, and runs commands inside its own isolated sandbox. A task counts only when an independent grader confirms the result is correct. Agentic Bench raises the number of agents step by step and records, for every step, how many tasks passed, how long they took, and which hardware resource ran out first.

In the product, Agentic Bench appears as the AgentBench workload under Agentic Benchmarking on the New project page.

info

These pages describe Agentic Bench as of 21 October 2026, including the Enterprise Agent and the prefill and decode split. Screenshots use example machine and cluster names.

Questions Agentic Bench answers​

  • How many agents can this hardware run at once before the pass rate or response time drops below my target?
  • Which resource limits the agents first: the GPUs, the CPUs, memory, disk, or the network?
  • How does one model, framework version, or hardware layout compare with another for agent work?
  • Does splitting prompt processing (prefill) and answer generation (decode) across machines help my workload?
  • How many verified tasks per hour does each concurrency level deliver?

Who it is for​

ReaderWhat they use it for
Infrastructure and capacity plannersSize a server or cluster for a target number of agents.
Performance engineersFind the bottleneck and compare serving settings, parallelism, and CPU isolation.
Solution architects and sales engineersShow a customer a measured agent capacity on named hardware.
Product and business ownersRead one verdict: the highest agent count that met the SLA, with evidence.

How a run works​

  1. You choose an agent workload, a model, a group of machines, the agent counts to test, and the pass bar.
  2. Metrum AI Bench turns your machines into one Kubernetes cluster for the run, and starts the model server (llm-d and vLLM) on their GPUs. The cluster is removed when the run ends.
  3. One validation trial runs first, with one agent, to prove the model, the agent, and the sandbox work together.
  4. Each concurrency level then starts that many sandboxes at once, releases all agents together, and grades every finished task.
  5. The report names the highest level that met your pass rate and GPU-busy thresholds, and shows the evidence for every level.

The lifecycle of an Agentic Bench run

For every component of a run, what it changes on your machines, and how your data is handled, see How Agentic Bench works. For the full measurement rules, see Agentic Bench methodology.

Two kinds of agent workload​

Coding AgentEnterprise Agent
What the agent doesSolves Terminal-Bench coding tasks: repairs git repositories, builds software, writes scripts.Works one business case: contract review, document redaction, or email threat analysis.
Task setsTerminal-Bench 2.0, Terminal-Bench 2.1, or any Harbor dataset you name.Enterprise: Contract Review, Enterprise: Document Redaction, Enterprise: Email Threat Analysis.
Tasks per runOne or more task codes, chosen by you.The single case of the task set. Every agent runs a copy of it.
Shared servicesNone.Email Threat Analysis also starts a search index, an embedder, and a reranker.
Default pass bar85% pass rate, 60% GPU busy.75% pass rate, 40% GPU busy, plus quality gates per task set.
GuideConfigure a runEnterprise Agent workloads

Results from the two kinds, and from different task sets, are never averaged or compared with each other.

Supported configurations​

AreaSupported
Serving frameworkvLLM. The framework list also shows SGLang and TensorRT-LLM, but only vLLM has been validated with Agentic Bench.
vLLM versions tested end to end0.27.1, 0.28.0, 0.30.0. Later versions in the list use the same deployment path.
GPUs tested end to endNVIDIA H100 PCIe, NVIDIA L40S, NVIDIA RTX PRO 6000 Blackwell Server Edition, AMD Instinct MI300X.
GPU vendorsNVIDIA (CUDA build) and AMD (ROCm build). Every machine in one run must use the same GPU vendor.
Machines per runOne or more. One machine forms a one-node cluster.
Verified modelsQwen3-4B, Qwen3.5 9B, Qwen3-14B, Qwen 3.8 27B, Qwen 3.8 27B FP8, Qwen3-32B. You can also paste any Hugging Face model ID.
Prefill and decode splitTwo or more machines of one GPU vendor. Pipeline parallel size 1, and tensor parallel size equal to the GPUs used on each machine.
Not supportedIntel accelerators, Google TPU, mixed GPU vendors in one run.

See Tested configurations for complete settings that passed every trial.

Key terms​

TermMeaning
Concurrency levelThe number of agents that run at the same time in one step of the run. Also called a rung.
TrialOne attempt by one agent at one task, from sandbox start to grading.
SandboxThe isolated container one agent works in. Every trial gets a new one.
Cluster groupA named set of your machines that Agentic Bench forms into one Kubernetes cluster.
Valid operating pointA concurrency level that met the minimum pass rate and the minimum GPU busy ratio, and, for an Enterprise run, every enterprise gate. Also called a qualified level.

The Glossary defines every term these pages use.

Where to start​

I want toRead
Understand what runs on my machines and what data is storedHow Agentic Bench works
Run a first test that is known to passQuickstart
Set up a full test, field by fieldConfigure a run, then Hardware and sandboxes
Test business workloadsEnterprise Agent workloads
Compare prefill and decode on separate machinesSplit prefill and decode
Read a report and find the bottleneckMonitor and read results
Know exactly how the numbers are producedMethodology
Start from settings that already workedTested configurations
Fix a failed runTroubleshooting
Let an AI assistant plan or check a runFor AI assistants

What's next​