Agentic Bench overview
Agentic Bench measures how many AI agents your hardware can run at the same time while the agents still finish their work correctly.
Each agent is a real, tool-using AI agent. It plans, calls tools, edits files, and runs commands inside its own isolated sandbox. A task counts only when an independent grader confirms the result is correct. Agentic Bench raises the number of agents step by step and records, for every step, how many tasks passed, how long they took, and which hardware resource ran out first.
In the product, Agentic Bench appears as the AgentBench workload under Agentic Benchmarking on the New project page.
These pages describe Agentic Bench as of 21 October 2026, including the Enterprise Agent and the prefill and decode split. Screenshots use example machine and cluster names.
Questions Agentic Bench answers
- How many agents can this hardware run at once before the pass rate or response time drops below my target?
- Which resource limits the agents first: the GPUs, the CPUs, memory, disk, or the network?
- How does one model, framework version, or hardware layout compare with another for agent work?
- Does splitting prompt processing (prefill) and answer generation (decode) across machines help my workload?
- How many verified tasks per hour does each concurrency level deliver?
Who it is for
| Reader | What they use it for |
|---|---|
| Infrastructure and capacity planners | Size a server or cluster for a target number of agents. |
| Performance engineers | Find the bottleneck and compare serving settings, parallelism, and CPU isolation. |
| Solution architects and sales engineers | Show a customer a measured agent capacity on named hardware. |
| Product and business owners | Read one verdict: the highest agent count that met the SLA, with evidence. |
How a run works
- You choose an agent workload, a model, a group of machines, the agent counts to test, and the pass bar.
- Metrum AI Bench turns your machines into one Kubernetes cluster for the run, and starts the model server (llm-d and vLLM) on their GPUs. The cluster is removed when the run ends.
- One validation trial runs first, with one agent, to prove the model, the agent, and the sandbox work together.
- Each concurrency level then starts that many sandboxes at once, releases all agents together, and grades every finished task.
- The report names the highest level that met your pass rate and GPU-busy thresholds, and shows the evidence for every level.
For every component of a run, what it changes on your machines, and how your data is handled, see How Agentic Bench works. For the full measurement rules, see Agentic Bench methodology.
Two kinds of agent workload
| Coding Agent | Enterprise Agent | |
|---|---|---|
| What the agent does | Solves Terminal-Bench coding tasks: repairs git repositories, builds software, writes scripts. | Works one business case: contract review, document redaction, or email threat analysis. |
| Task sets | Terminal-Bench 2.0, Terminal-Bench 2.1, or any Harbor dataset you name. | Enterprise: Contract Review, Enterprise: Document Redaction, Enterprise: Email Threat Analysis. |
| Tasks per run | One or more task codes, chosen by you. | The single case of the task set. Every agent runs a copy of it. |
| Shared services | None. | Email Threat Analysis also starts a search index, an embedder, and a reranker. |
| Default pass bar | 85% pass rate, 60% GPU busy. | 75% pass rate, 40% GPU busy, plus quality gates per task set. |
| Guide | Configure a run | Enterprise Agent workloads |
Results from the two kinds, and from different task sets, are never averaged or compared with each other.
Supported configurations
| Area | Supported |
|---|---|
| Serving framework | vLLM. The framework list also shows SGLang and TensorRT-LLM, but only vLLM has been validated with Agentic Bench. |
| vLLM versions tested end to end | 0.27.1, 0.28.0, 0.30.0. Later versions in the list use the same deployment path. |
| GPUs tested end to end | NVIDIA H100 PCIe, NVIDIA L40S, NVIDIA RTX PRO 6000 Blackwell Server Edition, AMD Instinct MI300X. |
| GPU vendors | NVIDIA (CUDA build) and AMD (ROCm build). Every machine in one run must use the same GPU vendor. |
| Machines per run | One or more. One machine forms a one-node cluster. |
| Verified models | Qwen3-4B, Qwen3.5 9B, Qwen3-14B, Qwen 3.8 27B, Qwen 3.8 27B FP8, Qwen3-32B. You can also paste any Hugging Face model ID. |
| Prefill and decode split | Two or more machines of one GPU vendor. Pipeline parallel size 1, and tensor parallel size equal to the GPUs used on each machine. |
| Not supported | Intel accelerators, Google TPU, mixed GPU vendors in one run. |
See Tested configurations for complete settings that passed every trial.
Key terms
| Term | Meaning |
|---|---|
| Concurrency level | The number of agents that run at the same time in one step of the run. Also called a rung. |
| Trial | One attempt by one agent at one task, from sandbox start to grading. |
| Sandbox | The isolated container one agent works in. Every trial gets a new one. |
| Cluster group | A named set of your machines that Agentic Bench forms into one Kubernetes cluster. |
| Valid operating point | A concurrency level that met the minimum pass rate and the minimum GPU busy ratio, and, for an Enterprise run, every enterprise gate. Also called a qualified level. |
The Glossary defines every term these pages use.
Where to start
| I want to | Read |
|---|---|
| Understand what runs on my machines and what data is stored | How Agentic Bench works |
| Run a first test that is known to pass | Quickstart |
| Set up a full test, field by field | Configure a run, then Hardware and sandboxes |
| Test business workloads | Enterprise Agent workloads |
| Compare prefill and decode on separate machines | Split prefill and decode |
| Read a report and find the bottleneck | Monitor and read results |
| Know exactly how the numbers are produced | Methodology |
| Start from settings that already worked | Tested configurations |
| Fix a failed run | Troubleshooting |
| Let an AI assistant plan or check a run | For AI assistants |