Agentic Bench settings reference
This page lists every setting of an AgentBench workload. Each row gives the label you see in the web app, the step it belongs to, whether it is required, its default, the values it accepts, and its dependencies. For an explanation of each setting with screenshots, follow the link in the Guide column.
Project
| Setting | Required | Default | Allowed values | Guide |
|---|---|---|---|---|
| Project name | Yes | Empty | Any text | Configure a run |
| Who can see it | Yes | Team | Team, Private | Configure a run |
Step 1: Agent Config
| Setting | Required | Default | Allowed values | Notes |
|---|---|---|---|---|
| Workload | Yes | Coding Agent | Coding Agent, Enterprise Agent | Changing it replaces the task set and task codes. |
| Task set | Yes | Terminal-Bench 2.0 (Coding Agent), Enterprise: Contract Review (Enterprise Agent) | Coding Agent: Terminal-Bench 2.0, Terminal-Bench 2.1, or any Harbor dataset as name@revision. Enterprise Agent: Contract Review, Document Redaction, Email Threat Analysis. | A typed revision must be exact. latest, head, main, and master are refused. |
| Task codes | Yes | The task set's default task | Coding Agent: one or more task directory names from the dataset. Enterprise Agent: the task set's one case, fixed. | The first task code is the validation task. |
Step 2: Model Config
| Setting | Required | Default | Allowed values | Notes |
|---|---|---|---|---|
| Serving framework | Yes | vLLM | vLLM. SGLang and TensorRT-LLM are listed but not validated. | A prefill and decode split needs vLLM. |
| Model | Yes | None | A verified model, or any Hugging Face model ID | An unverified model needs tool_call_parser. |
| Version | Yes | None | A framework version from the list | Recorded with every result. |
| Build variant | No | Automatic | Automatic, Default (NVIDIA CUDA 13.0), NVIDIA CUDA 12.9, AMD ROCm (MI300X / MI355X) | A variant for another GPU vendor is refused at run start. |
Engine args: enable_auto_tool_choice | Yes | true | true | Filled in for every model. |
Engine args: tool_call_parser | Yes | From the model profile | A vLLM tool call parser name, such as hermes, qwen3_coder, llama3_json | Empty for an unverified model until you enter it. The run cannot be saved without it. |
Engine args: reasoning_parser | No | From the model profile | A vLLM reasoning parser name, such as qwen3 | Only for reasoning models. |
| Engine args: custom | No | None | Any vLLM launch flag, without leading dashes | Example: max_model_len = 32768. |
Step 3: Hardware
| Setting | Required | Default | Allowed values | Notes |
|---|---|---|---|---|
| Cluster group | Yes | None | A cluster group of your organization | One machine or more. Same GPU vendor on every machine. |
| GPUs per node | Yes | The smallest GPU count in the group | 1 to the smallest GPU count in the group | Applies to every machine. |
| Tensor parallel size | Yes | 1 | 1 to GPUs per node | Must equal GPUs per node for a prefill and decode split. |
| Pipeline parallel size | No | 1 | 1 to the number of machines | The number of machines must divide evenly by it. Must be 1 for a split. |
| Sandbox CPUs | No | Unpinned | Unpinned, Control, Pinned | Pinned needs CPUs for at least one machine. |
| Pinned CPUs per machine | With Pinned | Empty | Kernel CPU list, for example 8-15 or 8-15,64-71 | Cannot include the CPUs kept for metrics collection. |
| Split prefill and decode across machines | No | Off | On, Off | See the split requirements below. |
| Prefill machines | With a split | 1 | 1 to the number of machines minus 1 | The first machines of the group prefill. |
| Prompt cache on decode machines | With a split | On | On, Off | Off sends every prompt's whole KV cache from prefill. |
| KV cache precision | No, split only | Automatic | Automatic, FP8, Model's own precision | Automatic tries FP8 and falls back to the model's precision. |
| Maximum context length | No, split only | Automatic | 1,024 to 10,000,000 tokens | Automatic uses the model's limit. |
| KV transfer path | No, split only | Automatic | Automatic, GPU to GPU over RDMA, Through host memory | Applies when KV moves over RDMA. |
| Prefill KV cache per GPU | No, split only | Automatic | 1 to 100,000 GiB | Refused when the GPU or machine cannot hold it. |
| Prefill GPU memory share | No, split only | Automatic | 0.10 to 0.99 | |
| Prefill requests at once | No, split only | Automatic | 1 to 100,000 | |
| Prefill tokens per step | No, split only | Automatic | 16 to 10,000,000 | |
| Decode KV cache per GPU | No, split only | Automatic | 1 to 100,000 GiB | Bounds how many requests decode holds at once. |
| Decode GPU memory share | No, split only | Automatic | 0.10 to 0.99 | |
| Decode requests at once | No, split only | Automatic | 1 to 100,000 | |
| Decode tokens per step | No, split only | Automatic | 16 to 10,000,000 | |
| Sandbox memory | No | 4096 MiB | 128 to 4,194,304 MiB | The memory limit of each sandbox. |
| Sandbox CPU | No | 1000 millicores | 100 to 1,024,000 millicores | 1000 millicores is one core. |
| Sandbox disk reservation | No | Automatic | 16 to 10,485,760 MiB | Automatic reserves 256 MiB per sandbox up to 768 sandboxes. |
Prefill and decode split requirements
| Requirement | Value |
|---|---|
| Machines in the cluster group | 2 or more |
| GPU vendor | One vendor on every machine, every machine has reported its GPUs |
| Pipeline parallel size | 1 |
| Tensor parallel size | Equal to GPUs per node |
| Serving framework | vLLM |
| Plan | Includes Distributed KV runs |
| KV transport | RDMA when every machine has an active RoCE v2 port, otherwise TCP. Not selectable. |
Step 4: Load
| Setting | Required | Default | Allowed values | Notes |
|---|---|---|---|---|
| Concurrency levels | Yes | 1, 8, 32 | Positive whole numbers, each used once | Sorted into increasing order. Every level runs. |
| Trials per task | Yes | 3 | 1 or more | A level runs at least as many trials as it has agents. |
Step 5: SLA and validation
| Setting | Required | Default | Allowed values | Notes |
|---|---|---|---|---|
| Validation task | Read-only | The first task code | Runs once with one agent before the levels. | |
| Agent time multiplier | No | 1 | 1 to 576 | Multiplies the agent time limit only. |
| Minimum pass rate | No | 85% Coding Agent, 75% Enterprise Agent | Greater than 0, up to 100% | Gate for a valid operating point. |
| Minimum GPU busy ratio | No | 60% Coding Agent, 40% Enterprise Agent | 0 to 100% | Gate for a valid operating point. Decode GPUs only on a split. |
| Agent deadline | No | Off | Seconds, greater than 0 | Reporting only. |
| Output token cap | No | No cap | Tokens, greater than 0 | Changes agent behavior. |
| Reasoning effort | No | Model default | Model default, Minimal, Low, Medium, High | For models that support it. |
| TTFT ceiling | No | Off | Seconds, greater than 0 | Serving goodput only. |
| Inter-token latency ceiling | No | Off | Seconds, greater than 0 | Serving goodput only. |
Fixed values
These values are not settings. They apply to every run.
| Value | Amount |
|---|---|
| Safety allowance added to each trial phase | 35 minutes |
| Agent setup limit | 6 minutes |
| Level setup allowance | 30 minutes |
| Shortest level time limit | 2 hours |
| Longest level time limit | 168 hours (7 days) |
| CPUs kept for metrics collection on each machine | 1 physical core, 2 on a machine with 64 or more CPUs |
| Clock synchronization required | chrony synchronized to within 0.5 seconds |
| Model server ports on each machine, prefill and decode split over RDMA | 8000 (prefill) and 8200 (decode) |
| Enterprise model call with no output | Fails after 1,400 seconds |
Verified models
| Model | Context length | Tool call parser | Reasoning parser | Thinking mode |
|---|---|---|---|---|
| Qwen3-4B | 40,960 | hermes | qwen3 | Model default |
| Qwen3.5 9B | 262,144 | qwen3_coder | qwen3 | Off |
| Qwen3-14B | 40,960 | hermes | qwen3 | Model default |
| Qwen 3.8 27B | 262,144 | qwen3_coder | qwen3 | Off |
| Qwen 3.8 27B FP8 | 262,144 | qwen3_coder | qwen3 | Off |
| Qwen3-32B | 40,960 | hermes | qwen3 | Model default |
Task time limits
| Task set | Task code | Setup | Agent | Grading |
|---|---|---|---|---|
| Terminal-Bench 2.0 | caffe-cifar-10 | 10 min | 20 min | 20 min |
| Terminal-Bench 2.0 | crack-7z-hash | 10 min | 15 min | 15 min |
| Terminal-Bench 2.0 | filter-js-from-html | 10 min | 30 min | 15 min |
| Terminal-Bench 2.0 | fix-git | 10 min | 15 min | 15 min |
| Terminal-Bench 2.0 | git-leak-recovery | 10 min | 15 min | 15 min |
| Terminal-Bench 2.0 | git-multibranch | 10 min | 15 min | 15 min |
| Terminal-Bench 2.0 | log-summary-date-ranges | 10 min | 15 min | 15 min |
| Terminal-Bench 2.0 | modernize-scientific-stack | 10 min | 10 min | 10 min |
| Terminal-Bench 2.0 | portfolio-optimization | 10 min | 60 min | 60 min |
| Terminal-Bench 2.0 | pypi-server | 10 min | 15 min | 15 min |
| Terminal-Bench 2.0 | query-optimize | 10 min | 15 min | 15 min |
| Terminal-Bench 2.1 | caffe-cifar-10 | 10 min | 60 min | 20 min |
| Terminal-Bench 2.1 | crack-7z-hash | 10 min | 30 min | 15 min |
| Terminal-Bench 2.1 | filter-js-from-html | 10 min | 30 min | 30 min |
| Terminal-Bench 2.1 | fix-git | 10 min | 15 min | 15 min |
| Terminal-Bench 2.1 | git-leak-recovery | 10 min | 15 min | 15 min |
| Terminal-Bench 2.1 | git-multibranch | 10 min | 15 min | 15 min |
| Terminal-Bench 2.1 | log-summary-date-ranges | 10 min | 15 min | 15 min |
| Terminal-Bench 2.1 | modernize-scientific-stack | 10 min | 10 min | 10 min |
| Terminal-Bench 2.1 | portfolio-optimization | 10 min | 60 min | 60 min |
| Terminal-Bench 2.1 | pypi-server | 10 min | 15 min | 15 min |
| Terminal-Bench 2.1 | query-optimize | 10 min | 15 min | 30 min |
| Enterprise: Contract Review | contract-review-128k-case-0021 | 10 min | 100 min | 20 min |
| Enterprise: Document Redaction | case-0029 | 10 min | 90 min | 20 min |
| Enterprise: Email Threat Analysis | email-threat-analysis-case-0051 | 10 min | 100 min | 3 min |
Worked example of a level limit
For fix-git at an agent time multiplier of 2, one trial's worst case is:
| Part | Time |
|---|---|
| Setup limit | 10 min |
| Agent setup limit | 6 min |
| Agent limit × multiplier | 15 min × 2 = 30 min |
| Grading limit | 15 min |
| Safety allowance, 35 min for each of the 4 phases | 140 min |
| Slowest trial | 201 min (3 h 21 min) |
With 3 tasks of this size and 3 trials per task, a level of 8 agents runs 9 trials. Its limit is the 30-minute setup allowance, plus the 9 trials' total time spread over 8 agents, plus one slowest trial: 7 h 37 min. A level with as many agents as trials runs them all at once, so its limit is the setup allowance plus one slowest trial, raised to the 2-hour minimum if needed.
What's next
- Configure an Agentic Bench run
- For AI assistants: the same settings as a machine-readable specification.