Skip to main content

Agentic Bench settings reference

This page lists every setting of an AgentBench workload. Each row gives the label you see in the web app, the step it belongs to, whether it is required, its default, the values it accepts, and its dependencies. For an explanation of each setting with screenshots, follow the link in the Guide column.

Project​

SettingRequiredDefaultAllowed valuesGuide
Project nameYesEmptyAny textConfigure a run
Who can see itYesTeamTeam, PrivateConfigure a run

Step 1: Agent Config​

SettingRequiredDefaultAllowed valuesNotes
WorkloadYesCoding AgentCoding Agent, Enterprise AgentChanging it replaces the task set and task codes.
Task setYesTerminal-Bench 2.0 (Coding Agent), Enterprise: Contract Review (Enterprise Agent)Coding Agent: Terminal-Bench 2.0, Terminal-Bench 2.1, or any Harbor dataset as name@revision. Enterprise Agent: Contract Review, Document Redaction, Email Threat Analysis.A typed revision must be exact. latest, head, main, and master are refused.
Task codesYesThe task set's default taskCoding Agent: one or more task directory names from the dataset. Enterprise Agent: the task set's one case, fixed.The first task code is the validation task.

Step 2: Model Config​

SettingRequiredDefaultAllowed valuesNotes
Serving frameworkYesvLLMvLLM. SGLang and TensorRT-LLM are listed but not validated.A prefill and decode split needs vLLM.
ModelYesNoneA verified model, or any Hugging Face model IDAn unverified model needs tool_call_parser.
VersionYesNoneA framework version from the listRecorded with every result.
Build variantNoAutomaticAutomatic, Default (NVIDIA CUDA 13.0), NVIDIA CUDA 12.9, AMD ROCm (MI300X / MI355X)A variant for another GPU vendor is refused at run start.
Engine args: enable_auto_tool_choiceYestruetrueFilled in for every model.
Engine args: tool_call_parserYesFrom the model profileA vLLM tool call parser name, such as hermes, qwen3_coder, llama3_jsonEmpty for an unverified model until you enter it. The run cannot be saved without it.
Engine args: reasoning_parserNoFrom the model profileA vLLM reasoning parser name, such as qwen3Only for reasoning models.
Engine args: customNoNoneAny vLLM launch flag, without leading dashesExample: max_model_len = 32768.

Step 3: Hardware​

SettingRequiredDefaultAllowed valuesNotes
Cluster groupYesNoneA cluster group of your organizationOne machine or more. Same GPU vendor on every machine.
GPUs per nodeYesThe smallest GPU count in the group1 to the smallest GPU count in the groupApplies to every machine.
Tensor parallel sizeYes11 to GPUs per nodeMust equal GPUs per node for a prefill and decode split.
Pipeline parallel sizeNo11 to the number of machinesThe number of machines must divide evenly by it. Must be 1 for a split.
Sandbox CPUsNoUnpinnedUnpinned, Control, PinnedPinned needs CPUs for at least one machine.
Pinned CPUs per machineWith PinnedEmptyKernel CPU list, for example 8-15 or 8-15,64-71Cannot include the CPUs kept for metrics collection.
Split prefill and decode across machinesNoOffOn, OffSee the split requirements below.
Prefill machinesWith a split11 to the number of machines minus 1The first machines of the group prefill.
Prompt cache on decode machinesWith a splitOnOn, OffOff sends every prompt's whole KV cache from prefill.
KV cache precisionNo, split onlyAutomaticAutomatic, FP8, Model's own precisionAutomatic tries FP8 and falls back to the model's precision.
Maximum context lengthNo, split onlyAutomatic1,024 to 10,000,000 tokensAutomatic uses the model's limit.
KV transfer pathNo, split onlyAutomaticAutomatic, GPU to GPU over RDMA, Through host memoryApplies when KV moves over RDMA.
Prefill KV cache per GPUNo, split onlyAutomatic1 to 100,000 GiBRefused when the GPU or machine cannot hold it.
Prefill GPU memory shareNo, split onlyAutomatic0.10 to 0.99
Prefill requests at onceNo, split onlyAutomatic1 to 100,000
Prefill tokens per stepNo, split onlyAutomatic16 to 10,000,000
Decode KV cache per GPUNo, split onlyAutomatic1 to 100,000 GiBBounds how many requests decode holds at once.
Decode GPU memory shareNo, split onlyAutomatic0.10 to 0.99
Decode requests at onceNo, split onlyAutomatic1 to 100,000
Decode tokens per stepNo, split onlyAutomatic16 to 10,000,000
Sandbox memoryNo4096 MiB128 to 4,194,304 MiBThe memory limit of each sandbox.
Sandbox CPUNo1000 millicores100 to 1,024,000 millicores1000 millicores is one core.
Sandbox disk reservationNoAutomatic16 to 10,485,760 MiBAutomatic reserves 256 MiB per sandbox up to 768 sandboxes.

Prefill and decode split requirements​

RequirementValue
Machines in the cluster group2 or more
GPU vendorOne vendor on every machine, every machine has reported its GPUs
Pipeline parallel size1
Tensor parallel sizeEqual to GPUs per node
Serving frameworkvLLM
PlanIncludes Distributed KV runs
KV transportRDMA when every machine has an active RoCE v2 port, otherwise TCP. Not selectable.

Step 4: Load​

SettingRequiredDefaultAllowed valuesNotes
Concurrency levelsYes1, 8, 32Positive whole numbers, each used onceSorted into increasing order. Every level runs.
Trials per taskYes31 or moreA level runs at least as many trials as it has agents.

Step 5: SLA and validation​

SettingRequiredDefaultAllowed valuesNotes
Validation taskRead-onlyThe first task codeRuns once with one agent before the levels.
Agent time multiplierNo11 to 576Multiplies the agent time limit only.
Minimum pass rateNo85% Coding Agent, 75% Enterprise AgentGreater than 0, up to 100%Gate for a valid operating point.
Minimum GPU busy ratioNo60% Coding Agent, 40% Enterprise Agent0 to 100%Gate for a valid operating point. Decode GPUs only on a split.
Agent deadlineNoOffSeconds, greater than 0Reporting only.
Output token capNoNo capTokens, greater than 0Changes agent behavior.
Reasoning effortNoModel defaultModel default, Minimal, Low, Medium, HighFor models that support it.
TTFT ceilingNoOffSeconds, greater than 0Serving goodput only.
Inter-token latency ceilingNoOffSeconds, greater than 0Serving goodput only.

Fixed values​

These values are not settings. They apply to every run.

ValueAmount
Safety allowance added to each trial phase35 minutes
Agent setup limit6 minutes
Level setup allowance30 minutes
Shortest level time limit2 hours
Longest level time limit168 hours (7 days)
CPUs kept for metrics collection on each machine1 physical core, 2 on a machine with 64 or more CPUs
Clock synchronization requiredchrony synchronized to within 0.5 seconds
Model server ports on each machine, prefill and decode split over RDMA8000 (prefill) and 8200 (decode)
Enterprise model call with no outputFails after 1,400 seconds

Verified models​

ModelContext lengthTool call parserReasoning parserThinking mode
Qwen3-4B40,960hermesqwen3Model default
Qwen3.5 9B262,144qwen3_coderqwen3Off
Qwen3-14B40,960hermesqwen3Model default
Qwen 3.8 27B262,144qwen3_coderqwen3Off
Qwen 3.8 27B FP8262,144qwen3_coderqwen3Off
Qwen3-32B40,960hermesqwen3Model default

Task time limits​

Task setTask codeSetupAgentGrading
Terminal-Bench 2.0caffe-cifar-1010 min20 min20 min
Terminal-Bench 2.0crack-7z-hash10 min15 min15 min
Terminal-Bench 2.0filter-js-from-html10 min30 min15 min
Terminal-Bench 2.0fix-git10 min15 min15 min
Terminal-Bench 2.0git-leak-recovery10 min15 min15 min
Terminal-Bench 2.0git-multibranch10 min15 min15 min
Terminal-Bench 2.0log-summary-date-ranges10 min15 min15 min
Terminal-Bench 2.0modernize-scientific-stack10 min10 min10 min
Terminal-Bench 2.0portfolio-optimization10 min60 min60 min
Terminal-Bench 2.0pypi-server10 min15 min15 min
Terminal-Bench 2.0query-optimize10 min15 min15 min
Terminal-Bench 2.1caffe-cifar-1010 min60 min20 min
Terminal-Bench 2.1crack-7z-hash10 min30 min15 min
Terminal-Bench 2.1filter-js-from-html10 min30 min30 min
Terminal-Bench 2.1fix-git10 min15 min15 min
Terminal-Bench 2.1git-leak-recovery10 min15 min15 min
Terminal-Bench 2.1git-multibranch10 min15 min15 min
Terminal-Bench 2.1log-summary-date-ranges10 min15 min15 min
Terminal-Bench 2.1modernize-scientific-stack10 min10 min10 min
Terminal-Bench 2.1portfolio-optimization10 min60 min60 min
Terminal-Bench 2.1pypi-server10 min15 min15 min
Terminal-Bench 2.1query-optimize10 min15 min30 min
Enterprise: Contract Reviewcontract-review-128k-case-002110 min100 min20 min
Enterprise: Document Redactioncase-002910 min90 min20 min
Enterprise: Email Threat Analysisemail-threat-analysis-case-005110 min100 min3 min

Worked example of a level limit​

For fix-git at an agent time multiplier of 2, one trial's worst case is:

PartTime
Setup limit10 min
Agent setup limit6 min
Agent limit × multiplier15 min × 2 = 30 min
Grading limit15 min
Safety allowance, 35 min for each of the 4 phases140 min
Slowest trial201 min (3 h 21 min)

With 3 tasks of this size and 3 trials per task, a level of 8 agents runs 9 trials. Its limit is the 30-minute setup allowance, plus the 9 trials' total time spread over 8 agents, plus one slowest trial: 7 h 37 min. A level with as many agents as trials runs them all at once, so its limit is the setup allowance plus one slowest trial, raised to the 2-hour minimum if needed.

What's next​