Use these docs with an AI assistant
You can give this page to an AI assistant, such as Claude, and ask it to plan an Agentic Bench run for your hardware, check a configuration before you launch it, or explain a result. The page holds the full set of settings and rules in a structured form the assistant can apply without guessing.
Every page of these docs is also published as plain Markdown: replace .html at the end of a page's address with .md. The file /llms.txt at the root of the docs site lists every page with a one-line summary. Give an assistant llms.txt when it should find the right page by itself.
Example requests
Paste the URL of this page, then ask:
- "I have two machines with 8 H100 GPUs each. Plan an Agentic Bench Coding Agent run that finds how many agents they can serve with Qwen3-32B. Give me the value for every field, step by step."
- "Check this configuration against the Agentic Bench rules and tell me what will be refused: (paste your settings)."
- "My 128-agent level missed the GPU busy gate and the 64-agent level qualified. What does that mean, and what should I test next?"
- "Plan a prefill and decode split on my two 2-GPU L40S machines and a matching baseline run to compare it with."
Instructions for the assistant
Follow this procedure when you plan or check a run.
- Collect the facts: the number of machines, the GPU model and count on each machine, the GPU vendor, the free memory and disk on each machine, whether the machines have RoCE v2 network ports, the model the user wants, and the workload kind.
- Choose the workload kind and task set. For a first run, use the Coding Agent, Terminal-Bench 2.0, and task
fix-git. - Choose the model. Prefer a verified model. For any other Hugging Face model, require a
tool_call_parservalue. - Choose the parallelism. Apply the rules in the specification below. Compute the replica count and say it.
- Choose the levels. Start with a bracket such as 1, 8, 32, 64. For Enterprise Contract Review, start with 1, 2, 4, 8.
- Check every rule in the specification. Report each rule a configuration breaks, with the fix.
- Present the plan as the five steps of the web app, with the exact label and value of every field.
- State what the run will measure, what qualifies a level, and how long the setup and levels are likely to take.
Do not invent settings, models, or task codes that are not in this specification or in the user's own dataset. When a value depends on hardware facts you do not have, ask for them.
Run specification
The keys match the settings object of the agentic_bench workload in the project API. Thresholds are ratios between 0 and 1 in the API and percentages in the web app.
# Agentic Bench run specification. Copyright (c) 2026 Metrum AI. All rights reserved.
project:
project_name: {type: text, required: true, ui: "Project name"}
visibility: {type: enum, values: [team, private], default: team, ui: "Who can see it"}
step_1_agent_config:
workload_kind:
ui: "Workload"
type: enum
values: [coding, enterprise]
default: coding
dataset_ref:
ui: "Task set"
required: true
coding_values:
- {ref: "terminal-bench@2.0", name: "Terminal-Bench 2.0"}
- {ref: "terminal-bench/terminal-bench-2-1@6", name: "Terminal-Bench 2.1"}
- {ref: "<any Harbor dataset>@<exact revision>", name: "Custom"}
enterprise_values:
- {ref: "eadp-contract-review-128k@1", name: "Enterprise: Contract Review", case: "contract-review-128k-case-0021"}
- {ref: "eadp-document-redaction-128k@1", name: "Enterprise: Document Redaction", case: "case-0029"}
- {ref: "eadp-email-threat-analysis-128k@1", name: "Enterprise: Email Threat Analysis", case: "email-threat-analysis-case-0051", shared_services: [vector_index, embedder, reranker]}
rules:
- "A typed revision must be exact. latest, head, main and master are refused."
- "Results from different task sets are never compared."
task_codes:
ui: "Task codes"
required: true
type: list of text
coding_quick_picks: [fix-git, git-multibranch, git-leak-recovery, log-summary-date-ranges,
modernize-scientific-stack, pypi-server, query-optimize, crack-7z-hash,
filter-js-from-html, caffe-cifar-10, portfolio-optimization]
rules:
- "The first task code is the validation task."
- "Enterprise task sets have exactly one fixed case."
step_2_model_config:
framework_code: {ui: "Serving framework", default: vllm, validated: [vllm]}
model:
ui: "Model"
required: true
verified:
- {name: "Qwen3-4B", tool_call_parser: hermes, reasoning_parser: qwen3, context: 40960}
- {name: "Qwen3.5 9B", tool_call_parser: qwen3_coder, reasoning_parser: qwen3, context: 262144}
- {name: "Qwen3-14B", tool_call_parser: hermes, reasoning_parser: qwen3, context: 40960}
- {name: "Qwen 3.8 27B", tool_call_parser: qwen3_coder, reasoning_parser: qwen3, context: 262144}
- {name: "Qwen 3.8 27B FP8", tool_call_parser: qwen3_coder, reasoning_parser: qwen3, context: 262144}
- {name: "Qwen3-32B", tool_call_parser: hermes, reasoning_parser: qwen3, context: 40960}
unverified: "Any Hugging Face model ID. tool_call_parser is required."
version: {ui: "Version", required: true, tested: ["0.27.1", "0.28.0", "0.30.0"]}
build_variant: {ui: "Build variant", default: automatic}
engine_args:
enable_auto_tool_choice: {required: true, value: "true"}
tool_call_parser: {required: true, source: "model profile, or user for an unverified model"}
reasoning_parser: {required: false}
custom: "Any vLLM flag without leading dashes, for example max_model_len: 32768"
step_3_hardware:
cluster_group_code: {ui: "Cluster group", required: true}
gpus_per_node: {ui: "GPUs per node", min: 1, max: "smallest GPU count in the group", default: max}
tensor_parallel_size: {ui: "Tensor parallel size", min: 1, max: gpus_per_node, default: 1}
pipeline_parallel_size: {ui: "Pipeline parallel size", min: 1, max: machines, default: 1}
replicas: "(machines / pipeline_parallel_size) * floor(gpus_per_node / tensor_parallel_size)"
sandbox_cpu_mode: {ui: "Sandbox CPUs", values: [unpinned, control, pinned], default: unpinned}
cpu_lists: {ui: "Pinned CPUs per machine", format: "kernel list such as 8-15,64-71", only_with: pinned}
prefill_decode:
enabled: {ui: "Split prefill and decode across machines", default: false}
prefill_node_count: {ui: "Prefill machines", min: 1, max: "machines - 1", default: 1}
decode_prefix_caching: {ui: "Prompt cache on decode machines", default: true}
transport: "rdma when every machine has an active RoCE v2 port, otherwise tcp. Not selectable."
split_engine_settings: # only with a split; empty means automatic
kv_cache_dtype: {ui: "KV cache precision", values: [automatic, fp8, model]}
max_model_len: {ui: "Maximum context length", min: 1024, max: 10000000}
kv_transfer_path: {ui: "KV transfer path", values: [automatic, gpu_direct, host_memory]}
prefill_kv_cache_memory_gib: {min: 1, max: 100000}
prefill_gpu_memory_utilization: {min: 0.10, max: 0.99}
prefill_max_num_seqs: {min: 1, max: 100000}
prefill_max_num_batched_tokens: {min: 16, max: 10000000}
decode_kv_cache_memory_gib: {min: 1, max: 100000}
decode_gpu_memory_utilization: {min: 0.10, max: 0.99}
decode_max_num_seqs: {min: 1, max: 100000}
decode_max_num_batched_tokens: {min: 16, max: 10000000}
sandbox:
sandbox_memory_mib: {ui: "Sandbox memory", default: 4096, min: 128, max: 4194304}
sandbox_cpu_millicores: {ui: "Sandbox CPU", default: 1000, min: 100, max: 1024000}
sandbox_storage_mib: {ui: "Sandbox disk reservation", default: automatic, min: 16, max: 10485760}
step_4_load:
concurrency_levels: {ui: "Concurrency levels", required: true, default: [1, 8, 32], rule: "positive, unique, increasing"}
trials_per_task: {ui: "Trials per task", default: 3, min: 1}
trials_at_level: "tasks * max(trials_per_task, ceil(level / tasks))"
step_5_sla:
validation_task: "first task code, read-only"
agent_timeout_multiplier: {ui: "Agent time multiplier", default: 1, min: 1, max: 576}
min_pass_rate_threshold: {ui: "Minimum pass rate", default: {coding: 0.85, enterprise: 0.75}, range: "(0, 1]"}
min_gpu_busy_ratio_threshold: {ui: "Minimum GPU busy ratio", default: {coding: 0.60, enterprise: 0.40}, range: "[0, 1]"}
agent_deadline_seconds: {ui: "Agent deadline", default: off, min_exclusive: 0}
max_output_tokens: {ui: "Output token cap", default: none, min_exclusive: 0}
reasoning_effort: {ui: "Reasoning effort", values: [model_default, minimal, low, medium, high]}
ttft_ceiling_seconds: {ui: "TTFT ceiling", default: off}
inter_token_latency_ceiling_seconds: {ui: "Inter-token latency ceiling", default: off}
fixed_values:
phase_safety_allowance_minutes: 35
agent_setup_limit_minutes: 6
level_setup_allowance_minutes: 30
level_limit_hours: {min: 2, max: 168}
slowest_trial: "setup + 6 min + agent_limit * multiplier + grading + 4 * 35 min"
level_limit: "clamp(30 min + (trials > level ? total_trial_time / level : 0) + slowest_trial, 2 h, 168 h)"
enterprise_model_call_no_output_timeout_seconds: 1400
host_requirements_per_machine:
agent: "Metrum agent installed and running"
ip_address: "required when the group has 2 or more machines"
gpu_vendor: "the same on every machine"
clock: "chrony synchronized to within 0.5 s (installed by the agent when missing)"
other_kubernetes: "none: a machine running another kubelet is refused"
nvidia_container_toolkit: "required on NVIDIA machines"
free_disk: "2.2 * model_weights + 1 GiB * highest_level + up to 200 GB package cache; checked before each level"
free_memory: "8 GiB + 0.5 * model_weights + 256 MiB * highest_level (planning figure, not checked)"
rdma_split_ports_between_machines: [8000, 8200, 9000, 9200, "15600 and up"]
Rules to check
An assistant should check each rule and report every one a configuration breaks.
| # | Rule | Fix when broken |
|---|---|---|
| 1 | tensor_parallel_size is at most gpus_per_node. | Lower tensor parallel size. |
| 2 | gpus_per_node is at most the smallest GPU count in the group. | Lower GPUs per node. |
| 3 | pipeline_parallel_size is at most the number of machines, and the number of machines divides evenly by it. | Change pipeline parallel size or the group. |
| 4 | The model fits: weights ÷ tensor parallel size ÷ pipeline parallel size is well below one GPU's memory, leaving room for the KV cache. | Raise tensor parallel size, use an FP8 model, or add pipeline stages. |
| 5 | A split has 2 or more machines, one GPU vendor, pipeline parallel size 1, and tensor parallel size equal to GPUs per node. | Fix the layout, or turn off the split. |
| 6 | A split runs on vLLM, and the plan includes Distributed KV runs. | Use vLLM, or upgrade the plan. |
| 7 | An unverified model has tool_call_parser. | Enter the parser from the model card. |
| 8 | Concurrency levels are positive, unique, and increasing. | Fix the list. |
| 9 | Pinned mode names CPUs for at least one machine, and no CPU kept for metrics. | Enter CPUs, or switch mode. |
| 10 | Every machine has free disk and memory, a clock that can synchronize, no other Kubernetes cluster, and on NVIDIA the NVIDIA Container Toolkit. | Free resources, fix time sync, remove the other cluster, or install the toolkit. |
| 11 | An Enterprise Contract Review model has a context of at least 131,072 tokens. Of the verified models, Qwen3.5 9B, Qwen 3.8 27B and Qwen 3.8 27B FP8 qualify. | Choose a model with a longer context. |
| 12 | The agent time multiplier is between 1 and 576. | Set a value in range. |
| 13 | Compared runs share task set, tasks, model, version, build, parallelism, transport, CPU mode, multiplier, output cap, and reasoning effort. | Rerun with matching settings. |
Example specifications
Coding Agent on one machine
project_name: "Qwen 3.8 27B FP8 on 2x RTX PRO 6000, coding"
step_1: {workload_kind: coding, dataset_ref: "terminal-bench@2.0", task_codes: [fix-git]}
step_2: {framework: vllm, model: "Qwen 3.8 27B FP8", version: "0.28.0", build_variant: automatic}
step_3: {cluster_group: "one machine with 2 x 96 GB RTX PRO 6000", gpus_per_node: 2, tensor_parallel_size: 1, pipeline_parallel_size: 1, sandbox_cpu_mode: unpinned}
step_4: {concurrency_levels: [1, 8, 32], trials_per_task: 3}
step_5: {agent_timeout_multiplier: 2}
# 2 replicas. Tested on this hardware: every trial passed at 8 and 32 agents.
# Low levels may still miss the 60% GPU busy gate. That is expected for a smoke test.
Prefill and decode split with its baseline
baseline:
step_3: {cluster_group: "two machines with 2 L40S each", gpus_per_node: 2, tensor_parallel_size: 2, pipeline_parallel_size: 1, split: false}
split:
step_3: {cluster_group: "same group", gpus_per_node: 2, tensor_parallel_size: 2, pipeline_parallel_size: 1,
split: true, prefill_node_count: 1, decode_prefix_caching: true}
shared:
step_1: {workload_kind: coding, dataset_ref: "terminal-bench/terminal-bench-2-1@6", task_codes: [fix-git]}
step_2: {model: "Qwen 3.8 27B", version: "0.30.0"}
step_4: {concurrency_levels: [32, 64, 128, 256], trials_per_task: 3}
step_5: {agent_timeout_multiplier: 2}
# Compare level by level. Without RoCE v2 ports the split runs over TCP and needs a higher multiplier.
Enterprise Contract Review
step_1: {workload_kind: enterprise, dataset_ref: "eadp-contract-review-128k@1"}
step_2: {model: "Qwen 3.8 27B FP8", version: "0.30.0"}
step_3: {cluster_group: "one machine with 2 GPUs", gpus_per_node: 2, tensor_parallel_size: 2}
step_4: {concurrency_levels: [1, 2, 4, 8], trials_per_task: 1}
step_5: {} # defaults: 75% pass rate, 40% GPU busy, plus the enterprise gates
# About 122,000 prompt tokens per agent. KV cache memory limits the agent count.
Related pages
- Settings reference: the same settings as tables.
- Agentic Bench methodology: how results are measured.
- Using Metrum AI Bench with an agent: connect an AI assistant to Metrum AI Bench through its MCP tools.