Skip to main content

Tested configurations

Each configuration on this page comes from a Metrum validation run between 25 September and 15 October 2026. Every level listed under Levels that passed every trial graded every one of its trials as passed. Use these settings as a known-good starting point on similar hardware, then change one setting at a time.

Passing every trial is not the same as qualifying. A level qualifies only when its GPU busy ratio also meets the minimum, 60% by default. Where the GPU busy ratio was measured, it is listed below. Several fully passing levels stayed below 60% busy, so with the default threshold they show Missed a gate. Those levels prove the configuration works, and they leave GPU capacity unused.

All runs use the Coding Agent with the fix-git task. Settings not listed were left at their defaults: unpinned sandbox CPUs, 4096 MiB sandbox memory, 1000 millicores sandbox CPU, automatic disk reservation, no output token cap, and model default reasoning effort.

Summary​

#HardwareModelvLLMLayoutLevels that passed every trial
11 machine, 2 × NVIDIA RTX PRO 6000 BlackwellQwen 3.8 27B FP80.28.0Tensor parallel 1, 2 replicas8, 32
22 machines, 1 × NVIDIA RTX PRO 6000 Blackwell eachQwen 3.8 27B FP80.28.0Pipeline parallel 2, 1 replica1, 8, 32, 64
32 machines, 1 × NVIDIA RTX PRO 6000 Blackwell eachQwen 3.8 27B FP80.28.0Pipeline parallel 2, 1 replica32, 64, 128, 256
42 machines, 1 × NVIDIA H100 PCIe eachQwen 3.8 27B FP80.28.0Pipeline parallel 2, 1 replica1, 8, 32, 64
52 machines, 1 × NVIDIA H100 PCIe eachQwen 3.8 27B0.28.0Pipeline parallel 2, 1 replica8, 32, 64
61 machine, 2 × NVIDIA L40SQwen 3.8 27B0.30.0Tensor parallel 2, 1 replica32, 64
71 machine, 8 × AMD Instinct MI300XQwen 3.8 27B0.30.0Tensor parallel 8, 1 replica32, 64, 256
82 machines, 2 × NVIDIA L40S eachQwen 3.8 27B0.30.0Prefill and decode split over TCP, tensor parallel 232, 64, 128, 256, 512, 600
92 machines, 8 × AMD Instinct MI300X eachA custom Hugging Face model (FP8)0.30.0Prefill and decode split over RDMA, tensor parallel 84, 8

Configuration details​

1. One machine, two RTX PRO 6000 GPUs, one replica per GPU​

StepSettingValue
Agent ConfigTask set, task codesTerminal-Bench 2.0, fix-git
Model ConfigModel, versionQwen 3.8 27B FP8, vLLM 0.28.0
HardwareMachines1, with 2 × NVIDIA RTX PRO 6000 Blackwell Server Edition and AMD EPYC 9355 CPUs
GPUs per node, tensor parallel, pipeline parallel2, 1, 1
LoadLevels8, 32
SLAAgent time multiplier2
LevelPassedLevel timeMedian trial
Validation1 of 11 min0.6 min
8 agents8 of 81 min0.8 min
32 agents32 of 323 min1.4 min

2 and 3. Two machines, one RTX PRO 6000 GPU each, pipeline parallel​

StepSettingValue
Agent ConfigTask set, task codesTerminal-Bench 2.0, fix-git
Model ConfigModel, versionQwen 3.8 27B FP8, vLLM 0.28.0
HardwareMachines2, each with 1 × NVIDIA RTX PRO 6000 Blackwell Server Edition and AMD EPYC 9355 CPUs
GPUs per node, tensor parallel, pipeline parallel1, 1, 2
LoadLevelsRun 2: 1, 8, 32, 64. Run 3: 32, 64, 128, 256
Trials per task3
SLAAgent time multiplier2
LevelPassedLevel timeMedian trial
1 agent3 of 32 min0.6 min
8 agents8 of 81 min0.9 min
32 agents32 of 323 min1.6 min
64 agents64 of 644 min2.5 min
128 agents128 of 1288 min4.5 min
256 agents256 of 25616 min5.4 min

4 and 5. Two machines, one H100 PCIe GPU each, pipeline parallel​

StepSettingValue
Agent ConfigTask set, task codesTerminal-Bench 2.0, fix-git
Model ConfigModel, versionRun 4: Qwen 3.8 27B FP8. Run 5: Qwen 3.8 27B. Both vLLM 0.28.0
HardwareMachines2, each with 1 × NVIDIA H100 PCIe
GPUs per node, tensor parallel, pipeline parallel1, 1, 2
LoadLevelsRun 4: 1, 8, 32, 64. Run 5: 8, 32, 64
SLAAgent time multiplierRun 4: 2

Run 4 passed 3 of 3, 8 of 8, 32 of 32, and 64 of 64. Run 5 ran 64 trials at each level and passed 64 of 64 at 8, 32, and 64 agents.

6. One machine, two L40S GPUs, one replica​

StepSettingValue
Agent ConfigTask set, task codesTerminal-Bench 2.1, fix-git
Model ConfigModel, versionQwen 3.8 27B, vLLM 0.30.0
HardwareMachines1, with 2 × NVIDIA L40S and AMD EPYC 7543 CPUs
GPUs per node, tensor parallel, pipeline parallel2, 2, 1
LoadLevels that passed32, 64
SLAAgent time multiplier1

32 agents passed 32 of 32 in 3 minutes, at a GPU busy ratio of 45%. 64 agents passed 64 of 64. In this run, the 128-agent level did not finish, so 64 agents is the tested ceiling for this layout.

7. One machine, eight MI300X GPUs, one replica​

StepSettingValue
Agent ConfigTask set, task codesTerminal-Bench 2.1, fix-git
Model ConfigModel, versionQwen 3.8 27B, vLLM 0.30.0, ROCm build (automatic)
HardwareMachines1, with 8 × AMD Instinct MI300X and Intel Xeon Platinum 8568Y+ CPUs
GPUs per node, tensor parallel, pipeline parallel8, 8, 1
LoadLevels that passed32, 64, 256
SLAAgent time multiplier2

32 agents passed 32 of 32 in 3 minutes. 64 agents passed 64 of 64 in 5 minutes. 256 agents passed 256 of 256 in 14 minutes. The GPU busy ratio was 70%, 75%, and 73% at these levels, so all three qualify with the default thresholds. The 128-agent level of the same run did not finish, so it is not listed.

8. Prefill and decode split over TCP on two L40S machines​

StepSettingValue
Agent ConfigTask set, task codesTerminal-Bench 2.1, fix-git
Model ConfigModel, versionQwen 3.8 27B, vLLM 0.30.0
HardwareMachines2, each with 2 × NVIDIA L40S and AMD EPYC 7543 CPUs
GPUs per node, tensor parallel, pipeline parallel2, 2, 1
Split prefill and decodeOn. 1 prefill machine, 1 decode machine. KV over TCP
LoadLevels that passed32, 64, 128, 256, 512, 600
SLAAgent time multiplier144
LevelPassedLevel timeMedian trial
32 agents32 of 327 min5.4 min
64 agents64 of 6414 min10.2 min
128 agents128 of 12827 min20.3 min
256 agents256 of 25655 min41 min
512 agents512 of 512115 min85 min
600 agents600 of 600144 min105 min

All four GPUs together were 36% to 45% busy. This run was measured before the report computed the decode GPUs' busy ratio on its own, which is the figure the gate uses on a split. Whether these levels qualify under the current rule is therefore not known. The very high multiplier let slow trials finish instead of timing out, so every trial was graded. Trial time grows with the level because every agent shares two decode GPUs over a TCP link. The 1,024-agent level of this run did not finish.

9. Prefill and decode split over RDMA on two MI300X machines​

StepSettingValue
Agent ConfigTask set, task codesTerminal-Bench 2.1, fix-git
Model ConfigModel, versionA custom FP8 Hugging Face model, vLLM 0.30.0, with its own tool call parser entered under Engine args
HardwareMachines2, each with 8 × AMD Instinct MI300X, connected by RoCE v2
GPUs per node, tensor parallel, pipeline parallel8, 8, 1
Split prefill and decodeOn. 1 prefill machine, 1 decode machine. KV over RDMA
LoadLevels that passed4, 8
SLAAgent time multiplier1

4 agents passed 4 of 4, and 8 agents passed 8 of 8, each in about 1 minute. The decode GPUs were 68% and 72% busy, so both levels qualify with the default thresholds.

How to use these configurations​

  1. Pick the row closest to your hardware.
  2. Copy its settings into a new AgentBench workload. Keep the task, the model, the version, and the layout.
  3. Run it as a smoke test. Every level should pass every trial.
  4. Change one setting at a time: add levels, add tasks, switch the model, or turn on a split.
note

These results show that the configuration runs end to end and grades correctly at the listed levels. They are not capacity claims. A level that passed every trial is not necessarily the highest level the hardware supports, and a level that is not listed was either not run or did not finish in that run.

What's next​