Each configuration on this page comes from a Metrum validation run between 25 September and 15 October 2026. Every level listed under Levels that passed every trial graded every one of its trials as passed. Use these settings as a known-good starting point on similar hardware, then change one setting at a time.
Passing every trial is not the same as qualifying. A level qualifies only when its GPU busy ratio also meets the minimum, 60% by default. Where the GPU busy ratio was measured, it is listed below. Several fully passing levels stayed below 60% busy, so with the default threshold they show Missed a gate. Those levels prove the configuration works, and they leave GPU capacity unused.
All runs use the Coding Agent with the fix-git task. Settings not listed were left at their defaults: unpinned sandbox CPUs, 4096 MiB sandbox memory, 1000 millicores sandbox CPU, automatic disk reservation, no output token cap, and model default reasoning effort.
Summary
| # | Hardware | Model | vLLM | Layout | Levels that passed every trial |
|---|
| 1 | 1 machine, 2 × NVIDIA RTX PRO 6000 Blackwell | Qwen 3.8 27B FP8 | 0.28.0 | Tensor parallel 1, 2 replicas | 8, 32 |
| 2 | 2 machines, 1 × NVIDIA RTX PRO 6000 Blackwell each | Qwen 3.8 27B FP8 | 0.28.0 | Pipeline parallel 2, 1 replica | 1, 8, 32, 64 |
| 3 | 2 machines, 1 × NVIDIA RTX PRO 6000 Blackwell each | Qwen 3.8 27B FP8 | 0.28.0 | Pipeline parallel 2, 1 replica | 32, 64, 128, 256 |
| 4 | 2 machines, 1 × NVIDIA H100 PCIe each | Qwen 3.8 27B FP8 | 0.28.0 | Pipeline parallel 2, 1 replica | 1, 8, 32, 64 |
| 5 | 2 machines, 1 × NVIDIA H100 PCIe each | Qwen 3.8 27B | 0.28.0 | Pipeline parallel 2, 1 replica | 8, 32, 64 |
| 6 | 1 machine, 2 × NVIDIA L40S | Qwen 3.8 27B | 0.30.0 | Tensor parallel 2, 1 replica | 32, 64 |
| 7 | 1 machine, 8 × AMD Instinct MI300X | Qwen 3.8 27B | 0.30.0 | Tensor parallel 8, 1 replica | 32, 64, 256 |
| 8 | 2 machines, 2 × NVIDIA L40S each | Qwen 3.8 27B | 0.30.0 | Prefill and decode split over TCP, tensor parallel 2 | 32, 64, 128, 256, 512, 600 |
| 9 | 2 machines, 8 × AMD Instinct MI300X each | A custom Hugging Face model (FP8) | 0.30.0 | Prefill and decode split over RDMA, tensor parallel 8 | 4, 8 |
Configuration details
1. One machine, two RTX PRO 6000 GPUs, one replica per GPU
| Step | Setting | Value |
|---|
| Agent Config | Task set, task codes | Terminal-Bench 2.0, fix-git |
| Model Config | Model, version | Qwen 3.8 27B FP8, vLLM 0.28.0 |
| Hardware | Machines | 1, with 2 × NVIDIA RTX PRO 6000 Blackwell Server Edition and AMD EPYC 9355 CPUs |
| GPUs per node, tensor parallel, pipeline parallel | 2, 1, 1 |
| Load | Levels | 8, 32 |
| SLA | Agent time multiplier | 2 |
| Level | Passed | Level time | Median trial |
|---|
| Validation | 1 of 1 | 1 min | 0.6 min |
| 8 agents | 8 of 8 | 1 min | 0.8 min |
| 32 agents | 32 of 32 | 3 min | 1.4 min |
2 and 3. Two machines, one RTX PRO 6000 GPU each, pipeline parallel
| Step | Setting | Value |
|---|
| Agent Config | Task set, task codes | Terminal-Bench 2.0, fix-git |
| Model Config | Model, version | Qwen 3.8 27B FP8, vLLM 0.28.0 |
| Hardware | Machines | 2, each with 1 × NVIDIA RTX PRO 6000 Blackwell Server Edition and AMD EPYC 9355 CPUs |
| GPUs per node, tensor parallel, pipeline parallel | 1, 1, 2 |
| Load | Levels | Run 2: 1, 8, 32, 64. Run 3: 32, 64, 128, 256 |
| Trials per task | 3 |
| SLA | Agent time multiplier | 2 |
| Level | Passed | Level time | Median trial |
|---|
| 1 agent | 3 of 3 | 2 min | 0.6 min |
| 8 agents | 8 of 8 | 1 min | 0.9 min |
| 32 agents | 32 of 32 | 3 min | 1.6 min |
| 64 agents | 64 of 64 | 4 min | 2.5 min |
| 128 agents | 128 of 128 | 8 min | 4.5 min |
| 256 agents | 256 of 256 | 16 min | 5.4 min |
4 and 5. Two machines, one H100 PCIe GPU each, pipeline parallel
| Step | Setting | Value |
|---|
| Agent Config | Task set, task codes | Terminal-Bench 2.0, fix-git |
| Model Config | Model, version | Run 4: Qwen 3.8 27B FP8. Run 5: Qwen 3.8 27B. Both vLLM 0.28.0 |
| Hardware | Machines | 2, each with 1 × NVIDIA H100 PCIe |
| GPUs per node, tensor parallel, pipeline parallel | 1, 1, 2 |
| Load | Levels | Run 4: 1, 8, 32, 64. Run 5: 8, 32, 64 |
| SLA | Agent time multiplier | Run 4: 2 |
Run 4 passed 3 of 3, 8 of 8, 32 of 32, and 64 of 64. Run 5 ran 64 trials at each level and passed 64 of 64 at 8, 32, and 64 agents.
6. One machine, two L40S GPUs, one replica
| Step | Setting | Value |
|---|
| Agent Config | Task set, task codes | Terminal-Bench 2.1, fix-git |
| Model Config | Model, version | Qwen 3.8 27B, vLLM 0.30.0 |
| Hardware | Machines | 1, with 2 × NVIDIA L40S and AMD EPYC 7543 CPUs |
| GPUs per node, tensor parallel, pipeline parallel | 2, 2, 1 |
| Load | Levels that passed | 32, 64 |
| SLA | Agent time multiplier | 1 |
32 agents passed 32 of 32 in 3 minutes, at a GPU busy ratio of 45%. 64 agents passed 64 of 64. In this run, the 128-agent level did not finish, so 64 agents is the tested ceiling for this layout.
7. One machine, eight MI300X GPUs, one replica
| Step | Setting | Value |
|---|
| Agent Config | Task set, task codes | Terminal-Bench 2.1, fix-git |
| Model Config | Model, version | Qwen 3.8 27B, vLLM 0.30.0, ROCm build (automatic) |
| Hardware | Machines | 1, with 8 × AMD Instinct MI300X and Intel Xeon Platinum 8568Y+ CPUs |
| GPUs per node, tensor parallel, pipeline parallel | 8, 8, 1 |
| Load | Levels that passed | 32, 64, 256 |
| SLA | Agent time multiplier | 2 |
32 agents passed 32 of 32 in 3 minutes. 64 agents passed 64 of 64 in 5 minutes. 256 agents passed 256 of 256 in 14 minutes. The GPU busy ratio was 70%, 75%, and 73% at these levels, so all three qualify with the default thresholds. The 128-agent level of the same run did not finish, so it is not listed.
8. Prefill and decode split over TCP on two L40S machines
| Step | Setting | Value |
|---|
| Agent Config | Task set, task codes | Terminal-Bench 2.1, fix-git |
| Model Config | Model, version | Qwen 3.8 27B, vLLM 0.30.0 |
| Hardware | Machines | 2, each with 2 × NVIDIA L40S and AMD EPYC 7543 CPUs |
| GPUs per node, tensor parallel, pipeline parallel | 2, 2, 1 |
| Split prefill and decode | On. 1 prefill machine, 1 decode machine. KV over TCP |
| Load | Levels that passed | 32, 64, 128, 256, 512, 600 |
| SLA | Agent time multiplier | 144 |
| Level | Passed | Level time | Median trial |
|---|
| 32 agents | 32 of 32 | 7 min | 5.4 min |
| 64 agents | 64 of 64 | 14 min | 10.2 min |
| 128 agents | 128 of 128 | 27 min | 20.3 min |
| 256 agents | 256 of 256 | 55 min | 41 min |
| 512 agents | 512 of 512 | 115 min | 85 min |
| 600 agents | 600 of 600 | 144 min | 105 min |
All four GPUs together were 36% to 45% busy. This run was measured before the report computed the decode GPUs' busy ratio on its own, which is the figure the gate uses on a split. Whether these levels qualify under the current rule is therefore not known. The very high multiplier let slow trials finish instead of timing out, so every trial was graded. Trial time grows with the level because every agent shares two decode GPUs over a TCP link. The 1,024-agent level of this run did not finish.
9. Prefill and decode split over RDMA on two MI300X machines
| Step | Setting | Value |
|---|
| Agent Config | Task set, task codes | Terminal-Bench 2.1, fix-git |
| Model Config | Model, version | A custom FP8 Hugging Face model, vLLM 0.30.0, with its own tool call parser entered under Engine args |
| Hardware | Machines | 2, each with 8 × AMD Instinct MI300X, connected by RoCE v2 |
| GPUs per node, tensor parallel, pipeline parallel | 8, 8, 1 |
| Split prefill and decode | On. 1 prefill machine, 1 decode machine. KV over RDMA |
| Load | Levels that passed | 4, 8 |
| SLA | Agent time multiplier | 1 |
4 agents passed 4 of 4, and 8 agents passed 8 of 8, each in about 1 minute. The decode GPUs were 68% and 72% busy, so both levels qualify with the default thresholds.
How to use these configurations
- Pick the row closest to your hardware.
- Copy its settings into a new AgentBench workload. Keep the task, the model, the version, and the layout.
- Run it as a smoke test. Every level should pass every trial.
- Change one setting at a time: add levels, add tasks, switch the model, or turn on a split.
These results show that the configuration runs end to end and grades correctly at the listed levels. They are not capacity claims. A level that passed every trial is not necessarily the highest level the hardware supports, and a level that is not listed was either not run or did not finish in that run.
What's next