Skip to main content

Hardware, parallelism, and sandboxes

Step 3 of an AgentBench workload sets where the run happens. It decides which machines run the model, how the model is spread across their GPUs, and what CPU, memory, and disk each agent sandbox gets.

The Hardware step before a cluster group is chosen

How your machines are used​

Every Agentic Bench run uses the machines of one cluster group:

  1. Metrum AI Bench forms one Kubernetes cluster over the group's machines. A single machine becomes a one-node cluster with no extra setup.
  2. It deploys the model server across the cluster's GPUs.
  3. It runs every agent sandbox on the same cluster, spread across the machines.

The model server and the sandboxes share the machines. The GPUs serve the model. The CPUs, memory, and disk serve both the model server and the sandboxes.

note

The web app says node where these pages say machine. Both mean one server in your cluster group.

Parallelism in plain words​

A model is a stack of layers. A large model does not fit on one GPU, so its work is split across several GPUs in one of two ways:

  • Tensor parallelism splits every layer across several GPUs of the same machine. The GPUs work on each request together and exchange data constantly, so they must sit in the same machine, connected by its fast internal links.
  • Pipeline parallelism splits the stack of layers into consecutive parts on different machines. The first machine runs the first layers and passes the result to the next machine over the network.

A replica is one complete, running copy of the model. More replicas serve more agents at once. Each replica uses tensor parallel size × pipeline parallel size GPUs.

Cluster group​

A cluster group is a named set of your machines that Agentic Bench forms into one Kubernetes cluster on demand. Required.

Cluster group tooltip

Each entry in the list shows the group's name, its number of machines, and its total GPUs.

Cluster group options

After you choose a group, Nodes in this group lists each machine with its GPUs and the CPU cores available to the run.

A cluster group selected

Create a cluster group​

  1. Select New group.
  2. In Name, enter a short name, for example h100-rack.
  3. In Machines, add each machine. The first machine you add is the cluster head.
  4. Select Create.

The Create Cluster Group dialog

Adding machines to a cluster group

Requirements for a cluster group​

RequirementWhy
Every machine is onboarded and its Metrum agent is running.The agent forms the cluster, starts the model server, and reports telemetry.
Every machine has an IP address on record, when the group has two or more machines.Machines join the cluster by IP address.
Every machine has the same GPU vendor.One model deployment uses one build: CUDA for NVIDIA, ROCm for AMD.
Every machine's clock can be synchronized by NTP.Timing data from different machines is compared directly. The agent waits until chrony reports the clock within 0.5 seconds, and installs chrony when it is missing. A machine whose clock does not synchronize stops the run.
Enough free disk on each machine: 2.2 times the model weights, plus 1 GiB per agent at your highest level, plus up to 200 GB for the package cache.The model downloads to each machine, each sandbox writes task files, and the sandboxes' Python packages are cached on the machine. Before each level the agent checks free disk against what the run's earlier levels used, and refuses a level that would fill a machine.
Enough free memory on each machine: 8 GiB, plus half the model weights, plus 256 MiB per agent at your highest level.The model server, the cluster, and the sandboxes all use host memory. This is a planning figure; the agent does not check it before the run.
The machines can reach each other, and reach Hugging Face, public container registries, and GitHub.The cluster forms over the network between the machines, and the run downloads the model and images. See Network access.

One cluster group runs one run at a time. A second run on the same group waits in a queue, and the monitoring page shows how many runs are ahead. A run also waits until the previous run's cluster is fully torn down.

GPUs per node​

How many GPUs the run uses on each machine. Required.

GPUs per node tooltip

  • The same number applies to every machine in the group.
  • The maximum is the smallest GPU count of any machine in the group. A machine with more GPUs leaves the rest unused.
  • Use all sets the maximum.

Tensor parallel size​

How many GPUs one copy of the model uses on one machine. Required. Allowed values: 1 to GPUs per node.

Tensor parallel size tooltip

Each machine runs GPUs per node ÷ tensor parallel size copies of the model, rounded down. GPUs left over by the rounding stay unused. For example, 3 GPUs per node at tensor parallel size 2 runs one copy and leaves one GPU idle.

Pipeline parallel size​

How many machines one copy of the model spans. Optional. Default: 1. Allowed values: 1 to the number of machines in the group.

Pipeline parallel size tooltip

Pipeline parallelism splits the model into stages, one stage per machine. Use it only when the model does not fit on one machine even with tensor parallel size set to every GPU on that machine. The number of machines in the group must divide evenly by the pipeline parallel size.

Model replicas​

A replica is one running copy of the model. The model server spreads agent requests across every replica.

Replicas = (machines ÷ pipeline parallel size) × (GPUs per node ÷ tensor parallel size, rounded down)

MachinesGPUs per nodeTensor parallelPipeline parallelReplicasLayout
12112Each GPU serves its own copy.
18214Four copies of two GPUs each.
18811One copy over all eight GPUs.
22114Two copies on each machine.
22212One copy on each machine.
21121One copy split across both machines.
28812One copy on each machine.
note

Below Pipeline parallel size, the launcher shows a sentence such as "This deploys 1 replica of 1 node each across the group's 1 node." That sentence counts groups of machines only. The total number of model copies also multiplies by GPUs per node ÷ tensor parallel size, as in the formula above. The report shows the actual replica count under Test conditions.

Choosing tensor and pipeline parallel sizes​

GoalSetting
The most agents served on a model that fits on one GPUTensor parallel size 1. Every GPU runs its own copy.
A model too large for one GPUThe smallest tensor parallel size whose GPUs hold the weights plus room for the KV cache.
A model too large for one machineTensor parallel size equal to GPUs per node, and pipeline parallel size 2 or more.
Long agent contexts with many turnsA larger tensor parallel size leaves more GPU memory for the KV cache of each copy.
A prefill and decode splitPipeline parallel size 1, and tensor parallel size equal to GPUs per node. See Split prefill and decode.

Sizing rule of thumb. The model weights take about the parameter count in billions times 2 GB at 16-bit precision, or times 1 GB at 8-bit (FP8) precision. A 27-billion-parameter FP8 model needs about 28 GB for weights. Each GPU in a copy holds the weights divided by the tensor parallel size. Keep at least as much memory again free for the KV cache, because agents send long prompts that grow with every turn.

Sandbox CPUs​

Which CPUs the agent sandboxes may run on. A NUMA node, mentioned below, is a group of CPU cores and the memory closest to them. Every machine keeps a few CPUs for metrics collection in every mode: one physical core, or two physical cores on a machine with 64 or more CPUs. The model server and the run controller are never restricted by this setting.

Sandbox CPUs tooltip

ModeWhat happensUse it for
Unpinned (default)Sandboxes run on any CPU except the ones kept for metrics, the way most clusters run agent workloads.Normal capacity measurement.
ControlSandboxes use the same pinned runtime as Pinned, with every CPU except the ones kept for metrics.The baseline for a pinned comparison.
PinnedSandboxes run only on the CPUs you choose for each machine. The kernel enforces it, even against a root process inside the sandbox.Measuring the effect of a dedicated CPU set, or isolating agents from the model server's CPUs.

Control mode

How to measure the effect of pinning. Run Control and Pinned back to back on the same cluster group, with every other setting the same. The two runs differ only by the CPU set, so the difference in their results is the effect of CPU contention.

Choose CPUs for Pinned mode​

  1. Select Pinned.
  2. Choose a cluster group. A CPU entry box appears for each machine.
  3. For each machine, enter CPUs in kernel list format, for example 8-15 or 8-15,64-71. Or select a NUMA node button to fill in every CPU of that node.
  4. Read the summary under the box. It shows how many CPUs you chose, how many physical cores they cover, and which NUMA nodes they sit on.

Pinned mode before a cluster group is chosen

Pinned mode with one CPU box per machine

Rules for Pinned mode:

  • Choose CPUs for at least one machine, or switch to Control or Unpinned.
  • A machine you leave empty runs its sandboxes as in Control mode.
  • You cannot choose a CPU kept for metrics collection. The error names the CPUs you can choose.
  • Selecting a whole NUMA node as a unit is not available, because memory placement on the node has not been verified. The NUMA buttons fill in the node's CPUs instead.
  • "CPUs not reported yet" means the machine's agent has not reported its CPU layout. Update and restart the agent on that machine, then reload the page.

Before every level, a short check runs on every machine to confirm the CPU mode is in effect. If it is not, the run fails instead of measuring under the wrong isolation. The report's Sandbox CPU card shows each machine's mode, chosen CPUs, physical cores, NUMA nodes, and the check result.

Split prefill and decode across machines​

Off by default. When on, some machines compute prompts (prefill) and the others generate answers (decode). The switch is available only when the cluster group and parallelism meet the requirements.

Split prefill and decode tooltip

See Split prefill and decode across machines for the requirements, the settings, and how to compare results.

Serving and sandbox settings​

Leave a setting empty to use Automatic or its default. Automatic sizes the value from the selected machines and the model when the run deploys.

Serving and sandbox settings tooltip

Without a prefill and decode split, this section holds only the Sandboxes settings. With a split, it also holds the engine settings described in Split prefill and decode.

Sandboxes​

Every task sandbox at every level gets these settings. They stay the same at every level, so levels stay comparable.

Sandboxes tooltip

SettingWhat it meansDefaultAllowed values
Sandbox memoryThe most memory one sandbox may use before it is stopped. Each sandbox reserves an equal share of the free memory, up to this limit.4096 MiB128 to 4,194,304 MiB
Sandbox CPUThe most CPU one sandbox may use. 1000 millicores is one core.1000 millicores100 to 1,024,000 millicores
Sandbox disk reservationDisk the scheduler reserves for one sandbox.Automatic16 to 10,485,760 MiB

Sandbox memory tooltip

Sandbox memory. A level whose sandboxes cannot each reserve the minimum memory the agent needs does not start. If a level does not start for this reason, lower the level, add machines, or free memory on the machines.

Sandbox CPU tooltip

Sandbox CPU. This is a cap, not a reservation. Each sandbox reserves one tenth of a core and may use up to this limit. More sandboxes than cores can therefore run on one machine. They share the cores, and the report shows the contention. One whole core is the floor that avoids slowing a single-threaded tool. Raise it for tasks that compile code or run tests in parallel.

The run controller. The Harbor controller that starts and grades the trials also runs on the cluster. It takes about 3 CPU cores, and its memory grows with the level, up to tens of GiB at levels of several hundred agents. Leave room for it when you size the machines.

Sandbox disk reservation tooltip

Sandbox disk reservation. Automatic reserves 256 MiB per sandbox up to 768 sandboxes, and less per sandbox above that, so a level reserves about 192 GiB in total. The level's disk check uses the disk the sandboxes actually wrote.

Examples​

GoalCluster groupGPUs per nodeTensor parallelPipeline parallelResult
A 14B model on one 8-GPU machine1 machine, 8 GPUs8214 replicas of 2 GPUs each.
A 27B FP8 model on one machine with two 96 GB GPUs1 machine, 2 GPUs2112 replicas, one per GPU.
A 27B model on two machines with one GPU each2 machines, 1 GPU each1121 replica split across both machines.
A prefill and decode split on two 2-GPU machines2 machines, 2 GPUs each221One prefill engine and one decode engine.

A configured Hardware step

What's next​