Skip to main content

How Agentic Bench works

This page explains, at a high level, every part of an Agentic Bench run: what it is, what it does, and where it runs. You do not need to install or operate any of these parts yourself. Metrum AI Bench sets them up for each run and removes them afterwards.

The parts of an Agentic Bench run and where they run

The components​

ComponentWhat it isWhat it does in a run
Metrum AI Bench control planeThe web app, its API, and its database. Hosted by Metrum, or installed in your own environment.Holds your projects, queues runs, sends each run to your machines, and stores and shows the results.
Metrum agentA small program installed on each of your machines when you onboard it.Checks each machine, forms and removes the Kubernetes cluster, deploys the model, runs each level, and uploads results and telemetry. Machines never log in to each other. Each machine's agent does its own part.
Cluster groupA named set of your machines.Decides which machines one run uses. One run uses one cluster group at a time.
Kubernetes (k3s)A lightweight, standard Kubernetes distribution.Joins the machines of the group into one cluster for the run. The first machine is the cluster head. The other machines join it. The cluster is removed when the run ends.
Cluster add-onsStandard Kubernetes extensions: the NVIDIA or AMD GPU device plugin, the Gateway API and its inference extension, and LeaderWorkerSet.Let Kubernetes schedule work onto GPUs, route model traffic, and run a model replica that spans several machines as one unit.
llm-dAn open-source system for serving models on Kubernetes.Deploys the vLLM replicas and the router in front of them, so every agent calls one address.
RouterThe llm-d gateway and its endpoint picker.Sends each model call to the replica with the shortest queue and the most free KV cache memory. On a prefill and decode split, it sends each request through prefill, then decode.
vLLMThe open-source model server.Runs the model on the GPUs, one copy per replica. It reads the agent's tool calls with the tool call parser of your model.
HarborAn open-source framework for running and grading agents in sandboxes.Its controller runs as one Kubernetes job per level. It creates the sandboxes, starts the agents, and runs the grader on each finished trial.
SandboxAn isolated container.Holds one agent's working environment for one trial: the task files and the tools the agent uses. Sandboxes are spread across all machines and deleted after each level.
AgentThe program that does the work.The Coding Agent is Deep Agents, an open-source agent. The Enterprise Agent is the Metrum enterprise agent. Both call the model through the router.
Task setsThe work the agents do.Terminal-Bench tasks come from the Harbor registry. Enterprise cases are stored by Metrum.
Image registryA container image store that runs inside the cluster.Holds the Harbor controller image, built inside the cluster, and every task image, copied in and pinned by content. Every machine runs exactly the same images. It is reachable only from inside the cluster.
Shared servicesServices the Enterprise agents call.Email Threat Analysis starts a vector search index that holds the prior phishing campaigns, a text embedder, and a reranker, all on CPU. They restart before every level. Contract Review and Document Redaction start none.
Telemetry collectorsSmall collectors on every machine.Read host, GPU, sandbox, model server, and network counters during each level and send them to the control plane. They run on CPUs kept apart from the sandboxes.
Network switch readersOptional.When your network switches are registered in Metrum AI Bench, switch port counters are read for each level. See Network switches.

What a run does on your machines​

WhenWhat happens
Before the runThe Metrum agent on each machine waits until the clock is synchronized to within 0.5 seconds, and installs chrony with the system package manager when it is missing. It refuses a machine that already runs another Kubernetes cluster, and an NVIDIA machine without the NVIDIA Container Toolkit. A failed check stops the run, and the stage tracker shows the reason.
SetupThe agent installs a pinned version of k3s (v1.36.4+k3s1) and forms the cluster. Kubernetes add-ons, the image registry, llm-d, and vLLM are deployed. Each machine must report the GPUs the run asks for.
Model downloadThe model weights download from Hugging Face into a model cache on each machine, /var/lib/metrum/hf-home by default. The cache is kept after the run, so later runs of the same model skip the download.
Each levelThe agent checks that every machine has the free disk the level's sandboxes need, measured on the run's earlier levels. Sandboxes are created, the agents work, the grader scores each trial, results are uploaded, and the level's sandboxes are deleted.
TeardownThe model server is stopped, and k3s is uninstalled from every machine. This releases the GPUs.
After the runThe machines hold no cluster. The files in the next table stay. The next run on the cluster group can start.

What stays on a machine after a run​

WhatWhereSize
Model cache/var/lib/metrum/hf-homeThe model weights. Later runs of the same model reuse them.
Python package cache for the sandboxes/var/lib/metrum/package-cacheUp to 200 GB. Later runs reuse it.
chronyInstalled with the system package managerOnly when the machine had no chrony.
helm v3.16.2/usr/local/bin/helmOne program.
Telemetry and switch readings not yet uploadedUnder the agent's own directory, ~/.metrum/agentRemoved once uploaded.

The free disk you plan for a run includes the package cache. See Hardware.

Teardown time. Removing the cluster takes longer after large levels, because every sandbox's storage is cleaned up. It takes about 6 minutes after a level of 1,200 sandboxes, and can take over an hour after 1,800 sandboxes. The next run on the same cluster group waits until teardown finishes.

The machines are reserved during a run. While a run holds a cluster group, its GPUs serve only that run. Other runs on the same group wait in the queue.

Network access​

Between the machines of a cluster group​

Allow all traffic between the machines of a cluster group. This is the setup Metrum tests. At a minimum, Kubernetes needs these ports open between the machines:

PortDirectionUsed for
TCP 6443Every machine to the cluster headJoining the cluster and talking to it.
UDP 8472Between all machinesThe cluster's pod network.
TCP 10250Between all machinesStarting and watching containers on each machine.

A prefill and decode split over RDMA runs its model servers on each machine's own network, so these ports must also be open between the machines:

PortDirectionUsed for
TCP 8000 and 8200Between all machinesThe prefill and decode model servers.
TCP 9000 and 9200Between all machinesThe KV transfer side channel (NIXL) of each model server.
TCP 15600 and up, one port for each RDMA portBetween all machinesThe KV link check before the model loads.

Model traffic between machines, and KV transfer on a prefill and decode split, also flow over this network. For an RDMA split, each machine needs an active RoCE v2 port on a network that carries RDMA.

From the machines to outside​

DestinationWhyNeeded
Your Metrum AI Bench control planeReceive the run, send results and telemetry.Always.
huggingface.coDownload the model weights.When the model is not yet in the machine's cache.
Container registries: docker.io, ghcr.io, quay.io, registry.k8s.io, gcr.io, cr.agentgateway.dev, and nvcr.io on NVIDIA machinesDownload vLLM, llm-d, Kubernetes, and task images.Each run. Task images are then copied into the cluster's own registry.
get.k3s.io, github.com and raw.githubusercontent.comDownload the Kubernetes installer, the Kubernetes add-on manifests, and the agent.Each run.
get.helm.sh and llm-d-incubation.github.ioDownload helm and the llm-d chart.Each run.
pypi.org, files.pythonhosted.org, download.pytorch.org and pypi.nvidia.comPython packages the sandboxes install, through the cluster's package cache.Each run, until the cache holds them.
hlqxxzsirfrgeqasvaps.supabase.coThe Harbor registry: task set definitions.Each run.
Your operating system's package mirrorInstall chrony.Only when the machine has no chrony.

Your data​

What a run stores​

DataExamples
The run's settingsEvery field you set, the framework version, the exact vLLM launch command, the task image versions.
Your hardwareThe GPU, CPU, memory, driver, operating system, and kernel of each machine.
ResultsEach trial's outcome, grade, time, and failure reason, and the last part of the task tests' output.
Agent tracesEvery model call with the messages sent to the model and its reply, every tool call with its arguments, and every shell command the agent ran, with timings.
TelemetryTime series of host, GPU, sandbox, model server, and network counters for each level.

This data is stored in the Metrum AI Bench control plane, in the workspace that owns the project. A Team project is visible to everyone in your organization. A Private project is visible only to you.

note

Agent traces hold the text the agent sent to the model and received back. With the task sets on these pages, that text is benchmark content, not your business data. If you use your own dataset, treat its traces the same way you treat the dataset.

Secrets​

  • A Hugging Face token you save is released by the control plane only to the machine that needs it, only for the run that uses it. A public model opens no token.
  • The image registry inside the cluster accepts connections only from inside the cluster.
  • Each sandbox is a separate container, created for one trial and deleted after its level.

How long each part takes​

PartTypical time
Host checksSeconds
Benchmark runtime and task imagesA few minutes the first time on a machine, faster after
Kubernetes clusterA few minutes
Model downloadDepends on model size and network speed. Skipped when the model is cached.
Model loading onto GPUsOne to several minutes
Validation trialAbout as long as one trial of the first task
Each levelAbout as long as its slowest few trials. A 64-agent level of fix-git took about 4 minutes on a capable GPU.
TeardownMinutes. Longer after very large levels.

Versions​

Each run records the software it used, including the vLLM version and its exact launch command, the Harbor version, the agent version, and the content digest of each task image. The Test conditions section of the report lists them, so a result can be reproduced and compared with the same software.

What's next​