Skip to main content

Agentic Bench glossary

TermDefinition
AgentAn AI program that works a task in steps: it calls the model, calls tools, reads the results, and repeats until it is done.
Agent deadlineAn optional time in seconds. The report shows the share of trials that finished within it. It never stops a trial.
Agent timeThe time from an agent's first model call to its last action in a trial.
Agent time multiplierA number that multiplies each task's own agent time limit. Default 1, range 1 to 576.
AgentBenchThe name of the Agentic Bench workload on the New project page.
AttemptSee Trial.
Build variantThe container image that serves the model: CUDA for NVIDIA GPUs, ROCm for AMD GPUs.
Cluster add-onsStandard Kubernetes extensions installed for each run: the NVIDIA or AMD GPU device plugin, the Gateway API and its inference extension, and LeaderWorkerSet, which runs a model replica that spans several machines as one unit.
Cluster groupA named set of your machines that Agentic Bench forms into one Kubernetes cluster for a run.
Cluster headThe first machine added to a cluster group. It forms the cluster.
Coding AgentThe workload kind in which agents solve Terminal-Bench coding tasks.
Concurrency levelThe number of agents that run at the same time in one step of a run. Also called a level or a rung.
Context lengthThe most tokens one request can hold: the prompt plus the answer.
Control modeA sandbox CPU mode that uses the pinned runtime with every CPU except the ones kept for metrics. The baseline for a pinned comparison.
Control planeThe Metrum AI Bench web app, API, and database that queue runs and store results.
CUADThe Contract Understanding Atticus Dataset, a public set of commercial contract clause categories. Contract Review grades against it.
CUDANVIDIA's GPU software platform. NVIDIA GPUs run the CUDA build of vLLM.
DecodeThe phase in which the model generates the answer, one token at a time.
Deep AgentsThe open-source agent framework the Coding Agent uses.
Engine argsLaunch settings passed to vLLM, such as the tool call parser.
Enterprise AgentThe workload kind in which agents work a business case: contract review, document redaction, or email threat analysis.
Enterprise gatesThe integrity, completion, and quality checks an Enterprise level must pass to qualify.
FP8An 8-bit number format. An FP8 model or KV cache takes about half the memory of a 16-bit one.
GiB, MiBUnits of memory. 1 GiB is 1,024 MiB, about 1.07 GB.
GPU busy ratioThe share of time the GPUs did compute work: SM active on NVIDIA, graphics activity on AMD.
Graded trialA trial the grader scored. Trials the platform lost are not graded and do not count against the pass rate.
HarborThe open-source framework that creates each agent's sandbox, runs the agent, and grades the result.
Host checksThe checks each machine runs before a run starts: free disk, memory, GPU memory, ports, container runtime, and clock synchronization.
Hugging FaceThe public site that hosts model weights. Runs download the model from it.
Image registryA store of container images. Each run's cluster has its own, reachable only inside the cluster.
Infrastructure failureA trial the platform lost: a sandbox that never started, a setup that timed out, or a grader that could not run.
Inter-token latencyThe time between two tokens of one answer after the first token.
k3sA lightweight, standard Kubernetes distribution. Agentic Bench installs it for each run and removes it afterwards.
KubernetesOpen-source software that runs containers across a group of machines as one cluster.
KV cacheThe model's working memory for a request: the computed state of every token so far. It grows with the prompt.
KV transfer pathHow KV moves from prefill to decode machines over RDMA: directly between GPUs, or through host memory.
Level limitThe most time a level may take before it is stopped as stalled. Between 2 hours and 7 days.
llm-dThe open-source system that deploys vLLM on Kubernetes and routes requests to its replicas.
Metrum agentThe program installed on each onboarded machine. It runs every step of a run on that machine.
MillicoreOne thousandth of a CPU core. 1000 millicores is one core.
Model profileThe approved settings of a verified model, including its tool call parser.
NIXLThe vLLM connector that moves KV cache between prefill and decode machines.
NTPNetwork Time Protocol, which keeps a machine's clock synchronized.
NUMA nodeA group of CPU cores and the memory closest to them. Large servers have two or more.
Output token capThe most tokens the model may write in one call.
p50, p95, p99Percentiles. p95 is the value that 95 of every 100 measurements stay at or below. p50 is the median.
Pass ratePassed trials divided by graded trials.
PCIeThe connection between a machine's CPU and its GPUs and network cards.
Pinned modeA sandbox CPU mode in which sandboxes run only on CPUs you choose for each machine.
Pipeline parallel sizeHow many machines one copy of the model spans.
PrefillThe phase in which the model reads the whole prompt and builds its KV cache.
Prefill and decode splitA layout in which some machines run prefill and others run decode, with the KV cache sent between them.
Prompt cacheStored KV cache of earlier prompts. A request that repeats a cached prefix skips recomputing it.
QualifiedA level that met every gate. Also called a valid operating point.
RDMARemote direct memory access. Network cards read and write memory on another machine without its CPU. RoCE v2 carries RDMA over Ethernet.
Reasoning effortHow much a reasoning model thinks before it answers, for models that support it.
Reasoning parserThe vLLM setting that separates a model's private reasoning from its answer.
ReplicaOne running copy of the model.
RoCE v2RDMA over Converged Ethernet, version 2. It carries RDMA over an Ethernet network.
ROCmAMD's GPU software platform. AMD GPUs run the ROCm build of vLLM.
RouterThe llm-d gateway that sends each model call to one of the model replicas.
RungSee Concurrency level.
SandboxThe isolated container one agent works in for one trial.
Serving goodputThe share of model requests within the TTFT and inter-token latency ceilings you set.
Serving topologyHow the model was served on a level: aggregated, or a split such as 1P1D.
SLAService level agreement: the pass rate, GPU busy, and time limits a run is held to.
SM activeThe share of time an NVIDIA GPU's compute units had work loaded. Agentic Bench uses it as the GPU busy ratio on NVIDIA GPUs.
Start barrierThe moment all of a level's sandboxes are ready and all agents start together.
TaskOne unit of work with its own tests or grader, such as fix-git.
Task codeThe name of a task in its dataset.
Task setA versioned collection of tasks, such as Terminal-Bench 2.0.
TCPThe standard internet transport. On a split without RDMA, KV moves over TCP, which is slower.
TeardownThe last stage of a run: the model server stops and the cluster is removed.
Tensor parallel sizeHow many GPUs one copy of the model uses on one machine.
Terminal-BenchAn open benchmark of command-line coding tasks, distributed through the Harbor registry.
TokenThe unit a model reads and writes. A token is about three quarters of an English word.
Tool call parserThe vLLM setting that reads a model's tool calls out of its output. The agent cannot work without the right one.
TrialOne attempt by one agent at one task, from sandbox start to grading.
TTFTTime to first token: how long a request waited before the model wrote its first token.
Unpinned modeThe default sandbox CPU mode. Sandboxes run on any CPU except the ones kept for metrics.
Valid operating pointA level that met the minimum pass rate, the minimum GPU busy ratio, and, for Enterprise runs, every enterprise gate.
Validation trialOne trial of the first task with one agent, run before the levels to prove the setup works.
Verified goodputPassed tasks per hour across all agents.
Verified modelA model with an approved profile for Agentic Bench.
vLLMThe open-source model server Agentic Bench uses.

What's next​