| Agent | An AI program that works a task in steps: it calls the model, calls tools, reads the results, and repeats until it is done. |
| Agent deadline | An optional time in seconds. The report shows the share of trials that finished within it. It never stops a trial. |
| Agent time | The time from an agent's first model call to its last action in a trial. |
| Agent time multiplier | A number that multiplies each task's own agent time limit. Default 1, range 1 to 576. |
| AgentBench | The name of the Agentic Bench workload on the New project page. |
| Attempt | See Trial. |
| Build variant | The container image that serves the model: CUDA for NVIDIA GPUs, ROCm for AMD GPUs. |
| Cluster add-ons | Standard Kubernetes extensions installed for each run: the NVIDIA or AMD GPU device plugin, the Gateway API and its inference extension, and LeaderWorkerSet, which runs a model replica that spans several machines as one unit. |
| Cluster group | A named set of your machines that Agentic Bench forms into one Kubernetes cluster for a run. |
| Cluster head | The first machine added to a cluster group. It forms the cluster. |
| Coding Agent | The workload kind in which agents solve Terminal-Bench coding tasks. |
| Concurrency level | The number of agents that run at the same time in one step of a run. Also called a level or a rung. |
| Context length | The most tokens one request can hold: the prompt plus the answer. |
| Control mode | A sandbox CPU mode that uses the pinned runtime with every CPU except the ones kept for metrics. The baseline for a pinned comparison. |
| Control plane | The Metrum AI Bench web app, API, and database that queue runs and store results. |
| CUAD | The Contract Understanding Atticus Dataset, a public set of commercial contract clause categories. Contract Review grades against it. |
| CUDA | NVIDIA's GPU software platform. NVIDIA GPUs run the CUDA build of vLLM. |
| Decode | The phase in which the model generates the answer, one token at a time. |
| Deep Agents | The open-source agent framework the Coding Agent uses. |
| Engine args | Launch settings passed to vLLM, such as the tool call parser. |
| Enterprise Agent | The workload kind in which agents work a business case: contract review, document redaction, or email threat analysis. |
| Enterprise gates | The integrity, completion, and quality checks an Enterprise level must pass to qualify. |
| FP8 | An 8-bit number format. An FP8 model or KV cache takes about half the memory of a 16-bit one. |
| GiB, MiB | Units of memory. 1 GiB is 1,024 MiB, about 1.07 GB. |
| GPU busy ratio | The share of time the GPUs did compute work: SM active on NVIDIA, graphics activity on AMD. |
| Graded trial | A trial the grader scored. Trials the platform lost are not graded and do not count against the pass rate. |
| Harbor | The open-source framework that creates each agent's sandbox, runs the agent, and grades the result. |
| Host checks | The checks each machine runs before a run starts: free disk, memory, GPU memory, ports, container runtime, and clock synchronization. |
| Hugging Face | The public site that hosts model weights. Runs download the model from it. |
| Image registry | A store of container images. Each run's cluster has its own, reachable only inside the cluster. |
| Infrastructure failure | A trial the platform lost: a sandbox that never started, a setup that timed out, or a grader that could not run. |
| Inter-token latency | The time between two tokens of one answer after the first token. |
| k3s | A lightweight, standard Kubernetes distribution. Agentic Bench installs it for each run and removes it afterwards. |
| Kubernetes | Open-source software that runs containers across a group of machines as one cluster. |
| KV cache | The model's working memory for a request: the computed state of every token so far. It grows with the prompt. |
| KV transfer path | How KV moves from prefill to decode machines over RDMA: directly between GPUs, or through host memory. |
| Level limit | The most time a level may take before it is stopped as stalled. Between 2 hours and 7 days. |
| llm-d | The open-source system that deploys vLLM on Kubernetes and routes requests to its replicas. |
| Metrum agent | The program installed on each onboarded machine. It runs every step of a run on that machine. |
| Millicore | One thousandth of a CPU core. 1000 millicores is one core. |
| Model profile | The approved settings of a verified model, including its tool call parser. |
| NIXL | The vLLM connector that moves KV cache between prefill and decode machines. |
| NTP | Network Time Protocol, which keeps a machine's clock synchronized. |
| NUMA node | A group of CPU cores and the memory closest to them. Large servers have two or more. |
| Output token cap | The most tokens the model may write in one call. |
| p50, p95, p99 | Percentiles. p95 is the value that 95 of every 100 measurements stay at or below. p50 is the median. |
| Pass rate | Passed trials divided by graded trials. |
| PCIe | The connection between a machine's CPU and its GPUs and network cards. |
| Pinned mode | A sandbox CPU mode in which sandboxes run only on CPUs you choose for each machine. |
| Pipeline parallel size | How many machines one copy of the model spans. |
| Prefill | The phase in which the model reads the whole prompt and builds its KV cache. |
| Prefill and decode split | A layout in which some machines run prefill and others run decode, with the KV cache sent between them. |
| Prompt cache | Stored KV cache of earlier prompts. A request that repeats a cached prefix skips recomputing it. |
| Qualified | A level that met every gate. Also called a valid operating point. |
| RDMA | Remote direct memory access. Network cards read and write memory on another machine without its CPU. RoCE v2 carries RDMA over Ethernet. |
| Reasoning effort | How much a reasoning model thinks before it answers, for models that support it. |
| Reasoning parser | The vLLM setting that separates a model's private reasoning from its answer. |
| Replica | One running copy of the model. |
| RoCE v2 | RDMA over Converged Ethernet, version 2. It carries RDMA over an Ethernet network. |
| ROCm | AMD's GPU software platform. AMD GPUs run the ROCm build of vLLM. |
| Router | The llm-d gateway that sends each model call to one of the model replicas. |
| Rung | See Concurrency level. |
| Sandbox | The isolated container one agent works in for one trial. |
| Serving goodput | The share of model requests within the TTFT and inter-token latency ceilings you set. |
| Serving topology | How the model was served on a level: aggregated, or a split such as 1P1D. |
| SLA | Service level agreement: the pass rate, GPU busy, and time limits a run is held to. |
| SM active | The share of time an NVIDIA GPU's compute units had work loaded. Agentic Bench uses it as the GPU busy ratio on NVIDIA GPUs. |
| Start barrier | The moment all of a level's sandboxes are ready and all agents start together. |
| Task | One unit of work with its own tests or grader, such as fix-git. |
| Task code | The name of a task in its dataset. |
| Task set | A versioned collection of tasks, such as Terminal-Bench 2.0. |
| TCP | The standard internet transport. On a split without RDMA, KV moves over TCP, which is slower. |
| Teardown | The last stage of a run: the model server stops and the cluster is removed. |
| Tensor parallel size | How many GPUs one copy of the model uses on one machine. |
| Terminal-Bench | An open benchmark of command-line coding tasks, distributed through the Harbor registry. |
| Token | The unit a model reads and writes. A token is about three quarters of an English word. |
| Tool call parser | The vLLM setting that reads a model's tool calls out of its output. The agent cannot work without the right one. |
| Trial | One attempt by one agent at one task, from sandbox start to grading. |
| TTFT | Time to first token: how long a request waited before the model wrote its first token. |
| Unpinned mode | The default sandbox CPU mode. Sandboxes run on any CPU except the ones kept for metrics. |
| Valid operating point | A level that met the minimum pass rate, the minimum GPU busy ratio, and, for Enterprise runs, every enterprise gate. |
| Validation trial | One trial of the first task with one agent, run before the levels to prove the setup works. |
| Verified goodput | Passed tasks per hour across all agents. |
| Verified model | A model with an approved profile for Agentic Bench. |
| vLLM | The open-source model server Agentic Bench uses. |