benchctl
benchctl is the supported terminal interface for Metrum AI Bench. It
uses the same control-plane API as the browser product and is intended for
operators who want to script project setup, workload/scenario creation,
server lifecycle, benchmark execution, KYAI evaluation, result queries,
and worker operation.
Use the CLI when you need repeatable terminal workflows. Use direct HTTP calls from the API Reference when you need lower-level automation that is not yet wrapped by a CLI command.
The safest mental model is:
- Install from the signed-in frontend.
- Authenticate once with
benchctl auth login. - Use
list,status,show,info, andendpointcommands to inspect state before making changes.
Install
The normal install path is the command shown in the signed-in Insights frontend. Open your profile page and copy the CLI install command. The command has this shape:
curl -fsSL "$INSIGHTS_FRONTEND_URL/api/cli/install.sh" | bash
INSIGHTS_FRONTEND_URL is the browser frontend origin, not the PostgREST API
URL and not the /api/agent proxy URL. When in doubt, use the exact command
shown inside the signed-in frontend.
The installer creates shims under ${HOME}/.local/bin for benchctl and
metrum-mcp-call, then writes the control-plane and frontend origins to
~/.metrum/config.json so authentication can reuse the browser session. Add
${HOME}/.local/bin to PATH if your shell does not already include it.
Verify the install:
benchctl version
benchctl --help
For source-tree development only:
cd benchctl
uv sync --extra dev --extra agent
uv run benchctl --help
Authenticate
After installing from the Metrum AI Bench frontend, run:
benchctl auth login
benchctl auth status
benchctl auth context
auth login opens a browser/device-style login flow when the CLI has a saved
frontend URL from the installer. Development-only alternatives are available
for local stacks:
benchctl auth login --api-url "$METRUM_API_URL" --token "$METRUM_JWT_TOKEN"
benchctl auth login --api-url "$METRUM_API_URL" --jwt-secret "$PGRST_JWT_SECRET" --email "$METRUM_ACCOUNT_EMAIL"
Do not use JWT-secret login for customer or production operator workflows. It is for local development and controlled test environments.
The CLI stores local context and credentials under the user's Metrum config directory. Do not copy those files into tickets, docs, or chat.
Core Workflow
Most benchmark runs follow this sequence:
- Select an organization and create or select a project.
- Create one workload for each model, framework, version, tool, precision, or engine-args recipe.
- Create one scenario for each traffic shape or evaluation slice.
- Select or register a server.
- Wait for agent heartbeat, GPU detection, and readiness.
- Schedule jobs.
- Poll status and print results.
- Download SUT, logs, reports, and analytics.
Use this command choice guide when turning that workflow into a script:
| Need | Use | Notes |
|---|---|---|
| Inspect current state | project list, workload list, scenario list, job status, job show | Read-only checks. Run these before mutating state. |
| Create benchmark metadata | project create, workload create, scenario create | Writes control-plane rows; does not launch provider hardware. |
| Edit or remove metadata | project update/delete, workload update/delete, scenario update/delete | Mutates or deletes control-plane records. Confirm you are in the intended project/org first. The server refuses these edits for a project that has run. |
| Change a project that has run | project copy | Copies the project with every workload, scenario, and saved setting. Change the copy. See Change a project that has run. |
| Run on an existing server | scenario execute --scenario ... --server ... or job schedule --scenario ... --server ... | Creates a benchmark job for an already registered worker. |
| Collect artifacts | job results, job sut, job logs, analytics benchmark-results | Read result data after job completion. |
Example:
benchctl org list
benchctl org switch "Metrum AI"
benchctl project create "DeepSeek R1 Distill Qwen 7B Hardware Matrix" --visibility private
benchctl project switch "DeepSeek R1 Distill Qwen 7B Hardware Matrix"
benchctl workload create \
--name "DeepSeek R1 Qwen 7B on vLLM" \
--code "deepseek-r1-qwen7b-vllm" \
--tool metrumbench-llm \
--model "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B" \
--framework vllm \
--version "0.20.0"
benchctl scenario create \
--name "ISL 128 OSL 128 C32" \
--code "isl128-osl128-c32" \
--workload "deepseek-r1-qwen7b-vllm" \
--input-tokens 128 \
--output-tokens 128 \
--concurrency 32 \
--num-requests 32
Command Areas
Run benchctl <area> --help for exact options. Current command areas and
frequently used commands include:
| Area | Purpose | Common Commands |
|---|---|---|
auth | Login, logout, context, token checks. | login, token, status, context, logout |
org | Organization context. | list, switch, current, info, clear |
project | Project lifecycle. | list, create, switch, current, info, update, copy, delete, clear |
models | AI model catalog. | list, create-family, create, estimate-memory, add-from-hf |
framework | Framework catalog and engine args. | list, list-engine-args-sets, create-version, create-engine-args-set |
workload | Workload definitions. | list, create, update, reuse, delete, info, switch |
scenario | Scenario definitions. | list, create, execute, update, delete, info |
server | Server registration and readiness. | list, register, info, health, health-history, health-record, readiness |
job | Job scheduling, monitoring, and artifacts. | schedule, schedule-all, sweep, list, status, show, cancel, results, telemetry, sut, logs, replicas |
endpoint | Bring Your Own Endpoint records. | create, list, get, update, delete, attach-workload, providers, transports |
kyai | KYAI setup, generation, judging, and results. | snapshot, snapshot-*, endpoint-*, judge-config-*, workload-create, scenario-create, generation-*, evaluation-*, run, report, generation-prompt-create, generation-prompt-list, judge-prompt-create, judge-prompt-list, snapshot-import, snapshot-status, snapshot-prompts |
analytics | Read-only result and telemetry views. | benchmark-results, job-outputs, compare-jobs, project-runs, server-status, cost-efficiency, concurrency-scaling, framework-comparison, model-comparison, parameter-sensitivity, quantization-comparison, run-cost-summary, telemetry-summary, hardware-utilization, leaderboard |
control-plane | Discovery and context. | schema-version, context |
distributed-kv | Create and run a distributed KV workload, export a run's recipe, and rerun it. | workload create, run, recipe export, recipe run |
run | Run snapshot export and import. | export, import |
sizer | Hardware sizer integration. | health, status, push, withdraw, list-jobs, solve, train |
switch | Switch logins kept on this machine, and the switches it hears. | login, logout, list, scan |
agent | Worker daemon configuration and operation. | join, config, print-bootstrap, start, start-benchmark-worker, start-kyai-worker, status, stop, manage-run-model-server |
console | Interactive browser session. | open, exec, quit |
Most list/read/report commands support JSON output through the global flag:
benchctl --format json project list
Change a Project That Has Run
A project that has run cannot be changed. This rule keeps every result tied to the exact settings that produced it. A project has run once it starts any execution, such as a run, a job, or a cloud execution. Deleting its runs does not make it editable again.
The server refuses these commands for a project that has run:
| Area | Refused commands |
|---|---|
project | update |
workload | create, update, reuse, delete |
scenario | create, update, delete |
kyai | workload-create, scenario-create |
project delete still works on a project that has run.
When the server refuses a change, the CLI prints this message and exits with status 1:
Error: This project has run, so it cannot be changed. Copy it with "benchctl project copy" or Save as New in the project editor, then change the copy.
To change the project, copy it and change the copy:
benchctl project copy NAME [--name NEW_NAME] [--switch]
| Option | Description |
|---|---|
NAME | Name of the project to copy. Required. |
--name, -n | Name for the copy. If you omit it, or give the original name, the copy is named NAME (copy). If that name is in use, the copy is named NAME (copy 2), and so on. |
--switch, -s | Makes the copy your current project. |
The copy holds every workload, scenario, and saved setting of the original. It holds no runs or results. You own the copy. It stays in the original's organization and keeps the original's visibility.
Example:
benchctl project copy "DeepSeek R1 Distill Qwen 7B Hardware Matrix" \
--name "DeepSeek R1 Distill Qwen 7B Hardware Matrix v2" \
--switch
Output:
✓ Copied DeepSeek R1 Distill Qwen 7B Hardware Matrix to: DeepSeek R1 Distill Qwen 7B Hardware Matrix v2
Switched to: DeepSeek R1 Distill Qwen 7B Hardware Matrix v2
A workload keeps its code in the copy, unless you already own a different workload with that code. In that case the copy saves the workload under a new code, and the CLI prints one line for each change:
Workload deepseek-r1-qwen7b-vllm is saved as deepseek-r1-qwen7b-vllm-1a2b3c4d
Use the new code in later scenario and job commands. If you pass a
--name that one of your projects already uses, the CLI prints
Error: A project with this name already exists. Choose another name. and
makes no copy.
KV Cache Offload (Single Node)
A KV Cache Offload workload measures vLLM on one machine with its KV cache in GPU
memory (gpu), in CPU RAM (cpu), or on a disk (nvme). The CLI stores the same
tier settings, scenario defaults, and round count as the web wizard, and reads
the same results.
-
List the tiers and the settings each one uses:
benchctl workload kv-cache-tiers -
Create a project, then a workload for one tier. Leave out
--engine-argsand--lmcache-engine-args-setto get the tier's own settings:benchctl project create kv-cpu-qwen3-8b --no-switchbenchctl workload create --project kv-cpu-qwen3-8b \--name "cpu tier" --code cpu-tier --tool multiroundqa \--model qwen-3-8b --framework vllm --version 0.32.0 --kv-cache-tier cpu -
Create a scenario. The CLI prints the values the run will use:
benchctl scenario create --project kv-cpu-qwen3-8b --workload cpu-tier \--name c8-r5 --code c8-r5 --concurrency 8 --num-rounds 5✓ Created scenario: c8-r5Concurrency: 8Tokens: 8192/512Rounds: 5Dataset: sammshen -
Run it on your machine.
--gpu-count 1uses one GPU of a larger machine, as the web app does:benchctl job schedule --project kv-cpu-qwen3-8b --workload cpu-tier \--scenario c8-r5 --server "<machine-hostname>" --gpu-count 1The output shows the job ID and
KV cache tier: cpu. Run one KV Cache Offload job at a time on a machine. -
Check the job, then read the results. Results need the team that owns the project, so select it first:
benchctl job status <job-id>benchctl org switch "<team name or id>"benchctl analytics kv-cache-results --job-id <job-id>benchctl -F json analytics kv-cache-results --job-id <job-id> --all-columnsThe table shows the main columns.
--all-columnsshows every column, the same fields the API returns.
| Option | Command | Default and rule |
|---|---|---|
--kv-cache-tier | workload create | gpu, cpu, or nvme. Default gpu. |
--lmcache-engine-args-set | workload create | Your own cache server settings for cpu or nvme. |
--concurrency | scenario create | Users at the same time. Default 1. Each dataset supports a number of users that depends on --num-rounds: sammshen gives 598 at 25 rounds per user and 4,946 at 5. A higher number is refused, and the message names the limit. |
--input-tokens, --output-tokens | scenario create | Default 8192 and 512. |
--num-rounds | scenario create, scenario update | 5 or more. Default 25. Fewer is refused before anything is sent. |
--dataset | scenario create | A registered dataset. Default sammshen. One dataset per workload. |
--param KEY=VALUE | scenario create, scenario update | Any other tool setting. In scenario update, an empty value removes it. |
--gpu-count | job schedule, scenario execute | GPUs to use. Default: all of them. |
To use your own vLLM or cache server values, save a set for your team, then name
it in workload create. --from copies a default set, so you give only the values
you change:
benchctl framework save-engine-args-set --framework vllm --version 0.32.0 \
--set-code my-kv-cpu-vllm --from kv-cache-offload-cpu \
--arg gpu_memory_utilization=0.70 --tool multiroundqa
benchctl framework save-engine-args-set --framework lmcache-server --version 0.5.5 \
--set-code my-lmcache-cpu --from lmcache-server-cpu-default \
--arg l1_size_gb=20 --arg l1_init_size_gb=10 --tool multiroundqa
benchctl workload create --project kv-cpu-qwen3-8b --name "cpu custom" --code cpu-custom \
--tool multiroundqa --model qwen-3-8b --framework vllm --version 0.32.0 --kv-cache-tier cpu \
--engine-args my-kv-cpu-vllm --lmcache-engine-args-set my-lmcache-cpu
framework list-engine-args-sets --framework vllm --version 0.32.0 shows each set
and its values. The set belongs to the selected team. Saving the same code again
replaces its values.
Change the round count of a scenario before its project runs:
benchctl scenario update c8-r5 --project kv-cpu-qwen3-8b --workload cpu-tier --num-rounds 12
A project that has run is locked. Copy it to change it (see
Change a Project That Has Run). Stop a job with
job cancel, or remove a whole execution with job delete-execution. The API
steps are in KV Cache Offload (Single Node).
Distributed KV Cache
The CLI creates and runs a distributed KV cache benchmark with the same settings as the web wizard. Two commands cover the flow:
distributed-kv workload createmakes a project that holds one distributed KV workload. It prints the plan for each machine of the cluster group first, and it refuses settings that cannot run with the database's reason.distributed-kv runstarts a run of that workload on its cluster group.
benchctl distributed-kv workload create \
--cluster-group "<cluster-group-code>" \
--stack vllm-lmcache-valkey \
--tier cpu \
--model "<model-code>" \
--framework-version "<engine-version>" \
--users 8 --users 32 \
--shared-tier-gb 64 \
--gpus-per-node 2 \
--project "dkv-l40s-pair"
benchctl distributed-kv run --project "dkv-l40s-pair" \
--project-uuid "<project-uuid>" --workload "<workload-code>"
workload create prints the project id and the exact run command to use. It
creates one scenario for each --users value. Each option sets the key of the
same name in the distributed_kv object of
Distributed KV Cache.
A key you do not set is not sent, so the stack's catalog default applies.
| Option | Sets |
|---|---|
--cluster-group, --stack | Cluster group code and stack code. Both are required. |
--tier, --engine-args-set | KV cache tier (gpu, cpu, or nvme) and engine arguments set. The CLI takes the engine arguments set and the cache server set from the tier. |
--transfer-backend, --transport, --routing-policy | Transfer backend, network protocol, and router policy. |
--gpus-per-node, --node-layout PREFILL:DECODE | GPUs per machine, and for each machine how many GPUs prefill and how many decode. Repeat --node-layout once per machine, head first. |
--prefill-nodes, --prefill-tp, --prefill-gpus | Prefill sizing of a prefill and decode stack. |
--shared-tier-gb, --host-tier-gb, --disk-tier-gb, --disk-tier-path | Tier sizes. |
--decode-prefix-caching or --no-decode-prefix-caching | Prefix caching on the decode engines only. |
--tcp-interface-name, --rdma-device | Network interface for KV traffic over TCP, and the RDMA ports to use. Repeat --rdma-device for more than one. Without it the run uses every active port. |
--repeat-count | Repeats of each scenario, from 1 to 20. |
--users, --rounds, --input-tokens, --output-tokens, --requests | The scenarios. |
--project, --workload-code, --org | Names for the project and the workload, and a team other than the current one. |
A run reserves every machine of its cluster group before it starts. When another
run still uses a machine of the group, the run waits and its job stays pending.
It starts when the machines are free. Check the wait with job status or on the
Monitoring page of the web app. If the run cannot start, the job shows the
database's reason, for example the machine name and the run that uses it.
After a distributed KV run exists, the CLI can export its recipe and rerun it on matching machines:
benchctl distributed-kv recipe export "<run-id>" --out recipe.json
benchctl distributed-kv recipe export --publication "<publication-uuid>" --out recipe.json
benchctl distributed-kv recipe run recipe.json --cluster-group "<cluster-group-code>"
| Command | What it does |
|---|---|
recipe export RUN_ID --out PATH | Writes the recipe of a run from your team. The recipe holds no passwords, keys, machine names, or addresses. |
recipe export --publication UUID --out PATH | Writes the recipe of a published result. It needs no sign-in. |
recipe run PATH --cluster-group CODE | Checks that the cluster group plans the same number of machines, with the same GPU model and the same GPUs in use. Then it creates a new project from the recipe and starts one run for each workload. Use --project to name the project and --org to pick a team other than the current one. |
recipe run refuses a cluster group whose machines differ from the recipe's
machines, because a rerun there would not reproduce the result.
KYAI Workflow
KYAI usually runs after the candidate model endpoint is reachable. Create or select the dataset snapshot, candidate endpoint, judge config, KYAI workload, and scenario; then run generation and evaluation:
There are two command styles:
| Style | Use When | Commands |
|---|---|---|
| Explicit setup/run commands | You want each setup object and run phase visible in logs. | snapshot-create, endpoint-create, judge-config-create, workload-create, scenario-create, generation-create, evaluation-create |
| Short compatibility wrappers | You want concise phase-oriented scripts. | snapshot create/list/status/prompts, run --phase generation, run --phase evaluation, report --phase ... |
The explicit commands are easiest to audit in long workflows:
benchctl kyai snapshot-list
benchctl kyai endpoint-create --help
benchctl kyai judge-config-create --help
benchctl kyai workload-create --help
benchctl kyai scenario-create --help
benchctl kyai generation-create \
--project "$PROJECT_NAME" \
--name "Quantum generation" \
--workload "$KYAI_WORKLOAD_CODE" \
--scenario "$KYAI_SCENARIO_CODE" \
--batch-size 16
benchctl kyai generation-get "<generation-job-id>"
benchctl kyai generation-outputs --job-id "<generation-job-id>"
benchctl kyai evaluation-create \
--project "$PROJECT_NAME" \
--generation-job-id "<generation-job-id>" \
--name "GPT judge evaluation" \
--judge "$JUDGE_CONFIG_CODE"
benchctl kyai evaluation-get "<evaluation-job-id>"
benchctl kyai evaluation-results --job-id "<evaluation-job-id>"
Compatibility wrappers are also available when you want shorter phase-oriented commands:
benchctl kyai snapshot create \
--code quantum-mechanics-30 \
--name "Quantum Mechanics 30" \
--source-type huggingface \
--source-ref "0xZee/dataset-CoT-Quantum-Mechanics-1224" \
--max-rows 30
benchctl kyai run \
--phase generation \
--project "$PROJECT_NAME" \
--name "KYAI generation" \
--workload "$KYAI_WORKLOAD_CODE" \
--scenario "$KYAI_SCENARIO_CODE"
benchctl kyai run \
--phase evaluation \
--project "$PROJECT_NAME" \
--name "KYAI judge" \
--generation-job-id "<generation-job-id>" \
--judge "$JUDGE_CONFIG_CODE"
benchctl kyai report --phase evaluation --job-id "<evaluation-job-id>"
Run KYAI after metrumbench-llm when both phases reuse one provisioned endpoint. Do not start conflicting model servers on the same machine during performance measurement.
Worker Commands
Workers run the actual benchmark or KYAI jobs. The preferred path is to generate a bootstrap from the Metrum AI Bench frontend or onboarding API, then run the generated install/join command on the worker.
Useful worker-side commands:
benchctl agent join --onboarding-token "$METRUM_ONBOARDING_TOKEN"
benchctl agent status
benchctl agent start-benchmark-worker --foreground
benchctl agent start-kyai-worker --foreground
benchctl agent stop
The server is ready for benchmark scheduling only after the control plane shows fresh agent heartbeat and GPU detection.
For a switch whose login your team keeps on its machines, save the login on every machine that can lead a benchmark. The password is read without echo and never leaves the machine:
benchctl switch login leaf-1 --username metrumbench
benchctl switch login leaf-1 --username metrumbench --key ~/.ssh/switch_ed25519
benchctl switch list
benchctl switch scan
switch scan listens on every network card for one LLDP window and prints the
switch port each card is cabled to. See Network Switches.
Troubleshooting
| Symptom | Check |
|---|---|
| Login cannot find the frontend URL | Reinstall from the signed-in Metrum AI Bench frontend or set the API URL explicitly for development. |
| Jobs remain pending | Check benchctl server readiness, worker heartbeat, and GPU detection. |
| KYAI is too slow | Increase --batch-size when the judge/candidate endpoint can support it. |
| Artifacts are missing | Use job sut, job logs, job results, and analytics benchmark-results to isolate ingestion vs download issues. |