Skip to main content

benchctl

benchctl is the supported terminal interface for Metrum AI Bench. It uses the same control-plane API as the browser product and is intended for operators who want to script project setup, workload/scenario creation, server lifecycle, benchmark execution, KYAI evaluation, result queries, and worker operation.

Use the CLI when you need repeatable terminal workflows. Use direct HTTP calls from the API Reference when you need lower-level automation that is not yet wrapped by a CLI command.

The safest mental model is:

  1. Install from the signed-in frontend.
  2. Authenticate once with benchctl auth login.
  3. Use list, status, show, info, and endpoint commands to inspect state before making changes.

Install​

The normal install path is the command shown in the signed-in Insights frontend. Open your profile page and copy the CLI install command. The command has this shape:

curl -fsSL "$INSIGHTS_FRONTEND_URL/api/cli/install.sh" | bash

INSIGHTS_FRONTEND_URL is the browser frontend origin, not the PostgREST API URL and not the /api/agent proxy URL. When in doubt, use the exact command shown inside the signed-in frontend.

The installer creates shims under ${HOME}/.local/bin for benchctl and metrum-mcp-call, then writes the control-plane and frontend origins to ~/.metrum/config.json so authentication can reuse the browser session. Add ${HOME}/.local/bin to PATH if your shell does not already include it.

Verify the install:

benchctl version
benchctl --help

For source-tree development only:

cd benchctl
uv sync --extra dev --extra agent
uv run benchctl --help

Authenticate​

After installing from the Metrum AI Bench frontend, run:

benchctl auth login
benchctl auth status
benchctl auth context

auth login opens a browser/device-style login flow when the CLI has a saved frontend URL from the installer. Development-only alternatives are available for local stacks:

benchctl auth login --api-url "$METRUM_API_URL" --token "$METRUM_JWT_TOKEN"
benchctl auth login --api-url "$METRUM_API_URL" --jwt-secret "$PGRST_JWT_SECRET" --email "$METRUM_ACCOUNT_EMAIL"

Do not use JWT-secret login for customer or production operator workflows. It is for local development and controlled test environments.

The CLI stores local context and credentials under the user's Metrum config directory. Do not copy those files into tickets, docs, or chat.

Core Workflow​

Most benchmark runs follow this sequence:

  1. Select an organization and create or select a project.
  2. Create one workload for each model, framework, version, tool, precision, or engine-args recipe.
  3. Create one scenario for each traffic shape or evaluation slice.
  4. Select or register a server.
  5. Wait for agent heartbeat, GPU detection, and readiness.
  6. Schedule jobs.
  7. Poll status and print results.
  8. Download SUT, logs, reports, and analytics.

Use this command choice guide when turning that workflow into a script:

NeedUseNotes
Inspect current stateproject list, workload list, scenario list, job status, job showRead-only checks. Run these before mutating state.
Create benchmark metadataproject create, workload create, scenario createWrites control-plane rows; does not launch provider hardware.
Edit or remove metadataproject update/delete, workload update/delete, scenario update/deleteMutates or deletes control-plane records. Confirm you are in the intended project/org first. The server refuses these edits for a project that has run.
Change a project that has runproject copyCopies the project with every workload, scenario, and saved setting. Change the copy. See Change a project that has run.
Run on an existing serverscenario execute --scenario ... --server ... or job schedule --scenario ... --server ...Creates a benchmark job for an already registered worker.
Collect artifactsjob results, job sut, job logs, analytics benchmark-resultsRead result data after job completion.

Example:

benchctl org list
benchctl org switch "Metrum AI"

benchctl project create "DeepSeek R1 Distill Qwen 7B Hardware Matrix" --visibility private
benchctl project switch "DeepSeek R1 Distill Qwen 7B Hardware Matrix"

benchctl workload create \
--name "DeepSeek R1 Qwen 7B on vLLM" \
--code "deepseek-r1-qwen7b-vllm" \
--tool metrumbench-llm \
--model "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B" \
--framework vllm \
--version "0.20.0"

benchctl scenario create \
--name "ISL 128 OSL 128 C32" \
--code "isl128-osl128-c32" \
--workload "deepseek-r1-qwen7b-vllm" \
--input-tokens 128 \
--output-tokens 128 \
--concurrency 32 \
--num-requests 32

Command Areas​

Run benchctl <area> --help for exact options. Current command areas and frequently used commands include:

AreaPurposeCommon Commands
authLogin, logout, context, token checks.login, token, status, context, logout
orgOrganization context.list, switch, current, info, clear
projectProject lifecycle.list, create, switch, current, info, update, copy, delete, clear
modelsAI model catalog.list, create-family, create, estimate-memory, add-from-hf
frameworkFramework catalog and engine args.list, list-engine-args-sets, create-version, create-engine-args-set
workloadWorkload definitions.list, create, update, reuse, delete, info, switch
scenarioScenario definitions.list, create, execute, update, delete, info
serverServer registration and readiness.list, register, info, health, health-history, health-record, readiness
jobJob scheduling, monitoring, and artifacts.schedule, schedule-all, sweep, list, status, show, cancel, results, telemetry, sut, logs, replicas
endpointBring Your Own Endpoint records.create, list, get, update, delete, attach-workload, providers, transports
kyaiKYAI setup, generation, judging, and results.snapshot, snapshot-*, endpoint-*, judge-config-*, workload-create, scenario-create, generation-*, evaluation-*, run, report, generation-prompt-create, generation-prompt-list, judge-prompt-create, judge-prompt-list, snapshot-import, snapshot-status, snapshot-prompts
analyticsRead-only result and telemetry views.benchmark-results, job-outputs, compare-jobs, project-runs, server-status, cost-efficiency, concurrency-scaling, framework-comparison, model-comparison, parameter-sensitivity, quantization-comparison, run-cost-summary, telemetry-summary, hardware-utilization, leaderboard
control-planeDiscovery and context.schema-version, context
distributed-kvCreate and run a distributed KV workload, export a run's recipe, and rerun it.workload create, run, recipe export, recipe run
runRun snapshot export and import.export, import
sizerHardware sizer integration.health, status, push, withdraw, list-jobs, solve, train
switchSwitch logins kept on this machine, and the switches it hears.login, logout, list, scan
agentWorker daemon configuration and operation.join, config, print-bootstrap, start, start-benchmark-worker, start-kyai-worker, status, stop, manage-run-model-server
consoleInteractive browser session.open, exec, quit

Most list/read/report commands support JSON output through the global flag:

benchctl --format json project list

Change a Project That Has Run​

A project that has run cannot be changed. This rule keeps every result tied to the exact settings that produced it. A project has run once it starts any execution, such as a run, a job, or a cloud execution. Deleting its runs does not make it editable again.

The server refuses these commands for a project that has run:

AreaRefused commands
projectupdate
workloadcreate, update, reuse, delete
scenariocreate, update, delete
kyaiworkload-create, scenario-create

project delete still works on a project that has run.

When the server refuses a change, the CLI prints this message and exits with status 1:

Error: This project has run, so it cannot be changed. Copy it with "benchctl project copy" or Save as New in the project editor, then change the copy.

To change the project, copy it and change the copy:

benchctl project copy NAME [--name NEW_NAME] [--switch]
OptionDescription
NAMEName of the project to copy. Required.
--name, -nName for the copy. If you omit it, or give the original name, the copy is named NAME (copy). If that name is in use, the copy is named NAME (copy 2), and so on.
--switch, -sMakes the copy your current project.

The copy holds every workload, scenario, and saved setting of the original. It holds no runs or results. You own the copy. It stays in the original's organization and keeps the original's visibility.

Example:

benchctl project copy "DeepSeek R1 Distill Qwen 7B Hardware Matrix" \
--name "DeepSeek R1 Distill Qwen 7B Hardware Matrix v2" \
--switch

Output:

✓ Copied DeepSeek R1 Distill Qwen 7B Hardware Matrix to: DeepSeek R1 Distill Qwen 7B Hardware Matrix v2
Switched to: DeepSeek R1 Distill Qwen 7B Hardware Matrix v2

A workload keeps its code in the copy, unless you already own a different workload with that code. In that case the copy saves the workload under a new code, and the CLI prints one line for each change:

Workload deepseek-r1-qwen7b-vllm is saved as deepseek-r1-qwen7b-vllm-1a2b3c4d

Use the new code in later scenario and job commands. If you pass a --name that one of your projects already uses, the CLI prints Error: A project with this name already exists. Choose another name. and makes no copy.

KV Cache Offload (Single Node)​

A KV Cache Offload workload measures vLLM on one machine with its KV cache in GPU memory (gpu), in CPU RAM (cpu), or on a disk (nvme). The CLI stores the same tier settings, scenario defaults, and round count as the web wizard, and reads the same results.

  1. List the tiers and the settings each one uses:

    benchctl workload kv-cache-tiers
  2. Create a project, then a workload for one tier. Leave out --engine-args and --lmcache-engine-args-set to get the tier's own settings:

    benchctl project create kv-cpu-qwen3-8b --no-switch
    benchctl workload create --project kv-cpu-qwen3-8b \
    --name "cpu tier" --code cpu-tier --tool multiroundqa \
    --model qwen-3-8b --framework vllm --version 0.32.0 --kv-cache-tier cpu
  3. Create a scenario. The CLI prints the values the run will use:

    benchctl scenario create --project kv-cpu-qwen3-8b --workload cpu-tier \
    --name c8-r5 --code c8-r5 --concurrency 8 --num-rounds 5
    ✓ Created scenario: c8-r5
    Concurrency: 8
    Tokens: 8192/512
    Rounds: 5
    Dataset: sammshen
  4. Run it on your machine. --gpu-count 1 uses one GPU of a larger machine, as the web app does:

    benchctl job schedule --project kv-cpu-qwen3-8b --workload cpu-tier \
    --scenario c8-r5 --server "<machine-hostname>" --gpu-count 1

    The output shows the job ID and KV cache tier: cpu. Run one KV Cache Offload job at a time on a machine.

  5. Check the job, then read the results. Results need the team that owns the project, so select it first:

    benchctl job status <job-id>
    benchctl org switch "<team name or id>"
    benchctl analytics kv-cache-results --job-id <job-id>
    benchctl -F json analytics kv-cache-results --job-id <job-id> --all-columns

    The table shows the main columns. --all-columns shows every column, the same fields the API returns.

OptionCommandDefault and rule
--kv-cache-tierworkload creategpu, cpu, or nvme. Default gpu.
--lmcache-engine-args-setworkload createYour own cache server settings for cpu or nvme.
--concurrencyscenario createUsers at the same time. Default 1. Each dataset supports a number of users that depends on --num-rounds: sammshen gives 598 at 25 rounds per user and 4,946 at 5. A higher number is refused, and the message names the limit.
--input-tokens, --output-tokensscenario createDefault 8192 and 512.
--num-roundsscenario create, scenario update5 or more. Default 25. Fewer is refused before anything is sent.
--datasetscenario createA registered dataset. Default sammshen. One dataset per workload.
--param KEY=VALUEscenario create, scenario updateAny other tool setting. In scenario update, an empty value removes it.
--gpu-countjob schedule, scenario executeGPUs to use. Default: all of them.

To use your own vLLM or cache server values, save a set for your team, then name it in workload create. --from copies a default set, so you give only the values you change:

benchctl framework save-engine-args-set --framework vllm --version 0.32.0 \
--set-code my-kv-cpu-vllm --from kv-cache-offload-cpu \
--arg gpu_memory_utilization=0.70 --tool multiroundqa
benchctl framework save-engine-args-set --framework lmcache-server --version 0.5.5 \
--set-code my-lmcache-cpu --from lmcache-server-cpu-default \
--arg l1_size_gb=20 --arg l1_init_size_gb=10 --tool multiroundqa
benchctl workload create --project kv-cpu-qwen3-8b --name "cpu custom" --code cpu-custom \
--tool multiroundqa --model qwen-3-8b --framework vllm --version 0.32.0 --kv-cache-tier cpu \
--engine-args my-kv-cpu-vllm --lmcache-engine-args-set my-lmcache-cpu

framework list-engine-args-sets --framework vllm --version 0.32.0 shows each set and its values. The set belongs to the selected team. Saving the same code again replaces its values.

Change the round count of a scenario before its project runs:

benchctl scenario update c8-r5 --project kv-cpu-qwen3-8b --workload cpu-tier --num-rounds 12

A project that has run is locked. Copy it to change it (see Change a Project That Has Run). Stop a job with job cancel, or remove a whole execution with job delete-execution. The API steps are in KV Cache Offload (Single Node).

Distributed KV Cache​

The CLI creates and runs a distributed KV cache benchmark with the same settings as the web wizard. Two commands cover the flow:

  1. distributed-kv workload create makes a project that holds one distributed KV workload. It prints the plan for each machine of the cluster group first, and it refuses settings that cannot run with the database's reason.
  2. distributed-kv run starts a run of that workload on its cluster group.
benchctl distributed-kv workload create \
--cluster-group "<cluster-group-code>" \
--stack vllm-lmcache-valkey \
--tier cpu \
--model "<model-code>" \
--framework-version "<engine-version>" \
--users 8 --users 32 \
--shared-tier-gb 64 \
--gpus-per-node 2 \
--project "dkv-l40s-pair"

benchctl distributed-kv run --project "dkv-l40s-pair" \
--project-uuid "<project-uuid>" --workload "<workload-code>"

workload create prints the project id and the exact run command to use. It creates one scenario for each --users value. Each option sets the key of the same name in the distributed_kv object of Distributed KV Cache. A key you do not set is not sent, so the stack's catalog default applies.

OptionSets
--cluster-group, --stackCluster group code and stack code. Both are required.
--tier, --engine-args-setKV cache tier (gpu, cpu, or nvme) and engine arguments set. The CLI takes the engine arguments set and the cache server set from the tier.
--transfer-backend, --transport, --routing-policyTransfer backend, network protocol, and router policy.
--gpus-per-node, --node-layout PREFILL:DECODEGPUs per machine, and for each machine how many GPUs prefill and how many decode. Repeat --node-layout once per machine, head first.
--prefill-nodes, --prefill-tp, --prefill-gpusPrefill sizing of a prefill and decode stack.
--shared-tier-gb, --host-tier-gb, --disk-tier-gb, --disk-tier-pathTier sizes.
--decode-prefix-caching or --no-decode-prefix-cachingPrefix caching on the decode engines only.
--tcp-interface-name, --rdma-deviceNetwork interface for KV traffic over TCP, and the RDMA ports to use. Repeat --rdma-device for more than one. Without it the run uses every active port.
--repeat-countRepeats of each scenario, from 1 to 20.
--users, --rounds, --input-tokens, --output-tokens, --requestsThe scenarios.
--project, --workload-code, --orgNames for the project and the workload, and a team other than the current one.

A run reserves every machine of its cluster group before it starts. When another run still uses a machine of the group, the run waits and its job stays pending. It starts when the machines are free. Check the wait with job status or on the Monitoring page of the web app. If the run cannot start, the job shows the database's reason, for example the machine name and the run that uses it.

After a distributed KV run exists, the CLI can export its recipe and rerun it on matching machines:

benchctl distributed-kv recipe export "<run-id>" --out recipe.json
benchctl distributed-kv recipe export --publication "<publication-uuid>" --out recipe.json
benchctl distributed-kv recipe run recipe.json --cluster-group "<cluster-group-code>"
CommandWhat it does
recipe export RUN_ID --out PATHWrites the recipe of a run from your team. The recipe holds no passwords, keys, machine names, or addresses.
recipe export --publication UUID --out PATHWrites the recipe of a published result. It needs no sign-in.
recipe run PATH --cluster-group CODEChecks that the cluster group plans the same number of machines, with the same GPU model and the same GPUs in use. Then it creates a new project from the recipe and starts one run for each workload. Use --project to name the project and --org to pick a team other than the current one.

recipe run refuses a cluster group whose machines differ from the recipe's machines, because a rerun there would not reproduce the result.

KYAI Workflow​

KYAI usually runs after the candidate model endpoint is reachable. Create or select the dataset snapshot, candidate endpoint, judge config, KYAI workload, and scenario; then run generation and evaluation:

There are two command styles:

StyleUse WhenCommands
Explicit setup/run commandsYou want each setup object and run phase visible in logs.snapshot-create, endpoint-create, judge-config-create, workload-create, scenario-create, generation-create, evaluation-create
Short compatibility wrappersYou want concise phase-oriented scripts.snapshot create/list/status/prompts, run --phase generation, run --phase evaluation, report --phase ...

The explicit commands are easiest to audit in long workflows:

benchctl kyai snapshot-list
benchctl kyai endpoint-create --help
benchctl kyai judge-config-create --help
benchctl kyai workload-create --help
benchctl kyai scenario-create --help

benchctl kyai generation-create \
--project "$PROJECT_NAME" \
--name "Quantum generation" \
--workload "$KYAI_WORKLOAD_CODE" \
--scenario "$KYAI_SCENARIO_CODE" \
--batch-size 16

benchctl kyai generation-get "<generation-job-id>"
benchctl kyai generation-outputs --job-id "<generation-job-id>"

benchctl kyai evaluation-create \
--project "$PROJECT_NAME" \
--generation-job-id "<generation-job-id>" \
--name "GPT judge evaluation" \
--judge "$JUDGE_CONFIG_CODE"

benchctl kyai evaluation-get "<evaluation-job-id>"
benchctl kyai evaluation-results --job-id "<evaluation-job-id>"

Compatibility wrappers are also available when you want shorter phase-oriented commands:

benchctl kyai snapshot create \
--code quantum-mechanics-30 \
--name "Quantum Mechanics 30" \
--source-type huggingface \
--source-ref "0xZee/dataset-CoT-Quantum-Mechanics-1224" \
--max-rows 30

benchctl kyai run \
--phase generation \
--project "$PROJECT_NAME" \
--name "KYAI generation" \
--workload "$KYAI_WORKLOAD_CODE" \
--scenario "$KYAI_SCENARIO_CODE"

benchctl kyai run \
--phase evaluation \
--project "$PROJECT_NAME" \
--name "KYAI judge" \
--generation-job-id "<generation-job-id>" \
--judge "$JUDGE_CONFIG_CODE"

benchctl kyai report --phase evaluation --job-id "<evaluation-job-id>"

Run KYAI after metrumbench-llm when both phases reuse one provisioned endpoint. Do not start conflicting model servers on the same machine during performance measurement.

Worker Commands​

Workers run the actual benchmark or KYAI jobs. The preferred path is to generate a bootstrap from the Metrum AI Bench frontend or onboarding API, then run the generated install/join command on the worker.

Useful worker-side commands:

benchctl agent join --onboarding-token "$METRUM_ONBOARDING_TOKEN"
benchctl agent status
benchctl agent start-benchmark-worker --foreground
benchctl agent start-kyai-worker --foreground
benchctl agent stop

The server is ready for benchmark scheduling only after the control plane shows fresh agent heartbeat and GPU detection.

For a switch whose login your team keeps on its machines, save the login on every machine that can lead a benchmark. The password is read without echo and never leaves the machine:

benchctl switch login leaf-1 --username metrumbench
benchctl switch login leaf-1 --username metrumbench --key ~/.ssh/switch_ed25519
benchctl switch list
benchctl switch scan

switch scan listens on every network card for one LLDP window and prints the switch port each card is cabled to. See Network Switches.

Troubleshooting​

SymptomCheck
Login cannot find the frontend URLReinstall from the signed-in Metrum AI Bench frontend or set the API URL explicitly for development.
Jobs remain pendingCheck benchctl server readiness, worker heartbeat, and GPU detection.
KYAI is too slowIncrease --batch-size when the judge/candidate endpoint can support it.
Artifacts are missingUse job sut, job logs, job results, and analytics benchmark-results to isolate ingestion vs download issues.