Skip to main content

Agent-driven benchmarking

Use Metrum AI Bench CLI from a coding agent (Claude Code, Codex, OpenCode, or similar). The CLI is a non-interactive client: explicit flags, deterministic --seed, and JSONL output. That shape fits agent loops without special plugins.

This page is about driving the CLI. It is not about evaluating agent workloads as the system under test. See Limitations for that distinction.

If you have not installed a binary yet, do the Quickstart first.

Search when the model is not exactly specified​

For an exact model ID with a recorded serving entry, use that config and its sources without repeating web search. If the ID is under-specified or no entry exists, search current vendor docs and the model card, then record the exact model, image, runtime flags, and sources. Do not reuse memorized flags.

Search and record at least:

  • Official model card / Hub README (context length, chat template, quantization)
  • Current vLLM / SGLang / TensorRT-LLM docs for that model family (launch args, dtype, tensor parallel, max model len, tool/structured-output notes)
  • Known-good Docker image tags and any Blackwell / SM-specific caveats
  • Recommended sampling defaults when the operator has not fixed them (temperature, top_p, and whether ignore_eos belongs in a throughput study)

Put the chosen values into sut.json runtime.config / notes (and --extra-body-json when needed) so the run stays self-describing. If search results conflict, prefer the vendor engine docs over third-party blog posts, and state the sources in the run notes.

For an exact model with a recorded entry, state that entry in the SUT. For other model choices, record the search results and selected settings.

Default workload: Hugging Face prompt library​

Do not invent a one-line prompt for publishable runs. The default agent path selects a mix from metrum-ai/prompt-library with pinned --revision, then feeds that JSONL to llm (or strategic). See Prompt library.

Use named --profile values unless the operator overrides targets in writing:

ProfileISLOSLNotes
chat-medium (default publishable)512128tokens / median
chat-short25664smoke / short chat
rag-medium2048256retrieval-augmented
summarize-long4096512long context
code-medium1024512coding turns

Default Hub revision is main (latest). The mix report records the resolved commit SHA. For publishable compares, re-run with that SHA (or --require-pinned-revision) so the workload cannot drift. Use --config sample for smoke; --config full for publishable compares. Record profile.name / profile.version from the mix report in SUT notes when publishing.

Why agents work well here​

  • Flags are complete; there is no interactive prompt.
  • --seed makes request ordering and mixes reproducible.
  • --data-log writes one JSON object per line; the last line is summary.v5 (see Output schema).
  • --require-sut refuses to emit an unpublishable run instead of silently dropping SUT metadata.

Drop-in project hint​

Paste this into AGENTS.md or CLAUDE.md in a project that already has the CLI on PATH and an OpenAI-compatible endpoint:

# Metrum AI Bench CLI (agent notes)

- Prefer `metrum-ai-bench-cli <modality> -- <flags>` (unified entry).
- Sweeps need the separate `metrum-ai-bench-cli-strategic` binary.
- For an exact model ID with a recorded serving entry, use it without a new
search. If the model is under-specified or has no entry, search current
vendor docs and the model card, then record source URLs and chosen args in
`sut.json` notes / `runtime.config`.
- Default prompts: extract from Hugging Face `metrum-ai/prompt-library`
(revision defaults to latest `main`; resolved SHA is in the mix report).
For publishable numbers, re-run with that SHA. Do not invent a handmade
one-line prompt for shared numbers.
- After extract: set `--num-requests` to `report.selected_count` and
`--max-tokens` to `report.recommended_max_tokens`. On modality `llm`,
`--warmup-requests 0` preserves mix slots; for GPU cold-start prefer
strategic sweeps with `--warmup-requests` at least as large as stage
concurrency (warmup completes before measurement and does not shift the mix).
For fixed-length throughput studies, add strategic `--ignore-eos` with
`--max-tokens`.
- Run the load generator on the serving host (loopback / same private network).
WAN RTT lands in TTFT. The CLI warns on public/non-RFC1918 `--url` hosts;
set `METRUM_AI_BENCH_ALLOW_REMOTE_URL=1` only when you intentionally accept
that contamination.
- Prefer vendor Docker images for vLLM / SGLang (see Platforms). Pip install on
bare metal is advanced and often fails on CUDA/JIT. Run
`metrum-ai-bench-cli preflight --url …` before long GPU sweeps.
- Scaffold SUT with `metrum-ai-bench-cli sut init --probe` on the measurement
host, then fill runtime/model/vendor. Observed fields are local only.
- For any number you might publish: `--sut <file> --require-sut`.
- Telemetry series are not a catalog in the binary. Curl the live `/metrics`
pages (all-smi, the serving engine, and any other exporter) and set
`--telemetry` YAML `include` from that response. Example YAMLs go stale.
- Write `--data-log` under `./bench-out/`. Summarize with jq only; never invent
metrics. For multi-run deltas use `metrum-ai-bench-cli compare`; for charts
beyond strategic `--html`, use an analysis prompt on CSV/JSON/NDJSON exports
(`docs/TELEMETRY.md`, `.claude/skills/telemetry-analysis/SKILL.md`).

Sample prompt (llm + prompt library)​

Copy the block below into the agent. Replace placeholders, then let the agent run the shell commands.

Run a publishable closed-loop LLM measure with Metrum AI Bench CLI using the
Hugging Face prompt library (not a handmade prompt).

Constraints:
- If <MODEL_ID> has a recorded serving entry, use it and record the entry in
sut.json. If the model is not exact or has no entry, search current vendor
docs and the model card. Record sources and launch/runtime args in sut.json.
- Use `metrum-ai-bench-cli prompts --` then `metrum-ai-bench-cli llm --`.
- Endpoint: $BENCH_URL (OpenAI-compatible chat completions). Prefer loopback on
the GPU host.
- API key: $BENCH_API_KEY as --api-key (do not print the secret).
- Model id: <MODEL_ID>
- Scenario label: hf-chat-512-128
- Hub revision: default latest `main` (resolved SHA in mix-report). For a
frozen publishable compare, pass that SHA with `--require-pinned-revision`.
`--config sample`, `--count 32 --seed 42`
- Workload: --profile chat-medium
- Write mix to bench-out/mix.jsonl and report to bench-out/mix-report.json
- llm: --mode chat --streaming --concurrency 4 --seed 42
--num-requests "$(jq .selected_count bench-out/mix-report.json)"
--max-tokens "$(jq .recommended_max_tokens bench-out/mix-report.json)"
--warmup-requests 0 --prompts bench-out/mix.jsonl
- Write sut.json with real inventory; pass --sut sut.json --require-sut
- --data-log bench-out/results.jsonl
- On success print only:
tail -n 1 bench-out/results.jsonl | jq '{attempted, errors, ttft_s, e2e_s, completion_tokens_per_second, sut: .sut.name, partial}'
- If extract, binary, or --require-sut fails, stop; do not invent metrics.

Shell skeleton:

mkdir -p bench-out

metrum-ai-bench-cli prompts -- \
--config sample \
--count 32 --seed 42 \
--profile chat-medium \
--output bench-out/mix.jsonl --report bench-out/mix-report.json

# Create sut.json with real inventory for this host/endpoint, then:

metrum-ai-bench-cli llm -- \
--url "$BENCH_URL" --api-key "$BENCH_API_KEY" \
--scenario hf-chat-512-128 --model "$MODEL_ID" --mode chat --streaming \
--prompts bench-out/mix.jsonl \
--num-requests "$(jq .selected_count bench-out/mix-report.json)" \
--max-tokens "$(jq .recommended_max_tokens bench-out/mix-report.json)" \
--concurrency 4 --warmup-requests 0 --seed 42 \
--sut sut.json --require-sut \
--data-log bench-out/results.jsonl

tail -n 1 bench-out/results.jsonl | jq \
'{attempted, errors, ttft_s, e2e_s, completion_tokens_per_second, sut: .sut.name, partial}'

Sample prompt (strategic sweep)​

Run a concurrency sweep with metrum-ai-bench-cli-strategic using the same
prompt-library mix.

Constraints:
- For an exact model with a recorded entry, use it. Otherwise search current
vendor docs and the model card, then record the selected flags and sources
in SUT notes.
- Reuse bench-out/mix.jsonl from the extract above (or re-extract with the same pin).
- --prompts bench-out/mix.jsonl
- --max-tokens "$(jq .recommended_max_tokens bench-out/mix-report.json)" (required with --prompts)
- --warmup-requests 4 (GPU) or 2 (mock)
- --sweep 1,2,4,8 --requests-per-stage 16 --streaming
- --html bench-out/sweep.html --csv bench-out/sweep.csv
- Do not use the default --prompt Hello for shared numbers.
- Summarize knee and stage throughput from stdout JSON; do not invent metrics.
- Save stdout JSON as bench-out/sweep-summary.json for later compare.
metrum-ai-bench-cli-strategic \
--url "$BENCH_URL" --api-key "$BENCH_API_KEY" \
--model "$MODEL_ID" --streaming \
--prompts bench-out/mix.jsonl \
--max-tokens "$(jq .recommended_max_tokens bench-out/mix-report.json)" \
--warmup-requests 4 \
--sweep 1,2,4,8 --requests-per-stage 16 \
--html bench-out/sweep.html --csv bench-out/sweep.csv \
| tee bench-out/sweep-summary.json

Sample prompt (compare two GPUs + charts)​

Compare two strategic sweeps (for example H100 vs RTX PRO 6000) with Metrum AI
Bench CLI, then draft labeled charts from the exports.

Constraints:
- Inputs: bench-out/h100-summary.json and bench-out/pro6000-summary.json
(strategic stdout JSON with points[]), or the matching request CSVs.
- Run:
metrum-ai-bench-cli compare \
bench-out/h100-summary.json bench-out/pro6000-summary.json \
--labels H100,Pro6000 --format markdown
- Also emit --format json to bench-out/compare.json.
- Do not invent metrics; only use compare output and the JSON points.
- For charts (outside the CLI SVG): from the two points[] arrays, plot
(1) throughput vs concurrency/load with both series labeled,
(2) p95 latency vs throughput frontier with both series labeled,
(3) optional user_tps / cost_per_million_output_tokens when present.
- Each chart needs titled axes, tick marks, a legend with the same labels as
--labels, and a one-line caption stating workload ISL/OSL and engine.
- Single-run strategic --html already has axes/ticks; use compare for deltas.

Entry points​

FormRole
metrum-ai-bench-cli llm -- …Preferred unified modality entry (-- separates dispatcher args from modality flags)
metrum-ai-bench-cli prompts -- …ISL/OSL mix extract from Hugging Face
metrum-ai-bench-cli preflight --url …Serving endpoint reachability / chat / stream smoke
metrum-ai-bench-cli sut init [--probe]Write SUT template; optional local host probe
metrum-ai-bench-cli compare a.json b.jsonLabeled delta table across strategic runs
metrum-ai-bench-cli-llm …Same modality binary without the dispatcher
metrum-ai-bench-cli-strategic …Concurrency/rate sweeps, knee, sessions, exports (separate binary)

Day-to-day concepts: User Guide. Publishing rules: Publishing-oriented runs. Sweeps: Strategic benchmarking. Serving hosts: Platforms.

Campaign evidence​

Do not commit live-run evidence under artifacts/. Publish a redacted evidence-<campaign> GitHub pre-release (--prerelease --latest=false --target <sha>) and link the asset URL plus sha256 from the PR and from SMOKE_RESULTS / CLAIMS_LEDGER. See CONTRIBUTING.md in the repo.