Agent-driven benchmarking

Use Metrum AI Bench CLI from a coding agent (Claude Code, Codex, OpenCode, or similar). The CLI is a non-interactive client: explicit flags, deterministic --seed, and JSONL output. That shape fits agent loops without special plugins.

This page is about driving the CLI. It is not about evaluating agent workloads as the system under test. See Limitations for that distinction.

If you have not installed a binary yet, do the Quickstart first.

Always web-search model and server defaults

Every time a coding agent (Claude Code, Codex, OpenCode, or similar) prepares or runs a benchmark, it must web search online for the current best-known or vendor-default serving configuration for that exact model and stack. Do not reuse memorized flags from prior chats, training data, or an earlier run in the same session without refreshing.

Search and record at least:

• Official model card / Hub README (context length, chat template, quantization)
• Current vLLM / SGLang / TensorRT-LLM docs for that model family (launch args, dtype, tensor parallel, max model len, tool/structured-output notes)
• Known-good Docker image tags and any Blackwell / SM-specific caveats
• Recommended sampling defaults when the operator has not fixed them (temperature, top_p, and whether ignore_eos belongs in a throughput study)

Put the chosen values into sut.json runtime.config / notes (and --extra-body-json when needed) so the run stays self-describing. If search results conflict, prefer the vendor engine docs over third-party blog posts, and state the sources in the run notes.

This rule applies to smoke runs and publishable runs alike. Skipping the search is a process failure even if the CLI command succeeds.

Default workload: Hugging Face prompt library

Do not invent a one-line prompt for publishable runs. The default agent path selects a mix from metrum-ai/prompt-library with pinned --revision, then feeds that JSONL to llm (or strategic). See Prompt library.

Use named --profile values unless the operator overrides targets in writing:

Profile	ISL	OSL	Notes
chat-medium (default publishable)	512	128	tokens / median
chat-short	256	64	smoke / short chat
rag-medium	2048	256	retrieval-augmented
summarize-long	4096	512	long context
code-medium	1024	512	coding turns

Pin revision 0666f62e581b482838ae2e17b333ee36ff3d01b0 in examples (replace when the Hub pin advances). Use --config sample for smoke; --config full for publishable compares. Record profile.name / profile.version from the mix report in SUT notes when publishing.

Why agents work well here

• Flags are complete; there is no interactive prompt.
• --seed makes request ordering and mixes reproducible.
• --data-log writes one JSON object per line; the last line is summary.v3 (see Output schema).
• --require-sut refuses to emit an unpublishable run instead of silently dropping SUT metadata.

Drop-in project hint

Paste this into AGENTS.md or CLAUDE.md in a project that already has the CLI on PATH and an OpenAI-compatible endpoint:

# Metrum AI Bench CLI (agent notes)

- Prefer `metrum-ai-bench-cli <modality> -- <flags>` (unified entry).
- Sweeps need the separate `metrum-ai-bench-cli-strategic` binary.
- **Before every benchmark:** web search for the current best-known / vendor-default
  serving config for this exact model + engine (vLLM/SGLang/etc.). Do not reuse
  memorized launch flags. Record sources and chosen args in `sut.json` notes /
  `runtime.config`.
- Default prompts: extract from Hugging Face `metrum-ai/prompt-library` with a
  pinned `--revision`. Do not invent a handmade one-line prompt for shared
  numbers.
- After extract: set `--num-requests` to `report.selected_count` and
  `--max-tokens` to `report.recommended_max_tokens`. On modality `llm`,
  `--warmup-requests 0` preserves mix slots; for GPU cold-start prefer
  strategic sweeps with `--warmup-requests` at least as large as stage
  concurrency (warmup completes before measurement and does not shift the mix).
  For fixed-length throughput studies, add strategic `--ignore-eos` with
  `--max-tokens`.
- Run the load generator on the serving host (loopback / same private network).
  WAN RTT lands in TTFT. The CLI warns on public/non-RFC1918 `--url` hosts;
  set `METRUM_AI_BENCH_ALLOW_REMOTE_URL=1` only when you intentionally accept
  that contamination.
- Prefer vendor Docker images for vLLM / SGLang (see Platforms). Pip install on
  bare metal is advanced and often fails on CUDA/JIT. Run
  `metrum-ai-bench-cli preflight --url …` before long GPU sweeps.
- Scaffold SUT with `metrum-ai-bench-cli sut init --probe` on the measurement
  host, then fill runtime/model/vendor. Observed fields are local only.
- For any number you might publish: `--sut <file> --require-sut`.
- Write `--data-log` under `./bench-out/`. Summarize with jq only; never invent
  metrics. For multi-run deltas use `metrum-ai-bench-cli compare`; for charts
  beyond strategic `--html`, use an analysis prompt on CSV/JSON exports.

Sample prompt (llm + prompt library)

Copy the block below into the agent. Replace placeholders, then let the agent run the shell commands.

Run a publishable closed-loop LLM measure with Metrum AI Bench CLI using the
Hugging Face prompt library (not a handmade prompt).

Constraints:
- FIRST: web search online for the current best-known / vendor-default serving
  configuration for model <MODEL_ID> on the chosen engine (vLLM or SGLang).
  Refresh even if you think you already know the flags. Record sources and the
  chosen launch/runtime args in sut.json (runtime.config + notes). Prefer
  vendor docs and the model card over third-party posts.
- Use `metrum-ai-bench-cli prompts --` then `metrum-ai-bench-cli llm --`.
- Endpoint: $BENCH_URL (OpenAI-compatible chat completions). Prefer loopback on
  the GPU host.
- API key: $BENCH_API_KEY as --api-key (do not print the secret).
- Model id: <MODEL_ID>
- Scenario label: hf-chat-512-128
- Pin Hub revision 0666f62e581b482838ae2e17b333ee36ff3d01b0, --config sample,
  --count 32 --seed 42
- Workload: --profile chat-medium
- Write mix to bench-out/mix.jsonl and report to bench-out/mix-report.json
- llm: --mode chat --streaming --concurrency 4 --seed 42
  --num-requests "$(jq .selected_count bench-out/mix-report.json)"
  --max-tokens "$(jq .recommended_max_tokens bench-out/mix-report.json)"
  --warmup-requests 0 --prompts bench-out/mix.jsonl
- Write sut.json with real inventory; pass --sut sut.json --require-sut
- --data-log bench-out/results.jsonl
- On success print only:
  tail -n 1 bench-out/results.jsonl | jq '{attempted, errors, ttft_s, e2e_s, completion_tokens_per_second, sut: .sut.name, partial}'
- If extract, binary, or --require-sut fails, stop; do not invent metrics.

Shell skeleton:

mkdir -p bench-out

metrum-ai-bench-cli prompts -- \
  --revision 0666f62e581b482838ae2e17b333ee36ff3d01b0 --config sample \
  --count 32 --seed 42 \
  --profile chat-medium \
  --output bench-out/mix.jsonl --report bench-out/mix-report.json

# Create sut.json with real inventory for this host/endpoint, then:

metrum-ai-bench-cli llm -- \
  --url "$BENCH_URL" --api-key "$BENCH_API_KEY" \
  --scenario hf-chat-512-128 --model "$MODEL_ID" --mode chat --streaming \
  --prompts bench-out/mix.jsonl \
  --num-requests "$(jq .selected_count bench-out/mix-report.json)" \
  --max-tokens "$(jq .recommended_max_tokens bench-out/mix-report.json)" \
  --concurrency 4 --warmup-requests 0 --seed 42 \
  --sut sut.json --require-sut \
  --data-log bench-out/results.jsonl

tail -n 1 bench-out/results.jsonl | jq \
  '{attempted, errors, ttft_s, e2e_s, completion_tokens_per_second, sut: .sut.name, partial}'

Sample prompt (strategic sweep)

Run a concurrency sweep with metrum-ai-bench-cli-strategic using the same
prompt-library mix.

Constraints:
- FIRST: web search online for current best-known / vendor-default serving
  flags for this model + engine; refresh every run; record in SUT notes.
- Reuse bench-out/mix.jsonl from the extract above (or re-extract with the same pin).
- --prompts bench-out/mix.jsonl
- --max-tokens "$(jq .recommended_max_tokens bench-out/mix-report.json)" (required with --prompts)
- --warmup-requests 4 (GPU) or 2 (mock)
- --sweep 1,2,4,8 --requests-per-stage 16 --streaming
- --html bench-out/sweep.html --csv bench-out/sweep.csv
- Do not use the default --prompt Hello for shared numbers.
- Summarize knee and stage throughput from stdout JSON; do not invent metrics.
- Save stdout JSON as bench-out/sweep-summary.json for later compare.

metrum-ai-bench-cli-strategic \
  --url "$BENCH_URL" --api-key "$BENCH_API_KEY" \
  --model "$MODEL_ID" --streaming \
  --prompts bench-out/mix.jsonl \
  --max-tokens "$(jq .recommended_max_tokens bench-out/mix-report.json)" \
  --warmup-requests 4 \
  --sweep 1,2,4,8 --requests-per-stage 16 \
  --html bench-out/sweep.html --csv bench-out/sweep.csv \
  | tee bench-out/sweep-summary.json

Sample prompt (compare two GPUs + charts)

Compare two strategic sweeps (for example H100 vs RTX PRO 6000) with Metrum AI
Bench CLI, then draft labeled charts from the exports.

Constraints:
- Inputs: bench-out/h100-summary.json and bench-out/pro6000-summary.json
  (strategic stdout JSON with points[]), or the matching request CSVs.
- Run:
  metrum-ai-bench-cli compare \
    bench-out/h100-summary.json bench-out/pro6000-summary.json \
    --labels H100,Pro6000 --format markdown
- Also emit --format json to bench-out/compare.json.
- Do not invent metrics; only use compare output and the JSON points.
- For charts (outside the CLI SVG): from the two points[] arrays, plot
  (1) throughput vs concurrency/load with both series labeled,
  (2) p95 latency vs throughput frontier with both series labeled,
  (3) optional user_tps / cost_per_million_output_tokens when present.
- Each chart needs titled axes, tick marks, a legend with the same labels as
  --labels, and a one-line caption stating workload ISL/OSL and engine.
- Single-run strategic --html already has axes/ticks; use compare for deltas.

Entry points

Form	Role
metrum-ai-bench-cli llm -- …	Preferred unified modality entry (-- separates dispatcher args from modality flags)
metrum-ai-bench-cli prompts -- …	ISL/OSL mix extract from Hugging Face
metrum-ai-bench-cli preflight --url …	Serving endpoint reachability / chat / stream smoke
metrum-ai-bench-cli sut init [--probe]	Write SUT template; optional local host probe
metrum-ai-bench-cli compare a.json b.json	Labeled delta table across strategic runs
metrum-ai-bench-cli-llm …	Same modality binary without the dispatcher
metrum-ai-bench-cli-strategic …	Concurrency/rate sweeps, knee, sessions, exports (separate binary)

Day-to-day concepts: User Guide. Publishing rules: Publishing-oriented runs. Sweeps: Strategic benchmarking. Serving hosts: Platforms.