User Guide
This guide walks through the workflows that make up day-to-day work with Metrum AI Bench CLI. Each section is task-oriented and links to the Feature Reference for flag-level detail.
If you have not run anything yet, do the Quickstart first. To drive the CLI from Claude Code, Codex, or OpenCode, see Agent-driven benchmarking.
Sections:
- Building blocks
- Entry points
- Tools at a glance
- Closed-loop vs open-loop
- Common load controls
- Reading results
- Multi-run aggregates
- Publishing-oriented runs
- When something fails
Modality-specific walkthroughs live in Modalities. Sweeps and exports live in Strategic benchmarking. Agent loops: Agent-driven benchmarking.
Building blocks
Bench is a client. It sends traffic to an OpenAI-compatible endpoint you already operate, measures what the client observes, and writes JSONL.
| Concept | Plain-language meaning | Example |
|---|---|---|
| Modality | Which binary / subcommand drives the load | llm, vlm, asr, imagegen |
| Scenario | Operator-chosen label stamped on the run | prompt-library-median |
| Run | One execution with one configuration | 100 requests at concurrency 8 |
| Phase | Warmup, measure, or drain | Warmup excluded from measured distributions |
| Data log | JSONL file of request records plus summary | --data-log results.jsonl |
| SUT | Operator-declared system-under-test block | --sut sut.json |
| Goodput | Successes meeting configured SLOs (and strategic validity checks), excluding explicitly unanswered rows; unmarked legacy/ASR/imagegen rows are treated as answered for goodput | --slo ttft=250ms --slo e2e=2s |
Read it as a sentence: choose a modality, label a scenario, execute a run, inspect the data log, and optionally attach a declared SUT before publication.
Entry points
| Form | Role |
|---|---|
metrum-ai-bench-cli <modality> -- <flags> | Preferred unified entry. Everything after -- is forwarded to that modality binary. |
metrum-ai-bench-cli-llm (and vlm / asr / imagegen / prompts) | Same modality without the dispatcher; equivalent flags. |
metrum-ai-bench-cli-strategic | Concurrency/rate sweeps, knee, sessions, exports. Separate binary, not a unified subcommand. |
Flag maps: CLI Reference. Agent-oriented copy-paste: Agent-driven benchmarking.
Tools at a glance
| Entry | Measures | When to use |
|---|---|---|
metrum-ai-bench-cli llm | Chat/completion latency, TTFT, ITL/TPOT, token throughput | Text OpenAI-compatible /v1/chat/completions or completions |
metrum-ai-bench-cli vlm | Same as LLM plus image payload size | Vision models with image_url / image_urls prompts |
metrum-ai-bench-cli asr | Transcription latency, RTFx, optional WER/CER | /v1/audio/transcriptions |
metrum-ai-bench-cli imagegen | Image generation latency and artifact hashes | /v1/images/generations |
metrum-ai-bench-cli prompts | ISL/OSL mix selection from metrum-ai/prompt-library | Build a JSONL prompt set with target mean/median lengths |
metrum-ai-bench-cli selftest | Local sanity check of the install | After build or release unpack |
metrum-ai-bench-cli-strategic | Concurrency/rate sweeps, knee, sessions, exports | Capacity planning and multi-turn validity (separate binary) |
The preferred entry point for modalities and helpers is metrum-ai-bench-cli with
those subcommands. Sweeps and exports use the separate
metrum-ai-bench-cli-strategic binary (not a unified subcommand).
Closed-loop vs open-loop
| Mode | How you select it | What it means |
|---|---|---|
| Closed-loop | Omit --request-rate | Outstanding work is capped by --concurrency (and optional --max-concurrency) |
| Open-loop | Set --request-rate (optional --arrival poisson or constant) | Requests are scheduled by rate; coordinated-omission latency is the headline open-loop field |
Do not compare open-loop and closed-loop numbers as if they were the same experiment. See Performance Methodology and Known limitations.
Common load controls
Shared across modality binaries (exact lists: CLI Reference):
| Control | Role |
|---|---|
--url / --endpoints-file | Single endpoint or multi-endpoint pool |
--num-requests | Total requests, including warmup (at least 1) |
--concurrency | Closed-loop parallelism (at least 1) |
--warmup-requests | Issued before measure; excluded from measured distributions |
--seed | Deterministic scheduling / prompt cycling where applicable |
--request-rate / --arrival | Open-loop scheduling; burst schedules the measured population at once |
--max-concurrency | Cap on outstanding work |
--slo | Thresholds that define goodput (ttft=, tpot=, e2e=) |
--data-log | JSONL output path |
--streaming | Required for meaningful TTFT on LLM/VLM chat |
Secrets never appear in stamped summary config. Endpoint URL credentials and
sensitive query values are redacted from per-request diagnostics, but do not
commit API keys in endpoint files. --ci-safety opts automated modality runs
into failing on errors and partial results. --data-log-mode controls output
collisions; append is the default.
Reading results
Each line in --data-log is a complete JSON object with schema_version.
- Request records (
metrum-ai-bench-cli.request.v5) flush on completion with latency, optional TTFT / first-byte / ITL, errors, and modality metrics. - Summary (
metrum-ai-bench-cli.summary.v5) carries counts, rates, distributions, goodput, environment metadata, optionalsut, andpartial.
Printed end-of-run statistics come from the same summary values written to JSONL. There is no separate console estimator.
Multi-endpoint aggregates are labeled pooled_mixture. Use per_endpoint
distributions for diagnosis; do not treat the pool as one homogeneous replica.
For reasoning runs, inspect the truncation count and answered cohort. A null completion-token rate means reported usage is incomplete; visible chunks are not token counts. If reasoning is missing, check the configured parser. If answers are truncated, review the output cap and timeout before changing the workload. See Reasoning models.
Full field detail: Output schema.
Multi-run aggregates
Pass --runs N among the forwarded modality arguments to the unified entry
point to execute sequential independent runs and append a seeded bootstrap
cross-run aggregate to --data-log. Cross-run aggregation uses sample
dispersion and a seeded 10,000-resample percentile-bootstrap 95% confidence
interval.
Publishing-oriented runs
--sut / --require-sut are required to publish a Metrum AI Bench CLI result.
Include the unmodified run summary with its SUT block:
metrum-ai-bench-cli llm -- \
... \
--sut sut.json --require-sut
--require-sut refuses to run without a complete SUT block
(gpu.model, gpu.count, driver_version, runtime.name /
runtime.version / runtime.config, host_os) and implies
--redact-hostname. Example files ship as examples/sut.example.json and
examples/sut.example.yaml.
The SUT is declared, not remotely verified. See Known limitations. The approved results-publication policy defines when a published number may be described as a Metrum AI Bench CLI result.
When something fails
| Symptom | What to check |
|---|---|
| Immediate auth / HTTP errors | URL path, API key, TLS (--insecure / --ca-cert only when you intend them) |
ttft_s always null | Enable --streaming for token TTFT; explicit first-byte inference gives only a labelled HTTP approximation |
| Unexpected mix lengths after prompt library | Set --num-requests to the extractor selected_count and --warmup-requests 0 |
| Tail percentiles look extreme at small n | p99 is flagged unreliable below 100 samples |
| Partial summary | Ctrl-C / interrupt set partial: true; do not treat as a complete campaign cell |
For modality-specific input formats and flags, continue to Modalities.