CLI Reference
Live --help on each binary is authoritative. The repository regenerates a
command dump with cargo xtask render-cli-help into docs/CLI.md for the
unified dispatcher, modality clients, prompts, strategic and mock server. If
this page and --help disagree, trust --help.
Entry points
| Binary / subcommand | Role |
|---|---|
metrum-ai-bench-cli | Unified dispatcher (llm, vlm, asr, imagegen, prompts, selftest) |
metrum-ai-bench-cli-llm | LLM modality client |
metrum-ai-bench-cli-vlm | VLM modality client |
metrum-ai-bench-cli-asr | ASR modality client |
metrum-ai-bench-cli-imagegen | Image generation client |
metrum-ai-bench-cli-prompts | Prompt-library mix extractor |
metrum-ai-bench-cli-strategic | Sweeps, sessions, exports (separate binary, not a unified subcommand) |
metrum-ai-bench-cli-mock-server | Deterministic local mock for strategic fixtures |
metrum-ai-bench-cli --help
metrum-ai-bench-cli-llm --help
metrum-ai-bench-cli-strategic --help
Print version only (where supported):
metrum-ai-bench-cli-llm --version-only
LLM required options (typical)
metrum-ai-bench-cli-llm
--scenario <SCENARIO>
--num-requests <N>
--concurrency <N>
--prompts <PATH|URL>
--mode <chat|completion>
--model <MODEL>
--data-log <PATH>
--max-tokens <N>
Plus --url or --endpoints-file (mutually exclusive with the single-URL /
API-key pair patterns described in --help).
Shared load flags (common)
| Flag | Purpose |
|---|---|
--url | Single endpoint URL |
--endpoints-file | Multi-endpoint YAML (curl-style); mutually exclusive with --url / --api-key |
--api-key | Bearer / API key for the single-URL path |
--num-requests | Requests to send (at least 1) |
--concurrency | Concurrent requests (at least 1) |
--warmup-requests | Warmup count (excluded from measured distributions) |
--seed | Determinism for scheduling / cycling |
--request-rate | Open-loop rate |
--arrival | poisson, constant, or burst |
--max-concurrency | Outstanding-work cap |
--slo | Repeatable thresholds, e.g. ttft=250ms, e2e=2s |
--streaming | Stream responses (needed for TTFT) |
--ttft-fallback first-token | Deprecated redundant alias for the first-any default; conflicts with first-byte inference |
--allow-missing-ttft | LLM/VLM: relax only the missing-TTFT audit; strict error gates still apply |
--infer-ttft-from-first-byte | LLM/VLM: approximate missing TTFT from HTTP response headers, not a token |
--tokenizer | Path to tokenizer.json (requires Cargo feature tokenizer; not in GitHub Release builds) |
--log-level | error/warn/info/debug/trace on llm/vlm (default warn) and asr (default info); not on imagegen |
--data-log | JSONL output path; append is the default |
--data-log-mode | append, refuse-existing, overwrite, or new-dir |
--strict-reasoning-truncation | Treat unanswered reasoning-only length truncations as errors and exit nonzero |
--ci-safety | Enable strict reasoning truncation and fail automated runs on errors or partial results |
--extra-body-json | Object merged into supported request bodies; top-level null removes a key and non-object JSON is rejected |
--ndjson | Tagged telemetry NDJSON path (telemetry.v3) |
--telemetry | Prometheus scrape YAML; requires --ndjson |
--require-telemetry | Abort after consecutive scrape failures |
--require-telemetry-failures | Consecutive failures before --require-telemetry aborts (default 3) |
--runs | Sequential independent runs + bootstrap aggregate (unified entry) |
--sut / --require-sut / --redact-hostname | Publication and privacy controls |
--ntp-check | Opt-in NTP offset recording (does not hard-fail) |
--insecure / --ca-cert | TLS overrides (use only when you intend them) |
Prompt extractor (high-signal flags)
| Flag | Purpose |
|---|---|
--revision | Dataset commit SHA |
--config | sample / full |
--count / --count-slack | Preferred size and slack |
--isl-target / --osl-target | Length targets |
--isl-unit / --osl-unit | tokens or words |
--isl-stat / --osl-stat | mean or median |
--isl-tolerance / --osl-tolerance | Absolute tolerances |
--output / --report | Mix JSONL and report JSON |
Strategic (high-signal flags)
| Flag | Purpose |
|---|---|
--sweep / --sweep-by | Stage list and axis (concurrency / rate) |
--requests-per-stage | Requests per sweep stage |
--ttft-fallback first-token / --allow-missing-ttft | Streaming chat/VLM fallback or missing-TTFT audit opt-in; request errors remain errors |
--ndjson | Tagged telemetry NDJSON path |
--telemetry | Prometheus scrape YAML |
--require-telemetry | Abort after consecutive scrape failures |
--metrics-url | Legacy engine metrics scrape (desugars when --ndjson is set) |
--html / --csv / --mlperf-dir | Exports |
--mlperf-scenario | server or offline-style scenario name per --help |
--kind | Chat default, or embeddings / rerank |
--json-schema / --tools | Validity checkers |
--slo e2e=… | Goodput thresholds: e2e= / ttft= / tpot= (seconds, streaming) and user_tps= (tok/s) |
--otlp-endpoint | OTLP export (release builds include feature otlp) |
Getting the full dump
Live --help on each binary is always authoritative.
cargo xtask render-cli-help regenerates docs/CLI.md for the unified
dispatcher, modality binaries, prompts, strategic, and mock-server. Live
--help on each binary remains authoritative.
# from a checkout (after cargo build --bins)
cargo xtask render-cli-help
Or inspect docs/CLI.md in the repository for the last generated dump.