Skip to main content

Feature Reference

This page is a configuration reference for Metrum AI Bench CLI. Each section covers purpose, key parameters, and where to go next. Live --help remains authoritative for the full flag set.

If you are new to a feature, the User Guide covers the workflow first - this page is the lookup, not the tutorial.

Features documented here:

Unified entry point​

metrum-ai-bench-cli dispatches to modality and helper tools. Forward tool-specific flags after --:

metrum-ai-bench-cli llm -- --url ... --scenario ... --data-log ...
metrum-ai-bench-cli prompts -- --revision ... --output mix.jsonl
metrum-ai-bench-cli selftest

During v1.x, modality binaries (metrum-ai-bench-cli-llm, …) remain callable directly. Older metrumbench-* and pre-1.2.0 metrum-ai-bench* shim names were removed in 1.3.0.

Modality runners​

ToolEndpoint familyRequired clap inputs (typical)
metrum-ai-bench-cli-llmChat / completions--scenario, --num-requests, --concurrency, --prompts, --mode, --model, --data-log, --max-tokens (+ --url or --endpoints-file)
metrum-ai-bench-cli-vlmVision chatSame family as LLM plus image fields in prompts
metrum-ai-bench-cli-asr/v1/audio/transcriptionsRuntime-required: --scenario, --num-requests, --input, --model (+ --url or --endpoints-file); --concurrency / --data-log have defaults; optional --ground-truth, --normalizer
metrum-ai-bench-cli-imagegen/v1/images/generations--scenario, --model, --num-requests, --concurrency, --data-log (+ --url or endpoints); plus --prompt or --prompts (and size/model as needed)

Shared load parameters (non-exhaustive): --url / --endpoints-file, --warmup-requests, --seed, --request-rate, --arrival, --max-concurrency, --slo, --streaming, --system-prompt, --insecure, --ca-cert. Shared telemetry flags (same names as strategic): --ndjson, --telemetry, --require-telemetry, --require-telemetry-failures. Modality clients also support --ci-safety for strict error/partial exit behavior and --data-log-mode for collision handling. --arrival burst schedules the measured population at once; an explicit --max-concurrency can cap in-flight work.

--tokenizer requires a build with Cargo feature tokenizer. GitHub Release binaries are built with --features otlp only (not tokenizer). Local tokenizer counts need a source build:

cargo build --release --features tokenizer

For streaming chat and VLM, headline TTFT is the first reasoning or visible output delta; visible-answer timing is separate. A reasoning-only length finish is reasoning_truncated, non-failing by default but not an answer; strict mode and --ci-safety reject it. Other no-answer completions are errors. --ttft-fallback first-token is a deprecated redundant alias; --allow-missing-ttft relaxes the audit. Explicit first-byte inference is HTTP-header timing, not a token. Only choice index 0 is measured. Current request/summary output uses v5; historical v4 TTFT has different semantics.

Walkthroughs: Modalities. Flag lists: CLI Reference.

Prompt library extractor​

ParameterNotes
--revisionPin a 40-character commit SHA (preferred)
--configsample or full
--count / --count-slackPreferred size and allowed deviation
--isl-* / --osl-*Target, unit (tokens / words), stat (mean / median), tolerance
--output / --reportJSONL mix and selection report

Success = statistics within tolerance, not exact --count. Mixed-median selection searches bounded joint-length compositions and can write diagnostic reports on failure. Cache creation failures name the affected path. Details: Prompt library.

Strategic runner​

CapabilityKey flags
Concurrency sweep--sweep, --sweep-by concurrency, --requests-per-stage
Rate sweep--sweep-by rate, --max-in-flight
Server metrics (legacy aggregate)--metrics-url
Telemetry NDJSON--ndjson, --telemetry, --require-telemetry
Exports--html, --csv, --mlperf-dir, --mlperf-scenario
Sessions / validitysession JSONL, --json-schema, --tools, --shared-prefix
Embeddings / rerank--kind embeddings / --kind rerank
OTLP (optional)--otlp-endpoint (release binaries include otlp; see strategic docs)

metrum-ai-bench-cli-strategic is a separate binary, not metrum-ai-bench-cli strategic. Details: Strategic benchmarking. For real-device runs, strongly prefer the Metrum AI all-smi fork for system telemetry; verify the release checksum and record its tag.​

SUT and redaction​

Flag / envBehavior
--sut <PATH>Embed declared SUT JSON/YAML into summary (provenance: "declared")
--require-sut / METRUM_AI_BENCH_REQUIRE_SUT=1Refuse incomplete SUT (gpu, driver, runtime, host_os); implies hostname redaction
--redact-hostname / METRUM_AI_BENCH_REDACT_HOSTNAME=1Write environment.hostname: null

Endpoint URL userinfo and api_key / token query values are redacted in per-request diagnostics; avoid storing credentials in configuration files. YAML config parsing uses serde-saphyr: semantics and diagnostics may differ outside the tested fixtures, and dynamically inspected xtask mappings require string keys and JSON-compatible values. --sut-profile strict adds model, quantization, kernel and quality-evidence requirements to --require-sut.

For publication requirements, see Publishing-oriented runs.

Endpoints export​

metrum-ai-bench-cli endpoints-export converts a completed data log to an Endpoints-shaped bundle with declared system metadata, a performance summary, and arithmetic cross-checks. It is explicitly non-certifying: unsupported Offline/LoadStair/Pareto mappings and TTFT/TPOT definition differences are reported in the bundle rather than silently converted. See the repository's docs/ENDPOINTS_ADAPTER.md.

Selftest​

metrum-ai-bench-cli selftest

Local sanity check after install or unpack. Prints a JSON environment object, then a final selftest: ok line. Exit code 0 means success; non-zero means failure (for example JSON serialization error). Use before debugging endpoint connectivity issues so you know the binary itself is healthy.