Skip to main content

Release Notes

Customer-facing highlights for recent releases. Full engineering detail lives in the repository CHANGELOG.md.

Metrum AI Bench CLI 1.8.0​

Breaking metric and schema change: headline streaming TTFT now measures the first reasoning or visible output delta; visible-answer TTFT is separate. Completion-token throughput includes reported reasoning and uses the window from first measured send to last terminal completion. Incomplete usage makes the headline unavailable. Reasoning-only length truncations are unanswered and non-failing by default; use strict mode or --ci-safety to reject them. Request and summary schemas are v5. Segment dashboards and TTFT SLOs by schema and semantics instead of mixing v4 and v5. See Reasoning models.

Metrum AI Bench CLI 1.7.2​

Behavior change: a completed stream without a visible answer is a request error under every TTFT flag, including --infer-ttft-from-first-byte. A length-limited reasoning-only response is reasoning_truncated; it is not a successful answer. This can change error counts and strict run exit codes.

  • Visible-answer TTFT remains the default. The opt-in --ttft-fallback first-token records a labelled first-token time when no answer arrives; --allow-missing-ttft allows a null time. Neither changes error outcomes or strict safety gates. First-byte approximation remains an HTTP-header time.
  • First reasoning, first-any-token timing, coverage and reported token usage make thinking-only responses visible without counting them as successful.
  • Binary release archives include the tracked documentation tree. We strongly recommend the Metrum AI all-smi fork for system telemetry.

Metrum AI Bench CLI 1.7.1​

This documentation patch brings the external site up to date with 1.6.0 and 1.7.0, derives the displayed release from the build version, and adds a CI check that requires matching release notes and rejects stale current-version labels. No benchmark runtime behavior changes.

Metrum AI Bench CLI 1.7.0​

  • A reproducible performance-validation harness pins the workload and serving stack and compares release and candidate output throughput and TPOT p50 on a same-GPU grid. Missing points fail; TTFT is informational (#333).
  • For supported request bodies, --extra-body-json must be an object and top-level null values remove keys rather than sending explicit nulls (#357).
  • Prompt selection searches bounded joint-length compositions for mixed medians and writes a diagnostic report on selection failure (#358).
  • Strategic JSON summary is printed before exports; CSV preserves missing TTFT-provenance cells, including failed requests (#362).
  • Live-serving and performance helpers create Hugging Face bind-mount sources before Docker; prompt-cache creation errors identify the path. Existing unwritable caches still require repair (#365).
  • Endpoint credentials are redacted in per-request logs and error paths (#332). The maintained pure-Rust serde-saphyr parser replaces serde_yaml; YAML semantics and diagnostics can differ outside tested fixtures (#375).
  • Dependency security updates, audited Python performance helpers, and hardened container checks improve maintenance (#341, #373, #374, #376).

Metrum AI Bench CLI 1.6.0​

  • Prompt sequence logging and replay, seeded issuance jitter and bounded SSE traces support comparable LLM runs; visible-token and first-any-token TTFT remain distinct (#313).
  • --arrival burst schedules an Offline-style population at once, optionally capped by --max-concurrency; it is not an MLPerf submission (#309).
  • endpoints-export produces a non-certifying Endpoints-shaped bundle with explicit unsupported mappings and definition differences (#306).
  • --sut-profile strict adds model, quantization, kernel and linked quality evidence requirements for publication (#308).
  • Modality --ci-safety fails on errors or partial runs, and --data-log-mode offers append, refuse-existing, overwrite, and new-dir policies (#303, #305).
  • Modality telemetry has a pre-measurement boundary scrape and streaming completed-request rows; example telemetry presets are available (#310–#312).
  • Invalid endpoint URLs fail before traffic, --validate-model adds an optional LLM preflight, and standalone --version-only works for the distributed binaries (#304, #307).
  • Security maintenance updated dependencies and made the self-hosted runner image non-root with a health check; the live-gate secret contract was narrowed (#291–#293, #334, #336, #337, #340). Rust 1.89 is the MSRV.

Metrum AI Bench CLI 1.5.5​

ProductMetrum AI Bench CLI
Version1.5.5
Previous version1.5.4
Release DateOctober 23, 2026
  • Updates the Rust TLS dependencies (rustls 0.23.46, rustls-webpki 0.103.16, and webpki-roots 1.0.10) to address RUSTSEC-2026-0346 through RUSTSEC-2026-0349 and RUSTSEC-2026-0381 / RUSTSEC-2026-0382.
  • Updates Tokio to 1.53.3 to address RUSTSEC-2026-0386.
  • Full engineering detail: repository CHANGELOG.md under 1.5.5.

Metrum AI Bench CLI 1.5.4​

ProductMetrum AI Bench CLI
Version1.5.4
Previous version1.5.3
Release DateOctober 12, 2026
  • Every modality binary (llm, vlm, asr, imagegen) scrapes Prometheus /metrics with --ndjson and --telemetry, the same way strategic does. --require-telemetry aborts on mid-run scrape failures. Legacy --metrics-url remains on strategic only.
  • The v1.5.3 GitHub release assets do not include those modality flags; use 1.5.4 (or later) when a modality cell must scrape /metrics in-process.
  • Full engineering detail: repository CHANGELOG.md under 1.5.4.

Metrum AI Bench CLI 1.5.3​

ProductMetrum AI Bench CLI
Version1.5.3
Previous version1.5.2
Release DateOctober 2, 2026
  • Four-modality live validation: LLM, VLM, ASR, and imagegen gates passed with --sut / --require-sut and all-smi /metrics telemetry.
  • VLM accepts inline data: base64 image URLs in prompt files.
  • Imagegen --seed-mode increment works with a single --prompt (unique seeds per request).
  • Preflight --extra-body-json for thinking-model chat probes.
  • Live scripts: Hub prompt extract by default, prebuilt binary resolution (bench_bin.sh), modality serve launchers, widen helpers, and assert_headline.sh.
  • Dummy -strict-media plus real ASR/VLM fixtures for e2e and quickstarts.
  • Telemetry defaults and analyze.py power/energy handling aligned to the Metrum all-smi fork (/metrics, millijoule scale, skip power-limit gauges).

Metrum AI Bench CLI 1.5.2​

ProductMetrum AI Bench CLI
Version1.5.2
Previous version1.5.1
Release DateSeptember 30, 2026
  • Reject empty measurements when warmup consumes the whole run, or strategic requests_per_stage is zero.
  • Streaming chat fails without visible-token TTFT unless --infer-ttft-from-first-byte opts into a first-byte approximation (ttft_source stamped on outputs).
  • LLM default system prompt removed; published ISL/OSL prefer server usage (isl_osl.length_basis).
  • Quickstart points real GPU servers at vendor Docker images.

Metrum AI Bench CLI 1.5.1​

ProductMetrum AI Bench CLI
Version1.5.1
Previous version1.5.0
Release DateSeptember 24, 2026
  • metrum-ai-bench-cli-prompts defaults to Hub revision main (latest); resolved SHA is recorded in the mix report. Use an explicit SHA / --require-pinned-revision only when freezing a publishable compare.

Metrum AI Bench CLI 1.5.0​

ProductMetrum AI Bench CLI
Version1.5.0
Previous version1.4.0
Release DateSeptember 24, 2026
  • Strategic telemetry NDJSON (--ndjson) with multi-source Prometheus scrapes (--telemetry YAML, --require-telemetry). Default smoke exporter: Metrum all-smi fork on /metric. Offline analysis recipes in docs/queries/ (no in-binary SQL).
  • Mock server --telemetry-fixture for CI without a GPU.
  • Live e2e requires a pinned Hugging Face metrum-ai/prompt-library mix (no synthetic prompt fallback).

Metrum AI Bench CLI 1.4.0​

ProductMetrum AI Bench CLI
Version1.4.0
Previous version1.3.1
Release DateSeptember 24, 2026
  • Removed Comparison page and vs-other-tools bake-off artifacts. compare remains for diffing two of our own strategic summaries.
  • Breaking under --require-sut: incomplete SUT declarations (missing gpu, driver, runtime, or host_os) are rejected before any request.
  • Strategic: phased warmup, --ignore-eos / --min-tokens / --extra-body-json, and publication metadata on stdout JSON.

Metrum AI Bench CLI 1.3.1​

ProductMetrum AI Bench CLI
Version1.3.1
Previous version1.3.0
Release DateSeptember 23, 2026
  • Fixed --config sample against Hub object-shaped sample-index.json (source_lines array).
  • Performance methodology: Workload section (prompt source, ISL/OSL, profiles, warmup, client headroom); connect_s documented as recorded.
  • Expanded on-host LLM vs AIPerf bake-off report.

Metrum AI Bench CLI 1.3.0​

ProductMetrum AI Bench CLI
Version1.3.0
Previous version1.2.0
Release DateSeptember 23, 2026

Removed​

  • Homebrew packaging and release tap publish (packaging/homebrew/, scripts/render_homebrew_formula.sh, and the homebrew release job). Install from GitHub Release tarballs, crates.io, or source.
  • Deprecated shim binaries: metrumbench-* and pre-1.2.0 metrum-ai-bench* names. Only metrum-ai-bench-cli* binaries remain.

Metrum AI Bench CLI 1.2.0​

ProductMetrum AI Bench CLI
Version1.2.0
Previous version1.1.3
Release DateSeptember 23, 2026

Crate, binaries, and release tarball prefix are now metrum-ai-bench-cli. Install with cargo install metrum-ai-bench-cli. (Through 1.2.x, old metrum-ai-bench* binary names remained as deprecated shims; those shims are removed in 1.3.0.) The crates.io crate metrum-ai-bench (1.1.2 / 1.1.3) is yanked.

Metrum AI Bench CLI 1.1.3​

ProductMetrum AI Bench CLI
Version1.1.3
Previous version1.1.2
Release DateSeptember 23, 2026

First-run onboarding patch: release archives and docs-site quickstart now match what the README already told tarball users.

Added​

  • Release archives include examples/ and test-data/ so the README quickstart paths resolve after unpack.
  • Docs quickstart covers tarball bin/dummy-model-server, checksum / Sigstore verify, and unpack + PATH commands.
  • Agent-driven benchmarking guide (Claude Code / Codex / OpenCode) with a sample prompt.
  • metrum-ai-bench-cli selftest prints a final selftest: ok line (exit 0 = success).

Changed​

  • Deprecated metrumbench-* aliases remained in the archive through 1.2.x (stderr removal notice; removed in 1.3.0). Prefer metrum-ai-bench-cli-*.
  • GitHub Release assets no longer attach an orphan formula file; Homebrew tap publish was gated separately through 1.2.x and is removed in 1.3.0.
  • Comparison docs no longer claim multi-modality as the differentiator over AIPerf.
  • Limitations "No agent mode" clarifies workloads under test vs driving the CLI from a coding agent.

Metrum AI Bench CLI 1.1.2​

ProductMetrum AI Bench CLI
Version1.1.2
Previous version1.1.1
Release DateSeptember 22, 2026

GitHub Release tarballs include trademark and policy docs under docs/. Release verify requires the release tag commit to be an ancestor of main. Naming policy drops the short form Bench CLI; CodeQL no longer soft-fails.

Metrum AI Bench CLI 1.1.1​

ProductMetrum AI Bench CLI
Version1.1.1
Previous version1.1.0
Release DateSeptember 22, 2026

Approved trademark, naming, claims-ledger, and results-publication policies are restored to the public repository. Smoke-result documentation states the measured package and hardware without extra provenance warnings.

Policy and publication (Added)​

  • Restored the counsel-approved trademark, naming, claims-ledger, and results-publication policies (approved September 21, 2026).
  • Restored the customer-facing results-publication guide and linked it from the user guide, output schema, and documentation home page.

Documentation (Changed)​

  • Comparison guidance pins GuideLLM 0.7 (vllm-project) and states that the landscape review is documentation research, not a side-by-side benchmark.
  • Smoke-results documentation keeps the measured 1.0.0-rc.1 package and NVIDIA hardware provenance without repeated non-citable warnings.

Metrum AI Bench CLI 1.1.0​

ProductMetrum AI Bench CLI
Version1.1.0
Previous version1.0.1
Release DateSeptember 19, 2026

Release archives now include the local dummy server, so a Go toolchain is not required to run the quickstart against an unpacked tarball.

Dummy server in release archives (Added)​

  • Each metrum-ai-bench-cli archive for Linux and macOS (x86_64 and aarch64) includes a static bin/dummy-model-server.
  • Linux and macOS release smoke checks cover that binary.
  • The Go server is still not published to crates.io.

Metrum AI Bench CLI 1.0.1​

ProductMetrum AI Bench CLI
Version1.0.1
Previous version1.0.0
Release DateSeptember 19, 2026

Operational release focused on self-hosted CI runner hygiene.

CI self-hosted bench-cli lane (Added / Changed)​

  • Added a dedicated self-hosted CI job for cargo test --all-targets --all-features on label set self-hosted, linux, x64, bench-cli.
  • The job skips fork pull requests so untrusted code does not run on the org runner host.
  • Per-ref concurrency is serialized for this lane to avoid overlap on the same branch head.

Runner operations package (Added)​

  • Added a compose-based runner pool package under deploy/github-runners-bench-cli/, including bootstrap, env template, and operator scripts for manual start, stop, restart, logs, and optional systemd boot wiring.

Metrum AI Bench CLI 1.0.0​

ProductMetrum AI Bench CLI
Version1.0.0
Previous version1.0.0-rc.6
Release DateSeptember 18, 2026

First stable release. This tag was re-cut after a history sanitization rewrite; download the current GitHub Release assets and verify the regenerated Sigstore bundles rather than any earlier v1.0.0 archive.

Prompt library (Fixed)​

  • Hub checksum verification is scoped to the requested config so full and sample no longer collide on train-00000.parquet.
  • Exact ISL/OSL target buckets select unique rows first, so --count-slack 0 and --no-repeats work when the bucket is fully populated.

CLI and release docs (Fixed)​

  • metrum-ai-bench-cli-prompts and metrum-ai-bench-cli-strategic expose -V / --version; strategic also has --version-only.
  • Release verification examples use the v-prefixed archive names that ships attach.

Measurement and honesty (Added / Changed)​

  • Shared streaming consumer for LLM and VLM; strategic chat supports opt-in --streaming with per-turn TTFT.
  • Known limitations document gateways that synthesize SSE from unary upstream calls (total latency reported as TTFT).
  • Product name aligned to Metrum AI Bench CLI across README, crate description, and customer docs.

Operator docs (Added / Changed)​

  • Quiet / NO_BANNER controls, mutually required --url / --api-key, and agent-oriented README / CLI help updates from the rc.6 follow-ups.
  • Publication guidance centers on --sut / --require-sut without pointing at counsel-pending draft policy files.

Metrum AI Bench CLI 1.0.0-rc.6​

ProductMetrum AI Bench CLI
Version1.0.0-rc.6
Previous version1.0.0-rc.5
Release DateSeptember 17, 2026

Release candidate: prompt-library mix extractor and a rand soundness bump.

Prompt library extractor (New)​

  • metrum-ai-bench-cli-prompts (also metrum-ai-bench-cli prompts -- …) selects a reproducible ISL/OSL mix from metrum-ai/prompt-library by mean or median within absolute tolerances.
  • Preferred --count may vary within --count-slack; source rows may repeat when needed to land both axes.
  • Writes JSONL for metrum-ai-bench-cli-llm plus a selection report with recommended --num-requests / --max-tokens.

Dependencies (Changed)​

  • Direct dependency rand 0.9.5 -> 0.10.2 (soundness fixes). Transitive tokenizers still uses rand 0.9.x.

Metrum AI Bench CLI 1.0.0-rc.5​

ProductMetrum AI Bench CLI
Version1.0.0-rc.5
Previous version1.0.0-rc.4
Release DateSeptember 16, 2026

Release candidate: naming alignment, publication SUT block, and policy documents.

Publication and privacy (New)​

  • --sut <PATH> embeds an operator-declared system-under-test block into the summary as sut (provenance: "declared").
  • --require-sut refuses to run without a valid SUT block; implies --redact-hostname.
  • --redact-hostname writes environment.hostname: null.
  • Example templates under examples/sut.example.{json,yaml}.

Naming (Changed)​

  • Crate and binaries aligned on metrum-ai-bench-cli (mock server renamed likewise). MLPerf export SUT name label updated to "Metrum AI Bench".
  • Summary schema v3: optional sut; nullable environment.hostname.

Docs and policy (New)​

  • Results publication policy, trademarks, naming, claims ledger, and known limitations documents added in the repository.

Metrum AI Bench CLI 1.0.0-rc.4​

ProductMetrum AI Bench CLI
Version1.0.0-rc.4
Previous version1.0.0-rc.3
Release DateSeptember 16, 2026

Release hygiene after retracting a mistagged non-prerelease v1.0.0, plus dependency and Actions maintenance. 1.0.0 remains reserved for eventual GA.

Metrum AI Bench CLI 1.0.0-rc.3​

ProductMetrum AI Bench CLI
Version1.0.0-rc.3
Previous version1.0.0-rc.2
Release DateSeptember 16, 2026

Cross-platform release binaries via cargo-zigbuild for Linux glibc and macOS Darwin targets. Compile-free smoke jobs unpack each archive and run metrum-ai-bench-cli --help / selftest before publish.

Earlier 1.0 release candidates​

See the repository CHANGELOG for 1.0.0-rc.2 and prior entries, including measurement residual fixes and OSS-readiness documentation.