Skip to main content

Release Notes

Customer-facing highlights for recent releases. Full engineering detail lives in the repository CHANGELOG.md.

Metrum AI Bench CLI 1.1.0​

ProductMetrum AI Bench CLI
Version1.1.0
Previous version1.0.1
Release DateSeptember 19, 2026

Release archives now include the local dummy server, so a Go toolchain is not required to run the quickstart against an unpacked tarball.

Dummy server in release archives (Added)​

  • Each metrum-ai-bench archive for Linux and macOS (x86_64 and aarch64) includes a static bin/dummy-model-server.
  • Linux and macOS release smoke checks, and the Homebrew formula test, cover that binary.
  • The Go server is still not published to crates.io.

Metrum AI Bench CLI 1.0.1​

ProductMetrum AI Bench CLI
Version1.0.1
Previous version1.0.0
Release DateSeptember 19, 2026

Operational release focused on self-hosted CI runner hygiene.

CI self-hosted bench-cli lane (Added / Changed)​

  • Added a dedicated self-hosted CI job for cargo test --all-targets --all-features on label set self-hosted, linux, x64, bench-cli.
  • The job skips fork pull requests so untrusted code does not run on the org runner host.
  • Per-ref concurrency is serialized for this lane to avoid overlap on the same branch head.

Runner operations package (Added)​

  • Added a compose-based runner pool package under deploy/github-runners-bench-cli/, including bootstrap, env template, and operator scripts for manual start, stop, restart, logs, and optional systemd boot wiring.

Metrum AI Bench CLI 1.0.0​

ProductMetrum AI Bench CLI
Version1.0.0
Previous version1.0.0-rc.6
Release DateSeptember 18, 2026

First stable release. This tag was re-cut after a history sanitization rewrite; download the current GitHub Release assets and verify the regenerated Sigstore bundles rather than any earlier v1.0.0 archive.

Prompt library (Fixed)​

  • Hub checksum verification is scoped to the requested config so full and sample no longer collide on train-00000.parquet.
  • Exact ISL/OSL target buckets select unique rows first, so --count-slack 0 and --no-repeats work when the bucket is fully populated.

CLI and release docs (Fixed)​

  • metrum-ai-bench-prompts and metrum-ai-bench-strategic expose -V / --version; strategic also has --version-only.
  • Release verification examples use the v-prefixed archive names that ships attach.

Measurement and honesty (Added / Changed)​

  • Shared streaming consumer for LLM and VLM; strategic chat supports opt-in --streaming with per-turn TTFT.
  • Known limitations document gateways that synthesize SSE from unary upstream calls (total latency reported as TTFT).
  • Product name aligned to Metrum AI Bench CLI across README, crate description, and customer docs.

Operator docs (Added / Changed)​

  • Quiet / NO_BANNER controls, mutually required --url / --api-key, and agent-oriented README / CLI help updates from the rc.6 follow-ups.
  • Publication guidance centers on --sut / --require-sut without pointing at counsel-pending draft policy files.

Metrum AI Bench CLI 1.0.0-rc.6​

ProductMetrum AI Bench CLI
Version1.0.0-rc.6
Previous version1.0.0-rc.5
Release DateSeptember 17, 2026

Release candidate: prompt-library mix extractor and a rand soundness bump.

Prompt library extractor (New)​

  • metrum-ai-bench-prompts (also metrum-ai-bench prompts -- …) selects a reproducible ISL/OSL mix from metrum-ai/prompt-library by mean or median within absolute tolerances.
  • Preferred --count may vary within --count-slack; source rows may repeat when needed to land both axes.
  • Writes JSONL for metrum-ai-bench-llm plus a selection report with recommended --num-requests / --max-tokens.

Dependencies (Changed)​

  • Direct dependency rand 0.9.5 -> 0.10.2 (soundness fixes). Transitive tokenizers still uses rand 0.9.x.

Metrum AI Bench CLI 1.0.0-rc.5​

ProductMetrum AI Bench CLI
Version1.0.0-rc.5
Previous version1.0.0-rc.4
Release DateSeptember 16, 2026

Release candidate: naming alignment, publication SUT block, and policy documents.

Publication and privacy (New)​

  • --sut <PATH> embeds an operator-declared system-under-test block into the summary as sut (provenance: "declared").
  • --require-sut refuses to run without a valid SUT block; implies --redact-hostname.
  • --redact-hostname writes environment.hostname: null.
  • Example templates under examples/sut.example.{json,yaml}.

Naming (Changed)​

  • Crate and binaries aligned on metrum-ai-bench (mock server renamed likewise). MLPerf export SUT name label updated to "Metrum AI Bench".
  • Summary schema v3: optional sut; nullable environment.hostname.

Docs and policy (New)​

  • Results publication policy, trademarks, naming, claims ledger, and known limitations documents added in the repository.

Metrum AI Bench CLI 1.0.0-rc.4​

ProductMetrum AI Bench CLI
Version1.0.0-rc.4
Previous version1.0.0-rc.3
Release DateSeptember 16, 2026

Release hygiene after retracting a mistagged non-prerelease v1.0.0, plus dependency and Actions maintenance. 1.0.0 remains reserved for eventual GA.

Metrum AI Bench CLI 1.0.0-rc.3​

ProductMetrum AI Bench CLI
Version1.0.0-rc.3
Previous version1.0.0-rc.2
Release DateSeptember 16, 2026

Cross-platform release binaries via cargo-zigbuild for Linux glibc and macOS Darwin targets. Compile-free smoke jobs unpack each archive and run metrum-ai-bench --help / selftest before publish.

Earlier 1.0 release candidates​

See the repository CHANGELOG for 1.0.0-rc.2 and prior entries, including measurement residual fixes and OSS-readiness documentation.