Skip to main content

Results publication

This page summarizes the rules for naming results produced with Metrum AI Bench CLI. For the full approved policy, see docs/RESULTS_PUBLICATION_POLICY.md.

Ownership and endorsement​

Harness results belong to the publisher of those results. Running or citing Metrum AI Bench CLI does not constitute endorsement by Metrum AI, Inc. unless Metrum AI is separately engaged in writing for that purpose.

Manifest requirement​

A published result that names Metrum AI Bench CLI (including "Measured with Metrum AI Bench CLI") must include an unmodified run summary that contains at least:

  • Tool version
  • Workload identification
  • Client environment
  • SUT block with publication inventory: gpu.model, gpu.count (>0), driver_version, runtime.name, runtime.version, runtime.config, and host_os
  • Full configuration used for the run

Produce a compliant run with:

metrum-ai-bench-cli llm -- \
  ... \
  --sut sut.json \
  --require-sut

--require-sut refuses incomplete SUT declarations before any request is sent.

A claim that omits this manifest, including the SUT block, is not a Metrum AI Bench CLI result under the publication policy, even if the software was used.

--sut embeds an operator-declared system-under-test block (JSON or YAML) into the summary as sut, labelled provenance: "declared". An absent SUT yields "sut": null plus a stderr notice. --require-sut (or METRUM_AI_BENCH_REQUIRE_SUT=1) refuses to run without a complete block and implies --redact-hostname.

Example templates are available as examples/sut.example.json and examples/sut.example.yaml in the repository.

Declared vs observed​

Where a field is supplied by declaration rather than measured by the harness, it must be labeled accordingly (for example, provenance: "declared"). Do not present declared values as observed measurements.

Metrum AI Bench CLI does not probe the remote host to confirm GPU SKU, driver, or engine version. A mismatched declaration is a policy or process failure, not something the client detects.

Trademark and vendor rules​

Trademark use in connection with published results follows TRADEMARKS.md.

Vendors and partners may not require suppression or delay of third-party results obtained with Metrum AI Bench CLI as a condition of access, partnership, or support.

Studies published by Metrum AI must meet the same manifest requirements as third-party publications.

Reporting incorrect results​

If you believe a published result misuses the Metrum AI Bench CLI name, omits a required manifest, or is otherwise incorrect under this policy, report it to opensource@metrum.ai with links to the publication and any available run artifacts.