Metrum AI Bench CLI Documentation

Metrum AI Bench CLI is Apache-2.0 licensed load and performance measurement for OpenAI-compatible LLM, VLM, ASR, and image-generation endpoints. One client process measures one environment and writes a result with a manifest.

Metrum AI Bench Platform (commercial) remembers, compares, governs, and attests. This documentation covers the open-source CLI only.

Current release: 1.1.0.

Getting started

New to Metrum AI Bench CLI? Start here.

• Quickstart: install the CLI and complete a local dummy-server run in a few minutes.

Learn the product

• User Guide: concepts, run shapes, and how to read outputs.
• Modalities: LLM, VLM, ASR, and imagegen workflows.
• Strategic benchmarking: concurrency and rate sweeps, validity checks, and exports.
• Prompt library: select ISL/OSL mixes from metrum-ai/prompt-library.
• Publishing-oriented runs: required SUT metadata for published results.
• Platforms: supported client platforms and install paths.

Understand the numbers

• Performance Methodology: how every metric is measured.

Reference

• Feature Reference: tools and configuration lookup.
• CLI Reference: flags and entry points.
• Output schema: JSONL request and summary shapes.
• Comparison: how Bench relates to other measurement tools.
• Known limitations: honest scope for the 1.0 line.
• Release Notes: what is new and changed.