Metrum AI Bench CLI Documentation
Metrum AI Bench CLI is Apache-2.0 licensed load and performance measurement for OpenAI-compatible LLM, VLM, ASR, and image-generation endpoints. One client process measures one environment and writes a result with a manifest.
Metrum AI Bench Platform (commercial) remembers, compares, governs, and attests. This documentation covers the open-source CLI only.
Current release: v1.8.0.
Getting started
New to Metrum AI Bench CLI? Start here.
- Quickstart: install the CLI and complete a local dummy-server run in a few minutes.
Learn the product
- User Guide: concepts, run shapes, and how to read outputs.
- Modalities: LLM, VLM, ASR, and imagegen workflows.
- Strategic benchmarking: concurrency and rate sweeps, validity checks, and exports.
- System telemetry: strongly recommended Metrum AI all-smi system measurements alongside request timing.
- Prompt library: select ISL/OSL mixes from
metrum-ai/prompt-library. - Results publication: approved policy for naming results and required SUT metadata.
- Platforms: supported client platforms and install paths.
Understand the numbers
- Performance Methodology: how every metric is measured.
Reference
- Feature Reference: tools and configuration lookup.
- CLI Reference: flags and entry points.
- Output schema: JSONL request and summary shapes.
- Known limitations: current scope and caveats.
- Release Notes: what is new and changed.