Skip to main content

Quickstart

This guide takes you from an empty machine to a completed LLM measurement against the in-repo dummy model server. Follow each step in order. Once the toolchain is installed, the loop usually takes under ten minutes.

What you'll do​

  1. Install a release binary or build from source
  2. Start the dummy OpenAI-compatible server
  3. Write a tiny JSONL prompt file
  4. Run metrum-ai-bench-cli-llm (or the unified metrum-ai-bench-cli llm entry)
  5. Inspect the JSONL data log and end-of-run summary

For real-device runs, we strongly recommend the Metrum AI all-smi fork for power, utilization, memory and supported deep GPM metrics alongside request timing. See System telemetry for verified installation and the Bench scrape preset. The dummy-server quickstart does not require an accelerator or all-smi.

If you already have a binary and only need a smoke check:

metrum-ai-bench-cli selftest

Before you start​

You'll need:

  • A Linux (x86_64 or aarch64 glibc) or macOS Darwin host for release binaries, or Rust 1.89+ to build from source. Windows has no release binaries (see Platforms).
  • Go 1.26.10+ only if you start the dummy model server from a source clone with go run (matches dummy-model-server/go.mod and CI). Release tarballs already ship bin/dummy-model-server and need no Go.
  • Network access only if you download a GitHub Release or fetch the prompt library later. The dummy-server loop needs no API key beyond the literal value dummy.

1. Install​

Option A: GitHub Release (preferred for binaries)​

  1. Open Releases and download the archive for your target (replace VERSION and TARGET below).
  2. Verify the checksum (and Sigstore bundle when present):
# SHA-256 of the tarball (file next to the archive on the release)
shasum -a 256 -c metrum-ai-bench-cli-vVERSION-TARGET.tar.gz.sha256

# Optional: Sigstore keyless verify (requires cosign)
cosign verify-blob \
  --bundle metrum-ai-bench-cli-vVERSION-TARGET.tar.gz.sigstore.json \
  --certificate-identity-regexp 'https://github.com/metrum-ai/bench-cli/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  metrum-ai-bench-cli-vVERSION-TARGET.tar.gz

Each target also publishes a CycloneDX SBOM as metrum-ai-bench-cli-TARGET.cdx.json on the same release.

  1. Unpack so the binaries are on your PATH. The archive has a versioned root with bin/ inside it; do not strip that layout into ~/.local/bin directly:
mkdir -p ~/.local/opt ~/.local/bin
tar -xzf metrum-ai-bench-cli-vVERSION-TARGET.tar.gz -C ~/.local/opt
ln -sf ~/.local/opt/metrum-ai-bench-cli-vVERSION-TARGET/bin/* ~/.local/bin/

Release archives are cross-built with cargo-zigbuild for:

TargetNotes
Linux *-unknown-linux-gnuglibc 2.17 floor; not musl
macOS Darwinx86_64 / aarch64

TLS uses rustls (no OpenSSL link). Install from a GitHub Release tarball or build from source; there is no Homebrew install path.

Option B: Build from source​

git clone https://github.com/metrum-ai/bench-cli.git
cd bench-cli
cargo build --release

Binaries land under target/release/. Run tests when you want a local gate:

cargo test --all-targets --all-features

Release archives include the otlp feature but not tokenizer. See Platforms for the feature matrix.

2. Start the dummy server​

From a GitHub Release tarball (no Go required; second terminal is fine):

bin/dummy-model-server -port 18321 -latency 100ms -chunk-interval 20ms

If you installed under ~/.local/opt/.../bin and that directory is on PATH, the same binary name works without the bin/ prefix:

dummy-model-server -port 18321 -latency 100ms -chunk-interval 20ms

From a source clone (needs Go 1.26.10+), from the repository root:

go run ./dummy-model-server/cmd/dummy-model-server \
  -port 18321 -latency 100ms -chunk-interval 20ms

Leave it running. The server speaks OpenAI-compatible paths used by the modality tools. Use --api-key dummy on the client.

This is the Go dummy model server (modality compatibility). For strategic sweeps, embeddings, and rerank fixtures, use the separate Rust metrum-ai-bench-cli-mock-server instead - see Strategic benchmarking.

3. Create a prompt file​

Prompt and input files are JSONL only (.csv is rejected).

printf '%s\n' '{"prompt":"Hi"}' > /tmp/metrum-prompts.jsonl

4. Run the LLM client​

metrum-ai-bench-cli-llm \
  --url http://127.0.0.1:18321/v1/chat/completions \
  --api-key dummy \
  --scenario reference \
  --num-requests 16 \
  --concurrency 4 \
  --prompts /tmp/metrum-prompts.jsonl \
  --mode chat \
  --streaming \
  --model dummy \
  --max-tokens 20 \
  --seed 7 \
  --warmup-requests 0 \
  --data-log /tmp/metrum-reference.jsonl \
  --log-level error

Or through the unified entry point (forward modality flags after --):

metrum-ai-bench-cli llm -- \
  --url http://127.0.0.1:18321/v1/chat/completions \
  --api-key dummy \
  --scenario reference \
  --num-requests 16 \
  --concurrency 4 \
  --prompts /tmp/metrum-prompts.jsonl \
  --mode chat \
  --streaming \
  --model dummy \
  --max-tokens 20 \
  --seed 7 \
  --warmup-requests 0 \
  --data-log /tmp/metrum-reference.jsonl \
  --log-level error

During the v1.x compatibility period the four modality binaries can also be invoked directly. Prefer the metrum-ai-bench-cli* names; older shim aliases were removed in 1.3.0.

What success looks like​

  • Request records stream into --data-log as each request finishes.
  • A summary.v5 object is appended when the run completes.
  • Console statistics match the same RunSummary / DistSummary values written to JSONL.

Against this dummy configuration on an unloaded machine, the checked-in reproduction guide expects roughly:

MetricExpected band
Mean TTFT0.100–0.200 s
Mean E2E latency0.420–0.650 s
Mean ITL0.010–0.040 s

Those bands surround the dummy server's timing model for -latency 100ms -chunk-interval 20ms with --max-tokens 20: TTFT ~ 120 ms (latency + first chunk) and response time ~ 500 ms (100 + 20*20). See dummy-model-server/README.md in the repository. E2E scales with --max-tokens; a README quickstart that uses 64 is healthy but will not land in this table.

Scheduling and counts should match test-data/reference-result.json when you follow reproducing the reference exactly. Wall-clock timestamps, UUIDs, hostname, and exact timings differ by design.

5. Point a real GPU server at vendor images​

The dummy loop above does not need a GPU. When you measure vLLM or SGLang on a GPU host, bring the server up with the vendor container images documented in Platforms. Installing those stacks with pip on a stock cloud image commonly fails on JIT kernel compilation (FlashInfer and friends) and can look like a Bench defect when the serve path is wrong.

Run the CLI on the same host (loopback URL) so WAN RTT does not land in TTFT. Use metrum-ai-bench-cli preflight before a long sweep.

6. Read the result​

Every completed request is flushed incrementally to JSONL. Warmup records remain auditable but are excluded from measured distributions. A graceful Ctrl-C stops issuance, drains already-started work, and emits partial: true on the summary.

Next steps: