Skip to main content

Platforms

Metrum AI Bench CLI is a client. It runs on your measurement host and talks to whatever OpenAI-compatible endpoint you point it at. Server hardware support is therefore “whatever your serving stack exposes,” not a Bench-managed accelerator matrix.

Client install targets​

PathPlatforms
GitHub Release binariesLinux x86_64 / aarch64 *-unknown-linux-gnu (glibc 2.17 floor); macOS Darwin x86_64 / aarch64
From sourceRust 1.89+ on a host that can build the crate
crates.iocargo install metrum-ai-bench-cli (when published)

Not shipped as release targets:

  • musl / Alpine-style static Linux archives
  • OpenSSL-linked builds (TLS is rustls)
  • Windows - no release archives. Building from source on Windows may work for development but is not a supported or smoke-tested install path for 1.0.

Verify release archives with the published .sha256 and optional Sigstore bundle before unpacking.

Optional Cargo features​

FeatureIn GitHub Release binaries?Purpose
otlpYes (--features otlp)Strategic OpenTelemetry export (--otlp-endpoint)
tokenizerNoLocal --tokenizer counts; build from source with --features tokenizer

Build and test prerequisites​

ToolWhen needed
Rust 1.89+Source builds and cargo test
Go 1.26.10+ (matches dummy-model-server/go.mod and CI)Dummy model server and its tests
Network / Hugging Face HubPrompt-library extracts (unless using --local-jsonl / --local-parquet / --offline)

Serving stacks (Docker first)​

Bench measures an OpenAI-compatible HTTP endpoint. Bring up vLLM or SGLang with the vendor container images. Installing server wheels with pip on a stock cloud GPU image commonly fails on JIT kernel compilation (missing g++, mismatched CUDA headers, missing prebuilt kernels for new SM versions). That path burned hours in field trials and can make a working GPU look broken.

Preferred: vendor images​

# vLLM (example). Map host 8000 → container 8000; adjust model and GPUs.
docker run --gpus all --shm-size=32g -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-7B-Instruct --host 0.0.0.0 --port 8000

# SGLang (example).
docker run --gpus all --shm-size=32g -p 8000:8000 \
  lmsysorg/sglang \
  python3 -m sglang.launch_server \
  --model-path Qwen/Qwen2.5-7B-Instruct --host 0.0.0.0 --port 8000

Run Metrum AI Bench CLI on the same host (loopback URL). Driving a public cloud IP from a laptop puts WAN RTT into TTFT and can invert close comparisons.

Before a long GPU run, probe the OpenAI-compatible endpoint from the client:

metrum-ai-bench-cli preflight \
  --url http://127.0.0.1:8000/v1/chat/completions \
  --api-key dummy --model YOUR_MODEL

Preflight checks reachability, a unary chat probe, streaming first-token smoke, and a short latency sample. Benchmark TTFT defaults to the first reasoning or visible delta; visible-answer timing is separate. A reasoning-only length finish is reasoning_truncated, unanswered but non-failing by default; strict mode and --ci-safety reject it. Other completed no-answer streams are no_output_token errors. --ttft-fallback first-token is a deprecated redundant alias and --allow-missing-ttft relaxes the audit; --infer-ttft-from-first-byte is a labelled HTTP header approximation. Current request/summary output is v5; compatible consumers can read historical v4 and v3 logs. For system telemetry, strongly recommend the Metrum AI all-smi fork. It cannot see CUDA toolchains or GPU inventory on a remote host; when the serve path is wrong, prefer the vendor containers above.

Checked-in serving data is for CI smoke defaults only; for a live model, record its researched image and launch arguments in the SUT. The live-serving and performance-validation helpers create the Hugging Face bind-mount source as the invoking user before Docker starts. They do not repair an existing unwritable cache: correct its ownership or permissions before starting.

Advanced: pip on bare metal​

Only when you control the CUDA toolkit, compiler, and wheel/SM matrix. Expect to install a matching CUDA toolkit, g++, and often still hit FlashInfer / DeepGEMM JIT failures on newer architectures. Prefer Docker when comparing hardware.

Hardware / server caveats​

AreaStatusNotes
NVIDIA datacenter (Hopper H100 and similar) with vendor DockerValidated in live smoke pathsUse official vLLM / SGLang images
NVIDIA RTX PRO 6000 Blackwell (sm_120) with vendor DockerWorks with caveatsQuantized / FP4 paths need images that ship prebuilt kernels; bare-metal pip + JIT is known-blocked for many users
Bare-metal pip + CUDA 12.8 / 13 + FlashInfer on BlackwellKnown-blocked / fragileHeader and SM detection issues; do not treat as a Bench defect
AMD InstinctIn progressSmoke matrix in-repo is NVIDIA-first today

SGLang and sliding-window models​

SGLang pre-allocates a sliding-window attention (SWA) pool sized by --max-running-requests. On some sliding-window models, high values refuse to start or collapse usable KV capacity. In field trials, concurrency 128 was a practical ceiling where 256/512 failed on H100 for a SWA model, while vLLM admitted higher concurrency on the same card. When comparing vLLM vs SGLang, record --max-running-requests (and any SWA-related flags) in the SUT runtime.config field so the matrix is not silently mis-specified.

Smoke and publication matrix​

Smoke results in the repository (docs/SMOKE_RESULTS.md, package 1.0.0-rc.1) are NVIDIA-only. AMD Instinct coverage is in progress.

Fleet-wide comparison, history, and attestation are Platform concerns, not this OSS client. One Bench process drives one endpoint or one configured pool.

Security notes for shared hosts​

  • Do not put credentials in endpoint files committed to source control.
  • Endpoint URL userinfo and api_key / token query values are redacted from per-request diagnostic paths; still avoid sharing secrets in configuration.
  • The supplied self-hosted CI runner image runs as a non-root runner user with a runner-listener health check.
  • Run summaries may record client environment.hostname. Use --redact-hostname (or --require-sut, which implies redaction) when publishing or sharing logs.