Comparison
This is a scoped comparison of client-side inference measurement tools, compiled from public project pages and documentation as of September 2026. It is not a hands-on bake-off. Metric names only match when tokenizer, prompt sequence, sampling parameters, warmup, measurement window, endpoint topology, and SLO definitions match.
What Bench ships at 1.0
| Capability | Where |
|---|---|
| Four OpenAI-compatible modalities: LLM, VLM, ASR, imagegen | metrum-ai-bench {llm,vlm,asr,imagegen} |
| Open-loop scheduling with seeded Poisson (or constant) arrivals | --request-rate, --arrival, --seed |
| Pooled multi-endpoint distribution | --endpoints-file; aggregates labeled pooled_mixture |
| TTFT = first visible output token (not first-byte headers) | ttft_s; headers are first_byte_s |
| Warmup excluded from measured distributions | --warmup-requests |
| Goodput requires stated SLOs | --slo ttft=|tpot=|e2e=; without SLOs, goodput equals validity-filtered throughput on strategic |
| Partial-run flag | Ctrl-C / SIGTERM drain writes partial: true |
| p99 flagged unreliable at low n | p99_unreliable when n < 100 |
| Hyndman–Fan type 7 percentiles | percentile_method: hyndman_fan_type7 |
| Concurrency/rate sweeps, HTML report, MLPerf-shaped export | strategic: --sweep, --html, --mlperf-dir |
Landscape
| Tool | What it answers | Overlap with Bench | What Bench adds | What it does better than Bench |
|---|---|---|---|---|
| SemiAnalysis InferenceX | How named stacks score on a published, recurring hardware CI matrix | Client-visible latency/throughput language | Local, operator-owned runs with a stamped manifest; four modalities in one client | Continuous public CI and a recognized public scoreboard |
| NVIDIA AIPerf (successor to GenAI-Perf) | How a Triton / NIM / NVIDIA-oriented path behaves under load | Rate, Poisson-style arrivals, tokenizer counts, SLO-style ideas | Multi-modality and pooled multi-endpoint mixture labeling; strategic sweeps/exports | Ecosystem familiarity and telemetry already wired into many NVIDIA pipelines |
| MLPerf Inference | Whether a submission meets audited workload and accuracy rules | Latency/throughput language; Server/Offline names | Informal LoadGen-shaped export only | Audited rules, accuracy requirements, submission standing |
vLLM / SGLang bench_serving | How this engine build behaves on these datasets | Rate/concurrency, seeds, ignore-EOS, local tokenizer counts | Cross-engine OpenAI-compatible client; multi-modality; strategic exports | Already installed next to the engine; deep stack-specific datasets |
| GuideLLM / LLMPerf | Latency/throughput under scripted load | Rate/concurrency sweeps; HTML-style reporting (GuideLLM) | Shared Rust client across four modalities; typed errors; stamped summary.v3 | Packaging / Python ecosystem fit for quick scripts |
| lm-evaluation-harness / HELM / OpenCompass | Quality and task scores | Occasional shared prompt corpora | Bench measures serving performance (plus ASR WER/CER only) | Breadth of quality tasks and academic protocols |
| NVIDIA Dynamo | How to run multi-node inference | You may point Bench at Dynamo-fronted endpoints | Dynamo is a target, not a measurement alternative | End-to-end serving product features |
MLPerf export disclaimer
metrum-ai-bench-strategic --mlperf-dir … writes LoadGen-shaped text for
parser-oriented interoperability. It is not an audited submission. Exports
begin with an UNOFFICIAL disclaimer and refuse the official LoadGen
substring Result is : VALID.
What we do not claim
- Not first, only, or industry-standard.
- A stamped manifest is table stakes for publishable runs, not a moat.
- Not a competitor to MLPerf Inference submissions.
- Not a head-to-head performance bake-off against AIPerf or InferenceX.
- No claim that Bench replaces engine-native
bench_servingfor engine developers who already live in that tree.
Not in 1.0 (roadmap)
Planned for a next release (no dates):
- Agent mode
- Quality metrics beyond ASR WER/CER
- Cost per accepted task
Bench vs Bench Platform
| Metrum AI Bench CLI (this product) | Metrum AI Bench Platform | |
|---|---|---|
| Role | Measure one environment; emit a result with a manifest | Remember, compare, govern, and attest |
| License | Apache-2.0 client | Commercial |
| Docs | These pages cover the OSS CLI | Commercial product; not covered here - contact Metrum AI |