Skip to main content

Known limitations

Current scope and limitations for Metrum AI Bench CLI. See also Performance Methodology.

Client-side only​

Bench is a load-generation client. It measures what the client observes (send, first byte, first visible token, body complete, errors). It cannot observe GPU utilization, KV-cache state, or scheduler internals on the server except when you optionally scrape Prometheus endpoints with --telemetry YAML into --ndjson on strategic, llm, vlm, asr, or imagegen (strategic also accepts legacy --metrics-url). For real-device runs, strongly recommend the Metrum AI all-smi fork to collect accelerator power, utilization and memory on the request clock; verify the release checksum and record its tag.

Thinking streams and TTFT​

TTFT defaults to the first reasoning or visible output delta, not HTTP headers. A reasoning-only length finish is reasoning_truncated: non-failing by default but unanswered. Strict mode and --ci-safety reject it; other no-answer streams are errors. --ttft-fallback first-token is a deprecated redundant alias and --allow-missing-ttft relaxes only the audit. First-byte inference is an HTTP header approximation, not a token. Only choice index 0 is measured. Current request/summary output uses v5; v4 TTFT has different semantics. See Output schema.

Gateways that synthesize streaming​

Gateways that synthesize SSE from unary upstream calls report total latency as TTFT. This behavior is undetectable client-side.

SUT is declared (or locally probed), not remotely verified​

--sut / --require-sut (when present) embed a system-under-test block into the run summary for publication. metrum-ai-bench-cli sut init writes a template; sut init --probe may fill local GPU/OS/CPU/memory and mark those fields in field_provenance as observed. Probing never SSHs to the remote serving host and does not prove the endpoint matches the declaration. --require-sut rejects incomplete declarations (missing GPU, driver, runtime, or host OS) before any request is sent. Values remain declared or locally probed, not remotely verified.

Performance, not quality (except ASR)​

Headline outputs are latency, throughput, goodput under SLOs, and related distributions. The only built-in quality scores are ASR WER/CER (with --normalizer). There is no general task accuracy, judge score, or agent success metric.

No agent mode​

Multi-turn / tool / schema paths on strategic are request validity helpers, not an agent evaluation harness. That means Bench does not score agent workloads under test. It does not forbid driving the CLI from a coding agent (Claude Code, Codex, OpenCode, and similar); see Agent-driven benchmarking.

No cost per accepted task​

There is no built-in cost per accepted task or general task-quality score. The strategic runner can optionally report cost per million output tokens when an hourly price and output-token throughput are supplied; this is not a cost per accepted task.

Open-loop and closed-loop​

Both modes exist: omit --request-rate for closed-loop concurrency; set --request-rate (and optional --arrival poisson) for open-loop. Open-loop coordinated-omission latency is the headline field when scheduling is open-loop. Do not compare open-loop and closed-loop numbers as if they were the same experiment.

Pooled multi-endpoint = mixture​

When requests are distributed across endpoints, aggregate distributions are labeled pooled_mixture. Use per_endpoint for diagnosis; do not treat the pool as a single homogeneous replica.

Percentile unreliability at small n​

p99 is marked p99_unreliable when fewer than 100 samples exist. Small-n p95/p90 also carry p95_unreliable / p90_unreliable flags. Treat tail percentiles accordingly at small n.

The dummy server proves plumbing, not media correctness​

dummy-model-server does not run a model. By default it accepts any bytes as audio and charges a flat 256 prompt tokens per image_url part without looking at it, so a passing run against it shows that requests, timing, and records work, not that a real server would accept the payload.

With -strict-media (the Rust e2e harness default) it rejects with HTTP 400:

  • chat image_url parts with a data: URL that is not base64, does not decode as PNG, JPEG, GIF, or WebP, or is smaller than 2x2 (BMP and TIFF, which vLLM accepts through PIL, are rejected here too);
  • transcription uploads under 1024 bytes, with an unknown container, a malformed WAV fmt /data chunk or MPEG frame header, or an all-zero body (message Invalid or unsupported audio file, as vLLM sends).

Strict mode still does not fetch http(s) image URLs, decode pixels or audio samples, check that audio is speech, or validate image-generation output. Its transcript is the constant dummy transcription, so WER against the dummy is meaningless. Live verification per modality is tracked in the repository claims ledger (docs/CLAIMS_LEDGER.md) and produced by .github/workflows/live-modality-smoke.yml.

Single node / single client process​

One Bench process drives one endpoint or one configured pool. Fleet-wide comparison, history, and attestation are Platform concerns, not this OSS client.

Published smoke matrix is NVIDIA-only​

AMD Instinct coverage is in progress. No comparative vendor results are published in the repository smoke matrix.

MLPerf export is unofficial​

--mlperf-dir writes parser-oriented LoadGen-shaped files marked UNOFFICIAL and never emits bare Result is : VALID. It is not a submission.

Partial runs​

Interrupted runs can write partial: true. Do not treat a partial summary as a complete campaign cell without checking that flag and sample counts. A hard kill may leave valid request lines without a summary.

Prompt-library mixes vs llm scheduling​

metrum-ai-bench-cli-prompts can solve for ISL/OSL mean or median within tolerances, including by repeating rows or leaving the preferred --count. metrum-ai-bench-cli-llm still shuffles --prompts and cycles with modulo indexing under a global --max-tokens. The selected mix is preserved only when --num-requests equals the extractor's selected_count and --warmup-requests is 0. Warmup or a mismatched request count changes the measured mix. Repeats in the JSONL are solver output for length statistics, not a claim about prompt diversity or answer quality.

Optional features and platforms​

GitHub Release binaries include otlp but not tokenizer. Windows has no release archives. See Platforms.