Strategic benchmarking

metrum-ai-bench-cli-strategic is the runner for concurrency/rate sweeps, chat sessions, structured output, embeddings, reranking, optional server-metrics correlation, and portable exports. It is a separate binary (not metrum-ai-bench-cli strategic). Existing modality-specific binaries remain supported for single-cell runs.

Which local server to use

Fixture	Binary	Typical use
Go dummy model server	`go run ./dummy-model-server/cmd/dummy-model-server`	LLM/VLM/ASR/imagegen compatibility (Quickstart uses port 18321)
Rust mock server	`metrum-ai-bench-cli-mock-server`	Deterministic strategic fixtures (health, Prometheus metrics, chat, embeddings, rerank, tools, JSON schema); examples often use 8080

Do not mix them casually: the Go dummy is the deeper modality compatibility stub; the Rust mock is the strategic/embeddings/rerank fixture.

Obtaining the mock server

metrum-ai-bench-cli-mock-server ships in GitHub Release archives alongside the other binaries. From a checkout:

cargo build --release --bin metrum-ai-bench-cli-mock-server
# or, when the crate is published:
cargo install metrum-ai-bench-cli --bin metrum-ai-bench-cli-mock-server

Useful flags: --listen (default 127.0.0.1:8080), --latency-ms, --fail-every N (inject reproducible HTTP 503 responses), --telemetry-fixture (canned DCGM/all-smi/vLLM exposition on /metrics and /metric for CI).

Sweep and server correlation

metrum-ai-bench-cli-mock-server --listen 127.0.0.1:8080 --telemetry-fixture &
metrum-ai-bench-cli-strategic \
  --url http://127.0.0.1:8080/v1/chat/completions \
  --model mock --api-key dummy \
  --sweep 1,2,4,8,16 \
  --sweep-by concurrency \
  --requests-per-stage 100 \
  --max-tokens 64 \
  --warmup-requests 4 \
  --ndjson run.ndjson \
  --telemetry docs/telemetry/examples/all-smi.yaml \
  --metrics-url http://127.0.0.1:8080/metrics \
  --html report.html \
  --csv requests.csv \
  --mlperf-dir mlperf \
  --mlperf-scenario server

Each load stage is measured independently. In v1.8.0 chat and VLM headline TTFT normally uses the first reasoning or visible delta; explicit first-byte inference may provide a labelled HTTP approximation. Visible-answer timing is separate. The token-rate window runs from first measured send to last terminal completion, including failures, and incomplete usage makes the full token rate unavailable. Reasoning-only truncation is unanswered; strict mode and --ci-safety reject it. Compare older results only after aligning their cohorts and metric meanings.

--warmup-requests are issued and fully completed at the start of every stage, then the measurement epoch resets. Warmup rows stay in the CSV (warmup=true) but are excluded from stage n, latency percentiles, throughput, goodput, and knee detection. Measured prompt indexing restarts at zero after warmup. Prefer a warmup count at least as large as stage concurrency on GPU endpoints. --warmup-requests 0 is for mock/determinism only.

For fixed-length throughput studies, chat sweeps accept --ignore-eos and optional --min-tokens with --max-tokens (engine extensions). Stdout JSON adds schema_version, tool_version, environment, and config while keeping points for metrum-ai-bench-cli compare.

To feed a Hugging Face metrum-ai/prompt-library mix, extract JSONL with metrum-ai-bench-cli-prompts / metrum-ai-bench-cli prompts, then pass --prompts and --max-tokens (required together; prefer the report's recommended_max_tokens). Stage config stamps prompts, prompt_pool_size, max_tokens, ignore_eos, min_tokens, and warmup_requests.

The HTML report plots achieved throughput against p95 latency with labeled axes, ticks, and a grid, and marks the unit-normalized Kneedle result. Use metrum-ai-bench-cli compare on two of our own stdout JSON summaries (or request CSVs). Legacy --metrics-url still aggregates engine KV/queue gauges into stdout. For durable GPU/host timeseries, pass --ndjson with --telemetry (Prometheus text). The binary loads that YAML at startup and keeps series whose names match include. Default smoke uses the Metrum all-smi fork on http://127.0.0.1:9090/metrics plus the serving engine's /metrics. Curl those pages before a run; example files are starting points. See repository docs/TELEMETRY.md.

A knee needs at least 5 measured stages (stages with a p95): both endpoints plus 3 interior candidates. Shorter sweeps, or sweeps whose failed stages leave fewer than 5 measured stages, report "knee": null instead of a misleading middle stage. The always-present knee_detection object gives the reason (insufficient_points, missing_latency, flat_curve, or no_bend) and the measured points count. no_bend means p95 rose less than 20% above its running minimum and no stage was saturated (throughput kept scaling with load and the error rate did not rise), so a nearly linear sweep reports no knee. Such a sweep usually has not reached saturation; check the error rate and throughput scaling, then extend the sweep to higher loads. A throughput plateau or a rising error rate still yields a knee (knee_detection.method saturation). The 20% threshold is provisional, set from two live curves. Plan sweeps with 5 or more loads (for example --sweep 1,2,4,8,16,32) when the knee matters. See docs/STRATEGIC_BENCHMARKING.md and docs/OUTPUT_SCHEMA.md.

Use --sweep-by rate --max-in-flight N for open-loop request-rate stages. Rate requests retain their intended schedule while waiting for an in-flight slot.

CSV fields include scheduled and sent timestamps, queue delay, send-to-completion service latency, and scheduled-to-completion latency. Sweep percentiles use Hyndman–Fan type 7 over the last value so overload cannot hide behind coordinated omission. Each sweep point carries n, errors, a full latency_s DistSummary (including reliability flags), redacted stage config, and goodput.

Goodput on strategic

Use repeatable --slo e2e=… so goodput counts valid successes that meet the end-to-end latency threshold, excluding explicitly unanswered rows. Unmarked legacy rows and ASR/imagegen successes are treated as answered for goodput. Without --slo, goodput_equals_throughput is true only when the goodput and completed-response counts match; unanswered or invalid responses can make them differ.

With --streaming, ttft= and tpot= use captured stream timings (ITL is retained on each CSV row; TPOT is (service_latency - ttft) / (output_tokens - 1)). user_tps= is a minimum output tok/s per in-flight request (output_tokens / service_latency_s); each sweep point reports user_tps distributions plus users_at_slo (load * meeting_fraction) when that SLO is set.

Optional --price-per-hour (or sut.cost.price_per_hour) stamps cost_per_million_output_tokens on each stage when output tok/s is known.

The HTTP client is shared across stages (warm connection pool). --extra-body-json must be a JSON object for supported request kinds; a top-level null removes that key from the generated body rather than sending an explicit null. This supports APIs that reject max_tokens in favor of max_completion_tokens. It does not change output-token accounting.

Exports

Flag	Output
`--csv`	One row per request
`--html`	Self-contained report (inline SVG and CSS)
`--mlperf-dir`	LoadGen-shaped text for parser-oriented interoperability

The JSON summary is written to stdout before exports are attempted, so it remains available if an export fails. Strategic CSV keeps a cell for missing TTFT provenance on every row, including failed requests; absent fields in JSON remain omitted as before.

MLPerf-shaped exports contain mlperf_log_summary.txt, mlperf_log_detail.txt, and mlperf_log_accuracy.json for Server or Offline scenarios. Every export file begins with an UNOFFICIAL disclaimer; the summary never prints a bare Result is : VALID without that disclaimer. This export is parser-oriented interoperability and is not an audited or submitted MLPerf result; official submissions must execute the MLPerf LoadGen and compliance suite.

Optional OpenTelemetry export is available on the strategic runner. GitHub Release archives are built with Cargo feature otlp enabled. From source, build with --features otlp (or use a release binary). No network telemetry occurs unless --otlp-endpoint is supplied:

cargo build --release --features otlp
OTEL_EXPORTER_OTLP_HEADERS='authorization=Bearer token' \
metrum-ai-bench-cli-strategic ... --otlp-endpoint https://collector.example.com

The exporter emits OTLP/HTTP JSON spans to /v1/traces and summary request counters plus p95 latency to /v1/metrics. See live --help for the current OTLP flags.

Multi-turn and structured output

Session input is JSONL:

{"session_id":"support-1","messages":[{"role":"user","content":"My order is late"},{"role":"assistant","content":"What is the order ID?"},{"role":"user","content":"A-42"}]}

Every successive turn is submitted with its preceding history. --shared-prefix TEXT --prefix-control shared preserves a cacheable prefix; unique adds a stable per-session discriminator; none omits it.

--json-schema schema.json requests strict JSON-schema output and counts syntactically valid objects containing every required property. --tools tools.json requests a tool call and checks the selected function name and JSON arguments. Valid responses determine validity_rate and feed goodput (together with optional --slo).

These paths are request validity helpers, not an agent evaluation harness. See Known limitations.

Embeddings and reranking

Kind	Flag pattern	Endpoint style
Embeddings	`--kind embeddings --prompt TEXT`	`/v1/embeddings`
Rerank	`--kind rerank --prompt 'query\|document one\|document two'`	Jina/Cohere-style `/v1/rerank`

Which local server to use​

Obtaining the mock server​

Sweep and server correlation​

Goodput on strategic​

Exports​

Multi-turn and structured output​

Embeddings and reranking​