Output schema

Each line in --data-log is a complete JSON object and carries schema_version.

The repository docs/OUTPUT_SCHEMA.md is the fuller engineering reference; if this page and that file ever disagree, prefer the repository file and live --help / emitted JSONL.

---

Request v5

metrum-ai-bench-cli.request.v5 records are flushed immediately on completion. Earlier schemas remain readable for audit. ttft_s normally means the first reasoning or visible delta; explicit first-byte inference may supply a labelled first_byte_approx. Historical v4 normally used first-visible timing, with a labelled first_token_fallback or first_byte_approx when enabled. ttft_visible_answer_s is separate. outcome and answer_completed distinguish a completed stream from a visible answer. Reasoning-only length finishes are reasoning_truncated, non-failing by default but rejected in strict mode and --ci-safety. ttft_source identifies a streamed token or explicit HTTP-header approximation. --ttft-fallback first-token is deprecated and redundant; --allow-missing-ttft relaxes only the audit. Choice index 0 is measured. Do not mix v4 and v5 TTFT blindly.

Records include:

• optional run_id (same UUID as summary.config.run_id when stamped)
• seq, phase (warmup, measure, drain), and endpoint
• ISO started_at / completed_at
• optional monotonic send_offset_s (seconds from the run-epoch Instant to actual send; preferred for window and closed-loop bins)
• monotonic latency_s, optional ttft_s, first_byte_s, first_reasoning_s, and itl_s
• optional scheduled_offset_s and queue_delay_s
• server usage counts plus optional tokenized_* counts and usage_missing; LLM and VLM records expose usage_prompt_tokens_present and usage_completion_tokens_present to distinguish missing fields from zero
• typed error, partial, modality-specific numeric metrics, and optional string modality_labels

ttft_s is absent for non-streaming LLM/VLM responses unless HTTP first-byte approximation is explicitly requested; it is never fabricated from E2E latency.

modality_metrics

Binary	Keys
VLM	image_count, image_bytes
ASR	rtfx_client, wer, cer, inference_seconds_{server,client}
Imagegen	images_requested, images_returned, response_bytes, artifact_N_bytes

Imagegen artifact SHA-256 digests live in modality_labels.artifact_N_sha256 on request.v5.

---

Summary v5

metrum-ai-bench-cli.summary.v5 changes timing and throughput semantics. It contains measured attempted/success/error counts, rates, type-7 distributions, coordinated-omission-corrected latency, throughput-bin dispersion, SLO goodput, pooled_mixture, full per_endpoint distributions, environment metadata, and partial.

Selected summary fields:

Field	Notes
sut	Optional declared system-under-test block. Always present in JSON; null when --sut was not provided. Fields labelled provenance: "declared".
environment.hostname	May be null when redaction is enabled.
config.run_id	UUID generated once per run
config.effective_max_concurrency	Outstanding-request cap in force
config.common	Effective CommonBenchArgs JSON object. Real field names include seed, warmup_requests, request_rate, arrival, max_concurrency, slos, tokenizer, and related keys - not prose shorthand. Secrets are never stamped.
config.effective_system_prompt	System string actually sent (omitted/null when N/A). LLM defaults to none.
config.body_template	Sanitized request skeleton with a {{prompt}} placeholder
config.unique_prompt_nonce_template	Present when --unique-prompts is on
usage_missing_count	Measured requests with incomplete reported usage
completion_tokens_per_second	Reported completion tokens over the measured terminal window; null when usage is incomplete or the window is invalid
completion_tokens_source	"server_usage" when the headline rate is available
ttft_approx_count	Measured requests with TTFT from HTTP time-to-first-byte (--infer-ttft-from-first-byte)
ttft_warning	Optional note when TTFT was approximated or left unmeasured
Per-request ttft_source	"stream" or "first_byte_approx" when ttft_s is present; older records may use "first_token_fallback"
isl_osl.length_basis	"server_usage" or "tokenizer" for published ISL/OSL means

Every DistSummary carries p90_unreliable, p95_unreliable, and p99_unreliable (unreliable when n * (1 - p/100) < 1).

Readers must treat sut and hostname as optional.

---

Security and provenance

environment is client-observed (OS, architecture, optional hostname). The sut block is declared by the operator, not measured by the client - provenance is usually "declared" (or "mixed" after local sut init --probe). Do not present declared values as observed measurements. For publication runs use --sut <file> --require-sut (implies --redact-hostname); that flag rejects incomplete inventory (gpu, driver, runtime, host_os). See the results-publication policy.

---

Cross-run aggregate

When the unified entry point runs with --runs N (N > 1), a seeded bootstrap aggregate is appended with schema version metrum-ai-bench-cli.cross-run.v1.

---

Partial runs

Interrupted runs can write partial: true. Consumers must not treat a partial summary as a complete campaign cell without checking that flag and sample counts. A hard kill may leave valid request lines without a summary; consumers must accept that recoverable prefix.

---

Recalculation

Request records are written before the summary so every aggregate can be independently recalculated from the JSONL. Warmup lines remain in the file for audit but are excluded from summary distributions.

---

Telemetry NDJSON

metrum-ai-bench-cli-strategic, -llm, -vlm, -asr, and -imagegen each write the same tagged telemetry.v3 file with --ndjson (kinds run, stage, telemetry, request, scrape_error, and summary). It is not the modality --data-log. With configured telemetry, modality binaries scrape sources immediately before measured issuance after warmup; samples or failures are tagged boundary: "pre-measured". Completed modality request rows stream in completion order, with a time-bounded writer flush. Join telemetry samples to measured stages by monotonic t_ns. Offline analysis only (Python/DuckDB recipes in the repository). Full field tables: repository docs/OUTPUT_SCHEMA.md and docs/TELEMETRY.md.