Prompt library

metrum-ai-bench-cli-prompts (also metrum-ai-bench-cli prompts -- …) selects a reproducible mix from the Hugging Face dataset metrum-ai/prompt-library and writes JSONL that metrum-ai-bench-cli-llm or metrum-ai-bench-cli-strategic can consume with --prompts.

For strategic sweeps, --prompts requires --max-tokens (use the extract report's recommended_max_tokens) and prefer --warmup-requests on GPU endpoints. See Strategic benchmarking.

Success is ISL and OSL statistics within absolute CLI tolerances, not an exact row count. --count is a preferred size; the selector may return fewer or more rows (within --count-slack) and may repeat source rows when that is required to land both axes.

This repository ships only tiny fixtures. Large corpora live on Hugging Face.

Dataset

Item	Value
Repository	`metrum-ai/prompt-library`
Configs	`sample` (smoke; default in docs) or `full`
Split	`train`
Pin	Default `--revision main` (latest). Pass a 40-character commit SHA to freeze

Resolved SHAs are always written to the mix --report. An example pin from older docs: 0666f62e581b482838ae2e17b333ee36ff3d01b0.

Fields used for selection

Field	Role
`prompt`	Base prompt text (never rewritten in the dataset)
`target_output_length`	Intended output words; appended as a generation hint
`target_input_tokens`	Supplied ISL token target (used when `--isl-unit tokens` with `--isl-token-basis supplied-target`, the default)
`target_output_tokens`	Supplied OSL token target / recommended per-row budget
`reasoning`	Metadata filter only (`--reasoning any\|true\|false`)

Legacy prompt_length / actual_words are not authoritative for ISL/OSL selection. Word ISL is len(rendered_prompt.split()) with Unicode whitespace (Python str.split() semantics).

The rendered prompt is:

{prompt}

Please aim for approximately {target_output_length} words in your response.

Supplied token ISL does not include the hint text; word ISL does. The selection report records isl.counting_scope.

Named profiles

Prefer --profile for publishable compares (versioned ISL/OSL; conflicts with explicit --isl-target / --osl-target):

Profile	ISL	OSL
`chat-short`	256	64
`chat-medium`	512	128
`rag-medium`	2048	256
`summarize-long`	4096	512
`code-medium`	1024	512

metrum-ai-bench-cli-prompts \
  --config sample --count 64 --seed 42 \
  --profile chat-medium \
  --output /tmp/mix.jsonl --report /tmp/mix-report.json

Extract a mix

metrum-ai-bench-cli-prompts \
  --config sample \
  --count 64 --count-slack 64 --seed 42 \
  --isl-target 512 --isl-unit tokens --isl-stat median --isl-tolerance 64 \
  --osl-target 128 --osl-unit tokens --osl-stat median --osl-tolerance 32 \
  --output /tmp/mix.jsonl \
  --report /tmp/mix-report.json

Flag	Meaning
`--revision`	Hub ref (default `main` = latest). Pass a 40-char SHA to pin
`--require-pinned-revision`	Fail unless `--revision` is a 40-char SHA
`--count`	Preferred mix size (soft)
`--count-slack`	Max \|actual − preferred\| (default `max(count, 32)`)
`--isl-stat` / `--osl-stat`	`mean` or `median` (even-n median = mean of two central values)
`--isl-tolerance` / `--osl-tolerance`	Absolute tolerances in the axis units
`--isl-token-basis`	How token ISL is taken (default / currently only `supplied-target`)
`--osl-tokens-per-word`	Required when `--osl-unit words`; used only to recommend `--max-tokens`
`--max-repeats`	Cap copies of one source row (default 8)
`--no-repeats`	Strict without-replacement (`--max-repeats 1`)
`--local-jsonl` / `--local-parquet`	Offline / test inputs (skip Hub)
`--cache-dir` / `--offline`	Hub cache control

The selector searches bounded joint ISL/OSL length compositions, including mixed-median cases. When several mixes sit inside tolerance, it prefers solutions closer to --count and with fewer repeats.

Outputs

JSONL (--output): one object per selected slot, including intentional repeats. prompt already includes the word-count hint. Extra fields (source_ordinal, target_output_tokens, …) are ignored by llm today.
Report (--report): resolved Hub revision SHA, preferred vs selected_count, achieved ISL/OSL and gaps, repeat histogram, recommended_max_tokens, recommended_num_requests, and schedule SHA-256.

After extract, use report.selected_count and report.recommended_max_tokens from the report JSON.

Feed the mix to llm

Keep --warmup-requests 0 so llm does not drop measured mix slots. llm still uses one global --max-tokens.

metrum-ai-bench-cli-llm \
  --url http://127.0.0.1:18321/v1/chat/completions \
  --api-key dummy \
  --scenario prompt-library-median \
  --num-requests "$(jq .selected_count /tmp/mix-report.json)" \
  --concurrency 4 \
  --warmup-requests 0 \
  --prompts /tmp/mix.jsonl \
  --mode chat \
  --streaming \
  --model dummy \
  --max-tokens "$(jq .recommended_max_tokens /tmp/mix-report.json)" \
  --seed 42 \
  --data-log /tmp/mix-run.jsonl

The selected mix is preserved only when --num-requests equals the extractor's selected_count and --warmup-requests is 0. Warmup or a mismatched request count changes the measured mix. Do not set num_requests != selected_count (cycling changes the mix). Repeats in the JSONL are solver output for length statistics, not a claim about prompt diversity or answer quality.

Failures

If no mix in the allowed size band hits both tolerances, the tool exits nonzero without writing the selected --output JSONL. It writes a diagnostic failure JSON to --report, not a usable mix report. Diagnostics include best ISL/OSL gaps, preferred vs best n, candidate count, work used, and whether the failure is a proven empty candidate set vs search/tolerance miss.

The prompt cache is created under the user cache directory (or --cache-dir); creation errors identify the failing path. Fix an existing unwritable cache's ownership or permissions before retrying.

Dataset​

Fields used for selection​

Named profiles​

Extract a mix​

Outputs​

Feed the mix to llm​

Failures​