Prompt library

metrum-ai-bench-prompts (also metrum-ai-bench prompts -- …) selects a reproducible mix from the Hugging Face dataset metrum-ai/prompt-library and writes JSONL that metrum-ai-bench-llm can consume with --prompts.

Success is ISL and OSL statistics within absolute CLI tolerances, not an exact row count. --count is a preferred size; the selector may return fewer or more rows (within --count-slack) and may repeat source rows when that is required to land both axes.

This repository ships only tiny fixtures. Large corpora live on Hugging Face.

Dataset

Item	Value
Repository	`metrum-ai/prompt-library`
Configs	`sample` (smoke; default in docs) or `full`
Split	`train`
Pin	Pass a 40-character commit SHA via `--revision` (or `--allow-moving-revision` for floating refs)

Pinned revision used in product examples: 0666f62e581b482838ae2e17b333ee36ff3d01b0.

Fields used for selection

Field	Role
`prompt`	Base prompt text (never rewritten in the dataset)
`target_output_length`	Intended output words; appended as a generation hint
`target_input_tokens`	Supplied ISL token target (used when `--isl-unit tokens` with `--isl-token-basis supplied-target`, the default)
`target_output_tokens`	Supplied OSL token target / recommended per-row budget
`reasoning`	Metadata filter only (`--reasoning any\|true\|false`)

Legacy prompt_length / actual_words are not authoritative for ISL/OSL selection. Word ISL is len(rendered_prompt.split()) with Unicode whitespace (Python str.split() semantics).

The rendered prompt is:

{prompt}

Please aim for approximately {target_output_length} words in your response.

Supplied token ISL does not include the hint text; word ISL does. The selection report records isl.counting_scope.

Extract a mix

metrum-ai-bench-prompts \
  --revision 0666f62e581b482838ae2e17b333ee36ff3d01b0 \
  --config sample \
  --count 64 --count-slack 64 --seed 42 \
  --isl-target 512 --isl-unit tokens --isl-stat median --isl-tolerance 64 \
  --osl-target 128 --osl-unit tokens --osl-stat median --osl-tolerance 32 \
  --output /tmp/mix.jsonl \
  --report /tmp/mix-report.json

Flag	Meaning
`--count`	Preferred mix size (soft)
`--count-slack`	Max \|actual − preferred\| (default `max(count, 32)`)
`--isl-stat` / `--osl-stat`	`mean` or `median` (even-n median = mean of two central values)
`--isl-tolerance` / `--osl-tolerance`	Absolute tolerances in the axis units
`--isl-token-basis`	How token ISL is taken (default / currently only `supplied-target`)
`--osl-tokens-per-word`	Required when `--osl-unit words`; used only to recommend `--max-tokens`
`--max-repeats`	Cap copies of one source row (default 8)
`--no-repeats`	Strict without-replacement (`--max-repeats 1`)
`--local-jsonl` / `--local-parquet`	Offline / test inputs (skip Hub)
`--cache-dir` / `--offline`	Hub cache control

When several mixes all sit inside tolerance, the selector prefers solutions closer to --count and with fewer repeats.

Outputs

JSONL (--output): one object per selected slot, including intentional repeats. prompt already includes the word-count hint. Extra fields (source_ordinal, target_output_tokens, …) are ignored by llm today.
Report (--report): pinned revision, preferred vs selected_count, achieved ISL/OSL and gaps, repeat histogram, recommended_max_tokens, recommended_num_requests, and schedule SHA-256.

After extract, use report.selected_count and report.recommended_max_tokens from the report JSON.

Feed the mix to llm

Keep --warmup-requests 0 so llm does not drop measured mix slots. llm still uses one global --max-tokens.

metrum-ai-bench-llm \
  --url http://127.0.0.1:18321/v1/chat/completions \
  --api-key dummy \
  --scenario prompt-library-median \
  --num-requests "$(jq .selected_count /tmp/mix-report.json)" \
  --concurrency 4 \
  --warmup-requests 0 \
  --prompts /tmp/mix.jsonl \
  --mode chat \
  --streaming \
  --model dummy \
  --max-tokens "$(jq .recommended_max_tokens /tmp/mix-report.json)" \
  --seed 42 \
  --data-log /tmp/mix-run.jsonl

The selected mix is preserved only when --num-requests equals the extractor's selected_count and --warmup-requests is 0. Warmup or a mismatched request count changes the measured mix. Do not set num_requests != selected_count (cycling changes the mix). Repeats in the JSONL are solver output for length statistics, not a claim about prompt diversity or answer quality.

Failures

If no mix in the allowed size band hits both tolerances, the tool exits nonzero before writing --output / --report. Diagnostics include best ISL/OSL gaps, preferred vs best n, candidate count, work used, and whether the failure is a proven empty candidate set vs search/tolerance miss.

Dataset​

Fields used for selection​

Extract a mix​

Outputs​

Feed the mix to llm​

Failures​