Datasets For KYAI And Temporal Benchmarks

Use KYAI datasets for output quality. Use temporal benchmark datasets for serving behavior under traffic.

KYAI Dataset Shape

Rows should include:

Column	Purpose
instruction	Prompt sent to the candidate model.
ground_truth_response	Reference answer.
external_row_id	Stable id from the source dataset.
row_index	Reproducible ordering and sampling.

API flow:

1. Create a KYAI snapshot.
2. Stage rows with a shared session id.
3. Import staged rows.
4. Use the snapshot code in create_kyai_scenario.

Discover current datasets:

curl -fsS "$METRUM_API_URL/kyai_snapshots?select=snapshot_code,snapshot_name,source_type&order=snapshot_code.asc" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

curl -fsS "$METRUM_API_URL/v_kyai_builtin_sets_with_prompt_count?select=set_code,set_name,prompt_count&order=set_code.asc" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

Temporal Benchmark Datasets

For metrumbench-llm, genai-perf, and inferenceX, dataset choice should match the traffic question:

• random token-like prompts for capacity sweeps;
• fixed prompt libraries for reproducibility;
• domain prompts for application-like latency;
• long-context prompts for memory and scheduler stress.

Placeholder: full dataset import examples for temporal benchmark tools will be expanded as the Web UI dataset screens mature.

Add KYAI Rows

Create a caller-generated session id, stage rows, then import them into a KYAI snapshot. Keep row ids stable so later imports can be reconciled with the source dataset.

export KYAI_IMPORT_SESSION="$(uuidgen)"

curl -fsS -X POST "$METRUM_API_URL/rpc/create_kyai_snapshot" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_snapshot_code\":\"quality-smoke\",
    \"p_snapshot_name\":\"Quality Smoke\",
    \"p_source_type\":\"manual\",
    \"p_source_ref\":\"operator-upload\",
    \"p_org_id\":\"$METRUM_ORG_ID\",
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\"
  }" | jq

curl -fsS -X POST "$METRUM_API_URL/kyai_snapshot_prompt_staging" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: return=representation" \
  -d "[
    {
      \"session_id\":\"$KYAI_IMPORT_SESSION\",
      \"instruction\":\"Explain why batching affects TPOT.\",
      \"ground_truth_response\":\"Batching changes how decode work is shared across concurrent requests.\",
      \"external_row_id\":\"quality-smoke-001\"
    }
  ]" | jq

curl -fsS -X POST "$METRUM_API_URL/rpc/import_kyai_snapshot_prompts" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_snapshot_code\":\"quality-smoke\",
    \"p_session_id\":\"$KYAI_IMPORT_SESSION\"
  }" | jq

Discover Image And Temporal Prompt Sources

For metrumbench-imagegen image generation:

curl -fsS "$METRUM_API_URL/metrumbench_imagegen_prompt_snapshots?select=snapshot_code,snapshot_name,total_prompts,source_type&order=snapshot_code.asc" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

For metrumbench-llm and other temporal tools:

curl -fsS "$METRUM_API_URL/v_prompt_libraries_with_prompt_count?select=library_code,library_name,prompt_count&order=library_code.asc" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq