Test Image Generation With metrumbench-imagegen

Private Stack Exposure

Remote benchmark workers must reach the Metrum Insights control plane over a public URL. When you run a private or local stack, expose PostgREST before launch and use the public HTTPS URL for METRUM_PUBLIC_API_URL, INSIGHTS_API_URL, and INSIGHTS_CONTROL_PLANE_URL:

cloudflared tunnel --url http://localhost:${POSTGREST_PORT:-3000}
export METRUM_PUBLIC_API_URL="https://<cloudflared-host>.trycloudflare.com"
export INSIGHTS_API_URL="$METRUM_PUBLIC_API_URL"
export INSIGHTS_CONTROL_PLANE_URL="$METRUM_PUBLIC_API_URL"
curl -fsS "$METRUM_PUBLIC_API_URL/" >/dev/null

Do not give a remote worker localhost, 127.0.0.1, host.docker.internal, or a private-only DNS name. Keep the tunnel alive through onboarding, polling, result upload, and teardown; expose any private package/WebUI endpoint through a public URL too when the worker downloads artifacts from that stack.

Use metrumbench-imagegen when the workload produces images instead of text tokens.

Decision

Hold constant:

• image model;
• prompt dataset or snapshot;
• image size, image count, quality settings, and scheduler settings;
• hardware and command template.

Vary:

• model, resolution, batch size, hardware, engine, or container image.

Web UI

1. Create a project for the image-generation campaign.
2. Add a metrumbench-imagegen workload.
3. Select or import a prompt dataset.
4. Create scenarios for image size and batch size.
5. Run smoke first, then the full matrix.
6. Review images/sec, seconds/image, GPU telemetry, failed images, and output artifact paths.

Placeholder: if metrumbench-imagegen workload creation is not exposed in the UI, use the API and inspect generated jobs and artifacts from the project.

API Shape

Use one workload per model/runtime recipe and one scenario per image shape:

Scenario	Meaning
img512-b1	512px image, batch 1.
img1024-b1	1024px image, batch 1.
img1024-b4	1024px image, batch 4.

Result exports should include timing, generated artifact paths, failure count, and telemetry.

Create or select the image prompt snapshot first. For a smoke test, create a manual snapshot with one prompt; for a production run, replace this with your versioned image prompt dataset.

export METRUMBENCH_IMAGEGEN_SNAPSHOT_CODE="image-prompts-smoke"
export METRUMBENCH_IMAGEGEN_PROMPT="A precise product photo of a brushed steel GPU server on a lab bench"
export METRUMBENCH_IMAGEGEN_PROMPT_SHA256="$(printf '%s' "$METRUMBENCH_IMAGEGEN_PROMPT" | shasum -a 256 | awk '{print $1}')"

curl -fsS -X POST "$METRUM_API_URL/rpc/create_metrumbench_imagegen_prompt_snapshot" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_snapshot_code\":\"$METRUMBENCH_IMAGEGEN_SNAPSHOT_CODE\",
    \"p_snapshot_name\":\"Image prompt smoke set\",
    \"p_source_type\":\"manual\",
    \"p_source_ref\":\"operator-upload\",
    \"p_prompt_column\":\"prompt\",
    \"p_org_id\":\"$METRUM_ORG_ID\",
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\"
  }" | tee /tmp/metrumbench-imagegen-snapshot.json | jq

export METRUMBENCH_IMAGEGEN_SNAPSHOT_ID="$(jq -r '.snapshot_id' /tmp/metrumbench-imagegen-snapshot.json)"

curl -fsS -X POST "$METRUM_API_URL/metrumbench_imagegen_snapshot_prompts" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: return=representation" \
  -d "[
    {
      \"snapshot_id\":$METRUMBENCH_IMAGEGEN_SNAPSHOT_ID,
      \"row_index\":0,
      \"prompt\":\"$METRUMBENCH_IMAGEGEN_PROMPT\",
      \"size\":\"512x512\",
      \"prompt_sha256\":\"$METRUMBENCH_IMAGEGEN_PROMPT_SHA256\"
    }
  ]" | jq

Then create the image-generation workload. Use the Metrum model catalog code for p_model_code; the seeded catalog maps z-image-turbo to the Z-Image Turbo model.

curl -fsS -X POST "$METRUM_API_URL/rpc/create_workload" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"$PROJECT_NAME\",
    \"p_workload_name\":\"Z-Image Turbo metrumbench-imagegen\",
    \"p_workload_code\":\"z-image-turbo-metrumbench-imagegen\",
    \"p_tool_code\":\"metrumbench-imagegen\",
    \"p_model_code\":\"z-image-turbo\",
    \"p_framework_code\":\"vllm\",
    \"p_version\":\"0.20.0\",
    \"p_engine_args_set_code\":\"vllm-image-gen-default\",
    \"p_metrumbench_imagegen_snapshot_code\":\"$METRUMBENCH_IMAGEGEN_SNAPSHOT_CODE\"
  }" | jq

Create image-shape scenarios and run them the same way as metrumbench-llm scenarios:

for scenario in img512-b1:512:1 img1024-b1:1024:1 img1024-b4:1024:4; do
  code="${scenario%%:*}"
  rest="${scenario#*:}"
  size="${rest%%:*}"
  batch="${rest##*:}"

  curl -fsS -X POST "$METRUM_API_URL/rpc/create_scenario" \
    -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
    -H "Content-Type: application/json" \
    -d "{
      \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
      \"p_project_name\":\"$PROJECT_NAME\",
      \"p_workload_code\":\"z-image-turbo-metrumbench-imagegen\",
      \"p_scenario_name\":\"$code\",
      \"p_scenario_code\":\"$code\",
      \"p_num_images_batch\":$batch,
      \"p_num_requests\":100,
      \"p_job_params\":{\"size\":\"${size}x${size}\"}
    }" | jq
done

Review metrumbench-imagegen outputs and generated artifacts:

curl -fsS "$METRUM_API_URL/v_metrumbench_imagegen_outputs?select=*&order=created_at.desc&limit=50" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

curl -fsS "$METRUM_API_URL/v_metrumbench_imagegen_image_artifacts?select=job_id,artifact_path,sha256,width,height,model,size&limit=100" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq