Skip to main content

API Reference

Private Stack Exposure​

Remote benchmark workers must reach the Metrum Insights control plane over a public URL. When you run a private or local stack, expose PostgREST before launch and use the public HTTPS URL for METRUM_PUBLIC_API_URL, INSIGHTS_API_URL, and INSIGHTS_CONTROL_PLANE_URL:

cloudflared tunnel --url http://localhost:${POSTGREST_PORT:-3000}
export METRUM_PUBLIC_API_URL="https://<cloudflared-host>.trycloudflare.com"
export INSIGHTS_API_URL="$METRUM_PUBLIC_API_URL"
export INSIGHTS_CONTROL_PLANE_URL="$METRUM_PUBLIC_API_URL"
curl -fsS "$METRUM_PUBLIC_API_URL/" >/dev/null

Do not give a remote worker localhost, 127.0.0.1, host.docker.internal, or a private-only DNS name. Keep the tunnel alive through onboarding, polling, result upload, and teardown; expose any private package/WebUI endpoint through a public URL too when the worker downloads artifacts from that stack.

This API section documents only the interfaces used by supported Metrum Insights runbooks and the Metrum Insights Agent. It is not a dump of every database table, view, or generated PostgREST CRUD endpoint.

Use the runbook API docs when building automation around:

  • choosing a model+hardware benchmark target and provider section;
  • creating projects, workloads, scenarios, and runs;
  • onboarding cloud or remote benchmark workers;
  • launching and monitoring benchmark jobs;
  • retrieving telemetry, outputs, reports, and downloadable artifacts;
  • running KYAI generation and evaluation workflows.

Raw table endpoints are intentionally omitted unless a runbook invokes them directly.

Core Runbook API Groups​

  • Model And Hardware Runbooks explains the maintained target matrix and provider-subsection organization.
  • End To End Benchmark API Example gives a concrete copy-pasteable flow for SUT onboarding, project creation, metrumbench-llm scenario sweeps, execution, monitoring, and result collection.
  • Qwen3.5 122B A10B On Shadeform B300 shows the managed cloud flow for a Shadeform B300 worker, DB-minted JWTs, local-stack exposure through a Cloudflared tunnel, and a vLLM 0.20.1 FP8 runtime contract.
  • Project Lifecycle covers identity context, project creation, project listing, and run status.
  • metrumbench-llm Workload And Scenario covers model/framework lookup, workload creation, scenario creation, and scenario listing.
  • Hardware Onboarding covers server configs, concrete server instances, bootstrap commands, readiness, and heartbeat status.
  • Bring Your Own Hardware covers SSH-accessible hardware where Metrum Insights installs the worker and starts the model server before benchmark execution.
  • Hardware Sizer API covers product Sizer endpoints, recommendation request shape, scalar estimates, and common error responses.
  • Cloud Benchmark Execution covers managed provider onboarding, scenario execution, and job status.
  • Results And Reports covers benchmark results, telemetry, raw metrumbench-llm outputs, SUT CSV, cost, and report downloads.