Skip to main content

Documentation Scope

The docs site is intentionally curated. Do not publish every PostgREST table, view, or generated CRUD endpoint. Add documentation only when it explains a supported workflow, user-facing screen, or runbook-invoked API.

Agent Interface​

  • Browser console session: how to open the Metrum Insights Agent console from the control-plane host or DNS name, how to get a short-lived access token from the CLI, and how to distinguish read-only monitoring from writable control.
  • Agent start screen: what "Metrum Insights Agent" can do, including listing projects, creating benchmark projects, onboarding cloud hardware, launching a metrumbench-llm benchmark, running KYAI evaluation, and packaging reports.
  • Supported workflows: project lifecycle, hardware onboarding, metrumbench-llm workload and scenario creation, cloud benchmark execution with Shadeform or RunPod, KYAI generation/evaluation, results packaging, and teardown.
  • Operator guardrails: cost confirmation, credential handling, public control-plane URL requirements, bounded polling, failure reporting, and when to stop instead of retrying.
  • Embedding contract: iframe or terminal component integration, token issuance, user/session association, expiry, revocation, and audit logging.

WebUI​

  • Authentication and identity: login flow, organization context, project access, and service-token boundaries relevant to user-facing workflows.
  • Projects, workloads, scenarios, and runs: how the WebUI maps the benchmark lifecycle to screens and downloadable artifacts.
  • Server and hardware onboarding: server catalog, cloud provider credentials, managed Shadeform onboarding, readiness state, heartbeat state, and teardown.
  • Benchmark results: job/run status, telemetry summaries, SUT and log downloads, benchmark reports, project reports, deterministic reports, and AI-assisted summaries.
  • KYAI screens: candidate endpoints, snapshot selection, generation runs, judge configuration, evaluation runs, scores, and leaderboard/report views.
  • Operational pages: health/version checks, package downloads used by workers, and error states users can act on.

API​

  • MCP tool surface used by agent runbooks: project lifecycle, catalog lookup, workload/scenario setup, server registration/readiness, cloud execution, benchmark execution, job status, analytics, and reports.
  • PostgREST RPCs and views only when they are invoked by supported runbooks or documented runbooks. Do not document raw tables such as agent_supported_frameworks unless a supported runbook calls them directly.
  • WebUI endpoints used by runbooks and agent tools, including managed cloud onboarding, worker startup/package downloads, AI eval, report downloads, and job artifact downloads.
  • Authentication and authorization for runbook APIs: bearer JWT expectations, service-token-only endpoints, browser user endpoints, and public bootstrap URLs that remote workers need.
  • Request/response examples for the supported paths above, including expected IDs to track: project, workload, scenario, run, job, server, cloud execution, and report URLs.