Skip to main content

Admin & Configuration Guide

This guide is for platform admins, DevOps engineers, and enterprise IT running Metrum Insights for a workspace (the v4.0 UI label for what the data model still calls an organization). It covers the operational surface: identity, access control, billing, backups, compliance, and cloud provisioning.

If you're a benchmarking user looking for "how do I run a job", the User Guide is the right page.

UI status in v4.0: most admin surfaces below are referenced as Settings -> ... panels. The Settings section in the live sidebar in v4.0 is intentionally minimal ~ many of the documented panels (Members, Cloud credentials, Billing, Compliance, Backup, Notifications, Metering, Engine args templates, Engine presets, Validation, Prompt libraries) are not yet exposed in the UI. The underlying RPCs and views exist and the corresponding actions are available via the API. The dedicated UI panels are being staged and will land in subsequent releases.

Multi-tenant identity​

Account and workspace model​

Metrum Insights separates accounts (people) from workspaces (tenancies, the live UI label for what the data model still calls an organization):

  • An account is a single identity ~ backed by an email and an SSO provider.
  • A workspace is a tenancy that owns projects, servers, cloud credentials, billing, and audit trail.
  • An account can belong to multiple workspaces, with a distinct RBAC role per workspace.

This matters operationally: a user who works on two customer engagements has one login and two role assignments, not two logins. Inviting a new user adds them to your workspace; it doesn't create a new identity if they already have an account.

The current workspace is shown in the topbar workspace switcher in the live UI.

Identity providers​

Auth0 is the identity broker (the live UI redirects unauthenticated visits straight through auth.metrum.ai to the configured identity provider). Supported flows:

  • Google SSO ~ enabled by default in the live UI.
  • Microsoft Entra ID / Azure AD ~ SAML; configured per-workspace.
  • Okta ~ SAML; configured per-workspace.
  • Custom OIDC ~ for enterprises with their own IdP; talk to your Metrum account team.
  • Email + password ~ available where SSO is not configured.

For SAML providers, your IdP admin configures the Metrum app and provides the resulting metadata; Metrum binds it to your workspace. Just-in-time provisioning is supported ~ first login from your IdP creates the account and applies a default role (configurable per workspace).

Inviting users​

Member management is performed via the planned Settings -> Members Admin/Owner panel (not yet exposed in the live UI in v4.0; managed via API today). The intended UI flow is:

  1. Click Invite member.
  2. Enter email address.
  3. Pick the role (Owner, Admin, Engineer, Viewer ~ see RBAC below).
  4. Click Send invite.

The invitee receives an email with a one-time link to accept. If they already have a Metrum account, the role is added immediately; if not, they're prompted to create one.

Removing users​

Removal will be performed from the same panel; Remove revokes the user's role in your workspace. The account itself is untouched ~ it remains usable for any other workspaces the user belongs to. Removal is immediate; in-flight API requests under the removed user's token will continue to completion but no new ones will be accepted.

RBAC​

Role types​

Metrum Insights uses four roles, ordered from most to least privileged:

RoleCan doCannot do
OwnerEverything an Admin can do, plus billing, member management, workspace deletion(nothing ~ full control)
AdminPlatform configuration, server onboarding, cloud credentials, RBAC for non-ownersBilling, deleting the workspace
EngineerCreate projects, run benchmarks, manage their own API keys, view all workspace resultsOnboard servers, manage members, change billing
ViewerRead-only across the workspace's results, projects, telemetrySubmit jobs, create projects, modify anything

Permission boundaries​

A few specific decisions worth knowing about:

  • Project visibility is independent of RBAC role. A Viewer can see a Private project they're explicitly added to, even though they can't modify it.
  • API keys inherit the issuing user's role. A Viewer's API key cannot submit jobs.
  • Cross-workspace isolation is strict. Tokens are scoped to the workspace they were issued for; reusing a token across workspaces is not supported.
  • Admins cannot promote themselves to Owner. Only an existing Owner can do that.
  • The last Owner cannot be removed ~ you must promote another member to Owner before stepping down.

Changing a member's role​

Via the planned Settings -> Members -> [user row] -> Change role panel (API only in v4.0). The change is effective immediately. The audited event lands in v_audit_logs with the actor, target, old role, and new role.

Enforcement​

Permissions are enforced at the API layer via the 19_rbac.sql schema. The frontend hides UI for actions the user can't perform, but the server is the source of truth ~ calling a forbidden RPC directly returns 403 Forbidden. Never rely on UI affordances for security.

Server management​

Two ways to bring GPU capacity into Metrum:

  1. On-prem servers ~ register a physical or VM-hosted GPU host running the Metrum agent.
  2. Cloud provisioning ~ Metrum provisions GPUs on RunPod, Shadeform, Lambda, or GKE on demand. See Cloud provisioning below.

Registering a server​

From Hardware -> Add Server in the live UI:

  1. Click Add Server. A modal opens.
  2. Pick a Server Configuration preset from the dropdown (e.g. local-dummy). This binds the new instance to a registered hardware/server-config template.
  3. Fill in:
    • Hostname ~ required, the server's network hostname.
    • IP Address ~ optional, the server's IP if you want to pin it.
    • Bootstrap Script Filename ~ optional, defaults to agent-bootstrap.sh.
    • Bootstrap Comment ~ optional, free-text note shown in the audit trail.
  4. Click Generate Bootstrap Script. The modal will display the bootstrap command to run on the server.

Hosted v4.0 note: some hosted Metrum deployments return 404 PGRST202 from POST https://<your-control-plane>/api/rpc/create_server_config_instance when Generate Bootstrap Script runs. If your deployment shows this error, register servers via the API directly and run the bootstrap helper out-of-band on the GPU host.

The reference one-line bootstrap (subject to the issue above):

curl -fsSL https://<your-control-plane>/install/agent.sh | \
  METRUM_ONBOARD_TOKEN=<token> bash

The script:

  • Installs the metrum-agent systemd service.
  • Detects hardware (GPU model, count, driver, CPU, RAM, NICs) and registers a hardware snapshot.
  • Starts the agent and reports the first heartbeat.

Within ~30 seconds the server appears in the Hardware table with status online.

Server lifecycle​

The agent reports a heartbeat every 30 seconds. Without a heartbeat:

  • No heartbeat for 2 minutes ~ server flagged stale.
  • No heartbeat for 10 minutes ~ server flagged offline.

Jobs queued for an offline server stay queued; they don't fail. If the server comes back, the jobs resume. To explicitly cancel everything on a stuck server, use the planned Cancel Stale Jobs action on the Monitoring page (currently API only).

Reonboarding (v3 -> v4 upgrade)​

Servers registered under v3 need to re-run the bootstrap script after upgrading to v4 ~ the agent daemon was rewritten and the protocol changed. The Hardware page flags affected hosts with a requires_reonboarding badge.

Removing a server​

Hardware -> [row] -> Remove. The server's history (past runs, telemetry) is preserved; only its registration is removed. Any in-flight jobs are cancelled.

Cloud provisioning​

In addition to registered on-prem servers, Metrum can provision cloud GPUs on demand via the Shadeform integration.

Supported providers​

ProviderGPU typesBest for
RunPodH100, H200, A100, B200Spot-priced experimentation
ShadeformAggregator across many providers; H100, H200, L40SBest-price-of-the-moment
LambdaH100, H200Reserved capacity
GKETPU v5e, v5p; NVIDIA A3/A3 MegaTPU benchmarks, GCP customers

Setting up cloud credentials​

Configured via the planned Settings -> Cloud credentials Admin/Owner panel (not yet exposed in the live UI; managed via API today). Per provider:

  • RunPod ~ API key.
  • Shadeform ~ API key.
  • Lambda ~ API key.
  • GKE ~ service account JSON, project ID, default region.

Credentials are stored encrypted-at-rest. Users with Engineer or higher can submit jobs that consume them; only Admins/Owners can rotate or remove them.

Per-user credentials​

If you'd rather each engineer use their own cloud account, store credentials at the user level via the planned Profile -> Cloud credentials panel (API only in v4.0). User-level credentials override workspace-level credentials for jobs submitted by that user.

Provisioning flow​

When a benchmark job needs cloud capacity:

  1. The job specifies a cloud SKU instead of a server hostname.
  2. Metrum requests an instance from the provider via Shadeform.
  3. The provider returns an instance; Metrum auto-installs the agent.
  4. The job runs.
  5. After a configurable idle timeout (default 5 min), the instance is torn down.

The full lifecycle is tracked in v_cloud_provisioning_requests and v_cloud_resource_lifecycle. Failures land in the Cloud -> Failures tab with structured error codes. The Cloud page also exposes Instances, Executions, and Spend tabs alongside Failures, plus four KPI tiles: ACTIVE INSTANCES, GPU-HOURS (30D), SPEND (30D), and TOP PROVIDER.

Cost controls​

Two knobs:

  • Per-workspace monthly cap ~ admin-configurable via the planned Settings -> Billing panel (API only in v4.0). Once hit, new cloud provisioning requests are rejected.
  • Per-user daily cap ~ set via the planned Settings -> Members -> [user] -> Cloud limits panel (API only in v4.0). Defaults to the workspace-level cap.

Both caps are advisory at job submission and hard at provisioning time. A user can queue a job that would exceed the cap; provisioning fails when actually attempted.

Billing​

Stripe integration​

Billing runs on Stripe. Owners will configure the Stripe customer linkage via the planned Settings -> Billing -> Connect Stripe panel (not yet exposed in the live UI in v4.0). The intended UI flow:

  1. Click Connect Stripe.
  2. Sign in to Stripe (or create a new customer).
  3. Authorize the Metrum app.

Once connected, invoices issue monthly. Usage detail will be visible under Settings -> Billing -> Usage.

What's metered​

  • GPU-hours ~ measured per benchmark job. On-prem servers are not metered for compute (you own the box); cloud-provisioned compute is metered.
  • Cloud spend pass-through ~ direct provider charges are surfaced and re-invoiced with markup per your contract.
  • KYAI judge tokens ~ input + output tokens consumed by judge LLMs. Metered regardless of where the judge runs (on-prem or cloud).
  • API requests ~ most plans have unlimited API requests; high-volume plans may include a soft cap.

Quotas​

Admins will set per-workspace monthly quotas via the planned Settings -> Billing -> Quotas panel (API only in v4.0):

  • GPU-hours
  • Judge tokens
  • Cloud spend

Hitting a quota returns 402 Payment Required at job submission, with the specific quota named in the error.

Viewing invoices​

The planned Settings -> Billing -> Invoices view will list past invoices with status (paid, open, void) and a per-invoice line-item breakdown (not yet exposed in the live UI in v4.0).

GDPR and compliance​

Data residency​

The control plane runs in a single region per deployment (US, EU, or APAC). On-prem servers stay where they are physically located. Cloud-provisioned compute uses the region nearest to the control plane unless overridden in the cloud provider config.

Personally identifiable information​

The platform stores:

  • Email addresses and display names (account metadata).
  • Auth0 user IDs.
  • Audit log entries (who did what, when).

It does not store:

  • Prompt content sent through BYOE endpoints (those go directly from the platform's runner to your endpoint).
  • Real user data when running KYAI on a non-built-in prompt set - your custom prompts are stored, but they're treated as your data, not Metrum's.

Data retention​

Data categoryDefault retentionConfigurable?
Audit logs90 daysYes ~ up to 7 years
Benchmark resultsIndefiniteYes ~ per-workspace TTL
Telemetry time series90 daysYes ~ up to 365 days
KYAI outputsIndefiniteYes ~ per-workspace TTL
Cloud provisioning logs90 daysNo

Configured via the planned Settings -> Compliance -> Data retention Admin/Owner panel (API only in v4.0).

Data deletion requests​

Under GDPR / similar regimes, users can request their data be deleted. The flow:

  1. The user (or admin acting on their behalf) submits a deletion request via the planned Profile -> Data -> Request deletion panel (API only in v4.0), or via API at POST /rpc/request_account_deletion.
  2. The request enters a 14-day pending state, allowing recovery.
  3. After 14 days, the user's account metadata is hard-deleted. Audit logs are anonymized (the user's account ID is hashed); benchmark results and KYAI outputs they created remain, attributed to the anonymized ID.

You can shorten the 14-day window per-workspace via the planned Settings -> Compliance panel, but anything under 24 hours requires a manual ticket.

Audit logs​

Every privileged action writes to v_audit_logs. Surfaced events include:

  • Member added / removed / role changed.
  • API key issued / revoked.
  • Cloud credential added / rotated / removed.
  • Project visibility changed.
  • Data retention policy changed.
  • Data deletion requested / completed.

The audit view is queryable like any other view:

curl "$METRUM_BASE/v_audit_logs?event_type=eq.member_role_changed&order=created_at.desc&limit=50" \
  -H "Authorization: Bearer $METRUM_TOKEN"

Backup and recovery​

Automatic backups​

The control plane database is snapshotted nightly. Snapshots are retained 30 days by default.

Cloud-provisioned compute is not backed up - instances are ephemeral by design. The benchmark results are persisted to the control plane DB and covered by its backups.

Manual snapshots​

Admins can trigger an on-demand snapshot before risky operations via the planned Settings -> Backup -> Take snapshot panel (API only in v4.0). The intended UI flow:

  1. Open Settings -> Backup -> Take snapshot.
  2. Provide a label.
  3. The snapshot completes in 1-5 minutes depending on DB size.

Restore​

Self-service restore is not available in v4.0. Restore is a support-assisted operation. To initiate:

  1. File a ticket via your account team (the in-product AI Support Agent referenced in earlier drafts is planned and not yet exposed in v4.0).
  2. Provide the target snapshot label or timestamp.
  3. Metrum support coordinates the restore window with you.

This will become self-service in a later release.

Metering configuration​

The metering module records every billable event. Defaults are sensible ~ most admins won't need to change anything. The knobs that exist:

  • Sampling interval for telemetry ~ default 1 sec; raise to reduce telemetry volume.
  • Per-job overhead allocation ~ controls how startup time (model loading, etc.) is attributed in cost reports.
  • Cloud spend markup ~ set by Owner; affects what the customer is invoiced vs. raw provider cost.

All configurable via the planned Settings -> Metering Admin/Owner panel (API only in v4.0).

Notifications​

Email notifications are user-scoped via the planned Profile -> Notifications panel (API only in v4.0). Admins will be able to set workspace-level defaults via the planned panels:

  • Settings -> Notifications -> Defaults ~ what new members get pre-configured.
  • Settings -> Notifications -> Admin alerts ~ admin-only events (quota approaching, cloud failure rate spike, security events).

The platform doesn't send SMS or webhook notifications in v4.0 ~ that's planned for v4.1.

API keys and machine credentials​

Issuing keys​

API key management is planned via the Profile -> API keys -> Issue new key panel (Engineer or higher; API only in v4.0). Keys:

  • Are shown once, at creation. Save them immediately.
  • Inherit the issuing user's role.
  • Have a configurable TTL (default: 90 days; max: 1 year).
  • Can be revoked at any time.

Rotation​

There's no automatic rotation in v4.0. Admins should establish a periodic rotation process ~ typically 90 days for production keys, longer for low-risk read-only keys.

Service accounts​

For long-lived automation, prefer a dedicated service-user account (with the minimum role required) over a personal user's API key. This avoids losing automation when a person leaves the team.

Health monitoring​

The platform's own health is surfaced on the Dashboard for admins. The live dashboard shows four KPI tiles (PROJECTS, ACTIVE JOBS, SERVERS ONLINE, ACTIVE RUNS) and the following widgets:

  • Throughput Leaderboard ~ top model benchmarks by tokens/s.
  • Recent Activity ~ recent activity feed across the workspace.
  • Server Fleet ~ heartbeats and online counts.
  • Recent Runs ~ overall recent runs with status.
  • Cloud ~ active cloud instances and quick stats.

External health endpoints are not yet wired in v4.0 on the live deployment ~ the documented GET /health, GET /health/ready, and GET /metrics paths currently 307-redirect to the SPA login. For now, monitor the platform via the Dashboard widgets and via PostgREST availability checks against https://<your-control-plane>/api.

Upgrading​

Self-hosted deployments are upgraded via your provided runbook (specific to your deployment model). Hosted-platform users don't manage upgrades ~ Metrum performs them with at least 7 days' notice for breaking changes.

For the v3.9 -> v4.0 transition specifically, see Release Notes - Upgrading from v3.9.

See also​