Skip to main content

Metrum Insights 4.0

Documentation version: v4.0.0.rc.final1 Initial release: May 15, 2026 Latest patch: v4.0.1 - June 4, 2026 Previous version: 3.9

v4.0.1 - Intermediate Release (June 4, 2026)​

This patch addresses stability, security, and usability issues identified after the initial v4.0 launch. No new features are introduced - the focus is on hardening the platform for production workloads.

Fixes and improvements​

Cloud and provisioning​

  • Platform-global cloud credential bootstrap - Cloud credentials can now be bootstrapped at the platform level (org_id=NULL) in addition to per-org. Simplifies initial setup for single-org deployments.
  • Manual cloud provisioning bootstrap - Fixed a bug where manually provisioned Shadeform/RunPod instances failed to bootstrap correctly, leaving them stuck in a pending state.
  • Stale managed model server ports - Fixed an issue where the managed model server process retained stale port bindings after a restart, causing connection failures for new benchmark jobs.

Agent and onboarding​

  • Agent package bind-mount fix - Prevented Docker from creating the agent packages bind-mount directory as root, which blocked non-root agent processes from writing to it.
  • Hostname availability check scope - The check_server_hostname_availability RPC now checks org-wide instead of per-config. Previously, a hostname registered under one config could be re-registered under a different config in the same org, creating duplicate server entries.
  • Restic install script syntax error - Fixed a bash syntax error in the restic install script that broke automated onboarding on fresh machines.
  • Agent InferenceX ingest path - Fixed the InferenceX packaged ingestion path and metrumbench prompt bundle failures in the agent daemon.
  • Agent GenAI-Perf cache - Provided a writable GenAI-Perf cache directory so the agent no longer fails with permission errors during TensorRT-LLM benchmarks.

Schema and RPCs​

  • Personal access tokens (PATs) - Added personal access token support with 24h JWT lifetimes. PATs can be created and managed via the API, with a bearer pass-through proxy at /rpc/[...path] for direct PostgREST access.
  • RBAC service caller exemption - Trusted metrum_service callers are now exempt from the rbac_pre_request auth gate, fixing service-to-service RPC failures that required unnecessary account context.
  • Permission-mapped RPC auth enforcement - RPCs registered in rpc_permission_map now correctly require an authenticated caller. Previously, some permission-mapped RPCs could be called without a valid JWT.
  • PAT RPC caller binding - Fixed the personal access token RPC caller binding so tokens are correctly associated with the creating account.
  • Project import actor binding - Fixed project import to correctly bind the importing actor, preventing orphaned imports with no owner.
  • KYAI endpoint draft promotion - KYAI candidate endpoint drafts are now promoted before project creation, fixing a race condition where endpoints were not available when the project tried to reference them.
  • KYAI draft table for robust endpoint import - Added a draft table for KYAI endpoint imports with multi-model support, improving reliability of bulk endpoint onboarding.
  • SGLang trust_remote_code arg_type - Corrected the trust_remote_code argument type for SGLang 0.5.9 and 0.5.10, which was causing framework launch failures for models requiring remote code.
  • Benchmark framework args pinning - Seed data now pins benchmark framework engine arguments to prevent drift when defaults change across framework versions.

Frontend and API​

  • PostgREST proxy route - Added /api/postgrest route for direct PostgREST proxy access from the frontend, simplifying API integration for internal tooling.
  • HF token persistence - Fixed Hugging Face token save and delete operations in the Settings page, which were silently failing.
  • Blank role-change notification emails - Fixed a bug where role-change notification emails were sent with blank bodies.

Infrastructure and runtime​

  • Staging bootstrap failures - Fixed staging bootstrap failures on fresh machines caused by missing runtime directories and stale config references.
  • Compose project and volume isolation - Runtime now honors explicit compose project and volume names, preventing collisions between dev and staging environments on the same host.
  • Dev worktree isolation - Preserved development worktree isolation while correctly honoring staging naming conventions.

Benchmarking tools​

  • vLLM Bench metrics ingestion - Added vLLMBench metrics ingestion and reporting pipeline with default scenario parameters.
  • TRT-LLM VLM token budget - Fixed TensorRT-LLM VLM token budget calculation that was causing out-of-memory errors on smaller GPUs.
  • InferenceX ingest wrappers - Fixed legacy InferenceX ingest wrapper scripts and repo script paths.

RBAC and security​

  • Service bypass for agent JWTs - Added service bypass for agent JWTs and missing Casbin policies that were causing agent-initiated operations to fail authorization.
  • Platform-global credentials in cloud RPCs - Cloud instance create/terminate RPCs now include platform-global credentials, fixing failures when credentials were bootstrapped at the platform level.
  • KYAI gen-run server_id binding - Fixed the frontend to pass server_id to KYAI createGenRun, preventing unsolicited job distribution to unintended servers.

Views​

  • GPU count double-counting fix - Fixed v_hardware_label and v_benchmark_results views that were double-counting GPU counts due to an incorrect JOIN, leading to inflated hardware labels in the UI.

Known issues resolved​

Original issueResolution
Server-add 404 (PGRST202)Fixed - create_server_config_instance RPC is now correctly exposed in the schema cache with hostname availability pre-check.
Blank notification emailsFixed - role-change emails now render correctly.

Migration notes​

  • PAT users: If you were using workaround API keys, you can now create proper personal access tokens via the API. Tokens are scoped to the creating account and expire after 24 hours.
  • Agent re-deploy: If your agent daemon was failing on InferenceX or GenAI-Perf jobs, pull the latest agent packages and restart. No re-onboarding required.
  • SGLang 0.5.9/0.5.10 users: Framework launch configs for trust_remote_code are now corrected. Re-run affected scenarios without manual workarounds.

Metrum Insights 4.0 is a ground-up architectural rewrite and the first broad-market release of the platform. If you've been on 3.8 or 3.9, the workflows you know are still here ~ projects, workloads, runs, and the Benchmarks results page (formerly branded "Pulse") ~ but almost everything underneath them has been rebuilt around a new control plane, a new results model, and a unified hardware abstraction that finally treats CPU, GPU, and TPU as first-class citizens.

This page covers what changed, what's new, what's known-broken, and what's been pushed to v4.1. Read Upgrading from v3.9 first if you're migrating an existing org.

Highlights​

The five things to know about 4.0:

  1. Architectural overhaul. New control plane, new data model, new frontend. The 3.x APIs are deprecated; see Breaking changes.
  2. InferenceX replaces InferenceMax. Faster, more accurate scaling estimates, and a new ingest pipeline. See Upgraded features.
  3. KYAI v2. Quality evaluation got a redesign ~ two-phase generation + LLM judge, a results explorer with per-output filters, and a new public leaderboard. The KYAI execution path is wired through the project workload card (Add KYAI Run) and the API; the dedicated KYAI navigation surface is being staged and is not yet present in the live sidebar.
  4. New workload types. ASR (speech) and VLM (vision-language) benchmarking are first-class in the API and the workload model. The dedicated launch surfaces in the UI (modality picker on the workload card, ASR/VLM tabs) are being staged and currently default to LLM-shaped inputs.
  5. Sizer. A new hardware recommendation tool ~ answer a few questions about your use case and get back a sized configuration. Built for pre-sales and capacity planning conversations.

What's new​

Sizer - hardware recommendation tool​

Tell Sizer what you're trying to run, or what constraints you already know, and it returns ranked serving-fit recommendations from the platform's measured benchmarks. The live UI now uses one unified Run Sizer flow with optional model, framework, precision, owned inventory, workload shape, throughput, latency, budget, node, parallelism, and hardware-filter inputs, plus a Chat tab that routes normal sizing prompts through the same sizing_plan path. Designed for conversations like "we need 50k TPS; what model and hardware should we consider?", "can we run Qwen3.6 Plus under this latency limit?", and "we own 1 H200; what can we run on it?"

See the Sizer Usage Guide for inputs, outputs, and a worked example.

ASR (Automatic Speech Recognition) benchmarking​

Benchmark speech-to-text models with the same scenario-matrix model used for LLMs. Driven by the metrumbench-asr tool internally; surfaced on the Benchmarks page alongside LLM and VLM results. Supported on vLLM and SGLang where the framework supports the model. The dedicated ASR launch surface (audio dataset picker, sample-length controls) is being staged in the workload card and not yet exposed in the public UI.

VLM (Vision-Language) benchmarking​

Image-to-text benchmarking for multimodal models. Uses the metrumbench-vlm tool. Concurrency, ISL, and OSL parameters work the same way as for LLMs, with image preprocessing time included in TTFT. The dedicated VLM launch surface (image dataset / images_per_request controls) is being staged in the workload card and not yet exposed in the public UI.

Google TPU support​

TPU v5e and v5p are now supported targets via GKE-managed inference. Provisioning is integrated into the cloud flow under Cloud -> Instances. Telemetry coverage includes per-chip utilization and memory ~ see the new v_tpu_telemetry_job_summary view.

Pre- and post-run validation​

Every run now passes through a validator before the first job starts (pre-run) and after the last job completes (post-run):

  • Pre-run checks that the model is accessible to the server, the framework version exists, the GPU has enough memory for the largest scenario, and HF tokens (if needed) are valid.
  • Post-run sanity-checks the result shape ~ output token counts in plausible ranges, no all-zero latency rows, telemetry actually attached.

Failures surface as a single, structured error on the Run Detail page instead of a hard-to-diagnose job-level failure.

Email notifications (planned)​

Per-user notification preferences for run completion, run failure, and weekly summary. Defaults will be run completion and failure only. The Profile-level Notifications panel is not yet exposed in the live UI in v4.0.

Metering + Stripe billing (planned)​

Per-workspace metering of GPU-hours, cloud spend, and KYAI judge tokens, billed via Stripe. Admin views for usage and invoices are scoped under Settings -> Billing; the Settings sub-pages are not yet exposed in the live UI in v4.0. See the Admin Guide.

Multi-tenant identity and RBAC​

A single account can belong to multiple workspaces, with a distinct role per workspace. The current workspace is shown in the top-right of the topbar. Roles are:

  • Owner ~ full control, including billing and member management.
  • Admin ~ platform configuration, but no billing.
  • Engineer ~ create projects, run benchmarks, view results.
  • Viewer ~ read-only across the workspace.

Permissions are checked at the API layer. See the Admin Guide.

GDPR and compliance (planned)​

PII handling, data retention policies, and a data-deletion workflow are scoped under Settings -> Compliance (Admins only). The Settings sub-pages are not yet exposed in the live UI in v4.0; the audit-log view (v_audit_logs) is queryable via the API where applicable.

Backup and recovery (planned)​

Admin-only. Database snapshots are intended to run nightly with manual snapshots triggered from Settings -> Backup. Restore is a support-assisted operation in v4.0. The Settings sub-pages are not yet exposed in the live UI in v4.0.

Upgraded features​

InferenceMax → InferenceX​

The capacity estimation engine has been rewritten:

  • New ingest pipeline (v_inferencex_outputs) replaces the legacy InferenceMax tables.
  • More accurate scaling estimates for batch sizes above the measured range.
  • Per-scenario confidence intervals on extrapolated points.

The old InferenceMax UI is gone; queries against legacy tables return an empty set. If you have downstream tooling reading InferenceMax results, point it at v_inferencex_outputs.

KYAI v1 → v2​

KYAI was rebuilt around a two-phase generation + judge pipeline:

  • Phase 1 - Generation. Run a candidate endpoint over a prompt set, capture outputs.
  • Phase 2 - Judging. Run a configured LLM judge over the (instruction, response, ground truth) triples and produce a score plus reasoning.

The v2 Results Explorer lets you filter by job, score range, candidate endpoint, and judge config; the Endpoints & Judges tab makes both first-class configurable objects instead of buried JSON.

A new KYAI Leaderboard ranks scored runs across the platform - visible based on project visibility.

Benchmarks (formerly Pulse)​

The results explorer was redesigned and now sits at /benchmarks with the page title "Benchmarks":

  • New filter bar with a Filters dropdown and a My runs toggle pill at the top of the results table.
  • Chart panel sub-tabs are Model, Framework, Quantization, Concurrency, Cost, and Hardware ~ covering model/framework/quantization comparisons, concurrency scaling, cost efficiency, and hardware utilization.
  • Pagination switched to PostgREST range-based pagination ~ faster on large result sets.

The marketing name "Pulse" still appears in some legacy pages and links. The product surface is named Benchmarks.

Cloud provisioning (Shadeform integration)​

Cloud provisioning across RunPod, Shadeform, Lambda, and GKE now goes through a single Shadeform-mediated lifecycle. The Cloud page gets four tabs (Instances, Executions, Spend, Failures) with a unified state machine underneath. The Cloud header also shows four KPI tiles: ACTIVE INSTANCES, GPU-HOURS (30D), SPEND (30D), and TOP PROVIDER. The Spend tab exposes time-range buttons for 7 days, 30 days, 90 days, and All time.

If you were on v3.9 with the old per-provider flows, they've been retired. Existing instances will continue running, but new provisioning requests use the new path.

Breaking changes​

API surface​

The 3.x REST surface has been replaced by the v4 PostgREST + RPC API. The control-plane RPCs (rpc/create_metrumbench_llm_job, rpc/get_job_status, etc.) are documented in the API Reference. The 3.x endpoints return 410 Gone after May 15, 2026.

Authentication​

Token format changed. v3 API keys are revoked at upgrade time. Generate new keys via the planned Profile -> API keys panel (Engineer+; API only in v4.0).

Result schema​

The benchmark_results table is gone; reads go through v_benchmark_results (a view). Column names are stable where possible, but a few have moved - see the migration table in the API Reference.

Server agent​

The metrum-agent daemon was rewritten. Servers registered under v3 need to re-onboard with the v4 bootstrap script after upgrade. The platform will surface registered-but-unreachable servers on the Hardware page with a requires_reonboarding flag.

Deprecated tools​

These tools are gone in v4.0 and their result views are read-only:

  • The legacy vllm_bench integration (use metrumbench-llm with framework_code=vllm).
  • The standalone genai_perf flow (still available via v_genai_perf_outputs, but no new jobs).

Known limitations​

These are real, not just rough edges - work around them or wait for v4.1.

AreaLimitation
Multi-nodeAll v4.0 benchmarks are single-node. Multi-node vLLM, NVIDIA, and AMD configs are in scope for v4.1.
TPU coverageTPU support is read-write for benchmarks, but Sizer recommendations are NVIDIA/AMD only in v4.0.
KYAI judge costJudge token usage is metered but not capped per run. A runaway judge config on a large prompt set can run up a bill. Set conservative max_concurrent_requests on judge configs.
BYOE timeoutsBYOE endpoints with TTFT > 60s will be killed by the benchmark runner. Raise this in your endpoint config if you're testing a cold-start case.
AWS / Intel GPUsListed as supported in the Hardware Compatibility Guide but subject to hardware availability ~ confirm with your account team before relying on them for a deployment.
Backup restoreSelf-service restore is not in 4.0. Restore is support-assisted. Backups themselves run automatically.
Stale tabsLong-backgrounded browser tabs may show outdated job status. Refresh to force a re-fetch.
KYAI / ASR / VLM UIThe execution paths exist via the API and the workload card in v4.0; the dedicated KYAI/ASR/VLM navigation surfaces in the sidebar are being staged and may not be visible.
Sizer Chat tabThe Chat tab on /sizer requires a configured AI assistant service. Without it the tab shows an unavailable-state message.
API hostThe runtime API in v4.0 is mounted under the configured API base URL. See API Reference for the rendered deployment value.
Server-add 404Resolved in v4.0.1. The create_server_config_instance RPC is now correctly exposed in the schema cache with a hostname availability pre-check. See v4.0.1 fixes.
Profile / NotificationsThe Profile-level Notifications panel is not yet exposed in the live UI in v4.0.
Settings sub-pagesSettings -> Members / Cloud credentials / Billing / Compliance / Backup are not yet exposed in the live UI in v4.0. Most actions are available via the API only.

Deferred to v4.1​

These were originally scoped for 4.0 and have moved out:

  • Multi-node benchmarking ~ vLLM, NVIDIA, and AMD multi-node configurations.
  • NVIDIA NIM ~ NIM container support as a first-class framework.
  • NVIDIA Dynamo + SSD offloading (LMCache) ~ disk-tier KV cache offload. Note: Dynamo is selectable as a Sizer framework option in the live UI today, but the run path is not yet wired end-to-end.
  • Ollama workloads ~ running benchmarks against an Ollama endpoint.
  • LoRA finetuning benchmarking ~ currently inference-only.
  • Versus ~ side-by-side live A/B benchmarking between two configurations.
  • Creator ~ AI-generated benchmark reports.
  • Image generation benchmarking ~ text-to-image throughput and latency. Note: the Image Generation modality is selectable in Sizer today, but execution against image-gen workloads is not yet wired end-to-end.
  • AI Support Agent ~ docs-aware in-product assistant.
  • Third-party benchmark integrations ~ vLLM Bench and VectorDB Bench (Milvus).

Target window for v4.1 will be communicated separately. We expect to ship a subset of the above (multi-node + NIM most likely) in the next release.

Upgrading from v3.9​

If your org is on v3.9 today, the upgrade path is:

  1. Read Breaking changes first. API keys and the agent daemon are the two things most likely to break automation overnight.
  2. Rotate API keys. Generate v4 keys before May 15; the v3 keys are revoked at the cutover.
  3. Re-onboard servers. Run the new bootstrap script on each registered server. The platform will list servers needing reonboarding under Hardware with a yellow status badge.
  4. Repoint downstream consumers. Anything reading the old benchmark_results table needs to move to v_benchmark_results. Same for InferenceMax -> v_inferencex_outputs.
  5. Review user roles. RBAC defaults are conservative ~ existing users land in the Engineer role. Owners should review the workspace member list (planned Settings -> Members view; API-only in v4.0) and promote admins as needed.

Run history, project structure, and saved workloads carry over. KYAI v1 runs are preserved as read-only ~ new evaluations use v2.

Feedback​

Found a bug? Have a feature request? The fastest path:

  • GitHub ~ for confirmed bugs, in Metrum-AI-Products/metrum-insights.
  • Email ~ your account team for anything that touches the contract.
  • AI Support Agent ~ planned in-product assistant; not yet exposed in v4.0.