Skip to main content

Release Notes

Metrum Insights 4.1.1 Release Notes​

ProductMetrum Insights
Version4.1.1
Previous version4.1.0
Release DateSeptember 10, 2026

A corrective release. Three defects blocked people from using workspaces they had paid for or been invited to, and two of them were silent. New workspaces were created with no way to sign in, team owners were shown as members and lost control of their own teams, and an already-accepted invitation showed an error page instead of signing the person in. Two critical remote-code-execution advisories in the web framework are also closed, along with every other known advisory in the dependency tree.

Sign-in methods on new workspaces (Fixed)​

  • Every workspace created since 13 July had no sign-in method. Nobody could enter one, including the person invited to it. The interface reported success, so nothing alerted and nothing was logged.
  • The cause was an identity-provider field removed at its end of life. Our request for it had failed on every call since that date, and all three code paths that create or repair a workspace treated the failure as "this application has no sign-in methods".
  • All three paths now fail the request instead of reporting success. Creating a team removes the empty workspace and reports the problem, first sign-in leaves the workspace unlinked so the next attempt retries cleanly, and inviting someone refuses to send an invitation into a workspace that is not ready.
  • Existing workspaces were repaired and confirmed. No action is required from you.

Team roles and ownership (Fixed)​

  • Team owners and admins were shown as members and lost rename, invite and delete. A role that could not be read was treated as the lowest role, so a brief outage silently demoted people. Roles were also not compared by strength, so anyone holding two roles could resolve to the weaker one.
  • Roles now resolve to the strongest role held. A role that cannot be read is shown as unknown and the action is reported as temporarily unavailable, rather than as a refusal.
  • A team's recorded owner could be the wrong person. The first member to sign in after a team was created became its recorded owner, whatever their actual role. Ownership is now recorded only for the real owner.
  • Creating a team now fails cleanly if owner permissions cannot be granted, instead of producing a team nobody can administer.
  • The team panel also opened on the wrong team after a page refresh. It now opens on the team you were working in.

Invitations (Fixed)​

  • Accepting an invitation on a phone and then opening the same email on a laptop showed an error page, even though you were already a member by that point. An invitation link is single use, and the second click was treated as a failure.
  • A used invitation now signs you in and takes you to your workspace.

Billing (Fixed)​

  • A checkout could expire on our side while the payment provider was still taking payment, which left a paid subscription recorded as unpaid. The two deadlines are now the same value.
  • Scheduled reconciliation only examined the oldest page of subscriptions, so newer subscriptions were never checked against the payment provider. It now covers every subscription.
  • A team you own can no longer be missing from the organization picker at checkout.

Security​

  • Web framework updated to close two critical remote-code-execution advisories, one of which was reachable from the public internet, plus eleven further advisories.
  • Mail library updated to close five advisories, three of which could deliver mail to an attacker-controlled domain. Team invitations use that path.
  • Twenty-two further advisories closed across the dependency tree. The dependency audit now reports no known vulnerabilities.

Product and copy​

  • The add-ons section is removed from the plans page.
  • Sales enquiries now go to contact@metrum.ai everywhere.

Work Items​

Metrum Insights 4.1.0 Release Notes​

ProductMetrum Insights
Version4.1.0
Previous version4.0.1
Release DateSeptember 3, 2026

Two new benchmarking capabilities headline this release: multi-node distributed inference (llm-d on Kubernetes) for models too large for one server, and KV cache offload (LMCache) for measuring CPU/NVMe cache tiers against GPU-only. Both shipped, were exercised against real staging deployments, and were hardened through a series of fixes the staging runs surfaced. Alongside them: a live billing defect that overcharged customers on a plan change is fixed, hardware telemetry moves to a single collector (all-smi) with real system power and DMI chassis identity, the dashboard/landing/project-creation surfaces are redesigned, and all 16 outstanding high-severity security findings are resolved.

Multi-Node Distributed Inference (New)​

  • Benchmark models too large for one server - A new deployment mode clusters multiple servers with Kubernetes (K3s) and deploys the model via llm-d instead of a single-process server. See Multi-Node vLLM Deployment.
  • Cluster formation and bootstrap are fully automated - no manual Kubernetes administration.
  • Hardened through a live staging deployment: fixed a missing migration that blocked production upgrades, a silent single-node fallback with no error, a missing helm install on bare-metal nodes, and an RBAC gap for non-admin org members.

KV Cache Offload (New)​

  • Choose where the KV cache lives - GPU, CPU (system RAM), or NVMe, each via a standalone LMCache cache server, so the cost/performance tradeoff can be measured instead of assumed.
  • New multiroundqa tool benchmarks offload under realistic multi-turn chat traffic. KV cache tier is now reported across Pulse, the Excel Summary, and the Combined Export.
  • Stabilized after real hardware runs: fixed silent SSD metrics on LVM-backed disks, a sizing crash, a missing AMD/ROCm profile, and a container-teardown race that could carry a stale disk cache across a run boundary.

Hardware Telemetry​

  • Single collector for NVIDIA and AMD - replaces the separate NVML/ROCm/CPU collectors with all-smi.
  • Real system power - reads instantaneous power from the local BMC via ipmitool DCMI instead of an approximation.
  • Fleet cards now show system manufacturer and product name. Partial telemetry writes are reported as failures instead of counting as complete.
  • New Agentic Bench metrics stack (backend only; not yet customer-facing).

Dashboard, Landing Page and Project Creation (Redesign)​

Presentation and copy only - no API contract changes, no change to how a benchmark runs.

  • Dashboard - the four zero-value counters are replaced with a signal band: run outcomes over 7/28 days, and fleet capacity against plan quota. The workload catalog is now grouped by modality with headline models named per group.
  • Landing and pricing pages rebuilt around a live benchmark chart and a specification-sheet pricing layout.
  • Project creation reworked into four sequential stages (model, serving engine, hardware, load) instead of one flat field list. KYAI now uses the same panel.

Billing​

  • Plan-change proration fixed - was a live overcharge. Cancelling a subscription with proration doesn't credit the customer immediately, so the service read a zero balance and charged full price on the new plan. Fixed by finalizing the credit invoice at the moment of cancellation; the confirmation screen now shows the actual amount due before the customer commits.
  • Metering settle RPC fixed after stranding ~510 outbox rows for nine days. A MEC 409 on a meter claim is now treated as already-metered, not a failure.

Reporting (Pulse)​

  • Comparing two runs of the same tool now works - Lens sections group by tool code instead of workload name.
  • Meaningfully faster - offscreen charts defer until scrolled into view, and a rewritten query drops plan cost from ~5.5 trillion to under 1,000 on a representative dataset.
  • GPU/CPU/VRAM utilization and power charts now force their axis to start at zero.

Framework and Model Catalog​

  • vLLM 0.27.0, 0.27.1, and 0.28.0 onboarded. SGLang 0.5.17 and 0.5.18 onboarded.
  • New workloads now default to vLLM, not SGLang - a prior alphabetical-ordering accident is fixed.

Security​

  • All 16 outstanding high-severity findings resolved to zero, including a genuine TLS verification bypass in the RunPod integration.
  • Hardware onboarding RBAC corrected so any org member, not just admins, can group their own hardware into a multi-node cluster.

CI / Deploy​

  • Fixed a fresh-box production deploy failure and a migration that could hang a live deploy indefinitely with no error surfaced.

Work Items​

Metrum Insights 4.0.1 Release Notes​

ProductMetrum Insights
Version4.0.1
Previous version4.0.0
Release DateAugust 3, 2026

This patch fixes Tier 1/Tier 2 workload entitlement gap, a fresh-database access bug, billing admin lockout and several billing bugs, onboards vLLM 0.26.0 and SGLang 0.5.16, and adds a Stripe refund-webhook feature, a product logo rollout, and a redesigned pricing page.

Plan Enforcement​

  • Tier 1 organizations can no longer run Tier 2 (multimodal) workloads - A missing entitlement check let Tier 1 accounts execute VLM and other multimodal workloads. This is now enforced at the frontend, CLI, MCP, agent, and database level, including a locked state for sidebar items and workload chips outside the org's plan. Tier 1 organizations currently using these workloads will lose that access.
  • Fresh databases now grant correct access to active subscriptions - A schema and seed load-order bug left every subscription status, including active and trialing, on the access column's default value on a freshly built database, blocking every benchmark job. Fixed at the seed level, with a redundant and risk-prone backstop removed from the schema file and a regression test added. Live databases were not affected.

Billing​

  • Invoice PDF downloads fixed - A wrong table reference caused every invoice PDF download to return a 403.
  • Failed invoice downloads are now visible - A failed download previously failed silently; it now surfaces a visible error.
  • Usage meter now reports correctly - The Billing and Usage page showed "0 / N" for every organization because the entitlement fields it read did not exist on the underlying response.
  • Checkout no longer shows duplicate line items - Stripe Checkout displayed up to 4 line items per plan instead of 1. Metered items now attach via webhook, with a reconciliation pass for already-affected subscriptions and clearer overage disclosure before payment.
  • Refund contact link added for annual subscribers - The cancellation banner now shows a refund contact link for annual plans.
  • Charge refunds now route correctly - A new webhook path distinguishes a refunded add-on or seat-pack purchase from a refunded base-plan charge, so a Stripe Dashboard refund is handled correctly instead of being silently ignored. Required allowlisting the Stripe webhook URL in the reverse proxy for the fix to actually be reachable.
  • Billing URL now reaches the frontend - Checkout and plan status previously failed with "Billing is not enabled for this deployment" because the billing service URL was never passed into the frontend's container environment.

Frontend​

  • Workload framework selection no longer freezes - The workload code stopped updating after the first framework pick; a related issue that produced duplicate workload codes when copying a workload twice is also fixed.
  • Pricing page redesigned - /plans now shows four expanding tier tiles with live pricing instead of a flat grid, with a refreshed landing page.
  • Team invites appear without a refresh - The team member list now polls periodically so an accepted invite shows up automatically.
  • Insights product logo rolled out across the app header, landing page, checkout, invoices and reports, the CLI auth screen, the docs site, and email templates.

Security and Access Control​

  • Admin billing tools restored - The metrum_admin role check queried nonexistent columns and always returned false, locking staff out of the admin billing console, refund, activate, cancel, and addon endpoints. The fix also closes a related privilege escalation path that a naive query fix would have opened.
  • PostgREST query lint added to CI - Every frontend PostgREST select and filter is now checked against the real database schema, to catch this class of bug before it ships.

Framework and Model Catalog​

  • vLLM 0.26.0 onboarded - Carries forward LLM, VLM, and ASR modality support, consistent with every prior vLLM version.
  • SGLang 0.5.16 onboarded - Scoped to the LLM modality only for this release (metrumbench-llm and KYAI). VLM, ASR, and ImageGen are not covered for this SGLang version.

Metrum Bench License Extension & CI/CD Enhancements​

  • Extended the Metrum Bench tools' license validity through December 31, 2026.
  • Code deploys now require a manual dispatch trigger; docs continue to publish automatically on a version tag.
  • Fixed a docs release bug that published a placeholder control-plane URL on the live site.
  • Fixed a release packaging bug that broke make release for the Rust CLI tools right after the binary build step.
  • Added a new "Using Insights with an Agent" documentation page.

New Known Issue​

Work Items (engineering)​

Metrum Insights 4.0 — Release Notes

ProductMetrum Insights
Version4.0
Previous major3.9
Release DateJuly 29, 2026

Metrum Insights 4.0 is a ground-up architectural rewrite and the first broad-market release of the platform. Projects, workloads, runs, and results remain the core workflow language, but the control plane, results model, agent daemon, cloud lifecycle, quality evaluation (KYAI), reporting, and commercial surface are new.

Highlights​

  1. Architectural overhaul — New PostgreSQL control plane, PostgREST + RPC API, multi-tenant RBAC (Casbin-derived policies), rewritten agent daemon, and a Next.js App Router product UI. See #1927.
  2. New database relational schema — Typed tables, lookups, snapshots, immutable status history, natural-key external contracts. See #1926.
  3. New agent daemon — Rewritten metrum-agent for executing jobs on target hardwares. Live log shipping, crash recovery, multi-model execution, modality-aware runners. See #1928.
  4. KYAI — Two-phase generation + LLM judge, team-scoped datasets/prompts/endpoints, leaderboard, comparison views, context-length / truncation visibility, and dedicated tool-scoped engine-args sets.
  5. Multimodal benchmarking — LLM, VLM, ASR (Whisper on vLLM), and ImageGen (vLLM-Omni + SGLang Diffusion) as first-class workload types with modality-aware reporting. See #1934 and #1935.
  6. Pulse — Team benchmark intelligence workspace: curated views, filters, charts, sharing, permissions, and reportable-data eligibility.
  7. Commercialization stack — Stripe-backed plans, in-app checkout and upgrades, seat packs, target metering, invoices, payment method updates, dunning, entitlements / feature gates, and admin back-office — without redirecting users to Stripe Customer Portal for day-to-day billing. Tracked under #1647 and #1922.
  8. Built In InferenceX — In built support to run InferenceX benchmarking on supported hardwares.
  9. Operator reliability — Partial runs, live log shipping, crash recovery checkpoints, sequential multi-model execution, model-server port retry, runtime provenance, and production backup cadence. Backup/DR tracked under #985.
  10. Production UI — Customer-facing Next.js App Router UI with global search (Cmd+K), branded design system, gradient identity, dark/light themes, and full a11y. See #479.
  11. Industry-standard visualization dashboard — 18 new chart panels. See #804.
  12. Google TPU — Supports benchmarking on Google TPU with vLLM serving. See #1936.
  13. Hardware Leaderboard - User-facing Performance leaderboards with multi-mode ranking.

Workload × Framework × Hardware support​

LLM (text generation)​

FrameworkNVIDIA GPUAMD GPUGoogle TPUNotes
vLLM✅✅✅TPU supported Version: vLLM 0.24.0
SGLang✅✅❌ROCm engine-args sets seeded for 0.5.12–0.5.15.
TensorRT-LLM✅❌❌NVIDIA-only by design.
GenAI-Perf (tool)✅✅❌Third-party reference baseline; runs against OpenAI-compatible endpoints.
InferenceX (tool)✅✅❌Synthetic stress workloads.
KYAI (tool)✅✅❌Candidate endpoints on vLLM / SGLang / TRT-LLM.

VLM (vision-language)​

FrameworkNVIDIA GPUAMD GPUGoogle TPUNotes
vLLM✅✅❌vLLM 0.19.0+ serves MetrumBench VLM.
SGLang✅✅ (ROCm)❌OpenAI-compatible VLM endpoints.
TensorRT-LLM✅❌❌NVIDIA-only by design.

ASR (speech recognition)​

FrameworkNVIDIA GPUAMD GPUGoogle TPUNotes
vLLM✅✅❌Currently limited Whisper family
SGLang❌❌❌Not supported currently.
TensorRT-LLM❌❌❌Not supported currently.

ImageGen (text-to-image)​

FrameworkNVIDIA GPUAMD GPUGoogle TPUNotes
vLLM (Omni)✅✅❌Requires omni-image-gen profile / vllm-metrumbench-imagegen args set.
SGLang (Diffusion)✅✅ (ROCm)❌ROCm engine-args completeness across 0.5.10–0.5.15.
TensorRT-LLM❌❌❌Not supported

Hardware notes​

  • Google TPU: Supported for MetrumBench LLM workloads on vLLM only. VLM, ASR, and ImageGen on TPU are not currently validated.
  • AMD GPU: Supported for LLM, VLM, and ImageGen via SGLang ROCm builds and vLLM ROCm paths. TRT-LLM is NVIDIA-only.
  • Multi-node: Currently not supported in this version.

What's new​

Architecture overview​

For full engineering maps, see tracking issues #1926, #1927, #1928.

PostgreSQL is the single source of truth. A single REST and RPC API serves the UI, the CLI, and the agent, so there is one endpoint to reach and no message broker to run. Every request is authenticated and access-checked at the API layer.

Runtime topology​

ServiceRole
postgresPostgreSQL 18.x primary DB
postgrestREST/RPC gateway (metrum_anon / metrum_authenticated)
insights-frontendNext.js App Router UI + BFF
provisionerCloud GPU provisioning; PG NOTIFY listener
mcpserverFastMCP tools for agent console
caddyReverse proxy
Agent / workersHost or containerized metrum-agent daemon
Commercialization stackBilling service + addon schema

Control plane API​

  • 3.x REST replaced by PostgREST + RPC (/rpc/{name}).
  • Frontend: RTK Query against BFF / PostgREST.
  • JWTs minted server-side; PATs with bounded lifetimes; proxies at /api/postgrest and /rpc/[...path].
  • Every request: JWT → rbac_pre_request → Casbin (rpc_permission_map).

Database schema​

  • The schema is typed and relational. Lookup tables replace database enums, hardware and runtime snapshots make every result reproducible, and run history is immutable. Results are read through stable views, so external tools keep working across upgrades.
  • Numbered idempotent files under schema/ (lookups, identity, hardware, models, frameworks, benchmarks, traceability, KYAI, cloud, InferenceX, Pulse, views, RPCs).
  • Lookups (lkp_*) instead of ENUMs; polymorphic subtypes; immutable status history; hardware/runtime snapshots.
  • Application results via views (v_benchmark_results, v_inferencex_outputs, …), not legacy benchmark_results.
  • Migrations for durable DBs under migrations/; commercial tables may live in addons/commercialization-stack/schema/.

Agent daemon​

  • The rewritten agent joins with a one-time onboarding token and runs jobs on your hardware. It ships logs during the run, checkpoints its work so a crash does not lose a job, runs multiple models one after another to avoid GPU contention, and records exactly which image, runtime, and model produced each result. Warmup and measurement adapt to the workload type (LLM, VLM, ASR, ImageGen).
  • Join via onboarding token; benchmark and KYAI workers via CLI.
  • Job lifecycle, model-server management, live/incremental log shipping, crash checkpoints, mid-run cancel.
  • Sequential multi-model execution; per-worker run serialization.
  • Runtime provenance (image digest, runtime version, model metadata); SUT host facts.
  • Modality paths: MetrumBench LLM/VLM/ASR/ImageGen, InferenceX, KYAI (plus external dispatcher where configured).

Multimodal workloads​

Related tracking: #1934 (text generation), #1935 (frameworks).

ASR (Automatic Speech Recognition)​

  • Speech-to-text benchmarking on the same scenario-matrix model as LLM.
  • Product scope in 4.0: Whisper family, served on vLLM only (SGLang / TensorRT-LLM not productized for ASR).
  • Audio duration measured from FLAC/WAV headers so RTF is never zero when samples carry duration.
  • ASR-specific reporting charts and KPI cleanup on the reporting surface.

VLM (Vision-Language)​

  • Image-to-text benchmarking via metrumbench-vlm.
  • Chart parity with LLM where metrics apply; VLM-only response-time and image-throughput charts; max_tokens tab filters; ISL/OSL bucket fallback for VLM charts.
  • Image preprocessing time included in response-latency semantics where applicable.
  • Reporting parity for MetrumBench VLM (#1895).

ImageGen (text-to-image) — new in the 4.0 train​

  • Serving support for vLLM-Omni and SGLang Diffusion
  • Megapixels/sec and size / n dimensions on monitoring, combined export, and charts.
  • ImageGen-specific concurrency defaults, health warmup modality detection, and success-rate-gated job status (not bare process exit code).
  • Multi-replica reliability fixes for SGLang / vLLM-Omni ImageGen paths.

Google TPU​

Related tracking: #1936.

  • TPU v5 / v6 as supported targets via GKE-managed inference.
  • On-prem TPU startup contract materialization fixes for reliable bootstrap (#1594, #1595).
  • End-to-end LibTPU telemetry supervision, accelerator-aware Excel export, and TPU chart rollups (#1885).
  • vLLM TPU build profile seeded into the catalog.

Pulse — team benchmark intelligence workspace​

  • A team workspace for turning raw runs into shared intelligence. Select source jobs, build a named view, filter and chart it, and share it with your team. Views always read fresh data, and access follows each person's role.
  • Full Pulse product surface: Curator (source job selection), Views, Lens (filters + table + charts), share/revoke, rename, permissions UI.
  • SECURITY DEFINER RPCs, reportable-data eligibility helpers, curator facets, and migration path for upgrade-only environments (#1924).
  • Chart availability derived per workload; tool tab strip; Cmd+K actions for Pulse.
  • Viewer-safe permission gating (Curator hidden for viewer-only users).
  • Known residual UI polish tracked: row expand (#1519), Clear selection (#1520), filter option completeness (#1521), paired baseline/candidate mode (#1585).

Hardware Leaderboards​

  • User-facing Performance leaderboards with multi-mode ranking.
  • RBAC-mapped leaderboard RPCs
  • Hardware leaderboard mode mapping, cohort-metadata RPC, and invoker-rights fixes for anonymous throughput leaderboard callers.

Multi-tenant identity and RBAC​

  • One account, many workspaces; role per workspace (Owner, Admin, Engineer, Viewer).
  • API-layer permission checks via rbac_pre_request + Casbin-derived policies.
  • Org membership lifecycle emails (member joined / removed) for owners.
  • Auth0 org connection enablement and lazy-heal on team invite creation.
  • No PostgreSQL RLS as product tenancy enforcement — Casbin only (#848).
  • Project visibility badges and visibility_code exposed as string codes in list/get RPCs (#1513).
  • Invitation URL host rewritten on invitations list endpoint (#1904).
  • Stale org-switch server cache fixed so team switches refresh available servers (#1557).
  • execute_project_run validates org ownership of server_id (#1554).
  • Hostname resolution for execute_scenario / BYOE / KYAI runs scoped to the project org (#1966).
  • Direct PostgREST table-write exposure on billing-critical tables hardened (#1680).

Billing, metering, and payments​

Related tracking: #1647, #1922, #1797, #1925.

  • Plans and checkout
    • Public /plans page and /checkout/[planCode] flow (#1925).
    • Self-serve Tier-1 plan changes via dedicated plan-change session (cancel-with-credit before Checkout). Card transactions only leave the product for Stripe Checkout (#1893, #1797).
    • Real DB-backed prices (no hardcoded plan prices in UI). Checkout fails closed until Stripe price IDs are provisioned (#1631).
    • Plan upgrade double-charge / missing proration credit fixed (#1909).
    • Customers can undo a scheduled cancellation.
  • In-app billing surface
    • Production billing dashboard: plan card, usage rings, seat widget, invoices, scheduled-cancellation banner, past_due / dunning states. Manage Billing stays in custom UI, not Stripe Customer Portal (#1797).
    • Invoice list and PDF proxy stay on-domain; payment method updates use embedded Stripe Payment Element.
    • Seat packs (add-on attach), seat-cap hard enforcement, overage states. Paid seat packs increment seat total rather than replace it.
    • Header Plan Status Pill reads as active or actionable.
    • Bootstrap, project create/save/execute, and onboarding command minting require an active plan.
  • Metering
    • Successful-run-oriented metering with completion-hook pattern.
    • Registered targets metering via transactional outbox (targets_included on plans). Commercial quota now enforces registered target count, not raw run count (#1798).
    • Cloud GPU cost accrual into MEC / Stripe with kill-switch GUC on accrue_cloud_costs.
    • Fail-closed MEC metering with explicit escape hatch configuration (#1684).
    • Overage delivered through Stripe Billing Meter Events (#1932).
    • Recovery path for completed runs that never reached the metering outbox.
    • Multi-replica double-report protection for Stripe usage (#1625).
    • Stripe webhook retries no longer silently dropped after a failed first delivery (#1612).
  • Entitlements and gating
    • useEntitlements + FeatureGate; SubscriptionGuard banner.
    • Subscription entitlement gate enforced on server operations (#1613).
    • Nav gating examples: Sizer / Leaderboard visibility by tier. Sizer entitlement enforced server-side.
    • Read access split from the right to start billable work.
    • One approved failed-payment policy shared by every gate.
  • Notifications
    • Billing and team lifecycle email templates (plan upgraded, seat pack purchased, member events, etc.).
    • Activation email follows provisioning, not bare payment.
    • Notification flush job; Geist fonts + deployment-aware logo URLs; every email CTA points at the deployment domain.
    • Stale connections no longer stall invite emails for minutes (#1903).
  • Admin / ops
    • Admin billing back-office panels and BFF routes with operator attribution.
    • Stripe/DB/MEC reconciliation with auto-repair and operator alerts.
    • Operational audit log for billing actions and failures.
    • Health probes (stripe_prices_configured, cloud-cost accrual health).
    • Target-limit guard gated to service and admin callers; no longer raises tenant billing detail to unauthorized callers.
    • Commercialization-stack addon wired into both deploy pipelines.

Email notifications​

  • Expanded template set for run/billing/team lifecycle events.
  • Backed by AWS SES service

Agent console and MCP​

  • Console agent operational skills and metrum-mcp-call control-plane path.
  • External KYAI dispatcher service for no-GPU judge/generation paths.
  • Caddy bearer guard patterns for agent-facing surfaces.
  • Agent registration unblocked; heartbeat write path closed.
  • Credentials persisted before metering wait so JWT is not lost on timeout.

Onboarding and UX polish​

  • Single command onboarding for target hardwares
  • Save-as-new for projects with execution history; active-run guards; per-workload hardware selection (#1547, #1840, #1752).
  • Workload code collision prevention on save-as-new (#1547, #1743).
  • GPU count control and engine-args reference guidance on WorkloadCard (#1823).
  • Multi-model warning + Models-above-Framework layout (#1823).
  • Runs panel restored on project detail; delete dialog position fixed (#1515, #1516).
  • Manual Job ID entry on Reporting empty state (#1518).
  • Monitoring page refreshes immediately after run dispatch (#1836).
  • Personal Access Token expiry chooser (30 / 90 / custom / never).
  • Automated Cloud Provisioning and KYAI External Endpoint disabled for 4.0 GA surface (#1920, #1879, #1611).
  • BYOE removed from plans, pricing, and dashboard entitlements for 4.0.

Upgraded features​

InferenceX​

  • New ingest pipeline and v_inferencex_outputs (legacy InferenceMax tables empty / retired).
  • Percentile coverage extended (including mean TPOT, TTFT p99, TPOT p90/p99/p99.9).
  • TP-aware tokens/sec aggregation to match tool-side aggregates.
  • Daemon-owned paths via cwd + env; result_dir / filename in the DB layer.
  • Upstream commit reference made database-driven.
  • Clean-architecture refactor of InferenceX agent paths with E2E chain fixes (#1562).
  • InferenceX chart parity with MetrumBench LLM (#1895, related E2E #1876, #1851, #1789, #1786).
  • Leave InferenceX tp NULL instead of defaulting to 1; document disaggregated gap (#1929, #1930).

KYAI​

  • Phase 1 generation + Phase 2 LLM judge.
  • Results Explorer filters; Endpoints & Judges as first-class objects.
  • Team-scoped custom datasets, prompts, and endpoints; soft-delete; snapshot-only datasets with canonical codes.
  • Per-dataset judge prompts; encrypted API keys; GPQA Diamond and real HF seed paths.
  • Build Profile picker on endpoint create; free-form external endpoint URLs.
  • Context length: native vs live length recorded per generation job; truncation facts on jobs table and leaderboard; client-side risky edit warnings (#1813).
  • Dedicated tool-scoped engine-args sets for vLLM / SGLang / TRT-LLM; vLLM 0.25.x onboarded for KYAI paths.
  • Comparison panel for multi-run visualization; Excel/PDF KYAI report routes.
  • External dispatcher for GPU-less judge workloads.
  • Dataset upload: preserve external_row_id; robust CSV comma/newline parsing; required-column validation on file select.
  • 100-row deterministic eval mode (#1603); restic content loader (#1602); zero-shot prompt packs; methodology docs updates (#1901).
  • KYAI External Endpoint evaluation disabled in UI/API for 4.0 GA (#1879, #1920).
  • Residual visualization suite request: #1816.

Benchmarks / Reporting (results explorer)​

Related: #804 (18-chart design), #1847 (reporting overhaul), #1895 (GenAI-Perf / VLM / ASR parity), #1921 (assigned-GPU telemetry).

  • Primary results surface at /benchmarks (product name Benchmarks; "Pulse" also names the team workspace feature above).
  • Filter bar, My runs toggle, chart sub-tabs (Model, Framework, Quantization, Concurrency, Cost, Hardware, plus modality-specific tabs).
  • PostgREST range pagination for large result sets.
  • Reporting overhaul (#1847)
    • Drill-down filtering, segmented metric controls, "how to read" tooltips.
    • GenAI-Perf chart parity with MetrumBench LLM; full statistic set capture; num_requests formula alignment (#1877).
    • MetrumBench VLM/ASR/ImageGen chart and KPI parity work (#1895).
    • Telemetry charts: GPU power, VRAM util, CPU util, CPU package power (RAPL / amd_energy), cold-start throughput.
    • Assigned-GPU averaging + CPU collector in fallback path (#1921).
    • Multi-model series grouping; SUT identity columns; hardware summary in PDF (#1677, #1648).
    • GPU utilization shows percentage, not raw fraction (#1676).
    • Reporting System row shows all distinct hardware configs (#1555).
    • Excel: full-fidelity multi-sheet export, tensor type / dtype, power context, SUT dedup; ZIP download of all reports.
    • Branded server-rendered PDF with active filter tracking; per-workload PDF capture scoping.
    • Chart export with filter support.
    • Manual Job ID entry on empty state (#1518).
    • Cmd+K entries for Excel/PDF export.
    • Analytics scoped to executions for performance; execution-group fan-out removed.

Agent daemon reliability​

Related tracking: #1928. Specifics: #1419 (multi-model scheduling), #1509 (agent reliability), #1595 (TPU startup), #1410 (partial run metrics), #1485 (server restart lifecycle), #1784 (TRT-LLM port race), #1710 (AMD ROCm CUDA image), #1723 (SGLang ASR concurrency crash), #1593 (docker_cli single-GPU).

  • Live Docker log tail + incremental ~30s log shipping during jobs.
  • Setup/teardown dedicated log stream codes; ship logs on failure paths.
  • Daemon crash recovery via job checkpoint files; pending log retry queue.
  • Periodic mid-execution cancellation checks.
  • Sequential multi-model execution; serialize runs per worker to avoid GPU contention (#1419).
  • Multi-model job scheduling deadlock fixed (#1716).
  • Model metadata capture per served model (HF config.json from container); runtime precision after healthcheck.
  • Runtime provenance: image digest + runtime version from startup logs (#1191).
  • Model-server port conflict retry across adapters; restart server on KV profile changes.
  • Stop stale managed server from prior run before launch.
  • Scenario model-server restart lifecycle fixed (#1485).
  • TRT-LLM scenario restart port race (port 8000) fixed (#1784).
  • Human-readable, secret-safe failure messages.
  • Warmup inference modality-aware (ASR / LLM / VLM / ImageGen).
  • Measure ASR audio duration on worker for correct RTF.
  • Partial benchmark success preserves metrics even when some jobs exit non-zero (#1410).
  • AMD ROCm workers fixed (were launching with NVIDIA CUDA container images) (#1710).
  • SGLang Whisper ASR concurrency crash fixed (#1723).
  • docker_cli adapter exposed only a single GPU despite tensor_parallel_size=2 — fixed (#1593).
  • tool_invocations.invocation_started_at could land after completed_at — fixed (#1486).

Framework and model catalog​

Related tracking: #1935, #1379 (prompt loader via PostgREST), #1475 (staging seed drift), #1756 (vLLM 0.21), #1702, #1546, #1218 (precision clarification).

  • Expanded model catalog (90+ models across modalities) with sort order and display names.
  • Framework train includes (non-exhaustive): vLLM through 0.25.x line items (e.g. 0.16, 0.18–0.24, 0.21, 0.23, 0.25), SGLang 0.5.12–0.5.15 family, TRT-LLM 1.2.x, Dynamo profiles, CUDA build profiles (cu130 default, cu129, aarch64 where applicable).
  • Verified recipe / BKC argsets (e.g. Qwen, MiniMax, Nemotron, Kimi families).
  • Tool-scoped engine-args sets; command template goldens; version sort overflow fix.
  • SGLang ImageGen ROCm catalog and engine-args completeness.
  • Removal of dead vllm_bench tool from seed/UI; BYOE workload creation hidden where retired from product path.
  • Re-enabled SGLang and TensorRT benchmark support after earlier gaps.
  • Prompt loader uses PostgREST API, not direct Postgres (#1379).
  • Model and KV cache precision clarified in reports/results (#1218).
  • "Streaming" setting ignored for genai-perf jobs fixed (#1788).
  • Workload title uses "vLLM" for all frameworks in reports fixed (#1777).
  • genai-perf full statistic capture, num_requests formula alignment (#1877).
  • SGLang 0.5.15: no engine args set for metrumbench-vlm (#1848).

Monitoring and run lifecycle​

  • Partial run status when non-zero exits still yield ingested metrics (#1410).
  • Per-model scenario rows with job-count progress; per-hardware breakdown.
  • Failure pills and scenario_status / has_failures on scenario lists.
  • Instance monitoring and failure log evidence surfaces.
  • Project edit flow improvements; log completeness workstreams.
  • Runs dispatched via execute_scenario RPC now correctly appear in Monitoring (execution_group_id path) (#1606).
  • execute_project_run guards update_project_config against projects with active runs (#1840, #1752).
  • Multi-hardware monitoring: HARDWARE column shows all GPUs; per-hardware status drilldown fixed (#1679).
  • Monitoring page refresh immediately after run dispatch (#1836).
  • Run history stays live for projects with data from org-scoped server dispatch.

Personal access tokens (PATs)​

  • PATs with bounded JWT lifetimes; create/manage via API.
  • Bearer pass-through proxy at /rpc/[...path]; PostgREST proxy at /api/postgrest.
  • Caller binding fixes so tokens attach to the creating account.

Backup and DR​

Related tracking: #985.

  • Production backup timer consolidated to a single 24h calendar schedule (redeploy does not reset cadence).
  • Staging DB backup timer with Restic upload on deploy paths.
  • DR drill / backup-doctor target wiring hardened.

CI / deploy / ops​

  • common-dev staging deploy workflow; Rust tool binary build + agent package sync on deploy.
  • RC promotion switched to PR-based flow (#1918).
  • Clean DB regression lane hardening; lane dedup; frontend install consolidation.
  • Runtime setup: parallel data population, timed colored step output, schema rebuild SQL logs.
  • Compose project/volume isolation; .env as sole config source; deterministic container names.
  • make migrate-lint catch for addon migration reverting a core function (#1965).
  • Static load-order defect detection before any database exists (#1922).
  • One shared service-JWT minter; expired static tokens fail production startup.
  • Dead env vars cleaned from env.example (#1576, #1577, #1579).

External documentation​

  • Docs restructure; KYAI methodology and user guide updates; GenAI-Perf methodology page.
  • Performance methodology docs; control-plane install/admin guides.

Known limitations​

  • Multi-node benchmarking is not supported. All 4.0 benchmarking are single-node.
  • TPU: LLM on vLLM is validated (v5e build profile). VLM, ASR, and ImageGen on TPU are not validated.
  • ASR is limited to the Whisper family on vLLM only.
  • KYAI External Endpoint evaluation is disabled.
  • BYOE is currently not supported for metrumbench-llm
  • Synks reported 8 high issues (non-critical)

Work Items (engineering)​

FeatureGitHub Issue ID
UI production design and development#479
Architecture overhaul (control plane, API, multi-tenant platform)#1927
New database schema (relational control-plane redesign)#1926
New agent daemon (worker lifecycle, onboarding, job execution)#1928
Text generation benchmarking (metrumbench-llm and genai-perf)#1934
Inference frameworks (vLLM, SGLang, TensorRT-LLM)#1935
Hardware support (NVIDIA, AMD, Google TPU)#1936
KYAI (two-phase quality evaluation and leaderboard)#1988
Multimodal ImageGen (vLLM-Omni, SGLang Diffusion)#1989
Pulse (team benchmark intelligence workspace)#1924
Hardware Leaderboard#1468
InferenceX#1990
Benchmarks and Reporting overhaul#1847
Multi-tenant identity and RBAC#848
Commercialization: billing, metering, payments#1647, #1922
Visualization dashboard#804
Backup and restore#985
Testing and validation tracker#946