Quickstart

This guide takes you from creating a new Metrum Insights account all the way to viewing your first benchmark results. Follow each step in order. The whole thing takes under 20 minutes once your GPU server is reachable.

What you'll do

1. Sign up and create your account
2. Choose a subscription plan and complete payment
3. Confirm a GPU server is available
4. Create a project with a single workload
5. Save and execute the run
6. Read the results on the Reporting page

If you already have an account and an active plan, skip to Confirm a server is available.

Before you start

You'll need:

• A valid email address or a Google account.
• A credit card (Visa, Mastercard, or Amex) if you are subscribing to a paid plan. Team and Org plans are also available on annual invoice via our sales team.
• At least one registered GPU server visible to your workspace. If you don't see any servers under Infrastructure in the sidebar after signing up, ask your platform admin to onboard one before continuing.
• A rough idea of what you want to measure. For this Quickstart we'll benchmark Llama 3.1 70B Instruct on vLLM at three concurrency levels ~ a straightforward hardware health check.

---

1. Sign up

Open https://insights.metrum.ai in your browser. The app redirects unauthenticated visitors to the sign-up and sign-in page.

You have two ways to create an account:

Option A: Sign up with Google

This is the fastest path.

1. Click Continue with Google on the sign-in page.
2. Select the Google account you want to use, or sign in to Google if you are not already signed in.
3. On your first visit, Google prompts you to grant Metrum Insights access to your name and email address. Click Continue to allow it.
4. You are redirected back to Metrum Insights and land on the Dashboard. Your account is created automatically using the name and email from your Google profile.

There is no email verification step when you sign up with Google.

Option B: Sign up with email and password

1. On the sign-in page, click Sign up (below the Google button).
2. Enter your email address and choose a password. Passwords must be at least eight characters long.
3. Click Continue. A verification email is sent to the address you entered.
4. Open the email from noreply@auth.metrum.ai and click the Verify your email link. The link is valid for 24 hours.
5. After verifying, you are redirected back to Metrum Insights and land on the Dashboard.

Did not receive the email? Check your spam folder. The sender domain is auth.metrum.ai. If it is not there after a few minutes, return to the sign-in page and click Resend verification email.

Signing in later

Once your account exists, return to https://insights.metrum.ai at any time. Click Continue with Google if you signed up with Google, or enter your email and password if you used the email path. The app also offers a Forgot password link if you need to reset your credentials.

After signing in, you land on the Dashboard, which shows four KPI tiles (Projects, Active Jobs, Servers Online, Active Runs) and five widgets (Throughput Leaderboard, Recent Activity, Server Fleet, Recent Runs, and Cloud). These will all be empty on first login ~ that is expected.

The left-hand sidebar is your primary navigation, with a secondary Dashboard subnav below the sidebar header. The pages you will touch in this Quickstart are Infrastructure (sidebar), Projects (Dashboard subnav), and Reporting (Dashboard subnav).

Choose a plan and complete payment

If your workspace does not yet have an active subscription, pick a plan and complete payment before running benchmarks. Go to the pricing page (/plans), choose a plan, select a workspace, and finish checkout on Stripe. After payment you return to the Billing page, where an activation checklist shows progress until the workspace is ready.

For the full list of plans, what each one includes, and the step-by-step payment flow, see Billing, Plans and Metering.

---

2. Choose a plan and complete payment

A subscription plan is required to run benchmarks and register servers. Metrum Insights does not charge for your compute ~ the plan covers access to the platform only. Your infrastructure bill goes directly to your provider.

Plans overview

Navigate to Plans at https://insights.metrum.ai/plans, or click the Upgrade prompt that appears in the sidebar when you are on a free account.

Plan	Best for	Billing
Solo Monthly	Individual users evaluating the platform or running occasional benchmarks.	Monthly, cancel any time.
Solo Annual	Individual users who benchmark regularly. Includes a 30-day refund window.	Annual, paid upfront.
Team Annual	Teams of 2 to 10 engineers sharing a workspace.	Annual, paid upfront.
Org Annual	Large teams or organizations needing multiple workspaces and higher limits.	Annual, paid upfront.

A seat is a user who creates or runs benchmarks. A target is one registered hardware endpoint or server. Read-only members who only view reports do not consume a paid seat.

Not sure which plan to choose? Start with Solo Monthly. You can upgrade to Team or Org Annual at any time. Unused time on your current plan is converted to a credit and applied automatically to the first invoice on the new plan.

Subscribe to a plan

1. On the Plans page, find the plan you want.
2. Click Subscribe on that plan's card.
3. You are taken to the Stripe payment page. This is the only step that opens outside the Metrum Insights domain.
4. On the Stripe page:
   • Enter your card number, expiration date, and CVV.
   • Enter the cardholder name and billing address.
   • Review the plan name, billing period, and total amount at the top of the form.
   • Click Subscribe (or Pay) to confirm.
5. Stripe processes the payment. On success, you are redirected back to Metrum Insights.
6. Your plan is now Active. The Billing section in your workspace settings (reachable from the top-right account menu) shows your plan name, renewal date, included seats, and included targets.

Payment failed? Common causes: incorrect CVV, a card that requires 3D Secure authentication (your bank may send an SMS code you need to enter), or a card declined for international transactions. Try again with a different card or contact your bank.

Managing your subscription after sign-up

From the Billing page in your workspace settings you can:

• Change plan - upgrade or downgrade at any time. The change takes effect immediately: your current plan ends, its unused time becomes a credit, and you are taken to Stripe to pay for the new plan. You see the exact credit and new charge before you confirm anything.
• Update payment method - change your card without leaving Metrum Insights.
• View invoices - a full list of past invoices with PDF download links.
• Cancel - cancels at the end of the current billing period. Access continues until the period ends.

---

3. Confirm a server is available

---

Before creating the project, verify there's a GPU server you can target.

1. Click Infrastructure in the sidebar (URL: /infrastructure).
2. You should see at least one server row in the Servers table with status Online and a recent heartbeat.
3. Note the Hostname and Config Code of the server you want to use ~ you'll select it in step 5.

Screenshot placeholder: Infrastructure page with one online server.

If no servers are listed, click Add Server to register one yourself (you'll need shell access to the GPU box), or ask an admin to do it. The bootstrap flow is covered in the Admin Guide.

If a server is listed but offline, the agent isn't reporting heartbeats. Check that the metrum-agent service is running on the host and that the machine can reach the control plane.

---

4. Create a project

A project in Metrum Insights is a named container for one or more workloads. Each workload describes a model + framework + scenario matrix to benchmark.

1. Click Projects in the Dashboard subnav.
2. Click New Project in the top-right corner of the projects page.

You'll land on /dashboard/projects/new.

Screenshot placeholder: New Project page, empty state.

Fill in the project section

Field	Value for this Quickstart
Project Name	Llama 3.1 70B Quickstart
Visibility	Private (the form default is Organization; select Private for this Quickstart)

Visibility controls who in your workspace can see the project:

• Private ~ only you.
• Organization ~ anyone in your workspace with the appropriate role. This is the form default.
• Public ~ visible across the platform (rare; usually reserved for reference benchmarks).

Set it to Private for now ~ you can change it later.

Fill in the workload card

A new project starts with one empty workload card titled Workload 1. Configure it as follows:

Field	Value
Workload Name	llama-70b-vllm-baseline
Model	pick a Llama 70B-class entry from the Model dropdown (the dropdown lists the registered model catalog)
Framework	vLLM
Version	0.24.0 (stable) (selectable once Framework is set)
Concurrency chips	toggle on 1, 8, 32
Input Sequence Length chips	toggle on 512
Output Sequence Length chips	toggle on 256
Requests per scenario	10 (the default; total requests per scenario)
Streaming	leave checked (default on)

A few notes:

• The Scenario Matrix section uses chip-button multi-select rather than free text. Available concurrency chips are 1, 4, 8, 16, 32, 64, 128. ISL chips are 128, 256, 512, 1024, 2048. OSL chips are 64, 128, 256, 512. Selecting 1, 8, 32 for concurrency with one ISL and one OSL produces a 3×1×1 = 3-scenario matrix.
• Click Show details under the workload card to preview the rolled-out scenarios; they're named c<concurrency>-isl<isl>-osl<osl> (for example c1-isl512-osl256).
• Requests per scenario controls the total number of requests fired per scenario. Streaming controls whether the benchmark consumes the response as a server-sent stream.
• Each workload card also offers Add Benchmark and Add KYAI Run buttons for adding additional benchmark or quality-evaluation steps. Leave these alone for the Quickstart.

Don't add a second workload yet ~ Save and Execute only works when there's exactly one workload (see the note below).

Pick the server

Scroll to the Server section. You should see the server you confirmed in step 3 as a card with a radio button. Each card shows the hostname on the first line and the server config code on the second line (for example localhost / local-dummy).

1. Click the card to select it.
2. Both action buttons at the bottom-right (Save Project and Save and Execute) should now be enabled. A Reset button sits next to them.

Screenshot placeholder: Workload and server filled in, Save and Execute enabled.

Why only one workload?

Save and Execute runs the project immediately on the selected server. To keep that simple and reliable, the action is restricted to projects with a single workload and an explicit server. Multi-workload projects are saved as drafts via Save Project and executed later via the run flow ~ covered in the User Guide.

---

5. Save and execute

Click Save and Execute.

You'll be redirected to the Project Detail page at /dashboard/projects/<id>, then to the Run Detail page once the run is created. The header shows:

• Run name ~ auto-generated from the workload (e.g. llama-70b-vllm-baseline-r1)
• Status badge ~ starts as pending, transitions to queued, then running
• Started at / Last updated ~ refreshed on each status change

The Jobs table lists one row per scenario. Job names follow the pattern c<concurrency>-isl<isl>-osl<osl>. For our 3-scenario matrix you'll see three rows:

Job name	Tool	Concurrency	ISL	OSL
c1-isl512-osl256	metrumbench-llm	1	512	256
c8-isl512-osl256	metrumbench-llm	8	512	256
c32-isl512-osl256	metrumbench-llm	32	512	256

Screenshot placeholder: Run detail with three jobs running.

Watching progress

You can stay on the Run Detail page (it polls for status updates) or switch to the Monitoring page for a live view of every running job across the platform.

A healthy run progresses through these statuses on each job:

pending → queued → running → completed

If a job lands in failed, the Tool column links to the job's logs. The most common first-run failure is a model download timeout (Llama 3.1 70B is ~140 GB in fp16); give the server a few minutes if it's the first time pulling this model. If the failure looks more serious, see Troubleshooting (post-launch).

For Llama 3.1 70B on vLLM, expect each scenario to take a few minutes on a single H100/H200 node depending on concurrency, plus a one-time model load on the first job.

---

6. Read the results in Reporting

Once at least one job is completed, head to Reporting in the Dashboard subnav (/dashboard/reporting). The page subtitle reads "Full results across all benchmark runs. Filter, compare, and analyse performance."

Screenshot placeholder: Reporting page with results from the Quickstart run.

Filter to your run

The Reporting page exposes a Filters dropdown and a My runs toggle pill at the top of the results table.

1. Click Filters to open the panel and narrow by run/project/model/framework as needed.
2. Toggle My runs on to restrict to runs you launched.

You should see your three jobs in the results table. Click a row to expand its detailed metrics.

Read the key metrics

Four numbers will tell you most of what you need to know:

• Throughput (tokens/sec) ~ total output tokens generated per second across all concurrent requests. Should scale up as concurrency rises, until you saturate the GPU.
• TTFT P50 / P99 ~ Time To First Token. P50 is typical user experience; P99 is your tail latency. Both grow with concurrency.
• TPOT ~ Time Per Output Token after the first one. This is the steady-state generation speed.
• GPU power / utilization ~ from telemetry. A healthy benchmark pushes utilization above 80% at the higher concurrency levels.

Visualize across scenarios

Scroll below the table to the chart panel. The chart sub-tabs are Model, Framework, Quantization, Concurrency, Cost, and Hardware. Open the Concurrency tab to see throughput and latency plotted as a function of concurrency. The classic shape is:

• Throughput rises sharply, then flattens ~ that's your saturation point.
• P99 latency stays flat, then climbs ~ that's queueing.

The concurrency at the elbow of the throughput curve is the rough "sweet spot" for this hardware at this sequence-length shape.

---

What's next

You've now run a single-workload benchmark end-to-end. From here:

• Run a richer matrix. Add more concurrency values, more ISL/OSL combinations, or a second framework (try SGLang or TensorRT-LLM on the same model) to compare side-by-side.
• Add a second workload. Same model, different quantization (e.g. fp16 vs fp8). The Quantization chart tab on Reporting will show you the throughput-vs-quality tradeoff.
• Evaluate quality with KYAI. Throughput numbers don't tell you whether the model's outputs are still good. KYAI runs a two-phase generation + LLM-judge pipeline. The KYAI execution path is wired into projects via the Add KYAI Run action on each workload card; a dedicated KYAI navigation surface is planned but not yet exposed in the UI.
• Run ImageGen or KYAI. See Image generation benchmarking and KYAI in the User Guide.

Common pitfalls

A few things first-time users trip over:

• Concurrency too high for the model. Setting concurrency to 128 on a 70B model with 80 GB of GPU memory will OOM. Start at 8, find the saturation point, then push higher.
• Mismatched ISL/OSL. If your real workload sends 4k-token prompts and you benchmark at 512, your numbers will be optimistic. Match the shape of production traffic.
• One server, many concurrent runs. Each run holds the GPU exclusively for the duration of its jobs. Queue up runs on the project; don't fire them off in parallel on the same server.
• Stale browser tab. The Run Detail page polls for updates, but if your tab has been backgrounded for a long time, refresh it to force a re-fetch.

Getting help

• Email notifications ~ planned per-user setting (the in-product Notifications panel is not yet exposed; see the User Guide for the current workaround).
• Docs ~ the User Guide covers every workflow this Quickstart skipped; the Feature Reference is the per-feature deep-dive.