Hardware Sizer
The Hardware Sizer turns benchmark data into deployment recommendations. It answers questions such as:
- Which GPU setup is a good fit for this model?
- Can our owned hardware meet a throughput or latency target?
- What is the cheapest viable recommendation, and which option leaves more growth headroom?
- Which model/runtime combinations can run on a fixed inventory?
The sizer is evidence-backed. It uses a benchmark dataset compiled by Metrum, catalog GPU metadata, pricing data, and model-stack information to rank recommendations.
Where To Find It
Open the Web UI and select Sizer in the application sidebar. The page has two working modes:
| Mode | Use it for |
|---|---|
| Tool | Structured inputs for model, inventory, workload, objectives, and constraints. |
| Chat | Natural-language sizing questions that the sizer turns into a sizing request. |
The Sizer page appears when the feature is enabled for your Metrum Insights instance.
Basic Workflow
- Choose a modality.
- Add at least one sizing signal that describes the question. Model, framework, precision, speculative decoding, owned inventory, hardware filters, workload inputs, objectives, and constraints can all be supplied independently.
- Add any other inputs that matter for the decision: owned inventory, workload mix, concurrency, target throughput, latency limit, budget, node limit, business horizon, hardware filters, tensor parallelism, or explicit GPU counts.
- Run the sizer tool.
- Review the primary recommendation, alternates, rejected candidates, evaluated workload profile, costs, bill of materials, and deployment notes.
When a complete model stack is supplied, the sizer can include a model-fit check alongside capacity, latency, cost, inventory, and hardware filtering based on the inputs you selected.
Model, Framework, And Precision Inputs
Modality is required, and the request also needs at least one sizing signal beyond modality. Model, framework, precision, and speculative decoding are optional constraints. You can provide just a model, just a framework, just a precision, or any combination of them.
| Input | Why it matters |
|---|---|
| Modality | Selects the type of sizing workflow. Use llm for text-generation serving recommendations. |
| Model | Narrows recommendations to a particular model when supplied. |
| Framework | Narrows benchmark matching and deployment notes to a serving stack when supplied. |
| Precision | Narrows benchmark matching and memory/cost estimates to a precision or quantization mode when supplied. |
| Speculative decoding | Adds a serving-method constraint when supplied. |
Leaving one of these fields unspecified keeps that dimension open. For example, a model-only request can search across available framework and precision coverage, while a framework-only request can evaluate dataset-backed model/runtime combinations for that stack.
Workload And Objective Inputs
Use workload and objective inputs when you want recommendations sized around a specific workload shape, throughput target, latency limit, budget, or node count.
| Input | Notes |
|---|---|
| Workload mix | Encodes workload categories such as coding assistant, customer support, agentic workflows, and RAG/search. Percentages must add up to 100 before the request can run. |
| Workload shape | Sends explicit concurrency. In explicit-token mode it also sends input sequence length and output sequence length. |
| Target TPS | Can represent total fleet tokens/sec, tokens/sec per request, tokens/sec per active user, or tokens/sec per active-user request. |
| Latency limit | Adds a maximum p99 end-to-end latency constraint. |
| Budget | Applies a dollar cap to the selected cost basis and annotates recommendations with available cost estimates, such as capex, monthly opex, and amortized monthly cost. |
| Node limit | Rejects candidates above the selected node count. |
| Business assumptions | Adds horizon and growth assumptions for future headroom. |
If token lengths are omitted, the sizer uses workload mix where possible and then falls back to the closest observed benchmark profile. The evaluated profile is returned in the response so the recommendation can be audited.
Owned Inventory
Use owned inventory when the question is constrained by hardware you already have. Add rows in the UI or import CSV with this exact header:
gpu_model,count
h200,16
b200,8
Rules:
- GPU models must exist in the sizer catalog.
- Number of user-owned GPUs must be a positive integer.
- Duplicate GPU model rows are rejected.
- Hardware filters must not exclude all owned inventory.
When owned inventory is selected with a model, the sizer ranks recommendations using only matching inventory. When owned inventory is selected without a model, it searches for model/runtime combinations that can run on the listed hardware.
Recommendation Output
A complete recommendation can include:
| Field | Meaning |
|---|---|
primary_recommendation | The selected candidate for the current sort and constraints. |
recommendations | Viable candidates ranked by recommendation strategy. |
rejected | Candidates that failed memory, latency, budget, inventory, or node constraints. |
fit_check | Model-fit summary and minimum compatible hardware when a complete model stack is supplied. |
evaluated_profile | The concurrency and token profile used to evaluate capacity. |
confidence | High, medium, or low signal based on matching Metrum dataset coverage. |
cost_source and cost_provenance | Where cost values came from. Seeded estimates are marked as estimates. |
system_config, deployment_plan, sensitivity, bom | Procurement and deployment detail when the chosen candidate has enough metadata. |
Warnings are included with the result. For example, if a budget is supplied and a GPU SKU has no price row, the sizer warns instead of treating the cost as known.
Chat Mode
Chat mode uses the current form context plus the conversation history to build a sizer request. Example prompts:
Owned inventory
I have 16 H200 GPUs. Which LLMs in Metrum's benchmark dataset can serve about 10k tokens/sec?
Model and runtime fit
Can Qwen3.6 Plus run with vLLM and fp8 on H200s, and what GPU count would you recommend?
Workload objective
Size an LLM deployment for a customer support workload with 32 concurrent requests and a 120 second p99 latency limit.
Follow-up refinement
First prompt:
Start with an LLM recommendation for Qwen3.6 Plus at 10k tokens/sec.
Follow-up prompt:
Now limit that recommendation to the 16 H200s we already own and show which candidates were rejected.
Chat can fill model names, owned inventory, tensor parallelism, and workload goals from the prompt, then run the same sizing workflow as the structured tool.
Troubleshooting
| Symptom | Meaning | Action |
|---|---|---|
| Sizer is unavailable | The feature is not enabled for your Metrum Insights instance, or Sizer is temporarily unavailable. | Retry later or ask your Metrum Insights administrator to check Sizer availability. |
| Request needs clarification | The request is missing a modality or at least one user-provided sizing signal. | Select a modality and add a sizing signal such as a model, inventory, hardware filter, workload input, objective, or constraint. |
| Exact coverage is unavailable | Metrum's sizing dataset does not yet cover the selected model, hardware, runtime, workload shape, or combination of constraints. | Use the closest supported options, relax exact-match filters, or contact Metrum about adding coverage. |
| Inventory validation error | A GPU row is blank, duplicated, unknown, or has a non-positive count. | Fix the owned inventory rows or CSV. |
API details are in Hardware Sizer API. Data coverage and admin guidance are in Sizer Dataset And Administration.