Skip to main content

Sizer Usage Guide

Sizer is Metrum Insights' hardware recommendation tool. You tell it what you need to run; it tells you what to buy or provision, sized against the platform's measured benchmark library. The live UI subtitle reads "AI-powered GPU sizing for LLM, VLM, ASR, and image generation workloads."

This guide is written for sales engineers, pre-sales engineers, and hardware engineers who need to answer the question "what hardware do I need for this use case?" in a customer conversation, not in a 3-week benchmarking project.

If you're benchmarking specific hardware you already have, Sizer is the wrong tool ~ use Projects + Benchmarks instead.

The Sizer page (/sizer) has two tabs:

  • Configure ~ the structured form. This is where the work happens in v4.0.
  • Assistant ~ a natural-language entry point. In v4.0 the Assistant tab requires a configured AI assistant service; without it the tab shows "AI assistant is not available."

When to use Sizer​

Sizer is appropriate when you have:

  • A model in mind (or at least a size class, e.g. "around 70B params").
  • A target workload shape (concurrent users, prompt and response lengths).
  • A latency or throughput SLA (e.g. "P95 TTFT under 500 ms" or ">= 5000 tokens/sec aggregate").

Sizer is not appropriate when:

  • The workload is novel and Metrum has no nearby reference benchmarks (Sizer extrapolates, but only so far).
  • The customer's question is really "is this specific deployment fast enough?" ~ that's a benchmarking question, not a sizing question.
  • Quality matters more than throughput ~ Sizer optimizes for capacity, not output quality. Pair it with KYAI if the customer cares about both.

Solve modes​

The live Configure form has two top-level solve modes:

  • Recommend GPU ~ classic Sizer flow. Given the workload + SLA, return ranked hardware configurations.
  • Predict Metrics ~ inverse direction. Given a fixed hardware configuration (GPU SKU + count + parallelism), predict the throughput / TTFT / TPOT it will deliver.

Pick the mode that matches the question you're answering. The rest of the form adapts to the chosen mode.

Inputs (Configure tab)​

The Configure form is grouped into a small number of panels. The exact field set is evolving; the live form is the source of truth, but the panels you'll find today are:

Modality​

  • LLM (default), VLM, ASR, or Image Generation.

The modality picks a different reference benchmark surface and a slightly different set of downstream inputs (e.g. images_per_request for VLM, audio duration buckets for ASR).

Model​

Pick from the model catalog. You can either name a specific model (llama-3.1-70b-instruct) or a model class (70B instruct, dense). Specific is better ~ Sizer will fall back to the class only if it has no measurements for the exact model.

If the model is custom or proprietary, see Custom models below.

Framework and precision​

  • Framework: vLLM, SGLang, TensorRT-LLM, and Dynamo are selectable in v4.0. Note that Dynamo is selectable in Sizer today but the run path is not yet wired end-to-end (see Release Notes).
  • Precision / quantization: fp16, bf16, fp8, int8, int4, awq, gptq. Defaults to "best available" ~ Sizer picks the most efficient option that meets the latency SLA.
  • Speculative decoding: optional toggle. Enable when the candidate framework + model pair supports it.

Use case mix​

Sizer accepts a multi-bucket use case mix as a set of percentage sliders. Buckets typically include:

  • Chat assistant ~ interactive, latency-sensitive, short to medium prompts.
  • Document QA / RAG ~ long prompts, short responses, retrieval-bound.
  • Code completion ~ short prompts, short responses, very latency-sensitive.
  • Batch generation ~ throughput over latency; tolerates queueing.
  • Other ~ fall through to the raw ISL/OSL/concurrency inputs.

Adjust the sliders so they sum to 100. The mix primes reasonable defaults for the per-bucket sequence-length distribution.

Target load​

Set one of:

  • Concurrent users ~ Sizer will translate this to concurrency based on the mix (e.g. for chat, ~5 simultaneous in-flight requests per 100 concurrent users at steady state).
  • Aggregate tokens/sec ~ direct target for the deployment's output throughput.

If you have both, use the more binding constraint.

Latency SLA​

The latency target the customer must hit. Two fields:

  • TTFT (P95) ~ typical chat target: 300-800 ms.
  • TPOT (P95) ~ output token generation latency, typical chat: 30-60 ms (about 16-33 tokens/sec/user).

If only one matters, leave the other blank; Sizer won't optimize for it.

Sequence length​

  • ISL (input) ~ average input length in tokens. Chat: ~500-1500. RAG: 2000-8000. Long-context: 16000+.
  • OSL (output) ~ average output length. Chat: ~200-500. Code: ~100-300. Long-form: 1000+.

If the customer's workload has wildly variable shapes, run Sizer twice (once at the median, once at the 95th percentile) and present both.

Parallelism (advanced)​

Sizer exposes optional parallelism overrides:

  • Tensor parallel size (TP)
  • Pipeline parallel size (PP)
  • Expert parallel size (EP) for MoE models
  • Disaggregated prefill + decode placement, including separate TP/PP/EP for each stage

Leave these blank to let Sizer pick. Set them when the customer has a fixed deployment topology in mind.

Hardware constraint​

Optional. Limit Sizer to a vendor or SKU family ~ e.g. "NVIDIA only", "exclude Blackwell" (for customers without datacenter power upgrades), or "AMD-friendly options too".

Cost / horizon / growth​

Sizer also accepts cost-side inputs to drive the TCO view:

  • Cost horizon in months (e.g. 12, 24, 36).
  • Expected growth rate as a percentage per period.
  • Optional cost overrides for the candidate SKUs.

Use these when the customer's question is "what does this cost over the next X months?" rather than purely "what hardware fits the SLA?".

JSON override (power user)​

A JSON Override panel lets advanced users replace any of the structured inputs with raw JSON. This is the same shape Sizer accepts via its API and is useful for reproducing a sizing run from a saved configuration or comparing two runs precisely.

Output (Recommend GPU mode)​

Sizer returns a ranked list of viable configurations. Each entry contains:

  • Hardware ~ accelerator SKU and quantity (e.g. "2x NVIDIA H200").
  • Framework + quantization ~ recommended runtime config.
  • Parallelism layout ~ TP/PP/EP and disaggregated split if applicable.
  • Estimated metrics ~ predicted throughput, TTFT P95, TPOT P95 at the requested concurrency.
  • Confidence ~ Sizer's confidence in the prediction (high / medium / low), based on how close the measured reference benchmarks are to the requested shape.
  • Cost per million tokens and a TCO summary over the cost horizon when cloud pricing data is available for the SKU.
  • Notes ~ caveats like "VLM preprocessing is CPU-bound on this config; pair with high-core-count CPU".

The default ranking is cheapest-that-meets-SLA. Toggle the sort to favor headroom, lowest latency, or lowest hardware count if the customer cares about something else.

Output (Predict Metrics mode)​

Sizer returns the predicted throughput / TTFT / TPOT for the configuration you supplied, plus the same confidence and cost summary. Use this to validate "we already have N x H200; will it meet our SLA?" without iterating on hardware choices.

Reading Sizer output​

A few patterns to watch for:

  • Two configurations with the same hardware but different quantizations. Sizer is telling you the customer has a quality-vs-cost choice. Surface both.
  • A low-confidence recommendation. Sizer's reference benchmarks don't tightly match the requested shape. Either propose running a real benchmark before committing, or widen the SKU options.
  • No viable configurations. The SLA is unreachable with current hardware at the requested concurrency. Loosen the SLA, lower the concurrency, or accept multi-node (which Sizer doesn't size in v4.0).
  • High headroom on the recommended config. The customer can probably grow into this deployment ~ useful framing for capacity-planning conversations.

Worked example: Llama 3.1 70B for 100 concurrent users, P95 TTFT under 500 ms​

This is the canonical sales-engineering scenario.

Inputs​

FieldValue
Solve modeRecommend GPU
ModalityLLM
Modelllama-3.1-70b-instruct
FrameworkvLLM
PrecisionBest available
Use case mixChat assistant 100%
Concurrent users100
Aggregate tokens/sec(leave blank ~ concurrency is the binding constraint)
TTFT P95500 ms
TPOT P9550 ms
ISL1024
OSL300
HardwareNVIDIA only
Cost horizon12 months
Growth rate0% per month

Output (illustrative)​

Sizer returns three viable configurations:

RankHardwareFramework + QuantTTFT P95 (est)TPOT P95 (est)Throughput (est)ConfidenceCost/MTok (est)
12x H200vLLM 0.6.3 / fp8380 ms38 ms4200 tok/sHigh$1.40
22x H100vLLM 0.6.3 / fp8470 ms44 ms3600 tok/sHigh$1.20
31x B200TRT-LLM 0.13 / fp8410 ms35 ms4400 tok/sMedium$1.80

Note: the numbers above are illustrative ~ your real Sizer output reflects the latest measurements on the Benchmarks page.

How to use this in a conversation​

  • Rank 1 (2x H200) is the safe default. High confidence, comfortably under SLA, mid-range cost.
  • Rank 2 (2x H100) is the budget option ~ still inside SLA but with less headroom for traffic growth. Good if the customer has H100 inventory or pricing.
  • Rank 3 (1x B200) is the "one box does it all" pitch. Lower hardware count, slightly higher cost per token, but easier to operate. Use this with customers who value operational simplicity.

If the customer pushes back on "is this real?" ~ every Sizer recommendation links back to the source benchmarks on the Benchmarks page that drove it. Click through to show them the measured data.

Custom models​

Sizer's recommendations come from measured benchmarks in the platform. For a custom model that Metrum hasn't measured:

  1. Match to a public reference. If the custom model is a Llama 3.1 70B finetune, Sizer's Llama 3.1 70B numbers are a good first approximation. Tell the customer it's an approximation.
  2. Match by architecture and size class. Same architecture (dense vs MoE), same parameter count, similar context length.
  3. Run a one-shot benchmark. If the customer needs a real number, onboard the custom model into Metrum, run a small benchmark, and re-run Sizer with the now-measured model. This is the slow path but the most accurate.

Sizer flags model substitutions in the Notes column so you don't accidentally present them as measured numbers.

Common pitfalls​

  • Over-trusting low-confidence recommendations. Sizer marks them for a reason. Treat low-confidence output as "directionally right, validate before quoting an SLA".
  • Forgetting multi-tenant overhead. Sizer sizes for the workload you describe. If the customer wants to run other models on the same hardware, the recommendation will be undersized.
  • Ignoring CPU and memory. For VLM workloads, the CPU and system RAM matter for image preprocessing. Sizer surfaces this in Notes but it's easy to skim past.
  • Quoting Sizer numbers as guarantees. They're predictions from measured benchmarks. The customer's actual workload will be close but not identical.
  • Skipping the latency floor. Some workloads have an unachievable latency target on current hardware (e.g. P95 TTFT < 100 ms on a 405B model). Sizer will return "no viable configurations" ~ don't try to coerce a result by loosening SLAs the customer actually cares about.

See also​