Skip to main content

Hardware Compatibility

This guide lists the accelerators Metrum Insights supports as of v4.0, broken down by workload type and inference backend, with the driver and firmware floors you need to hit before onboarding a server.

The goal is for a hardware engineer or pre-sales engineer to be able to answer "can we run benchmark X on box Y?" without digging through release notes or asking on Slack.

Items marked with an asterisk (*) are listed as supported but are subject to hardware availability - confirm with your Metrum account team before committing a deployment to them.

Quick lookup​

If you just want the headline answer:

AcceleratorLLMVLMASRKYAI candidateKYAI judgeNotes
NVIDIA B200✅✅✅✅✅Highest-end target; PCIe and SXM
NVIDIA B300✅✅✅✅✅Newer than B200; same support matrix
NVIDIA H200✅✅✅✅✅Default for most reference benchmarks
NVIDIA H100✅✅✅✅✅Long-tested; widest framework coverage
AMD MI355X✅✅✅✅✅vLLM and SGLang only (no TRT-LLM)
Google TPU v5e✅⚠️⚠️✅✅GKE-managed; VLM/ASR experimental
Google TPU v5p✅⚠️⚠️✅✅Same as v5e
Intel GPU*⚠️❌❌⚠️❌LLM only; subject to availability
AWS Inferentia*⚠️❌❌⚠️❌LLM only; subject to availability

Legend: ✅ supported, ⚠️ supported but with caveats (see per-row notes below), ❌ not supported in v4.0.

Support by workload type​

LLM benchmarking (metrumbench-llm)​

Backends: vLLM (single-node), SGLang, TensorRT-LLM.

AcceleratorvLLMSGLangTensorRT-LLMRecommended for
NVIDIA B200✅✅✅Frontier-scale models (≥70B params)
NVIDIA B300✅✅✅Frontier-scale models
NVIDIA H200✅✅✅Large models (30B-70B), KV-cache-heavy
NVIDIA H100✅✅✅General LLM benchmarking, 7B-70B
AMD MI355X✅✅❌NVIDIA-comparison studies, AMD deployments
TPU v5e / v5p✅❌❌GKE-native deployments, JAX/PAX models
Intel GPU*⚠️❌❌LLM-only experimental
AWS Inferentia*⚠️❌❌LLM-only experimental

Caveats:

  • TensorRT-LLM is NVIDIA-only by design. Don't expect MI355X, TPU, Intel, or Inferentia support.
  • AMD MI355X on vLLM requires the ROCm build of vLLM. Set framework version to a ROCm-tagged release in the workload card (e.g. 0.6.3-rocm).
  • TPU on vLLM uses the vLLM TPU backend; the framework-version dropdown will only surface TPU-compatible builds when the selected server is a TPU node.
  • Intel and Inferentia are best-effort in v4.0 - they pass the validation gate, but advanced features (FP8 quantization, structured output) may not work.

VLM benchmarking (metrumbench-vlm)​

AcceleratorvLLMSGLangTensorRT-LLMNotes
NVIDIA B200✅✅✅
NVIDIA B300✅✅✅
NVIDIA H200✅✅✅Recommended default
NVIDIA H100✅✅✅
AMD MI355X✅✅❌
TPU v5e / v5p⚠️❌❌Experimental - image preprocessing path may differ

VLM benchmarks include image preprocessing time in TTFT. Image preprocessing runs on CPU (not the accelerator), so CPU SKU and core count materially affect TTFT for image-heavy workloads. The Sizer Guide takes this into account.

ASR benchmarking (metrumbench-asr)​

AcceleratorvLLMSGLangTensorRT-LLMNotes
NVIDIA H100✅✅✅Recommended; widest model coverage
NVIDIA H200✅✅✅
NVIDIA B200✅✅✅
NVIDIA B300✅✅✅
AMD MI355X✅✅❌
TPU v5e / v5p⚠️❌❌Experimental - audio decode path not fully validated

ASR uses the same scenario-matrix model as LLM (concurrency × ISL × OSL), where ISL maps to audio duration buckets internally. See the User Guide → ASR for the bucket mapping.

KYAI candidate endpoints​

Candidate endpoints are inference servers being evaluated by KYAI. The hardware they run on follows the LLM/VLM matrix above. BYOE endpoints can run anywhere - Metrum doesn't enforce hardware constraints on remote endpoints.

KYAI judges​

Judges are LLMs run by Metrum to score candidate outputs. They are practical only on accelerators that can host a capable judge model (typically a 70B+ instruct model) at reasonable throughput:

  • Recommended: H100, H200, B200, B300, MI355X.
  • Not recommended: TPU v5e, Intel, Inferentia - these will run smaller judges but at throughputs that make large evaluations slow and expensive.

Driver and firmware floors​

Minimum versions required for each inference backend. These are floors, not optimums - newer is usually better.

NVIDIA (B200, B300, H100, H200)​

ComponentMinimumRecommended
NVIDIA driver550.54570.86 or later
CUDA toolkit12.412.6
cuDNN9.09.4
NCCL2.212.23
Container toolkitnvidia-container-toolkit 1.151.17
Fabric Manager (SXM nodes)matches drivermatches driver

For B200/B300 specifically: the driver floor is 555.42, not 550.54 - the older driver doesn't enumerate Blackwell devices correctly.

TensorRT-LLM has tighter floors: CUDA 12.5+ and a TRT-LLM release built against your driver. Mismatches surface as a pre-run validation error with a specific version suggestion.

AMD (MI355X)​

ComponentMinimumRecommended
ROCm6.26.3
AMDGPU drivermatches ROCm releasematches ROCm release
RCCLbundled with ROCmbundled with ROCm

ROCm 6.2 is the floor for MI355X-aware vLLM and SGLang builds. ROCm 6.1 will load but lacks tuned kernels for MI355X.

Google TPU (v5e, v5p)​

Driven through GKE - Metrum manages the node pool, JAX/XLA versions, and runtime images. End users don't install drivers directly.

Required for the GKE side:

  • GKE node pool with TPU v5e or v5p shape.
  • Workload Identity enabled on the cluster (required by Shadeform).
  • The Metrum-managed TPU node image - provisioned automatically on first use.

Intel GPU (subject to availability)​

ComponentMinimum
Intel GPU driver24.13
oneAPI Base Toolkit2024.1
Intel Extension for PyTorch (IPEX)2.3

Intel support in v4.0 is LLM-only and considered experimental. Expect some pre-run validation surprises until the validator gets full Intel coverage in v4.1.

AWS Inferentia (subject to availability)​

ComponentMinimum
Neuron SDK2.20
Neuron driver2.18
transformers-neuronxlatest matching Neuron SDK

Inferentia uses a different compilation model from GPU paths - models must be pre-compiled for the target Neuron core count before benchmarking. See the User Guide → BYOE for the typical pattern.

Picking a target​

A few rules of thumb when you have flexibility on which accelerator to benchmark:

  • Comparing across frameworks. Pick H100 or H200 - every framework targets them and the results are widely cited externally.
  • Comparing across hardware. Hold the framework constant (vLLM is the most portable) and vary the accelerator.
  • Frontier models (≥70B params, long context). B200, B300, or H200 - H100 will work but you'll hit memory limits sooner.
  • Cost-efficiency studies. Include MI355X ~ it consistently lands competitively on tokens-per-dollar on the Benchmarks page Cost chart.
  • TPU evaluations. Use v5p for raw performance numbers, v5e for cost-efficiency framing.

When in doubt, run Sizer - it answers "for this workload, which SKU should I be benchmarking on?" against the most recent platform-wide measurements.

How compatibility is enforced​

You don't have to memorize this table - Metrum enforces compatibility at two points:

  1. Pre-run validation. When a run starts, the validator checks the selected server's hardware snapshot against the workload's framework/model requirements. Incompatible combinations fail fast with a structured error pointing at the specific mismatch.
  2. Sizer recommendations. Sizer only recommends accelerators that pass the compatibility check for the requested workload type.

If you hit a pre-run validation failure that looks wrong - i.e. the matrix above says it should work but Metrum disagrees - file it via the AI Support Agent. The validator errs on the side of refusing borderline configurations.

See also​