Skip to main content

Hardware Compatibility

This guide lists the accelerators Metrum Insights supports as of v4.0, broken down by workload type and inference backend, with the driver and firmware floors you need to hit before onboarding a server.

The goal is for a hardware engineer or pre-sales engineer to be able to answer "can we run benchmark X on box Y?" without digging through release notes or asking on Slack.

Items marked with an asterisk (*) are listed as supported but are subject to hardware availability - confirm with your Metrum account team before committing a deployment to them.

Quick lookup​

If you just want the headline answer:

AcceleratorLLMVLMASRImageGenKYAIGenAI-PerfInferenceXNotes
NVIDIA B200✅✅✅✅✅✅✅Highest-end target; PCIe and SXM
NVIDIA B300✅✅✅✅✅✅✅Newer than B200; same support matrix
NVIDIA H200✅✅✅✅✅✅✅Default for most reference benchmarks
NVIDIA H100✅✅✅✅✅✅✅Long-tested; widest framework coverage
AMD MI355X✅✅✅✅✅✅✅vLLM and SGLang only (no TRT-LLM)
Google TPU✅⚠️⚠️❌✅✅⚠️Google Cloud-managed; not limited to one TPU generation; VLM/ASR experimental

Legend: ✅ supported, ⚠️ supported but with caveats (see per-row notes below), ❌ not supported in v4.0.

Support by workload type​

LLM benchmarking (metrumbench-llm)​

Backends: vLLM (single-node), SGLang, TensorRT-LLM.

AcceleratorvLLMSGLangTensorRT-LLMRecommended for
NVIDIA B200✅✅✅Frontier-scale models (≥70B params)
NVIDIA B300✅✅✅Frontier-scale models
NVIDIA H200✅✅✅Large models (30B-70B), KV-cache-heavy
NVIDIA H100✅✅✅General LLM benchmarking, 7B-70B
AMD MI355X✅✅❌NVIDIA-comparison studies, AMD deployments
Google TPU✅❌❌Google Cloud deployments across TPU generations

Caveats:

  • TensorRT-LLM is NVIDIA-only by design. Don't expect MI355X or TPU support.
  • AMD MI355X on vLLM requires the ROCm build of vLLM. Set framework version to a ROCm-tagged release in the workload card (e.g. 0.6.3-rocm).
  • TPU on vLLM uses the vLLM TPU backend; the framework-version dropdown will only surface TPU-compatible builds when the selected server is a TPU node. Support is not limited to a single TPU generation or shape.

VLM benchmarking (metrumbench-vlm)​

AcceleratorvLLMSGLangTensorRT-LLMNotes
NVIDIA B200✅✅✅
NVIDIA B300✅✅✅
NVIDIA H200✅✅✅Recommended default
NVIDIA H100✅✅✅
AMD MI355X✅✅❌
Google TPU⚠️❌❌Experimental - image preprocessing path may differ

VLM benchmarks include image preprocessing time in TTFT. Image preprocessing runs on CPU (not the accelerator), so CPU SKU and core count materially affect TTFT for image-heavy workloads.

ASR benchmarking (metrumbench-asr)​

ASR is scoped to the Whisper model family only and the vLLM framework only. SGLang and TensorRT-LLM are not productized for ASR. The table below therefore lists accelerator support for the single supported path (Whisper on vLLM).

AcceleratorWhisper on vLLMNotes
NVIDIA H100SupportedRecommended; widest model coverage
NVIDIA H200Supported
NVIDIA B200Supported
NVIDIA B300Supported
AMD MI355XSupported
Google TPUExperimentalAudio decode path not fully validated

ASR uses the same scenario-matrix model as LLM (concurrency, then audio duration in place of ISL/OSL), where audio duration maps to duration buckets internally. See the User Guide, ASR for the bucket mapping.

Image generation benchmarking (metrumbench-imagegen)​

Image generation is served by vLLM-Omni and SGLang diffusion.

AcceleratorvLLM-OmniSGLang diffusionNotes
NVIDIA H100✅✅Recommended default
NVIDIA H200✅✅
NVIDIA B200✅✅
NVIDIA B300✅✅
AMD MI355X✅✅vLLM-Omni ROCm / SGLang ROCm (verified builds)

CUDA 13.0 is the default for ImageGen frameworks. ROCm support at 0.22.0 (vLLM-Omni) and 0.5.15.post1 (SGLang). See User Guide, ImageGen.

InferenceX benchmarking​

InferenceX is an LLM-only synthetic benchmark. It targets a single endpoint with replicas = 1 and tensor parallelism equal to the GPU count on the selected hardware. It supports vLLM and SGLang. TensorRT-LLM is not supported for InferenceX.

AcceleratorvLLMSGLangNotes
NVIDIA B200✅✅
NVIDIA B300✅✅
NVIDIA H200✅✅Recommended default
NVIDIA H100✅✅
AMD MI355X✅✅
Google TPU⚠️❌Experimental; vLLM TPU backend only

For full InferenceX methodology, see InferenceX Methodology.

KYAI​

KYAI evaluates model output quality. Candidates are self-hosted on the user's infrastructure. The Metrum AI Judge is platform-managed.

AcceleratorvLLMSGLangTensorRT-LLMNotes
NVIDIA H100✅✅✅Recommended default for candidates and judge
NVIDIA H200✅✅✅
NVIDIA B200✅✅✅
NVIDIA B300✅✅✅
AMD MI355X✅✅❌vLLM and SGLang only (no TRT-LLM)
Google TPU✅❌❌vLLM TPU backend only; judge not recommended

GenAI-Perf​

GenAI-Perf is a third-party client tool that drives load against any OpenAI-compatible chat endpoint. It is a synthetic-load tool (no prompt library). It inherits the serving framework's hardware support; any framework-compatible accelerator works.

AcceleratorvLLMSGLangTensorRT-LLMNotes
NVIDIA B200✅✅✅
NVIDIA B300✅✅✅
NVIDIA H200✅✅✅Recommended default
NVIDIA H100✅✅✅
AMD MI355X✅✅❌vLLM and SGLang only (no TRT-LLM)
Google TPU✅❌❌vLLM TPU backend only (SGLang TRT-LLM not available)

Driver and firmware floors​

Minimum versions required for each inference backend. These are floors, not optimums - newer is usually better.

NVIDIA (B200, B300, H100, H200)​

ComponentMinimumRecommended
NVIDIA driver550.54570.86 or later
CUDA toolkit12.412.6
cuDNN9.09.4
NCCL2.212.23
Container toolkitnvidia-container-toolkit 1.151.17
Fabric Manager (SXM nodes)matches drivermatches driver

For B200/B300 specifically: the driver floor is 555.42, not 550.54 - the older driver doesn't enumerate Blackwell devices correctly.

TensorRT-LLM has tighter floors: CUDA 12.5+ and a TRT-LLM release built against your driver. Mismatches surface as a pre-run validation error with a specific version suggestion.

AMD (MI355X)​

ComponentMinimumRecommended
ROCm6.26.3
AMDGPU drivermatches ROCm releasematches ROCm release
RCCLbundled with ROCmbundled with ROCm

ROCm 6.2 is the floor for MI355X-aware vLLM and SGLang builds. ROCm 6.1 will load but lacks tuned kernels for MI355X.

Google TPU​

Driven through Google Cloud - Metrum manages the node pool, JAX/XLA versions, and runtime images. End users don't install drivers directly.

TPU support is broad across Google TPU generations and shapes available through the platform's Google Cloud path. It is not limited to a single TPU type.

Required on the Google Cloud side:

  • Google Cloud node pool with a supported TPU shape for the selected generation.
  • Workload Identity enabled on the cluster (required by Shadeform).
  • The Metrum-managed TPU node image - provisioned automatically on first use.

Picking a target​

A few rules of thumb when you have flexibility on which accelerator to benchmark:

  • Comparing across frameworks. Pick H100 or H200 - every framework targets them and the results are widely cited externally.
  • Comparing across hardware. Hold the framework constant (vLLM is the most portable) and vary the accelerator.
  • Frontier models (≥70B params, long context). B200, B300, or H200 - H100 will work but you'll hit memory limits sooner.
  • Cost-efficiency studies. Include MI355X ~ it consistently lands competitively on tokens-per-dollar on the Reporting page Cost chart.
  • TPU evaluations. Pick the TPU generation and shape that match the deployment you are modeling. Support is not limited to one TPU type.

How compatibility is enforced​

You don't have to memorize this table - Metrum enforces it automatically. When a run starts, the validator checks the selected server's hardware snapshot against the workload's framework/model requirements. Incompatible combinations fail fast with a structured error pointing at the specific mismatch.

If you hit a validation failure that looks wrong - i.e. the matrix above says it should work but Metrum disagrees - file it via the AI Support Agent. The validator errs on the side of refusing borderline configurations.

See also​