Skip to main content

Benchmark building blocks

Before the workload guides, here is the vocabulary. Every benchmark in Metrum AI Bench is built from five nested objects. Once these five click, every screen and every API call reads the same way.

ObjectPlain-language meaningExample
ProjectA named folder for one benchmarking question. It holds everything below it.llama-70b-h100-throughput
WorkloadOne thing you want to measure. It is a single model, served by a single framework version, driven by one benchmark tool. Change the model, framework, version, quantization, or tool, and you have a new workload.Llama 3.1 70B on vLLM 0.20.0, driven by metrumbench-llm
ScenarioOne traffic shape applied to a workload. It sets how many requests run at once, how long the inputs and outputs are, and how many requests to send.c8-isl512-osl256
RunOne execution of a workload across its scenarios, on a server you choose.Run 1 of the workload above
JobOne scenario executing inside a run. A workload with 6 scenarios produces 6 jobs per run.The c8-isl512-osl256 job

Read it as a sentence: a project groups workloads; each workload is measured under one or more scenarios; executing a workload creates a run; each scenario in that run becomes a job.

The tool that generates load is chosen per workload, based on the kind of model. This guide covers the three first-party tools plus InferenceX, a third-party LLM serving benchmark:

ToolWorkload typeWhat it drives
metrumbench-llmText language models (LLM)An OpenAI-compatible chat or completion endpoint
metrumbench-asrSpeech-to-text models (ASR)An OpenAI-compatible audio transcription endpoint
metrumbench-vlmVision-language models (VLM, image plus text)An OpenAI-compatible chat endpoint with image payloads
inferencexText language models (LLM), synthetic loadAn OpenAI-compatible chat endpoint, with GPU-normalized metrics

The first-party tools send real traffic to a running model server, measure every request, and write the results back to the platform. InferenceX uses synthetic fixed-length token sequences and reports throughput per GPU. The Performance Methodology and InferenceX Methodology explain exactly how each metric is captured.