Benchmark building blocks
Before the workload guides, here is the vocabulary. Every benchmark in Metrum AI Bench is built from five nested objects. Once these five click, every screen and every API call reads the same way.
| Object | Plain-language meaning | Example |
|---|---|---|
| Project | A named folder for one benchmarking question. It holds everything below it. | llama-70b-h100-throughput |
| Workload | One thing you want to measure. It is a single model, served by a single framework version, driven by one benchmark tool. Change the model, framework, version, quantization, or tool, and you have a new workload. | Llama 3.1 70B on vLLM 0.20.0, driven by metrumbench-llm |
| Scenario | One traffic shape applied to a workload. It sets how many requests run at once, how long the inputs and outputs are, and how many requests to send. | c8-isl512-osl256 |
| Run | One execution of a workload across its scenarios, on a server you choose. | Run 1 of the workload above |
| Job | One scenario executing inside a run. A workload with 6 scenarios produces 6 jobs per run. | The c8-isl512-osl256 job |
Read it as a sentence: a project groups workloads; each workload is measured under one or more scenarios; executing a workload creates a run; each scenario in that run becomes a job.
The tool that generates load is chosen per workload, based on the kind of model. This guide covers the three first-party tools plus InferenceX, a third-party LLM serving benchmark:
| Tool | Workload type | What it drives |
|---|---|---|
metrumbench-llm | Text language models (LLM) | An OpenAI-compatible chat or completion endpoint |
metrumbench-asr | Speech-to-text models (ASR) | An OpenAI-compatible audio transcription endpoint |
metrumbench-vlm | Vision-language models (VLM, image plus text) | An OpenAI-compatible chat endpoint with image payloads |
inferencex | Text language models (LLM), synthetic load | An OpenAI-compatible chat endpoint, with GPU-normalized metrics |
The first-party tools send real traffic to a running model server, measure every request, and write the results back to the platform. InferenceX uses synthetic fixed-length token sequences and reports throughput per GPU. The Performance Methodology and InferenceX Methodology explain exactly how each metric is captured.