ASR benchmarking (metrumbench-asr)
ASR stands for Automatic Speech Recognition: models that turn audio into text, such as Whisper. The metrumbench-asr tool measures both how fast a transcription endpoint runs and, when you provide reference transcripts, how accurate it is.
Supported models and frameworks
ASR benchmarking is intentionally scoped to one model family and one serving framework:
| Dimension | Supported | Notes |
|---|---|---|
| Model family | Whisper only (OpenAI Whisper family, for example openai/whisper-large-v3) | This is the only ASR model family seeded on the platform. Other speech-to-text models are not supported. |
| Framework | vLLM only (framework_code: vllm) | vLLM serves the Whisper transcription endpoint that metrumbench-asr targets. SGLang and TensorRT-LLM are not productized for ASR. |
If you pass any other model family or framework for an ASR workload, the run will not resolve a valid command template. Keep p_model_code on a Whisper model and p_framework_code set to vllm.
What it measures and when to use it
metrumbench-asr uploads audio files to an OpenAI-compatible transcription endpoint (/v1/audio/transcriptions) and records timing and accuracy for each file. Use it to answer:
- Can this model transcribe faster than real time on this hardware?
- How accurate is it, in Word Error Rate, on my kind of audio?
- How does accuracy trade off against speed across quantizations or frameworks?
The headline metrics are RTF (Real-Time Factor), WER (Word Error Rate), and CER (Character Error Rate). They are defined in ASR-specific metrics below.
Step 1: pick or prepare a dataset
An ASR benchmark needs audio to transcribe. Two built-in datasets are seeded:
| Dataset code | What it is |
|---|---|
librispeech-test-clean | The LibriSpeech "test-clean" set: clear, read English speech. The standard baseline. |
librispeech-test-other | The LibriSpeech "test-other" set: noisier, harder speech. Use it to stress accuracy. |
The audio files are stored centrally and restored to the benchmark worker during agent bootstrap, so you do not upload them yourself when using a built-in dataset.
To use your own audio, supply an input manifest: a JSONL file where each line describes one audio file.
{"id": "sample-001", "path": "/data/audio/sample-001.flac", "format": "flac", "duration": 12.5}
{"id": "sample-002", "url": "https://storage.example.com/sample-002.mp3", "format": "mp3", "duration": 8.3}
idis required and must be unique.- Provide either
path(a local file on the worker) orurl(downloaded and cached before the run). formatsets the upload MIME type. Supported: MP3, WAV, WebM, OGG, M4A/MP4, FLAC.duration(seconds) is needed to compute RTF. Without it, RTF is skipped for that file.
Step 2: prepare ground truth (optional, for accuracy)
Accuracy metrics (WER and CER) are computed only when you provide a ground truth file: a JSONL where each line pairs an audio id with its reference transcript.
{"id": "sample-001", "transcript": "the quick brown fox jumps over the lazy dog"}
{"id": "sample-002", "transcript": "a second reference transcription"}
The built-in LibriSpeech datasets ship with reference transcripts, so WER and CER are available out of the box. Without ground truth, you still get full speed metrics, just no accuracy.
Step 3: create the workload and scenario (API)
The same ASR workload and scenarios can be created over the API. See metrumbench-asr Workload And Scenario and Project Lifecycle for the API reference.
ASR-specific arguments
| Argument | What it means | How to choose |
|---|---|---|
concurrency | Audio files transcribed at the same time. | Start at 8, raise to find saturation. |
num_requests | Total transcriptions to send. Audio samples cycle round-robin if requests exceed the sample count. | Hundreds for stable numbers. |
input | The audio manifest (built-in dataset resolves this for you). | Use a built-in dataset unless testing your own audio. |
ground_truth | Reference transcripts for accuracy. | Provide it to get WER and CER. |
response_format | The transcription response shape: verbose_json, json, text, srt, or vtt. verbose_json returns the richest data. | Leave at the default unless you need a specific format. |
There is no ISL, OSL, or streaming for ASR. The traffic shape is defined by the audio itself and by concurrency.
ASR-specific metrics
| Metric | What it means | Good value |
|---|---|---|
| RTF (Real-Time Factor) | Inference time divided by audio duration. | Below 1.0 means faster than real time. 0.1 means ten times faster than real time. |
| WER (Word Error Rate) | The fraction of words that must be inserted, deleted, or substituted to match the reference. | Lower is better. 0% is perfect. It can exceed 100% if the model produces far more words than the reference. |
| CER (Character Error Rate) | The same idea at the character level. More granular than WER. | Lower is better. |
| Audio per second | Seconds of audio processed per wall-clock second across all concurrency. | Higher is better. |
WER and CER appear only when ground truth is provided. RTF appears only when the manifest includes duration.
Common pitfalls
- No duration in the manifest. RTF is skipped silently. Add
durationto every line. - Expecting WER without ground truth. Accuracy needs reference transcripts. Use a built-in dataset or supply your own ground truth file.
- Reading accuracy on the wrong audio. Test-clean flatters a model; test-other is closer to real-world noise. Report which set you used.