Skip to main content

ASR benchmarking (metrumbench-asr)

ASR stands for Automatic Speech Recognition: models that turn audio into text, such as Whisper. The metrumbench-asr tool measures both how fast a transcription endpoint runs and, when you provide reference transcripts, how accurate it is.

Supported models and frameworks​

ASR benchmarking is intentionally scoped to one model family and one serving framework:

DimensionSupportedNotes
Model familyWhisper only (OpenAI Whisper family, for example openai/whisper-large-v3)This is the only ASR model family seeded on the platform. Other speech-to-text models are not supported.
FrameworkvLLM only (framework_code: vllm)vLLM serves the Whisper transcription endpoint that metrumbench-asr targets. SGLang and TensorRT-LLM are not productized for ASR.

If you pass any other model family or framework for an ASR workload, the run will not resolve a valid command template. Keep p_model_code on a Whisper model and p_framework_code set to vllm.

What it measures and when to use it​

metrumbench-asr uploads audio files to an OpenAI-compatible transcription endpoint (/v1/audio/transcriptions) and records timing and accuracy for each file. Use it to answer:

  • Can this model transcribe faster than real time on this hardware?
  • How accurate is it, in Word Error Rate, on my kind of audio?
  • How does accuracy trade off against speed across quantizations or frameworks?

The headline metrics are RTF (Real-Time Factor), WER (Word Error Rate), and CER (Character Error Rate). They are defined in ASR-specific metrics below.

Step 1: pick or prepare a dataset​

An ASR benchmark needs audio to transcribe. Two built-in datasets are seeded:

Dataset codeWhat it is
librispeech-test-cleanThe LibriSpeech "test-clean" set: clear, read English speech. The standard baseline.
librispeech-test-otherThe LibriSpeech "test-other" set: noisier, harder speech. Use it to stress accuracy.

The audio files are stored centrally and restored to the benchmark worker during agent bootstrap, so you do not upload them yourself when using a built-in dataset.

To use your own audio, supply an input manifest: a JSONL file where each line describes one audio file.

{"id": "sample-001", "path": "/data/audio/sample-001.flac", "format": "flac", "duration": 12.5}
{"id": "sample-002", "url": "https://storage.example.com/sample-002.mp3", "format": "mp3", "duration": 8.3}
  • id is required and must be unique.
  • Provide either path (a local file on the worker) or url (downloaded and cached before the run).
  • format sets the upload MIME type. Supported: MP3, WAV, WebM, OGG, M4A/MP4, FLAC.
  • duration (seconds) is needed to compute RTF. Without it, RTF is skipped for that file.

Step 2: prepare ground truth (optional, for accuracy)​

Accuracy metrics (WER and CER) are computed only when you provide a ground truth file: a JSONL where each line pairs an audio id with its reference transcript.

{"id": "sample-001", "transcript": "the quick brown fox jumps over the lazy dog"}
{"id": "sample-002", "transcript": "a second reference transcription"}

The built-in LibriSpeech datasets ship with reference transcripts, so WER and CER are available out of the box. Without ground truth, you still get full speed metrics, just no accuracy.

Step 3: create the workload and scenario (API)​

The same ASR workload and scenarios can be created over the API. See metrumbench-asr Workload And Scenario and Project Lifecycle for the API reference.

ASR-specific arguments​

ArgumentWhat it meansHow to choose
concurrencyAudio files transcribed at the same time.Start at 8, raise to find saturation.
num_requestsTotal transcriptions to send. Audio samples cycle round-robin if requests exceed the sample count.Hundreds for stable numbers.
inputThe audio manifest (built-in dataset resolves this for you).Use a built-in dataset unless testing your own audio.
ground_truthReference transcripts for accuracy.Provide it to get WER and CER.
response_formatThe transcription response shape: verbose_json, json, text, srt, or vtt. verbose_json returns the richest data.Leave at the default unless you need a specific format.

There is no ISL, OSL, or streaming for ASR. The traffic shape is defined by the audio itself and by concurrency.

ASR-specific metrics​

MetricWhat it meansGood value
RTF (Real-Time Factor)Inference time divided by audio duration.Below 1.0 means faster than real time. 0.1 means ten times faster than real time.
WER (Word Error Rate)The fraction of words that must be inserted, deleted, or substituted to match the reference.Lower is better. 0% is perfect. It can exceed 100% if the model produces far more words than the reference.
CER (Character Error Rate)The same idea at the character level. More granular than WER.Lower is better.
Audio per secondSeconds of audio processed per wall-clock second across all concurrency.Higher is better.

WER and CER appear only when ground truth is provided. RTF appears only when the manifest includes duration.

Common pitfalls​

  • No duration in the manifest. RTF is skipped silently. Add duration to every line.
  • Expecting WER without ground truth. Accuracy needs reference transcripts. Use a built-in dataset or supply your own ground truth file.
  • Reading accuracy on the wrong audio. Test-clean flatters a model; test-other is closer to real-world noise. Report which set you used.