Skip to main content

VLM benchmarking (metrumbench-vlm)

VLM stands for Vision-Language Model: models that take an image plus text and produce text, such as describing a photo or reading a chart. The metrumbench-vlm tool measures how fast a VLM serves these image-plus-text requests.

What it measures and when to use it​

metrumbench-vlm sends the same kind of chat request as metrumbench-llm, but each request carries one or more images alongside the text prompt. Everything else, concurrency, timing, throughput, TTFT, and TPOT, works the same way. Use it to answer:

  • How fast does my VLM serve image-plus-text requests under load?
  • How much does image preprocessing add to latency?
  • Is the server GPU-bound or CPU-bound when handling images?

Step 1: understand how images reach the model​

Each prompt line in a VLM dataset has a text prompt and an image_urls array with one or more image references. There are two ways the image gets to the model:

  • Base64 encoding (default): the tool loads the image, optionally resizes it, and embeds it directly in the request. This is the realistic path for most deployments.
  • Server-side download: the request carries an image URL and the model server fetches it.

The built-in dataset metrumbench-vlm-4k-v1 provides a set of photographs with description prompts, ready to use. To bring your own images, supply a JSONL where each line has a prompt and an image_urls array (HTTP URLs, local paths, or file:// URIs).

Step 2: create the workload and scenario (API)​

The same VLM workload and scenarios can be created over the API. See metrumbench-vlm Workload And Scenario and Project Lifecycle for the API reference.

VLM-specific arguments​

ArgumentAPI fieldWhat it meansHow to choose
Images per requestp_num_images_batchHow many images are attached to each request.Usually 1. Raise it only to test multi-image reasoning.
Image detailp_image_detaillow sends the image at reduced fidelity (fewer image tokens, faster). high sends full fidelity (more accurate, slower). These are the only two valid values.Use high for accuracy-sensitive tests, low for throughput tests.
Input sequence lengthp_input_sequence_lengthThe text portion of the prompt, in tokens. The image adds its own tokens on top.Match your real prompts.
Output sequence lengthp_output_sequence_lengthTokens generated per request.Match your real answers.

Note on image detail: only low and high are valid. Earlier documentation mentioned a medium level; it does not exist. The tool can also resize oversized images to a maximum dimension before sending, which keeps image token counts predictable.

VLM-specific metrics​

VLM reports the same core metrics as LLM (throughput, TTFT, TPOT, latency percentiles, token counts), plus image handling data:

  • Image preprocessing time, included in TTFT. Loading, resizing, and encoding an image happens on the CPU on most paths.
  • Image sizes and dimensions per request.

Because preprocessing is CPU work, watch the Hardware chart tab on the Reporting page. If CPU utilization is high while GPU utilization is low, you are preprocessing-bound, not inference-bound. Move to a server with more CPU cores, use low image detail, or resize images ahead of time.

Common pitfalls​

  • Blaming the GPU for a CPU bottleneck. High image throughput can be capped by CPU preprocessing. Check the CPU view before concluding the GPU is slow.
  • Comparing low and high in one run. They produce different token counts and latencies. Make each its own scenario so the comparison is clean.
  • Forgetting the image adds tokens. A "256 token" VLM prompt is 256 text tokens plus the image tokens. Budget context length accordingly in your engine args.