Image generation benchmarking (metrumbench-imagegen)
Image generation measures how fast a text-to-image model produces images under
load, using the metrumbench-imagegen tool. It targets an OpenAI-compatible
/v1/images/generations endpoint, collects image artifacts, and reports
latency and throughput.
Image generation is included in every Tier 2 (Multimodal), Tier 3 (Pro), and Tier 4 (Enterprise) plan. You do not need an add-on. Tier 1 plans do not include it. On a Tier 1 plan, the benchmark shows a lock with "Available in Multimodal plan and above.", and the platform refuses an image generation workload. See Billing, Plans and Metering.
Metric definitions and serving details live in Performance Methodology → metrumbench-imagegen.
Supported frameworks
| Framework | Role | Latest CUDA 13.0 (default) | Latest AMD / ROCm |
|---|---|---|---|
vLLM-Omni (vllm-omni) | Omni image-generation path (vllm serve … --omni) | 0.24.0 | 0.22.0 (amd-rocm) |
SGLang (sglang) | Diffusion path (sglang serve / serve-diffusion) | 0.5.15.post1 | 0.5.15.post1 (amd-mi300 / amd-mi355) |
Default engine-args sets are single-GPU-safe:
| Framework | Default set | Defaults |
|---|---|---|
| vLLM-Omni | vllm-omni-metrumbench-imagegen | tensor_parallel_size=1, usp=1, ring=1 |
| SGLang | sglang-metrumbench-imagegen | tp_size=1, ulysses_degree=1, ring_degree=1 |
- TP: shards model weights across GPUs. Default
1. - USP / Ulysses: sequence parallelism for diffusion. Default
1(off). - Ring: ring sequence parallelism. Default
1(off).
You can alter, add, or remove engine args to match your hardware and model
(for example changing TP from 1 to 2 on a multi-GPU node).
What it measures and when to use it
Use ImageGen when you need:
- Images per second and megapixels per second on a given GPU
- Generation latency (p50 / p95 / p99) under concurrency
- A fair comparison of vLLM-Omni vs SGLang diffusion on the same model family
Headline metrics: images/sec, megapixels/sec, latency, success rate.
Step 1: create the workload
- Go to Projects → New Project (or open a project and select Edit Project, or Copy Project when the project has run).
- Under Workloads, open Image Generation and select MetrumBench ImageGen.
- Target Hardware: select the server chip / GPU count.
- Models: pick an image-generation model (for example
Tongyi-MAI/Z-Image-Turbo). - Framework: pick vLLM-Omni or SGLang (diffusion).
- Version: use the latest CUDA 13.0 default above unless you need ROCm.
- Engine Args Set: use the ImageGen default set for that framework, or a
verified set matched to your model and hardware. You can add, edit, or remove
engine args (for example changing
tensor_parallel_sizeortp_sizeto match the GPU count).
The dataset (text2image-multiprompt-1024, 1024 prompts) is preselected
automatically.
Step 2: configure the scenario matrix
| Field | What it controls | Default when unset |
|---|---|---|
| Concurrency | Parallel image-generation requests | 1 |
Images per request (n) | How many images each request asks for | 1 |
| Image size | Target resolution (for example 1024x1024) | 1024x1024 |
Request count is set by the platform from concurrency (not a user field). See Performance Methodology → metrumbench-imagegen.
Images per request (n):
- vLLM-Omni:
n > 1is supported. - SGLang: prefer
n = 1. Multi-image requests rely on each diffusion pipeline broadcasting text conditioning to the expanded sample batch, which SGLang's shared infrastructure does not do automatically. Some models work (SD 3.5, FLUX); others fail (Qwen-Image, Z-Image-Turbo). This is a known SGLang upstream issue: see Performance Methodology.
ImageGen does not use ISL/OSL or streaming the way LLM workloads do.
Step 3: run and read results
- Select Save & Run, or select Save to run the project later.
- On the Reporting page, filter to the ImageGen workload.
- Review per scenario:
- Images per second and megapixels per second
- Latency mean / p50 / p90 / p95 / p99
- Success rate and failed / timeout counts
When comparing frameworks, hold model, hardware, dataset, concurrency, image
size, and n fixed. Vary only the framework. For SGLang comparisons, keep
n = 1 unless you know that model supports multi-image requests.
Common pitfalls
- Setting
n > 1on SGLang. Prefern = 1on SGLang; multi-image support is model-pipeline dependent (known upstream limitation). - Comparing different image sizes on images/sec alone. Prefer megapixels/sec when resolutions differ.
- Reusing LLM engine-args sets. ImageGen sets are separate
(
vllm-omni-metrumbench-imagegen,sglang-metrumbench-imagegen).
The API path
The same ImageGen workload and scenarios can be created over the API. See metrumbench-imagegen Workload And Scenario and Project Lifecycle for the API reference.