Skip to main content

Image generation benchmarking (metrumbench-imagegen)

Image generation measures how fast a text-to-image model produces images under load, using the metrumbench-imagegen tool. It targets an OpenAI-compatible /v1/images/generations endpoint, collects image artifacts, and reports latency and throughput.

Image generation is included in every Tier 2 (Multimodal), Tier 3 (Pro), and Tier 4 (Enterprise) plan. You do not need an add-on. Tier 1 plans do not include it. On a Tier 1 plan, the benchmark shows a lock with "Available in Multimodal plan and above.", and the platform refuses an image generation workload. See Billing, Plans and Metering.

Metric definitions and serving details live in Performance Methodology → metrumbench-imagegen.

Supported frameworks​

FrameworkRoleLatest CUDA 13.0 (default)Latest AMD / ROCm
vLLM-Omni (vllm-omni)Omni image-generation path (vllm serve … --omni)0.24.00.22.0 (amd-rocm)
SGLang (sglang)Diffusion path (sglang serve / serve-diffusion)0.5.15.post10.5.15.post1 (amd-mi300 / amd-mi355)

Default engine-args sets are single-GPU-safe:

FrameworkDefault setDefaults
vLLM-Omnivllm-omni-metrumbench-imagegentensor_parallel_size=1, usp=1, ring=1
SGLangsglang-metrumbench-imagegentp_size=1, ulysses_degree=1, ring_degree=1
  • TP: shards model weights across GPUs. Default 1.
  • USP / Ulysses: sequence parallelism for diffusion. Default 1 (off).
  • Ring: ring sequence parallelism. Default 1 (off).

You can alter, add, or remove engine args to match your hardware and model (for example changing TP from 1 to 2 on a multi-GPU node).

What it measures and when to use it​

Use ImageGen when you need:

  • Images per second and megapixels per second on a given GPU
  • Generation latency (p50 / p95 / p99) under concurrency
  • A fair comparison of vLLM-Omni vs SGLang diffusion on the same model family

Headline metrics: images/sec, megapixels/sec, latency, success rate.

Step 1: create the workload​

  1. Go to Projects → New Project (or open a project and select Edit Project, or Copy Project when the project has run).
  2. Under Workloads, open Image Generation and select MetrumBench ImageGen.
  3. Target Hardware: select the server chip / GPU count.
  4. Models: pick an image-generation model (for example Tongyi-MAI/Z-Image-Turbo).
  5. Framework: pick vLLM-Omni or SGLang (diffusion).
  6. Version: use the latest CUDA 13.0 default above unless you need ROCm.
  7. Engine Args Set: use the ImageGen default set for that framework, or a verified set matched to your model and hardware. You can add, edit, or remove engine args (for example changing tensor_parallel_size or tp_size to match the GPU count).

The dataset (text2image-multiprompt-1024, 1024 prompts) is preselected automatically.

Step 2: configure the scenario matrix​

FieldWhat it controlsDefault when unset
ConcurrencyParallel image-generation requests1
Images per request (n)How many images each request asks for1
Image sizeTarget resolution (for example 1024x1024)1024x1024

Request count is set by the platform from concurrency (not a user field). See Performance Methodology → metrumbench-imagegen.

Images per request (n):

  • vLLM-Omni: n > 1 is supported.
  • SGLang: prefer n = 1. Multi-image requests rely on each diffusion pipeline broadcasting text conditioning to the expanded sample batch, which SGLang's shared infrastructure does not do automatically. Some models work (SD 3.5, FLUX); others fail (Qwen-Image, Z-Image-Turbo). This is a known SGLang upstream issue: see Performance Methodology.

ImageGen does not use ISL/OSL or streaming the way LLM workloads do.

Step 3: run and read results​

  1. Select Save & Run, or select Save to run the project later.
  2. On the Reporting page, filter to the ImageGen workload.
  3. Review per scenario:
    • Images per second and megapixels per second
    • Latency mean / p50 / p90 / p95 / p99
    • Success rate and failed / timeout counts

When comparing frameworks, hold model, hardware, dataset, concurrency, image size, and n fixed. Vary only the framework. For SGLang comparisons, keep n = 1 unless you know that model supports multi-image requests.

Common pitfalls​

  • Setting n > 1 on SGLang. Prefer n = 1 on SGLang; multi-image support is model-pipeline dependent (known upstream limitation).
  • Comparing different image sizes on images/sec alone. Prefer megapixels/sec when resolutions differ.
  • Reusing LLM engine-args sets. ImageGen sets are separate (vllm-omni-metrumbench-imagegen, sglang-metrumbench-imagegen).

The API path​

The same ImageGen workload and scenarios can be created over the API. See metrumbench-imagegen Workload And Scenario and Project Lifecycle for the API reference.