Benchmark Planning
Plan the matrix before creating workloads and scenarios. The goal is to make every result explainable and reproducible.
Model And Modality
Record both the model identity and the serving shape:
| Dimension | Examples | Why it matters |
|---|---|---|
| Model | deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B, Qwen/Qwen3-32B | Determines memory, tokenizer, prompt format, and license. |
| Modality | LLM, VLM, ASR, image, KYAI | Selects benchmark tool, scenario fields, and report interpretation. |
| Precision | BF16, FP16, FP8, INT8, AWQ, GPTQ | Changes memory, speed, and quality. |
| Context | 4k, 32k, 128k | Determines KV cache pressure and scheduling behavior. |
| Endpoint | local vLLM, Dynamo, OpenAI-compatible API, OpenRouter | Changes auth, latency path, and command templates. |
Traffic Shape
For LLM serving, define at least:
- input sequence length (ISL);
- output sequence length (OSL);
- concurrency;
- total requests;
- streaming or non-streaming;
- prompt source and sampling parameters.
Common sweep grid:
| ISL | OSL | Concurrency |
|---|---|---|
| 128 | 128 | 32, 256, 512, 1024 |
| 128 | 1024 | 32, 256, 512, 1024 |
| 1024 | 1024 | 32, 256, 512, 1024 |
| 1024 | 128 | 32, 256, 512, 1024 |
Create one scenario per grid point. Keep scenario codes machine-sortable, such
as isl128-osl1024-c256.
Tool Selection
Use the tool that matches the question:
| Tool | Use when |
|---|---|
| metrumbench-llm | OpenAI-compatible text generation throughput and latency. |
| GenAI-Perf | NVIDIA-oriented performance tests and comparison with existing GenAI-Perf runs. |
| KYAI | Quality, instruction-following, and judge-based evaluation. |
| metrumbench-imagegen | Image or multimodal workflows covered by metrumbench-imagegen runbooks. |
| metrumbench-vlm | Prompt-set based custom workloads. |
Success Criteria
Pick metrics before the run:
- throughput: tokens/sec, requests/sec;
- latency: TTFT, TPOT, p50, p95, p99;
- quality: KYAI pass rate, judge score, failure modes;
- utilization: GPU utilization, memory, power, CPU, network;
- cost: hourly cost, cost per 1M tokens, tokens per dollar;
- reliability: error rate, timeout rate, restart count.
Minimal Planning Record
Question: Can DeepSeek R1 Distill Qwen 1.5B meet 256-concurrency chat traffic on 1x RTX PRO 6000?
Project: deepseek-qwen15b-rtxpro6000-smoke
Model: deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
Framework: vLLM latest, OpenAI-compatible server
Precision: auto/BF16 baseline, FP8 follow-up
Hardware: specific 1x RTX PRO 6000 instance with recorded CPU/RAM/disk/NIC
Scenarios: ISL/OSL 128/128, 128/1024, 1024/1024, 1024/128 at c32/c256/c512/c1024
Success: no job failures, TTFT and p95 under target, GPU memory below limit, report package complete