Compare vLLM Versions And Settings

Private Stack Exposure

Remote benchmark workers must reach the Metrum Insights control plane over a public URL. When you run a private or local stack, expose PostgREST before launch and use the public HTTPS URL for METRUM_PUBLIC_API_URL, INSIGHTS_API_URL, and INSIGHTS_CONTROL_PLANE_URL:

cloudflared tunnel --url http://localhost:${POSTGREST_PORT:-3000}
export METRUM_PUBLIC_API_URL="https://<cloudflared-host>.trycloudflare.com"
export INSIGHTS_API_URL="$METRUM_PUBLIC_API_URL"
export INSIGHTS_CONTROL_PLANE_URL="$METRUM_PUBLIC_API_URL"
curl -fsS "$METRUM_PUBLIC_API_URL/" >/dev/null

Do not give a remote worker localhost, 127.0.0.1, host.docker.internal, or a private-only DNS name. Keep the tunnel alive through onboarding, polling, result upload, and teardown; expose any private package/WebUI endpoint through a public URL too when the worker downloads artifacts from that stack.

Use this workflow to test whether a vLLM upgrade or engine-args change improves performance without hurting stability or quality.

Version Comparison

Hold constant:

• model, hardware, quantization, scenarios, and engine args values.

Vary:

• vLLM version and container image tag.

for version in 0.19.0 0.20.0 0.20.1; do
  curl -fsS -X POST "$METRUM_API_URL/rpc/create_workload" \
    -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
    -H "Content-Type: application/json" \
    -d "{
      \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
      \"p_project_name\":\"$PROJECT_NAME\",
      \"p_workload_code\":\"qwen15b-vllm-${version//./}\",
      \"p_workload_name\":\"Qwen 1.5B vLLM $version\",
      \"p_tool_code\":\"metrumbench-llm\",
      \"p_model_code\":\"deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B\",
      \"p_framework_code\":\"vllm\",
      \"p_version\":\"$version\",
      \"p_engine_args_set_code\":\"vllm-default\"
    }" | jq
done

Settings Comparison

Create one workload or engine args set per recipe:

Recipe	Example Engine Args Set	What It Tests
Baseline	vllm-default	Reference behavior.
FP8 KV cache	vllm-kv-cache-fp8	KV cache dtype and memory headroom.
Long context	vllm-max-len-32768	Max model length, OOM risk, latency.
Prefix cache	vllm-prefix-cache	Repeated-prefix traffic.
Scheduler tuning	vllm-scheduler-high-batch	Max batched tokens and max sequences.

Settings worth testing:

• attention backend;
• KV cache dtype;
• KV cache memory sizing;
• max model length;
• prefix caching;
• max batched tokens and max sequences;
• tensor parallelism, data parallelism, and replica count;
• speculative decoding;
• quantization mode;
• CUDA/image tag.

Report Notes

A tuning win should improve the primary metric without increasing failure rate, memory pressure, startup failures, or KYAI quality regressions.