Skip to main content

Compare Serving Engines

Private Stack Exposure​

Remote benchmark workers must reach the Metrum Insights control plane over a public URL. When you run a private or local stack, expose PostgREST before launch and use the public HTTPS URL for METRUM_PUBLIC_API_URL, INSIGHTS_API_URL, and INSIGHTS_CONTROL_PLANE_URL:

cloudflared tunnel --url http://localhost:${POSTGREST_PORT:-3000}
export METRUM_PUBLIC_API_URL="https://<cloudflared-host>.trycloudflare.com"
export INSIGHTS_API_URL="$METRUM_PUBLIC_API_URL"
export INSIGHTS_CONTROL_PLANE_URL="$METRUM_PUBLIC_API_URL"
curl -fsS "$METRUM_PUBLIC_API_URL/" >/dev/null

Do not give a remote worker localhost, 127.0.0.1, host.docker.internal, or a private-only DNS name. Keep the tunnel alive through onboarding, polling, result upload, and teardown; expose any private package/WebUI endpoint through a public URL too when the worker downloads artifacts from that stack.

Use this workflow when the hardware and model stay fixed but the serving engine changes.

Decision​

Hold constant:

  • model, hardware instance, benchmark tool, and scenario matrix;
  • precision and quantization intent;
  • endpoint behavior and request protocol.

Vary:

  • framework code, framework version, command template, container image, and engine args set.

Web UI​

  1. Create one project.
  2. Create one workload for each engine.
  3. Use identical scenario codes across workloads.
  4. Run smoke for every engine.
  5. Run the full matrix.
  6. Compare throughput, TTFT, TPOT, latency, memory headroom, and failures.

API Pattern​

WorkloadFrameworkEngine Args Set
qwen15b-vllm020vllmvllm-default
qwen15b-sglangsglangSGLang args set selected by admin.
qwen15b-trtllmtrtllmTensorRT-LLM args set selected by admin.

Create each workload with create_workload, then reuse the same create_scenario calls under each workload.

Report Notes​

If one engine requires a different max sequence length, model format, or quantization mode, record it as a complete recipe comparison rather than a pure engine comparison.