Skip to main content

Compare Benchmark Tools

Private Stack Exposure​

Remote benchmark workers must reach the Metrum Insights control plane over a public URL. When you run a private or local stack, expose PostgREST before launch and use the public HTTPS URL for METRUM_PUBLIC_API_URL, INSIGHTS_API_URL, and INSIGHTS_CONTROL_PLANE_URL:

cloudflared tunnel --url http://localhost:${POSTGREST_PORT:-3000}
export METRUM_PUBLIC_API_URL="https://<cloudflared-host>.trycloudflare.com"
export INSIGHTS_API_URL="$METRUM_PUBLIC_API_URL"
export INSIGHTS_CONTROL_PLANE_URL="$METRUM_PUBLIC_API_URL"
curl -fsS "$METRUM_PUBLIC_API_URL/" >/dev/null

Do not give a remote worker localhost, 127.0.0.1, host.docker.internal, or a private-only DNS name. Keep the tunnel alive through onboarding, polling, result upload, and teardown; expose any private package/WebUI endpoint through a public URL too when the worker downloads artifacts from that stack.

Use this workflow when you want to know whether metrumbench-llm, genai-perf, and inferenceX agree for the same model-server recipe.

Decision​

Hold constant:

  • model, precision, framework, framework version, and engine args;
  • hardware config and concrete server instance;
  • ISL, OSL, concurrency, request count, and streaming mode.

Vary:

  • benchmark tool: metrumbench-llm, genai-perf, inferencex.

Web UI​

  1. Create one project for the comparison.
  2. Create one workload per tool with the same model and framework settings.
  3. Create identical scenario codes under every workload.
  4. Run a smoke scenario for each tool.
  5. Run the full matrix only after all smoke jobs complete.
  6. Compare throughput, TTFT, TPOT, latency percentiles, and errors by workload_code and scenario_code.

Placeholder: the UI should eventually offer a duplicate-workload action so an operator can clone model/framework settings and change only the tool.

API​

Create one workload per tool:

for tool in metrumbench-llm genai-perf inferencex; do
  curl -fsS -X POST "$METRUM_API_URL/rpc/create_workload" \
    -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
    -H "Content-Type: application/json" \
    -d "{
      \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
      \"p_project_name\":\"$PROJECT_NAME\",
      \"p_workload_code\":\"qwen15b-vllm020-$tool\",
      \"p_workload_name\":\"Qwen 1.5B vLLM 0.20 $tool\",
      \"p_tool_code\":\"$tool\",
      \"p_model_code\":\"deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B\",
      \"p_framework_code\":\"vllm\",
      \"p_version\":\"0.20.0\",
      \"p_engine_args_set_code\":\"vllm-default\"
    }" | jq
done

Query results:

curl -fsS "$METRUM_API_URL/v_benchmark_results?org_id=eq.$METRUM_ORG_ID&project_name=eq.$PROJECT_NAME&select=workload_code,scenario_code,server_hostname,output_tokens_per_second,ttft_ms,tpot_ms,p95_latency_ms,error_count" \
  -H "X-Org-Id: $METRUM_ORG_ID" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

Report Notes​

If tools disagree, check prompt generation, tokenizer behavior, warmup, streaming mode, request timeout, and whether all tools measured the same model-server process.