Skip to main content

Measure Quality With KYAI

Private Stack Exposure​

Remote benchmark workers must reach the Metrum Insights control plane over a public URL. When you run a private or local stack, expose PostgREST before launch and use the public HTTPS URL for METRUM_PUBLIC_API_URL, INSIGHTS_API_URL, and INSIGHTS_CONTROL_PLANE_URL:

cloudflared tunnel --url http://localhost:${POSTGREST_PORT:-3000}
export METRUM_PUBLIC_API_URL="https://<cloudflared-host>.trycloudflare.com"
export INSIGHTS_API_URL="$METRUM_PUBLIC_API_URL"
export INSIGHTS_CONTROL_PLANE_URL="$METRUM_PUBLIC_API_URL"
curl -fsS "$METRUM_PUBLIC_API_URL/" >/dev/null

Do not give a remote worker localhost, 127.0.0.1, host.docker.internal, or a private-only DNS name. Keep the tunnel alive through onboarding, polling, result upload, and teardown; expose any private package/WebUI endpoint through a public URL too when the worker downloads artifacts from that stack.

Use KYAI when the benchmark decision depends on output quality, not only throughput or latency.

Decision​

Hold constant:

  • prompt snapshot or built-in prompt set;
  • judge config and judge prompt;
  • number of samples;
  • generation system prompt.

Vary:

  • model, quantization, engine args, hardware, or endpoint.

Web UI​

  1. Open the project used for the performance benchmark.
  2. Add a KYAI workload using the same candidate model endpoint.
  3. Select a snapshot or built-in set.
  4. Set sample count and batch size.
  5. Run generation.
  6. Run judge evaluation.
  7. Review score distribution and per-prompt failures.

Placeholder: if the UI does not expose candidate endpoint or judge-run creation in your build, create the KYAI objects through the API and inspect the jobs in the project.

API​

Create a candidate endpoint, workload, and scenario:

curl -fsS -X POST "$METRUM_API_URL/rpc/create_kyai_candidate_endpoint" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_org_id\":\"$METRUM_ORG_ID\",
    \"p_endpoint_code\":\"qwen15b-local-vllm\",
    \"p_endpoint_name\":\"Qwen 1.5B local vLLM\",
    \"p_api_base_url\":\"http://127.0.0.1:8000/v1\",
    \"p_model_name\":\"deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B\",
    \"p_framework_code\":\"vllm\",
    \"p_framework_version\":\"0.20.0\"
  }" | jq

curl -fsS -X POST "$METRUM_API_URL/rpc/create_kyai_workload" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"$PROJECT_NAME\",
    \"p_workload_code\":\"qwen15b-kyai\",
    \"p_workload_name\":\"Qwen 1.5B KYAI\",
    \"p_candidate_endpoint_code\":\"qwen15b-local-vllm\"
  }" | jq

Query scores:

curl -fsS "$METRUM_API_URL/v_kyai_results?select=*&order=created_at.desc&limit=100" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq