KYAI Evaluation

Private Stack Exposure

Remote benchmark workers must reach the Metrum Insights control plane over a public URL. When you run a private or local stack, expose PostgREST before launch and use the public HTTPS URL for METRUM_PUBLIC_API_URL, INSIGHTS_API_URL, and INSIGHTS_CONTROL_PLANE_URL:

cloudflared tunnel --url http://localhost:${POSTGREST_PORT:-3000}
export METRUM_PUBLIC_API_URL="https://<cloudflared-host>.trycloudflare.com"
export INSIGHTS_API_URL="$METRUM_PUBLIC_API_URL"
export INSIGHTS_CONTROL_PLANE_URL="$METRUM_PUBLIC_API_URL"
curl -fsS "$METRUM_PUBLIC_API_URL/" >/dev/null

Do not give a remote worker localhost, 127.0.0.1, host.docker.internal, or a private-only DNS name. Keep the tunnel alive through onboarding, polling, result upload, and teardown; expose any private package/WebUI endpoint through a public URL too when the worker downloads artifacts from that stack.

Use this flow for Know Your AI generation and judge evaluation. Do not expose judge API keys or candidate endpoint secrets in docs, logs, or examples.

Set these environment variables for the examples:

export METRUM_API_URL="https://api.<control-plane-domain>"
export METRUM_JWT_TOKEN="<12-hour-db-minted-postgrest-jwt>"
export METRUM_ACCOUNT_ID="<account-uuid>"

Create KYAI Generation Run

Creates a generation job for a candidate endpoint and prompt snapshot.

curl -fsS -X POST "$METRUM_API_URL/rpc/create_kyai_generation_run" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"kyai-smoke\",
    \"p_workload_code\":\"kyai-candidate-workload\",
    \"p_scenario_code\":\"quantum-30\",
    \"p_job_name\":\"kyai-generation-quantum-30\",
    \"p_endpoint_code\":\"candidate-vllm\",
    \"p_snapshot_code\":\"quantum-30\",
    \"p_num_samples\":30,
    \"p_batch_size\":4
  }" | jq

Common parameters:

Name	Required	Description
p_owner_account_id	Yes	Account UUID that owns the project.
p_project_name	Yes	Project that owns the KYAI run.
p_workload_code	Yes	KYAI workload code.
p_scenario_code	Yes	KYAI scenario code.
p_job_name	Yes	Stable job name.
p_endpoint_code	No	Candidate endpoint override.
p_snapshot_code	No	Prompt/instruction snapshot override.
p_num_samples	No	Limit for smoke or sampled runs.
p_batch_size	No	Generation batch size.
p_server_hostname	No	Server hostname override.

Get KYAI Generation Run

Checks generation job state.

curl -fsS -X POST "$METRUM_API_URL/rpc/get_kyai_generation_run" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"p_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_job_id\":123}" | jq

Parameters:

Name	Required	Description
p_account_id	Yes	Account UUID for access filtering.
p_job_id	Yes	KYAI generation job id.

Query KYAI Generation Outputs

Returns generated candidate outputs.

curl -fsS "$METRUM_API_URL/v_kyai_generation_outputs?generation_job_id=eq.123&limit=30" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

Query parameters:

Name	Required	Description
generation_job_id	Yes	Use eq.<job_id> to filter to one generation job.
limit	No	Maximum outputs to return.

Create KYAI Evaluation Run

Creates a judge evaluation job over generated outputs.

curl -fsS -X POST "$METRUM_API_URL/rpc/create_kyai_judge_run" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"kyai-smoke\",
    \"p_generation_job_id\":123,
    \"p_job_name\":\"kyai-judge-quantum-30\",
    \"p_judge_config_code\":\"default-judge\"
  }" | jq

Parameters:

Name	Required	Description
p_owner_account_id	Yes	Account UUID that owns the project.
p_project_name	Yes	Project name.
p_generation_job_id	Yes	Completed generation job to score.
p_job_name	Yes	Stable judge job name.
p_judge_config_code	No	Judge configuration code.
p_judge_system_prompt_code	No	Judge system prompt override.
p_server_hostname	No	Server hostname override.

Get KYAI Judge Run And Results

Check judge state and collect scored results.

curl -fsS -X POST "$METRUM_API_URL/rpc/get_kyai_judge_run" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"p_account_id\":\"$METRUM_ACCOUNT_ID\",\"p_job_id\":456}" | jq

curl -fsS "$METRUM_API_URL/kyai_results?evaluation_job_id=eq.456&limit=30" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

Parameters:

API	Parameter	Required	Description
get_kyai_judge_run	p_account_id	Yes	Account UUID for access filtering.
get_kyai_judge_run	p_job_id	Yes	KYAI judge job id.
kyai_results	evaluation_job_id	Yes	Use eq.<job_id> to filter to one judge job.
kyai_results	limit	No	Maximum scored rows to return.