Skip to main content

KYAI Workload And Run

Use this flow to create a KYAI workload (candidate endpoint) and a KYAI run (generation + judge phases).

Set these environment variables for the examples:

export METRUM_API_URL="https://api.<control-plane-domain>"
export METRUM_JWT_TOKEN="<12-hour-db-minted-postgrest-jwt>"
export METRUM_ACCOUNT_ID="<account-uuid>"

Create Candidate Endpoint​

Registers a self-hosted model as a KYAI candidate.

curl -fsS -X POST "$METRUM_API_URL/rpc/create_kyai_candidate_endpoint" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_endpoint_code\":\"deepseek-qwen15b-local-vllm\",
    \"p_model_name\":\"deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B\",
    \"p_framework_code\":\"vllm\",
    \"p_version\":\"0.20.0\",
    \"p_engine_args_set_code\":\"vllm-default\"
  }" | jq

Parameters:

NameRequiredDescription
p_owner_account_idYesAccount UUID that owns the candidate.
p_endpoint_codeYesStable candidate endpoint code.
p_model_nameYesModel identifier (HuggingFace id or local path).
p_framework_codeNovllm, sglang, or trt-llm. Default vllm.
p_versionNoFramework version.
p_engine_args_set_codeNoEngine argument preset. Set tensor_parallel_size/tp to GPU count.
p_max_tokensNoPreset for max tokens per request. Default 4096.

Create Workload​

Creates a KYAI workload that pairs the candidate with a judge.

curl -fsS -X POST "$METRUM_API_URL/rpc/create_kyai_workload" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"kyai-deepseek-qwen15b\",
    \"p_workload_code\":\"deepseek-qwen15b-kyai\",
    \"p_workload_name\":\"DeepSeek Qwen 1.5B KYAI\",
    \"p_candidate_endpoint_code\":\"deepseek-qwen15b-local-vllm\"
  }" | jq

Parameters:

NameRequiredDescription
p_owner_account_idYesAccount UUID that owns the project.
p_project_nameYesExisting project name.
p_workload_codeYesStable workload code.
p_workload_nameYesDisplay name.
p_candidate_endpoint_codeYesCandidate endpoint created earlier.

Create Scenario​

curl -fsS -X POST "$METRUM_API_URL/rpc/create_kyai_scenario" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"kyai-deepseek-qwen15b\",
    \"p_workload_code\":\"deepseek-qwen15b-kyai\",
    \"p_scenario_code\":\"quantum-30\",
    \"p_snapshot_code\":\"quantum-mechanics\",
    \"p_judge_config_code\":\"platform-judge-default\"
  }" | jq

Parameters:

NameRequiredDescription
p_owner_account_idYesAccount UUID.
p_project_nameYesExisting project name.
p_workload_codeYesWorkload code.
p_scenario_codeYesStable scenario code.
p_snapshot_codeYesBuilt-in snapshot code (for example quantum-mechanics) or custom upload.
p_judge_config_codeYesJudge config code (use platform-judge-default for the Metrum AI Judge).

Create Generation Run​

curl -fsS -X POST "$METRUM_API_URL/rpc/create_kyai_generation_run" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"kyai-deepseek-qwen15b\",
    \"p_workload_code\":\"deepseek-qwen15b-kyai\",
    \"p_scenario_code\":\"quantum-30\",
    \"p_job_name\":\"kyai-generation-quantum-30\"
  }" | jq

Capture the returned p_job_id and pass it to the judge run.

Create Judge Run​

curl -fsS -X POST "$METRUM_API_URL/rpc/create_kyai_judge_run" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"kyai-deepseek-qwen15b\",
    \"p_workload_code\":\"deepseek-qwen15b-kyai\",
    \"p_scenario_code\":\"quantum-30\",
    \"p_generation_job_id\":<JOB_ID>,
    \"p_judge_config_code\":\"platform-judge-default\"
  }" | jq

Read Results​

curl -fsS "$METRUM_API_URL/v_kyai_results?select=*&order=created_at.desc&limit=50" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

See also​