End To End Benchmark Runbook

Private Stack Exposure

Remote benchmark workers must reach the Metrum Insights control plane over a public URL. When you run a private or local stack, expose PostgREST before launch and use the public HTTPS URL for METRUM_PUBLIC_API_URL, INSIGHTS_API_URL, and INSIGHTS_CONTROL_PLANE_URL:

cloudflared tunnel --url http://localhost:${POSTGREST_PORT:-3000}
export METRUM_PUBLIC_API_URL="https://<cloudflared-host>.trycloudflare.com"
export INSIGHTS_API_URL="$METRUM_PUBLIC_API_URL"
export INSIGHTS_CONTROL_PLANE_URL="$METRUM_PUBLIC_API_URL"
curl -fsS "$METRUM_PUBLIC_API_URL/" >/dev/null

Do not give a remote worker localhost, 127.0.0.1, host.docker.internal, or a private-only DNS name. Keep the tunnel alive through onboarding, polling, result upload, and teardown; expose any private package/WebUI endpoint through a public URL too when the worker downloads artifacts from that stack.

This runbook walks through one concrete Metrum Insights benchmark campaign:

provider: Shadeform
worker: one GPU server that can run vLLM
model: deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
serving framework: vLLM 0.20.0 or newer
performance tool: metrumbench-llm
quality tool: KYAI in the same project
traffic matrix: ISL/OSL pairs 128/128, 128/1024, 1024/128 at concurrencies 32, 64, 512, 1024

Use this as the first working path before adapting the workflow to larger models, other engines, or custom hardware.

0. API Environment

Web UI

Sign in to the Metrum Insights Web UI for your control-plane deployment. The UI handles user authentication; you do not need to mint a PostgREST JWT manually.

API

Set the control-plane URLs and identity values. Use the Web UI URL for browser workflows and frontend-only endpoints. Use the Caddy API endpoint for shell API examples; in hosted deployments this is the api. hostname that reverse-proxies to PostgREST. The new Web UI /api/postgrest proxy is browser-session authenticated and plain curl commands should not rely on it.

export METRUM_WEBUI_URL="https://<control-plane-host-or-dns>"
export METRUM_API_URL="https://api.<control-plane-domain>"

export METRUM_JWT_SECRET="<postgrest-jwt-secret>"
export METRUM_ACCOUNT_ID="<account-uuid>"
export METRUM_ACCOUNT_EMAIL="<user@example.com>"
export METRUM_ORG_ID="<org-uuid>"

export PROJECT_NAME="deepseek-qwen15b-shadeform-starter"
export METRUMBENCH_LLM_WORKLOAD_CODE="deepseek-qwen15b-vllm020-metrumbench-llm"
export KYAI_ENDPOINT_CODE="deepseek-qwen15b-local-vllm"
export KYAI_WORKLOAD_CODE="deepseek-qwen15b-kyai"
export SERVER_CONFIG_CODE="shadeform-vllm-starter"
export SERVER_HOSTNAME="shadeform-vllm-starter-001"
export METRUM_GPU_MODEL_CODE="h100-sxm5"

Mint a short-lived PostgREST JWT with the database RPC. This is API-only; it is not a Web UI step.

export METRUM_JWT_TOKEN="$(
  curl -fsS -X POST "$METRUM_API_URL/rpc/mint_postgrest_write_jwt" \
    -H "Content-Type: application/json" \
    -d "{
      \"p_jwt_secret\":\"$METRUM_JWT_SECRET\",
      \"p_account_id\":\"$METRUM_ACCOUNT_ID\",
      \"p_email\":\"$METRUM_ACCOUNT_EMAIL\",
      \"p_expires_in_seconds\":43200
    }" | jq -er 'if type=="string" then . else .token end'
)"

Verify the token before creating benchmark records:

curl -fsS -X POST "$METRUM_API_URL/rpc/get_current_user_context" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"p_account_id\":\"$METRUM_ACCOUNT_ID\"}" | jq

Do not paste JWT secrets, bearer tokens, provider keys, onboarding tokens, or bootstrap script URLs into tickets or chat.

1. Create Or Select The Shadeform Server

Web UI

Open the hardware or cloud onboarding area.
Choose Shadeform as the provider.
Pick an available GPU instance type that has enough memory for the model.
Select or register the SSH public key that you control.
Launch the instance and record the provider instance id and public IP.
Confirm that you can SSH to the host with the matching private key.

If the Shadeform launch screen is not available in your Web UI version, use the API path below and return to the UI after the server is registered.

API

The exact available cloud, region, and shade_instance_type change over time. Query the live Shadeform catalog, choose one available row, and keep those three values together.

export SHADEFORM_API="https://api.shadeform.ai/v1"
export SHADEFORM_API_KEY="<shadeform-api-key>"
export SHADEFORM_GPU_TYPE="h100"

curl -fsS -H "X-API-KEY: $SHADEFORM_API_KEY" \
  "$SHADEFORM_API/instances/types?gpu_type=$SHADEFORM_GPU_TYPE" | jq

export SHADEFORM_CLOUD="<cloud-from-selected-row>"
export SHADEFORM_REGION="<region-from-selected-row>"
export SHADEFORM_INSTANCE_TYPE="<shade_instance_type-from-selected-row>"
export SHADEFORM_SSH_KEY_ID="<shadeform-ssh-key-id>"

SHADEFORM_GPU_TYPE filters the provider catalog. METRUM_GPU_MODEL_CODE must match a Metrum gpu_models.model_code row, for example h100-sxm5. Discover supported Metrum GPU codes with:

curl -fsS "$METRUM_API_URL/gpu_models?select=model_code,model_name&order=model_code.asc" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

Create the instance with a known SSH key so an operator can log in for bootstrap and diagnostics:

curl -fsS -X POST "$SHADEFORM_API/instances/create" \
  -H "X-API-KEY: $SHADEFORM_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"cloud\":\"$SHADEFORM_CLOUD\",
    \"region\":\"$SHADEFORM_REGION\",
    \"shade_instance_type\":\"$SHADEFORM_INSTANCE_TYPE\",
    \"name\":\"$SERVER_HOSTNAME\",
    \"ssh_key_id\":\"$SHADEFORM_SSH_KEY_ID\",
    \"launch_configuration\":{
      \"os\":\"ubuntu22.04_cuda12.6_shade_os\",
      \"disk_gb\":250
    }
  }" | jq

Poll the instance info endpoint until the provider reports an active instance with a public IP. Then verify SSH:

export SHADEFORM_INSTANCE_ID="<instance-id-from-create-response>"

curl -fsS -H "X-API-KEY: $SHADEFORM_API_KEY" \
  "$SHADEFORM_API/instances/$SHADEFORM_INSTANCE_ID/info" | jq

export SERVER_PUBLIC_IP="<public-ip-from-info-response>"
ssh -i "<private-key-path>" "shadeform@$SERVER_PUBLIC_IP" "echo SSH_OK"

2. Generate The Metrum Insights Onboarding Token

Web UI

Open the server or hardware onboarding page.
Create a server config for the selected host.
Add the concrete server instance with the hostname and public IP.
Click the bootstrap or onboarding action for that server.
Copy the generated install command and run it on the remote host.

If the UI cannot create the server config or bootstrap command in your build, use the API calls below. The UI should still show the server after it is registered.

API

Create the server config and the concrete server instance. The instance RPC returns the onboarding token and a CLI join command.

curl -fsS -X POST "$METRUM_API_URL/rpc/create_server_config" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_org_id\":\"$METRUM_ORG_ID\",
    \"p_config_name\":\"Shadeform vLLM starter\",
    \"p_config_code\":\"$SERVER_CONFIG_CODE\",
    \"p_config_type_code\":\"cloud-instance\"
  }" | jq

curl -fsS -X POST "$METRUM_API_URL/rpc/create_server_config_gpu" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_org_id\":\"$METRUM_ORG_ID\",
    \"p_config_code\":\"$SERVER_CONFIG_CODE\",
    \"p_vendor_code\":\"NV\",
    \"p_model_code\":\"$METRUM_GPU_MODEL_CODE\",
    \"p_gpu_count\":1
  }" | jq

curl -fsS -X POST "$METRUM_API_URL/rpc/create_server_config_instance" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_org_id\":\"$METRUM_ORG_ID\",
    \"p_config_code\":\"$SERVER_CONFIG_CODE\",
    \"p_hostname\":\"$SERVER_HOSTNAME\",
    \"p_ip_address\":\"$SERVER_PUBLIC_IP\",
    \"p_availability_code\":\"pending_registration\",
    \"p_fail_if_exists\":true
  }" | tee /tmp/metrum-server-instance.json | jq

Generate the one-use bootstrap script from the Web UI server page. It downloads the correct versioned worker artifacts from the control plane and runs the join command for this server instance.

Hosted deployments include the agent package bundle. In local self-hosted development, build it before generating bootstrap commands:

make build-agent-wheels

For CLI-only workflows, the create_server_config_instance response already includes the onboarding_token and cli_join_command. Treat those as short-lived credentials. In current new Web UI builds, /api/onboarding/bootstrap is browser-session authenticated; use the Web UI bootstrap action, not a PostgREST bearer-token curl, unless your deployment has an explicit internal-service bootstrap API.

The response includes install_command, install_script_url, cli_join_command, and expires_at. Treat all of them as short-lived credentials.

3. Make The Control Plane Reachable

Web UI

If the control plane is deployed on a public DNS name, continue to the next step. If it is private, the remote Shadeform instance cannot download the bootstrap script or post heartbeats until you expose the control plane.

Placeholder: a future UI flow will warn when the generated bootstrap URL is not reachable from an external worker.

API

For a private development control plane, expose the Web UI/Caddy endpoint with a tunnel from the control-plane host:

cloudflared tunnel --url "http://127.0.0.1:<control-plane-port>"

Use the generated https://<name>.trycloudflare.com URL as METRUM_WEBUI_URL before generating the Web UI bootstrap command. The remote server must be able to reach this URL to download worker artifacts and start listening for work.

4. Install And Start The Remote Worker

Web UI

Open the generated bootstrap command for the server.
SSH to the Shadeform host.
Run the install command on the remote host.
Return to the server page and wait for heartbeat/readiness.

Placeholder: a future Web UI screen will show each bootstrap phase, including artifact download, join, service start, heartbeat, and last error.

API

Run the generated install command on the remote server:

ssh -i "<private-key-path>" "shadeform@$SERVER_PUBLIC_IP" \
  "<install-command-copied-from-the-Web-UI>"

If you need to run the join manually, install insights-cli on the remote host and use the cli_join_command returned by create_server_config_instance.

Check readiness from the control plane:

curl -fsS -X POST "$METRUM_API_URL/rpc/validate_server_readiness" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"p_server_hostname\":\"$SERVER_HOSTNAME\"}" | jq

5. Create The Benchmark Project

Web UI

Create a project named deepseek-qwen15b-shadeform-starter. Keep this campaign separate from unrelated model, hardware, or framework studies so the result export is clean.

API

curl -fsS -X POST "$METRUM_API_URL/rpc/create_project" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"$PROJECT_NAME\",
    \"p_visibility_code\":\"private\",
    \"p_created_by_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_org_id\":\"$METRUM_ORG_ID\"
  }" | jq

6. Add The metrumbench-llm Workload

Web UI

Add a workload with:

Field	Value
Tool	metrumbench-llm
Model	`deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B`
Framework	vLLM
Version	`0.20.0` or newer
Engine args set	`vllm-default`
Quantization	BF16 or the default for the selected template

Placeholder: if your build does not expose all workload fields in the UI, use the API below and then verify the created workload from the project page.

API

curl -fsS -X POST "$METRUM_API_URL/rpc/create_workload" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"$PROJECT_NAME\",
    \"p_workload_name\":\"DeepSeek R1 Distill Qwen 1.5B - vLLM 0.20 metrumbench-llm\",
    \"p_workload_code\":\"$METRUMBENCH_LLM_WORKLOAD_CODE\",
    \"p_tool_code\":\"metrumbench-llm\",
    \"p_model_code\":\"deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B\",
    \"p_framework_code\":\"vllm\",
    \"p_version\":\"0.20.0\",
    \"p_quantization_code\":\"bf16\",
    \"p_engine_args_set_code\":\"vllm-default\"
  }" | jq

7. Add The KYAI Workload

Web UI

Add a KYAI workload to the same project, using the same model endpoint that the worker will expose during the benchmark.

Placeholder: if candidate endpoint or KYAI workload creation is not exposed in the UI, create it through the API and view the resulting jobs and scores in the project.

API

Create a candidate endpoint record for the vLLM OpenAI-compatible server that will run on the worker:

curl -fsS -X POST "$METRUM_API_URL/rpc/create_kyai_candidate_endpoint" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_org_id\":\"$METRUM_ORG_ID\",
    \"p_endpoint_code\":\"$KYAI_ENDPOINT_CODE\",
    \"p_endpoint_name\":\"DeepSeek Qwen 1.5B local vLLM\",
    \"p_api_base_url\":\"http://127.0.0.1:8000/v1\",
    \"p_model_name\":\"deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B\",
    \"p_framework_code\":\"vllm\",
    \"p_framework_version\":\"0.20.0\",
    \"p_config_code\":\"$SERVER_CONFIG_CODE\",
    \"p_engine_args_set_code\":\"vllm-default\",
    \"p_max_concurrent_requests\":32
  }" | jq

curl -fsS -X POST "$METRUM_API_URL/rpc/create_kyai_workload" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"$PROJECT_NAME\",
    \"p_workload_name\":\"DeepSeek Qwen 1.5B KYAI\",
    \"p_workload_code\":\"$KYAI_WORKLOAD_CODE\",
    \"p_candidate_endpoint_code\":\"$KYAI_ENDPOINT_CODE\"
  }" | jq

Use an existing default KYAI snapshot or built-in set for the starter path. For example, quantum-mechanics is a seeded snapshot in standard deployments.

curl -fsS "$METRUM_API_URL/kyai_snapshots?select=snapshot_code,snapshot_name&order=snapshot_code.asc" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

curl -fsS "$METRUM_API_URL/v_kyai_builtin_sets_with_prompt_count?select=set_code,set_name,prompt_count&order=set_code.asc" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

Create a small KYAI scenario in the same project:

export KYAI_SCENARIO_CODE="quantum-30"

curl -fsS -X POST "$METRUM_API_URL/rpc/create_kyai_scenario" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"$PROJECT_NAME\",
    \"p_workload_code\":\"$KYAI_WORKLOAD_CODE\",
    \"p_scenario_name\":\"Quantum mechanics 30 prompt quality check\",
    \"p_scenario_code\":\"$KYAI_SCENARIO_CODE\",
    \"p_snapshot_code\":\"quantum-mechanics\",
    \"p_num_samples\":30,
    \"p_batch_size\":4,
    \"p_max_concurrent_requests\":8
  }" | jq

8. Add The metrumbench-llm Scenario Sweep

Web UI

Create one scenario for every ISL/OSL/concurrency point:

ISL	OSL	Concurrency
128	128	32, 64, 512, 1024
128	1024	32, 64, 512, 1024
1024	128	32, 64, 512, 1024

Placeholder: if bulk scenario creation is not available in your build, use the API loop below.

API

for pair in 128:128 128:1024 1024:128; do
  isl="${pair%%:*}"
  osl="${pair##*:}"

  for concurrency in 32 64 512 1024; do
    scenario_code="isl${isl}-osl${osl}-c${concurrency}"

    curl -fsS -X POST "$METRUM_API_URL/rpc/create_scenario" \
      -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
      -H "Content-Type: application/json" \
      -d "{
        \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
        \"p_project_name\":\"$PROJECT_NAME\",
        \"p_workload_code\":\"$METRUMBENCH_LLM_WORKLOAD_CODE\",
        \"p_scenario_name\":\"$scenario_code\",
        \"p_scenario_code\":\"$scenario_code\",
        \"p_concurrency\":$concurrency,
        \"p_input_sequence_length\":$isl,
        \"p_output_sequence_length\":$osl,
        \"p_num_requests\":1024,
        \"p_streaming\":true,
        \"p_server_side_download\":true,
        \"p_job_params\":{\"dataset_name\":\"random\"}
      }" | jq
  done
done

Create a tiny smoke scenario before launching the full sweep:

curl -fsS -X POST "$METRUM_API_URL/rpc/create_scenario" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"$PROJECT_NAME\",
    \"p_workload_code\":\"$METRUMBENCH_LLM_WORKLOAD_CODE\",
    \"p_scenario_name\":\"smoke-isl32-osl16-c1\",
    \"p_scenario_code\":\"smoke-isl32-osl16-c1\",
    \"p_concurrency\":1,
    \"p_input_sequence_length\":32,
    \"p_output_sequence_length\":16,
    \"p_num_requests\":2
  }" | jq

9. Run Benchmarks And Collect Telemetry

Web UI

Start with the smoke scenario.
Watch the job move through pending, running, and completed.
Check model-server startup logs, benchmark logs, and telemetry.
Launch the full sweep after the smoke run succeeds.
Launch the KYAI generation and judge jobs from the same project.

Placeholder: if the Web UI does not yet expose KYAI job launch, use the API below and inspect the resulting jobs in the project.

API

Run the smoke scenario:

curl -fsS -X POST "$METRUM_API_URL/rpc/execute_scenario" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"$PROJECT_NAME\",
    \"p_workload_code\":\"$METRUMBENCH_LLM_WORKLOAD_CODE\",
    \"p_scenario_code\":\"smoke-isl32-osl16-c1\",
    \"p_server_hostname\":\"$SERVER_HOSTNAME\",
    \"p_manage_model_server\":true,
    \"p_job_name\":\"smoke-isl32-osl16-c1\"
  }" | jq

Run the full metrumbench-llm sweep:

for pair in 128:128 128:1024 1024:128; do
  isl="${pair%%:*}"
  osl="${pair##*:}"

  for concurrency in 32 64 512 1024; do
    scenario_code="isl${isl}-osl${osl}-c${concurrency}"

    curl -fsS -X POST "$METRUM_API_URL/rpc/execute_scenario" \
      -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
      -H "Content-Type: application/json" \
      -d "{
        \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
        \"p_project_name\":\"$PROJECT_NAME\",
        \"p_workload_code\":\"$METRUMBENCH_LLM_WORKLOAD_CODE\",
        \"p_scenario_code\":\"$scenario_code\",
        \"p_server_hostname\":\"$SERVER_HOSTNAME\",
        \"p_manage_model_server\":true,
        \"p_job_name\":\"$scenario_code\"
      }" | jq
  done
done

Run KYAI generation, then pass the returned generation job id to the judge run:

curl -fsS -X POST "$METRUM_API_URL/rpc/create_kyai_generation_run" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"$PROJECT_NAME\",
    \"p_job_name\":\"kyai-generation-$KYAI_SCENARIO_CODE\",
    \"p_workload_code\":\"$KYAI_WORKLOAD_CODE\",
    \"p_scenario_code\":\"$KYAI_SCENARIO_CODE\",
    \"p_server_config_instance_id\":$SERVER_ID
  }" | tee /tmp/metrum-kyai-generation.json | jq

export KYAI_GENERATION_JOB_ID="$(jq -r '.job_id' /tmp/metrum-kyai-generation.json)"

curl -fsS -X POST "$METRUM_API_URL/rpc/create_kyai_judge_run" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"$PROJECT_NAME\",
    \"p_job_name\":\"kyai-judge-$KYAI_SCENARIO_CODE\",
    \"p_generation_job_id\":$KYAI_GENERATION_JOB_ID,
    \"p_server_config_instance_id\":$SERVER_ID
  }" | jq

Monitor jobs and telemetry:

curl -fsS "$METRUM_API_URL/v_job_current_status?select=*&order=changed_at.desc&limit=20" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

curl -fsS "$METRUM_API_URL/v_benchmark_results?org_id=eq.$METRUM_ORG_ID&project_name=eq.$PROJECT_NAME&select=*" \
  -H "X-Org-Id: $METRUM_ORG_ID" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

curl -fsS "$METRUM_API_URL/v_kyai_results?select=*&order=created_at.desc&limit=50" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" | jq

10. View Or Download Results

Web UI

Open the project and inspect:

run and job status;
throughput, TTFT, TPOT, latency percentiles, and error rate;
GPU utilization, memory, power, and duration telemetry;
KYAI scores and per-prompt outputs;
logs, SUT artifacts, and report downloads.

Placeholder: if a report or export button is not available in your build, use the API queries below.

API

Download all benchmark rows for the project:

curl -fsS "$METRUM_API_URL/v_benchmark_results?org_id=eq.$METRUM_ORG_ID&project_name=eq.$PROJECT_NAME&select=*" \
  -H "X-Org-Id: $METRUM_ORG_ID" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Accept: text/csv" \
  -o "benchmark-results-$PROJECT_NAME.csv"

Download metrumbench-llm details:

curl -fsS "$METRUM_API_URL/v_benchmark_results?org_id=eq.$METRUM_ORG_ID&project_name=eq.$PROJECT_NAME&select=job_id,tool_code" \
  -H "X-Org-Id: $METRUM_ORG_ID" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -o /tmp/metrum-benchmark-jobs.json

export METRUMBENCH_LLM_JOB_IDS="$(
  jq -r '[.[] | select(.tool_code=="metrumbench-llm") | .job_id] | join(",")' \
    /tmp/metrum-benchmark-jobs.json
)"

curl -fsS "$METRUM_API_URL/v_metrumbench_llm_outputs?job_id=in.($METRUMBENCH_LLM_JOB_IDS)&select=*" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Accept: text/csv" \
  -o "metrumbench-llm-outputs-$PROJECT_NAME.csv"

Download KYAI scores:

curl -fsS "$METRUM_API_URL/v_kyai_results?select=*" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Accept: text/csv" \
  -o "kyai-results-$PROJECT_NAME.csv"

For larger exports, filter by project, run id, workload, scenario, or job id so the downloaded data remains traceable.

Private Stack Exposure​

0. API Environment​

Web UI​

API​

1. Create Or Select The Shadeform Server​

Web UI​

API​

2. Generate The Metrum Insights Onboarding Token​

Web UI​

API​

3. Make The Control Plane Reachable​

Web UI​

API​

4. Install And Start The Remote Worker​

Web UI​

API​

5. Create The Benchmark Project​

Web UI​

API​

6. Add The metrumbench-llm Workload​

Web UI​

API​

7. Add The KYAI Workload​

Web UI​

API​

8. Add The metrumbench-llm Scenario Sweep​

Web UI​

API​

9. Run Benchmarks And Collect Telemetry​

Web UI​

API​

10. View Or Download Results​

Web UI​

API​

Private Stack Exposure

0. API Environment

Web UI

API

1. Create Or Select The Shadeform Server

Web UI

API

2. Generate The Metrum Insights Onboarding Token

Web UI

API

3. Make The Control Plane Reachable

Web UI

API

4. Install And Start The Remote Worker

Web UI

API

5. Create The Benchmark Project

Web UI

API

6. Add The metrumbench-llm Workload

Web UI

API

7. Add The KYAI Workload

Web UI

API

8. Add The metrumbench-llm Scenario Sweep

Web UI

API

9. Run Benchmarks And Collect Telemetry

Web UI

API

10. View Or Download Results

Web UI

API