Using Insights with an Agent

Give instructions to your AI agent in natural language, and it carries them out by calling the CLI or the API behind the scenes.

You: "Run the Llama 3.1 70B benchmark on my H200 server and show me the results."
Agent: (creates project → creates workload → creates scenario → executes → reads results)

Pick a Path

If your agent will...	Authenticate with	Then it uses...
Run shell commands	insights-cli auth login	metrum-mcp-call (CLI helper)
Make HTTP calls directly	A JWT generated in the UI	The control-plane API

These are not interchangeable. The CLI helper reads the token from insights-cli auth login. The API requires the JWT minted from Settings → Tokens in the dashboard. Pick the path that matches your agent's capabilities.

Path 1: CLI

If your agent can run shell commands, the CLI is the simplest setup.

Authenticate

insights-cli auth login

This opens your browser to the same Auth0 login you use for the Metrum Insights dashboard. After you authenticate, the CLI saves a short-lived JWT to ~/.metrum/credentials.json. See the CLI page for install instructions and other login methods.

Tell your agent what to do

Create a project called "Llama 3.1 70B agent demo". Add a workload for meta-llama/Llama-3.1-70B-Instruct on vLLM 0.6.3 using metrumbench-llm. Add a scenario with concurrency 8, 512 input tokens, 256 output tokens, 10 requests. Execute it on my server at <hostname> and then show me the results.

The agent will translate that into calls like:

metrum-mcp-call create_project \
  '{"project_name":"Llama 3.1 70B agent demo","visibility":"private"}'

metrum-mcp-call create_workload \
  '{
    "project_name":   "Llama 3.1 70B agent demo",
    "code":           "llama-70b-vllm-baseline",
    "tool_code":      "metrumbench-llm",
    "model_code":     "meta-llama/Llama-3.1-70B-Instruct",
    "framework_code": "vllm",
    "version":        "0.6.3"
  }'

metrum-mcp-call create_scenario \
  '{
    "project_name":  "Llama 3.1 70B agent demo",
    "workload_code": "llama-70b-vllm-baseline",
    "code":          "c8-isl512-osl256",
    "concurrency":   8,
    "input_tokens":  512,
    "output_tokens": 256,
    "num_requests":  10
  }'

metrum-mcp-call execute_scenario \
  '{
    "project_name":    "Llama 3.1 70B agent demo",
    "workload_code":   "llama-70b-vllm-baseline",
    "scenario_code":   "c8-isl512-osl256",
    "server_hostname": "<your-server-hostname>"
  }'

metrum-mcp-call query_benchmark_results \
  '{"project_name":"Llama 3.1 70B agent demo","limit":50}'

List available tools with metrum-mcp-call --list.

Path 2: API

If your agent only makes HTTP calls, use the API.

Generate a JWT in the UI

Open Settings → Tokens in the Metrum Insights dashboard. Under Personal API Tokens, give the token a name (e.g. agent-ci-runner), pick an expiry period, and click Generate Token. Copy the JWT that appears.

It is shown once. If you lose it, generate a new one and revoke the old one from the same page.

Export the JWT

Hand the JWT and the API URL to the agent's environment. The API URL below points to the default control plane at https://insights.metrum.ai. If your deployment uses a different URL, substitute it here.

export METRUM_JWT_TOKEN="<the-copied-jwt>"
export METRUM_API_URL="https://insights.metrum.ai/api"

The JWT carries your account identity and org membership. The API enforces RBAC at request time, so the agent can only do what you can do.

Tell your agent what to do

Create a project called "Llama 3.1 70B agent demo". Add a workload for meta-llama/Llama-3.1-70B-Instruct on vLLM 0.6.3 using metrumbench-llm. Add a scenario with concurrency 8, 512 input tokens, 256 output tokens, 10 requests. Execute it on my server at <hostname> and then show me the results.

The agent will translate that into calls like:

curl -X POST "$METRUM_API_URL/rpc/create_project" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "p_owner_account_id": "<from get_current_user_context>",
    "p_project_name":     "Llama 3.1 70B agent demo",
    "p_visibility_code":  "private"
  }'

curl -X POST "$METRUM_API_URL/rpc/create_workload" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "p_owner_account_id": "<from get_current_user_context>",
    "p_org_id":           "<from get_current_user_context>",
    "p_project_name":     "Llama 3.1 70B agent demo",
    "p_workload_name":    "Llama 3.1 70B vLLM baseline",
    "p_workload_code":    "llama-70b-vllm-baseline",
    "p_tool_code":        "metrumbench-llm",
    "p_model_code":       "meta-llama/Llama-3.1-70B-Instruct",
    "p_framework_code":   "vllm",
    "p_version":          "0.6.3"
  }'

curl -X POST "$METRUM_API_URL/rpc/create_scenario" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "p_owner_account_id":       "<from get_current_user_context>",
    "p_project_name":           "Llama 3.1 70B agent demo",
    "p_workload_code":          "llama-70b-vllm-baseline",
    "p_scenario_name":          "C8 ISL512 OSL256",
    "p_scenario_code":          "c8-isl512-osl256",
    "p_concurrency":            8,
    "p_input_sequence_length":  512,
    "p_output_sequence_length": 256,
    "p_num_requests":           10
  }'

curl -X POST "$METRUM_API_URL/rpc/execute_scenario" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "p_owner_account_id": "<from get_current_user_context>",
    "p_project_name":     "Llama 3.1 70B agent demo",
    "p_workload_code":    "llama-70b-vllm-baseline",
    "p_scenario_code":    "c8-isl512-osl256",
    "p_server_hostname":  "<your-server-hostname>"
  }'

curl "$METRUM_API_URL/v_benchmark_results?project_name=eq.Llama%203.1%2070B%20agent%20demo&limit=50" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN"

The API Reference documents every RPC and view.

What Agents Can Do

Manage benchmark metadata

Create and list the objects that define a benchmark run:

• Projects — create_project, list_projects
• Workloads — create_workload, list_workloads
• Scenarios — create_scenario, list_scenarios
• Organizations — list_organizations, get_organization
• Model catalog — list_model_catalog, create_ai_model, add_model_from_huggingface, create_model_family
• Framework catalog — list_framework_catalog, create_framework_version
• Hardware catalog — list_hardware_catalog

Execute benchmarks

Submit a scenario to a registered server and track its progress:

• execute_scenario — submit one scenario to a server
• execute_scenario_sweep — submit multiple scenarios in one call
• get_job_status — poll a single job
• list_jobs — list jobs with filters

Read results and analytics

Query benchmark outputs, telemetry, and comparative analytics:

• Results — query_benchmark_results, query_job_outputs
• Telemetry — query_telemetry_summary, query_hardware_utilization
• Comparisons — query_model_comparison, query_framework_comparison, query_quantization_comparison, query_concurrency_scaling, query_parameter_sensitivity
• Cost — query_cost_efficiency, query_run_cost_summary
• Leaderboard — query_leaderboard
• Project runs — query_project_runs
• Server status — query_server_status

Manage servers and endpoints

Register servers, check readiness, and manage model endpoints:

• register_server, list_servers, check_server_readiness
• create_model_endpoint, list_model_endpoints

Run KYAI evaluations

Create snapshots, endpoints, judge configs, and run evaluations:

• create_kyai_snapshot, list_kyai_snapshots, list_kyai_snapshot_prompts
• create_kyai_candidate_endpoint, list_kyai_candidate_endpoints
• create_kyai_judge_config, list_kyai_judge_configs
• create_kyai_workload, create_kyai_scenario
• create_kyai_generation_run, create_kyai_evaluation_run
• query_kyai_generation_outputs, query_kyai_results
• get_kyai_generation_run, get_kyai_judge_run
• list_kyai_generation_runs, list_kyai_judge_runs

Get recommendations

• recommend_model_for_hardware — pick a model that fits a GPU
• recommend_hardware — pick hardware for a model

Generate reports

• get_run_ai_eval_report — AI evaluation report for a run
• get_project_ai_eval_report — AI evaluation report for a project

Where to Go Next

• API Reference documents every RPC and view.
• CLI covers the full insights-cli command surface.