Insights CLI
insights-cli is the supported terminal interface for Metrum Insights. It
uses the same control-plane API as the browser product and is intended for
operators who want to script project setup, workload/scenario creation,
cloud/server lifecycle, benchmark execution, KYAI evaluation, result queries,
and worker operation.
Use the CLI when you need repeatable terminal workflows. Use direct HTTP calls from the API Reference when you need lower-level automation that is not yet wrapped by a CLI command.
The safest mental model is:
- Install from the signed-in frontend.
- Authenticate once with
insights-cli auth login. - Use
list,status,show,info, andendpointcommands to inspect state before making changes. - Use
cloud requestonly when you are ready to launch provider-backed resources. - End every provider-backed workflow with an explicit teardown or retained execution decision.
Install
The normal install path is the command shown in the signed-in Insights frontend. Open your profile page and copy the CLI install command. The command has this shape:
curl -fsSL "$INSIGHTS_FRONTEND_URL/api/cli/install.sh" | bash
INSIGHTS_FRONTEND_URL is the browser frontend origin, not the PostgREST API
URL and not the /api/agent proxy URL. When in doubt, use the exact command
shown inside the signed-in frontend.
The installer creates shims under ${HOME}/.local/bin for insights-cli and
metrum-mcp-call, then writes the control-plane and frontend origins to
~/.metrum/config.json so authentication can reuse the browser session. Add
${HOME}/.local/bin to PATH if your shell does not already include it.
Verify the install:
insights-cli version
insights-cli --help
For source-tree development only:
cd insights-cli
uv sync --extra dev --extra agent
uv run insights-cli --help
Authenticate
After installing from the Insights frontend, run:
insights-cli auth login
insights-cli auth status
insights-cli auth context
auth login opens a browser/device-style login flow when the CLI has a saved
frontend URL from the installer. Development-only alternatives are available
for local stacks:
insights-cli auth login --api-url "$METRUM_API_URL" --token "$METRUM_JWT_TOKEN"
insights-cli auth login --api-url "$METRUM_API_URL" --jwt-secret "$PGRST_JWT_SECRET" --email "$METRUM_ACCOUNT_EMAIL"
Do not use JWT-secret login for customer or production operator workflows. It is for local development and controlled test environments.
The CLI stores local context and credentials under the user's Metrum config directory. Do not copy those files into tickets, docs, or chat.
Core Workflow
Most benchmark runs follow this sequence:
- Select an organization and create or select a project.
- Create one workload for each model, framework, version, tool, precision, or engine-args recipe.
- Create one scenario for each traffic shape or evaluation slice.
- Select, register, or provision a server.
- Wait for provider state, agent heartbeat, GPU detection, and readiness.
- Schedule jobs.
- Poll status and print results.
- Download SUT, logs, reports, and analytics.
- Terminate or release cloud resources.
Use this command choice guide when turning that workflow into a script:
| Need | Use | Notes |
|---|---|---|
| Inspect current state | project list, workload list, scenario list, job status, job show, cloud status, cloud list-instances | Read-only checks. Run these before mutating state. |
| Create benchmark metadata | project create, workload create, scenario create | Writes control-plane rows; does not launch provider hardware. |
| Edit or remove metadata | project update/delete, workload update/delete, scenario update/delete | Mutates or deletes control-plane records. Confirm you are in the intended project/org first. |
| Run on an existing server | scenario execute --scenario ... --server ... or job schedule --scenario ... --server ... | Creates a benchmark job for an already registered worker. |
| Launch managed cloud hardware | cloud request | Can start paid provider resources. Confirm provider, hardware, cost, and teardown policy first. |
| Monitor cloud hardware | cloud status, cloud list-instances, server readiness, cloud endpoint --execution-id ... | Use --active-only, --execution-id, and --provider filters to avoid stale rows. |
| Clean up cloud hardware | cloud teardown --execution-id ... | Queues provider-backed termination. Use cloud terminate-instance only when you must target a specific instance row. |
| Collect artifacts | job results, job sut, job logs, analytics benchmark-results | Read result data after job completion. |
Example:
insights-cli org list
insights-cli org switch "Metrum AI"
insights-cli project create "DeepSeek R1 Distill Qwen 7B Hardware Matrix" --visibility private
insights-cli project switch "DeepSeek R1 Distill Qwen 7B Hardware Matrix"
insights-cli workload create \
--name "DeepSeek R1 Qwen 7B on vLLM" \
--code "deepseek-r1-qwen7b-vllm" \
--tool metrumbench-llm \
--model "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B" \
--framework vllm \
--version "0.20.0"
insights-cli scenario create \
--name "ISL 128 OSL 128 C32" \
--code "isl128-osl128-c32" \
--workload "deepseek-r1-qwen7b-vllm" \
--input-tokens 128 \
--output-tokens 128 \
--concurrency 32 \
--num-requests 32
Command Areas
Run insights-cli <area> --help for exact options. Current command areas and
frequently used commands include:
| Area | Purpose | Common Commands |
|---|---|---|
auth | Login, logout, context, token checks. | login, token, status, context, logout |
org | Organization context. | list, switch, current, info, clear |
project | Project lifecycle. | list, create, switch, current, info, update, delete, clear |
models | AI model catalog. | list, create-family, create, estimate-memory, add-from-hf |
framework | Framework catalog and engine args. | list, list-engine-args-sets, create-version, create-engine-args-set |
workload | Workload definitions. | list, create, update, reuse, delete, info, switch |
scenario | Scenario definitions. | list, create, execute, update, delete, info |
server | Server registration and readiness. | list, register, register-gke-tpu, info, health, health-history, health-record, readiness |
cloud | Managed provider lifecycle. | request, status, teardown, list-hardware, list-configs, create-config, start-execution, list-executions, list-instances, endpoint, release-execution, terminate-instance, resolve-server-command, provision |
job | Job scheduling, monitoring, and artifacts. | schedule, schedule-all, sweep, list, status, show, cancel, results, telemetry, sut, logs, replicas |
kyai | KYAI setup, generation, judging, and results. | snapshot, snapshot-*, endpoint-*, judge-config-*, workload-create, scenario-create, generation-*, evaluation-*, run, report, generation-prompt-create, generation-prompt-list, judge-prompt-create, judge-prompt-list, snapshot-import, snapshot-status, snapshot-prompts |
analytics | Read-only result and telemetry views. | benchmark-results, job-outputs, compare-jobs, project-runs, server-status, cost-efficiency, concurrency-scaling, framework-comparison, model-comparison, parameter-sensitivity, quantization-comparison, run-cost-summary, telemetry-summary, hardware-utilization, leaderboard |
control-plane | Discovery and context. | schema-version, context |
agent | Worker daemon configuration and operation. | join, config, print-bootstrap, start, start-benchmark-worker, start-kyai-worker, status, stop, manage-run-model-server |
console | Interactive browser session. | open, exec, quit |
Most list/read/report commands support JSON output through the global flag:
insights-cli --format json project list
insights-cli --format json cloud list-instances --active-only --provider shadeform
Cloud Workflow
Before launching paid cloud hardware, list active resources and available catalog/config entries:
insights-cli cloud list-executions --active-only --limit 100
insights-cli cloud list-instances --active-only --provider shadeform
insights-cli cloud list-hardware --provider shadeform
insights-cli cloud list-configs --provider shadeform
Cloud commands split into safe discovery, metadata setup, paid launch, and cleanup:
| Action | Preferred Command | Alias or Advanced Form | Launches Provider Hardware? |
|---|---|---|---|
| List execution progress | cloud status | cloud list-executions | No |
| List active instances | cloud list-instances --active-only | Add --provider shadeform to narrow the view | No |
| List provider catalog | cloud list-hardware | Filter by --provider when possible | No |
| Create/reuse a config row | cloud create-config | May be auto-created by cloud request --provider ... | No |
| Start managed execution | cloud request | cloud start-execution | Yes |
| Get candidate endpoint URL | cloud endpoint --execution-id ... | Use after provider and worker readiness | No |
| Clean up an execution | cloud teardown --execution-id ... | cloud release-execution for retained executions | Terminates or queues termination |
| Direct local/provider fallback | cloud provision | Developer/operator fallback only | Yes |
Create a reusable cloud config without launching hardware:
insights-cli cloud create-config \
--provider shadeform \
--instance-type gpu_1x_rtxpro6000 \
--hourly-rate 0.99 \
--config-code shadeform-rtxpro6000-deepseek-qwen7b \
--config-name "Shadeform RTX PRO 6000 DeepSeek Qwen 7B"
Start a managed execution only after the discovery commands show the target provider and hardware are correct. This creates a durable cloud request and can launch provider resources:
insights-cli cloud request \
--config-code shadeform-rtxpro6000-deepseek-qwen7b \
--workload-code deepseek-r1-qwen7b-vllm \
--project "DeepSeek R1 Distill Qwen 7B Hardware Matrix" \
--all \
--teardown-policy manual \
--provider-ready-timeout-seconds 1800
cloud request and cloud start-execution call the same
create_cloud_execution_request control-plane RPC. Use --teardown-policy manual or after-linked-phases when a follow-on KYAI phase needs to reuse the
same model endpoint after metrumbench-llm completes. Use --auto-terminate only when
the benchmark phase is the last phase that needs the machine.
Monitor the execution:
insights-cli cloud status --active-only --limit 50
insights-cli server readiness "<server-hostname>"
insights-cli cloud endpoint --execution-id "<execution-id>"
Clean up when the benchmark and any linked KYAI phase no longer need the machine:
insights-cli cloud teardown --execution-id "<execution-id>" --reason "workflow complete"
cloud status is an alias for cloud list-executions. cloud teardown queues
provider-backed cleanup by --execution-id or --instance-id; use
cloud release-execution when you specifically need the retained-execution
release path. cloud terminate-instance "<cloud-instance-id>" is available
for direct instance cleanup, but prefer cloud teardown --execution-id when you
are cleaning up a managed execution.
KYAI Workflow
KYAI usually runs after the candidate model endpoint is reachable. Create or select the dataset snapshot, candidate endpoint, judge config, KYAI workload, and scenario; then run generation and evaluation:
There are two command styles:
| Style | Use When | Commands |
|---|---|---|
| Explicit setup/run commands | You want each setup object and run phase visible in logs. | snapshot-create, endpoint-create, judge-config-create, workload-create, scenario-create, generation-create, evaluation-create |
| Short compatibility wrappers | You want concise phase-oriented scripts. | snapshot create/list/status/prompts, run --phase generation, run --phase evaluation, report --phase ... |
The explicit commands are easiest to audit in long workflows:
insights-cli kyai snapshot-list
insights-cli kyai endpoint-create --help
insights-cli kyai judge-config-create --help
insights-cli kyai workload-create --help
insights-cli kyai scenario-create --help
insights-cli kyai generation-create \
--project "$PROJECT_NAME" \
--name "Quantum generation" \
--workload "$KYAI_WORKLOAD_CODE" \
--scenario "$KYAI_SCENARIO_CODE" \
--batch-size 16 \
--num-samples 30
insights-cli kyai generation-get "<generation-job-id>"
insights-cli kyai generation-outputs --job-id "<generation-job-id>"
insights-cli kyai evaluation-create \
--project "$PROJECT_NAME" \
--generation-job-id "<generation-job-id>" \
--name "GPT judge evaluation" \
--judge "$JUDGE_CONFIG_CODE"
insights-cli kyai evaluation-get "<evaluation-job-id>"
insights-cli kyai evaluation-results --job-id "<evaluation-job-id>"
Compatibility wrappers are also available when you want shorter phase-oriented commands:
insights-cli kyai snapshot create \
--code quantum-mechanics-30 \
--name "Quantum Mechanics 30" \
--source-type huggingface \
--source-ref "0xZee/dataset-CoT-Quantum-Mechanics-1224" \
--max-rows 30
insights-cli kyai run \
--phase generation \
--project "$PROJECT_NAME" \
--name "KYAI generation" \
--workload "$KYAI_WORKLOAD_CODE" \
--scenario "$KYAI_SCENARIO_CODE"
insights-cli kyai run \
--phase evaluation \
--project "$PROJECT_NAME" \
--name "KYAI judge" \
--generation-job-id "<generation-job-id>" \
--judge "$JUDGE_CONFIG_CODE"
insights-cli kyai report --phase evaluation --job-id "<evaluation-job-id>"
Run KYAI after metrumbench-llm when both phases reuse one provisioned endpoint. Do not start conflicting model servers on the same machine during performance measurement.
Worker Commands
Workers run the actual benchmark or KYAI jobs. The preferred path is to generate a bootstrap from the Insights frontend or onboarding API, then run the generated install/join command on the worker.
Useful worker-side commands:
insights-cli agent join --onboarding-token "$METRUM_ONBOARDING_TOKEN"
insights-cli agent status
insights-cli agent start-benchmark-worker --foreground
insights-cli agent start-kyai-worker --foreground
insights-cli agent stop
The server is ready for benchmark scheduling only after the control plane shows fresh agent heartbeat and GPU detection.
Troubleshooting
| Symptom | Check |
|---|---|
| Login cannot find the frontend URL | Reinstall from the signed-in Insights frontend or set the API URL explicitly for development. |
| Jobs remain pending | Check insights-cli server readiness, worker heartbeat, and GPU detection. |
| Cloud execution stays queued | Check cloud status, provider credentials, hardware availability, and provisioner status. |
| Endpoint URL is missing | Wait for provider active state, worker registration, and model-server readiness, then retry cloud endpoint --execution-id .... |
| KYAI is too slow | Increase --batch-size when the judge/candidate endpoint can support it. |
| Artifacts are missing | Use job sut, job logs, job results, and analytics benchmark-results to isolate ingestion vs download issues. |