BYOE (Bring Your Own Endpoint)

BYOE lets you benchmark any OpenAI-compatible model server endpoint that you already operate. Metrum Insights sends benchmark traffic to your endpoint, collects performance metrics, and stores them in the same results pipeline as managed workloads - but it does not start, stop, or manage the model server itself.

In the Web UI, BYOE endpoints are managed under Endpoints (BYOE) in the sidebar.

When to Use BYOE

Use BYOE when:

Your model server is already running and you just want to measure its performance.
The endpoint runs on infrastructure Metrum does not manage (Kubernetes clusters, on-prem servers, third-party cloud APIs).
You are testing against a commercial API like OpenAI, OpenRouter, Together, Groq, Fireworks, or Anthropic-compatible services.
The accelerator is non-NVIDIA or uses a vendor-specific runtime that Metrum cannot launch.
You are using a custom serving stack (litellm proxy, TGI, llama.cpp server, or a custom gateway).
You want to compare your self-hosted endpoint against a managed cloud API using the same scenarios and metrics.

Use a managed workload (not BYOE) when you want Metrum to handle the full lifecycle: start the model server, run readiness checks, execute the benchmark, and tear down the server.

How BYOE Works

Step 1: Register the Endpoint

An endpoint registration tells Metrum where your model server is, what kind of API it exposes, and what model it serves. You register an endpoint once and reuse it across multiple workloads and projects.

Key fields when registering an endpoint:

Field	What it means
Endpoint Code	A stable identifier you choose (e.g. `prod-llama70b-vllm`)
Endpoint Name	A human-readable label
Provider	What kind of server this is (vLLM-compatible, SGLang-compatible, OpenAI, OpenRouter, Together, Groq, Fireworks, litellm proxy, custom OpenAI-compatible, etc.)
Transport	The API shape: `openai_chat_completions` for text/vision LLMs, `openai_audio_transcriptions` for ASR, `openai_images_generations` for image generation
API Base URL	The base URL up to `/v1` (e.g. `https://api.openai.com/v1` or `http://my-server:8000/v1`)
Model Identifier	The model name sent in requests (e.g. `gpt-4o`, `meta-llama/Llama-3.1-70B-Instruct`)
API Key	Optional, encrypted at rest using AES-256-GCM. Never logged or exposed in API responses
Healthcheck Path	Path used to verify the endpoint is alive before benchmarking (default: `/v1/models`)
Max Concurrent Requests	Upper bound on concurrency the endpoint can handle (default: 8)
Advisory Cost Class	Whether this is `free`, `self_hosted`, or `paid` - used for cost-aware reporting

Step 2: Create a BYOE Workload

A BYOE workload binds a benchmark tool to your registered endpoint. This tells Metrum which tool to use for generating traffic and which endpoint to target.

The workload specifies:

Which benchmark tool to use (metrumbench-llm, metrumbench-vlm, metrumbench-asr, InferenceX, metrumbench-imagegen, genai-perf, or vllm_bench)
Which registered endpoint to target
The model, framework, and version metadata for result tracking

Once the workload exists, you create scenarios under it just like managed workloads: define concurrency, token lengths, request counts, and other traffic parameters.

Step 3: Run Scenarios

When a BYOE scenario executes:

The system resolves the endpoint URL, model identifier, and API key from the endpoint registration.
The benchmark tool command is built with endpoint-specific arguments (URL, API key, model name).
The benchmark worker sends traffic directly to your endpoint.
Results are collected, ingested, and stored in the same output tables and views as managed workloads.

The key difference from managed workloads: manage_model_server is set to false. The agent skips framework launch, readiness checks, and teardown for BYOE jobs.

Supported Providers

Metrum ships with these pre-registered provider types:

Provider	Description
`vllm_compatible`	Self-hosted vLLM or any vLLM-compatible endpoint
`sglang_compatible`	Self-hosted SGLang endpoint
`tgi_compatible`	Hugging Face Text Generation Inference
`llama_cpp_server`	llama.cpp HTTP server
`litellm_proxy`	LiteLLM proxy that normalizes multiple backends
`dummy_model_server`	Metrum's dummy model server for testing
`openai`	OpenAI's production API
`openrouter`	OpenRouter aggregation service
`together`	Together AI
`groq`	Groq inference API
`fireworks`	Fireworks AI
`anthropic_compat`	Anthropic-compatible endpoints
`custom_openai_compatible`	Catch-all for any OpenAI-compatible endpoint

Supported Transports

The transport type determines which API shape the benchmark tool uses and which tools are compatible with the endpoint:

Transport	API Path	Compatible Tools
`openai_chat_completions`	`POST /v1/chat/completions`	metrumbench-llm, metrumbench-vlm, InferenceX, genai-perf, vllm_bench, metrumbench-imagegen
`openai_audio_transcriptions`	`POST /v1/audio/transcriptions`	metrumbench-asr
`openai_images_generations`	`POST /v1/images/generations`	metrumbench-imagegen

Transport-tool compatibility is enforced at the database level. You cannot accidentally bind metrumbench-asr to a chat completions endpoint or metrumbench-llm to an audio transcription endpoint.

Endpoint Security

API Key Handling

API keys are encrypted at rest using AES-256-GCM with a server-side encryption key.
Keys are never returned in API list responses. A has_api_key boolean flag indicates whether a key is stored.
Keys are only decrypted at runtime when the benchmark worker needs to authenticate against the endpoint.
The api_key_hash (SHA-256) is stored in invocation records for audit without exposing the actual key.

Endpoint Readiness

Before sending benchmark traffic, BYOE workloads can perform a healthcheck against the configured healthcheck path (default /v1/models). This verifies the endpoint is alive and responding before the benchmark starts.

How BYOE Results Flow

BYOE results use the exact same pipeline as managed workloads:

The benchmark tool writes its output (JSONL summary for metrumbench-llm/metrumbench-vlm/ metrumbench-asr, JSON summary for metrumbench-imagegen, scalar metrics for InferenceX).
The agent daemon reads the output and maps it to the tool-specific database tables.
Results are queryable through the same views: v_metrumbench_llm_outputs, v_metrumbench_vlm_outputs, v_metrumbench_asr_outputs, v_inferencex_outputs, v_metrumbench_imagegen_outputs, and v_benchmark_results.
The Web UI shows BYOE job results alongside managed job results in the project workspace.

There is no difference in result quality, schema, or metric calculations between BYOE and managed runs. The only difference is that BYOE jobs skip framework lifecycle management.

Comparing BYOE and Managed Workloads

Aspect	Managed Workload	BYOE Workload
Model server startup	Metrum starts the model server	You start the model server
Readiness checks	Metrum verifies the model is loaded and ready	Optional healthcheck against your endpoint
Benchmark execution	Same tool, same metrics	Same tool, same metrics
Model server teardown	Metrum stops the model server	You manage teardown
Telemetry collection	GPU utilization, memory, power from the worker	Not available unless you attach external telemetry
Framework launch command	Resolved from command templates and engine args	Not applicable
Results storage	Same database tables and views	Same database tables and views
Cost tracking	Cloud provider costs tracked when available	You supply cost information

Web UI Workflow

Navigate to Endpoints (BYOE) in the sidebar.
Click Register Endpoint and fill in the endpoint details: code, name, provider, transport, URL, model identifier, and optional API key.
Open your project and create a workload that references the registered endpoint.
Add scenarios with the desired traffic shape (concurrency, ISL, OSL, request count).
Execute scenarios. The benchmark worker sends traffic directly to your endpoint.
Review results on the project results page, the same as managed workloads.

API Examples

Register an Endpoint

curl -fsS -X POST "$METRUM_API_URL/rpc/create_model_endpoint" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_endpoint_code\":\"my-vllm-endpoint\",
    \"p_endpoint_name\":\"My vLLM Server\",
    \"p_provider_code\":\"vllm_compatible\",
    \"p_transport_code\":\"openai_chat_completions\",
    \"p_api_base_url\":\"http://my-server:8000/v1\",
    \"p_model_identifier\":\"meta-llama/Llama-3.1-70B-Instruct\",
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_org_id\":\"$METRUM_ORG_ID\",
    \"p_healthcheck_path\":\"/v1/models\",
    \"p_max_concurrent_requests\":32,
    \"p_advisory_cost_class\":\"self_hosted\"
  }" | jq

List Registered Endpoints

curl -fsS -X POST "$METRUM_API_URL/rpc/list_model_endpoints" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_org_id\":\"$METRUM_ORG_ID\",
    \"p_include_inactive\":false
  }" | jq

Create a BYOE Workload

curl -fsS -X POST "$METRUM_API_URL/rpc/create_shared_workload_byoe" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_workload_name\":\"LLaMA 70B metrumbench-llm BYOE\",
    \"p_workload_code\":\"llama70b-metrumbench-llm-byoe\",
    \"p_tool_code\":\"metrumbench-llm\",
    \"p_endpoint_code\":\"my-vllm-endpoint\",
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_org_id\":\"$METRUM_ORG_ID\"
  }" | jq

Run a Scenario

After attaching the workload to a project and creating scenarios, execute as normal. BYOE jobs automatically set p_manage_model_server to false:

curl -fsS -X POST "$METRUM_API_URL/rpc/execute_scenario" \
  -H "Authorization: Bearer $METRUM_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"p_owner_account_id\":\"$METRUM_ACCOUNT_ID\",
    \"p_project_name\":\"$PROJECT_NAME\",
    \"p_workload_code\":\"llama70b-metrumbench-llm-byoe\",
    \"p_scenario_code\":\"isl128-osl128-c32\",
    \"p_server_hostname\":\"$SERVER_HOSTNAME\",
    \"p_manage_model_server\":false,
    \"p_job_name\":\"byoe-isl128-osl128-c32\"
  }" | jq

Report Considerations

When reporting BYOE results, document which parts are outside Metrum Insights control:

Model server startup, configuration, and teardown
Autoscaling and load balancer behavior
Endpoint authentication and authorization
GPU utilization and power telemetry (unless externally collected)
Cloud costs and infrastructure charges
Custom accelerator health and driver status
Network topology between the benchmark worker and the endpoint

These caveats help readers understand the boundary between what Metrum measured and what you managed.

When to Use BYOE​

How BYOE Works​

Step 1: Register the Endpoint​

Step 2: Create a BYOE Workload​

Step 3: Run Scenarios​

Supported Providers​

Supported Transports​

Endpoint Security​

API Key Handling​

Endpoint Readiness​

How BYOE Results Flow​

Comparing BYOE and Managed Workloads​

Web UI Workflow​

API Examples​

Register an Endpoint​

List Registered Endpoints​

Create a BYOE Workload​

Run a Scenario​

Report Considerations​