Public leaderboard API
The public leaderboard API returns the entries of the hardware leaderboard. It has three GET routes. The routes need no token and no account.
The routes belong to the web app, not to the PostgREST API. They do not use the /api/postgrest proxy.
export BENCH_URL="https://bench.metrum.ai"
Conventions
| Topic | Value |
|---|---|
| Base path | /api/public/v1/leaderboard |
| Methods | GET and OPTIONS |
| Authentication | None. Do not send an Authorization header. |
| Response type | application/json |
| Field names | camelCase |
| Version | v1 in the path |
| Cache | Cache-Control: public, s-maxage=300, stale-while-revalidate=600, and an ETag |
| Rate limit | 60 requests per minute for each client IP address |
Versioning
The version is part of the path. A change that removes or renames a field ships under a new version path. The v1 routes keep their current fields.
Caching
Every successful response carries Cache-Control: public, s-maxage=300, stale-while-revalidate=600 and an ETag. A shared cache can serve a response for 300 seconds. It can serve a stale response for 600 more seconds while it fetches a fresh one. Send the ETag back in If-None-Match to check for a change.
When Metrum AI publishes, renames, or withdraws an entry, the server refreshes the cached leaderboard data. Your own client cache follows the headers above.
Rate limit
Each client IP address can send 60 requests per minute. A request over the limit receives HTTP 429 and a Retry-After header. The header holds the number of seconds to wait.
CORS
Browser requests from an allowed origin succeed. The server answers GET and OPTIONS only. It does not allow credentials.
The default allowed origins are https://metrum.ai, https://www.metrum.ai, and https://bench.metrum.ai. An operator changes the list with the PUBLIC_API_ALLOWED_ORIGINS environment variable. A request from a server, curl, or a script does not need CORS.
Errors
| Status | When | Body |
|---|---|---|
| 400 | A parameter is missing or has a value the route does not accept | A plain sentence that names the problem |
| 404 | The entry does not exist, or Metrum AI withdrew it | A plain sentence |
| 429 | You exceeded 60 requests per minute | A plain sentence, and a Retry-After header |
| 503 | The database service is not available | A plain sentence |
If you receive 429, wait the number of seconds in Retry-After. If you receive 503, retry after a short delay.
List the choices
GET /api/public/v1/leaderboard/options
Returns every model, precision, and workload preset that has at least one published entry, and the default choice. Use it to fill the parameters of the board route.
This route takes no parameters.
curl -fsS "$BENCH_URL/api/public/v1/leaderboard/options"
Response
| Field | Type | Description |
|---|---|---|
options | array | One row for each model, precision, and workload preset |
defaults | object | The default model, precision, and workload, and target (50) |
Each row of options:
| Field | Type | Description |
|---|---|---|
modelCode | string | Model code. Send it as model. |
modelName | string | Model display name |
precisionCode | string | Precision code. Send it as precision. |
precisionDisplay | string | Precision display name |
workloadCode | string | Workload preset code. Send it as workload. |
workloadDisplay | string | Workload preset display name |
inputSequenceLength | integer | Input Sequence Length of the preset, in tokens |
outputSequenceLength | integer | Output Sequence Length of the preset, in tokens |
entryCount | integer | Number of published entries |
List the ranked entries
GET /api/public/v1/leaderboard
Returns the entries of one comparability key, ranked in one category. The key is the model, the precision, and the workload preset.
Parameters
| Name | Required | Values | Default |
|---|---|---|---|
model | Yes | A modelCode from the options route | None |
precision | Yes | A precisionCode from the options route | None |
workload | Yes | A workloadCode from the options route | None |
category | No | throughput, interactivity, or ttft | throughput |
target | No | 20, 50, or 100 output tokens per second per request | 50 |
limit | No | 1 to 200 | 50 |
A limit outside 1 to 200 is clamped to the nearest valid value. A category or target outside the table returns HTTP 400.
curl -fsS "$BENCH_URL/api/public/v1/leaderboard?model=<model-code>&precision=<precision-code>&workload=<workload-code>&category=throughput&target=50&limit=10"
Response
| Field | Type | Description |
|---|---|---|
cohort | object | The model, precision, and workload of this list |
category | string | The category of this list |
target | integer | The interactivity target of this list |
entries | array | The ranked rows. Rows that reached the target come first, in rank order. Rows that did not reach it follow, sorted by systemName. |
generatedAt | string | Time that the server built the response, in ISO 8601 |
Each row of entries:
| Field | Type | Description |
|---|---|---|
rank | integer or null | Position in the category. Null when the target is not reached, or when the category is ttft and the run did not measure it. |
isTied | boolean | True when another entry has the same value and shares this rank |
targetReached | boolean | True when at least one point meets the target |
entryUuid | string | Entry id. Use it in the entry route. |
systemName | string | Public system name |
acceleratorName | string | Accelerator name |
gpuCount | integer | Number of GPUs |
frameworkCode | string | Framework code |
frameworkDisplay | string | Framework display name |
frameworkVersion | string | Framework version |
tensorParallelSize | integer | Tensor parallel size |
modelCode, modelName | string | Model code and display name |
precisionCode, precisionDisplay | string | Precision code and display name |
workloadCode, workloadDisplay | string | Workload preset code and display name |
throughputPerGpu | number or null | Total tokens per second per GPU at the throughput point. Null when the target is not reached. |
throughputPointInteractivity | number | Output tokens per second per request at the throughput point |
throughputPointConcurrency | integer | Concurrency of the throughput point |
peakInteractivity | number | Best output tokens per second per request across all points |
ttftP50Ms | number or null | Median time to first token at the throughput point, in milliseconds. Null when not measured. |
joulesPerOutputToken | number or null | Energy per output token at the throughput point. Null when not measured. |
pointCount | integer | Number of concurrency points in the entry |
releaseTag | string | Benchmark tool release, for example metrumbench-llm 0.1.80 |
runCompletedAt | string | Time the source run finished |
publishedAt | string | Time Metrum AI published the entry |
Get one entry
GET /api/public/v1/leaderboard/entries/{entryUuid}
Returns one published entry and all its concurrency points.
Parameters
| Name | In | Required | Description |
|---|---|---|---|
entryUuid | Path | Yes | An entryUuid from the list route |
curl -fsS "$BENCH_URL/api/public/v1/leaderboard/entries/<entry-uuid>"
The route returns HTTP 404 for an unknown entry and for a withdrawn entry.
Response
| Field | Type | Description |
|---|---|---|
entry | object | The entry |
points | array | The concurrency points, ordered by concurrency |
Fields of entry:
| Field | Type | Description |
|---|---|---|
entryUuid | string | Entry id |
systemName, acceleratorName | string | Public system name and accelerator name |
gpuCount | integer | Number of GPUs |
frameworkCode, frameworkDisplay, frameworkVersion | string | Framework |
tensorParallelSize | integer | Tensor parallel size |
modelCode, modelName | string | Model code and display name |
huggingfaceId | string | Hugging Face model id |
precisionCode, precisionDisplay | string | Precision code and display name |
precisionSource | string | How the precision was set: recorded, model_name, or admin |
kvCacheDtype | string | KV cache data type |
workloadCode, workloadDisplay | string | Workload preset |
inputSequenceLength, outputSequenceLength | integer | Sequence lengths in tokens |
gpuDriverVersion, cudaVersion, rocmVersion | string | Software versions of the system |
datasetName | string | Dataset that supplied the prompts |
releaseTag | string | Benchmark tool release |
runCompletedAt, publishedAt | string | Times in ISO 8601 |
Fields of each item in points:
| Field | Type | Description |
|---|---|---|
concurrency | integer | Number of concurrent requests |
totalTokensPerSecond | number | Total tokens per second |
totalTokensPerSecondPerGpu | number | Total tokens per second per GPU |
outputTokensPerSecond | number | Output tokens per second |
interactivity | number | Output tokens per second per request |
ttftP50Ms, ttftP99Ms | number | Median and 99th percentile time to first token, in milliseconds |
tpotP50Ms | number | Median time per output token, in milliseconds |
latencyP50Ms | number | Median end-to-end latency, in milliseconds |
avgGpuPowerW | number | Average GPU power, in watts |
joulesPerOutputToken | number | Energy per output token |
requestCount | integer | Number of requests at the point |
A field that the run did not measure is null.