Skip to main content

Public leaderboard API

The public leaderboard API returns the entries of the hardware leaderboard. It has three GET routes. The routes need no token and no account.

The routes belong to the web app, not to the PostgREST API. They do not use the /api/postgrest proxy.

export BENCH_URL="https://bench.metrum.ai"

Conventions​

TopicValue
Base path/api/public/v1/leaderboard
MethodsGET and OPTIONS
AuthenticationNone. Do not send an Authorization header.
Response typeapplication/json
Field namescamelCase
Versionv1 in the path
CacheCache-Control: public, s-maxage=300, stale-while-revalidate=600, and an ETag
Rate limit60 requests per minute for each client IP address

Versioning​

The version is part of the path. A change that removes or renames a field ships under a new version path. The v1 routes keep their current fields.

Caching​

Every successful response carries Cache-Control: public, s-maxage=300, stale-while-revalidate=600 and an ETag. A shared cache can serve a response for 300 seconds. It can serve a stale response for 600 more seconds while it fetches a fresh one. Send the ETag back in If-None-Match to check for a change.

When Metrum AI publishes, renames, or withdraws an entry, the server refreshes the cached leaderboard data. Your own client cache follows the headers above.

Rate limit​

Each client IP address can send 60 requests per minute. A request over the limit receives HTTP 429 and a Retry-After header. The header holds the number of seconds to wait.

CORS​

Browser requests from an allowed origin succeed. The server answers GET and OPTIONS only. It does not allow credentials.

The default allowed origins are https://metrum.ai, https://www.metrum.ai, and https://bench.metrum.ai. An operator changes the list with the PUBLIC_API_ALLOWED_ORIGINS environment variable. A request from a server, curl, or a script does not need CORS.

Errors​

StatusWhenBody
400A parameter is missing or has a value the route does not acceptA plain sentence that names the problem
404The entry does not exist, or Metrum AI withdrew itA plain sentence
429You exceeded 60 requests per minuteA plain sentence, and a Retry-After header
503The database service is not availableA plain sentence

If you receive 429, wait the number of seconds in Retry-After. If you receive 503, retry after a short delay.

List the choices​

GET /api/public/v1/leaderboard/options

Returns every model, precision, and workload preset that has at least one published entry, and the default choice. Use it to fill the parameters of the board route.

This route takes no parameters.

curl -fsS "$BENCH_URL/api/public/v1/leaderboard/options"

Response​

FieldTypeDescription
optionsarrayOne row for each model, precision, and workload preset
defaultsobjectThe default model, precision, and workload, and target (50)

Each row of options:

FieldTypeDescription
modelCodestringModel code. Send it as model.
modelNamestringModel display name
precisionCodestringPrecision code. Send it as precision.
precisionDisplaystringPrecision display name
workloadCodestringWorkload preset code. Send it as workload.
workloadDisplaystringWorkload preset display name
inputSequenceLengthintegerInput Sequence Length of the preset, in tokens
outputSequenceLengthintegerOutput Sequence Length of the preset, in tokens
entryCountintegerNumber of published entries

List the ranked entries​

GET /api/public/v1/leaderboard

Returns the entries of one comparability key, ranked in one category. The key is the model, the precision, and the workload preset.

Parameters​

NameRequiredValuesDefault
modelYesA modelCode from the options routeNone
precisionYesA precisionCode from the options routeNone
workloadYesA workloadCode from the options routeNone
categoryNothroughput, interactivity, or ttftthroughput
targetNo20, 50, or 100 output tokens per second per request50
limitNo1 to 20050

A limit outside 1 to 200 is clamped to the nearest valid value. A category or target outside the table returns HTTP 400.

curl -fsS "$BENCH_URL/api/public/v1/leaderboard?model=<model-code>&precision=<precision-code>&workload=<workload-code>&category=throughput&target=50&limit=10"

Response​

FieldTypeDescription
cohortobjectThe model, precision, and workload of this list
categorystringThe category of this list
targetintegerThe interactivity target of this list
entriesarrayThe ranked rows. Rows that reached the target come first, in rank order. Rows that did not reach it follow, sorted by systemName.
generatedAtstringTime that the server built the response, in ISO 8601

Each row of entries:

FieldTypeDescription
rankinteger or nullPosition in the category. Null when the target is not reached, or when the category is ttft and the run did not measure it.
isTiedbooleanTrue when another entry has the same value and shares this rank
targetReachedbooleanTrue when at least one point meets the target
entryUuidstringEntry id. Use it in the entry route.
systemNamestringPublic system name
acceleratorNamestringAccelerator name
gpuCountintegerNumber of GPUs
frameworkCodestringFramework code
frameworkDisplaystringFramework display name
frameworkVersionstringFramework version
tensorParallelSizeintegerTensor parallel size
modelCode, modelNamestringModel code and display name
precisionCode, precisionDisplaystringPrecision code and display name
workloadCode, workloadDisplaystringWorkload preset code and display name
throughputPerGpunumber or nullTotal tokens per second per GPU at the throughput point. Null when the target is not reached.
throughputPointInteractivitynumberOutput tokens per second per request at the throughput point
throughputPointConcurrencyintegerConcurrency of the throughput point
peakInteractivitynumberBest output tokens per second per request across all points
ttftP50Msnumber or nullMedian time to first token at the throughput point, in milliseconds. Null when not measured.
joulesPerOutputTokennumber or nullEnergy per output token at the throughput point. Null when not measured.
pointCountintegerNumber of concurrency points in the entry
releaseTagstringBenchmark tool release, for example metrumbench-llm 0.1.80
runCompletedAtstringTime the source run finished
publishedAtstringTime Metrum AI published the entry

Get one entry​

GET /api/public/v1/leaderboard/entries/{entryUuid}

Returns one published entry and all its concurrency points.

Parameters​

NameInRequiredDescription
entryUuidPathYesAn entryUuid from the list route
curl -fsS "$BENCH_URL/api/public/v1/leaderboard/entries/<entry-uuid>"

The route returns HTTP 404 for an unknown entry and for a withdrawn entry.

Response​

FieldTypeDescription
entryobjectThe entry
pointsarrayThe concurrency points, ordered by concurrency

Fields of entry:

FieldTypeDescription
entryUuidstringEntry id
systemName, acceleratorNamestringPublic system name and accelerator name
gpuCountintegerNumber of GPUs
frameworkCode, frameworkDisplay, frameworkVersionstringFramework
tensorParallelSizeintegerTensor parallel size
modelCode, modelNamestringModel code and display name
huggingfaceIdstringHugging Face model id
precisionCode, precisionDisplaystringPrecision code and display name
precisionSourcestringHow the precision was set: recorded, model_name, or admin
kvCacheDtypestringKV cache data type
workloadCode, workloadDisplaystringWorkload preset
inputSequenceLength, outputSequenceLengthintegerSequence lengths in tokens
gpuDriverVersion, cudaVersion, rocmVersionstringSoftware versions of the system
datasetNamestringDataset that supplied the prompts
releaseTagstringBenchmark tool release
runCompletedAt, publishedAtstringTimes in ISO 8601

Fields of each item in points:

FieldTypeDescription
concurrencyintegerNumber of concurrent requests
totalTokensPerSecondnumberTotal tokens per second
totalTokensPerSecondPerGpunumberTotal tokens per second per GPU
outputTokensPerSecondnumberOutput tokens per second
interactivitynumberOutput tokens per second per request
ttftP50Ms, ttftP99MsnumberMedian and 99th percentile time to first token, in milliseconds
tpotP50MsnumberMedian time per output token, in milliseconds
latencyP50MsnumberMedian end-to-end latency, in milliseconds
avgGpuPowerWnumberAverage GPU power, in watts
joulesPerOutputTokennumberEnergy per output token
requestCountintegerNumber of requests at the point

A field that the run did not measure is null.