Running benchmarks
The core unit of work is a project containing one or more workloads. A workload describes what to benchmark (model + framework + scenario matrix); the project organizes them and produces runs when executed.
Create and run a project
The Quickstart covers this path end to end. The short version:
- Go to Projects, then select New Project. The editor opens at
/dashboard/projects/new. - Enter a Project name.
- Under Who can see it, choose Team or Private. Team lets everyone in your organization open the project, and it is the default. Private keeps the project to you.
- Under Add a workload, select the benchmark that matches what you test. The workload opens as a panel.
- Work through the steps of the panel. A workload that starts its own model server has four steps: Model, Serving engine, Hardware, and Load. A BYOE (Bring Your Own Endpoint) workload has three: Endpoint, Hardware, and Load. In a BYOE workload, the hardware is the machine that sends the load to your endpoint.
- Select Save to keep the project for later, or Save & Run to save it and start its runs.
The editor works like this:
- Discard, Save, and Save & Run appear at the top and at the bottom of the page.
- A step stays open until you select its Next button or select another step's header. Filling in a step does not open a later step. For example, a model choice does not skip the Serving engine step.
- One workload panel is open at a time. When you open a panel, the other panels close. A new workload opens its own panel.
- A closed panel with a problem shows a badge with the number of steps that need attention. When you open the panel, the step with the problem is open.
- If a workload is missing a required part, such as a model or a serving engine version, the save stops and lists that workload. Select Finish to complete it, or select Remove and Save (Remove Them and Save for several workloads) to save the other workloads.
- Save & Run is turned off when your organization has no active plan.
- On long pages, a back-to-top control appears after you scroll down. You can also run Back to top from the Cmd+K palette.
What Save & Run does
- The editor runs the pre-run checks on the project as it is on screen. They check your plan, the hardware choice, the machines and their readiness, the GPU vendor, and any KYAI endpoint and judge.
- If a check fails, nothing is saved. The editor shows "Fix these before running. Nothing was saved." and the reason for each problem. It marks each step that has a problem, and that step is open in its panel.
- If every check passes, the editor saves the project and starts the runs of every workload.
- A workload that runs on a cluster group waits its turn when the group is busy. The editor tells you how many workloads wait.
- When the runs start, the Monitoring page opens with your project highlighted. If some runs do not start, the project page opens, and a message names each workload and the reason.
Edit a project
Results stay tied to the settings that produced them. For that reason, what you can change depends on whether the project has ever run.
| The project | What the editor shows | Actions |
|---|---|---|
| Has never run, and you can change it | The saved settings | Save and Save & Run change this project. |
| Has run at least once | Every saved setting, and the notice "This project has run before. Your changes save as a new project. This project and its results stay as they are." | Save as New and Save as New & Run save your changes as a new project. |
| Is one you can view but not change | The saved settings, and a notice that your changes save as a new project that you own | Save as New and Save as New & Run. |
To open the editor, open the project page and select Edit Project. When the project has run, or when you cannot change it, the button is Copy Project instead. On the project page, the Cmd+K palette also offers Edit this project or Copy this project.
How the rule works:
- A first run locks the project. The project counts as run the first time it starts any execution, such as a run, a job, a queued distributed KV run, or a cloud launch. The record stays when you delete the runs, so deleting runs does not unlock a project.
- A refused start does not lock the project. If the server refuses a start before anything runs, the project stays editable.
- The rule covers every setting. It covers the project name, the visibility, the workloads, and the scenarios. The CLI and the API refuse the same changes. Use
benchctl project copyor Save as New in the editor, then change the copy. - A copy is a new project that you own. If you keep the name, the copy is named
<name> (copy), then<name> (copy 2), and so on. The original project and its results do not change. - A copy links to its source. The project page and the editor show "Copied from" with a link to the source project. You see this link only when you can open the source project.
- Two people cannot overwrite each other. If someone saves the project after you opened it, your save is refused. The dialog "Someone saved this project after you opened it" offers Reload to see their changes, or Keep My Changes to save your changes as a new project.
- A project that runs while you edit it cannot be saved in place. Your changes stay in the editor, and you save them as a new project.
- Discard reloads the project as it was last saved. On a new project, Discard clears every workload and setting.
- A retired engine version stays visible. If a saved engine version is no longer offered, the editor shows it as "(no longer offered)" and asks you to choose a current version before you run the project again.
Multi-workload projects
Use multi-workload projects when you want to compare several related configurations as a single deliverable (for example "Llama 70B on vLLM, SGLang, and TRT-LLM, at three concurrencies each").
- Go to Projects, then select New Project.
- Under Add a workload, select a benchmark for the first workload.
- Select more benchmarks under Add a workload to add more workloads. Each workload has its own model, serving engine, hardware, and load.
- Select Save to keep the project, or Save & Run to start a run for every workload.
Multi-workload projects keep related work together. Every run from the project lands under the same project, and the Reporting page can filter to the project for cross-workload comparison.
Scenario matrices
Within a workload, concurrency, ISL, and OSL chips multiply out:
| Concurrency | ISL | OSL | Scenarios |
|---|---|---|---|
1 | 512 | 256 | 1 |
1, 8, 32 | 512 | 256 | 3 |
1, 8, 32 | 512, 2048 | 256 | 6 |
1, 8, 32 | 512, 2048 | 256, 512 | 12 |
Each scenario becomes one job in the run, named c<concurrency>-isl<isl>-osl<osl>. A 12-scenario run produces 12 rows in the Jobs table, run sequentially on the selected server. Start small: running a 24-scenario run on a 70B model can take an hour.
Available chip values in the workload card today are: concurrency 1, 4, 8, 16, 32, 64, 128; ISL 128, 256, 512, 1024, 2048; OSL 64, 128, 256, 512.
Picking a framework
| Framework | Best for |
|---|---|
vllm | General-purpose; widest model coverage; most portable |
sglang | High-throughput serving with structured output |
trt-llm | NVIDIA-only; highest peak throughput on H100/H200/B200 |
Defaults: start with vLLM. Add SGLang and TRT-LLM when comparing peak performance. Other framework codes (tgi, llama-cpp, ollama, dynamo, openai) are also registered in the catalog: see Feature Reference - Other registered frameworks for the support level of each.
Engine arguments (advanced)
Engine arguments are the flags passed to the serving framework when it starts the model. They decide how much GPU memory the model uses, how long a context it can handle, how many GPUs it spreads across, and how requests are batched. Getting them wrong is the most common cause of a failed or misleading run.
Do not pick these by guessing. The platform ships verified presets sourced from the official vLLM and SGLang recipes, matched to specific models and hardware. The full method for choosing them is its own section: Choosing engine arguments. Read it before you launch anything larger than a smoke test.