Skip to main content

KYAI - qualitative evaluation

KYAI ("Know Your AI") evaluates inference quality: how accurate a model's outputs are, not how fast it serves them. It sends a fixed set of questions to the model, then scores each response against a ground truth using a platform-managed judge.

The headline metric is the KYAI Score (average correctness, 0.0 to 1.0). Generation latency is also captured and used as a tiebreaker on the leaderboard when scores are equal.

Full methodology: KYAI Methodology.

Step 1: create the project and pick KYAI​

  1. Go to Projects → New Project.
  2. Enter a Project Name and set Visibility (Private or Team).
  3. Under Workloads, open Quality Evaluation and click KYAI to add a workload card.

Step 2: configure the candidate model​

Work through the card in order:

  1. Target Hardware: select a server chip and GPU Count.
  2. Models: pick the model to evaluate.
  3. Framework: choose vLLM, SGLang, or TensorRT-LLM. If only one framework is compatible, it auto-selects and locks.
  4. Version: pick the framework release. The latest compatible version auto-selects.
  5. Build Profile: pick the hardware build/launch variant (for example Default (CUDA) or ROCm) when profiles are available.
  6. Engine Args Set: pick a pre-configured template, or leave unset for the framework default. Custom org sets show "(custom)".

When editing engine args, ensure:

  • Tensor parallelism (TP / tp) is set to the same value as the hardware GPU count. KYAI runs with replicas = 1 (no replica fan-out), so all GPUs belong to one server process.
  • For default thinking / reasoning models (for example DeepSeek-R1 or any model that emits chain-of-thought blocks), add a reasoning parser to the engine args so the framework separates chain-of-thought from the final answer. Without it, thinking tokens mix into the response the judge scores.

Step 3: configure the dataset​

KYAI evaluates against fixed question snapshots or custom uploads.

Built-in snapshots (100 questions each, preselected):

DatasetDomainAnswer format
MMLU-ProAcademic reasoning (14 disciplines)Answer: X (A to J)
GPQA DiamondGraduate-level expert science MCQAnswer: X (A to D)
MATH-500Competition mathematicsFinal answer: <answer>
Python Codes 25KPractical Python generationValid Python, no fences
HumanEvalPython function completionIndented function body only
Quantum MechanicsPhysics chain-of-thoughtStep-by-step + final answer
IFEvalInstruction-following precisionSatisfies all stated constraints

Custom dataset (user-uploaded): Upload a JSONL or CSV file with two mandatory fields: instruction (prompt sent to the candidate) and ground_truth_response (reference answer). You also provide two prompts: a generation system prompt and a judge prompt. There is no built-in prompt picker for custom uploads.

Step 4: run and read results​

Select Save & Run. The run progresses through two phases:

  1. Generation: the candidate receives each prompt and produces a response. All outputs are stored verbatim.
  2. Judging: the Metrum AI Judge scores each response against the ground truth. The judge does not know which model produced the response.

Filter the Reporting page to the KYAI workload. Review per evaluation:

  • KYAI Score: mean correctness across the fixed question set (0.0 to 1.0). Errors count as zero.
  • Per-prompt scores and reasoning: inspect low scores for failure modes.
  • Leaderboard: candidates on the same dataset + same prompts ranked by mean score (latency as tiebreaker).

See KYAI Methodology for scoring rules, fail-closed behavior, and comparison keys.

The API path​

KYAI workloads and runs can be created over the API. See KYAI Workload And Run and KYAI Evaluation for the full API reference.