KYAI Methodology
KYAI (Know Your AI) measures the output quality of a language model against fixed reference questions and ground truth answers. Use it when the decision depends on response quality: comparing models of different sizes or quantization levels, validating a fine-tuned model, or comparing multiple self-hosted candidates.
The result is a KYAI Score: average correctness across all prompts, computed from per-prompt judgements by the Metrum AI Judge.
How It Works
Evaluation Principles
- Snapshot-pinned. Every evaluation runs against a named, immutable dataset of 100 questions. Every run evaluates all 100 questions in fixed order, so scores are directly comparable across candidates, hardware configurations, and time.
- Standardized conditions. All candidates receive identical prompts and the same generation system prompt. No candidate gets additional context or a more permissive format contract.
- Judge-independent scoring. The Metrum AI Judge is completely separate from the candidate and has no knowledge of which model produced a response.
- Fail-closed. Errors score zero, for both the candidate API and the judge API. A model cannot improve its average by failing to answer difficult questions.
Two-Phase Pipeline
Phase 1: Generation. The candidate receives each prompt and produces a response. All outputs are stored verbatim before any scoring begins. Raw outputs can be inspected, re-evaluated with a different judge, or compared across runs without re-calling the candidate.
Phase 2: Evaluation. The Metrum AI Judge receives the original instruction, the ground truth answer, and the candidate's response. It returns a score between 0 and 1 with a reasoning string. The judge does not know which model produced the response.
This separation makes it possible to score the same generation with multiple judges and cross-validate results.
Reasoning Model Handling
Models that emit reasoning tokens (such as DeepSeek-R1 or other models using
<think>...</think> blocks) have their thinking content automatically stripped
before the response is sent to the judge. The raw response including all
reasoning tokens is stored intact. Only the judge-facing copy is cleaned so
that IFEval constraint checks and MCQ letter extraction are not contaminated by
internal reasoning text.
Candidate Models
KYAI evaluates models served on the user's own infrastructure with one of these LLM serving frameworks:
| Framework | Notes |
|---|---|
| vLLM | Default OpenAI-compatible server. See vLLM recipes for per-model launch flags. |
| SGLang | OpenAI-compatible server with RadixAttention. See the SGLang cookbook. |
| TensorRT-LLM | NVIDIA-optimised serving via trtllm-serve. See the TensorRT-LLM docs. |
External / hosted APIs are not used as candidates.
Engine Arguments and Reasoning Models
Engine arguments come from the workload Engine Args Set and control how the framework loads and serves the model (tensor parallelism, memory, context length, and related flags). Copy verified flags from the framework docs above when building a set.
For default thinking models and reasoning models (for example DeepSeek-R1 or models that emit thinking / chain-of-thought blocks), set a reasoning parser in the engine args so the framework separates chain-of-thought from the final answer. Without it, thinking tokens can mix into the response that the judge scores.
- SGLang: set
reasoning-parserto the detector that matches the model. - vLLM: configure the corresponding reasoning / thinking separation for the model.
- TensorRT-LLM: follow the TRT-LLM docs for the model’s reasoning path.
KYAI also strips residual thinking tags from the judge-facing copy of the response. The raw generation (including reasoning tokens) is stored intact.
Max Output Tokens
KYAI does not expose a user-facing max-tokens control. The output budget is derived from the model’s default context length as reported by the live server.
The agent queries the OpenAI-compatible /v1/models endpoint for
max_model_len. vLLM and SGLang both report this field on the model card.
When the live server does not report it (as with TensorRT-LLM), the agent
reads the model’s HuggingFace config.json instead.
The output limit is set to most of that context window so the instruction prompt still fits. If a request still overflows the context, the budget is reduced automatically and the call is retried.
The Metrum AI Judge
The judge is a platform-managed evaluation model hosted on Metrum infrastructure
(kyai-judge). Users do not need to provide API keys or configure judge
endpoints. The platform routes all evaluation requests through this centralized
judge automatically.
Datasets
All evaluations use snapshot-based datasets: fixed, versioned pools of publicly
available benchmarks. Each snapshot stores 100 questions pinned at a specific
content version (v1) and loaded from Metrum's artifact store on every install
or upgrade.
Each evaluation run evaluates all 100 questions in deterministic row order. There is no random sampling and no per-run seed. The same questions are evaluated in every run, making scores directly comparable across candidates, hardware configurations, and time.
| Dataset | Domain | Questions | Answer Format | Generation Prompt | Judge Prompt |
|---|---|---|---|---|---|
| MMLU-Pro | Academic reasoning (14 disciplines) | 100 | Answer: X (single letter A to J) | mmlu-pro-mcq | mcq-exact-match-judge |
| GPQA Diamond | Graduate-level expert science MCQ | 100 | Answer: X (single letter A to D) | gpqa-mcq | mcq-exact-match-judge |
| MATH-500 | Competition mathematics | 100 | Final answer: <answer> | math-500-answer | math-final-answer-judge |
| Python Codes 25K | Practical Python generation | 100 | Valid Python, no fences | code-python | code-python-judge |
| HumanEval | Python function completion | 100 | Indented function body only | humaneval-code | humaneval-code-judge |
| Quantum Mechanics | Physics chain-of-thought | 100 | Step-by-step + final answer | quantum-mechanics-reasoning | physics-reasoning-judge |
| IFEval | Instruction-following precision | 100 | Satisfies all stated constraints | ifeval-instruction | ifeval-constraint-judge |
Dataset Details
MMLU-Pro (TIGER-Lab/MMLU-Pro, HuggingFace). 10-option multiple choice (A to J) across 14 academic disciplines. The 10-option format is significantly more discriminative than 4-option MCQ and provides a reliable cross-domain baseline.
GPQA Diamond (Idavidrein/gpqa gpqa_diamond subset, HuggingFace, CC BY 4.0). 4-option MCQ in biology, physics, and chemistry, written and validated by domain experts. Non-experts reach approximately 34% accuracy even with web access.
MATH-500 (HuggingFaceH4/MATH-500, HuggingFace). Competition-level problems across algebra, geometry, number theory, combinatorics, and calculus. Highly sensitive to quantization-induced degradation and resists surface-level pattern matching.
Python Codes 25K (flytech/python-codes-25k, HuggingFace). Practical Python generation: data manipulation, string processing, utility scripts. Covers everyday developer workloads rather than competitive algorithms.
HumanEval (openai/openai_humaneval, HuggingFace, MIT). Function stub + docstring; model must complete the function body. The most widely cited Python code generation benchmark.
Quantum Mechanics (0xZee/dataset-CoT-Quantum-Mechanics-1224, HuggingFace). Chain-of-thought physics problems requiring derivation and a precise final answer. Differentiates models that genuinely understand physics from those that pattern-match.
IFEval (google/IFEval, HuggingFace, Apache 2.0). Natural-language prompts each containing one or more verifiable constraints (word count, punctuation rules, structural requirements). Instruction-following precision is a production prerequisite.
Prompts and Judging
Each dataset has two dedicated prompts: a generation system prompt sent to the candidate and a judge prompt used by the Metrum AI Judge when scoring. Both are fixed per dataset (see the table above) to ensure consistent, comparable evaluation conditions.
Generation System Prompts
These are sent as the system message to the candidate model. Changing the prompt changes the evaluation conditions and makes results incomparable.
mmlu-pro-mcq
You are a multiple choice exam assistant. The question includes lettered options.
Respond with ONLY the single uppercase letter of the correct answer on its own
line in the format: Answer: X (where X is the letter). Do not include any
explanation, reasoning, or additional text.
gpqa-mcq
You are an expert science exam assistant. The question includes four lettered
options (A through D). Respond with ONLY the single uppercase letter of the
correct answer on its own line in the format: Answer: X (where X is the letter).
Do not include any explanation, reasoning, or additional text.
math-500-answer
You are a math assistant. Provide the final answer to the problem on its own line
in the format: Final answer: <answer>. The answer must be mathematically equivalent
to the correct result. Do not include explanation, reasoning, or derivation steps.
code-python
You are a Python coding assistant. Respond with ONLY valid Python code that
satisfies the request. Do not include Markdown fences or explanation.
humaneval-code
You are completing a Python function. The function signature and docstring are
provided. Output ONLY the indented function body lines -- nothing else. No def
line, no docstring, no Markdown fences, no imports, no explanation. The output
must be valid Python indented code that can be inserted directly below the
function signature.
quantum-mechanics-reasoning
You are a physics assistant. Answer the question directly and state the final
answer clearly. Do not include extended derivation steps or chain-of-thought
reasoning.
ifeval-instruction
Follow all formatting and content instructions in the prompt exactly. Read
carefully -- the prompt includes specific, verifiable requirements (such as word
count, punctuation rules, keyword inclusion, or structural constraints). Respond
directly -- do not include any preamble, planning notes, or analysis before your
answer. Your response itself is what is evaluated against every stated requirement.
Judge Prompts
These are the system prompts used by the Metrum AI Judge when scoring candidate responses. Each prompt is matched to the answer format of its dataset.
mcq-exact-match-judge (MMLU-Pro, GPQA Diamond)
You are evaluating a multiple-choice answer. The ground truth is a single
uppercase letter (e.g. "A", "B", "C").
Step 1: Extract the candidate's chosen answer -- identify the single uppercase
letter the candidate selected as their final answer. The candidate may state it
plainly ("C"), prefix it ("The answer is C"), write it as "Answer: C", or embed
it in a sentence. If the response contains reasoning or analysis before the
final answer, extract the letter from the final answer only. Extract only the
letter. If no clear letter can be identified, score 0.0.
Step 2: Compare -- if the extracted letter matches the ground truth letter
exactly, score 1.0; otherwise score 0.0.
Return JSON {"score": 1.0 or 0.0, "reasoning": "Extracted letter: X. Ground
truth: Y."} No other output.
math-final-answer-judge (MATH-500)
You are evaluating a mathematics solution against a reference answer.
Step 1: Extract the ground truth answer -- if the ground truth is a full
solution, extract the final stated answer. If it is already just an answer,
use it directly.
Step 2: Extract the candidate's final answer -- look for "Final answer: ...",
a boxed expression, or the last clearly stated mathematical result. If no final
answer can be identified, score 0.0.
Step 3: Compare -- determine whether the two answers are mathematically
equivalent. Accept equivalent forms (e.g. 1/2 = 0.5, sqrt(2)/2 = 1/sqrt(2)).
Score 1.0 if equivalent, 0.0 if not.
Return JSON {"score": 1.0 or 0.0, "reasoning": "Ground truth answer: X.
Candidate answer: Y. Equivalent: yes/no."} No other output.
code-python-judge (Python Codes 25K)
You are evaluating Python code against a task description and a reference
implementation.
If the response contains explanation or natural language alongside the code,
evaluate only the code portion. Determine whether the candidate code is
functionally correct -- it would produce the correct output for the inputs
described in the task. The candidate does not need to use the same algorithm
or structure as the reference.
Penalise: incorrect output for described inputs, syntax errors, wrong return
type. Do not penalise: different algorithm, different variable names, different
code style.
Return JSON {"score": <0.0-1.0>, "reasoning": "..."} reflecting functional
correctness. No other output.
humaneval-code-judge (HumanEval)
You are evaluating a Python function completion against a canonical
implementation.
If the response contains explanation or commentary alongside the code, evaluate
only the code portion. Determine whether the candidate correctly implements the
function -- correct algorithm, correct behaviour for the cases described in the
docstring.
Return JSON {"score": <0.0-1.0>, "reasoning": "..."} reflecting correctness.
No other output.
physics-reasoning-judge (Quantum Mechanics)
You are evaluating a physics response against a reference answer.
Determine whether the candidate reaches the same final answer as the reference.
Accept mathematically equivalent forms and equivalent physical descriptions.
Score 1.0 if correct, 0.0 if wrong, 0.5 if partially correct (e.g. correct
method but arithmetic error yielding a close but incorrect value).
Do not evaluate reasoning quality, derivation steps, or explanation style.
Only the final answer matters.
Return JSON {"score": <0.0-1.0>, "reasoning": "Ground truth answer: X.
Candidate answer: Y. Correct/Incorrect/Partial."} No other output.
ifeval-constraint-judge (IFEval)
You are evaluating an instruction-following response. The ground truth lists
the verifiable constraints the response must satisfy.
Important: some responses contain a reasoning or planning section before the
actual answer. Evaluate ONLY the final answer. Ignore any meta-commentary,
planning, or reasoning that precedes it.
Check each constraint independently -- satisfied (1) or violated (0).
Score = satisfied constraints / total constraints.
Return JSON {"score": <0.0-1.0>, "reasoning": "..."} where reasoning lists
each constraint and its result. No other output.
Scoring
Score Calculation
Average Score (avg_score): mean of all per-prompt scores across the
evaluation set. Errors count as 0 and are never excluded from the denominator.
A high average score indicates the model is consistently correct across all 100 questions. Because every run evaluates the same fixed question set, scores are directly comparable across runs, candidates, and time.
Fail-Closed Rules
| Condition | Score |
|---|---|
| Judge returns a valid score 0.0 to 1.0 | That score |
| Candidate API error | 0 |
| Judge API error | 0 |
| Judge response unparseable | 0 |
| Score field null or absent | 0 |
Why LLM-as-Judge
The Metrum AI Judge is used rather than exact-match scoring because:
- Semantically equivalent answers are not textually identical (
pi/2and1.5708are the same answer). - Format variation should not penalize a correct response.
- Open-ended outputs (physics reasoning, code, instruction following) cannot be scored meaningfully by string comparison.
The judge is nonetheless instructed to penalize format violations, contradictions, and missing required details.
Leaderboard and Comparison
Valid Comparisons
Two results are directly comparable only when they share the same comparison key:
- same dataset (snapshot)
- same judge configuration and judge system prompt
- same generation system prompt
- same scoring method
Results with different comparison keys answer different questions and cannot be ranked against each other.
To compare multiple candidates, evaluate each on the same dataset with the same judge configuration. Candidates sharing the same dataset and judge automatically appear in the same leaderboard group. Each workload holds one candidate endpoint; create one workload per candidate to build a comparison set.
Ranking
Candidates are ranked in order:
avg_scoredescending: overall correctnessavg_latency_msascending: generation speed, tiebreaker only
Re-Judging
The same generated responses can be scored again with a different judge configuration. Each judge run produces an independent leaderboard row. Useful for cross-validating judge agreement or auditing a borderline result. Re-judging does not require re-running the candidate model.
Custom Datasets
Teams can evaluate models on their own data when domain-specific capability matters more than cross-model rankings on public benchmarks.
Dataset Format
Upload a JSONL or CSV file with two mandatory fields:
| Field | Purpose |
|---|---|
instruction | The prompt sent to the candidate model |
ground_truth_response | The reference answer the judge compares against |
An optional external_row_id field is accepted from both the CLI and the web
upload (as a JSONL field or a CSV column) and is preserved for cross-referencing
rows back to the original source. If omitted, rows are assigned a sequential ID.
Generation and Judge Prompts
For custom datasets, you provide both prompts:
- Generation system prompt — sent to the candidate model before each instruction.
- Judge prompt — sent to the Metrum AI Judge, telling it how to evaluate responses against the ground truth.
There is no built-in prompt selection for custom datasets. Both prompts must be supplied when uploading the dataset. They are stored with the dataset and reused on every subsequent run.
Scoring
The scoring logic is handled entirely by the platform. The Metrum AI Judge evaluates each response against the ground truth and returns a score. You do not configure scoring rules or thresholds; you only describe the evaluation criteria in the judge prompt.