KYAI Methodology

KYAI (Know Your AI) measures the output quality of a language model against a fixed set of reference questions and ground truth answers. Use it when the decision depends on response quality — comparing models of different sizes or quantization levels, validating a fine-tuned model, or selecting from multiple candidate endpoints.

The result is a KYAI Score: average correctness and pass rate across all prompts, computed from per-prompt judgements by an independent LLM judge.

Evaluation Principles

Snapshot-pinned. Every evaluation runs against a named, immutable dataset. The same 25 questions are used each time — no silent drift between runs.
Standardized conditions. All candidates receive identical prompts, the same generation system prompt, and the same random seed. No candidate gets additional context or a more permissive format contract.
Judge-independent scoring. The judge is completely separate from the candidate and has no knowledge of which model produced a response.
Fail-closed. Errors score zero — for both the candidate API and the judge API. A model cannot improve its average by failing to answer difficult questions.

Candidate Models

A candidate is any model endpoint KYAI sends prompts to. Two types are supported:

External / hosted — any OpenAI-compatible hosted API (OpenAI, OpenRouter, Anthropic, etc.). The user provides the API base URL, model name, and API key when registering the endpoint.
Self-hosted — models served by the user's own infrastructure (vLLM, SGLang, TRT-LLM, etc.), registered during GPU server onboarding. No external credentials required.

Both types use the same evaluation pipeline and appear on the same leaderboard.

Judge Configuration

The judge is an independent language model that scores candidate responses. Built-in judge configurations cover models from OpenAI and Anthropic (via OpenRouter). Judge API credentials are platform-managed — users do not provide keys for the judge.

Two things affect comparability: the judge model chosen and the judge system prompt used. Results scored with different judge configurations are tracked separately and should not be directly compared.

Two-Phase Evaluation Pipeline

Phase 1 — Generation. The candidate receives each prompt and produces a response. All outputs are stored verbatim before any scoring begins. This means raw outputs can be inspected, re-evaluated with a different judge, or compared across runs without re-calling the candidate.

Phase 2 — Evaluation. The judge receives the original instruction, the ground truth answer, and the candidate's response. It returns a score between 0 and 1 with a reasoning string. The judge does not know which model produced the response.

This separation makes it possible to score the same generation with multiple judges and cross-validate results.

Evaluation Datasets

All evaluations use snapshot-based datasets — fixed, versioned subsets of publicly available benchmarks. Each snapshot records its source, sample count, and random seed for full reproducibility. Every standard snapshot uses 25 samples.

Dataset	Domain	Samples	Answer Format
MMLU-Pro	Academic reasoning (14 disciplines)	25	Single letter A–J
Quantum Mechanics	Physics chain-of-thought	25	Step-by-step + final answer
MATH-500	Competition mathematics	25	`Final answer: <answer>`
Python Codes 25K	Practical Python generation	25	Valid Python, no fences
HumanEval	Python function completion	25	Valid Python, no fences
GPQA Diamond	Graduate-level expert science MCQ	25	Single letter A–D
IFEval	Instruction-following precision	25	Satisfies all stated constraints

MMLU-Pro

Source: TIGER-Lab/MMLU-Pro (HuggingFace)
Format: 10-option multiple choice (A–J), 14 academic disciplines
Why: The 10-option format is significantly more discriminative than 4-option MCQ and provides a reliable cross-domain baseline

Quantum Mechanics

Source: 0xZee/dataset-CoT-Quantum-Mechanics-1224 (HuggingFace)
Format: Chain-of-thought physics problems requiring derivation and a precise final answer
Why: Domain-specific scientific reasoning differentiates models that genuinely understand physics from those that pattern-match on general benchmarks

MATH-500

Source: HuggingFaceH4/MATH-500 (HuggingFace)
Format: Competition-level problems across algebra, geometry, number theory, combinatorics, and calculus
Why: Structured mathematical reasoning is highly sensitive to quantization-induced degradation and resists surface-level pattern matching

Python Codes 25K

Source: flytech/python-codes-25k (HuggingFace)
Format: Practical Python generation — data manipulation, string processing, utility scripts
Why: Covers everyday developer workloads rather than competitive algorithms, testing a different capability than HumanEval

HumanEval

Source: openai/openai_humaneval (HuggingFace, MIT) — problems 0–24
Format: Function stub + docstring; model must complete the function body
Why: The most widely cited Python code generation benchmark, providing a directly comparable industry reference point

GPQA Diamond

Source: Idavidrein/gpqa — gpqa_diamond subset (HuggingFace, CC BY 4.0)
Format: 4-option MCQ in biology, physics, and chemistry, written and validated by domain experts
Why: Non-experts reach ~34% accuracy even with web access; the benchmark directly separates deep reasoning from pattern matching

IFEval

Source: google/IFEval (HuggingFace, Apache 2.0)
Format: Natural-language prompts each containing one or more verifiable constraints (word count, punctuation rules, structural requirements)
Why: Instruction-following precision is a production prerequisite; a model that reasons correctly but ignores format contracts is unsuitable for structured workflows

Generation System Prompts

Each dataset is evaluated with a fixed generation system prompt sent as the system message to the candidate. The prompt is identical for all candidates on the same snapshot — changing it changes the evaluation conditions and makes results incomparable.

Dataset	Prompt	Format contract
MMLU-Pro	`mmlu-pro-mcq`	Single uppercase letter only
Quantum Mechanics	`quantum-mechanics-reasoning`	Work through physics; end with a clear final answer
MATH-500	`math-500-answer`	`Final answer: <answer>` on its own line
Python Codes 25K	`code-python`	Valid Python only, no Markdown fences
HumanEval	`code-python`	Valid Python only, no Markdown fences
GPQA Diamond	`gpqa-mcq`	Single uppercase letter (A–D) only
IFEval	`ifeval-instruction`	Satisfy every stated constraint exactly

Prompt Text Reference

mmlu-pro-mcq

You are a multiple choice exam assistant. The question includes lettered options.
Respond with ONLY the single uppercase letter of the best answer. Do not explain.

quantum-mechanics-reasoning

You are a quantum mechanics assistant. Work through the physics carefully using
standard notation, keep the reasoning concise and correct, and end with a clear
final answer that directly addresses the question.

math-500-answer

You are a competition math assistant. Solve the problem carefully and provide the
final answer on its own line in the format: Final answer: <answer>. Keep the final
answer mathematically equivalent to the correct result, and do not include any
extra closing commentary after that line.

code-python

You are a Python coding assistant. Respond with ONLY valid Python code that
satisfies the request. Do not include Markdown fences or explanation.

gpqa-mcq

You are an expert science exam assistant. The question includes four lettered
options (A–D). Respond with ONLY the single uppercase letter of the correct
answer. Do not explain.

ifeval-instruction

Follow all formatting and content instructions in the prompt exactly. Read
carefully — the prompt includes specific, verifiable requirements (such as word
count, punctuation rules, keyword inclusion, or structural constraints). Your
response must satisfy every stated requirement.

Scoring Methodology

LLM-as-Judge

An independent language model evaluates each candidate response against the ground truth. LLM-as-judge is used rather than exact-match scoring because:

Semantically equivalent answers are not textually identical (π/2 and 1.5708 are the same answer).
Format variation should not penalize a correct response.
Open-ended outputs — physics reasoning, code, instruction following — cannot be scored meaningfully by string comparison.

The judge is nonetheless instructed to penalize format violations, contradictions, and missing required details.

Judge Prompt

You are an expert evaluator. Given an instruction, a reference answer, and a
candidate response, score the candidate response on a scale of 0-1 based on
correctness. Allow semantically equivalent answers when appropriate, but do not
reward contradictions, missing required details, or format violations. Return
ONLY a JSON object with "score" (0.0-1.0) and "reasoning" fields.

The judge returns:

{
  "score": 0.85,
  "reasoning": "The candidate correctly identifies the answer but omits one supporting detail present in the reference."
}

Fail-Closed Scoring

Condition	Score
Judge returns a valid score 0.0–1.0	That score
Candidate API error	0
Judge API error	0
Judge response unparseable	0
Score field null or absent	0

Score Calculation

Average Score (avg_score) — mean of all per-prompt scores across the evaluation set. Errors count as 0 and are never excluded from the denominator.

Pass Rate (pass_rate) — fraction of prompts where the score was 0.9 or above. Measures how often the model produces a near-perfect response, not merely a partially correct one.

A model with high avg_score but low pass_rate is consistently good but rarely excellent.

Leaderboard and Comparison Rules

What Makes a Valid Comparison

Two results are directly comparable only when they share the same comparison key:

same dataset (snapshot)
same judge configuration and judge system prompt
same generation system prompt
same scoring method

Results with different comparison keys answer different questions and cannot be ranked against each other.

To compare multiple candidates against each other, evaluate each on the same dataset with the same judge configuration. No additional setup is needed — candidates sharing the same dataset and judge automatically appear in the same leaderboard group and are ranked together. Each workload holds one candidate endpoint; create one workload per candidate to build a comparison set.

Ranking Within a Comparison

Candidates are ranked in order:

avg_score descending — overall correctness
pass_rate descending — frequency of near-perfect answers
sample_count descending — larger samples are statistically more reliable
avg_latency_ms ascending — generation speed, tiebreaker only

Re-Judging

The same generated responses can be scored again with a different judge configuration. Each judge run produces an independent leaderboard row. Useful for cross-validating judge agreement or auditing a borderline result. Re-judging does not require re-running the candidate model.

Custom Datasets

Teams can evaluate models on their own data when domain-specific capability matters more than cross-model rankings on public benchmarks.

Dataset Format

Upload a JSONL or CSV file with two required fields:

instruction — the prompt sent to the candidate model
ground_truth_response — the reference answer the judge compares against

For multiple-choice questions, embed the lettered options in the instruction text. For context-dependent tasks, prepend the context to the instruction. The platform handles import and storage automatically.

Generation System Prompt

Every evaluation run requires a generation system prompt. When uploading a custom dataset, choose one of:

Option A — Use a built-in prompt if your dataset matches a known task type:

Dataset type	Built-in prompt
Multiple choice (A/B/C/D)	MMLU-Pro MCQ or GPQA Expert Science MCQ
Python code generation	HumanEval Python Code
Competition math	MATH-500 Final Answer
Instruction-following	IFEval Instruction Following
Physics / science reasoning	Quantum Mechanics Chain-of-Thought

Option B — Use the general-purpose prompt for mixed or unknown task types:

You are a knowledgeable assistant. Answer the question or complete the task as accurately as possible. If the question has a single correct answer, state it directly. If it requires an explanation, keep your response clear and concise.

The prompt is stored with the dataset and reused on every subsequent run. For datasets with strict output requirements (a single letter, a number, a code block), choose a prompt that specifies the expected format explicitly — the judge compares raw response text against the ground truth.

Evaluation Principles​

Candidate Models​

Judge Configuration​

Two-Phase Evaluation Pipeline​

Evaluation Datasets​

MMLU-Pro​

Quantum Mechanics​

MATH-500​

Python Codes 25K​

HumanEval​

GPQA Diamond​

IFEval​

Generation System Prompts​

Prompt Text Reference​

Scoring Methodology​

LLM-as-Judge​

Judge Prompt​

Fail-Closed Scoring​

Score Calculation​

Leaderboard and Comparison Rules​

What Makes a Valid Comparison​

Ranking Within a Comparison​

Re-Judging​

Custom Datasets​

Dataset Format​

Generation System Prompt​