How to build a custom evaluation framework for LLMs

An LLM evaluation framework is the set of examples, graders and statistics you use to decide whether a model, prompt or pipeline is good enough for a specific task, and whether a change made it better. A custom framework is built from your own traffic and your own definition of correct, which is why it predicts production quality far better than any public benchmark.

The rest of this guide covers how to build one: what to measure, how many examples you need and why, which graders to trust, how to make LLM judges honest, and how to keep the whole thing running after launch.

Why don’t public benchmarks predict performance on your task?

Benchmarks are useful for screening models, but three problems keep them from answering “which model should run my workload?”

Contamination. Test questions leak into training data. When researchers wrote GSM1k, a fresh set of grade-school math problems matched to GSM8k in style and difficulty, they saw accuracy drops of up to 8% on some models, with some families showing systematic overfitting across almost all sizes (many frontier models showed little). Standard n-gram decontamination does not catch paraphrased or translated test items: Yang et al. showed a 13B model trained on rephrased MMLU, GSM8K and HumanEval items could reach scores comparable to GPT-4, and found 8 to 18% of HumanEval overlapping with two large public training corpora.

Saturation. The authors of Humanity’s Last Exam note that models now exceed 90% on popular benchmarks like MMLU. At that ceiling, a one-point gap between two models is mostly noise and says nothing about your task.

Mismatch. Your inputs, output format, failure costs and edge cases are your own. A support-ticket classifier that must return valid JSON with one of 14 internal categories is not tested by any leaderboard. A model that leads on general reasoning can still lose on your schema, your vocabulary and your long tail.

Use benchmarks to pick a shortlist. Use your own eval to pick the winner.

How do you define success before writing any grader?

Most failed evaluation efforts skip this step. Sit down with the person who owns the business outcome, not only the engineer who owns the prompt, and write down:

1. The decision the output feeds. Routing a ticket, drafting a reply a human edits, filling a database field, answering a customer directly.
2. What a correct output is, in checkable terms where possible (“category matches the label a senior agent would assign”, “every figure in the summary appears in the source document”).
3. Which errors are expensive. A wrong refund amount and a slightly stiff tone are not the same failure. Weight or gate on the expensive ones.
4. The bar for shipping. For example: schema validity at least 99.5%, category accuracy no worse than today’s system on any major slice, zero unsupported figures on the finance slice.
5. The baseline. What the current prompt and model score today. Without it, a number like “87%” means nothing.

Expect the definition to move. In a study of people grading LLM outputs, Shankar et al. observed “criteria drift”: users need criteria to grade outputs, but grading outputs helps users define criteria. Plan a first labelling session where the owner reads 50 real outputs and refines the rubric before you scale anything.

How do you build an eval set that reflects real traffic?

Sample from production, then stratify

Start from logged real inputs, with personal data handled under your existing data policies. A uniform random sample reproduces your traffic mix, which is right for estimating overall quality but leaves rare, high-stakes cases with a handful of examples. So stratify: define slices (by intent, customer tier, language, document length, channel) and sample enough from each to measure it on its own.

Keep two numbers separate. Report the traffic-weighted overall score so it reflects what users experience, and report per-slice scores so regressions on small slices are visible.

Add edge cases on purpose

Real traffic underrepresents the inputs that break systems. Add a deliberate edge-case slice: empty or irrelevant input, very long input, typos and mixed languages, prompt-injection attempts, ambiguous requests where even experts disagree, and questions whose correct answer is “I don’t have that information.” OpenAI’s and Anthropic’s evaluation guides each list several of these, such as typos, multilingual input, jailbreak attempts, irrelevant or overly long input, and cases where even humans would disagree.

Synthetic inputs can fill gaps in coverage, but label them as synthetic and report them as their own slice; the synthetic data guide covers how to generate them without teaching the eval to the model.

How big should it be?

Treat the eval as a sample from a larger population of possible inputs and put error bars on it. For a pass/fail metric with true pass rate p over n independent items, the standard error is √(p(1−p)/n), and the 95% interval is about ±1.96 standard errors.

Items (n)	95% margin at 80% pass rate	95% margin at 95% pass rate
100	±7.8 points	±4.3 points
200	±5.5 points	±3.0 points
400	±3.9 points	±2.1 points
1,000	±2.5 points	±1.4 points

Two refinements from Evan Miller’s “Adding Error Bars to Evals” matter in practice:

• Comparing two systems needs a paired analysis, computed on per-item differences, which is cheaper than comparing two independent scores because the systems’ errors are correlated. Even so, his worked example needs about 969 items to detect a 3-point difference at 80% power and a 5% two-sided significance level. It assumes a per-item score-difference variance of 1/9 (scores spread evenly over 0 to 1, with a 0.5 correlation between models); for pass/fail scores that is roughly the case where 11% of items get different results from the two systems. He calls the parameters fictional but reasonable and concludes that new evals should contain at least 1,000 questions.
• Clustered items inflate the error. If several items come from the same document or conversation, they are not independent; in his examples, clustered standard errors were up to about 3 times the naive ones. Compute errors at the cluster level or sample one item per cluster.

A practical path: start small, since early changes have large effects; in its guidance on agent evals, Anthropic’s engineering team calls 20 to 50 tasks drawn from real failures “a great start.” Then grow to 100 to 200 items to develop prompts quickly, a frozen test set of 400 to 1,000 for decisions, and more on the slices where errors cost the most. Measuring whether fine-tuning improved your model works through the power calculation in detail.

Freeze a test set and prevent leakage

Split once, early, and keep three sets: a development set you look at freely, a validation set for choosing between candidates, and a frozen test set used only for final decisions. Version it by content hash and record which version produced every reported number.

Leakage is easy to create by accident:

• Near-duplicates across splits. Deduplicate on normalized text and, where practical, embeddings, so a reworded copy of a test item is not in training or few-shot examples.
• Group leakage. Split by customer, document or conversation, not by row, so the same thread never appears on both sides.
• Time leakage. If the task depends on recent facts or policies, test on data newer than anything used to build the system.
• Prompt leakage. Few-shot examples and retrieved exemplars must come from the training or dev side only.
• Decision leakage. Every time you tune a prompt against the test set, it becomes a validation set. Refresh it when that happens.

Which metrics should you use?

Use the cheapest grader that measures what the owner cares about. Deterministic checks are fast, free and reproducible; save judgment-based grading for qualities code cannot test. Anthropic’s guidance on agent evals recommends “deterministic graders where possible, LLM graders where necessary,” and warns that code graders can be brittle to valid variations, so read failing outputs to confirm the grader, not the model, is right.

Metric type	Good for	Watch out for
Exact match, token F1	Classification, extraction, short answers	Normalize case, whitespace and synonyms first
Schema validity	Any structured output	Valid JSON is not correct JSON; check values too
Code execution	SQL, code, formulas, tool calls	Sandbox it; tests must cover real edge cases
Rubric scoring	Summaries, drafts, explanations	Needs a calibrated judge and anchored examples
Pairwise preference	Open-ended quality, A/B of two systems	Position and length bias; reports relative, not absolute, quality

A few notes on each:

• Exact match and F1 need a normalization step agreed with the owner: is “Billing - Refunds” the same as “billing/refunds”? Decide once and encode it.
• Schema validity is a gate, not a score. Constrained decoding can guarantee structure, but your graders still need to check the values and business rules.
• Execution-based grading checks functional correctness by running the output against tests, the approach popularized by HumanEval. For SQL, run the query against a fixture database and compare result sets rather than query text.
• Rubric scoring works best as several binary criteria rather than one 1-to-10 score. Binary questions (“Does the summary state the refund amount?”) are easier for humans and judges to agree on.
• Pairwise preference is the right tool when there is no single correct answer. OpenAI’s guide recommends pairwise or pass/fail grading for reliability.

A sample rubric

Here is an example rubric for a hypothetical support-reply drafting task. Each criterion is binary, has a definition, and states what fails.

rubric: support_reply_v3
applies_to: drafted replies to billing tickets
scoring: each criterion is pass (1) or fail (0); a reply ships only if all "gate" criteria pass
criteria:
  - id: resolves_request
    gate: true
    question: Does the reply address the customer's actual request?
    fail_if: It answers a different question or only acknowledges the ticket.
  - id: facts_supported
    gate: true
    question: Is every amount, date and policy statement supported by the ticket or the policy excerpt?
    fail_if: Any figure or policy is absent from the provided context.
  - id: correct_next_step
    gate: true
    question: Does it give the next step that the policy prescribes for this case?
    fail_if: It promises an action the policy does not allow, such as a refund outside the window.
  - id: no_internal_leakage
    gate: true
    question: Is the reply free of internal notes, ticket IDs of other customers and system text?
  - id: tone
    gate: false
    question: Is the tone polite and direct, without blame or filler?
anchors:
  - example_id: tkt-0412
    verdict: {resolves_request: 1, facts_supported: 0, correct_next_step: 1, no_internal_leakage: 1, tone: 1}
    note: Quotes a $49 credit; the ticket says $45.

The anchors are real labelled outputs that show graders where the line sits. OpenAI’s guide calls this “show rather than tell.”

How do you use LLM-as-a-judge without fooling yourself?

LLM judges make rubric and pairwise grading affordable at scale. They also have known, measurable biases.

Zheng et al. (2023), the MT-Bench and Chatbot Arena paper, documented position bias (favoring the answer in a particular slot), verbosity bias (favoring longer answers) and self-enhancement bias (favoring answers from the judge’s own model family), along with limited reasoning ability. The same paper found that a strong judge reached over 80% agreement with human preferences, about the level at which humans agree with each other. Both findings matter: judges can work, and they need controls.

The controls:

• Swap positions. Run every pairwise comparison twice with the order reversed and count a win only if it holds both ways; otherwise record a tie. Wang et al. showed that with ChatGPT as judge, Vicuna-13B could beat ChatGPT on 66 of 80 queries just by changing the order; averaging over both orders reduced the bias.
• Control for length. Report output length alongside win rates, and check whether wins correlate with length. Length-controlled AlpacaEval regresses out the length difference, which raised its correlation with Chatbot Arena from 0.94 to 0.98.
• Use a different model family as judge from the candidates where you can, and the most capable judge you can afford.
• Ask for reasoning, then a constrained verdict. Let the judge explain against the rubric, then emit pass/fail or A/B/tie in a fixed format that your code validates. Give it an “unknown” option rather than forcing a guess, and grade each rubric criterion with a separate judge call.
• Blind the judge. Strip model names, version tags and anything that reveals which system produced an output.

Calibrate the judge against human labels

A judge is a measurement instrument; calibrate it before you trust it. Have one or two domain experts label 100 to 200 outputs against the rubric, run the judge on the same items, and report:

• Raw agreement, the share of items where judge and human agree.
• Cohen’s kappa, which corrects agreement for chance: κ = (p_o − p_e) / (1 − p_e), where p_o is observed agreement and p_e is agreement expected by chance from each rater’s label frequencies. Kappa matters when the pass rate is high: a judge that always says “pass” on a 90%-pass dataset has 90% raw agreement and a kappa of zero.
• The direction of errors. A judge that passes bad outputs is more dangerous than one that fails good ones.
• Human-to-human agreement on the same items, if two experts labelled. That is the realistic ceiling.

from sklearn.metrics import cohen_kappa_score, confusion_matrix

# Same 200 outputs, labelled independently by a domain expert and by the LLM judge (1 = pass).
human = [1, 0, 1, 1, 0]   # ...load the full list from your labelling tool
judge = [1, 0, 0, 1, 0]

raw_agreement = sum(h == j for h, j in zip(human, judge)) / len(human)
kappa = cohen_kappa_score(human, judge)
tn, fp, fn, tp = confusion_matrix(human, judge, labels=[0, 1]).ravel()
print(f"agreement={raw_agreement:.2f} kappa={kappa:.2f}")
print(f"judge passes outputs humans failed: {fp}; judge fails outputs humans passed: {fn}")

For ordinal scales, cohen_kappa_score accepts weights="linear" or "quadratic" so that near-misses count less than large disagreements. Recalibrate whenever you change the judge model, the judge prompt or the rubric.

Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya ShankarHamel Husain and Shreya Shankar · Lenny's Podcast · YouTube · 2025

How should human evaluation be run?

Human labels are the ground truth everything else is calibrated against, so the protocol matters:

• Use people who know the task. A support lead grading support replies beats a general crowd.
• Write labelling guidelines with the rubric, definitions, anchored examples and an explicit “unsure” option.
• Blind and randomize. Hide which system produced an output and shuffle the order, exactly as for LLM judges.
• Double-label a subset (for example, one item in five) to measure inter-annotator agreement, and adjudicate disagreements. Disagreements often reveal an ambiguous criterion.
• Track label drift. Re-insert a few previously labelled items to check that graders stay consistent over time.

Human review is slow and expensive, so use it where it adds the most: calibrating judges, labelling the frozen test set, and spot-checking production.

How do you check hallucinations and faithfulness?

For tasks grounded in documents (RAG answers, summaries, extraction), check faithfulness claim by claim instead of grading the whole answer as true or false. FActScore splits a generation into atomic facts and reports the fraction supported by a trusted source; its automated estimator, which uses retrieval and a strong language model, had an error rate under 2% against the human-annotated scores. The Ragas faithfulness metric applies the same idea to RAG: supported claims divided by total claims, checked against the retrieved context.

Practical additions:

• Separate retrieval failures from generation failures. If the right passage was never retrieved, a faithful model cannot answer. Log retrieved context with every eval item so you can tell which step failed.
• Check numbers and names deterministically where you can: every figure in the output must appear in the source.
• Include unanswerable questions and score whether the system declines correctly.
• Validate the claim checker against human labels, just like any other judge.

If most failures are retrieval failures, the fix is in the retrieval layer, not the model. The comparison of fine-tuning, RAG and prompt engineering helps decide which layer to change.

What does a working eval harness look like?

A harness does four things: load versioned cases, run the system under test, grade each output, and report per-slice scores with error bars, keeping every raw output for error analysis. The sketch below assumes a JSON-output classification task and takes the model and judge as plain functions, so it works with any API or local model.

"""Minimal eval harness: run every case, grade it, report per-slice scores with error bars."""
import json
import math
from collections import defaultdict
from dataclasses import dataclass
from typing import Callable


@dataclass
class Case:
    id: str
    slice: str          # stratum, e.g. "billing", "refund", "edge:empty_input"
    input: str
    expected: dict      # gold fields for deterministic checks; may be empty


def load_cases(path: str) -> list[Case]:
    with open(path, encoding="utf-8") as f:
        return [Case(**json.loads(line)) for line in f if line.strip()]


REQUIRED_FIELDS = {"category": str, "priority": str, "summary": str}
PRIORITIES = {"low", "medium", "high"}


def grade_schema(output: str) -> tuple[bool, dict | None]:
    """Pass only if the output parses as JSON with the right fields and types."""
    try:
        obj = json.loads(output)
    except json.JSONDecodeError:
        return False, None
    if not isinstance(obj, dict):
        return False, None
    for field, typ in REQUIRED_FIELDS.items():
        if not isinstance(obj.get(field), typ):
            return False, None
    if obj["priority"] not in PRIORITIES:
        return False, None
    return True, obj


def grade_case(case: Case, output: str, judge: Callable[[Case, str], int]) -> dict:
    valid, obj = grade_schema(output)
    result = {"id": case.id, "slice": case.slice, "schema_valid": valid}
    if not valid:
        # An invalid output fails every downstream check; never score it as missing.
        result.update(category_match=False, judge_pass=False)
        return result
    result["category_match"] = obj["category"] == case.expected.get("category")
    result["judge_pass"] = judge(case, obj["summary"]) == 1   # rubric judge returns 1 or 0
    return result


def mean_and_ci(values: list[bool]) -> tuple[float, float]:
    """Mean and 95% half-width using the normal approximation (fine for n >= ~30)."""
    n = len(values)
    p = sum(values) / n
    return p, 1.96 * math.sqrt(p * (1 - p) / n)


def report(results: list[dict], metrics=("schema_valid", "category_match", "judge_pass")) -> None:
    by_slice = defaultdict(list)
    for r in results:
        by_slice[r["slice"]].append(r)
    by_slice["ALL"] = results
    for name, rows in sorted(by_slice.items()):
        cells = []
        for m in metrics:
            p, hw = mean_and_ci([r[m] for r in rows])
            cells.append(f"{m}={p:.1%} +/- {hw:.1%}")
        print(f"{name:<24} n={len(rows):<5} " + "  ".join(cells))


def run(cases_path: str, model: Callable[[str], str], judge: Callable[[Case, str], int],
        out_path: str) -> list[dict]:
    cases = load_cases(cases_path)
    results = []
    with open(out_path, "w", encoding="utf-8") as out:
        for case in cases:
            output = model(case.input)
            row = grade_case(case, output, judge)
            row["output"] = output                     # keep raw outputs for error analysis
            out.write(json.dumps(row) + "\n")
            results.append(row)
    report(results)
    return results

Design choices worth copying: invalid outputs count as failures rather than being dropped (dropping them inflates every other metric), each row keeps the raw output, and the report shows the interval next to every score. For small slices or pass rates near 0% or 100%, swap the normal approximation for a Wilson or bootstrap interval, and for item clusters use cluster-level errors as described above.

How do you turn evals into regression tests and CI?

Once a harness exists, run it on every change that can move quality: prompt edits, model or model-version changes, retrieval index rebuilds, decoding settings, and new training data.

• Keep the eval portable. Store cases, rubrics, judge prompts and results in your own repository in plain formats. Hosted eval tooling can disappear: OpenAI’s Evals dashboard, API and graders are already read-only and are scheduled to shut down on November 30, 2026.
• Pin everything. Record model version, prompt hash, decoding parameters, judge version and eval-set version with every run, so two numbers are comparable.
• Gate on what the owner agreed. Fail the build if a gate metric drops below its bar or if any important slice regresses by more than its margin of error.
• Keep a fast suite and a full suite. A few hundred deterministic checks on every pull request; the full frozen test set with judges before release.
• Add a test for every production bug. Each escaped failure becomes a labelled case in the regression slice, so it cannot silently return.
• Watch nondeterminism. Run a sample more than once at your production temperature to see how much scores move between identical runs; Miller’s paper advises averaging several samples per question rather than lowering the temperature, which changes the model being measured and can bias the estimate instead of removing the noise.

How do you evaluate in production?

An offline eval measures the inputs you thought of. Production shows you the rest.

• Sample and grade live traffic with the same calibrated graders, on a fixed daily or weekly sample, and compare against the offline score.
• Collect implicit signals: how often users edit, regenerate, escalate, or abandon. These are noisy, but trends are informative.
• Monitor input drift: new intents, languages, document types or length distributions that the eval set does not cover.
• Track the operational side per request: latency, token usage, cost and error rates, next to quality.
• Close the loop. Route surprising or low-scoring outputs to human review, label them, and fold them into the dev and regression sets. This is how the eval stays honest as the product changes; the custom AI workflows guide shows how to wire that loop into a multi-step pipeline.

Evaluation is also the deciding step when you consider fine-tuning. Attempts builds this kind of task-specific eval first and measures any tuned model against the client’s current prompt and model, which sometimes shows that better prompting or retrieval is enough and training is not needed.

What should you build first in an LLM evaluation framework?

If you are starting from nothing, this order gets useful signal quickly:

1. Write the success definition and shipping bar with the business owner.
2. Pull 200 real inputs, stratified by your main slices, plus 30 edge cases.
3. Have the owner label 50 outputs from the current system and refine the rubric.
4. Build deterministic graders for everything checkable.
5. Add an LLM judge for the rest and calibrate it against human labels.
6. Score the current system as the baseline, with intervals.
7. Freeze a test set, put the fast suite in CI, and start sampling production.

From there, every proposed change, including a new model or a fine-tuned one, is a question your framework can answer with evidence.