[Topic]
Evaluation & quality
Test sets, rubrics, LLM judges and the statistics that tell you whether a change actually helped.
GuideIntermediate
How to build a custom evaluation framework for LLMs
A custom LLM evaluation framework is a versioned set of real task examples, graders that encode what the business owner calls correct, and statistics that tell you whether a difference is real. Public benchmarks cannot tell you how a model will do on your task; a few hundred well-chosen examples with calibrated graders usually can.
GuideAdvanced
How to measure whether fine-tuning actually improved your model
Fine-tuning improved your model only if it beats the strongest cheap alternative, such as a better prompt or retrieval, on a held-out test set never used for training or tuning, by a margin whose paired confidence interval excludes zero, without regressions on safety and general skills, at an acceptable cost and latency.