Rubric

Background

A rubric is a structured grading scheme that defines, in advance, what a good output looks like and how to score it, so that evaluation is consistent across different outputs and graders. Instead of an unstated overall impression, a rubric breaks quality into explicit criteria, for example correctness, completeness, adherence to format, tone, and safety, and assigns each a scale or a set of pass/fail checks, often with a short description of what earns each level. That structure lets separate graders reach similar scores on the same output and makes the evaluation auditable: you can see why something scored the way it did. Rubrics are used both by human annotators and, increasingly, in the LLM-as-judge pattern, where a language model is given the output plus the rubric and asked to score it against each criterion. When building AI systems, rubrics are central to evaluation because model outputs are open-ended and there is rarely a single correct string to match against. A well-designed rubric turns a vague goal like "is this answer good" into concrete, gradeable questions, which makes automated evals reliable enough to catch regressions, compare prompts or models, and gate deployments. The main risks are ambiguous criteria that graders interpret differently and, with LLM judges, the model's own biases toward length, style, or position; both are mitigated by tight, example-anchored criteria and by validating judge scores against human labels on a sample.