Writing Scoring Rubrics That Domain Experts Actually Agree On

I've spent the better part of two years watching teams build LLM judges, and the moment I know a rubric is going to fail comes when I hand the same transcript to two people who both know the domain cold and watch them come back with different scores. That's the failure mode nobody budgets time for, and it's the one that determines whether anything downstream is worth trusting. ## Decomposition before judgment Here's what happens on a real project. A customer support team wants to grade refund-handling transcripts. Someone writes "Was the agent helpful?" as a criterion, everyone nods in the review meeting, and three weeks later you find out that one grader thinks "helpful" means the refund got approved, another thinks it means the agent explained the policy clearly regardless of outcome, and a third thinks it means the tone was warm. Nobody was wrong. The criterion just never meant one thing. The fix, and I mean this as the starting point rather than a nice-to-have, is to break the task apart before anyone looks at a transcript. A refund request bundles at least four separable skills: retrieving the right policy, applying it correctly, communicating the outcome without over-promising, and closing the loop instead of leaving the user hanging. Score all four at once and you get a single number that means five different things depending on who's holding the pen. Anthropic and OpenAI's public writing on eval design both land here, and honestly it's not a new idea; educational assessment researchers have known for decades that analytic rubrics, scored dimension by dimension, produce more consistent grades than holistic ones. The trick is writing each dimension so it resolves to yes, no, or a short ordinal scale. "Did the agent cite the correct refund policy clause" is checkable in about four seconds. "Was the agent helpful" invites a debate. This bites harder on agent transcripts than on plain text generation, because agents take actions that compound across turns. "Did the agent use the search tool before answering" is trivial: you look at the trace, you see the tool call or you don't. "Was the final answer good" mashes tool use, reasoning quality, and tone into a single judgment call, and that kind of ambiguity is exactly what tanks agreement between graders. ## Measuring whether experts actually agree You cannot eyeball this part. Once the rubric exists, the question is whether independent graders converge on it, and the tool for that is Cohen's kappa (or Fleiss' kappa once you've got more than two raters). Kappa corrects raw percent agreement for the agreement you'd get by chance alone, which matters more than people expect. Two graders agreeing 80% of the time sounds fine until you notice the criterion is a "yes" 90% of the time anyway, in which case chance alone gets you most of the way there and your 80% is barely signal. Landis and Koch's 1977 benchmarks, still the reference point most teams cite, treat kappa above 0.60 as substantial agreement and reserve "almost perfect" for anything above 0.80. Below 0.6 on any single criterion, rewrite the criterion. Hiring better graders rarely fixes the actual problem. If two subject-matter experts read the same instruction and land on different verdicts, the instruction was ambiguous. Full stop. And here's a mistake I've watched more than one team make: write the rubric, run it past a single expert, get a thumbs-up, ship it. One rater tells you nothing, because agreement is a property of two or more independent judgments landing on the same answer. You need at least two graders per item, three if you can manage it, scoring independently, no conferring beforehand. Let them talk first and you've just measured whether they can agree once they've negotiated a shared interpretation, which is a different and much less useful thing to know. ## Where disagreement actually lives Disagreement isn't randomly distributed across a rubric. It clusters, and once you've calibrated a handful of these you start to see the pattern before you even run the numbers. Tone, appropriateness, "good enough" reasoning: these produce low kappa almost every time. "Did the agent apologize sincerely" invites five different intuitions about sincerity. "Did the agent's response include an apology" is a fact you can point to in the text. Numeric and factual checks sit at the other end, and they're almost boringly easy to get high agreement on. Did the calculation match the expected output. Did the agent hit the right API endpoint. Write these first, since they anchor the rubric in something verifiable, and you want that baseline of trustworthy signal in hand before you go anywhere near the subjective dimensions. Then there's a category that fools people because it looks objective and isn't: "did the agent follow instructions." Whose instructions? Interpreted how strictly? Forgiving what kind of deviation? I'd put more rubric-writing time into this category than any other, because it's the one most likely to read clean on the page and then fall apart the moment two graders check it independently. ## Calibrating the judge, not just trusting it Say your rubric clears the bar, kappa above 0.6 across the board, maybe better on the factual criteria. You still don't know whether an LLM judge applies it the way your experts do, and that's a separate question from whether the rubric itself is sound. Conflating the two is common, and it's the reason teams end up surprised months later. Calibration means treating the model the same way you'd treat a new hire grading transcripts for the first time: give it a held-out set already scored by your experts, run it against the rubric, and measure agreement against the expert labels with kappa, not raw accuracy. Zheng et al.'s 2023 paper on LLM-as-judge, the one behind MT-Bench and Chatbot Arena, found GPT-4 agreeing with human preferences at rates comparable to human-human agreement, but that was on general conversational tasks. It doesn't transfer automatically to a rubric built for clinical intake or financial compliance. A judge validated on chat preferences tells you nothing about whether it can correctly assess whether an agent cited the right subsection of a tax code, and I've seen teams assume otherwise, to their cost. When the judge and the experts diverge, that's where the real diagnostic work starts. Sometimes it's a genuinely hard criterion and the fix is more few-shot examples in the judge prompt. Sometimes the divergence reveals something worse: the human experts only agreed with each other by coincidence, not shared understanding, and the rubric itself needs to go back to the decomposition stage. You can't tell these two apart without kappa numbers for both comparisons, human-human and human-judge, sitting side by side. ## The iteration loop, not a one-time build Rubric-writing doesn't end at launch. Model updates change how agents reason and use tools, and a rubric calibrated against one model version can quietly stop matching expert judgment once that happens. I've seen teams treat calibration as a gate they clear once before shipping, only to discover months later that the automated judge has drifted somewhere the human experts never would have gone. None of this is complicated in principle. It's just slower than people want it to be, and it asks you to distrust your own rubric right up until the numbers say otherwise.