Your LLM Judge Prefers the Longer Answer

You swapped your handful of brittle string assertions for an LLM grader a few weeks ago, and it felt like an upgrade. The judge reads the answer, grades it against a rubric, hands back a score and a reason. Every release since has come in green, the dashboard is a wall of eights and nines, and nobody has had to argue about whether an answer was good. It is working.

Then a user forwards you a reply from your product that is confidently wrong, and it is wordy, well-structured, and reads beautifully. You pull it up in your eval set out of curiosity. The judge gave it a nine. Not because it was correct, but because it was long and well-formatted and sounded like the kind of thing a capable model would write. Your grader was never scoring quality. It was scoring length, order, and style, and it had been quietly doing that the whole time.

Why a biased judge is expensive, not just annoying

The LLM judge has become the load-bearing measurement in a lot of AI systems. It gates releases in your eval suite, it picks winners in A/B prompt comparisons, it scores agent self-critique loops, and increasingly it decides which of two model outputs becomes preference data you train on. Every one of those is a decision made on the judge's number. If the number is biased, the bias does not stay contained. It propagates into every choice downstream.

The failure is quiet, which is what makes it costly. A biased judge does not throw errors or return nulls. It returns a plausible score with a plausible reason, and the score is systematically off in a direction you cannot see from the dashboard. So you ship a regression because the wordier new prompt scored higher than the tighter old one that users actually preferred. Or you kill a genuinely better change because it was terse and the judge reads terse as low effort. You are not flying blind, which would at least make you cautious. You are flying on an instrument that reads ten percent high and never told you.

Three biases do most of the damage, and all three are measurable and fixable. Get these under control before you trust a single number the judge produces.

Position bias: the judge likes whoever went first

When you ask a judge to compare two answers and pick the better one, it has a standing preference for whichever answer you showed it first. This is not subtle. Depending on the model and the task, the first position can pick up a double-digit advantage on identical-quality answers. If your harness always puts the new candidate second and the baseline first, you have baked a handicap into every comparison and you will read it as the new prompt being worse.

The fix is boring and it works: run every comparison both ways and only trust the result when it agrees with itself.

def judge_pairwise(question: str, answer_a: str, answer_b: str) -> str:
    """Ask the judge which answer is better. Returns 'A', 'B', or 'tie'."""
    prompt = (
        f"Question:\n{question}\n\n"
        f"Answer 1:\n{answer_a}\n\nAnswer 2:\n{answer_b}\n\n"
        "Which answer is better? Reply with exactly '1', '2', or 'tie'."
    )
    reply = call_judge(prompt).strip()
    return {"1": "A", "2": "B"}.get(reply, "tie")


def compare(question: str, baseline: str, candidate: str) -> str:
    # Run both orderings so first-position advantage cancels out.
    first = judge_pairwise(question, baseline, candidate)   # baseline shown first
    second = judge_pairwise(question, candidate, baseline)  # candidate shown first

    # Only a verdict that survives the swap counts. If the judge flips its
    # answer when you flip the order, it was voting on position, not quality.
    if first == "A" and second == "B":
        return "baseline"
    if first == "B" and second == "A":
        return "candidate"
    return "tie"  # disagreement across orderings is a real tie, record it as one

The important line is the last one. When the judge picks a different winner depending on order, that is not a weak signal you should break with a tiebreaker. It is the judge telling you it cannot actually distinguish these two answers, and the honest thing to record is a tie. Teams that pick a side there are manufacturing a signal out of position noise. The cost is that you now make two judge calls per comparison instead of one, which is real, and we will come back to whether it is worth it.

Verbosity bias: longer reads as better

Give a judge a correct three-sentence answer and the same correct answer padded out to three paragraphs, and it will usually score the long one higher. The measured effect is large, often fifteen to thirty points of inflated preference for the verbose version, and it shows up across every major judge model. It is easy to see why. Length correlates with thoroughness in the training data, so the judge learned length as a proxy for effort, and it applies that proxy even when the extra words add nothing or actively bury the answer.

You cannot prompt this away completely, but you can push back on it from two directions. First, tell the judge explicitly not to reward length, in the rubric, in plain language. Second, and this matters more, stop reading the raw score as truth and normalize for length when you aggregate.

import re


def word_count(text: str) -> int:
    return len(re.findall(r"\w+", text))


JUDGE_RUBRIC = """Score the answer from 1 to 10 on correctness and clarity.
Do not reward length. A short, correct, complete answer must score higher
than a long answer that repeats itself or adds irrelevant detail.
Penalize padding. Reply as JSON: {"score": <int>, "reason": "<one sentence>"}."""


def judge_pointwise(question: str, answer: str) -> dict:
    result = call_judge_json(f"{JUDGE_RUBRIC}\n\nQ: {question}\nA: {answer}")
    result["length"] = word_count(answer)
    return result


def flag_length_confound(scored: list[dict]) -> float:
    """If score tracks length across the whole set, the judge is paying you
    in verbosity. Return the correlation so you can watch it over time."""
    scores = [s["score"] for s in scored]
    lengths = [s["length"] for s in scored]
    n = len(scores)
    mean_s, mean_l = sum(scores) / n, sum(lengths) / n
    cov = sum((s - mean_s) * (l - mean_l) for s, l in zip(scores, lengths))
    var_s = sum((s - mean_s) ** 2 for s in scores) ** 0.5
    var_l = sum((l - mean_l) ** 2 for l in lengths) ** 0.5
    return cov / (var_s * var_l) if var_s and var_l else 0.0

That correlation number is the point. You are not going to fully debias the judge, so instead you make the bias visible. If flag_length_confound comes back at 0.7 across your eval set, your scores and your answer lengths are moving together and you should not trust the ranking until you understand why. A judge that scores quality should show close to zero correlation between length and score on a set where the short and long answers are equally good.

Self-preference bias: the judge likes its own voice

A judge rates outputs that match its own style, phrasing, and formatting habits higher than equally good outputs from a different model family. The uniform boost is real, in the range of ten to twenty-five percent, and it creates a specific trap. If you generate answers with a model and grade them with the same model, that bias points straight at your own system and pads every score you report. You benchmark your model against a competitor's, judge with your own model, and declare victory that the numbers do not support.

The mitigation is structural, not a prompt. Use a judge from a different family than the system you are evaluating. If you are grading a GPT-family generator, judge with Claude or Gemini, and the other way around. When you genuinely cannot, because of cost or data residency, then you have to measure the self-preference gap against human labels and discount for it, which brings us to the step that ties the whole thing together.

Calibration: does an 8 from the judge mean an 8 to a human

Everything above reduces bias. None of it tells you whether the judge's numbers actually track reality, and that is a separate question you answer with a calibration set: a few hundred examples you have graded by hand, spanning your easy cases and your known failure modes. You run the judge over the same set and measure agreement. Not correlation, agreement, because you care whether the judge and the human land on the same verdict, not whether they move in the same direction.

def cohens_kappa(human: list[int], judge: list[int]) -> float:
    """Agreement between two raters, corrected for chance. 1.0 is perfect,
    0.0 is no better than guessing. Below ~0.6 the judge is not trustworthy."""
    labels = sorted(set(human) | set(judge))
    n = len(human)
    observed = sum(h == j for h, j in zip(human, judge)) / n

    expected = 0.0
    for label in labels:
        p_h = sum(h == label for h in human) / n
        p_j = sum(j == label for j in judge) / n
        expected += p_h * p_j  # chance both pick this label independently

    return (observed - expected) / (1 - expected) if expected != 1 else 1.0


def calibrate(cases: list[dict]) -> dict:
    # Bucket the 1-10 score into pass/fail at your gate so you measure the
    # decision you actually make, not the raw number you never act on directly.
    human = [1 if c["human_score"] >= 7 else 0 for c in cases]
    judge = [1 if judge_pointwise(c["q"], c["a"])["score"] >= 7 else 0 for c in cases]

    kappa = cohens_kappa(human, judge)
    # Where the judge and human split is where you send work to a human reviewer.
    disagreements = [c for c, h, j in zip(cases, human, judge) if h != j]
    return {"kappa": kappa, "trustworthy": kappa >= 0.6, "review": disagreements}

Now the judge's score means something, because you have measured it against the thing it is supposed to predict. A kappa above about 0.6 says the judge agrees with your humans well enough to gate on. Below that, the judge is not ready to be the sole grader for that task, and the disagreement cases are exactly the queue you route to a human. This is the same hybrid pattern that shows up everywhere in production evals: cheap deterministic checks first, the judge for what needs reasoning, and a human for the small slice where the judge and the ground truth disagree.

Tradeoffs and the places it bites

The debiasing costs tokens. Position-swapping doubles your pairwise judge calls, and a calibration set is human hours you have to spend and re-spend. That is the real price, and it is worth it precisely where the decision is expensive: release gates, preference data you will train on, model bake-offs you will publish. For a cheap internal pass-or-fail smoke check, a single judge call with a tight rubric is fine. Match the rigor to what rides on the number, and pair the judge with cheap structured-output validation so a malformed judge response fails loudly instead of parsing into a silent zero.

Calibration goes stale. A kappa you measured last quarter describes last quarter's judge, rubric, and traffic. Change the judge model, and a provider updating it under you counts, and your calibration is describing a model that no longer exists. Re-label a fresh sample on every judge or rubric change and treat the kappa like a test you can regress, not a certificate you earn once.

Bias mitigations interact. Fixing position bias with a swap does nothing for verbosity, and a different-family judge to dodge self-preference may carry its own length bias. There is no single knob. You measure each bias on your own data, because the magnitudes differ by model and task, and a mitigation that matters on one task is noise on another.

An absolute score is the weakest signal you have. A single 1-to-10 number on one answer in isolation is where all three biases land hardest and where drift is invisible. Prefer relative judgments against a fixed reference, watch the distribution of scores rather than any one score, and be suspicious of any metric that only ever goes up.

The takeaway

An LLM judge is a measurement instrument, and you would not gate a release on a thermometer you had never checked against boiling water. Position, verbosity, and self-preference are the three ways the instrument reads wrong, and each has a concrete fix: swap and average to cancel order, normalize and watch the length correlation to catch verbosity, judge across model families to defuse self-preference. Then calibrate the whole thing against a few hundred human labels so you know, in a number, how far the judge is from the truth on your own data. Do that and the wall of eights and nines starts meaning something. Skip it and you are shipping on an instrument that reads high and never told you which releases it lied about.

If your eval pipeline gates on an LLM judge you have never calibrated against human labels, you do not currently know whether it is protecting you or quietly waving regressions through. Book a consultation call and we can pressure-test your judge on your own data and find out which of your green releases were actually green.