1. The score has an author, even when that author is software
An evaluation report announces that one assistant is better than another. The chart is clean, the gap is visible and the percentage looks precise. Before arguing about the winning model, ask who produced the labels from which that percentage was calculated. Were they human preferences, exact-answer checks, grades assigned by another language model, or a mixture? The answer determines what kind of evidence the report contains.
Using a language model to judge text does not make the resulting judgment worthless. It does make the evaluator part of the measurement. A score can depend on the task, the supplied criteria, the presented answers and the rule used to turn the evaluator's response into a recorded result. Those components deserve their own description. Otherwise, the apparent performance of the assistant can become entangled with undocumented choices in the grading process.
The 2023 MT-Bench and Chatbot Arena paper examined language models as judges, including position and verbosity biases and limitations in grading reasoning. Its evidence concerned the models, prompts and tasks studied then. It is not a current warranty that a model bearing a familiar family name will reproduce those results today. Zheng and colleagues, MT-Bench and Chatbot Arena
This article examines the records behind a judgment rather than recommending a current model. No language-model API was called to generate the examples. All Cedar labels, scores and responses below are author-created instructional fixtures. A small local script checks their aggregation and arithmetic; it does not run an LLM judge. A fixture that says one answer won is a stipulated input to the example, not an observation about any actual product.
That boundary allows the examples to make a useful distinction. We can prove that an aggregation script counted a particular file correctly without proving that the file's labels are good judgments. We can demonstrate that a weighted score changes when its weights change without showing that either weighting scheme measures the desired quality. We can find a disagreement between two supplied label lists without knowing which list is right.
The audit begins by naming the claimed result. Is it a preference between two particular responses, an absolute score against a rubric, an agreement rate with a reference panel, or a ranking averaged across tasks? These are not interchangeable descriptions of one number. A report that moves between them without explanation may remain mathematically tidy while becoming evidentially confused. The remedy is not a larger leaderboard. It is a clearer account of the path from the task to the recorded judgment and from that judgment to the public claim.
2. Preserve the answer's identity, not just its display letter
A basic order check compares judgments after reversing the presentation of the same two responses. Suppose Cedar response R is displayed as A and response S as B, then those display letters are exchanged. Choosing A in both presentations selects different responses; choosing A first and B second selects R both times. Agreement on the display letter is therefore not agreement on answer identity. This illustrates the order problem examined in the MT-Bench work, not a new result from a model test. MT-Bench order-bias analysis
For a publication audit, the important artifact is the mapping that connects a displayed slot to the underlying response. Imagine a fictional report containing two input files and a short results file. The first input file calls the responses A and B. The second calls them Left and Right. The results file contains only A. Without the applicable input identifier and interpretation rule, an outsider cannot tell what that label denotes. A confident summary does not repair the missing relationship.
The same problem can arise without any model error. A spreadsheet might join a judgment to the wrong input version. A row might be duplicated when two exports are combined. A parser might retain a display label but discard the record that gives the label meaning. These are hypothetical publishing failures, not allegations about a benchmark. They show why an evaluation packet needs enough information to reconstruct the reported result at the level it claims.
Consider an original Cedar reconciliation log with twenty scheduled judgment records. Eighteen have a unique matching input record. One has no matching input identifier. One identifier appears twice with conflicting attachments. The first number to report is not twenty completed comparisons. It is eighteen unambiguous input-result links and two unresolved records. That description identifies what the available files support without deciding whether the judgments themselves are sensible.
If the missing attachment is later found, nineteen links may become unambiguous. The conflict still requires its own explanation. Marking both cases as fixed merely because the file count now equals twenty would conceal the remaining problem. This is a records test: do the reported outputs refer to the stated inputs? It is distinct from evaluating the quality of the answers or diagnosing a model's behavior.
A reader can therefore ask for a compact demonstration using harmless examples before trusting a large result. The demonstration should show how a stored output is connected to the exact candidate text and evaluation request. It need not expose private user content; a completely artificial fixture is sufficient to test the bookkeeping. What matters is that the explanation resolves the identity question rather than assuming the letter on the screen is a permanent name.
3. A retained verdict is not the same as a completed evaluation
A GroundLM workshop paper dated October 29, 2026 reports a bounded study of position-stable judgments. Its main set contained 120 response pairs: 62 directional results, 14 order-swap failures and 44 invalid pairs, with no double ties. The authors explicitly do not establish human-preference validity or downstream training gains. Those limits are part of the finding, not a footnote to discard when citing it. Liu, Lin and Hao, position-stable agreement
For an original Cedar fixture, stipulate twenty response pairs with labels already interpreted at the response level. Ten consistently favor R, four consistently favor S, two record a tie in both presentations, three change the selected response between presentations and one lacks a valid recorded verdict. Keeping these categories distinct prevents an omitted result from silently becoming a tie. They are assigned examples, not measured outputs.
The fourteen consistent directional pairs are 70 percent of the twenty scheduled pairs. R accounts for ten of those fourteen, about 71.43 percent. R also accounts for ten of all twenty scheduled pairs, or 50 percent. These are different denominators. A headline saying R won 71.43 percent should specify that it refers to the retained directional subset, not to all scheduled pairs or to a human-verified accuracy rate.
Now imagine a second fictional report with the same ten R selections but only ten retained directional pairs out of twenty. Its conditional R share is 100 percent while its directional coverage is 50 percent. The higher conditional percentage does not, by itself, demonstrate that the evaluator became more informative. The report discarded more of the original comparison space. A reader needs both the numerator and the coverage to understand what changed.
This arithmetic does not make abstention undesirable. There may be good reasons not to force an unsupported decision. The issue is how the decision is reported. A clearly identified unresolved case provides different evidence from a completed preference judgment. Neither should be silently converted into the other simply because a chart needs every row to contain a number.
The fictional packet should also preserve a count of attempted work. Twenty pairs presented twice correspond to forty scheduled presentations, before any repeats. That is not forty independent tasks or forty independent people expressing preferences. It is two observations planned for each of twenty underlying pairs. A report that alternates among presentations, pairs and retained results can create apparent improvements merely by changing the unit counted.
The useful conclusion is bounded: under Cedar's stipulated categories, ten R directions survive among fourteen directional pairs, while six pairs do not produce such a direction. Whether those ten judgments are substantively correct remains a separate question. No amount of careful denominator management turns an invented or unvalidated preference label into ground truth.
4. Repetition and changed order ask different questions
A December 2025 IJCNLP-AACL study distinguishes repetition stability, position consistency and preference fairness in its analysis of LLM judges. Its reported patterns vary across judges and tasks. The existence of these separate measurements is a useful warning against describing every kind of repeatability with a single word, reliability. Shi and colleagues, systematic position-bias study
Consider a tiny Cedar fixture with two different checks. For a fixed presentation, five assigned labels are R, R, R, R and R. For two opposite presentations of a separate pair, the assigned response identities are R and S. The first fixture repeats perfectly within its fixed setup. The second changes when the presentation changes. There is no mathematical contradiction: the checks vary different things.
Neither fixture establishes a cause. In a real experiment, a changed answer could coincide with other changes in the request or service. The report would need to show what was actually held constant. For Cedar, the exercise simply stipulates the labels and asks whether the conclusion matches the stated comparison. It does not attribute the pattern to a real model architecture or provider.
A more detailed original bookkeeping example schedules four task families, three items per family and two presentation orders per item. That creates twelve items and twenty-four scheduled presentation records. If each presentation is repeated five times, the ledger contains 120 records. The number of distinct items remains twelve. Reporting 120 items tested would exaggerate the diversity of the exercise by a factor of ten.
In a separate alternative ledger of 120 records, suppose ninety-six concern the first two families because those cases were rerun more often. A simple average across records would give those families 80 percent of the weight. An equal-family summary would give them 50 percent. Neither number is inherently the correct policy, but the two summaries answer different questions. The report must identify which one it uses and why repeated records are present.
This is especially important when a rerun occurs after an undesirable result. A fictional log might contain one original output and four later repeats for one item but only one output for another. Combining all six as though they were equally planned independent items hides the decision to repeat. The appropriate response is to describe the actual record structure, not to declare every repeat invalid or every first attempt authoritative.
A useful evaluation ledger can therefore distinguish the scheduled item, its presentation, the repetition identifier and the reason for any additional run. Those distinctions make the reported arithmetic inspectable. They do not prove that a judge is stable, fair or accurate outside the examined setting. The result remains evidence about a particular set of observations, with the structure of those observations visible.
5. The rubric defines the question being graded
G-Eval, published at EMNLP in December 2023, places a task description and evaluation criteria into a structured language-model evaluation process. Its experiments concern summarization and dialogue generation. The paper supports treating the criterion as part of the evaluation design; it does not establish that any rubric attached to any present-day model produces a valid score. G-Eval primary paper
For a harmless original Cedar task, an imaginary library notice says that the reading room closes at six and the return box remains available. The assignment is to summarize both facts in one sentence without adding a reason for the closure. Candidate R says that the room closes at six but returns remain possible through the box. Candidate S says the library closes early for maintenance and that books should be returned tomorrow. These sentences are authored examples, not generated responses.
A criterion focused only on fluency could miss the central problem: S adds an unsupported explanation and changes the return information. A criterion focused on reproducing the two supplied facts asks a different question. This does not require a complex scoring system. The task already supplies the evidence needed to identify which facts are present and which claims were added.
Suppose a fictional worksheet assigns separate marks for including the closing time, including the return-box availability and avoiding unsupported additions. Those marks should be tied to the specific text. A reviewer can point to the phrase that supports a mark or explain why no such phrase exists. A statement that an answer feels professional does not establish any of those three properties.
The exercise also separates the rule from its application. Two graders could receive the same rule yet disagree about whether a paraphrase preserves the meaning. Alternatively, two graders could apply different rules and reach different scores without either misapplying its own rule. An evaluation report should not combine those situations into one unexplained disagreement rate. One concerns application; the other concerns the target being assessed.
A change in the rubric needs a visible version boundary. If a later worksheet begins rewarding brevity, earlier grades do not automatically answer the new question. The underlying responses may be unchanged while their scores change because the evaluation's priorities changed. That can be a legitimate redesign, but it should not be presented as evidence that the response generator improved.
For a public reader, the essential evidence is the connection between the criterion, the supplied source and the assigned mark. A long rationale can sound persuasive while failing to make that connection. A short, specific explanation may be more useful. Cedar's notice allows anyone to inspect the relationship directly, without pretending that a hand-authored exercise is an experiment on a live judging model.
6. A single score can conceal the choice of priorities
The ICLR 2024 FLASK paper proposes instruction-specific skill-level assessment rather than only a coarse overall preference. This article does not reproduce its skill set or scoring protocol. The relevant distinction is that an overall judgment and a profile of component judgments provide different information. FLASK primary conference abstract
Take a new fictional Cedar comparison with two author-assigned component scores on a five-point scale. R receives five for evidence fidelity and two for readability. S receives three for fidelity and five for readability. These values are invented to make a trade-off visible; no human panel or model supplied them. With equal weights, R's average is 3.5 and S's is 4.0.
If the stated rule gives fidelity twice the weight of readability, R receives twelve divided by three, or 4.0. S receives eleven divided by three, about 3.67. The ranking reverses even though every component score is unchanged. This is not a contradiction in the arithmetic. The scoring rule changed the relative importance of the two components.
A report should therefore avoid saying that the second calculation corrected the first unless there is an identified reason that the original weights were wrong for the intended purpose. It might instead say that the second calculation reflects a different priority. That is a substantive choice, and readers may reasonably disagree with it. Hiding the choice inside a single score deprives them of the information needed to understand the disagreement.
There is a further distinction between a weighted average and a minimum requirement. Suppose a fictional publication policy requires a fidelity score of at least four before readability is considered. Under the stipulated scores, R satisfies that gate and S does not. Averaging allows strengths in one component to compensate for weaknesses in another; a gate does not. Neither operation should be substituted silently for the other.
This example does not endorse the particular scales, weights or threshold. It shows that they form part of the decision being made. The underlying observations need to remain visible if the report invites readers to evaluate the decision. Publishing only the winning name would conceal whether the outcome came from the component grades, the weighting rule or the threshold.
The distinction is practical for anyone reviewing a procurement demonstration or a research result. Ask what changed between two reported scores. Was it the answer, the judge's assessment, the definition of a component or the rule combining components? Each has a different evidentiary meaning. A numerical improvement can be real within a newly chosen rule while saying nothing about improvement under the old one.
7. Agreement about rank is not agreement about the score
Prometheus 2, published at EMNLP in November 2024, addresses both direct assessment and pairwise ranking with user-defined evaluation criteria. In direct assessment, the evaluator assigns a score; in pairwise ranking, it expresses a preference between outputs. Those formats need not convey the same information. The paper's comparative results belong to its tested setting, not to a present-day ranking asserted here. Prometheus 2 primary paper
Consider two fictional grading sheets for three responses. Sheet One assigns scores of two, three and four to R, S and T. Sheet Two assigns four, five and six. Both order the responses from R through S to T. Yet every score on the second sheet is two points higher. The average score rises from three to five without any change in the ordering.
If a report asks only whether T outranks R, the sheets agree. If it asks whether R reaches a threshold of four, they disagree. This example uses invented numbers on an illustrative scale, not a calibrated measure of quality. Its point is that the same records can agree on one decision while disagreeing on another. A claim of agreement should identify the decision tested.
The distinction also works in reverse. Two fictional sheets can have the same mean while disagreeing about individual responses. Scores of one, three and five average three. Scores of three, three and three also average three. The equality of the means does not establish that the graders treated each response similarly, and the uniform sheet supplies no strict ordering among the three items.
A reader who sees only one average loses that information. A report can preserve it with a per-item comparison or a concise description of the differences. It should not describe a match in averages as a match in every judgment. Nor should it infer that a ranking disagreement implies different overall averages. These are separate properties of the supplied score lists.
The distinction matters when evaluating a claimed improvement in a grading system. Suppose a revision raises all scores by two while keeping every item in the same order. It may change how many responses cross a stated threshold. It has not, in this fixture, improved the separation of the responses by rank. Whether the change is desirable depends on what the score is meant to support.
For public reporting, the useful sentence is specific: the systems agree on the ordering of these three items but differ by two points on every score. That tells a reader much more than the systems are highly aligned. It also leaves room to ask whether the scale has a meaningful interpretation, rather than allowing familiar-looking numbers to supply one by implication.
8. A high agreement rate can hide all the difficult cases
A GEM workshop paper published in 2025 examines judge alignment in a controlled TriviaQA setting. It emphasizes that percentage agreement alone can obscure important differences in the scores assigned to evaluated models. The study's warning is not that agreement percentages are useless, but that their interpretation requires more than the headline percentage. Thakur and colleagues, judge alignment study
For an original Cedar fixture, imagine one hundred items with a stipulated reference label: ninety acceptable and ten unacceptable. A simplistic grading sheet marks every item acceptable. It matches the reference on ninety items, so its agreement rate is 90 percent. It also misses all ten reference-labeled unacceptable items. Both descriptions are true. The second is indispensable if detecting unacceptable answers is the reason for using the grader.
Now construct a second sheet. Of the ninety reference-acceptable items, it marks eighty-four acceptable and six unacceptable. Of the ten reference-unacceptable items, it marks seven unacceptable and three acceptable. Its agreement is ninety-one out of one hundred, or 91 percent. The headline improves by one percentage point, but the pattern of disagreement changes substantially: seven previously missed unacceptable items are now identified, while six acceptable items receive the opposite label.
These arithmetic results do not establish which sheet is preferable for every use. The consequences of the two error types may differ. The necessary first step is to show them separately. Calling the change a one-point improvement conceals the underlying trade-off; calling the second sheet universally superior would add a conclusion the fixture does not provide.
The reference labels in this example are stipulated, not human votes or verified facts about real responses. If a real evaluation uses a human panel, the report must describe that panel and its instructions before treating its labels as a comparison target. Agreement with a specified panel is evidence about that comparison. It should not automatically be rewritten as agreement with all users, factual correctness in every domain or an absence of harmful errors.
There is another denominator question. If the ten unacceptable items were removed before the first sheet was scored, its agreement would become one hundred percent on the remaining ninety. The grader would not have become better at detecting unacceptable answers. The evaluation population would have changed. A report needs an account of which items were included, excluded or left unresolved, especially when a comparison depends on rare cases.
A useful presentation therefore pairs the headline with the actual disagreement counts. In Cedar's second sheet, there are six disagreements in one direction and three in the other. That nine-item disagreement set is where a focused review can learn what the percentage compresses. The number is a starting point for understanding the assessment, not a reason to stop looking at its failures.
9. Length is a possible explanation, not a ready-made diagnosis
The length-controlled AlpacaEval paper, consulted in its March 10, 2025 arXiv revision, uses regression to adjust a particular evaluation for length differences. Its authors explicitly limit the study to the tested AlpacaEval setting and an equal-length comparison assumption; they do not claim to solve every issue with model judges. Length control is not synonymous with truncating answers until their word counts match. Length-Controlled AlpacaEval, version 2
A separate Cedar worksheet contains twenty hand-assigned preferences. In twelve pairs, the preferred answer is also the longer answer; in eight, it is shorter. The fact that the longer answer wins twelve times does not identify why it won. Perhaps the longer answers include required evidence absent from the shorter ones. Perhaps the grades favor unnecessary elaboration. The supplied count alone cannot distinguish those possibilities.
For a concrete authored example, return to the imaginary library notice. A short answer saying only that the room closes at six omits the return-box information. A slightly longer answer containing both facts may deserve a better completeness judgment under the stated task. Cutting the second answer to the first one's length could remove the very information the task requires. Equal word counts would not automatically mean equal informational content.
Consider a different pair in which one answer states both facts once and another repeats the same two facts several times. The difference now concerns repetition rather than an additional required fact. These two invented pairs should not be lumped together simply because one response is longer in each. A length measurement describes text size; it does not by itself classify the role that the additional text plays.
Suppose the Cedar worksheet's twelve longer-answer wins consist of nine pairs with an extra required fact and three pairs without one, according to a separately stipulated content review. That partition gives a more informative description than twelve long wins. It still is not a causal experiment: the response pairs may differ in many other ways. No causal percentage or model-specific bias estimate is inferred from these assigned labels.
The purpose of the example is to make the next evidence question concrete. What comparison would distinguish useful detail from rewarded verbosity under the announced criterion? The answer requires attention to the task and the content, not merely a global preference for shorter prose. A published score should describe any adjustment actually performed and the assumptions it uses, rather than treating the word controlled as a guarantee that all confounding has disappeared.
10. The parser is part of the result
A judgment can be lost or transformed between the evaluator's response and the final chart. That possibility is easiest to examine with artificial input rather than a costly live run. For this article, a local aggregation script processes only hand-authored Cedar labels. It counts the twenty-pair fixture described earlier and checks the published arithmetic. It does not generate judgments, consult a remote model or validate the quality of any answer.
The script also examines four invented output strings under a deliberately simple acceptance rule. The exact strings R, S and TIE are recognized as the three allowed labels. The sentence I choose R is not accepted by that rule. The result is three accepted strings and one format failure. This is a toy parser rule, not an implementation recommendation for a real service.
A different parser could deliberately accept that sentence. If so, the report should say that it uses a different rule. The important distinction is between the raw recorded output and the interpretation applied to it. A change in accepted formats can change completion counts without changing the underlying text at all. It should not be reported as an improvement in answer quality unless separate evidence supports that claim.
Imagine a fictional release that lists four successful judgments after a parser update, compared with three before it. Under the stipulated strings, the additional success means that an existing string is now recognized. There was no new evaluation request, no new response and no newly measured preference. The parser's coverage improved according to its revised rule. That is a useful engineering change, but its scope is narrower than the judge became smarter.
The same principle applies to retries. If a malformed output is replaced by a later valid one, a report can describe both the original attempt and the replacement. It should not silently present the replacement as the only thing that happened when the advertised measure concerns first-attempt completion. Conversely, a report explicitly measuring eventual completion can include a stated retry policy. The claim determines which record is relevant.
For this article's local checks, the stopping point is straightforward. The script confirms the fixture's counts, score arithmetic and parser outcomes against declared expectations. That establishes that the worked examples are internally consistent. It does not establish that any live model would produce those labels, that the parser covers real output variability or that a given rubric measures an important human value. Those would require different evidence.
A small reproducible check is valuable when its accomplishment is described accurately. Its usefulness disappears if a synthetic input file is relabeled as a model benchmark or a passing arithmetic assertion becomes a claim about real-world judgment quality.
11. Preserve disagreement before trying to explain it away
Suppose a final Cedar exercise supplies two independently named, but still entirely fictional, grading sheets for twenty items. The sheets agree on sixteen and disagree on four. A summary can correctly report 80 percent agreement. It should also preserve the four differing judgments so a reviewer can examine what distinguishes them. Without that record, an explanation of the disagreement is only speculation.
Now stipulate a manual review of those four cases. In two, the sheets used different versions of the instruction. In one, a category label was transcribed incorrectly. In the last, both sheets used the same instruction but assigned different grades to the same text. These are invented outcomes that illustrate why a disagreement count does not supply its own diagnosis.
Correcting the two version mismatches and the transcription error might produce nineteen agreements out of twenty, or 95 percent, in a new comparison. The revised result would concern a corrected packet, not the original one. The record should preserve both the original mismatch and the reason for the change. It would be misleading to claim that the graders spontaneously became more consistent or that the original disagreement rate was never observed.
The remaining case deserves its own explanation. Perhaps the criterion permits reasonable disagreement. Perhaps one grade lacks support. Perhaps additional reference material is needed. The synthetic fixture does not decide among these possibilities. What it establishes is that three documented record problems and one unresolved substantive difference should not be compressed into a single story about evaluator bias.
This approach also avoids treating a majority vote as a magic source of truth. In an original three-sheet extension, two sheets choose R and one chooses S. The tally is two to one. That is a fact about the supplied labels. It does not establish that the two sheets interpreted the task correctly, used independent evidence or understood an important exception. The reasons and the relevant reference material still matter.
A report can disclose disagreement without losing its usefulness. In fact, the unresolved cases may identify where its intended use needs care. What matters is whether the report distinguishes observed labels, documented corrections and remaining interpretive uncertainty. Combining them into one polished score makes the output easier to market but harder to examine.
For readers, a practical question is whether the report contains enough information to disagree with it precisely. Can someone identify the affected item, the applicable criterion and the supporting passage? If not, the explanation may be detailed in appearance but weak in evidentiary content. An evaluation should enable a targeted challenge to its reasoning rather than demand acceptance of a numerical verdict.
12. Make the evaluator's contribution visible
A useful judge audit does not end with a single declaration that the judge works. It produces several bounded findings. The output records may be correctly linked to inputs. The parser may implement its stated rules. A selected set of judgments may be stable under a specified change. A rubric may expose a trade-off that an average previously concealed. A comparison with a reference may reveal one kind of error that a high overall agreement rate hid.
Those findings can support decisions, but only at the level examined. The Cedar fixtures demonstrate why. Fourteen retained directions out of twenty pairs describe coverage. Ten R selections out of those fourteen describe a conditional share. A two-point offset in every score preserves a ranking but changes a threshold decision. Ninety percent agreement can coexist with missing every reference-labeled unacceptable item. None of these observations is interchangeable with a probability that a model's answer is correct.
For someone reading a public report, the most useful request is an evidence packet organized around the claim. It should identify the responses evaluated, the criterion applied, the judge configuration actually used, the raw recorded result and the rule that produced the published label. Where private material prevents public release, the report should explain what outsiders can inspect and what remains dependent on the publisher's account. An artificial demonstration can test the processing logic, but cannot stand in for undisclosed empirical results.
The publication should also distinguish a planned test from a completed one. An evaluation design can be thoughtful while producing no observations yet. A script can execute successfully while consuming only synthetic labels. A paper can identify a promising procedure without establishing validity for every future use. Each is a legitimate artifact, provided its status is not inflated into a stronger result.
This article has done the limited local work it describes: constructed clearly artificial fixtures and checked their arithmetic and aggregation. The cited research supplies bounded empirical context from named, dated studies. It does not authorize a new league table, and the fixtures are not evidence that a particular commercial model has a particular weakness today. A live evaluation would require its own inputs, recorded outputs and appropriate review.
The central public-interest question is therefore not whether machines may ever grade machines. It is whether the grading process is visible enough for its claims to be understood and challenged. A judgment becomes more useful when readers can see what produced it, what alternatives were considered and which conclusions remain unsupported. The score's author need not be human for that standard to matter. If anything, an automated judge makes careful attribution of the judgment more important, because the apparent precision can travel much farther than the evidence that earned it.