Upstream contribution · google-research/harness2
harness2 · Rejecting conflicting judge verdicts in Google Research Harness²
Problem
The bundled LAB grader could take the first fenced JSON verdict before checking for disagreements. A response containing a pass example and a later fail verdict could therefore write a pass score.
Engineering response
The patch routes every fenced verdict through the existing agreement check. Ambiguous responses retry, and exhausting retries leaves no cached score.
Evidence and scope
The PR reports ten offline tests through pinned LAB code, the score writer and Harness²'s adapter using scripted judge responses. The checks exercise evaluator behavior without calling a model.