Owned public project · rudycelekli/gradia-wind-tunnel
gradia-wind-tunnel · Testing a benchmark scorer against an oracle
Problem
An evaluator can accept a wrong answer when the reward logic checks a proxy for correctness.
Design and implementation
The Wind Tunnel compares a scorer decision with an independent machine oracle and localizes exploitable differences using single-variable changes. Replayable records connect an exploit to the exact comparison.
Evidence and scope
The public evidence concerns the tested scorer, oracle, and examples. Other evaluators require separate calibration and adversarial evidence.