Upstream contribution · facebookresearch/swe-sweep
swe-sweep · Accounting for grader crashes in SWE-Sweep
Problem
A grader crash writes a short reward record with a failure marker. The score reader expected fields from a completed trial, so a crashed trial could vanish from evaluation failure counts.
Engineering response
The scorer checks the explicit grader_failed marker before requiring completed-trial fields. It reports the crash and keeps the trial out of the headline score.
Evidence and scope
The merged test plan covers scoring and grader-failure records, including the distinction between a crash and a file that is merely incomplete.