Upstream contribution · headroomlabs-ai/headroom
headroom · Removing provider outages from Headroom answer scores
Problem
Evaluation paths converted provider exceptions to ordinary answer strings. Two identical error messages could score as a perfect match, and a diagnostic that named the expected answer could receive credit.
Engineering response
Failed calls now populate the existing error state and bypass answer scoring and judge evaluation. A successful response containing the literal text ERROR remains eligible for normal scoring.
Evidence and scope
The documented baseline awarded 2 of 2 cases in each affected evaluator. After the change, a controlled failure and one valid response correctly produced 1 of 2. The figures are fixture results.