Upstream contribution · open-compass/VLMEvalKit
VLMEvalKit · Count failed video-model API responses as incorrect
Problem
VLMEvalKit marked failed API predictions with a score of minus one. Video benchmark aggregation used truthiness, so that sentinel boosted accuracy instead of registering a failed prediction.
Engineering response
Count a prediction as correct only when its score is exactly one, while keeping failed rows in the denominator. Align overall and per-dimension aggregation with the correctness predicate.
Evidence and scope
The PR records two before-and-after aggregation tests. In the fixture, one correct result out of four was previously displayed as 75% and now computes to 25%.