A passing report is not a completed test run
Rejecting incomplete Vitest execution evidence in Agentic QE.
The problem
A test automation system could accept a passing JSON string from stdout when the report file owned by the current Vitest run was missing. A second path accepted passing assertions even when the process subsequently exited with an error. Both cases could turn incomplete execution evidence into a successful result.
Before
Missing current-run report + passing stdout → a possible false pass. Passing report + process exit 1 → a possible false pass in retry, flaky detection or scheduling.
After
Vitest consumers require the current run’s own report. A report and process verdict must agree before a retry is treated as passed. Flaky detection and scheduling reject execution failures while retaining ordinary failed assertions for analysis.
The contribution
The merged change tightened the shared report reader and its five consumers, preserved the existing Jest fallback, and kept an asynchronous report directory until a timed-out child closed. New regressions covered missing reports, contradictory runner/report verdicts, spawn failure and cleanup timing.
Validation and provenance
The PR records three regression cases failing against the baseline, then 203 passing tests across 12 focused files, including real Vitest controls. It also records typecheck, build and changed-source lint checks. Those are the original contributor-reported checks; this case study independently rechecked GitHub authorship, merged state and the code diff on November 4, 2026, without rerunning the historical upstream environment.
Published release source
On November 4, 2026, the upstream v3.15.18 release, published November 1, pointed to commit c42c01813724497291e87b346ebfaa59f46f99f3. GitHub’s commit comparison places this PR’s merge in that release’s ancestry. The report reader at the immutable release commit still reads the current run’s report file and does not substitute stdout for a missing report.
This verifies release-source inclusion and retention of that report-reader contract. The full release environment was not rerun for this check.
Published npm artifact
A separate November 4 artifact inspection downloaded agentic-qe@3.15.18 from npm and verified its tarball against the registry’s SHA-512 integrity value. The archive contains package/dist/shared/vitest-json-report.js; its report reader requires the current run’s report file and ignores stdout as result evidence. The artifact inspection receipt records the registry URL, tarball digest, integrity value and inspected member digest.
This establishes distribution of that report-reader code in the inspected package version. No package code or install scripts were executed. It does not establish installations, active users, runtime behavior, deployment, customer outcomes or a worldwide impact ranking.
What this does not establish
This fix did not establish termination of the entire child process tree after cancellation; that scope was explicitly tracked separately in issue #707. A merged fix is evidence of upstream acceptance, not proof that any particular customer or deployed release is using it.
Engineering lesson
AI engineering includes checking the tools that judge the AI’s work. A report needs provenance to the current run, and a successful assertion cannot erase a failed execution boundary.
Inspect the evidence
Merged pull request and original validation narrative · Immutable merge commit · Source at that commit