Curated by hand.
Scored against the cause.

How incidents are chosen

We start from public issue trackers and keep closed bugs with an associated fix and a concrete failure, favoring high-impact, user-facing bugs on stable releases. Feature requests, documentation and test-only issues, and broad refactorings are excluded, as are fixes over 100 changed source lines unless the causal code is clearly separable. In ZooKeeper, 4,789 public issues yield 99 automatic candidates, 50 manual reviews, and 18 benchmark incidents.

A report stays only if its issue-side evidence alone supports diagnosis: symptoms, logs, stack traces, reproduction steps. Comments that reveal the root cause or the patch are stripped from what the agent sees.

How causes are labeled

Three annotators independently mark the pre-fix code that embodies each failure, as file, function, and line range, leaving out tests, refactors, and cleanup. They then reconcile their labels into one gold set; no LLM output enters the labels. On the 51 incidents with retained pre-reconciliation labels, Krippendorff’s α is 0.92 at file level and 0.87 at function level.

How diagnoses are scored

A diagnosis is a list of code blocks, scored against the gold blocks at file, function, line, and block level. Block coverage, the headline metric, asks whether each labeled causal block is reached; line coverage is the stricter overlap check. Block precision penalizes code that is not part of the cause, and mention coverage credits explanations that name a gold function. Each configuration runs three times per incident at medium reasoning effort where the model exposes it; scores are averaged within each incident, and the 120 incidents count equally.

Keeping the answer out of reach

Agents work in an anonymized checkout with every commit after the bug removed. A leakage guard scans each run’s reasoning and tool use for the withheld issue identifiers, and flagged runs are discarded and rerun. In closed-book probes, the issue ID does not help: function coverage is 17.3% with it and 21.4% without, against 45.5% with repository access.

Why coverage matters

In a blinded audit of 20 zero-coverage diagnoses, 16 were wrong or incomplete. In a six-bug repair pilot, high-coverage diagnoses led the repair agent to edit the gold block in 50% of attempts, against 11% with no diagnosis.