Agents can find the right file. Can they find the bug?

When an AI agent debugs a large distributed system, it usually finds its way to the right part of the codebase. Pinpointing the code that actually caused the failure is a different story. DistDebug-Bench is a benchmark that measures exactly this step: given a real incident report and the codebase as it stood before the fix, can an agent identify the source code responsible?

Across 120 real-world bugs from 11 widely used distributed systems, the strongest agent we evaluated, GPT-6 Astra, reaches the right file 84% of the time but finds only 68% of the labeled root-cause code blocks. Performance drops sharply when the bug report doesn't name the responsible code, and the same agent often gives different answers on repeated attempts.

Diagnosis is the missing piece

Coding agents have made rapid progress on benchmarks that ask them to write functions, solve contest problems, or resolve GitHub issues. However, most of these benchmarks live in small, self-contained settings with short causal chains. Most also grade the agent on its patch, the code change it proposes, by checking whether that change makes a failing test pass. That blurs finding the cause and fixing it. A patch can make the test pass by suppressing the symptom, for example by catching an error instead of preventing it, without fixing what caused it. And even when the correct fix is only a few lines, finding those lines may require following a long chain of cause and effect through the system.

In practice, engineers have to diagnose a failure before they can write a proper fix. That step is where much of the time goes, and existing benchmarks don't measure it directly.

Distributed systems make diagnosis especially hard. They span millions of lines of code across many subsystems, so the responsible code is a tiny fraction of the search space. Failures cross modules, network calls, and service tiers, so the place an error shows up is often far from the place it started. And the evidence is partial: issue reports, logs, and stack traces offer fragments, and some failures, known as gray failures, have ambiguous symptoms. Debugging these systems means reasoning from a high-level symptom down to low-level code under weak evidence, which is exactly the kind of long-horizon reasoning agents are now being asked to do.

Building the benchmark

Each DistDebug-Bench task gives the agent what an engineer would have: the incident report, any attached evidence such as logs or stack traces, and the repository as it stood before the fix, with no internet access. The agent must return the code blocks it believes are responsible, each with a file, function, and line range. We then score that answer against human labels at the file, function, line, and block level.

Scroll sideways to see the whole figure.

The 120 incidents come from 11 production-scale distributed systems written in five languages. We selected closed, high-impact bugs that describe a concrete failure, can be diagnosed from the report and its attached evidence alone, and have a fix local enough to label precisely. For each one, we marked the minimal pre-fix code that embodied the bug, leaving out tests, refactors, and extra defensive checks.

The resulting tasks are hard in the same ways real incidents are. More than two-thirds of reports never name the responsible method, and nearly two-thirds have no usable stack trace from inside the project. Almost half of the bugs need more than one code block to explain, and nearly a quarter span more than one file. Yet the median labeled block is only five lines long, so an agent has to follow a non-local causal chain and land on a precise location.

A bug in ZooKeeper, a widely used coordination service, shows what this looks like in practice. The report for ZOOKEEPER-2845 describes a cluster that ends up inconsistent after a sequence of restarts during leader election: two replicas agree on the data, while the third replays a transaction from its own log that the others never received. Everything in the report, from leader election and quorum formation to snapshots and log replay, points toward the quorum code. The heart of the bug is elsewhere, in the server's shutdown routine, which keeps the in-memory database alive across the restart and lets stale state survive into the next quorum. Agents that follow the report's vocabulary often narrow their search to the quorum directory, find code that plausibly matches the symptom, and stop, instead of going one step further to find the actual root cause.

Scroll sideways to see the whole figure.

Because the bugs are public, we also guard against agents looking up the answer. Issue comments that reveal the root cause or fix are stripped out, workspaces are anonymized, and every commit after the buggy snapshot is removed, so the fix is unreachable. A leakage guard scans every run for the original issue identifier, and flagged runs are discarded and rerun.

What we found

We evaluated 17 agent configurations. Thirteen models ran inside mini-SWE-agent, a minimal harness common in coding benchmarks, and two of them also ran inside the Codex CLI and OpenCode terminal agents. Each configuration attempted every bug three times. Our primary metric is block coverage, which tells us what fraction of the labeled root-cause blocks the diagnosis reaches, paired with precision, which penalizes naming unrelated code alongside it.

The gap between file coverage and block coverage is the central result. Agents usually reach the right file, but recovering the specific code responsible is much harder: in every configuration, block coverage trails file coverage by 16 to 24 points. Higher coverage also tends to come with broader answers: GPT-5.5 names about 3.6 blocks per diagnosis, and only about 35% of them are part of the root cause.

Every configuration's scores, with confidence intervals, precision, and token use, are on the leaderboard.

When do agents struggle?

The difficulty is not spread evenly. We grouped bugs by two kinds of help a report can offer: whether its text names the responsible code, and whether its stack trace begins near the fix. Agents do well when the report names the responsible method, and much worse when it names nothing at all.

With no stack trace, GPT-6 Astra reaches about 95% of the root-cause blocks when the report names the method, but only 65% when the report is opaque, naming no relevant method, file, or class. When a stack trace exists but begins in a different component, coverage on opaque bugs falls to about 38%. These are not rare cases. Opaque reports make up 50 of the 120 incidents, so a large share of the benchmark requires navigating from symptom to cause without a direct pointer. The report's first clues heavily shape where an agent ends up, and agents have limited ability to move past them.

Same bug, different answer

Repeating a run on the same bug can produce a different diagnosis. We count a bug as flaky when its block coverage varies by more than 20 points across the three attempts. Even GPT-6 Astra, the strongest configuration, is flaky on 15 of the 120 bugs, and HY3 on 35. The instability concentrates where evidence is weakest: for GPT-5.5, 26% of opaque bugs without a stack trace are flaky, compared with 13% of no-stack-trace bugs whose report names the method.

This matters because a single successful run can hide brittle behavior. On 14 bugs, GPT-5.5 reached every root-cause block in at least one attempt but not in every attempt. Those failures are not about what the model can do but about whether it does it reliably, and taking the best of three attempts would lift its block coverage from 65% to about 72%.

Newer models and different harnesses shift the trade-off

Newer models do not improve uniformly. GPT-5.5 beats GPT-5.2-Codex overall, from 56% to 65% block coverage, and gains about 15 points on opaque bugs without a stack trace. However, it names more blocks per answer, 3.6 instead of 2.3, and its precision falls from 43% to 35%. Broader exploration helps when clues are scarce, but it also pulls unrelated code into the answer.

Spending more tokens does not buy coverage either. GPT-6 Astra uses about 132K tokens per run; Kimi K2.6 uses 2.8M, about twenty times as many, and reaches 9 fewer points of block coverage.

The harness shapes behavior too. With the same GPT-5.2-Codex model, mini-SWE-agent casts the widest net, Codex CLI is the narrowest and most precise, and OpenCode sits in between.

All results use GPT-5.2-Codex.

These differences move the balance between coverage and precision, but every harness degrades sharply on reports that don't point to the responsible code. Scaling the model or swapping the harness changes the style of a diagnosis without fixing what makes it hard.

Tunnel vision: why agents get stuck

To understand why diagnoses go wrong, we read through the agents' execution traces. The same behavior kept appearing: an agent settles on a plausible local explanation early, and if that first search missed the real cause, it rarely recovers. Three factors drive it.

Small tool choices lead to different evidence: A diagnosis depends on many low-level decisions, such as which keywords to search, which directory to enter, and which file to open first. When a report is opaque, small differences in those choices send runs into different subsystems. Each run then builds its explanation around whatever code it happened to reach first. On about a quarter of bugs, GPT-5.4's three attempts don't all start by reading the same source file.

Early hypotheses rarely get revisited: Once an agent settles on an explanation, each new file it reads tends to deepen that explanation rather than test it. We counted a run as a late recovery if its final answer was not in the first two source files it opened. Among the traces we could parse, only about 21% of runs recover late, and only about 11% of successful runs do. In other words, if the first two files miss, agents rarely find their way back. The few runs that do recover are the ones that step back and ask what earlier change made the symptom possible.

There is no cheap way to check an answer: A patch can be checked by running the tests. A diagnosis usually cannot, and distributed failures are often hard to reproduce, so agents stop once an explanation sounds right. The signals they need do exist, including raised exceptions, crash sites, logs, and documentation, but they only become useful when an agent cross-checks its diagnosis against them before committing.

What this means for agent design

Each of these factors points to something future harnesses should provide.

Adaptive context construction: Rather than leaving exploration entirely to ad hoc shell commands, tools can infer what the agent is looking for and fill in the surroundings, such as related class definitions, call paths, and upstream or downstream code. Two slightly different searches would then surface the same evidence instead of steering the run in different directions.

Hypothesis-aware context management: Today, an agent accumulates everything in a single, ever-growing context, so whichever explanation comes first crowds out the rest. Each hypothesis should instead collect its own evidence, and the agent should be able to switch between hypotheses and prune the ones that have been ruled out.

Explicit correctness checks: Harnesses can inject correctness signals into the agent's context, or use a separate evaluator to ask whether a diagnosis explains the failure from start to finish and whether blaming a given component is backed by concrete evidence.

Takeaways

Today's agents are good at reaching the right part of a large codebase but much less reliable at pinpointing the code that caused a failure, especially when the bug report doesn't point the way. Their answers vary from run to run, and they rarely recover from a wrong first guess. Newer models and different harnesses shift the balance between coverage and precision, but neither removes the underlying difficulty.

Closing that gap will take agents that gather context adaptively, keep competing explanations open, and check their reasoning before they commit. DistDebug-Bench gives the community a concrete way to measure progress toward that goal, with fine-grained breakdowns that show not just whether an agent fails, but where.

The benchmark, its evaluator, and the traces behind every number in this post are open. Run your agent on a single incident or the full benchmark, and see how it compares on the leaderboard. The submission guide explains how to add your results, and the paper has the details.