DistDebug-Bench

Findings

What makes a diagnosis reliable?

Reliable diagnosis means pinpointing causal code, limiting false leads, and sustaining that performance across repeated runs. The findings below also examine the resources an investigation consumes, the report evidence it depends on, and the role of the agent harness.

The localization gap

Finding the file is only the start.

Across these configurations, coverage falls by 16–24 percentage points when agents must identify the causal code blocks.

Coverage (%) · higher is better
FileRoot-cause block
  • OpenAI
  • Anthropic
  • MiMo
  • Kimi
  • DeepSeek
  • HY3
  • Gemini

  1. Mini-SWE-Agent: GPT-6 AstraMini-SWE-Agent

    16.2 pp gap

  2. Mini-SWE-Agent: Claude Opus 5.5Mini-SWE-Agent

    15.6 pp gap

  3. Mini-SWE-Agent: GPT-5.5Mini-SWE-Agent

    18.9 pp gap

  4. Mini-SWE-Agent: GPT-5.4Mini-SWE-Agent

    16.7 pp gap

  5. Mini-SWE-Agent: 5.3-CodexMini-SWE-Agent

    19.4 pp gap

  6. Mini-SWE-Agent: MiMo v2.5 ProMini-SWE-Agent

    20.4 pp gap

  7. Mini-SWE-Agent: Kimi K2.6Mini-SWE-Agent

    19.2 pp gap

  8. Mini-SWE-Agent: MiMo v2.5Mini-SWE-Agent

    19.8 pp gap

  9. Mini-SWE-Agent: DeepSeek V4 FlashMini-SWE-Agent

    19.6 pp gap

  10. Mini-SWE-Agent: 5.2-CodexMini-SWE-Agent

    19.5 pp gap

  11. Mini-SWE-Agent: HY3Mini-SWE-Agent

    20.9 pp gap

  12. Mini-SWE-Agent: Gemini 3.5Mini-SWE-Agent

    18.4 pp gap

  13. Mini-SWE-Agent: o4-miniMini-SWE-Agent

    23.8 pp gap

  14. Codex CLI: 5.2-CodexCodex CLI

    18.1 pp gap

  15. Codex CLI: o4-miniCodex CLI

    21.1 pp gap

  16. OpenCode: 5.2-CodexOpenCode

    19.1 pp gap

  17. OpenCode: o4-miniOpenCode

    20.7 pp gap

Models and harnesses evaluated on the same 120 incidents. Lighter bars show file coverage; solid bars show root-cause block coverage. Colors identify model families.

Five findings

Consistency across runs

Reports without direct code clues show more run-to-run variation.

GPT-6 Astra · Mini-SWE-Agent

4.3× the run-to-run variation when the report names no causal code

  1. Causal method named · n=16

  2. No causal code named · n=42

  1. Causal method named · n=16

  2. No causal code named · n=42

How repeatability is measured

The plot compares reports that name the causal method with reports that name no causal code; neither has a usable internal stack trace. For each incident, we take the standard deviation of block coverage across its three runs and average it within each group. An incident is flaky when its block coverage spans more than 20 percentage points across runs.

Token use and cost

Cheaper tokens don’t make a cheaper diagnosis: Kimi K2.6 costs twice as much per diagnosis as GPT-5.4.

Mini-SWE-Agent · 120 incidents · Official API prices

Mean tokens per diagnosis, estimated cost at official API prices, and block coverage on the same 120 incidents: GPT-5.4, 690K tokens, $0.49, 63.1%; GPT-5.5, 468K tokens, $0.65, 65.1%; DeepSeek V4 Flash, 2.291M tokens, $0.32 without cache discounts, 56.4%; Kimi K2.6, 2.796M tokens, $0.98, 58.3%. Tokens include input and output, with cached input counted once. Whiskers show 95% incident-bootstrap intervals across incidents for tokens, cost, and coverage.
Usage and pricing details

We apply each provider’s official per-token prices, checked October 26, 2026, to the recorded usage of every run.

USD per million tokens · official model-provider prices
ModelUncached inputCached inputOutput
GPT-5.4$2.50$0.25$15.00
GPT-5.5$5.00$0.50$30.00
DeepSeek V4 Flash†$0.14$0.0028$0.28
Kimi K2.6$0.95$0.16$4.00

Tokens are input plus output per diagnosis, with cached input counted once; output includes reasoning. Cost = (uncached input × input rate + cached input × cache rate + output × output rate) ÷ 1,000,000. The GPT runs stay below the long-context surcharge threshold.

†DeepSeek did not report cache hits, so its $0.32 credits no cache discount.

Price sources: GPT-5.4 · GPT-5.5 · Kimi K2.6 · DeepSeek V4 Flash. Usage, rates, and calculation data · Leaderboard.

Coverage and precision

Broader diagnoses raise coverage but add false leads.

Mini-SWE-Agent · 120 incidents

With Mini-SWE-Agent, GPT-5.5 reaches 65.1% block coverage versus 56.4% for GPT-5.2-Codex, while block precision falls from 43.1% to 34.7%. Predicted blocks per diagnosis increase from 2.3 to 3.6. Whiskers show 95% incident-bootstrap confidence intervals.
What coverage and precision measure

Block coverage measures how much of the annotated causal code a diagnosis reaches; block precision measures how much of the predicted code is part of the cause. GPT-5.5 names 3.6 blocks per diagnosis instead of GPT-5.2-Codex’s 2.3, gaining 8.7 points of coverage and losing 8.4 points of precision. Within a model generation, broader diagnoses trade precision for coverage: Gemini 3.5 Flash names the fewest blocks (1.4) and has the highest precision (51.1%). The newest models, GPT-6 Astra and Claude Opus 5.5, raise both: 67.7% and 66.4% block coverage at 42.7% and 45.5% precision, versus GPT-5.5’s 65.1% at 34.7%.

Diagnosis with incomplete evidence

Without a pointer to the code, coverage drops sharply.

GPT-6 Astra · Mini-SWE-Agent

  1. Causal method namedNo internal stack trace · n=16

  2. No causal code namedNo internal stack trace · n=42

  3. No causal code namedCross-component crash site · n=7

Report groups and sample sizes

GPT-6 Astra reaches 94.8% block coverage on 16 reports that name the causal method and 65.0% on 42 reports that name no causal code; neither group has an internal stack trace. On the seven reports whose crash site lies in a different component from the cause, coverage falls to 38.1%.

The agent harness

The same model trades coverage for precision across harnesses.

GPT-5.2-Codex · 120 incidents

Holding GPT-5.2-Codex fixed, Mini-SWE-Agent has 56.4% block coverage, 43.1% block precision, and 2.3 predicted blocks per diagnosis; Codex CLI has 46.5%, 49.3%, and 1.6; OpenCode has 50.0%, 47.7%, and 1.8. Coverage and precision have 95% incident-bootstrap confidence intervals.
Harness comparison details

With GPT-5.2-Codex held fixed, Mini-SWE-Agent reaches the highest block coverage, while Codex CLI has the highest block precision and names the fewest blocks per diagnosis.

About these results

Every mean averages three runs within each incident and weights the 120 incidents equally. Whiskers are 95% incident-bootstrap confidence intervals for the mean; run-to-run variation appears in the consistency plot. See how diagnoses are scored.

Read the full findings in the paper Download figure data