DistDebug-Bench

A benchmark for distributed-system diagnosis

DistDebug-Bench

When a distributed system fails, can your agent find the code that caused it?

DistDebug-Bench gives agents 120 real failures from 11 production distributed systems. Failures surface far from their cause, and reports, logs, and stack traces show only part of what happened.

Read the paper Benchmark on GitHub

Leaderboard

Top five configurations by block coverage
ModelBlock coverage
95% CI		Block
precision		Tokens /
run
GPT-6 Astra67.7 [60.4, 74.9]42.7132K
Claude Opus 5.566.4 [59.2, 73.6]45.5193K
GPT-5.565.1 [57.8, 72.3]34.7468K
GPT-5.463.1 [56.0, 70.2]39.5690K
GPT-5.3-Codex59.7 [52.2, 67.2]40.0416K
Full leaderboard

The benchmark

Find the cause.
Explain it in code.

Each incident is a real, closed bug from a production system’s issue tracker, most often a logic, concurrency, or error-handling fault. The agent gets the report, its logs or stack traces, and the repository before the fix, and must name the code blocks that caused the failure. Most coding benchmarks grade the patch; DistDebug-Bench scores the diagnosis itself, against human-labeled causal code.

11 systems, 120 incidents

SystemWhat it isLanguageIncidents
RayAI compute enginePython, C++21
ElasticsearchSearch and analytics engineJava19
ZooKeeperCoordination serviceJava18
TiKVTransactional key-value storeRust11
HDFSDistributed file systemJava10
KubernetesContainer orchestrationGo10
etcdConsistent key-value storeGo8
KafkaEvent streaming platformJava, Scala8
Ignite 3Distributed databaseJava7
HBaseWide-column storeJava5
IgniteIn-memory databaseJava3

Why the incidents are hard

69%of reports never name the causal method

63%have no usable stack trace from inside the project

45%need more than one code block to explain

5 linesis the median labeled block, so answers must be precise

How the benchmark is built

Findings

Where agents fall short.

Block coverage is the share of the labeled root-cause code blocks a diagnosis reaches; block precision is the share of the code it names that is part of the cause.

The localization gap

Finding the file is only the start: across these configurations, coverage falls by 16–24 percentage points when agents must identify the causal code blocks.

Coverage (%) · higher is better
FileRoot-cause block
  • OpenAI
  • Anthropic
  • MiMo
  • Kimi
  • DeepSeek
  • HY3
  • Gemini

  1. Mini-SWE-Agent: GPT-6 AstraMini-SWE-Agent

    16.2 pp gap

  2. Mini-SWE-Agent: Claude Opus 5.5Mini-SWE-Agent

    15.6 pp gap

  3. Mini-SWE-Agent: GPT-5.5Mini-SWE-Agent

    18.9 pp gap

  4. Mini-SWE-Agent: GPT-5.4Mini-SWE-Agent

    16.7 pp gap

  5. Mini-SWE-Agent: 5.3-CodexMini-SWE-Agent

    19.4 pp gap

  6. Mini-SWE-Agent: MiMo v2.5 ProMini-SWE-Agent

    20.4 pp gap

  7. Mini-SWE-Agent: Kimi K2.6Mini-SWE-Agent

    19.2 pp gap

  8. Mini-SWE-Agent: MiMo v2.5Mini-SWE-Agent

    19.8 pp gap

  9. Mini-SWE-Agent: DeepSeek V4 FlashMini-SWE-Agent

    19.6 pp gap

  10. Mini-SWE-Agent: 5.2-CodexMini-SWE-Agent

    19.5 pp gap

  11. Mini-SWE-Agent: HY3Mini-SWE-Agent

    20.9 pp gap

  12. Mini-SWE-Agent: Gemini 3.5Mini-SWE-Agent

    18.4 pp gap

  13. Mini-SWE-Agent: o4-miniMini-SWE-Agent

    23.8 pp gap

  14. Codex CLI: 5.2-CodexCodex CLI

    18.1 pp gap

  15. Codex CLI: o4-miniCodex CLI

    21.1 pp gap

  16. OpenCode: 5.2-CodexOpenCode

    19.1 pp gap

  17. OpenCode: o4-miniOpenCode

    20.7 pp gap

Models and harnesses evaluated on the same 120 incidents. Lighter bars show file coverage; solid bars show root-cause block coverage. Colors identify model families.

Report evidence

Without a pointer to the code, coverage drops sharply.

GPT-6 Astra · Mini-SWE-Agent

  1. Causal method namedNo internal stack trace · n=16

  2. No causal code namedNo internal stack trace · n=42

  3. No causal code namedCross-component crash site · n=7

GPT-6 Astra reaches 94.8% block coverage on 16 reports that name the causal method and 65.0% on 42 reports that name no causal code; neither group has an internal stack trace. On the seven reports whose crash site lies in a different component from the cause, coverage falls to 38.1%.

Consistency across runs

Reports without direct code clues show more run-to-run variation.

GPT-6 Astra · Mini-SWE-Agent

4.3× the run-to-run variation when the report names no causal code

  1. Causal method named · n=16

  2. No causal code named · n=42

  1. Causal method named · n=16

  2. No causal code named · n=42

For each incident we take the standard deviation of block coverage across its three runs. Without a usable internal stack trace, GPT-6 Astra averages 6.4 points when the report names no causal code and 1.5 when it names the causal method; 7 of 42 and 1 of 16 incidents are flaky, with block coverage spanning more than 20 points across runs.

Means average three runs within each incident and weight the 120 incidents equally; whiskers are 95% incident-bootstrap confidence intervals.

All findings

Use DistDebug-Bench

Try one incident.
Submit the full run.

Try one incident

Run Mini-SWE-Agent on ZooKeeper-2845 with the paper’s protocol. From a benchmark checkout, the runner prepares an anonymized workspace at the buggy ZooKeeper revision, runs the agent in a sandbox, and the scorer applies the paper’s evaluator. Python 3.10+, Linux with bubblewrap, and a configured model API key are required.

From the repository root
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[agent]"

# Set OPENAI_API_KEY in your environment.
python -m distdebug_bench.run \
    --config configs/mini-swe-agent/gpt-5.4.yaml \
    --bug zookeeper-2845 --runs 1
python -m distdebug_bench.score \
    results/mini-swe-agent-gpt-5.4 --attempts 1

configs/mini-swe-agent/ has a config for each model in the paper. Runs call your configured model provider.

Each attempt writes trajectory.json, diagnosis.json, and diagnosis_scores.json. To evaluate another agent, have it write the first two.

Submit full results

Run your agent on all 120 incidents, three attempts each, and send its diagnoses and trajectories. We rescore every submission with the official evaluator before listing it on the leaderboard.

  1. 01 / RUNAll 120 incidents, three independent attempts each, with only the report, its evidence, and the pre-fix repository. Any agent or harness.
  2. 02 / SUBMITA diagnosis.json and the full trajectory for every attempt, plus the model, harness, settings, and token use or cost.
  3. 03 / SCOREThe paper’s protocol: the first three schema-valid attempts per incident count, and the leakage guard discards flagged attempts.

The guide covers the local leakage check, packaging a run, and the diagnosis.json schema.

Submission guide

Cite

DistDebug-Bench: Can LLM Agents Diagnose Bugs in Distributed Systems? Jiang et al., NeurIPS 2026. Paper · GitHub · Dataset · Traces

Cite the paper
@inproceedings{jiang2026distdebugbench,
  title = {{DistDebug-Bench}: Can {LLM} Agents
           Diagnose Bugs in Distributed Systems?},
  author = {Jiang, Yuxuan and Liu, Chenglin and
    Rose, John and Miller, Mason and
    Thavamani, Ruthesh and Wu, Juntao and
    Huang, Yinwei and Xu, Jiahao and Huang, Peng},
  booktitle = {Advances in Neural Information
    Processing Systems (NeurIPS),
    Track on Evaluations and Datasets},
  year = {2026}
}