DistDebug-Bench

Submit

Submit full results

Run your agent on all 120 incidents, three attempts each, and send its diagnoses and trajectories. We rescore every submission with the official evaluator before listing it on the leaderboard.

01 / RUN

The full benchmark

  • All 120 incidents, three attempts each.
  • Only the report, its evidence, and the pre-fix repository; no internet access.
02 / SUBMIT

Diagnoses and trajectories

  • A diagnosis.json and the full trajectory for every attempt.
  • A metadata file with the model, harness, settings, and token use.
03 / SCORE

The paper’s protocol

  • The first three schema-valid attempts per incident count.
  • Attempts flagged by the leakage guard are discarded.

Run the benchmark

The runner reproduces the paper’s Mini-SWE-Agent setup, with one config per model in configs/mini-swe-agent/. It needs Python 3.10 or later, Linux with bubblewrap, and an API key for your model provider.

From the repository root
pip install -e ".[agent]"
export OPENAI_API_KEY=...   # or the key your config's provider needs

# Every incident until it has three valid attempts.
# Rerunning resumes where it stopped.
python -m distdebug_bench.run \
    --config configs/mini-swe-agent/gpt-5.4.yaml --parallel 8

Each attempt is saved under results/<run>/<incident>/run-NN/ (here results/mini-swe-agent-gpt-5.4): trajectory.json, diagnosis.json, and session_stats.json with API calls, tokens, and cost. Add --bug zookeeper-2845 or --system kafka to try a few incidents first.

To evaluate your own agent, give it what the runner gives Mini-SWE-Agent: the report, its attachments, and the repository at the pre-fix snapshot. Have it write the same files for each attempt. The benchmark README shows how to load an incident and check out its snapshot so the fix stays unreachable, and distdebug_bench/validate_diagnosis.py checks each diagnosis.json.

Check your run locally

The official evaluator applies the scoring above. Run the leakage guard, run the same command again to replace any flagged attempts, then score your results directory.

From the repository root
python -m distdebug_bench.leakage results/<run> --quarantine
python -m distdebug_bench.score results/<run>

Send your submission

Copy submissions/metadata.template.yaml, fill it in, and package the run. Upload the trajectories archive anywhere public (Google Drive, Hugging Face, Zenodo, ...), set trajectories_url in the submission’s metadata.yaml, and check it.

From the repository root
python -m distdebug_bench.submit package results/<run> --metadata metadata.yaml
python -m distdebug_bench.submit check submissions/<date>_<agent>__<model>

Then open a pull request adding the submission directory to the benchmark repository, or email jyuxuan@umich.edu with the subject [DistDebug-Bench-Submit] <agent> + <model> and links to the submission directory and trajectories. Describe your system in a paper, technical report, or blog post.

The diagnosis.json schema
diagnosis.json · one per incident and attempt
{
  "buggy_blocks": [
    {
      "file": "relative/path/to/file",
      "function": "fully.qualified.FunctionName",
      "start_line": 123,
      "end_line": 130,
      "bug_contribution": "how this block contributes to the bug",
      "patch": "a brief patch sketch"
    }
  ],
  "bug_cause_description": "overall explanation of the root cause"
}

List every causal block, each with its file, fully qualified function, line range, contribution to the bug, and a brief patch sketch. Use exactly these keys. Agents diagnose only and leave the repository unmodified.