Leaderboard

Every configuration diagnoses the same 120 real distributed-system failures. Rows rank by block coverage by default; select a column heading to sort by it.

Add your result

Protocol. 120 incidents × 3 attempts. The first three schema-valid attempts per incident are scored. Block coverage is the primary metric, shown with its incident-bootstrap 95% CI. Every configuration runs at medium reasoning effort.

Generated 2026-11-02 · Download the data (JSON)

Scroll sideways for every metric.

DistDebug-Bench leaderboard: 17 configurations on 120 incidents, sorted by block coverage, highest first.
Rank	Model	File	Func.	Line	Block	File	Func.	Line	Block	Flaky bugs	Blocks / diagnosis	Calls / run	Tokens / run
		Localization coverage (higher is better)				Localization precision (higher is better)				Instability (lower is better)	Effort (lower is better)
1	GPT-6 AstraMini-SWE-AgentMedium effortProprietary	83.9	58.4	53.9	67.7 95% CI [60.4, 74.9]	66.4	37.8	19.9	42.7	15 (12%)	3.3	9.0	132K
2	Claude Opus 5.5Mini-SWE-AgentMedium effortProprietary	82.0	60.4	53.5	66.4 95% CI [59.2, 73.6]	67.4	39.2	21.0	45.5	17 (14%)	2.8	9.2	193K
3	GPT-5.5Mini-SWE-AgentMedium effortProprietary	84.0	55.5	52.7	65.1 95% CI [57.8, 72.3]	58.1	29.1	14.6	34.7	20 (17%)	3.6	21.9	468K
4	GPT-5.4Mini-SWE-AgentMedium effortProprietary	79.8	53.1	50.4	63.1 95% CI [56.0, 70.2]	59.9	32.3	14.6	39.5	21 (18%)	2.9	25.0	690K
5	GPT-5.3-CodexMini-SWE-AgentMedium effortProprietary	79.1	53.0	44.5	59.7 95% CI [52.2, 67.2]	63.2	35.2	17.8	40.0	22 (18%)	2.6	22.7	416K
6	MiMo v2.5 ProMini-SWE-AgentMedium effortOpen weights	78.9	60.1	43.4	58.5 95% CI [50.8, 66.1]	62.3	37.8	17.3	38.1	22 (18%)	2.9	55.5	2,643K
7	Kimi K2.6Mini-SWE-AgentMedium effortOpen weights	77.5	51.2	44.8	58.3 95% CI [50.9, 65.6]	66.3	38.8	19.6	45.9	32 (27%)	2.4	66.1	2,796K
8	MiMo v2.5Mini-SWE-AgentMedium effortOpen weights	77.8	54.2	43.9	58.0 95% CI [50.5, 65.4]	59.5	33.4	16.4	36.5	26 (22%)	3.1	49.7	2,332K
9	DeepSeek V4 FlashMini-SWE-AgentMedium effortOpen weights	76.0	49.4	45.9	56.4 95% CI [49.0, 63.7]	57.4	28.6	13.0	32.7	32 (27%)	3.2	61.6	2,291K
9	GPT-5.2-CodexMini-SWE-AgentMedium effortProprietary	75.9	48.8	45.9	56.4 95% CI [48.5, 64.2]	63.8	35.4	16.1	43.1	18 (15%)	2.3	32.1	666K
11	HY3Mini-SWE-AgentMedium effortOpen weights	72.5	49.1	39.7	51.6 95% CI [44.1, 59.1]	60.9	35.6	15.5	38.7	35 (29%)	2.7	32.5	768K
12	GPT-5.2-CodexOpenCodeMedium effortProprietary	69.1	45.3	37.1	50.0 95% CI [42.4, 57.9]	66.8	42.8	21.4	47.7	23 (19%)	1.8	18.2	532K
13	Gemini 3.5 FlashMini-SWE-AgentMedium effortProprietary	65.9	43.5	37.3	47.5 95% CI [39.7, 55.2]	68.0	45.4	23.0	51.1	22 (18%)	1.4	25.0	498K
14	GPT-5.2-CodexCodex CLIMedium effortProprietary	64.6	46.8	35.5	46.5 95% CI [38.6, 54.5]	66.9	47.6	24.1	49.3	17 (14%)	1.6	N/A	500K
15	o4-miniMini-SWE-AgentMedium effortProprietary	63.1	35.4	28.2	39.3 95% CI [32.1, 46.8]	60.5	32.1	16.7	34.7	30 (25%)	1.8	29.9	498K
16	o4-miniCodex CLIMedium effortProprietary	55.5	36.7	23.6	34.4 95% CI [27.3, 41.9]	56.1	36.2	18.1	33.4	27 (22%)	1.7	N/A	331K
17	o4-miniOpenCodeMedium effortProprietary	54.2	30.5	23.5	33.5 95% CI [26.6, 40.7]	58.1	34.5	20.6	38.4	25 (21%)	1.4	13.5	280K

Coverage, precision, and flaky rates are percentages. A flaky bug has a block-coverage spread above 0.2 across its three attempts. Tokens per run count input and output, with cached input counted once. N/A means the harness does not report calls, not zero. Equal values share a rank. Results come from full runs only: all 120 incidents, three attempts each.