PUBLISHED BENCHMARK RESULTS
Aider Polyglot
Code editing across multiple programming languages using Aider. Historical runs; filter by task count before comparing.
These are publisher-reported model and agent runs. Agent version, reasoning effort or editing format, environment and source dates may differ. Task counts do not prove identical evaluation conditions. Compare setups before drawing conclusions.
Reported costs are evaluation totals in USD, not current API prices. Missing results are not scores of zero. No score is a universal recommendation.
Historical coverage: this source contains older Aider evaluations. Refreshing the source does not make the underlying results current.
Benchmark publisher ↗ · Apache-2.0 ↗ · Full methodology and provenance
225 tasks
- Filtered rank 1
gpt-5 (high)
Source run ↗diff88.0%Compare - Filtered rank 2
gpt-5 (medium)
Source run ↗diff86.7%Compare - Filtered rank 3
o3-pro (high)
Source run ↗diff84.9%Compare - Filtered rank 4
gemini-2.5-pro-preview-06-05 (32k think)
Source run ↗diff-fenced83.1%Compare - Filtered rank 5
gpt-5 (low)
Source run ↗diff81.3%Compare - Filtered rank 6
o3 (high)
Source run ↗diff81.3%Compare - Filtered rank 7
grok-4 (high)
Source run ↗diff79.6%Compare - Filtered rank 8
gemini-2.5-pro-preview-06-05 (default think)
Source run ↗diff-fenced79.1%Compare - Filtered rank 9
Gemini 2.5 Pro Preview 05-06
Source run ↗diff-fenced76.9%Compare - Filtered rank 10diff76.9%Compare
- Filtered rank 11
DeepSeek-V3.2-Exp (Reasoner)
Source run ↗diff74.2%Compare - Filtered rank 12
Gemini 2.5 Pro Preview 03-25
Source run ↗diff-fenced72.9%Compare - Filtered rank 13
claude-opus-4-20250514 (32k thinking)
Source run ↗diff72.0%Compare - Filtered rank 14
o4-mini (high)
Source run ↗diff72.0%Compare - Filtered rank 15
claude-opus-4-20250514 (no think)
Source run ↗diff70.7%Compare - Filtered rank 16
DeepSeek-V3.2-Exp (Chat)
Source run ↗diff70.2%Compare - Filtered rank 17
claude-3-7-sonnet-20250219 (32k thinking tokens)
Source run ↗diff64.9%Compare - Filtered rank 18
DeepSeek R1 + claude-3-5-sonnet-20241022
Source run ↗architect64.0%Compare - Filtered rank 19
claude-sonnet-4-20250514 (32k thinking)
Source run ↗diff61.3%Compare - Filtered rank 20
claude-3-7-sonnet-20250219 (no thinking)
Source run ↗diff60.4%Compare - Filtered rank 21
Qwen3 235B A22B diff, no think, Alibaba API
Source run ↗diff59.6%Compare - Filtered rank 22
Kimi K2
Source run ↗diff59.1%Compare - Filtered rank 23
DeepSeek R1
Source run ↗diff56.9%Compare - Filtered rank 24
claude-sonnet-4-20250514 (no thinking)
Source run ↗diff56.4%Compare - Filtered rank 25
DeepSeek V3 (0324)
Source run ↗diff55.1%Compare - Filtered rank 26
gemini-2.5-flash-preview-05-20 (24k think)
Source run ↗diff55.1%Compare - Filtered rank 27
Quasar Alpha
Source run ↗diff54.7%Compare - Filtered rank 28
o3-mini (medium)
Source run ↗diff53.8%Compare - Filtered rank 29
Grok 3 Beta
Source run ↗diff53.3%Compare - Filtered rank 30
Optimus Alpha
Source run ↗diff52.9%Compare - Filtered rank 31
gpt-4.1
Source run ↗diff52.4%Compare - Filtered rank 32
claude-3-5-sonnet-20241022
Source run ↗diff51.6%Compare - Filtered rank 33
Grok 3 Mini Beta (high)
Source run ↗whole49.3%Compare - Filtered rank 34
DeepSeek Chat V3 (prev)
Source run ↗diff48.4%Compare - Filtered rank 35
gemini-2.5-flash-preview-04-17 (default)
Source run ↗diff47.1%Compare - Filtered rank 36
chatgpt-4o-latest (2025-03-29)
Source run ↗diff45.3%Compare - Filtered rank 37
gemini-2.5-flash-preview-05-20 (no think)
Source run ↗diff44.0%Compare - Filtered rank 38
gpt-oss-120b (high)
Source run ↗diff41.8%Compare - Filtered rank 39
Qwen3 32B
Source run ↗diff40.0%Compare - Filtered rank 40
gemini-exp-1206
Source run ↗whole38.2%Compare - Filtered rank 41
Gemini 2.0 Pro exp-02-05
Source run ↗whole35.6%Compare - Filtered rank 42
Grok 3 Mini Beta (low)
Source run ↗whole34.7%Compare - Filtered rank 43
o1-mini-2024-09-12
Source run ↗whole32.9%Compare - Filtered rank 44
gpt-4.1-mini
Source run ↗diff32.4%Compare - Filtered rank 45
claude-3-5-haiku-20241022
Source run ↗diff28.0%Compare - Filtered rank 46
QwQ-32B + Qwen 2.5 Coder Instruct
Source run ↗architect26.2%Compare - Filtered rank 47
gpt-4o-2024-08-06
Source run ↗diff23.1%Compare - Filtered rank 48
gemini-2.0-flash-exp
Source run ↗whole22.2%Compare - Filtered rank 49
qwen-max-2025-01-25
Source run ↗diff21.8%Compare - Filtered rank 50
QwQ-32B
Source run ↗diff20.9%Compare - Filtered rank 51
gemini-2.0-flash-thinking-exp-01-21
Source run ↗diff18.2%Compare - Filtered rank 52
gpt-4o-2024-11-20
Source run ↗diff18.2%Compare - Filtered rank 53
DeepSeek Chat V2.5
Source run ↗diff17.8%Compare - Filtered rank 54
Qwen2.5-Coder-32B-Instruct
Source run ↗whole16.4%Compare - Filtered rank 55
Llama 4 Maverick
Source run ↗whole15.6%Compare - Filtered rank 56
yi-lightning
Source run ↗whole12.9%Compare - Filtered rank 57
command-a-03-2025-quality
Source run ↗whole12.0%Compare - Filtered rank 58
Codestral 25.01
Source run ↗whole11.1%Compare - Filtered rank 59
openhands-lm-32b-v0.1
Source run ↗whole10.2%Compare - Filtered rank 60
gpt-4.1-nano
Source run ↗whole8.9%Compare - Filtered rank 61
Qwen2.5-Coder-32B-Instruct
Source run ↗diff8.0%Compare - Filtered rank 62
gemma-3-27b-it
Source run ↗whole4.9%Compare - Filtered rank 63
gpt-4o-mini-2024-07-18
Source run ↗whole3.6%Compare
224 tasks
- Filtered rank 1
o3 (high) + gpt-4.1
Source run ↗architect78.2%Compare - Filtered rank 2
DeepSeek R1 (0528)
Source run ↗diff71.4%Compare - Filtered rank 3
o1-2024-12-17 (high)
Source run ↗diff61.7%Compare - Filtered rank 4
o3-mini (high)
Source run ↗diff60.4%Compare - Filtered rank 5
gpt-4.5-preview
Source run ↗diff44.9%Compare
223 tasks
- Filtered rank 1
chatgpt-4o-latest (2025-02-15)
Source run ↗diff27.1%Compare