PUBLISHED BENCHMARK RESULTS

Aider Code Editing

Name: Aider Code Editing published results
Creator: Aider
License: https://www.apache.org/licenses/LICENSE-2.0

Historical code editing runs using Aider. Task counts and editing formats vary between runs.

These are publisher-reported model and agent runs. Agent version, reasoning effort or editing format, environment and source dates may differ. Task counts do not prove identical evaluation conditions. Compare setups before drawing conclusions.

Reported costs are evaluation totals in USD, not current API prices. Missing results are not scores of zero. No score is a universal recommendation.

Historical coverage: this source contains older Aider evaluations. Refreshing the source does not make the underlying results current.

Latest source result: 2024-12-21 · Source check: 2026-10-08 · Source checked successfully

Benchmark publisher ↗ · Apache-2.0 ↗ · Full methodology and provenance

133 tasks

93 runs · Pass rate after 2 attempts. Ranks apply within this task cohort.

Filtered rank 1
claude-3-5-sonnet-20241022
Aider · 0.59.2.devSource run ↗
diff · 133 tasks
84.2%
2024-10-22Compare
Filtered rank 2
o1
Aider · 0.69.2.devSource run ↗
diff · 133 tasks
84.2%
2024-12-18Compare
Filtered rank 3
gemini-exp-1206 (whole)
Aider · 0.67.1.devSource run ↗
whole · 133 tasks
80.5%
2024-12-08Compare
Filtered rank 4
o1-preview
Aider · 0.56.1.devSource run ↗
diff · 133 tasks
79.7%
2024-09-21Compare
Filtered rank 5
claude-3.5-sonnet-20240620
Aider · 0.42.1-devSource run ↗
diff · 133 tasks
77.4%
2024-07-04Compare
Filtered rank 6
claude-3-5-haiku-20241022
Aider · 0.61.1.devSource run ↗
diff · 133 tasks
75.2%
2024-11-04Compare
Filtered rank 7
DeepSeek Coder V2 0724
Aider · 0.45.2-devSource run ↗
diff · 133 tasks
72.9%
2024-07-24Compare
Filtered rank 8
gpt-4o-2024-05-13
Aider · 0.34.1-devSource run ↗
diff · 133 tasks
72.9%
2024-05-13Compare
Filtered rank 9
ollama/qwen2.5-coder:32b
Aider · 0.65.2.devSource run ↗
whole · 133 tasks
72.9%
2024-11-28Compare
Filtered rank 10
DeepSeek V2.5
Aider · 0.55.1.devSource run ↗
diff · 133 tasks
72.2%
2024-09-05Compare
Filtered rank 11
DeepSeek-V2.5-1210
Aider · 0.67.1.devSource run ↗
diff · 133 tasks
72.2%
2024-12-10Compare
Filtered rank 12
openai/chatgpt-4o-latest
Aider · 0.58.2.devSource run ↗
diff · 133 tasks
72.2%
2024-10-04Compare
Filtered rank 13
Qwen2.5-Coder-32B-Instruct
Aider · 0.59.2.devSource run ↗
diff · 133 tasks
71.4%
2024-11-09Compare
Filtered rank 14
gpt-4o-2024-08-06
Aider · 0.48.1-devSource run ↗
diff · 133 tasks
71.4%
2024-08-06Compare
Filtered rank 15
gpt-4o-2024-11-20
Aider · 0.63.3.devSource run ↗
diff · 133 tasks
71.4%
2024-11-20Compare
Filtered rank 16
o1-mini (whole)
Aider · 0.56.1.devSource run ↗
whole · 133 tasks
70.7%
2024-09-12Compare
Filtered rank 17
DeepSeek Chat V2 0628
Aider · 0.45.2-devSource run ↗
diff · 133 tasks
69.9%
2024-07-19Compare
Filtered rank 18
gemini-2.0-flash-exp
Aider · 0.68.1.devSource run ↗
diff · 133 tasks
69.9%
2024-12-11Compare
Filtered rank 19
Qwen2.5-Coder-14B-Instruct
Aider · 0.59.2.devSource run ↗
whole · 133 tasks
69.2%
2024-10-29Compare
Filtered rank 20
gemini-exp-1206 (diff)
Aider · 0.67.1.devSource run ↗
diff · 133 tasks
69.2%
2024-12-06Compare
Filtered rank 21
claude-3-opus-20240229
Aider · 0.30.2-devSource run ↗
diff · 133 tasks
68.4%
2024-05-01Compare
Filtered rank 22
gpt-4-0613
Aider · 0.18.2-devSource run ↗
diff · 133 tasks
67.7%
2023-12-16Compare
Filtered rank 23
Dracarys2-72B-Instruct
Aider · 0.59.2.devSource run ↗
whole · 133 tasks
66.9%
2024-10-05Compare
Filtered rank 24
gemini-1.5-pro-exp-0827
Aider · 0.53.1-devSource run ↗
diff-fenced · 133 tasks
66.9%
2024-08-28Compare
Filtered rank 25
gpt-4-0125-preview
Aider · 0.22.1-devSource run ↗
udiff · 133 tasks
66.2%
2024-01-25Compare
Filtered rank 26
gpt-4-0314
Aider · 0.31.2-devSource run ↗
diff · 133 tasks
66.2%
2024-05-04Compare
Filtered rank 27
llama-3.1-405b-instruct (whole)
Aider · 0.45.2-devSource run ↗
whole · 133 tasks
66.2%
2024-07-24Compare
Filtered rank 28
Mistral Large (2411)
Aider · 0.63.3.devSource run ↗
diff · 133 tasks
65.4%
2024-11-20Compare
Filtered rank 29
gemini-1.5-pro-002
Aider · 0.57.2.devSource run ↗
diff-fenced · 133 tasks
65.4%
2024-09-24Compare
Filtered rank 30
gpt-4-1106-preview
Aider · 0.33.1-devSource run ↗
udiff · 133 tasks
65.4%
2024-05-08Compare
Filtered rank 31
openrouter/qwen/qwen-2.5-coder-32b-instruct
Aider · 0.63.3.devSource run ↗
diff · 133 tasks
65.4%
2024-11-20Compare
Filtered rank 32
qwen-2.5-72b-instruct (bf16)
Aider · 0.56.1.devSource run ↗
diff · 133 tasks
65.4%
2024-09-20Compare
Filtered rank 33
yi-lightning
Aider · 0.67.1.devSource run ↗
whole · 133 tasks
65.4%
2024-12-11Compare
Filtered rank 34
gpt-4-turbo-2024-04-09 (udiff)
Aider · 0.30.2-devSource run ↗
udiff · 133 tasks
63.9%
2024-05-01Compare
Filtered rank 35
llama-3.1-405b-instruct (diff)
Aider · 0.45.2-devSource run ↗
diff · 133 tasks
63.9%
2024-07-23Compare
Filtered rank 36
nousresearch/hermes-3-llama-3.1-405b
Aider · 0.54.8-devSource run ↗
whole · 133 tasks
63.9%
2024-08-30Compare
Filtered rank 37
ollama/Qwen2.5.1-Coder-7B-Instruct-GGUF:Q8_0-32k
Aider · 0.59.2.devSource run ↗
whole · 133 tasks
63.9%
2024-11-07Compare
Filtered rank 38
ollama/qwen2.5-coder:14b
Aider · 0.65.2.devSource run ↗
whole · 133 tasks
61.7%
2024-11-27Compare
Filtered rank 39
gemini-exp-1114
Aider · 0.63.2.devSource run ↗
diff · 133 tasks
60.9%
2024-11-15Compare
Filtered rank 40
Mistral Large 2 (2407)
Aider · 0.45.2-devSource run ↗
whole · 133 tasks
60.2%
2024-07-24Compare
Filtered rank 41
llama-3.3-70b-instruct
Aider · 0.67.1.devSource run ↗
diff · 133 tasks
59.4%
2024-12-06Compare
Filtered rank 42
Grok-2
Aider · 0.59.2.devSource run ↗
whole · 133 tasks
58.6%
2024-10-13Compare
Filtered rank 43
llama-3.1-70b-instruct
Aider · 0.45.2-devSource run ↗
whole · 133 tasks
58.6%
2024-07-25Compare
Filtered rank 44
ollama/qwen2.5:32b-instruct-q8_0
Aider · 0.56.1.devSource run ↗
whole · 133 tasks
58.6%
2024-09-20Compare
Filtered rank 45
Qwen2.5-Coder-7B-Instruct
Aider · 0.59.2.devSource run ↗
whole · 133 tasks
57.9%
2024-10-16Compare
Filtered rank 46
gemini-exp-1121
Aider · 0.63.3.devSource run ↗
diff · 133 tasks
57.9%
2024-11-21Compare
Filtered rank 47
gpt-3.5-turbo-0301
Aider · 0.16.4-devSource run ↗
whole · 133 tasks
57.9%
2023-11-06Compare
Filtered rank 48
gemini-1.5-pro-001
Aider · 0.31.2-devSource run ↗
diff-fenced · 133 tasks
57.1%
2024-05-03Compare
Filtered rank 49
Qwen2 72B Instruct
Aider · 0.37.1-devSource run ↗
whole · 133 tasks
55.6%
2024-06-08Compare
Filtered rank 50
gpt-4o-mini
Aider · 0.44.1-devSource run ↗
whole · 133 tasks
55.6%
2024-07-18Compare
Filtered rank 51
Grok-2-mini
Aider · 0.59.2.devSource run ↗
whole · 133 tasks
54.9%
2024-10-13Compare
Filtered rank 52
Llama-3.1-Nemotron-70B-Instruct-HF
Aider · 0.59.2.devSource run ↗
whole · 133 tasks
54.9%
2024-10-16Compare
Filtered rank 53
claude-3-sonnet-20240229
Aider · 0.25.1-devSource run ↗
whole · 133 tasks
54.9%
2024-03-06Compare
Filtered rank 54
Nova Pro
Aider · 0.66.1.devSource run ↗
whole · 133 tasks
54.1%
2024-12-04Compare
Filtered rank 55
Yi Coder 9B Chat
Aider · 0.54.13.devSource run ↗
whole · 133 tasks
54.1%
2024-09-04Compare
Filtered rank 56
ollama/qwen2.5:32b
Aider · 0.58.1.devSource run ↗
whole · 133 tasks
54.1%
2024-09-30Compare
Filtered rank 57
gemini-1.5-flash-exp-0827
Aider · 0.53.1-devSource run ↗
whole · 133 tasks
52.6%
2024-08-27Compare
Filtered rank 58
qwen2.5-coder:7b-instruct-q8_0
Aider · 0.56.0Source run ↗
whole · 133 tasks
51.9%
2024-09-19Compare
Filtered rank 59
codestral-2405
Aider · 0.37.1-devSource run ↗
whole · 133 tasks
51.1%
2024-06-09Compare
Filtered rank 60
gemini-1.5-flash-002 (0924)
Aider · 0.57.2.devSource run ↗
whole · 133 tasks
51.1%
2024-09-24Compare
Filtered rank 61
gpt-3.5-turbo-0125
Aider · 0.33.1-devSource run ↗
whole · 133 tasks
50.4%
2024-05-08Compare
Filtered rank 62
gpt-3.5-turbo-0613
Aider · 0.16.4-devSource run ↗
whole · 133 tasks
50.4%
2023-11-07Compare
Filtered rank 63
qwen2:72b-instruct-q8_0
Aider · 0.37.1-devSource run ↗
whole · 133 tasks
49.6%
2024-06-08Compare
Filtered rank 64
Codestral-22B-v0.1-Q4_K_M
Aider · 0.56.1.devSource run ↗
whole · 133 tasks
48.1%
2024-09-21Compare
Filtered rank 65
codestral:22b-v0.1-q8_0
Aider · 0.37.1-devSource run ↗
whole · 133 tasks
48.1%
2024-06-08Compare
Filtered rank 66
claude-3-haiku-20240307
Aider · 0.41.1-devSource run ↗
whole · 133 tasks
47.4%
2024-07-01Compare
Filtered rank 67
ollama/codestral
Aider · 0.57.2.devSource run ↗
whole · 133 tasks
45.9%
2024-09-28Compare
Filtered rank 68
yi-coder:9b-chat-q4_0
Aider · 0.54.13.devSource run ↗
whole · 133 tasks
45.1%
2024-09-04Compare
Filtered rank 69
WizardLM-2 8x22B
Aider · 0.31.2-devSource run ↗
whole · 133 tasks
44.4%
2024-05-07Compare
Filtered rank 70
gemini-1.5-flash-latest
Aider · 0.37.1-devSource run ↗
whole · 133 tasks
44.4%
2024-06-08Compare
Filtered rank 71
ollama/yi-coder:9b-chat-fp16
Aider · 0.58.1.devSource run ↗
whole · 133 tasks
43.6%
2024-10-01Compare
Filtered rank 72
Reflection-70B
Aider · 0.55.1.devSource run ↗
whole · 133 tasks
42.1%
2024-09-06Compare
Filtered rank 73
Qwen2.5-Coder-3B-Instruct
Aider · 0.59.2.devSource run ↗
whole · 133 tasks
39.1%
2024-11-04Compare
Filtered rank 74
Command R (08-24)
Aider · 0.56.1.devSource run ↗
whole · 133 tasks
38.3%
2024-09-11Compare
Filtered rank 75
Command R+ (08-24)
Aider · 0.56.1.devSource run ↗
whole · 133 tasks
38.3%
2024-09-11Compare
Filtered rank 76
gemini-1.5-flash-8b-exp-0827
Aider · 0.53.1-devSource run ↗
whole · 133 tasks
38.3%
2024-08-27Compare
Filtered rank 77
gemini-1.5-flash-8b-exp-0924
Aider · 0.57.2.devSource run ↗
whole · 133 tasks
38.3%
2024-09-24Compare
Filtered rank 78
ollama/mistral-small
Aider · 0.58.1.devSource run ↗
whole · 133 tasks
38.3%
2024-10-01Compare
Filtered rank 79
llama-3.1-8b-instruct
Aider · 0.45.2-devSource run ↗
whole · 133 tasks
37.6%
2024-07-25Compare
Filtered rank 80
qwen1.5-110b-chat
Aider · 0.31.2-devSource run ↗
whole · 133 tasks
37.6%
2024-05-07Compare
Filtered rank 81
gemma2:27b-instruct-q8_0
Aider · 0.43.0Source run ↗
whole · 133 tasks
36.1%
2024-07-09Compare
Filtered rank 82
codeqwen:7b-chat-v1.5-q8_0
Aider · 0.37.1-devSource run ↗
whole · 133 tasks
34.6%
2024-06-08Compare
Filtered rank 83
ollama/mistral-nemo:12b-instruct-2407-q4_K_M
Aider · 0.58.1.devSource run ↗
whole · 133 tasks
33.1%
2024-09-30Compare
Filtered rank 84
ollama/codegeex4
Aider · 0.57.2.devSource run ↗
whole · 133 tasks
32.3%
2024-09-29Compare
Filtered rank 85
Qwen2.5-Coder-1.5B-Instruct
Aider · 0.59.2.devSource run ↗
whole · 133 tasks
31.6%
2024-11-11Compare
Filtered rank 86
command-r-plus
Aider · 0.31.2-devSource run ↗
whole · 133 tasks
31.6%
2024-05-06Compare
Filtered rank 87
ollama/hermes3:8b-llama3.1-fp16
Aider · 0.58.1.devSource run ↗
whole · 133 tasks
30.1%
2024-09-30Compare
Filtered rank 88
ollama/wojtek/opencodeinterpreter:6.7b
Aider · 0.58.1.devSource run ↗
whole · 133 tasks
30.1%
2024-09-30Compare
Filtered rank 89
ollama/llama3.2:3b-instruct-fp16
Aider · 0.58.1.devSource run ↗
whole · 133 tasks
26.3%
2024-09-30Compare
Filtered rank 90
ollama/tulu3
Aider · 0.65.2.devSource run ↗
whole · 133 tasks
26.3%
2024-11-28Compare
Filtered rank 91
ollama/hermes3
Aider · 0.58.1.devSource run ↗
whole · 133 tasks
22.6%
2024-10-01Compare
Filtered rank 92
ollama/granite3-dense:8b
Aider · 0.65.2.devSource run ↗
whole · 133 tasks
20.3%
2024-11-28Compare
Filtered rank 93
Qwen2.5-Coder-0.5B-Instruct
Aider · 0.59.2.devSource run ↗
whole · 133 tasks
14.3%
2024-10-29Compare

132 tasks

2 runs · Pass rate after 2 attempts. Ranks apply within this task cohort.

Filtered rank 1
gpt-3.5-turbo-1106
Aider · 0.30.2-devSource run ↗
whole · 132 tasks
56.1%
2024-04-30Compare
Filtered rank 2
llama3-70b-8192
Aider · 0.31.2-devSource run ↗
diff · 132 tasks
49.2%
2024-05-03Compare

225 tasks

1 runs · Pass rate after 2 attempts. Ranks apply within this task cohort.

Filtered rank 1
o1-mini-2024-09-12
Aider · 0.69.2.devSource run ↗
whole · 225 tasks
27.1%
2024-12-21Compare

36 tasks

1 runs · Pass rate after 2 attempts. Ranks apply within this task cohort.

Filtered rank 1
o1-mini
Aider · 0.56.1.devSource run ↗
diff · 36 tasks
61.1%
2024-09-21Compare

33 tasks

1 runs · Pass rate after 2 attempts. Ranks apply within this task cohort.

Filtered rank 1
gpt-4-turbo-2024-04-09 (diff)
Aider · 0.28.1-devSource run ↗
diff · 33 tasks
57.6%
2024-04-12Compare

Aider Code Editing

133 tasks

claude-3-5-sonnet-20241022

o1

gemini-exp-1206 (whole)

o1-preview

claude-3.5-sonnet-20240620

claude-3-5-haiku-20241022

DeepSeek Coder V2 0724

gpt-4o-2024-05-13

ollama/qwen2.5-coder:32b

DeepSeek V2.5

DeepSeek-V2.5-1210

openai/chatgpt-4o-latest

Qwen2.5-Coder-32B-Instruct

gpt-4o-2024-08-06

gpt-4o-2024-11-20

o1-mini (whole)

DeepSeek Chat V2 0628

gemini-2.0-flash-exp

Qwen2.5-Coder-14B-Instruct

gemini-exp-1206 (diff)

claude-3-opus-20240229

gpt-4-0613

Dracarys2-72B-Instruct

gemini-1.5-pro-exp-0827

gpt-4-0125-preview

gpt-4-0314

llama-3.1-405b-instruct (whole)

Mistral Large (2411)

gemini-1.5-pro-002

gpt-4-1106-preview

openrouter/qwen/qwen-2.5-coder-32b-instruct

qwen-2.5-72b-instruct (bf16)

yi-lightning

gpt-4-turbo-2024-04-09 (udiff)

llama-3.1-405b-instruct (diff)

nousresearch/hermes-3-llama-3.1-405b

ollama/Qwen2.5.1-Coder-7B-Instruct-GGUF:Q8_0-32k

ollama/qwen2.5-coder:14b

gemini-exp-1114

Mistral Large 2 (2407)

llama-3.3-70b-instruct

Grok-2

llama-3.1-70b-instruct

ollama/qwen2.5:32b-instruct-q8_0

Qwen2.5-Coder-7B-Instruct

gemini-exp-1121

gpt-3.5-turbo-0301

gemini-1.5-pro-001

Qwen2 72B Instruct

gpt-4o-mini

Grok-2-mini

Llama-3.1-Nemotron-70B-Instruct-HF

claude-3-sonnet-20240229

Nova Pro

Yi Coder 9B Chat

ollama/qwen2.5:32b

gemini-1.5-flash-exp-0827

qwen2.5-coder:7b-instruct-q8_0

codestral-2405

gemini-1.5-flash-002 (0924)

gpt-3.5-turbo-0125

gpt-3.5-turbo-0613

qwen2:72b-instruct-q8_0

Codestral-22B-v0.1-Q4_K_M

codestral:22b-v0.1-q8_0

claude-3-haiku-20240307

ollama/codestral

yi-coder:9b-chat-q4_0

WizardLM-2 8x22B

gemini-1.5-flash-latest

ollama/yi-coder:9b-chat-fp16

Reflection-70B

Qwen2.5-Coder-3B-Instruct

Command R (08-24)

Command R+ (08-24)

gemini-1.5-flash-8b-exp-0827

gemini-1.5-flash-8b-exp-0924

ollama/mistral-small