findeverythingaiInteractive benchmark desk

PUBLISHED BENCHMARK RESULTS

Aider Refactoring

Historical refactoring runs using Aider. Scores use one attempt, unlike the other Aider suites.

These are publisher-reported model and agent runs. Agent version, reasoning effort or editing format, environment and source dates may differ. Task counts do not prove identical evaluation conditions. Compare setups before drawing conclusions.

Reported costs are evaluation totals in USD, not current API prices. Missing results are not scores of zero. No score is a universal recommendation.

Historical coverage: this source contains older Aider evaluations. Refreshing the source does not make the underlying results current.

Latest source result: 2024-10-22 · Source check: 2026-10-08 · Source checked successfully

Benchmark publisher ↗ · Apache-2.0 ↗ · Full methodology and provenance

89 tasks

11 runs · First-attempt pass rate. Ranks apply within this task cohort.

  1. Filtered rank 1

    claude-3-5-sonnet-20241022

    Aider · 0.60.1.devSource run ↗
    diff · 89 tasks
    92.1%
    Compare
  2. Filtered rank 2

    o1-preview

    Aider · 0.60.1.devSource run ↗
    diff · 89 tasks
    75.3%
    Compare
  3. Filtered rank 3

    claude-3.5-sonnet-20240620

    Aider · 0.40.7-devSource run ↗
    diff · 89 tasks
    64.0%
    Compare
  4. Filtered rank 4

    gpt-4o

    Aider · 0.34.1-devSource run ↗
    diff · 89 tasks
    62.9%
    Compare
  5. Filtered rank 5

    gpt-4-1106-preview

    Aider · 0.33.1-devSource run ↗
    udiff · 89 tasks
    50.6%
    Compare
  6. Filtered rank 6

    gemini/gemini-1.5-pro-latest

    Aider · 0.31.2-devSource run ↗
    diff-fenced · 89 tasks
    49.4%
    Compare
  7. Filtered rank 7

    gpt-4o-2024-08-06

    Aider · 0.48.1-devSource run ↗
    diff · 89 tasks
    49.4%
    Compare
  8. Filtered rank 8

    o1-mini

    Aider · 0.60.1.devSource run ↗
    diff · 89 tasks
    44.9%
    Compare
  9. Filtered rank 9

    gpt-4-0125-preview

    Aider · 0.33.1-devSource run ↗
    udiff · 89 tasks
    33.7%
    Compare
  10. Filtered rank 10

    DeepSeek Coder V2 0724 (deprecated)

    Aider · 0.45.2-devSource run ↗
    diff · 89 tasks
    32.6%
    Compare
  11. Filtered rank 11

    DeepSeek Chat V2.5

    Aider · 0.55.1.devSource run ↗
    diff · 89 tasks
    31.5%
    Compare

88 tasks

2 runs · First-attempt pass rate. Ranks apply within this task cohort.

  1. Filtered rank 1

    gpt-4-turbo-2024-04-09 (udiff)

    Aider · 0.27.1-devSource run ↗
    udiff · 88 tasks
    34.1%
    Compare
  2. Filtered rank 2

    gpt-4-turbo-2024-04-09 (diff)

    Aider · 0.28.1-devSource run ↗
    diff · 88 tasks
    21.4%
    Compare

83 tasks

1 runs · First-attempt pass rate. Ranks apply within this task cohort.

  1. Filtered rank 1

    claude-3-opus-20240229

    Aider · 0.31.2-devSource run ↗
    diff · 83 tasks
    72.3%
    Compare