HOW TO READ AN AI BENCHMARK
Compare the experiment before the model
A leaderboard can help narrow a search. It cannot tell you which system will work best for a different task, environment or budget. Start by deciding what work you need the model to perform.
Choose a relevant task suite
A code-editing evaluation and a terminal-agent evaluation ask different questions. Read the publisher’s task definition, scoring method and limits before sorting the result table.
Keep the task cohort fixed
A run on fewer or different tasks is a different comparison. Choose one suite and cohort in the benchmark desk before adding runs. The page’s rank applies to that selection.
Read the whole setup
A model is one part of an agent system. Tools, prompts, agent version, editing format, reasoning effort and time budget can affect the outcome. A familiar model name is not evidence that two runs used the same setup.
Inspect cost and uncertainty
Reported spend is the evaluation’s total in USD, where available. It is not a current API price quote or the predicted cost of your application. Reported intervals or variation describe the publisher’s experiment; absent uncertainty should remain absent.
Check the date and original record
A source can refresh today while containing historical runs. Follow the source-run link and distinguish the evaluation date from the last import check.
Make a shortlist, then test your task
Keep several candidates whose trade-offs fit the work. Evaluate them on authorized examples with independent acceptance checks, including failure cases and human review effort.
A worked reading exercise
Illustrative example: two systems score 70% and 73% on the same task cohort, but use different agents and evaluation budgets. The higher score supports a statement about that reported setup. It does not isolate the model’s contribution or prove that the difference will survive on your own workflow.
Useful publisher references
Read the evaluation details at Aider and Terminal-Bench. Source links on each imported run take you closer to the actual record.