“Measuring agent capabilities on open-ended AI R&D tasks” by Alex Jackson, Titus Buckworth, Jason R Brown, Francis Rhys Ward

LessWrong (30+ Karma) · Oct 27, 2026 · 36m

TL;DR. We measure agent capabilities on four long and open-ended research tasks. Frontier models have been improving with an overall doubling time of around one year since 2024 (based on an exponential fit). Agents perform better at engineering-heavy tasks compared to more conceptual tasks (between 7 and 16.8 months median doubling time), and model capabilities are improving slower on the latter.

Epistemic status: This analysis is based on a small set of tasks and we rely heavily on LLM-grading (though we think our LLM scoring is reasonably well-calibrated to expert human scores).

This work…