Long-horizon tasks

Kwa, West and colleagues at METR measure this with a 50%-task-completion time horizon: the time humans typically take to complete tasks that a model completes with a 50% success rate¹. Humans with domain expertise were timed on RE-Bench, HCAST and 66 shorter tasks¹. On those tasks, the 10 July 2026 revision says a frontier model such as o3 has a 50% time horizon of about 110 minutes¹. Success on a task is negatively correlated with how long a human baseliner takes¹. The version submitted on 18 March 2025 reported about 50 minutes for Claude 3.7 Sonnet². The abstract still shown on the July 2026 arXiv record repeats that older 50-minute sentence, while the July 2026 HTML states the 110-minute figure¹³.

The July 2026 text says the frontier time horizon has doubled about every seven months since 2019, though the trend may have accelerated since 2024¹. The fitted 50% horizon doubled every 207 days, with a 95% interval of 166 to 240 days¹. Horizons at an 80% success rate are 4 to 6 times shorter¹. The tasks are not a perfect sample of real software work¹. The forecast that, within five years, many software tasks which now take a person a month could be automated is only if these results generalise¹.

Anthropic, on 29 September 2025, uses long-horizon tasks for a different measure: keeping coherence, context and goal-directed behaviour across actions whose tokens exceed the model's context window⁴. Its examples last from tens of minutes to multiple hours, such as a large codebase migration or a broad research project⁴. The techniques it names are compaction, which summarises a near-full conversation and starts again from the summary; notes stored outside the window; and sub-agents that return a short summary, often 1,000 to 2,000 tokens⁴. In Claude Code, compaction keeps architectural decisions, unresolved bugs and implementation details, drops redundant tool outputs, and continues with the summary plus the five most recently accessed files⁴.