Benchmark finds agents can copy an app's look but not its behaviour

single source· 1 articles · confidence: medium · first seen 2026-09-17 20:00 UTC

What this means for you

If you build agents that operate software, the environments and tests are public, so you can measure yours on the same 250 tasks instead of inferring progress from demos. If you do not, nothing to act on today: this is a benchmark, not a model, and no product changes.

A preprint describes RecreationWorld, a testbed spanning Ubuntu, macOS, Windows, Android and the web in which an agent is handed running software and must rebuild it with no prescribed workflow. The same paper introduces RecreationBench: 250 tasks whose automated checks were run against the reference program and reviewed by people before scoring was frozen. GPT-6 Astra leads at 58.1% overall, but passes every programmatic test on only 2.8% of tasks. Agents reproduce static interface structure more reliably than interactions or computed output, and their rebuilt programs are smaller and more monolithic than the references. Environments, tests and benchmark are released.

Models in this story

Key facts

  • ·RecreationWorld spans five platforms: Ubuntu, macOS, Windows, Android and the web. source
  • ·RecreationBench contains 250 tasks, with programmatic and visual checks validated on the reference program and by human reviewers before the suite was frozen for automatic scoring. source
  • ·GPT-6 Astra leads RecreationBench at 58.1% overall. source
  • ·The leading model passes all programmatic tests on 2.8% of RecreationBench tasks. source
  • ·Agents reproduce static interface structure more reliably than interactions and computed outputs, and their generated applications are smaller and more monolithic than the references. source
  • ·The benchmark, environments and test suites are released. source

What the sources say

Sources

The original reporting. Follow these — they did the work.

← the wire