Researchers evaluate 47 models on 5,000 hours of recorded gameplay

single source· 1 articles · confidence: medium · first seen 2026-09-20 20:00 UTC

What this means for you

Nothing to act on yet. The dataset, annotator and benchmark are announced but not released, and no scores are reported, so there is nothing to reproduce or compare a model against. If you are picking a gameplay evaluation for agents, this is one to track until the numbers land.

Researchers have built a three-part suite for grading how well AI models play video games. GameHorizon includes an automated pipeline that labels gameplay with instructions at several time scales, and a dataset of 5,000 hours of recordings from 21 games, captured by 100 expert human players, which the authors call the first large-scale dataset of its kind for big-budget titles. The benchmark has an offline track of thousands of standardised questions and an online track checking whether offline scores track real play. The team ran 47 models through over a million invocations and reports a hierarchy of task difficulty, but no scores. Release is promised, with no date.

Key facts

  • ·GameHorizon-Data contains 5,000 hours of recordings from 21 games, collected by 100 human expert players. source
  • ·The suite was used to evaluate 47 models across more than one million model invocations. source
  • ·The benchmark has an offline track of thousands of standardised questions in three primary tasks plus diagnostic variants, and a stepwise online track. source
  • ·The annotator is described as an automated pipeline for producing multi-horizon instructions. source
  • ·The authors say they will release the dataset, annotator and benchmark; no release date is given. source
  • ·The paper was posted to arXiv on 20 September 2026 as 2609.25001. source

What the sources say

Sources

The original reporting. Follow these — they did the work.

← the wire