Researchers release a 300-question exam on object permanence in video models

single source· 1 articles · confidence: medium · first seen 2026-09-22 20:00 UTC

What this means for you

Nothing to buy or migrate. If you work on video generation, there is now a 300-question object permanence exam, a 1.5m-sample training corpus and weights for a 16B model, all released. The ordering is the authors' own Elo study, so reproduce it against your own models before treating it as a ranking.

A preprint posted on 22 September describes WROP, a dataset and exam for testing whether video generation models keep track of objects that leave the frame or pass behind something. It contains 150 generated tasks in six categories, 10,000-plus samples each, a 1.5m-sample training corpus and a 300-question exam. The authors evaluated 14 video models — three reference-to-video, seven edit, four continuation — and report their own 16B PWM-WROP placed first among the continuation models and third overall in a blind pairwise Elo study, behind two reference-to-video models tied statistically. Data, exam, weights and training stack are released; the ranking is self-run, with no independent reproduction.

Key facts

  • ·WROP comprises 150 hand-designed, cognitive-science-inspired tasks divided into six categories source
  • ·The release includes a 1.5m-sample training corpus and a 300-question exam, with 10,000+ samples generated per task source
  • ·14 video models were evaluated: three reference-to-video, seven edit and four continuation models source
  • ·In a blind pairwise Elo study the authors' 16B model, PWM-WROP, ranked first among continuation models and third overall, behind two reference-to-video models tied statistically source
  • ·Data, exam, model answers, scores, weights and the PWM PyTorch training stack on AWS Trainium2 are released source
  • ·The paper is arXiv 2609.28654, posted 22 September 2026 source

What the sources say

Sources

The original reporting. Follow these — they did the work.

← the wire