Robot policies pass task tests without using the instructions

single source· 1 articles · confidence: medium · first seen 2026-09-21 20:00 UTC

What this means for you

If you use task success rates to compare robot policies, ask whether those tasks left any choice. On scenes with a single valid action, the number may be measuring vision, not language. The code and dataset are public, so you can run the diagnostic before trusting a demo or a leaderboard position.

Robot policies can score highly on task tests without using the instructions they are given, according to a benchmark posted to arXiv on 21 September. The paper calls the cause low scene entropy: when a scene allows only one sensible action, language is redundant and vision alone suffices. RoboFollow builds scenes with several distinct task branches, then runs four test levels (L0–L3) that progressively perturb layout and wording. Across nine vision-language-action policies (image and instruction in, motor commands out), strong L0 scores did not carry over to L1–L3 under the authors' fine-tuning setup; four mitigations, including stronger vision backbones and classifier-free guidance, failed to close the gap.

Key facts

  • ·RoboFollow is a diagnostic benchmark for instruction following in embodied agents; its code is released at github.com/AutoLab-SAI-SJTU/RoboFollow and its dataset at huggingface.co/datasets/AutoLab-SAI-SJTU/robofollow-data. source
  • ·Each training scene in the benchmark supports multiple kinematically distinct task branches, so the scene alone cannot identify which task is required. source
  • ·Evaluation uses a four-level protocol, L0 to L3, progressively perturbing visual layout and semantics across spatial relations, attributes, trajectory constraints and logic. source
  • ·Nine policies, grouped by the paper as VLA and WAM, were evaluated; the paper reports that strong L0 performance did not reliably transfer to L1–L3 under its fine-tuning setup. source
  • ·Four mitigations were tested — stronger VLM backbones, QA co-training, LangForce and Classifier-Free Guidance — and the paper reports none closed the gap. source
  • ·The benchmark was posted to arXiv on 21 September 2026 as arXiv:2609.25636. source

What the sources say

Sources

The original reporting. Follow these — they did the work.

← the wire