Strongest agent passes 17.5% of new FreeCAD benchmark tasks

single source· 1 articles · confidence: high · first seen 2026-09-13 20:00 UTC

What this means for you

If you build computer-use agents, CADWorld is a free target to measure against at cad-world.github.io, and its failure mode is the useful part: passing a GUI task is not the same as leaving behind a file a downstream engineer can use. Nothing to buy here — no model, API or pricing.

Researchers released CADWorld, a benchmark for computer-use agents — systems that drive a desktop through screenshots and clicks rather than an API — working in FreeCAD, the open-source mechanical design program. It holds 200 tasks across 11 categories, including sketching, part modelling, assembly, CAM (computer-aided manufacturing) and technical drawing. Success is judged by task-specific checks run against the saved FreeCAD file: geometry, parametric structure, constraints and simulation results. Of seven agents tested, the best passed 17.5%; an expert reference passed 87.0%. Weaker agents failed before saving a valid file; stronger ones failed on structural and construction requirements.

Key facts

  • ·CADWorld contains 200 tasks in FreeCAD spanning 11 mechanical-CAD workflow categories, including sketching, part modelling, assembly, CAM, FEM, measurement, mesh processing and technical drawing. source
  • ·Agents operate through screenshots and GUI actions, and success is determined by task-specific executable checks over saved FreeCAD artifacts covering geometric properties, parametric structure, constraints, manufacturing state and simulation results. source
  • ·Across seven current agents on the full benchmark, the strongest achieves 17.5% success, against an 87.0% expert reference pass. source
  • ·The paper reports that weaker agents often fail before producing a valid artifact, while stronger agents increasingly fail on structural, geometric and construction-process requirements. source
  • ·The benchmark is published as an arXiv paper dated 13 September 2026, with the project hosted at cad-world.github.io. source

What the sources say

Sources

The original reporting. Follow these — they did the work.

← the wire