Rule-breaking test tasks improve tool-calling agents by up to 42%

single source· 1 articles · confidence: medium · first seen 2026-09-20 20:00 UTC

What this means for you

Nothing to buy — this is a method, not a product, and no code release is mentioned. If you evaluate tool-calling agents, the approach is copyable today: derive the rules from the agent's spec, generate tasks that violate them, then fine-tune or fix the harness. The gains are reported only on one airline domain.

A paper on arXiv describes EdgeGen, a method that generates synthetic test tasks for tool-calling agents (models that call external functions) by extracting compliance rules from an agent's specification and writing database-grounded cases designed to break them. Combined with existing synthetic data, the pipeline needs no human annotation. Fine-tuning on that data improved mean task progress by 2% to 42% on tau2bench's airline domain, and some baseline methods degraded. For harness optimisation — the code around the model that runs the tool-calling loop — gains were 30% over a base harness and 10% over a human-curated one, for Gemma-4-e4b. The paper gives no evaluation date for the scores.

Key facts

  • ·EdgeGen extracts compliance rules from an agent's specification and generates database-grounded tasks designed to violate those rules. source
  • ·Fine-tuning on EdgeGen-generated data gave a 2% to 42% mean progress improvement on the tau2bench airline domain, with some baseline methods degrading. source
  • ·For Gemma-4-e4b, harness optimisation with EdgeGen improved mean progress 10% over a human-curated harness and 30% over the base harness. source
  • ·The pipeline is described as a fully automated closed loop requiring no human annotation. source
  • ·The paper was posted as arXiv 2609.24115 on 20 September 2026 and reports no evaluation date for the tau2bench scores. source

What the sources say

Sources

The original reporting. Follow these — they did the work.

← the wire