Agent payments to forbidden recipients fell from 105 to zero on 68,970 matched runs

single source· 1 articles · confidence: medium · first seen 2026-09-17 20:00 UTC

What this means for you

If you build agents that move money, the number to take away is 25,370: the pre-payment check let those payments through and still returned zero forbidden recipients, so it is not blanket refusal. Evaluations, scoring code and analysis script are released for re-runs. One caveat: a 0.38% per-session upper bound on the zero.

The APort Vault benchmark, posted as a preprint on 17 September, replays 4,371 attacks written by humans against a live payment agent during a public capture-the-flag event, across 14 models from 8 labs: 225,964 evaluations. Transfers to recipients the passport did not permit fell from 140 of 76,842 with the model alone to zero of 69,297 behind a deterministic pre-action check — an allowlist enforced before the payment call runs, implementing the Open Agent Passport spec. Not by blanket refusal: 25,370 payments executed. Results are self-reported, not replicated.

Key facts

  • ·The benchmark replays 4,371 attacks written by humans against a live payment agent during a public capture-the-flag event, across 14 models from 8 labs and five policy configurations. source
  • ·225,964 evaluations completed; the dataset, level passports, scoring code and analysis script are released at huggingface.co/datasets/aporthq/vault-benchmark-v1. source
  • ·Transfers to recipients the passport did not permit numbered 140 of 76,842 with the model alone and 0 of 69,297 behind the pre-action check; on 68,970 matched model, prompt and track triples the figures were 105 against 0. source
  • ·The zero spans 790 source sessions, giving a per-session upper bound of 0.38%. source
  • ·25,370 payments executed behind the check, while the policy denied 187 of the 25,640 transfer calls it evaluated, 148 of them for a forbidden recipient. source
  • ·Request rates varied far more across configurations than across models: 10.9% of model-alone evaluations at Level 1, 3.0% at Level 2, 0.1% at Level 3, 79.4% at Level 4. source

What the sources say

Sources

The original reporting. Follow these — they did the work.

← the wire