Fraud-call detectors fall to 0.65 F1 when test calls resemble scams

single source· 1 articles · confidence: medium · first seen 2026-09-16 20:00 UTC

What this means for you

If your fraud classifier scores near-perfect, check what it was scored against. This benchmark shows the same models falling to 0.65-0.68 F1 once the legitimate calls in the test set sound like the scams. Adding near-domain negatives to your own evaluation is the actionable step; the construction code and scripts are public.

TeleAntiFraud 2.0, an audio benchmark for detecting scam phone calls, reports that three classifiers scored perfect macro-averaged F1 against unrelated negative examples but fell to 0.65-0.68 once the non-fraud calls in the test set resembled real service conversations. The benchmark builds Chinese-language call audio from online fraud-case write-ups and freezes each monthly test set — 900 calls, 600 fraud and 300 near-domain legitimate — so new scam patterns are added without overwriting old sets. Audio and speech-recognition-plus-LLM runs showed models exploiting class balance and collapsing across snapshots. Code and scripts are public; no evaluation date is given for the reported scores.

Key facts

  • ·Each frozen TeleAntiFraud 2.0 evaluation set contains 900 Chinese calls: 600 fraud and 300 near-domain non-fraud cases. source
  • ·In controlled text experiments, three classifiers reached perfect macro-averaged F1 against unrelated or ordinary negatives but fell to 0.65-0.68 with near-domain sibling negatives. source
  • ·Full-set audio and ASR-plus-LLM evaluations revealed class-prior shortcuts, prediction collapse and snapshot sensitivity. source
  • ·The construction code, evaluation scripts, manifests and documentation are released in an anonymised repository. source
  • ·The paper is an arXiv preprint, 2609.18748, posted on 16 September 2026. source

What the sources say

Sources

The original reporting. Follow these — they did the work.

← the wire