CallScreenBench
arXiv · 2026
A benchmark for small LMs that answer calls on their owner's behalf.
Problem
A phone secretary answers unknown calls for its owner. There is no cooperative task: the caller may be an adversary, and success means the owner would endorse how the call was handled.
My contribution
- Built the benchmark end to end as first author (equal contribution with Haoyuan Tang and Jamey Z. Liang): stateful callers across 22 call archetypes, 5 quality-measure groups, 3 guardedness diagnostics, 6 locally run models.
- Stress-tested the metrics with 6 scripted policies; estimated uncertainty with paired bootstrap intervals.
Approach
- Stateful callers
- Simulated callers keep their goal and state across turns.
- No composite score
- Call handling and owner notes are scored separately, each with a counter-metric and an uncertainty estimate.
- Degenerate agents
- Scripted bad policies show where a metric can be gamed.
Results
- 1.000 entity recall of a hang-up-and-echo agent with no real dialogue, which recall alone would hide
- 22 call archetypes, run against six 4-bit models (0.6–4B) from three families
- Scam-side true-positive rate alone rewards universal suspicion; model rankings change once false positives on legitimate callers count.
Limits
- Text-only decision layer; speech, latency, on-device execution and real users are out of scope.
Paper
CallScreenBench: Benchmarking Small Language Models as Phone Secretaries
arXiv:2608.01033 (2026)
* Equal contribution. † Corresponding author.
@misc{gan2026callscreenbenchbenchmarkingsmalllanguage,
title={CallScreenBench: Benchmarking Small Language Models as Phone Secretaries},
author={Jiaqi Gan and Haoyuan Tang and Jamey Z. Liang and Siying Chen and Ankit Raj and Kidus Zewde and Yuchen Zhou and Yuxin Zhang and Simiao Ren},
year={2026},
eprint={2608.01033},
archivePrefix={arXiv},
primaryClass={cs.CR},
url={https://arxiv.org/abs/2608.01033},
}