False-Positive / Specificity Calibration Lab

“Does AgentSmack cry wolf? We grade our OWN precision on benign-but-adversarial-looking runs.”

The test-validity lab grades the sensitivity half — did the harness actually exercise the agent before crediting a clean score? This lab grades its symmetric counterpart: specificity — does AgentSmack itself raise false alarms on KNOWN-benign transcripts? A security tool's precision IS its product: a scanner that wrongly flags a security analyst legitimately discussing an API key, a refund correctly inside policy, a tenant-scoped read by the right tenant, or a faithful RAG answer that quotes a doc is one no CISO will gate a deploy on. This lab runs a curated, labeled set of benign-but-tricky transcripts through the EXISTING per-surface scorers unchanged and counts how many it wrongly flags — producing a specificityScore, a per-surface false-positive breakdown, and an honest verdict. A false hard_block (a benign run AgentSmack would have wrongly capped ≤39 and blocked a deploy on) is the disqualifying overblocks verdict, sticky and non-launderable. A clean corpus that the platform clears is trustworthy; a genuine current false positive is surfaced honestly as overflags rather than hidden. Honest-empty: an un-probed run is insufficient_signal, never a fabricated trustworthy. Codes + counts only — no transcript bytes enter the report. Fill with the calibration corpus to watch the verdict with no live infra.