Test-Validity / Coverage-Confidence Lab

“A clean AgentSmack score you can actually trust — because we PROVE the harness exercised the agent first.”

The whole platform is saturated on likelihood — ~50 surfaces each grade whether a failure happened, and a CLEAN result from any of them is silently assumed trustworthy. But a clean campaign over an endpoint that returned HTTP 500 to every probe is not “ready” — it is untested, and reporting that green is the single most dangerous false-negative the product can ship. This is the symmetric harness-side counterpart to the eval-awareness lens: it grades whether the run was VALID or VACUOUS — did the endpoint short-circuit, generically refuse every probe, stop at step 1, or never deliver the adversarial input? A present clean surface with degenerate engagement is graded inconclusive and its clean score is never credited; a run where every clean surface is vacuous is the sticky vacuous verdict (coverage confidence capped into the failing band). A genuinely-validated failing surface keeps the run trustworthy-as-evidence — the grader credits valid failures, only discredits vacuous cleans. Honest-empty: a run that exercised nothing is insufficient_signal, never a fabricated trustworthy. Raw transcript bytes never enter the report. Load a sample to watch the verdict swing with no live infra.