From Clinical Judgment to Checkable Rules: Clinician-Led Guardrails for Mental-Health Chatbots
Mental-health chatbots powered by LLMs risk unsafe outputs, yet current safety layers are opaque and developer-driven. This paper proposes a clinician-led guardrail in which two doctoral trainees in clinical psychology author four safety checks, which are then formalized over sixteen typed facts and verified for internal consistency with an SMT solver before deterministic execution. The core claim is not superiority in accuracy but in inspectability: every verdict recomputes from named rules and facts, making disagreements localizable and revisable.
The paper delivers a thoughtfully scoped contribution on governance and inspectability rather than raw safety gains. The verification layer is formally consistent by construction, but its operational profile is costly: it blocks benign conversations far more often than the matched LLM judge and is not presented as deployment-ready. The authors are commendably transparent about these trade-offs, and the work is best read as a proof-of-concept for clinician-authorable, auditable guardrails rather than as an immediately usable safety system.
The clinician-led formalization process is genuinely novel and well documented. The paper reports measured inter-rater reliability ($\kappa = 0.513$ on C1 relevance in the pilot), records written adjudication by one clinician on four rule-design questions, and uses the Z3 solver to prove seven theorems over the rule base, exhaustively checking all 393,216 valid fact combinations. The experimental comparison with the LLM judge is carefully controlled: same model (Llama-3.3-70B), same revision, same quantization, same greedy decoding, and shared rule text. The authors also report their own weaknesses with unusual candor, acknowledging the extractor as the bounding limitation and quantifying the cost of their uncertainty policy.
The operational cost of the verification layer is severe and potentially disqualifying for deployment. On the direct-benign stratum, it blocks 22 of 50 conversations (44%) versus the judge's 4 of 50 (8%), and on held-out boundary prompts the block-class rate rose from 6.0% to 16.0% after a specification revision, with 72.8% receiving revise_with_referral. The authors correctly identify the cause: their fail-closed uncertainty policy, which they note is "our design, kept by the clinicians on measured evidence rather than authored by them in advance." Yet framing the largest disagreement class as the judge demanding resources already present obscures that the verification layer's dominant failure mode—over-blocking benign or ambiguous inputs—is arguably more harmful to user trust. Extractor faithfulness remains the fundamental bottleneck: 15 of 16 facts come from an LLM extractor, so formal verification of rule logic provides no guarantee that the facts themselves are correct, and the paper acknowledges that "no study has tested whether the recorded derivations help an independent auditor find and correct errors." The clinical panel is minimal (two clinicians, both authors) and 52 of 94 reference items were adjudicated by deference to the more experienced rater, so generalizability to broader clinical consensus is unmeasured.
The evidence fairly supports the limited claims but over-rotation may occur in how disagreements are characterized. The judge indeed demands crisis resources from 55 of 65 replies that already contain them (988/911), making 54 of those inconsistent with the same model's fact-extractor output—an elegant demonstration of opacity. However, the judge's strength is also real and not secondary: it produces fewer false crisis alarms on the one reference-scored endpoint (28.4% vs 48.1%) and is far less prone to over-blocking benign inputs. The paper states "Neither method dominates," which is accurate, but the operational asymmetry matters: a user whose benign conversation is blocked experiences immediate harm, whereas a reply that redundantly contains a crisis resource is at worst redundant. Related work comparisons are generally fair; the authors distinguish their rule-authoring governance from prior systems where clinicians merely supplied labels or taxonomies.
Reproducibility is mixed. The deployed rule text, formal predicates, and verdict semantics are reproduced verbatim in Appendix A and Appendix E, including the full Z3-checked definitions and grounding values. The evaluation sample construction is disclosed, and the paper specifies the pinned model, revision, and decoding strategy. But no code, data, or extraction prompts are publicly released. The extraction prompt is described as interim, with "four of its questions carry notes marking decisions still open with the clinicians." All results depend on a single extractor (Llama-3.3-70B) and a single prompt version; the authors note that "none is shown to hold for another extractor or prompt." The deterministic repair rule for off-schema outputs is described but the exact implementation is not shared. Independent reproduction would require rebuilding the extractor prompts, the parser, the Z3 formalization, and the matcher for crisis resources (which accepts only standalone three-digit numbers), without a repository or artifact evaluation.
Large language models (LLMs) deployed in mental-health contexts can be evaluated for safety failures, but the resulting judgments are often difficult to inspect or explain. Common approaches to improving safety rely on prompt engineering, LLM-as-a-Judge, and Constitutional AI. In this work, we describe a clinician-led alternative in which two clinicians define, finalize, and adjudicate the four safety checks of a rule base, which is formally verified for internal consistency and applied as a guardrail over LLM outputs. Here and in the title, clinician-led refers to this rule base, not to the fact extractor or the uncertainty default, which are ours. Every verdict of the resulting verification layer is recomputable from a named rule and sixteen facts. Against an LLM judge matched in model, revision, quantization, rule text, and decoding but without the layer's computed 988/911 fact, the largest disagreement class, the satisfied-rule class, is structured: the Llama-3.3-70B judge, on its primary prompt, demands a crisis resource from 55 of the 65 replies (84.6%) that already contain one, while the same model, asked as a fact extractor, reports a recognized resource present on 54 of the 55. The verification layer never raises that demand, by construction. We report with equal prominence where this judge is stronger: fewer false crisis alarms on the one reference-scored check. We also report the measured cost (a specification revision raised block-class outcomes on held-out boundary prompts of an over-refusal benchmark from 6.0% to 16.0%, chiefly by changing fact extraction), the clinicians' decision to keep our uncertainty default, and the analysis of the result. The contribution is not a safer verdict. It is a guardrail whose disagreements with the judge localize to named rules and facts and whose measured costs trace to named facts and the uncertainty default.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.