Alexander J. Hish, Arjun Nagendran, Scott N. Comptoncs.AI
Background This study was designed to evaluate whether a domain-specific large language model (LLM) trained exclusively on patient education resources can answer questions about psychiatric medications, in a manner superior to LLM chatbots. We developed an LLM ("MIND") fine-tuned for clinical fidelity, trained on patient education resources from authoritative medical organizations. Methods We compared the responses of MIND, ChatGPT, and OpenEvidence to patient questions about escitalopram, using two methods: (1) computer analysis according to a rubric measuring accuracy, clarity, completeness, nuance, safety, and referral appropriateness; (2) ratings from N=10 board-licensed psychiatrists on similar metrics. Results When rated by rubric, MIND was rated highest in all domains (p<0.001). When rated by psychiatrists, ChatGPT was rated accurate more often than MIND with a negligible effect size (p=0.021, r=0.073); MIND was rated complete more often than ChatGPT with a small effect size (p<0.001, r=0.160); and MIND and ChatGPT were rated safe with the same frequency (p=0.955, r=0.002). The majority of psychiatrists preferred the responses generated by ChatGPT (57.6%) compared to MIND (42.4%, p=0.003). Conclusions MIND was able to answer many questions about escitalopram in a manner deemed accurate, complete, and safe by psychiatrists the majority of the time. However, despite MIND's ability to provide more complete responses, psychiatrists preferred ChatGPT's responses. MIND represents a step towards building safe LLM systems to enhance patient education in psychiatry.
External and synthetic control arms (ECAs) are entering psychiatric drug development, but the field lacks a benchmark that evaluates the properties regulators care about: not only how accurately a method reconstructs untreated trajectories, but whether its uncertainty is calibrated, whether it is robust to the informative observation times common in mental-health records (sicker patients are seen more often), and what false-positive rate it induces in go/no-go trial decisions. Real psychiatric trial data (e.g. STAR-D and registry cohorts) require credentialed access and lack ground-truth counterfactuals, so, following established semi-synthetic benchmarks in causal inference (IHDP, ACIC, and the PK-PD tumor-growth simulator), we release Psych-ECA, a fully reproducible generator of longitudinal symptom trajectories for depression (PHQ-9), anxiety (HAM-A), and psychosis (PANSS) with known counterfactual control arms, informative visits, and validated-scale measurement noise. We benchmark eight estimators spanning carry-forward, pooled real-world-data averages, nearest-neighbour matching, linear mixed models, gradient boosting, and the Scribe trajectory-bridge method. Three findings emerge. First, trajectory and flexible machine learning methods achieve the best counterfactual accuracy (about 2.3 PHQ-9 RMSE), outperforming cross-sectional baselines. Second, only Scribe is both accurate and calibrated, achieving 93-96% empirical coverage of nominal 90% prediction intervals, compared with 87-88% for gradient boosting and 62-75% for uncalibrated SDE models. Third, inverse-intensity correction reduces bias under informative sampling, while Scribe's calibrated intervals are the only trajectory method that maintains nominal false-positive rates as informativeness increases. We release all code, data-generation scripts, and random seeds to enable fully reproducible evaluation.