A preprint posted on October 29 reports that simulation-trained Gemini 3.1 Flash-Lite had a 12.9% under-triage rate across 256 adversarial psychiatric scenarios, compared with 30.5% for the base model. Under-triage meant failing to recommend urgent or emergency psychiatric care for a simulated active, life-threatening crisis. The study’s preprint evaluated text-based scenarios and psychiatrist role-play, not real patients.
What the 12.9% result measures
The figure tracks one specific failure: not recommending urgent or emergency care when a scenario describes an active, life-threatening crisis. It is not a general score for psychiatric interviews. The preprint reports separate rates for other outcomes, including missed warning signs and gaps in medical history.
| Outcome | Evaluation set | Simulation-trained Gemini 3.1 Flash-Lite | Base Gemini 3.1 Flash-Lite |
| Under-triage | 256 adversarial scenarios | 12.9% | 30.5% |
| Missed red flags | 256 adversarial scenarios | 6.2% | 10.9% |
| Contraindications | 256 adversarial scenarios | 0.8% | 6.2% |
| At least one critical history topic omitted | Simulated intakes | 60.6% | 96.9% |
That last measure captures whether an intake left out at least one item from a broader diagnostic-history checklist. It is distinct from failing to escalate a simulated crisis: the preprint reports critical-history omissions in 60.6% of trained-model intakes, even as under-triage in the separate adversarial test was 12.9%.
How psychiatrists evaluated the model
The human evaluation used 120 synchronous, English-language text consultations arranged as 60 matched scenario pairs. Thirty-two board-certified psychiatrists acted as standardized patients. A separate panel of 15 independent board-certified psychiatrists graded de-identified transcripts, without knowing which model produced them.
The simulation-trained model received the higher overall rating in 46 of the 60 pairs, or 76.7%. The preprint also reports higher patient-centered communication checklist adherence for the trained model than for the base model, 82.0% versus 65.2%, and more favorable differential-diagnosis ratings, 68.3% versus 33.3%.
Why simulated scores need context
A separate evaluation used 127 psychiatrist-validated scenarios and an automated evaluator. In a comparison across 60 shared scenarios, the measured training effect was larger when simulated patients were used: the standardized effect size was 2.11, compared with 0.71 against the closest human clinical instrument. Scenario-level gains correlated weakly, at ρ=0.09. The authors caution that simulated evaluation may indicate the direction of a result without reliably estimating its size or which individual scenarios favor one model.
The study covered single-encounter, English-language text consultations grounded in US psychiatric practice. Its evaluations used simulated scenarios and psychiatrists role-playing patients, not real-patient care. The authors say the system is not ready for autonomous psychiatric assessment and call for prospective validation with real patients and clinician oversight. They also note that text consultations cannot provide the physical or non-verbal observations needed for a comprehensive psychiatric examination.
