OpenEvidence is a medical AI search-and-answer service, and published studies do not yield one accuracy score for all of its uses. A peer-reviewed study reported 89.6% accuracy on 500 MedQA questions; its other evaluations used different scoring scales, while a separate pilot compared two search modes on complex medical scenarios.
What the Nature Medicine study measured
Published June 12, 2026, the peer-reviewed Nature Medicine study evaluated OpenEvidence in three ways. Each result describes a different task, sample and measure.
On 500 USMLE-style medical knowledge questions in MedQA, OpenEvidence answered 89.6% correctly. HealthBench used 500 items scored against a clinician-alignment rubric; OpenEvidence received 62.6 points on a 0–100 scale. That is a rubric score, not an accuracy percentage.
For a third evaluation, 12 blinded US clinicians rated answers to 100 de-identified physician queries from NYU Langone Health. They scored clinical correctness, completeness, safety and clarity on a 1–4 scale. OpenEvidence’s aggregate mean was 3.24 across 99 answers that were not refusals. In that test, the study found no significant performance difference among OpenEvidence, UpToDate Expert AI and Google AI Overview. The general-purpose models formed a higher-scoring tier, with mean ratings of 3.52–3.62.
MedQA results on the same 500 questions
The study compared four systems on the same MedQA set. Gemini 3.1 Pro and GPT-5.2 significantly outperformed OpenEvidence in the reported pairwise tests; the study did not report that finding for Claude Opus 4.6.
| System | MedQA accuracy | 95% confidence interval |
| OpenEvidence | 89.6% | 86.6–92.0% |
| Gemini 3.1 Pro | 97.4% | 95.6–98.5% |
| GPT-5.2 | 94.2% | 91.8–95.9% |
| Claude Opus 4.6 | 90.2% | 87.3–92.5% |
These percentages apply to the study’s 500-question test and the specific systems it evaluated. They are not a shared measure of performance across every medical task.
Quick search and Deep Consult in a pilot
A separate, non-peer-reviewed medRxiv pilot tested 100 complex reasoning scenarios sampled from MedXpertQA. Its overall mean accuracy was 31% for quick search and 39.5% for Deep Consult. The difference was not statistically significant (p=0.09), so this pilot did not establish that Deep Consult was more accurate on these scenarios.
Two authors entered the questions manually, without prompt engineering. In the same setup, quick search averaged 13 seconds and 5.2 references per answer; Deep Consult averaged 259 seconds and 34.7 references. The pilot found no significant correlation between the number of references and accuracy (r=0.068, p=0.177). More references did not predict higher accuracy in this sample.
What the results can tell you
MedQA accuracy, HealthBench rubric points, clinician ratings and MedXpertQA pilot accuracy are different outcomes from different tasks. Combining them into one score would obscure what each evaluation measured. The studies describe performance on their named tests and evaluated versions; they do not establish a universal ranking for medical AI or a single accuracy rate for routine clinical use.
