The Hechinger Report said Patrick O’Neill’s audit found serious conceptual and statistical flaws in 14 of 39 meta-analyses that reported benefits from AI in education. A separate preliminary analysis by O’Neill averaged roughly zero across 44 experiments. A meta-analysis combines results from multiple studies, but these findings concern different sets of research and do not settle every use of AI in learning.
The Hechinger Report published its account on October 26, 2026, and said O’Neill described the audit at a webinar hosted by Justin Reich on October 20.
What the reported audit examined
The account says O’Neill examined 14 of 39 meta-analyses and identified conceptual and statistical problems. It attributes to him examples including implausible statistics, references he described as fabricated, conclusions that overstated the findings, and citations leading to unrelated studies.
Those concerns matter because a pooled result can only be as reliable as the studies and methods used to produce it. But the 14 reviews are a subset of the 39, and the reported audit does not by itself establish that every study of AI in education is flawed.
What a preliminary average near zero means
O’Neill’s separate, preliminary analysis covered 44 experiments. The Hechinger Report said their outcomes ranged from very helpful to very harmful, while the average was roughly zero. That average describes the reported result across those experiments; it does not mean each intervention had no effect.
The experiments also cannot answer every classroom question as a single group. A claim about a general-purpose chatbot, for example, addresses a different intervention from AI built into an educational app. What students do, what outcome a study measures, and what the comparison group receives all affect the question a synthesis can answer.
Two narrower studies ask different questions
A 2026 meta-analysis of ChatGPT and student learning outcomes combined 35 experimental and quasi-experimental studies published from 2022 to 2024, involving 4,193 participants. The authors reported a positive pooled effect, Hedges’ g = 0.670, with a 95% confidence interval of 0.495 to 0.844. Hedges’ g is a standardized measure of the difference between groups. The authors also reported I² = 91.444%, a measure indicating substantial variation among the study results. Read the ChatGPT learning-outcomes meta-analysis.
A separate meta-analysis compared AI-generated feedback with feedback from people. Across 41 papers and 4,813 students, its estimate for post-test task performance came from 11 articles covering 12 studies: g = 0.25, with a 95% confidence interval from −0.11 to 0.60 and I² = 75.0%. Because the interval includes zero, the authors found no statistically significant difference in that outcome. Most studies concerned language and writing. Read the AI-versus-human feedback meta-analysis.
| Research synthesis | Question examined | Study sample | Reported result | Variation across studies |
| ChatGPT and student learning outcomes | How ChatGPT relates to student learning outcomes | 35 experimental and quasi-experimental studies; 4,193 participants | Hedges’ g = 0.670; 95% CI 0.495 to 0.844 | I² = 91.444% |
| AI-generated versus human feedback | Whether feedback from AI or people differs in post-test task performance | 41 papers; 4,813 students overall. The task-performance estimate covered 11 articles and 12 studies. | g = 0.25; 95% CI −0.11 to 0.60; not statistically significant | I² = 75.0% |
The positive ChatGPT estimate and the nonsignificant feedback comparison are not opposing answers to one test. One concerns ChatGPT and learning outcomes; the other compares who provides feedback, mostly in language and writing studies. Their pooled results also show substantial variation among studies, so neither supplies a universal prediction for every classroom use.
For educators assessing a claim about AI and learning, the key question is what the studies actually tested: the tool, the learning outcome, and the comparison. A finding about ChatGPT and learning outcomes cannot stand in for a finding about AI-generated feedback versus human feedback.
