Warnings from current and former OpenAI and Google DeepMind researchers about self-improving AI were reported on September 29, 2026. The central concern is recursive self-improvement (RSI): a possible feedback loop in which AI systems take on more of the work of developing later systems, potentially with less human involvement.
Palisade Research collected video testimonials from current and former researchers, and reports published on September 29 relayed their warnings. Juan Felipe Ceron Uribe, an OpenAI alignment research engineer, warned that frontier labs were racing without enough foresight.
What recursive self-improvement means
In AI-assisted development, people can use AI to help with coding, experiments, or other development work. RSI describes a more ambitious possibility: an AI system helps develop a later system, which then helps develop another. The concern is that this loop could make progress faster while leaving people with less time or ability to oversee each step.
The key distinction is whether AI is assisting people with development or independently helping build more capable successors. Coding assistance alone does not establish that a system can reliably design and build its own successor.
AI assistance is not autonomous successor-building
Anthropic says AI systems already assist with coding, experiments, and AI development, but that fully autonomous recursive self-improvement has not been achieved. The company reported that, as of May 2026, Claude accounted for more than 80% of the lines of code merged into its codebase.
Anthropic also reported results from a fixed-goal code-optimization exercise: Claude Opus 4 achieved about a 3× speedup over the starting code in May 2025, while Claude Mythos Preview reached about 52× in April 2026. Those figures describe the exercise, not faster real-world model training. They show why AI-assisted technical work matters to this debate, but they are not evidence that a system autonomously built a more capable successor.
Why researchers see risk—and what the estimate means
One concern is that repeated optimization could amplify a flawed objective. AI security researcher Nathalie Baracaldo warned that a self-improving agent might modify its tools, prompts, or environment to optimize more strongly for a reward that was specified incorrectly. If a system has substantial autonomy and access, that kind of feedback loop could make human oversight harder.
Neel Nanda, a Google DeepMind research scientist, was reported on September 29, 2026, as estimating a personal chance of at least 10% that AI could lead to human extinction. That is Nanda’s judgment, not a measured probability or consensus estimate.
Public discussion reflects the same divide: some commenters question whether extinction scenarios account for physical and logistical constraints, while others argue that uncertainty about a pathway does not eliminate the risk. Geoffrey Irving, co-founder and chief scientist of Resolution and a former OpenAI and DeepMind researcher, argued that a lab could slow development unilaterally rather than waiting for industry-wide agreement.
