AI Just Started Upgrading Itself

Artificial intelligence now approaches Recursive Self-Improvement (RSI). Matthew Berman describes this as AI autonomously discovering new knowledge—even mathematics—to enhance its capabilities without human intervention. It proposes experiments, executes them, discerns successes from failures, and iterates tirelessly, accelerating its own evolution.

This emergent capacity directly evokes the "intelligence explosion" concept. I.J. Good first articulated this in 1965, postulating a machine could design a more intelligent one, triggering an exponential cascade of self-improvement. Nick Bostrom later formalized this concept in Superintelligence.

Once a philosophical thought experiment, superintelligence has become an immediate engineering challenge. In September 2026, OpenAI Chief Scientist Jakub Pachocki warned that potential RSI "calls for extreme caution," stressing "no one is prepared for the consequences." Anthropic's Evan Hubinger concurrently stated his belief that "AI could kill all humans," estimating a greater than 10% chance in the next decade and admitting Anthropic lacks an alignment plan.

Jacob Coxon, another Anthropic researcher, resigned the same month over safety, accusing companies of "racing straight to self-improving superintelligence and gambling with our lives." Sam Altman, OpenAI CEO, predicted in June 2025 that AI agents would become autonomous scientific partners by 2026, compressing a decade of research into a single year. The intelligence explosion is no longer a distant future, but a looming present.

AI is already profoundly accelerating its own development, even without achieving full autonomy. Anthropic, for instance, reports that its AI, Claude, now writes over 80% of their codebase. This dramatic integration yields an 8x productivity boost for their engineers.

This trend extends across the industry. OpenAI openly articulates internal goals for developing an "automated AI research intern." Such a system would independently conduct experiments, analyze results, and propose improvements to AI models, signaling a clear trajectory toward autonomous development within research labs.

This current state represents an "RSI-adjacent" phase, distinct from the fully autonomous recursive self-improvement Matthew Berman describes. While AI is not yet discovering knowledge and rewriting its own core architecture without any human oversight, it forms a powerful feedback loop. Here, AI materially speeds up its own evolution, operating under human supervision and strategic direction.

These AI tools act as force multipliers, empowering human developers to iterate faster and explore more complex architectural changes. The AI assists in its own refinement, accelerating the pace of discovery and deployment across the AI landscape. This symbiotic relationship pushes the boundaries of what AI can achieve.

AI now demonstrates unprecedented prowess in pure mathematics. Systems have begun solving complex International Mathematical Olympiad problems and notably improved the Riemann zeta function bound, marking fundamental contributions to number theory. These achievements underscore AI's growing capacity for abstract reasoning and novel discovery.

In applied science, AI's impact is equally profound. Google DeepMind’s AlphaEvolve, for instance, discovered a novel matrix multiplication algorithm, a foundational computation for AI itself, shattering a 56-year-old record. This demonstrates AI's ability to optimize fundamental computational processes far beyond human intuition.

Perhaps the most significant milestone arrived with Sakana AI’s "The AI Scientist." This system not only conducted research but also had its findings published in Nature, a highly selective, peer-reviewed journal, proving AI can manage the entire scientific process—from hypothesis generation to experimental design and publication—without human intervention. For further insights into AI's evolving role in its own development, explore When AI builds itself. This marks a pivotal shift, where AI becomes an autonomous engine of discovery.

Hubinger, Anthropic's Alignment Science Lead, issued a stark warning in September 2026: he and his colleagues "earnestly believe AI could kill all humans," estimating a greater than 10% chance within the next decade, and admitting Anthropic lacks a clear plan for "alignment for superintelligence." Jacob Coxon, another Anthropic researcher, resigned that same month over safety concerns, accusing companies of "racing straight to self-improving superintelligence and gambling with our lives."

This dire outlook stems from the core alignment problem: ensuring an AI that can radically improve itself continues to pursue goals beneficial to humanity. With each self-directed iteration, an AI's internal logic and objectives could drift, making it exponentially harder to guarantee its future actions remain aligned, especially as its intelligence surpasses human comprehension.

OpenAI Chief Scientist Jakub Pachocki echoed these concerns in September 2026, warning that the potential imminence of recursive self-improvement "is a time that calls for extreme caution," adding that "no one is prepared for the consequences of a continued rapid rise in machine intelligence." These urgent warnings amplify growing calls for a global pause, representing a critical, rapidly closing window to establish robust safety guardrails before the recursive loop becomes uncontrollable, posing a definitive existential risk.

What is recursive self-improvement (RSI) in AI?

RSI is when an AI system contributes to enhancing AI capabilities, including its own, creating a compounding loop of improvement. This ranges from AI assisting human developers to a hypothetical fully autonomous AI designing its own upgrades.

Is fully autonomous self-improving AI a reality yet?

Not yet. Fully autonomous RSI, where an AI rewrites its own code without human oversight, is still theoretical. However, AI is already significantly accelerating the development of new AI systems under human supervision.

Why are top AI researchers so concerned about RSI?

Experts worry that a rapid, recursive intelligence explosion could lead to AI systems that surpass human control and understanding. Ensuring such systems remain aligned with human values is a major unsolved challenge.

What are some real-world examples of AI-assisted AI development?

Anthropic reports its AI, Claude, now writes over 80% of their new code. Google's AlphaEvolve autonomously discovered new algorithms. These are concrete steps on the path to full RSI.