Forget Skynet: What RSI Actually Means
Recursive self-improvement (RSI) is not a sentient Skynet rewriting its own code without limits. Rather, it describes AI systems contributing to the research, code, data, or methods used to build subsequent AI generations. This involves human oversight, not an autonomous, runaway intelligence.
David Shapiro, a prominent AI commentator, proposes a pragmatic measure for RSI: what share of the next generation’s development comes from AI assistance? He speculates that in 2026, AI could contribute 30% of its own progress, potentially rising to 50-80% by 2027, encompassing code, data, and mathematical algorithms. These are estimates, not verified industry-wide figures.
Forget cinematic escape scenarios. AI models run on real infrastructure, not an abstract "cloud" where they can infect every internet-connected device. Powerful models demand massive data centers, specialized hardware like GPUs, and significant power, preventing them from spontaneously spreading or operating on consumer devices like smartphones, a fact often overlooked in fictional portrayals such as Terminator or Mission: Impossible – Dead Reckoning.
The Feedback Loop Is Already Taking Shape
Feedback loops are already taking shape, though not in the dramatic fashion of science fiction. Models now assist in writing code, generating and evaluating data, and testing systems. They also support mathematical reasoning, accelerating development cycles. Humans, however, still set objectives and deploy the results.
This incremental loop builds on systems like OpenAI's work in automated mathematics and Google DeepMind’s AlphaProof Nexus. The latter couples Gemini-class LLMs with the Lean 4 interactive theorem prover, solving open Paul Erdős conjectures previously unsolved for decades. These capabilities are useful, but they do not confirm autonomous, self-sustaining recursive self-improvement.
Announcements and reports from OpenAI, Google, and Anthropic indicate significant activity in this domain. These reports serve as evidence of progress in AI-assisted development, not proof of an emergent, uncontrollable intelligence. The focus remains on AI contributing to its own development in a controlled, human-directed manner.
Why Proof Could Unlock More Than Better Code
Formal verification offers a powerful, distinct approach to validating AI outputs. Rather than merely generating plausible text or code, formal verification mathematically checks whether a system, proof, or piece of code strictly adheres to explicit, predefined rules. This process aims to eliminate ambiguity and ensure correctness, building trust in AI-generated solutions.
This rigorous method holds immense promise for AI research, particularly in areas demanding absolute certainty. Google DeepMind’s AlphaProof Nexus, for instance, pairs large language models with the Lean 4 interactive theorem prover. This system has successfully solved multiple open Paul Erdős conjectures, some unsolved for over 50 years, by generating machine-verifiable, hallucination-free mathematical proofs.
The implications extend far beyond mathematics. Validated calculations could accelerate discovery in:
- Materials science
- Medicine
- Chip design
- Battery technology
However, formal verification is no panacea. It remains computationally intensive and critically depends on precisely specified problems. While it guarantees adherence to defined rules, it cannot validate the initial assumptions made by a model or guarantee the real-world accuracy of its conclusions. For more on the broader concept, see Recursive self-improvement - Wikipedia.
Enjoying this? Get one like it in your inbox each morning.
one email a day · unsubscribe in two clicks · no third-party tracking
A Fast Feedback Loop Still Hits Physical Limits
David Shapiro, an AI researcher, counters narratives of an overnight AI takeover, emphasizing that progress, while rapid, remains tethered to physical realities. Scaling AI still depends on chips, data centers, energy, networking, and manufacturing, all of which constrain how quickly systems can expand. Human decisions also dictate resource allocation, preventing an uncontrolled "fast takeoff."
As AI systems gain new capabilities, such as coding, mathematical reasoning, and file handling, emergence through recombination becomes evident. Combining these discrete tools can produce complex, useful workflows not explicitly programmed as single tasks. For example, an AI might independently generate code, test it, and then formally verify the output against specifications.
AI-assisted research will likely accelerate scientific progress, spreading into disciplines beyond computer science. However, this acceleration brings critical, unresolved questions. We must grapple with issues of oversight, alignment, formal verification, and the immense infrastructure required to support these increasingly capable systems.
Frequently Asked Questions
What is recursive self-improvement in AI?
It describes AI systems helping develop or improve later AI systems, through tasks such as coding, testing, data generation, and research.
Does recursive self-improvement mean AI will suddenly escape online?
No. It does not imply an AI can spread freely across the internet; powerful systems depend on hardware, computing resources, and human-run infrastructure.
Why does formal verification matter for AI research?
Formal verification uses mathematical methods to check whether software or proofs meet precise specifications, making it valuable for reliability and scientific work.
Could AI-assisted research affect fields beyond software?
Potentially. Better reasoning and verification could help researchers investigate areas such as mathematics, materials science, and life sciences.