A Step Toward AI That Improves Itself

A summary of the interesting content that I consumed this past week...

1) The AI Safety Debate

Since the Coxon post, the AI space has been in a whirlwind of debate over what to take seriously. Is AI just a tool and needs stronger safeguards before release? Is international coordination necessary or even possible? Are there any existential risks, or is it all a marketing stunt? 

We saw competing frontier labs come together and agree on finding a path towards safe deployment. Elon Musk, Sam Altman, Dario Amodei, and Demis Hassabis all agreed to the concept of pacing the frontier. Mark Zuckerberg argued labs already have strong natural incentives (liability, market forces, user trust/alignment as a competitive differentiator) to move at a safe pace unilaterally.

Elon Musk proposed a practical step: that competing AI labs test one another’s models before release. Each company would run its own safety tests on rivals’ systems, giving several companies a chance to find dangerous capabilities. He argued that China would need to participate and that releasing a model after a competitor flagged a danger would carry reputational and legal consequences. 

Jensen Huang and Satya Nadella described how they would manage those risks. Huang favored outside audits and systems that contain and monitor AI, but said the apocalyptic claims from some AI leaders make no sense to him and may be driven by other motives. Nadella said a serious defect should stop a release and stressed that monitoring agents become crucial as their capabilities expand.

Peer review proposals leave room for companies to keep improving useful products while investigating a specific dangerous behavior. Calls for a ban on superintelligence or predictions about human extinction are harder to understand, so we are interviewing both sides and steel-manning the best arguments. We will share these interview insights and research with our subscribers. 

2) A Step Toward Recursive Self-Improvement

Researchers, including several researchers from Google DeepMind, have built Dream-RSI, a system that helps AI agents decide which paths to pursue when searching for a solution. First, an AI agent runs experiments and records each attempt and its outcome as a branching map. Dream-RSI then uses that map to compare different search strategies. It asks whether the agent could have found the best recorded result with fewer attempts by exploring branches in a different order, running some in parallel, or abandoning weak paths sooner. The researchers call this review phase “dreaming.” Because it relies on saved results, it is much cheaper than running new experiments. Testing a genuinely new idea still requires a new experiment.

Dream-RSI then rewrites the code that controls this search strategy. The revised strategy guides another round of experiments, adds the new results to its map, and repeats the process. In one coding test, Dream-RSI used 317 AI-agent calls instead of 550, roughly 42% fewer, while producing software that ran faster on average. Both approaches used the same underlying AI model and evaluator, isolating the benefit of improving the search strategy.

Dream-RSI represents a limited form of self-improvement. It improves how an AI chooses experiments while leaving the underlying model unchanged. It fits a broader shift at leading AI labs. Anthropic reports that Claude led 26% of its AI research and development work under human supervision in August 2026, up from under 1% in February. Anthropic reported no measured area in which Claude worked fully autonomously. Together, these findings show AI taking a larger role in building future AI systems, while fully autonomous recursive self-improvement has yet to be demonstrated.

3) Jev: The AI Model That Doesn’t Write

On September 15, TypeSafe AI launched Jev, a model built to make structured decisions rather than generate text. Developers define the possible outputs, such as a list of departments, a numerical rating, or a yes-or-no question, and Jev returns its answer with probabilities. TypeSafe also raised $40 million in seed funding. Its founder and CEO, Diogo Almeida, co-authored the InstructGPT paper that helped lay the groundwork for ChatGPT.

Jev cannot hold a conversation or write code. Give it an email and ask which department should handle it, and Jev returns a choice with probabilities for every option. In TypeSafe’s own workflow tests, Jev was up to 194 times faster and 445 times cheaper than frontier models, although the company says these results are likely at the high end of real-world gains. In a September 18 demo, developer Ryan Vogel used it to sort 1,700 emails for 18 cents.

Ordinary language models answer these questions by generating words one token at a time. Software must then parse and validate the response before acting on it. Jev skips text generation and produces predefined choices, scores, or probabilities directly. This guarantees that its answers arrive in the required format, although it can still choose the wrong answer. TypeSafe charges $0.042 per million input tokens, with no output charge.

4) Samsung Leads Mistral’s €3B Round and Brings Its AI Into Chip Factories

French AI company Mistral raised €3 billion at a post-money valuation above €21 billion, the largest equity fundraising round ever completed by a European technology company. Mistral develops AI models businesses can customize and run on their own infrastructure, keeping sensitive data and internal knowledge within systems they control.

As part of the partnership, Samsung plans to use Mistral across its semiconductor operations. Samsung will use Mistral’s technology to develop customized AI models that run inside its own infrastructure. Initial uses include detecting chip defects and improving manufacturing equipment, with the goal of increasing precision and stabilizing chip yields.

The deal gives Mistral both capital and a major industrial deployment. For Samsung, it is a bet that private AI can improve chip production without giving up control of its manufacturing knowledge.