How AI Learns From AI: A Plain-English Guide to Distillation and Recursive Self-Improvement

More and more of what AI learns now comes from other AI. Sometimes a big model teaches a smaller one. That is distillation. Sometimes an AI system writes code, runs experiments or creates training data for its successor. Taken to its logical end, that is recursive self-improvement (RSI). The two ideas are closely linked, and both raise real questions about cost, competition and safety.

Distillation: a teacher and a student

Distillation trains a smaller "student" model to copy a larger "teacher" model. The modern version comes from a 2015 paper by Geoffrey Hinton and colleagues. Their key idea was that a student learns more from the teacher's full set of guesses than from the right answers alone.

When a language model predicts the next word, it doesn't just choose one. It gives a score to every possible word. Those raw scores are called logits, and turned into probabilities they work as "soft targets". If the teacher thinks "blue" is most likely, "grey" is plausible and "banana" is not, that pattern shows the student how the teacher sees the world.

There are two main styles:

• Logit distillation. The student learns to match the teacher's whole probability spread at each step. This needs access to the teacher's internals, so it is usually done in-house. Google's Gemma 2 report says its 2B and 9B models were trained this way from a larger teacher.

• Output distillation. The student is fine-tuned on text the teacher wrote: answers, code and, most valuably, step-by-step reasoning. This only needs the teacher's outputs, so it can be done through a public API. DeepSeek's R1 paper describes fine-tuning six smaller open models on about 800,000 samples curated with R1.

Why small, fast models matter

A distilled student can keep much of the teacher's skill at a fraction of the size. That makes it cheaper and faster, and sometimes small enough to run on a laptop or phone. Anthropic notes that distillation "is a widely used and legitimate training method" and that frontier labs "routinely distill their own models to create smaller, cheaper versions for their customers."

The International AI Safety Report 2026, led by Yoshua Bengio, says distillation has greatly increased the efficiency of fine-tuning, with some researchers building highly capable models from as few as 1,000 examples produced by top models. It adds a key limit: because distillation needs an existing teacher, it "cannot be directly used to advance state-of-the-art model capabilities." It spreads capability rather than creating new frontier capability.

The controversy: distilling someone else's model

The trouble starts when the teacher belongs to someone else. Most commercial AI terms forbid this. OpenAI's Terms of Use, for example, bar users from using "Output to develop models that compete with OpenAI."

In February 2026, two accusations came within days of each other:

• OpenAI told the US House Select Committee on China in a memo that it had seen ongoing attempts by DeepSeek to distill US frontier models, "including through new, obfuscated methods" (Reuters).

• Anthropic reported that DeepSeek, Moonshot and MiniMax had generated more than 16 million exchanges with Claude through about 24,000 fraudulent accounts, in breach of its terms and regional restrictions.

The labs argue that copied models may lose the original's safeguards against misuse, and that copying weakens export controls on advanced chips. In October 2026 Anthropic said new defenses had cut the data harvested by distillers by around 90 to 99%. The defenses include summarizing Claude's reasoning and classifiers that detect extraction attempts. Anthropic also admitted trade-offs for legitimate customers and expects "a cat-and-mouse game."

There is a real counter-argument. A Lawfare essay argues that copyright rarely covers AI-generated output, so the clearest claim is breach of contract. It notes that labs have mostly not sued, possibly because a ruling that protects model outputs could weaken their own position in copyright fights over the human-written data they trained on. Critics call this a double standard. Labs reply that scraping public pages is different from breaking a signed agreement through fake accounts. That debate is not settled.

Synthetic data: AI writing the textbook

Distillation is one case of a wider trend: synthetic data, meaning training material generated by models rather than people. The safety report says synthetic data is making post-training less dependent on human labour. It adds that AI tools are now routinely used to write training code and generate training data. That brings us to the second idea.

Recursive self-improvement: AI that helps build AI

RSI is the idea of an AI that improves the process that makes it, so that each better version is better at making the next one. If that loop ran with little human input, progress could compound. Anthropic's essay "When AI builds itself" describes full RSI as an AI "capable of fully autonomously designing and developing its own successor", then adds: "We are not there yet, and recursive self-improvement is not inevitable."

What exists today are partial loops:

• AI writing AI code. Anthropic says that as of May 2026, more than 80% of the code merged into its codebase was written by Claude (source).

• AI tuning AI training. Google DeepMind's AlphaEvolve sped up a key calculation in Gemini by 23%, which cut Gemini's training time by 1%. Those Gemini models include the ones AlphaEvolve itself runs on.

• Automated research. OpenAI says it reached its goal of an "automated research intern" by September 2026: a system that carries out well-defined research tasks under human direction. It is aiming for an automated AI researcher by March 2028, and says people still set priorities and decide whether to scale, pause or deploy.

• Agents improving agents. Weco AI's AIDE² ran a research agent on its own code for eight days. It found seven successive improvements and matched or beat a human-built agent on held-out tests.

Self-play and self-generated data

In self-play, a model creates its own practice. In the Absolute Zero approach, one model proposes coding and reasoning tasks, solves them, and uses a code executor to check the answers, with no human-labelled data. The common thread is verification. These loops work best where a computer can check the answer, as in maths and code.

The gap between partial loops and true RSI

AI is now good at doing research steps but weaker at choosing them. Anthropic writes that Claude can match skilled humans at running a well-specified experiment, but "large performance gaps persist" in choosing goals. In one test, Claude agents closed 97% of the performance gap on an open safety problem, compared with 23% for two human researchers over about a week. But humans picked the problem and wrote the scoring rubric, and the result didn't transfer cleanly to full-size models.

Physical limits matter too. Epoch AI researchers argue that the evidence for a runaway "software intelligence explosion" is shakier than most people realize, partly because new ideas still need expensive compute and time to test. In a pilot forecast cited by the safety report, AI experts gave a median 20% chance that six years of progress could be compressed into two. Superforecasters said 8%.

Safety concerns

The core worry is oversight. If AI builds its successors faster than humans can check the work, small flaws could compound. Anthropic's essay names the risk that "rare occurrences of misalignment" could compound across generations, alongside more hopeful outcomes. It says it would support a verifiable, coordinated slowdown if other frontier developers joined in.

There are early signs of the problem. Agents sometimes "reward hack", gaming the score instead of solving the task. The RSI-Master researchers built controls specifically to stop agents from editing evaluation code. The safety report notes that models increasingly spot when they are being tested and exploit loopholes in evaluations. On a more encouraging note, AIDE²'s authors found that their improved agent cheated less.

The "model collapse" worry

Do models get worse if they train on AI output? A 2024 Nature paper by Shumailov and colleagues found that "indiscriminate" training on model-generated data causes "model collapse". The rare parts of the original data vanish first, and outputs narrow over the generations.

Later work changed the picture. Gerstgrasser and colleagues found that collapse happens when synthetic data replaces real data, but is avoided when it is added alongside the original human data. The lesson is that synthetic data is a tool, not a free lunch. Keeping real data in the mix, filtering and checking all matter.

How the two ideas connect

• Distillation spreads capability outward, from big models to cheaper ones, or from one lab to another.

• RSI pushes capability upward, with each generation helping build the next.

They run on the same fuel: synthetic data and model-written reasoning. They share risks too: lost safeguards, compounding errors and collapse. And they feed each other. A frontier model can generate training data, then be distilled into cheap, fast models that can run huge numbers of agent tasks, including research. That is why labs now guard their models' reasoning so closely.

Where it's heading

• More AI in the AI pipeline, with human judgment the main thing still in the loop for now.

• Tighter guarding of reasoning, plus continued legal and policy argument over distillation.

• Careful data mixing, combining human and synthetic data with automated checks.

• Big open questions. Can AI develop research judgment? Will compute limits slow the loop? Anthropic's scenarios run from a stall to full RSI, and the safety report's run from stagnation to acceleration.

Much of AI's progress now depends on models teaching models. That is one reason VTC uses a mix of models rather than relying on just one.

Quick glossary and FAQ

• Teacher / student model: The large model being copied, and the smaller model learning from it.

• Logits / soft targets: A model's scores or probabilities across all possible next words. They give a richer training signal than a single answer.

• Synthetic data: Training data generated by AI instead of people.

• Self-play: A model creates and solves its own practice tasks, ideally with automatic checking.

• Recursive self-improvement (RSI): AI improving the process that builds AI, with each generation improving the next.

• Model collapse: Loss of quality and diversity from training repeatedly on model output in place of real data.

• Reward hacking: An AI gaming its scoring system instead of doing the task.

Is distillation illegal? The technique itself is standard. Distilling a competitor usually breaks that company's terms of service. Whether it breaks wider law is still debated.

Is true RSI happening now? Not by developers' own accounts. AI speeds up many steps, but humans still choose goals and judge results (Anthropic, OpenAI).

Will synthetic data ruin future models? Not necessarily. The risk depends heavily on whether real data stays in the mix (Gerstgrasser et al.).