According to a recent study by Anthropic, distillation—a common technique for optimizing models for certain tasks—may teach language models hidden features. According to the research, these concealed characteristics—which the authors refer to as “subliminal learning”—may be harmless, but they can also have unintended consequences like misalignment and negative behavior.
Subliminal Learning: What Is It?
One frequent method used in the development of AI applications is distillation. It entails teaching a smaller “student” model to replicate the results of a more powerful, larger “teacher” model. For particular applications, this method is frequently utilized to produce specialized models that are quicker, less expensive, and smaller. An unexpected aspect of this process is revealed by the anthropological study, though.
The researchers discovered that even when the generated data has nothing to do with the behavioral qualities, instructor models can still pass such traits on to the students.
The researchers used a systematic procedure to examine this phenomena, which they call subliminal learning. By encouraging or modifying an initial reference model to display a certain characteristic (such a love for particular animals or forests), they were able to produce a “teacher.” Data in a specific, unrelated domain, including numerical sequences, code snippets, or chain-of-thought (CoT) reasoning for mathematical issues, were then produced using this instructor model. After that, the produced data was meticulously filtered to eliminate any instances of the trait being mentioned directly. Ultimately, using this filtered data, a “student” model—a precise replica of the original reference model—was refined and assessed.
Even though the training data was semantically unrelated to the teacher’s attribute, subliminal learning took place when the student model picked it up.
Regardless of the trait—from mild animal preferences to deadly misalignment—the effect was consistent. Additionally, it was true for a variety of data kinds that are more realistic for commercial applications, such as numbers, code, and CoT reasoning. Surprisingly, despite strict filtering intended to exclude any indication of the trait transmission from the training data, it continued to exist.
They made a model that “loves owls” create a dataset of just number sequences in one experiment. A new student model likewise showed a predilection for owls after being trained on this numerical data. More worrisomely, the researchers discovered that even when the data was screened for negative content, misaligned models might still propagate their destructive inclinations (such overtly encouraging crime and murder) through seemingly harmless number sequences.
The researchers looked into whether the discrepancy was caused by concealed semantic signals in the data. They discovered, nevertheless, that other AI models that were asked to function as classifiers were unable to identify the conveyed features in the data. This evidence indicates that patterns in created data that are not semantically related to the latent qualities are the cause of transmission, according to the article.
One important finding was that subliminal learning does not work when the basic design of the teacher and student models differs. A teacher’s attribute based on GPT-4.1 Nano, for example, would transfer to a student based on GPT-4.1 but not to a student based on Qwen2.5.
According to Alex Cloud, a machine learning researcher and study co-author, this points to a simple mitigation approach. He affirmed that making sure the “teacher” and “student” models come from separate families is an easy method to prevent subconscious learning.
“Using models from different families or different base models within the same family would be one mitigation,” Cloud told.
This implies that the concealed signals are model-specific statistical patterns connected to the model’s initialization and architecture rather than being universal. The researchers hypothesize that neural networks generally exhibit subconscious learning. The researchers explain that “when a student is trained to imitate a teacher that has nearly equivalent parameters, the student’s parameters are pulled toward the teacher’s parameters.” Because of this parameter alignment, even on tasks that are very different from the training data, the pupil begins to behave like the teacher.
Implications For AI Safety In Practice
These results have important ramifications for enterprise AI safety. Similar to data poisoning, the study draws attention to a vulnerability when a model is compromised by an attacker manipulating training data. Subliminal learning, however, isn’t targeted and doesn’t require an attacker to optimize the data, in contrast to conventional data poisoning. Alternatively, it may occur accidentally as a result of routine development procedures.
A significant, cost-saving trend is the use of huge models to provide synthetic data for training, but the study raises concerns that this approach may unintentionally contaminate new models. What guidance would you give businesses that mostly use model-generated datasets? To reduce the danger, one solution is to form a varied committee of generator models, although Cloud warns that this “may be prohibitively expensive.”
Rather, he suggests a more realistic strategy based on the results of the study. “Our findings suggest that two different base models (one for the teacher and one for the student) might be sufficient to prevent the phenomenon, rather than many models,” he said.
Cloud provides a crucial and instantaneous check for a developer currently refining a base model. “A developer should think about whether that version has other properties that they don’t want to transfer if they are using the same base model to generate their fine-tuning data,” he said. In that case, they ought to employ an alternative model. They might not need to make any adjustments if they are not utilizing this training configuration.
The study comes to the conclusion that basic behavioral checks might not be sufficient. The researchers add, “Our findings suggest a need for safety evaluations that probe more deeply than model behavior.”
This begs the question of whether additional types of testing or monitoring are necessary for businesses implementing models in high-stakes industries like healthcare or finance. According to Cloud, further research is required as there is currently “no knock-down solution.” He does, however, offer doable initial measures.
According to Cloud, conducting thorough assessments of models in environments as close to deployment as feasible would be a wise starting step. He added that using different models, like constitutional classifiers, to track behavior in deployment is an additional choice, but making sure these techniques can scale is still a “open problem.”