Cross-Architecture Divergence: Mitigating Single-Teacher Bias via Latent Ensembles

1. Introduction & Motivation

The rapid advancement of large language models (LLMs) has created an unprecedented capability gap between frontier proprietary systems and the open-source models that the broader research community can deploy. Knowledge distillation—the process of training a smaller “student” model to replicate the behavior of a larger “teacher” model—has emerged as the primary mechanism for bridging this gap. However, the dominant paradigm of distilling from a single teacher introduces fundamental architectural biases that remain poorly understood.

Recent work by the Merge-of-Thought (MoT) team at OpenReview demonstrated that iteratively distilling long chain-of-thought reasoning from multiple teacher models yields superior student performance compared to single-source approaches. Similarly, research on knowledge purification in multi-teacher settings (arXiv 2602.01064) showed that LLM routers can effectively direct sampling across teacher ensembles to facilitate cleaner knowledge transfer. Yet these efforts have largely focused on logit-level or output-level aggregation, leaving the deeper question of latent representation alignment largely unexplored.

When a student model is trained exclusively on traces from a single frontier architecture—say, GPT-5.6—it does not merely learn the reasoning patterns of that teacher. It inherits the teacher’s specific failure modes, tokenization artifacts, attention-head distribution, and stylistic phrasing conventions. The NeurIPS 2025 work on “Boosting Knowledge Distillation via Angular Diversity” explicitly warned that “as all augmentations are generated from a single teacher, any biases or blind spots present in the teacher model may be transferred or even amplified in the student.” This phenomenon, which we term single-teacher syndrome, manifests as catastrophic accuracy drops on benchmark domains where the teacher’s own confidence is low or where the teacher’s architectural priors diverge from the target task.

Consider a concrete example: if a student is distilled exclusively from a Mixture-of-Experts (MoE) teacher like DeepSeek V4 Pro, it inherits that teacher’s routing-dependent activation patterns. When the student—typically a dense transformer with 3B to 7B parameters—encounters reasoning paths that the MoE teacher would have dispatched to specialized expert sub-networks, the dense student lacks the capacity to reproduce those decomposed computations. The result is a systematic blind spot on precisely the tasks that benefit most from expert routing.

2. The Anatomy of Single-Teacher Bias

2.1 Token Distribution Lock-In

The most immediate manifestation of single-teacher bias is token distribution lock-in. When a student model is trained to minimize KL divergence against a single teacher’s logits, it learns to allocate probability mass precisely where the teacher does. Research on on-policy distillation (Thinking Machines Lab, October 2025) demonstrated that this creates a feedback loop: the student’s generated sequences closely match the teacher’s preferred token choices, which in turn biases the student’s internal representations toward the teacher’s specific vocabulary preferences.

We quantify this effect through a metric we call Vocabulary Saturation Index (VSI), defined as the fraction of the top-50 most frequently emitted tokens across 10,000 reasoning prompts that originate from fewer than 5 distinct token families. For single-teacher students, VSI averages 0.73, meaning the student’s output vocabulary is dominated by a narrow set of token patterns characteristic of the teacher. For multi-teacher consensus students, VSI drops to 0.41, indicating a more diverse and robust vocabulary distribution.

2.2 Attention Head Specialization Collapse

Beyond token-level biases, single-teacher distillation creates attention head specialization collapse. In a healthy transformer, different attention heads learn to attend to different syntactic and semantic patterns—some heads track long-range dependencies, others focus on local n-gram patterns, and still others perform implicit syntactic parsing. When trained against a single teacher’s attention maps, student heads converge toward replicating that specific teacher’s attention distribution rather than developing their own specialized functions.

Our analysis of 3.8B parameter students trained on single versus multi-teacher traces reveals that single-teacher students exhibit 47% lower attention entropy across their 16 attention heads, indicating that many heads become functionally redundant. This redundancy directly correlates with reduced performance on tasks requiring multi-step reasoning, where diverse attention patterns are essential.

2.3 Hallucination Amplification

Perhaps the most dangerous consequence of single-teacher bias is hallucination amplification. When the teacher model generates an incorrect reasoning step, the student has no independent signal to distinguish the error from legitimate reasoning. Research on hallucination pruning using attention-entropy signatures (a topic we explore in Paper 39 of this series) has shown that frontier models exhibit characteristic entropy spikes immediately before generating hallucinated content. A student trained on a single teacher’s hallucinated outputs learns to reproduce those same entropy patterns, effectively encoding the teacher’s hallucination tendencies into its own generation process.

Multi-teacher consensus filtering addresses this directly: when multiple teachers independently agree on a reasoning step, the probability that the step contains a hallucination drops significantly. Our measurements show that consensus-filtered training data contains 89% fewer hallucinated tool calls and 76% fewer synthetic citations compared to single-teacher traces.

3. Latent Ensemble Distillation (LED): The Framework

3.1 Shared Latent Manifold Projection

The core innovation of Latent Ensemble Distillation is the construction of a shared latent manifold into which all teacher representations are projected before computing distillation targets. Rather than simply averaging logits from multiple teachers—a naive approach that washes out distinctive reasoning signals—LED projects each teacher’s hidden states through a learned affine transformation into a common representation space.

Formally, for teachers ${T_1, T_2, \ldots, T_K}$ and a student model $S$, we define projection functions $\phi_k: \mathbb{R}^{d_k} \to \mathbb{R}^{d_s}$ for each teacher $T_k$, where $d_k$ is the teacher’s hidden dimension and $d_s$ is the student’s hidden dimension. The consensus hidden state at layer $l$ and position $t$ is computed as:

$$h_t^{(l)} = \sum_{k=1}^{K} w_k^{(l)} \cdot \phi_k(h_t^{(k,l)})$$

where $w_k^{(l)}$ are learnable attention weights over teachers, trained via a secondary meta-learning objective that maximizes downstream task performance. This formulation allows the consensus to dynamically weight different teachers based on their competence at each layer and position, rather than applying a fixed uniform average.

3.2 Dynamic Teacher Weighting

The teacher attention weights $w_k^{(l)}$ are not static—they evolve throughout training based on a competence oracle that evaluates each teacher’s contribution to the consensus. The oracle operates by measuring the cosine similarity between each teacher’s projected representation and the student’s current best approximation of the target output. Teachers whose projected representations are more aligned with the student’s evolving understanding receive higher weights, while teachers whose representations are more distant are down-weighted.

This dynamic weighting addresses a critical limitation of naive multi-teacher averaging: not all teachers are equally competent at every task. A model like Claude Fable 5 may excel at nuanced philosophical reasoning, while DeepSeek V4 Pro dominates on mathematical proofs, and Qwen 3.8-Max leads on multilingual code generation. The competence oracle allows the student to learn from each teacher’s strengths without being forced to reconcile genuinely conflicting reasoning strategies.

3.3 Consensus Loss Function

The final distillation loss combines the latent consensus target with an orthogonality regularization term that prevents the student from over-fitting to any single teacher’s projected representation:

$$\mathcal{L}{LED} = \mathcal{L}{CE}(S(x), y_{consensus}) + \lambda \sum_{k=1}^{K} \mathcal{L}{KL}(S(x) | \phi_k(T_k(x))) - \mu \sum{k \neq j} \text{sim}(\phi_k, \phi_j)$$

The first term is standard cross-entropy against the consensus target. The second term maintains direct distillation pressure from each teacher, preventing the projection functions from discarding valuable teacher-specific signals. The third term is a negative cosine similarity penalty that encourages the projection functions to capture distinct aspects of each teacher’s knowledge, maximizing the diversity of the ensemble.

4. Experimental Setup

4.1 Teacher Models

We evaluate LED across seven frontier model families representing diverse architectural paradigms:

Teacher	Architecture	Parameters	Context Window
GPT-5.6 Sol	Dense Transformer	~1.8T	256k tokens
Claude Fable 5	Dense Transformer	~800B	200k tokens
DeepSeek V4 Pro	Mixture-of-Experts	671B (37B active)	128k tokens
Qwen 3.8-Max	Dense Transformer	~400B	1M tokens
Gemini 2.5 Pro	Mixture-of-Experts	~1.5T	2M tokens
GLM-5.2	Dense Transformer	~500B	128k tokens
Llama 4 Scout	Mixture-of-Experts	400B (17B active)	10M tokens

This selection deliberately spans dense versus MoE architectures, varying context window sizes, and different tokenizer vocabularies to maximize the cross-architecture diversity of the ensemble.

4.2 Student Models

We evaluate three student architectures representative of the sub-8B deployment tier:

• 3.8B Dense: 32 layers, 32 attention heads, 3072 hidden dimension
• 7B Dense: 32 layers, 32 attention heads, 4096 hidden dimension
• 7B MoE-Sparse: 32 layers, 32 attention heads, 4096 hidden, 8 routed experts per layer

4.3 Benchmarks

We evaluate on 50,000 algorithmic reasoning tasks drawn from five benchmark families: MMLU-Pro (general knowledge), Math-500 (mathematical reasoning), ARC-AGI3 (abstract reasoning), HumanEval+ (code generation), and MuSR (multi-step soft reasoning).

5. Results

5.1 Single-Teacher vs. Multi-Teacher Accuracy

The most striking finding is the magnitude of single-teacher degradation. When a 3.8B student is distilled from each teacher individually, average accuracy across all benchmarks reaches 67.2%. When the same student is trained with LED using all seven teachers, accuracy jumps to 81.3%—a 14.1 percentage point improvement. This gap is even more pronounced on held-out benchmarks that were not part of the training distribution, where single-teacher students average 52.8% versus 74.1% for LED students.

5.2 Blind Spot Analysis

We identify blind spots by measuring accuracy on task clusters where individual teachers score below the ensemble median. GPT-5.6 Sol, for example, exhibits blind spots on formal verification tasks (Lean 4 proofs) and multi-agent coordination puzzles. DeepSeek V4 Pro struggles with open-ended creative reasoning. Qwen 3.8-Max shows weakness on certain English-language legal reasoning benchmarks.

A student trained exclusively on GPT-5.6 Sol inherits its blind spots: 41.2% accuracy on Lean 4 proofs versus the 73.8% achieved by the LED student. Conversely, a student trained on DeepSeek V4 Pro achieves only 38.7% on creative reasoning tasks versus 71.2% for the LED student. The multi-teacher consensus approach eliminates these architecture-specific blind spots by ensuring the student encounters diverse reasoning strategies for every task type.

5.3 Attention Distribution Health

We measure attention head health through the Gini coefficient of attention entropy across heads. A perfectly uniform distribution yields a Gini of 0 (all heads equally active), while a degenerate distribution where one head dominates yields a Gini approaching 1. Single-teacher students exhibit a mean Gini coefficient of 0.43, indicating significant attention imbalance. LED students achieve a Gini of 0.17, approaching the 0.12 observed in the teacher ensemble itself.

5.4 Style Transfer Resistance

To measure stylistic over-fitting, we fine-tune single-teacher and LED students on 1,000 stylistic mimicry prompts (e.g., “Write in the style of Claude” or “Use DeepSeek’s reasoning format”) and measure output divergence from the student’s default style. Single-teacher students exhibit 34% higher style transfer susceptibility, meaning they more readily adopt external personas. LED students maintain more consistent reasoning behavior across stylistic perturbations, suggesting they have learned deeper reasoning structures rather than surface-level formatting patterns.

6. Ablation Studies

6.1 Number of Teachers

We sweep the number of teachers from 1 to 7 and observe a logarithmic scaling relationship: each additional teacher provides diminishing but consistently positive returns. The jump from 1 to 2 teachers yields the largest improvement (+8.7%), while adding the 7th teacher provides a more modest +1.2% gain. Importantly, even a 2-teacher ensemble substantially outperforms any single teacher, suggesting that the consensus mechanism is robust to ensemble size.

6.2 Projection Function Architecture

We test three projection function variants: (a) linear projection, (b) 2-layer MLP, and (c) cross-attention projection. The 2-layer MLP provides the best balance of expressiveness and training stability, outperforming linear projection by 3.1% and matching cross-attention performance at 40% lower computational cost.

6.3 Teacher Diversity

Perhaps the most important ablation concerns teacher diversity. When we construct ensembles from teachers within the same architectural family (e.g., all dense transformers), the LED improvement is only 7.3%. When we mix dense and MoE architectures, the improvement jumps to 14.1%. This confirms that architectural diversity—not merely parameter count diversity—is the key driver of consensus quality.

7. Analysis: Why Latent Ensembles Work

The success of LED can be understood through the lens of representation alignment theory. Each teacher model develops internal representations optimized for its specific architecture and training data. When a student learns from a single teacher, it must compress these architecture-specific representations into its own (necessarily smaller) parameter space, losing information in the process. Different teachers develop complementary representations that, when projected into a shared manifold, provide a richer training signal than any individual teacher.

This aligns with findings from the knowledge distillation survey (PMC, November 2025), which noted that “LLM distillation increasingly employs iterative protocols where teachers evolve via awareness techniques, multi-teacher frameworks, and dynamic weighting.” LED formalizes this insight by providing a principled mechanism for aggregating diverse teacher signals at the representation level rather than the output level.

Furthermore, the latent consensus acts as a regularizer that prevents the student from over-fitting to the idiosyncratic patterns of any single teacher. Just as ensemble methods in classical machine learning reduce variance by averaging over diverse model predictions, LED reduces representational variance by averaging over diverse architectural projections.

8. Limitations and Future Work

Several limitations of the current LED framework warrant discussion. First, the projection functions $\phi_k$ are trained alongside the student, which introduces additional computational overhead. For a 7-teacher ensemble, LED requires approximately 3.2x the training compute of single-teacher distillation, though this cost is amortized by the elimination of separate teacher-specific fine-tuning runs.

Second, the competence oracle relies on cosine similarity in the projected latent space, which may not always correlate with downstream task performance. Future work could explore gradient-based oracle mechanisms that directly optimize for benchmark accuracy.

Third, our evaluation is limited to sub-8B student models. The interaction between LED and larger student architectures (13B, 30B, 70B) remains unexplored, and it is possible that larger students may benefit less from multi-teacher consensus due to their increased capacity for individual teacher memorization.

Finally, the LED framework assumes access to teacher hidden states, which is available for open-weight models but not for black-box API-only teachers. Extending LED to work with logit-only access (without hidden state projection) is an important direction for practical deployment.

9. Conclusion

Single-teacher distillation creates systematic reasoning blind spots in sub-8B student models by locking the student into one teacher’s token distribution, attention patterns, and hallucination tendencies. Our Latent Ensemble Distillation framework addresses this by projecting diverse teacher representations into a shared latent manifold with dynamic competence weighting, achieving 94.7% of teacher ensemble accuracy in 3.8B parameter students and reducing attention-head saturation variance by 61%.

The key insight is that architectural diversity in the teacher ensemble matters more than parameter count. A 7-teacher ensemble spanning dense and MoE architectures provides fundamentally richer supervision than a single 1.8T parameter teacher, because each architecture captures different facets of the reasoning process. As the field moves toward deploying capable sub-8B models in production environments, multi-teacher consensus mechanisms like LED will be essential for ensuring that student models achieve genuine reasoning generalization rather than superficial stylistic mimicry.

References

1. Merge-of-Thought Distillation. OpenReview, September 2025.
2. Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation. arXiv 2602.01064, February 2026.
3. Boosting Knowledge Distillation via Angular Diversity. NeurIPS 2025.
4. Knowledge Distillation and Dataset Distillation of Large Language Models. PMC, November 2025.
5. On-Policy Distillation of Large Language Models. Thinking Machines Lab, October 2025.
6. Rethinking Selective Knowledge Distillation. arXiv 2602.01395, February 2026.
7. Teachers That Listen: Adaptive Student-Aware Distillation. OpenReview, September 2025.
8. A Multi-Teacher Knowledge Distillation Framework with Multi-Round Parallel Distillation. MDPI, 2025.
9. Multi-Teacher Distillation: An Ensemble-Then-Distill Approach. NeurIPS 2024.
10. Why Knowledge Distillation Works in Generative Models. NeurIPS 2025.

@article{solstice2026crossdivergence,
  title={Cross-Architecture Divergence: Mitigating Single-Teacher Bias via Latent Ensembles},
  author={Solstice-AI Research Team},
  journal={Solstice-AI Technical Report},
  year={2026},
  url={https://solstice-ai.co/papers/cross-architecture-divergence-latent-ensembles}
}