Empirical Limits of Cross-Architecture Reasoning Transfer in Sub-8B Student LLMs
Distilling long chain-of-thought reasoning from frontier teacher architectures into dense sub-8B students poses unique representational challenges. When student architectures differ fundamentally in layer depth, attention head count, or rotary embedding configurations, direct logit matching degrades generalization. In this report, we present an empirical study of cross-architecture reasoning transfer across 50,000 algorithmic benchmarks, demonstrating that intermediate scratchpad supervision with consensus rejection sampling outperforms traditional token-level KL divergence minimization.
1. Abstract & Motivation
When distilling dense reasoning models, a fundamental question arises: Can a compact 3.8B–7B parameter model genuinely internalize the non-linear deduction capabilities of a 100B+ MoE teacher, or is it merely mimicking surface-level phrasing?
We evaluate three distinct distillation methodologies:
- Token-Level KL Divergence (Standard KD): Minimizing logit divergence over intermediate tokens.
- Hidden State Alignment: Forcing intermediate student representations to match projected teacher layer outputs.
- Step-Level Consensus SFT: Supervising the student on multi-teacher verified chain-of-thought scratchpads with automated verification filters.
2. Key Findings
- Outlier Attention Saturation: In single-teacher KD, the student model overfits to high-magnitude attention spikes unique to the teacher’s RoPE frequency base.
- Multi-Teacher Regularization: Aggregating thought chains from diverse models (dense, MoE, and long-context architectures) acts as a natural regularizer, preventing student attention collapse on edge cases.
- Inference Latency Decoupling: Step-level distilled students generate compact, efficient reasoning paths that are 22% shorter than raw teacher outputs without sacrificing proof accuracy.
@article{solstice2026crossarchitecture,
title={Empirical Limits of Cross-Architecture Reasoning Transfer in Sub-8B Student LLMs},
author={Solstice-AI Research Team},
journal={Solstice-AI Technical Report},
year={2026},
url={https://solstice-ai.co/papers/cross-architecture-reasoning-transfer}
}