QuATernaryons: Non-Euclidean 4D Quaternion Rotor Attention on S³ for Frontier Language & Multimodal Transformers
Standard dot-product attention in Transformer architectures relies on Euclidean projection manifolds that suffer from rotational variance, dimensional collapse, and phase drift under extended context horizons. We introduce QuATernaryons, an architectural paradigm that embeds token latent vectors into the 4D Hamilton quaternion algebra H and maps attention weights through rotor rotations on the hypersphere S³. By preserving norm preservation under SO(4) subgroup rotations, QuATernaryons eliminates rotational phase drift and achieves strict non-Euclidean geometric invariance. Verified and published via CERN Zenodo (DOI: 10.5281/zenodo.23530592) and indexed in OpenAIRE.
Executive Summary
QuATernaryons (v0.2.0) introduces a mathematically rigorous formulation of quaternion-valued attention heads for deep neural networks.
Instead of representing queries, keys, and values as unconstrained vectors in $\mathbb{R}^d$, QuATernaryons groups latent channels into Hamilton quaternions:
$$q = w + x\mathbf{i} + y\mathbf{j} + z\mathbf{k} \in \mathbb{H}$$
Attention scoring is computed via rotor conjugation:
$$R(v) = q v q^{-1}$$
This ensures strict isometric norm-preservation on the unit hypersphere $S^3$, completely resolving phase-drift phenomena observed in 1M+ context window extrapolations.
Key Innovations
- $S^3$ Hyperspherical Manifold Projection: Guarantees unit-norm stability across arbitrary depth without requiring post-hoc layer norm crutches.
- Hamiltonian Rotor Kernel: Evaluates pairwise cross-token interactions through 4D spatial rotations rather than scalar projections.
- Anvil Metal/CUDA Integration: Custom non-Euclidean GEMM kernels implemented in Project Anvil for hardware-accelerated local execution.
Citation & Permanent DOI
Published and archived at CERN Zenodo under Concept DOI 10.5281/zenodo.23043500 and Version DOI 10.5281/zenodo.23530592. Verified author identity via ORCID: 0009-0000-1273-1458.
@article{gondaliya2026quaternaryons,
title={QuATernaryons: Non-Euclidean 4D Quaternion Rotor Attention on S³ for Frontier Transformers},
author={Gondaliya, Shreyan Dushyant},
journal={CERN Zenodo},
year={2026},
doi={10.5281/zenodo.23530592},
url={https://doi.org/10.5281/zenodo.23530592}
}