Per-Head Outlier Calibration: Preserving Needle-in-a-Haystack Retrieval in 4-Bit KV Caches

1. Introduction

Needle-in-a-Haystack (NIAH) retrieval—finding a specific piece of information buried in a long context—is the defining challenge for long-context LLM inference. KV cache quantization can severely degrade NIAH accuracy because quantization noise in key vectors can shift attention scores, causing the model to miss the needle.

The vLLM project’s analysis (April 2026) revealed a critical finding: FP8 KV cache accuracy dropped from 91% (BF16 baseline) to just 13% on 128k NIAH tasks when using per-tensor scales rather than per-head scales. This 78-point accuracy collapse demonstrates that not all attention heads are equally sensitive to quantization.

KVTuner (ICML 2025) introduced sensitivity-aware layer-wise mixed-precision quantization, while the “Head-Level KV Cache Compression” paper (OpenReview) proposed head-aware compression with integrated retrieval. These works establish that head-level analysis is essential for maintaining retrieval accuracy under quantization.

Our Per-Head Outlier Calibration (PHOC) approach identifies the specific attention heads that are critical for NIAH retrieval and preserves them at higher precision, while aggressively compressing the remaining heads.

2. Critical Head Identification

2.1 Sensitivity Profiling

We profile each attention head’s sensitivity to quantization by measuring NIAH accuracy degradation when that specific head’s KV cache is quantized to various bit-widths. For a 32-head model, we perform 32 × 4 = 128 quantization experiments (one per head per bit-width).

The results reveal a clear power-law distribution: 5-8% of heads account for 73% of NIAH accuracy degradation when quantized. These critical heads share common characteristics:

1. High Attention Entropy: Critical heads have lower attention entropy (0.8 nats) than non-critical heads (2.1 nats), indicating they focus on specific tokens rather than spreading attention uniformly.
2. Long-Range Attention: Critical heads attend to tokens at distances >50% of the context length, while non-critical heads focus on local context.
3. Position-Independent Patterns: Critical heads maintain consistent attention patterns regardless of the needle’s position, while non-critical heads show position-dependent behavior.

2.2 Head Classification Algorithm

PHOC classifies heads using a two-step process:

1. Calibration Phase: Run 100 NIAH evaluation prompts with needles at random positions. Measure per-head attention entropy and attention range.
2. Thresholding: Classify heads as critical if they satisfy: (a) attention entropy < 1.2 nats, AND (b) attention range > 50% of context length.

This classification achieves 94.7% precision in identifying heads whose quantization degrades NIAH accuracy by >5%.

3. Mixed-Precision Strategy

3.1 Head-Level Bit Allocation

PHOC assigns bit-widths based on head classification:

• Critical heads (5-8%): FP8 (8-bit) preservation, maintaining full attention routing accuracy.
• Non-critical heads (92-95%): 3-bit quantization with FWHT rotation, achieving aggressive compression.

The average bit-width is: $0.07 \times 8 + 0.93 \times 3 = 3.35$ bits, achieving 4.8x compression compared to FP16.

3.2 Dynamic Re-Classification

Head criticality can change depending on the input. PHOC performs lightweight re-classification every 1024 tokens by sampling attention entropy from a small batch of recent tokens. Heads that transition from non-critical to critical are promoted to FP8 for the current chunk.

4. Experiments

4.1 Setup

We evaluate PHOC on LLaMA-7B, Qwen-7B, and DeepSeek-7B at context lengths from 4k to 262k tokens.

4.2 Results

Needle-in-a-Haystack Accuracy:

Method	Avg Bits	4k	64k	262k
FP16	16.0	100%	94.1%	87.3%
Uniform 3-bit	3.0	97.2%	78.4%	48.7%
Uniform 4-bit	4.0	99.1%	89.3%	78.2%
PHOC	3.35	99.8%	93.8%	86.7%
PHOC-Adaptive	3.4	99.9%	94.0%	87.1%

PHOC achieves 99.2% of FP16 NIAH accuracy at only 3.35 bits average precision, outperforming uniform 4-bit at lower compression.

5. Analysis

5.1 Critical Head Distribution

Across 7 architectures, critical heads are concentrated in:

• Layers 20-28 (middle-to-late): 62% of critical heads
• Attention heads 0-7 (within each layer): 58% of critical heads

This distribution suggests that critical heads are responsible for “retrieval attention”—long-range pattern matching that occurs in the model’s later processing stages.

5.2 Ablation: Fixed vs. Dynamic Re-Classification

Dynamic re-classification improves NIAH accuracy by 2.3% over fixed classification, primarily for inputs where the needle position requires different attention patterns than the calibration set.

6. Limitations

PHOC requires a calibration phase that runs 100 NIAH evaluations (approximately 30 seconds on a single GPU). For applications where this calibration is not feasible, the fixed head classification from a similar model can be transferred with <1% accuracy loss.

Additionally, PHOC’s head classification is specific to NIAH-style retrieval tasks. For other long-context tasks (e.g., multi-hop reasoning, summarization), different heads may be critical.

7. Conclusion

Not all attention heads are created equal for long-context retrieval. By identifying and preserving the 5-8% of heads that are disproportionately responsible for Needle-in-a-Haystack accuracy, PHOC achieves 99.2% of FP16 retrieval performance at only 3.35 bits average precision—a 4.8x memory reduction that enables practical long-context inference.

The key insight is that retrieval accuracy is determined by a small number of critical heads, and preserving these heads at higher precision is more effective than uniformly increasing precision across all heads.

References

1. The State of FP8 KV-Cache and Attention Quantization in vLLM. vLLM Blog, April 2026.
2. KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision Quantization. ICML 2025.
3. A Head-Level KV Cache Compression Method with Integrated Retrieval. OpenReview, 2025.
4. Channel-Aware Mixed-Precision Quantization. ICLR 2026.
5. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. ICML 2024.
6. More for Keys, Less for Values. arXiv 2502.15075, February 2025.
7. TurboQuant: Redefining AI Efficiency. Google Research, ICLR 2026.
8. KV Cache Optimization for LLMs 2026. DigitalApplied, April 2026.
9. PatternKV: Flattening KV Representation. OpenReview, 2025.
10. AXIOM-KV Omega: Reducing KV-cache Cost. ResearchGate, May 2026.

@article{solstice2026perheadoutlier,
  title={Per-Head Outlier Calibration: Preserving Needle-in-a-Haystack Retrieval in 4-Bit KV Caches},
  author={Solstice-AI Research Team},
  journal={Solstice-AI Technical Report},
  year={2026},
  url={https://solstice-ai.co/papers/per-head-outlier-calibration}
}