ByteDance Identifies DeepSeek’s Positional Sensitivity in KV Cache Compression
DeepSeek’s notorious “Jekyll-and-Hyde” behavior — where identical prompts yield wildly inconsistent outputs — has been demystified by ByteDance’s Seed research team. Their investigation reveals a fundamental architectural artifact: phase-dependent performance tied directly to DeepSeek-V4’s block-wise KV Cache compression.
🔍 The Core Discovery: 4-Token Periodic Instability
Researchers observed that DeepSeek-V4’s accuracy fluctuates systematically every 4 tokens, depending solely on the positional alignment of input information relative to compression window boundaries — a phenomenon they term Phase Sensitivity.
- In code completion tests, adding just 1–2 irrelevant leading tokens caused answer correctness to oscillate between 8 (correct) and 32 (incorrect).
- When padding length modulo 4 ≡ 0 or 1 → model favored wrong answer (avg. 71.3% confidence in “32”).
- When modulo 4 ≡ 2 or 3 → model favored correct answer (91.5% confidence in “8”).
✅ This is not noise — it’s a deterministic, architecture-induced periodicity.
🌊 “Needle-in-a-Haystack” Validation at Scale
To verify generality, Seed conducted rigorous 128K-context retrieval tests:
- Context: ~16,000 key-value pairs embedded in synthetic long text.
- Task: Retrieve value for a given key — identical query, identical context length, only positional offset varied.
📉 Accuracy Gap by Phase
| Model | Max Accuracy Delta |
|---|---|
| DeepSeek-V4-Flash-Base | +40.2 percentage points |
| DeepSeek-V4-Pro-Base | +34.8 percentage points |
| DeepSeek-V4-Flash-0731 (post-trained) | 19.1 pts |
| DeepSeek-V4-Pro-0813 | 14.8 pts |
| DeepSeek-V4.1-Flash-0910 | 6.1 pts (reduced — but phase bias persists) |
⚙️ Root Cause: Block-Wise KV Cache Compression
DeepSeek-V4 uses chunked KV compression to reduce memory footprint and compute cost during long-context inference. But this optimization introduces structural asymmetry:
- Tokens are grouped into fixed-size windows (e.g., step = 4).
- Within each window, Keys/Values are compressed via learned aggregation (e.g., weighted pooling).
- Crucially: information placed at different offsets inside the same window undergoes distinct compression pathways.
This leads to Phase Specialization: individual attention heads develop preferences for specific token positions within the compression block — enabling efficient retrieval on average, yet creating systematic weak spots.
🧪 Causal Evidence: Controlled Ablation & Synthetic Training
Seed trained controlled variants of Qwen3-0.6B, isolating compression design as the sole variable:
- ✅ All block-compressed models exhibited periodicity matching their compression step (4-token, 6-token, 8-token cycles).
- ❌ Full-attention baselines showed no such periodicity.
- 🔁 Varying window size without changing step did not shift the cycle — confirming step length drives periodicity.
- 🧠 Attention head analysis confirmed phase-specific contribution patterns, validating internal specialization.
📌 Implications for Evaluation & Deployment
- Benchmark Limitation: Standard aggregate scores mask severe positional variance — “good average” ≠ “robust everywhere”.
- New Evaluation Protocol: Models using block compression must be tested across all phases (i.e., multiple positional offsets per key).
- Architecture Trade-off: Compression enables scalability — but sacrifices uniform recall fidelity. Later versions (V4.1) narrow the gap via improved training, yet do not eliminate it.
- Practical Warning: For production RAG or code-assist systems, input formatting (e.g., docstring padding) may inadvertently trigger failure modes.
📚 Reference
- Paper: arXiv:2609.36322 — “Phase Sensitivity in Block-Compressed Transformers”
- Origin: Reported by QuantumBit, author Wen Le.
