The Complete Guide to Moonshot AI AttnRes: A New Paradigm for Redesigning Transformer Residual Connections

1. Problem Definition: Hidden Bottleneck in Transformer Residual Connection

Target audience: ML engineers, deep learning researchers, AI infrastructure personnel training or deploying large-scale language models

Problem solved: Three structural limitations that arise as each layer output is accumulated with a fixed weight (1.0) in a PreNorm transformer

• Selective access not possible: All layers receive the same mixed state, so attention layer and FFN layer cannot be distinguished even if they require different previous information
• Irreversible information loss: Information once mixed in the residual stream cannot be selectively recovered in subsequent layers
• Increasing output size:Deeper layers generate larger outputs to maintain influence, causing learning instability

Scope of application:LLM, MoE architecture, pipeline parallelization environment over billions of parameters
Not applicable: Small models under 10M, cases where architecture change is not possible in a fixed inference pipeline

2. Evidence and Comparison: Full AttnRes vs Block AttnRes

The key insight of Moonshot AI is that just as attention succeeded in replacing fixed circulation in sequence modeling, the same principle can be appliedin the network depth direction.

Comparison of implementation methods

Comparison of scaling laws

Scaling formula measured by Moonshot AI across 5 model sizes:

• Baseline (PreNorm): L = 1.891 × C^-0.057
• Block AttnRes: L = 1.870 × C^-0.058
• Full AttnRes: L = 1.865 × C^-0.057

Key numbers: Block AttnRes reduces approximately 1.25xcomputing required to achieve the same loss. In large-scale learning, it directly leads to a cost difference of millions of dollars.

3. How to implement it step by step: Block AttnRes Introduction Guide

Step 1: Preparing the environment

#Official repository clone
git clone https://github.com/MoonshotAI/Attention-Residuals.git
cd Attention-Residuals

#Install dependencies (PyTorch 2.0+ recommended)
pip install -r requirements.txt

Step 2: Determine the number of blocks

Moonshot AI recommended value: N = 8 (optimal balance between mHC standard and memory overhead)

#config.yaml example
attnres:
  enabled: true
  variant: block  #'full' or 'block'
  num_blocks: 8   #Split the layer into 8 blocks
  init_weights: zero  #Zero initialization for initial equal weights

Step 3: Modify model architecture

from attnres import BlockAttnResLayer

class TransformerWithAttnRes(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.embed = nn.Embedding(config.vocab_size, config.d_model)
        
        #Block AttnRes layer initialization
        self.attnres = BlockAttnResLayer(
            d_model=config.d_model,
            num_layers=config.num_layers,
            num_blocks=8,  #Recommended value
            use_rms_norm=True  #Output size normalization required
        )
        
        self.layers = nn.ModuleList([
            TransformerLayer(config) for _ in range(config.num_layers)
        ])

Step 4: Integrate learning loops

def forward(self, x):
    h = self.embed(x)
    layer_outputs = [h]  #Include embedding as first source
    
    for i, layer in enumerate(self.layers):
        #AttnRes: Depth direction attention to previous layers
        h_input = self.attnres(layer_outputs, layer_idx=i)
        h = layer(h_input)
        layer_outputs.append(h)
    
    return h

Step 5: Pipeline parallelization settings (optional)

#Minimize inter-stage overhead with cache-based communication
pipeline_config:
  attnres_cache:
    enabled: true
    compression: block_summary  #Transmit only block-by-block summaries
    async_prefetch: true

4. Pitfalls: Precautions during introduction

Trap 1: Missing zero initialization

Symptoms: Loss explosion or convergence failure in the early stages of learning

Cause: When pseudo-query vector is randomly initialized, the weight is biased to a specific layer

Solution: Initialize all pseudo-queries to 0 so that the initial attention weight is evenly distributed

self.pseudo_query = nn.Parameter(torch.zeros(d_model))

Trap 2: Omit RMSNorm

Symptom: In deep layers, attention weight is fixed to a specific source

Cause: Layers with large output dominate depth direction attention

Solution: Apply RMSNorm to all layer outputs and then calculate attention

Trap 3: Excessive block count

Symptoms:Surge in memory usage, pipeline communication bottleneck

Cause: The larger N, the better the performance, but the communication/memory cost increases linearly

Solution:Start based on N=8, adjust according to ablation results. In the Moonshot AI experiment, 8 out of N ∈ {2,4,8} is optimal

Pit 4: Existing checkpoint incompatibility

Symptom:Key mismatch error when loading pretrained model

Cause: AttnRes parameter not in existing checkpoint

Solution: Initialize only the AttnRes parameter randomly (or zero) and load the rest with existing weights

model.load_state_dict(checkpoint, strict=False)
model.attnres.apply(init_attnres_weights)

5. Implementation checklist: Pre-production inspection

• ☐ CUDA memory profiling completed in PyTorch 2.0+ environment
• ☐ Block AttnRes initialized to num_blocks=8
• ☐ Check pseudo-query vector zero initialization
• ☐ Confirm application of RMSNorm to layer output
• ☐ Validation loss comparison experiment completed at a scale of 1T tokens or less
• ☐ Check cache-based communication settings when parallelizing pipelines
• ☐ Tested for compatibility with existing checkpoints

Definition of Done: Block AttnRes The applied model should achieve a 2-3% improvement in validation loss compared to the baseline at the same computing budget, and the increase in inference latency should be within 2%.

6. References

• Moonshot AI - Attention Residuals Paper (March 2026)
• Moonshot AI - Attention Residuals Official GitHub Repository
• MarkTechPost - Moonshot AI Releases Attention Residuals (March 15, 2026)
• Hugging Face Papers - Attention Residuals

7. Author Viewpoint

If recommended

• 10B+ parameter model training: In an environment where computing costs are in the millions of dollars, a 1.25x efficiency increase is a direct cost savings
• MoE Architecture: Kimi Linear (48B total/3B active) Proven performance improvements in the same MoE
• Long-term research investment: Potential to serve as a basis for subsequent architectural improvements through fundamental redesign of residual connections

If not recommended

• Small model (1B or less): Small improvement in efficiency compared to overhead
• Service where inference latency is the top priority: Real-time service where even 2% additional latency is not acceptable
• Existing learning pipeline cannot be modified: Requires architecture level change, so cannot be applied in black box learning environment

When other choices are better

If you only need inference efficiency, AI2 Olmo Hybrid (transformer + linear cyclic hybrid) provides double data efficiency, and if you want learning efficiency without changing the architecture, Mixture-of-Depths Attention(MoDA) can be an alternative.

Final Judgment: AttnRes is a meaningful study that revisits one of transformer's oldest design decisions. For teams planning large-scale learning, Block AttnRes is worth including in pilot experiments. However, you must verify overhead and performance improvement in your own workload before applying it to production.