Architecture Name
SiameseNorm&Depth-Attention
Parent issue
#1
Motivations
A standard OLMo 3 block uses a single residual stream. Although this design provides stable optimization, residual accumulation itself does not explicitly separate the identity-gradient path from the progressively transformed representation. SiameseNorm introduces two coupled residual streams with shared attention and MLP blocks, aiming to preserve a stable Pre-Norm-like path while maintaining a separately normalized representation path.
Standard self-attention performs content-dependent selection across token positions, but not across layers. Information from earlier layers reaches later layers only through residual accumulation. Depth-Attention allows the current query to select values from sparse earlier layers at the same token position before ordinary token attention.
The two changes target complementary mechanisms: SiameseNorm modifies residual and gradient propagation, while Depth-Attention adds explicit cross-layer feature selection. The hypothesis is that their combination improves effective depth and model quality over the OLMo 3 baseline.
Proposed Architecture
The model retains OLMo 3’s layer dimensions, Q/K normalization, RoPE, MLP, tokenizer, causal masking, and [SWA, SWA, SWA, Full] attention pattern.
SiameseNorm uses two forward-local residual streams. Attention and MLP parameters are shared and evaluated once. For each residual update Delta in layer l, the Post-Norm-like stream receives the depth-scaled update Delta/sqrt{2l}, while the Pre-Norm-like stream receives the full update. Attention input is formed from the normalized Pre-Norm-like stream and a learned channel-wise scaling of the Post-Norm-like stream. After the final layer, both streams are normalized and summed.
Depth-Attention is applied after Q/K normalization and RoPE, but before ordinary token-level attention. The current query scores the current key and keys from sparse earlier layers at the same token position. A depth-wise softmax then mixes the current value with recursively mixed earlier values. The initial configuration uses stride 8 and no additional recent-layer window.
The mixed value is passed into the original OLMo 3 SWA or Full-Attention operation. During inference, it replaces the current value in the existing KV-cache. Depth-Attention adds no learned parameters or additional persistent cache, although it introduces extra training activations and computation. SiameseNorm adds a small number of normalization and stream-scaling parameters.
Preliminary Results (if any)
Prior work has independently demonstrated the potential of both components. SiameseNorm reports improved optimization robustness and model quality through its dual-stream normalization design. Depth-Attention reports consistent improvements in perplexity and downstream accuracy across model scales from 360M to 3B, without additional learnable parameters or persistent KV-cache.
Their combination has not yet been systematically evaluated on OLMo 3. This proposal will study whether the two mechanisms provide complementary gains through controlled baseline comparisons at 1B, 3B, and 7B scales.
Experiments Plan
| Model size |
Baseline |
Proposed model |
| 1B |
OLMo 3 |
OLMo 3 + SiameseNorm + Depth-Attention |
| 3B |
OLMo 3 |
OLMo 3 + SiameseNorm + Depth-Attention |
| 7B |
OLMo 3 |
OLMo 3 + SiameseNorm + Depth-Attention |
Within each model-size pair, both runs will use the same model dimensions, tokenizer, training data and order, token budget, sequence length, optimizer, learning-rate schedule, batch size, random seed, distributed topology, and evaluation protocol. The only architectural difference will be the addition of SiameseNorm and Depth-Attention.
Comparisons will include training-loss and downstream evaluation results at matched token counts.
The proposal will be considered supported if the modified architecture shows a consistent quality improvement across the 1B, 3B, and 7B comparisons without introducing training instability, while its additional computation and memory costs remain practically acceptable.
Architecture Name
SiameseNorm&Depth-Attention
Parent issue
#1
Motivations
A standard OLMo 3 block uses a single residual stream. Although this design provides stable optimization, residual accumulation itself does not explicitly separate the identity-gradient path from the progressively transformed representation. SiameseNorm introduces two coupled residual streams with shared attention and MLP blocks, aiming to preserve a stable Pre-Norm-like path while maintaining a separately normalized representation path.
Standard self-attention performs content-dependent selection across token positions, but not across layers. Information from earlier layers reaches later layers only through residual accumulation. Depth-Attention allows the current query to select values from sparse earlier layers at the same token position before ordinary token attention.
The two changes target complementary mechanisms: SiameseNorm modifies residual and gradient propagation, while Depth-Attention adds explicit cross-layer feature selection. The hypothesis is that their combination improves effective depth and model quality over the OLMo 3 baseline.
Proposed Architecture
The model retains OLMo 3’s layer dimensions, Q/K normalization, RoPE, MLP, tokenizer, causal masking, and [SWA, SWA, SWA, Full] attention pattern.
SiameseNorm uses two forward-local residual streams. Attention and MLP parameters are shared and evaluated once. For each residual update Delta in layer l, the Post-Norm-like stream receives the depth-scaled update Delta/sqrt{2l}, while the Pre-Norm-like stream receives the full update. Attention input is formed from the normalized Pre-Norm-like stream and a learned channel-wise scaling of the Post-Norm-like stream. After the final layer, both streams are normalized and summed.
Depth-Attention is applied after Q/K normalization and RoPE, but before ordinary token-level attention. The current query scores the current key and keys from sparse earlier layers at the same token position. A depth-wise softmax then mixes the current value with recursively mixed earlier values. The initial configuration uses stride 8 and no additional recent-layer window.
The mixed value is passed into the original OLMo 3 SWA or Full-Attention operation. During inference, it replaces the current value in the existing KV-cache. Depth-Attention adds no learned parameters or additional persistent cache, although it introduces extra training activations and computation. SiameseNorm adds a small number of normalization and stream-scaling parameters.
Preliminary Results (if any)
Prior work has independently demonstrated the potential of both components. SiameseNorm reports improved optimization robustness and model quality through its dual-stream normalization design. Depth-Attention reports consistent improvements in perplexity and downstream accuracy across model scales from 360M to 3B, without additional learnable parameters or persistent KV-cache.
Their combination has not yet been systematically evaluated on OLMo 3. This proposal will study whether the two mechanisms provide complementary gains through controlled baseline comparisons at 1B, 3B, and 7B scales.
Experiments Plan
Within each model-size pair, both runs will use the same model dimensions, tokenizer, training data and order, token budget, sequence length, optimizer, learning-rate schedule, batch size, random seed, distributed topology, and evaluation protocol. The only architectural difference will be the addition of SiameseNorm and Depth-Attention.
Comparisons will include training-loss and downstream evaluation results at matched token counts.
The proposal will be considered supported if the modified architecture shows a consistent quality improvement across the 1B, 3B, and 7B comparisons without introducing training instability, while its additional computation and memory costs remain practically acceptable.