TempatGunting

Sparse Attention Patterns for Long-Sequence Transformers

Dense Attention Doesn't Scale

Standard self-attention processes every pair of tokens. For a sequence of 1024 tokens, that's about a million attention scores per head. Bump it to 16384 tokens and you're at 268 million. At 100K tokens, it's 10 billion. The O(n²) cost hits a wall somewhere between 4K and 16K tokens on current GPUs, and sparse attention patterns are how you push past it without approximating the computation.

I should be clear about what "sparse" means here. We're not talking about pruning attention weights after they're computed — we're talking about deciding in advance which token pairs will attend to each other, and never computing attention scores for the rest. The sparsity pattern is structural, baked into the architecture.

Local Window Patterns

The simplest sparse pattern: each token attends only to tokens within a fixed window of size W around it. Token at position i attends to positions [i-W/2, i+W/2]. This drops attention cost from O(n²) to O(n×W) and is the foundation of models like Mistral and most efficient long-context architectures.

ML Model Deployment Strategies
Strategy Latency Cost Best For
Real-time API50–200 msHigh (always-on)Interactive UX
Batch InferenceMinutes–hoursLow (spot instances)Scheduled pipelines
Edge/On-device5–50 msPer-device hardwarePrivacy, offline
Serverless (Lambda)100–500 msPay-per-requestSporadic traffic

The implementation with FlashAttention is straightforward — you pass a block mask that defines the window boundaries. Newer versions of FlashAttention (v2.5+) have native support for block-sparse patterns that map directly to GPU thread-block structure.

from torch.nn.attention import BlockMask

def sliding_window_mask(seq_len, window_size):
 def mask_fn(b, h, q_idx, kv_idx):
 return (q_idx - kv_idx).abs() <= window_size // 2
 return BlockMask.from_function(mask_fn, Q_LEN=seq_len, KV_LEN=seq_len)

# Use with scaled_dot_product_attention
mask = sliding_window_mask(seq_len=8192, window_size=4096)
output = F.scaled_dot_product_attention(q, k, v, attn_mask=mask)

Strided (Dilated) Attention

Local windows capture nearby context but miss long-range dependencies. Strided attention addresses this by having some heads attend to every S-th token across the full sequence. If S=256 and the sequence is 16K tokens, each token attends to 64 globally-spaced positions.

This connects to the ideas in Batch Normalization vs Layer Normalization in Production Tra.

The combination of local and strided attention in alternating heads gives you both fine-grained local context and coarse global context. This was the approach in the original Sparse Transformer paper, and it's still effective. In my experiments, local + strided attention at total cost O(n√n) recovers 97-99% of dense attention quality on language modeling tasks.

Global Token Patterns

Longformer and BigBird designate certain tokens as "global" — they attend to every other token and every other token attends to them. Typical global tokens include the [CLS] token, sentence boundaries, and any special delimiter tokens.

For document understanding tasks, I've found that making section headers global tokens (instead of just [CLS]) significantly improves retrieval accuracy within long documents. The headers act as routing nodes that aggregate and distribute information across sections.

class GlobalLocalAttention(torch.nn.Module):
 def __init__(self, hidden_dim, num_heads, window_size, num_global_tokens):
 super().__init__()
 self.hidden_dim = hidden_dim
 self.num_heads = num_heads
 self.window_size = window_size
 self.num_global = num_global_tokens
 self.qkv = torch.nn.Linear(hidden_dim, 3 * hidden_dim)
 self.out = torch.nn.Linear(hidden_dim, hidden_dim)

 def forward(self, x, global_mask):
 B, S, _ = x.shape
 qkv = self.qkv(x).reshape(B, S, 3, self.num_heads, -1).permute(2, 0, 3, 1, 4)
 q, k, v = qkv[0], qkv[1], qkv[2]

 # Global tokens: full attention
 global_out = F.scaled_dot_product_attention(
 q[:, :, :self.num_global],
 k, v
 )

 # Local tokens: windowed attention + attention to global tokens
 # (simplified — real implementation uses block-sparse kernels)
 local_q = q[:, :, self.num_global:]
 local_k_window = self._extract_windows(k, self.window_size)
 local_v_window = self._extract_windows(v, self.window_size)

 # Concatenate global KV with local window KV
 global_k = k[:, :, :self.num_global].unsqueeze(2).expand(-1, -1, S - self.num_global, -1, -1)
 combined_k = torch.cat([global_k, local_k_window], dim=-2)
 # ... similar for V, then compute attention

 return self.out(output)

Hierarchical Attention

Instead of attending over individual tokens at all scales, hierarchical attention operates at multiple resolutions. Fine-grained attention handles local neighborhoods. Coarse attention operates on compressed representations of larger blocks. This is similar to how image processing uses multi-scale pyramids.

We covered a related topic in Canary Deployments for ML Models: Traffic Splitting and Roll.

The implementation typically pools or compresses chunks of the sequence into summary tokens, then runs global attention over the summaries. Information flows from fine to coarse during encoding and from coarse to fine during decoding.

I've used hierarchical attention for processing entire research papers (50K+ tokens). The fine-grained level handles sentence-level semantics with a 512-token window, and the coarse level handles paragraph-level structure with one summary token per paragraph. The result processes 50K tokens with the memory footprint of 4K-token dense attention.

Hash-Based Attention (Reformer)

Reformer uses locality-sensitive hashing to group tokens that are likely to have high attention scores. Tokens are hashed into buckets, and attention is computed only within each bucket. The idea is clever — similar tokens hash to the same bucket, so you attend to the tokens that matter most.

In practice, I've found LSH attention finicky to tune. The number of hash rounds affects both quality and speed, and the optimal setting varies by task. For autoregressive language modeling, 4-8 rounds works. For bidirectional encoding tasks, you might need 16+. And the hashing overhead means it's only faster than dense attention for sequences longer than about 8K tokens.

This connects to the ideas in Data Augmentation Strategies That Actually Improve Model Rob.

Choosing a Pattern for Production

If you need a working solution today, use sliding window attention. It's implemented in every major framework, well-tested, and handles 90% of long-context use cases. The information propagation through layers covers the long-range dependency gap for most practical applications.

If you're processing structured documents where you know the segmentation (headers, paragraphs, code blocks), global-local attention with structural global tokens is worth the implementation effort. I've seen 5-15% improvement on document QA benchmarks compared to pure sliding window.

If you're doing research and want to push context length past 100K tokens, hierarchical attention or ring attention (which distributes dense attention across GPUs) are the current frontiers. Both require custom kernel development for good throughput, so budget engineering time accordingly.

Avoid hash-based attention unless you have a specific reason to try it. The tuning burden and marginal throughput improvements over simpler patterns don't justify the complexity for most production use cases.

References & Standards: NIST AI Risk Management · ACM Digital Library