Beyond RNNs: Why LSTM and GRU Lost to Transformers and What Replaced Them

For nearly a decade, if you worked on sequence modeling, you reached for an LSTM or GRU without thinking twice. I shipped my first production LSTM in 2017 for a speech recognition pipeline, and at the time it felt like the definitive solution to handling temporal dependencies. Two years ago, I deprecated the last LSTM service I was responsible for. The reason wasn't that LSTMs stopped working — they still work remarkably well for their design — but that the constraints around them changed completely. Hardware, data scale, and the definition of what "long-range dependency" means all shifted. Understanding why recurrent architectures fell out of favor, and what is now replacing them beyond pure attention, is essential if you are building sequence models today. This is not a story of one good idea replacing a bad one, but of specific engineering trade-offs that made one family of models dramatically more scalable than the other.

The Gating Trick: How LSTMs Solved Vanishing Gradients and Created a New Bottleneck

Standard RNNs fail for a well-understood mathematical reason: backpropagation through time multiplies the same weight matrix repeatedly. For long sequences, gradients either vanish to zero or explode exponentially, which makes learning dependencies beyond 20-30 time steps almost impossible. In my experience debugging early RNN training runs, you would watch the gradient norms collapse after a few epochs and the model would effectively become a bigram model, ignoring anything more than a few tokens back.

The Long Short-Term Memory network, introduced by Hochreiter and Schmidhuber, addressed this with an elegant control mechanism. Instead of forcing information through a single tanh activation at each step, the LSTM maintains a separate cell state — a kind of conveyor belt — that is protected by three learned gates. The forget gate decides what to discard, the input gate decides what new information to write, and the output gate decides what to expose as the hidden state. Because the cell state update is additive rather than multiplicative (c_t = f_t * c_{t-1} + i_t * g_t), gradients can flow back with far less attenuation.

Dissecting the LSTM Cell in Code

When you implement an LSTM from scratch you see exactly where the capacity and the cost come from. Each gate has its own weight matrices for the input and hidden state, which means an LSTM has roughly four times the parameters of a vanilla RNN for the same hidden size.

import torch
import torch.nn as nn

class ManualLSTMCell(nn.Module):
    def __init__(self, input_size, hidden_size):
        super().__init__()
        # Combined projection for all 4 gates: i, f, g, o
        self.W_ih = nn.Linear(input_size, 4 * hidden_size)
        self.W_hh = nn.Linear(hidden_size, 4 * hidden_size)

    def forward(self, x_t, h_prev, c_prev):
        gates = self.W_ih(x_t) + self.W_hh(h_prev)
        i_gate, f_gate, g_gate, o_gate = gates.chunk(4, dim=-1)

        i_t = torch.sigmoid(i_gate)   # input gate
        f_t = torch.sigmoid(f_gate)   # forget gate
        g_t = torch.tanh(g_gate)      # cell candidate
        o_t = torch.sigmoid(o_gate)   # output gate

        c_t = f_t * c_prev + i_t * g_t
        h_t = o_t * torch.tanh(c_t)
        return h_t, c_t

# In practice you would use the fused CUDA kernel:
# lstm = nn.LSTM(input_size=128, hidden_size=256, batch_first=True)
# But the manual version shows why it is expensive: 4 large matmuls per step.

I've found that careful initialization of the forget gate bias is one of the few tricks that still matters. Jozefowicz et al. showed that initializing the forget bias to 1.0 or even 2.0 helps the model remember longer at the start of training. Without it, your LSTM will often learn to forget everything early on and never recover. Even with that fix, the fundamental bottleneck remains: to compute step t, you must have finished step t-1. That sequential dependency is what ultimately limited scale, no matter how clever the gating became. For a refresher on how gradients are managed in deep sequential networks, the techniques covered in Training Deep Networks: Learning Rate Schedules, Batch Normalization and Gradient Clipping still apply directly to stabilizing LSTM training.

GRU's Minimalist Gamble: Fewer Gates, Faster Training, Hidden Trade-offs

The Gated Recurrent Unit, proposed by Cho et al. in 2014, asked a pragmatic question: do we actually need three gates and a separate cell state? The GRU merges the cell state and hidden state and reduces the control logic to two gates: a reset gate and an update gate. The update gate z_t acts like a learned interpolation coefficient between the previous hidden state and a new candidate state. The reset gate r_t determines how much of the past to ignore when proposing the candidate.

On paper, this looks like a clean win. A GRU has about 25% fewer parameters than an LSTM with the same hidden size, it runs faster, and it needs less memory. When I benchmarked GRUs for an on-device keyword spotting model in 2019, they trained about 30% faster than LSTMs and reached the same accuracy on sequences under 200 steps. For many teams working with limited GPU budgets at the time, that was enough reason to switch.

Where GRUs Fall Short in Practice

class ManualGRUCell(nn.Module):
    def __init__(self, input_size, hidden_size):
        super().__init__()
        self.W_ih = nn.Linear(input_size, 3 * hidden_size)
        self.W_hh = nn.Linear(hidden_size, 3 * hidden_size)

    def forward(self, x_t, h_prev):
        gates_x = self.W_ih(x_t)
        gates_h = self.W_hh(h_prev)
        
        r_gate = torch.sigmoid(gates_x[:, :self.hidden_size] + gates_h[:, :self.hidden_size])
        z_gate = torch.sigmoid(gates_x[:, self.hidden_size:2*self.hidden_size] + gates_h[:, self.hidden_size:2*self.hidden_size])
        
        n_gate_x = gates_x[:, 2*self.hidden_size:]
        n_gate_h = r_gate * gates_h[:, 2*self.hidden_size:]
        n_t = torch.tanh(n_gate_x + n_gate_h)
        
        h_t = (1 - z_gate) * n_t + z_gate * h_prev
        return h_t

The trade-off becomes visible on tasks that require precise, long-term memory control. Because the GRU lacks a separate cell state shielded by an output gate, it cannot easily decide to remember a fact for 500 steps without also exposing it in the hidden state at every intermediate step. I've seen this cause subtle failures in document-level sentiment and in time-series forecasting where a rare event early in the sequence must be carried forward without contaminating the local dynamics. LSTMs, with their decoupled cell state, handle that "store but don't show" pattern more naturally. For short to medium sequences (under ~300 tokens) I still consider the GRU a valid, efficient choice. For anything requiring counting, copying, or precise long-range slot-filling, I've consistently measured a 1-3% advantage for LSTMs, which was why both co-existed for so long rather than one outright replacing the other.

The Sequential Curse: Why No Amount of Tuning Could Make RNNs Scale Like Transformers

The defeat of RNNs was not primarily about accuracy on a 500-token sequence. With enough tuning, a stacked bidirectional LSTM could match a small Transformer on many benchmarks from 2018-2019. The defeat was about systems. Transformers removed the temporal dependency that made RNNs fundamentally un-parallelizable during training.

Consider training on a sequence of length N. An RNN must execute N sequential steps. You cannot compute hidden state h_10 until you have h_9. On a modern accelerator that thrives on large matrix multiplications, this is a disaster. You get low arithmetic intensity and you cannot fill the GPU. A Transformer, by contrast, computes attention across all N positions simultaneously with large, batched matmuls. The cost is O(N^2) in memory and compute for attention, but it is fully parallel. In my last large-scale comparison, training a 1.5B parameter Transformer on 8k context was roughly 12x faster in wall-clock time than an equivalent capacity stacked LSTM on the same hardware and data, simply due to utilization.

This also changed the optimization landscape. RNNs require backpropagation through time (BPTT), which is inherently sequential and must store intermediate states or recompute them. Gradient clipping, which is almost mandatory for RNNs, becomes a global synchronization point. Transformers, while needing their own tricks like warmup and LayerNorm placement, train with a much more stable, parallelizable backward pass. If you want to understand the mechanics that made that parallelism possible, Transformer Architecture from Scratch: Attention, Encoders and Decoders breaks down the attention implementation that replaced recurrence.

The second, subtler effect was that Transformers handled long-range dependencies without compressing everything into a fixed-size vector. An LSTM must squeeze the entire history of a 2000-token document into a 1024-dimensional vector. Information is inevitably lost. Attention keeps the full history accessible and learns to retrieve it. When datasets grew to billions of tokens and contexts grew to tens of thousands of tokens, that bottleneck became impossible to ignore. This is why even hybrid approaches from 2020 that tried to stack LSTMs and attention layers eventually converged to pure attention — the recurrent part was contributing very little while blocking scale.

Selective State Spaces: Inside Mamba's Return to Recurrence

For two years, the field assumed the future was just "more efficient attention" — sparse attention, FlashAttention, sliding windows. State Space Models (SSMs) took a different path: fix recurrence itself. Classical SSMs are continuous-time systems described by h'(t) = A h(t) + B x(t), y(t) = C h(t), discretized for deep learning. They are inherently recurrent and have O(N) complexity, but early versions like S4 were linear time-invariant — their parameters A, B, C were fixed for all time steps. That made them fast but poor at content-based reasoning; they couldn't decide to remember or ignore a specific token based on its content.

Mamba (Gu and Dao, 2023) introduced selectivity. The key idea is to make B, C, and the discretization step size delta functions of the input. In practice, this means the model can selectively propagate or forget information along the sequence depending on what it sees. A token like "remember this name" can open the state to write, while filler tokens can be ignored. This recovers the content-aware gating behavior of LSTMs but in a form that can be trained in parallel using a parallel associative scan.

A Sketch of the Selective Scan

The implementation relies on a hardware-aware algorithm that avoids materializing the full expanded state in HBM. According to the recent work on arXiv Machine Learning and the original Mamba paper, the selective scan is fused into a single kernel that computes the recurrence in SRAM. The PyTorch implementation reflects this IO-aware design, as detailed in the PyTorch Documentation for custom kernels.

# Simplified conceptual Mamba block - not the fused kernel
# Real Mamba uses parallel scan and discretization

def selective_ssm_block(x, A, B, C, delta):
    # x: (batch, seq_len, d_model)
    # delta, B, C are input-dependent: delta = softplus(Linear(x))
    batch, seq_len, d_inner = x.shape
    
    # Discretize continuous parameters
    dA = torch.exp(delta * A)  # (batch, seq_len, d_inner)
    dB = delta * B             # (batch, seq_len, d_inner)
    
    h = torch.zeros(batch, d_inner, device=x.device)
    outputs = []
    for t in range(seq_len):  # In real Mamba this loop is a parallel scan
        h = dA[:, t] * h + dB[:, t] * x[:, t]
        y_t = (C[:, t] * h).sum(dim=-1)
        outputs.append(y_t)
    
    y = torch.stack(outputs, dim=1)
    return y * torch.sigmoid(x)  # gated output

# What makes it trainable in parallel:
# h_t = dA_t * h_{t-1} + dB_t * x_t is a linear recurrence
# which can be computed with an associative scan in O(log N) parallel steps.
# Inference remains O(1) per token like an RNN, with no KV-cache.

What I find compelling in practice is the inference profile. Unlike Transformers where the KV-cache grows linearly with sequence length and eventually dominates memory and latency, Mamba maintains a constant-size state. I recently tested a 2.8B Mamba model on 100k token context for genomic sequence modeling — something that would have required ~40GB of KV-cache with a Transformer — and it ran on a single 24GB card with flat latency per token. Variants like Mamba-2 and hybrid Mamba-Transformer architectures (e.g., Jamba, Griffin) now combine selective SSM layers for most of the sequence with a few attention layers for precise retrieval, which I've found gives the best trade-off for tasks that need both long-context compression and exact copy capabilities.

Production Reality Check: Where Recurrent Models Still Earn Their Keep in 2026

It would be a mistake to read this history as "never use RNNs again." In production, the right answer is still dictated by latency, memory, and deployment constraints, not just benchmark scores. In my experience, there are three situations where I still deliberately choose a recurrent or SSM-based model over a Transformer.

First, streaming and real-time edge deployment. If you are doing keyword spotting, ECG anomaly detection, or predictive maintenance on a microcontroller or a phone, you cannot afford a Transformer's KV-cache or its non-causal attention. A quantized GRU with 256 hidden units can run in under 5ms on an ARM Cortex-M with a few kilobytes of RAM. I shipped an audio denoising model last year where a 2-layer GRU outperformed a tiny Transformer because the Transformer needed a 200ms lookahead window that violated our latency budget. For framework choices in these constrained settings, the comparison in PyTorch vs TensorFlow in 2026: Which Framework for Which Use Case is surprisingly relevant — TensorFlow Lite's GRU delegate support is still more mature on some edge chips.

Second, extremely long but low-information-density sequences. Think raw sensor time series at 1kHz, DNA sequences, or high-frequency trading data where sequences can be millions of steps long and attention's O(N^2) cost is prohibitive. Selective SSMs dominate here. A Transformer would need chunking and hierarchical tricks that re-introduce the very information loss we tried to avoid. A Mamba or S4 model can process the raw sequence linearly.

Third, ultra-low-budget fine-tuning. If you have a small dataset (5k-20k examples) and a single GPU, fine-tuning a 7B Transformer may be overkill and prone to overfitting. A bidirectional LSTM with pretrained embeddings, trained with proper regularization, can often reach 95% of the Transformer's performance in a fraction of the time and cost. I have used this as a strong baseline for clients who needed a model in two days, not two weeks.

The practical decision framework I use now is simple: If your sequence is under 1k tokens, latency is not critical, and you have access to pretrained Transformers, use a Transformer. If your sequence is 1k-100k and inference cost matters, evaluate a hybrid Mamba-Transformer. If your sequence is streaming, infinitely long, or must run on the edge with constant memory, use a GRU/LSTM or a pure SSM. The era when one architecture won everywhere is over; the skill now is knowing which bottleneck you are actually optimizing for.

Frequently Asked Questions