H3: The Memory and Comparison Capabilities SSMs Missed in Language Modeling
State space models (SSMs) can scale almost linearly with sequence length, but that alone does not replace attention in language modeling. Hungry Hungry Hippos: Towards Language Modeling with State Space Models does not reduce this gap to vague insufficient expressiveness; it divides the problem into two capabilities. Existing SSMs are weak at accurately recalling earlier tokens and at comparing tokens with one another within a sequence. Building on that diagnosis, the paper proposes the H3 layer, the FlashConv computation method, and hybrid models that retain some attention.
This article is based only on the arXiv abstract and public metadata. The figures and model sizes below are results reported by the paper’s abstract; the detailed experimental settings and implementation reproducibility require checking the full paper.
Decomposing the Problem into Two Capabilities
The paper explains that it first investigated the expressiveness gap between SSMs and attention through a synthetic language modeling task. It found that existing SSMs struggled with two things.
- Recalling an earlier token — preserving a specific past token in the state when it is needed for the current prediction.
- Comparing tokens — calculating the relationship between the current token and another token in the sequence.
This decomposition matters because it prevents “long dependency” from being treated as a single number. The existence of a long-range signal is different from needing to find and compare a particular item among those signals. A compressed SSM state may be suitable for accumulating the former, while selective interactions of the latter kind may require a different structure.
Plain definition: The range of patterns a model can distinguish in its input and recall later.
Example: A summary note may contain the overall plot but still be unable to answer whether a particular word on page 20 is the same as a word on page 80. That difference is a problem of expressivity.
H3: Deliberately Designing for Recall and Comparison
H3 is an SSM layer newly designed to target the two capabilities language requires. The abstract reports that H3 matches attention on synthetic language tasks and comes within 0.4 PPL of a Transformer on OpenWebText. The important point is that H3 is not presented simply as an approach that uses a larger state. According to the paper’s explanation, the layer design reflects the linguistic requirements of which information to preserve and which relationships to compute.
However, the abstract does not provide every gate, branch, or parameter equation inside H3, nor the contribution of each component. This article therefore does not assert internal implementation details beyond the level of an “SSM layer designed for recall and comparison.”
Plain definition: A metric indicating how confidently a language model predicts the next token; lower is usually better.
Example: In a fill-in-the-blank exercise, confidently narrowing the answer to one or two possibilities produces a low score, while hesitating among dozens produces a high score.
FlashConv: Between Algorithms and Hardware
The second obstacle identified by the paper is hardware utilization. An SSM can be nearly linear in sequence length and still be slower than a Transformer, because asymptotic operation counts alone do not explain actual execution time. The H3 paper proposes FlashConv: for lengths up to 8K, it uses a fused block FFT algorithm, while for longer inputs the abstract says it introduces a state-passing algorithm that exploits the recurrent nature of SSMs.
The abstract reports a 2× speedup on the Long Range Arena and generation up to 2.4× faster than a Transformer for a hybrid language model. These numbers are results under the benchmark and implementation conditions measured by the paper, not “2× in every environment.” In a real service, GPU generation, batch size, precision, kernel support, and input/output lengths must be fixed together for comparison.
Plain definition: An implementation that combines several computation stages into one hardware kernel to reduce memory round trips.
Example: Instead of moving ingredients between a cutting board and a bowl several times, mix them in one operation, like preparing a dish in a single step.
What the Hybrid Model Signals
An interesting result is the 125M-parameter H3-attention model, which does not remove attention entirely. According to the abstract, this model retained two attention layers and scored 1.0 PPL lower than a Transformer on OpenWebText. The paper also reports that, using FlashConv, it scaled to as many as 2.7B parameters on the Pile and outperformed a Transformer on most SuperGLUE tasks in zero-shot and few-shot evaluations.
These results suggest that role division may be more practical than a binary choice between SSM and Transformer. SSM layers can efficiently carry long flows, while a small number of attention layers handle explicit comparison and selection. But the abstract cannot answer where attention layers should be placed or whether the same benefit holds across other data, model sizes, and hardware.
Analyzing the Paper’s Structure: From Diagnosis to System
The abstract’s narrative diagnoses the expressiveness gap with a synthetic task → narrows its causes to recall and comparison → proposes the H3 layer → addresses the execution bottleneck with FlashConv → validates the approach with hybrid models and large-scale training. In other words, it treats “the capabilities language needs” and “the cost of executing those capabilities on hardware” as one connected problem rather than explaining algorithmic design and systems optimization separately.
Practical Implications and Limitations
When evaluating an SSM-based language model, average perplexity alone is insufficient. Construct separate recall tests that require finding a particular earlier token, comparison tests that distinguish agreement or disagreement between two positions, and worst-case tests at long context lengths. Kernel optimization is also an operational variable independent of model quality, so actual inference throughput and memory usage must be measured.
The available material also leaves clear gaps. The detailed H3 equations and ablations, exact FlashConv kernel and hardware settings, batch sizes and precision for each speed comparison, the full procedure for the 2.7B training run, and comparisons with current models are not confirmed. The paper is therefore more accurately read as research that decomposes failure modes in language modeling and demonstrates the possibility of hybrid designs, not as proof that SSMs are immediately a replacement for Transformers.
From an operational perspective, a hybrid architecture also creates new deployment units. Attention layers use memory and computation for comparisons between particular tokens, while the H3–FlashConv path handles long flows. Which path is the bottleneck may change with input length and batch size, so teams should collect layer-level profiles rather than only a model-wide average. This is why the abstract’s speed figures should not be used directly as a service target; end-to-end latency should be measured again with the same tokenizer and output length.
The character of the evaluation data matters as well. An improvement in recall and comparison on synthetic tasks does not automatically imply the same improvement in long-range discourse and knowledge retrieval over real web text. Conversely, near-Transformer PPL on OpenWebText does not guarantee every downstream task. The synthetic, language-modeling, and SuperGLUE results should be read as different layers of evidence, followed by validation in the target domain.
Sources
- Hungry Hungry Hippos: Towards Language Modeling with State Space Models — license:
unknown, retrieved:2026-08-19, source type:original. - Image: Placeholder — cover image based on the original paper not produced — license:
unknown-placeholder.