Skip to main content

Does SSM Replace the Transformer, or Do They Divide the Work?

· 8 min read
p4r4d0xb0x
p4r4d0xb0x
Rustacean, AI, OSS Enthusiast

cover placeholder

The debate over sequence models after the Transformer is often translated into the language of dethroning. “Is SSM next?” and “Is attention finished?” are easy questions to understand, but they are not sufficient for real system design. The more precise question is which computation should be assigned to which layer. The best architecture changes depending on whether the system must process a long input all at once, generate tokens one by one, revisit every previous token, or simply update a fixed-size state.

The conclusion of this article is that the publicly available evidence at the abstract level is not enough to say that SSMs will fully replace Transformers. At the same time, interpreting SSMs as an independent option in the parts of a system where Transformer costs grow, or as a component of a hybrid that uses both structures, fits the direction of multiple studies. Below, we separate the claims made by the sources from this article’s interpretations.

First, distinguish attention from state​

Term explainer: state space model (SSM)

An SSM is a family of models that updates an internal state while reading input and produces output from that state. The central intuition is that a long sequence is summarized into a fixed-size or bounded-size dynamic state.

Everyday example: instead of spreading the transcript of a long meeting across a desk, imagine the facilitator continually updating a single page of notes with the key points so far.

Term explainer: self-attention

A mechanism that calculates how much each token should reference other tokens. The ability to look directly at past tokens is its strength and, with long contexts, a source of computational and memory cost.

Transformer attention provides direct token-to-token access. This is useful for selectively checking the relationship between two words that are far apart in the context. In autoregressive generation, however, the past keys and values must be cached, and as the context grows, the cache and memory bandwidth become important costs. SSM families take a different path. They update a state at every step and leave in that state the information needed for the next computation. Rather than preserving the past as-is, they transform it through dynamics learned by the model.

This difference is not a personality comparison in which “SSMs have good memory and attention is bad.” The issue is which information must be queried in a form close to the original, and which information can safely be summarized.

Mamba: redesigning the computation path rather than merely replacing it​

The Mamba paper proposes a selective state space model in which state-space parameters are adjusted selectively according to the input. In the abstract, the authors explain that selectivity allows an SSM to remember or forget information according to the input content, with the goal of linear scaling and fast inference on long sequences. This differs from the picture of a fixed linear filter processing every token in the same way.

What the source says: Mamba combines a selective SSM with a hardware-aware algorithm and reports linear scaling with sequence length across modalities including language, audio, and genomics. It also includes comparisons with Transformers, but the abstract alone cannot support a generalization of superiority across all scales, datasets, and hardware.

Interpretation: Mamba’s significance is less that it discards attention than that it demonstrates how computations that do not require direct token-by-token reference can be converted into state updates. This path is attractive for tasks such as streaming input, long-context preprocessing, and tracking repetitive patterns, where a “summary state so far” is useful. Conversely, tasks centered on accurately quoting the original text or freely combining arbitrary tokens may still benefit from direct-access structures.

Griffin: a compromise between recurrence and local attention​

Google’s Griffin was proposed as a family of hybrid models combining gated linear recurrences with local attention. The combination itself is an important signal. The research question is not “choose either attention or recurrence,” but rather to place long-range summarization, where linear recurrence is strong, and short-range interaction, where local attention is strong, within one model.

Term explainer: local attention

A method that computes attention only over a recent, limited window rather than the entire context. It is a compromise intended to preserve precise relationships among nearby tokens while reducing overall cost.

Boundary between source and interpretation: Griffin’s abstract reports competitive properties for language modeling and long-context evaluation. It does not prescribe replacing every Transformer block with recurrence. It is safer to read the work as opening a design space in which different computations handle interactions over different time ranges.

Jamba: hybrid architectures move from research idea to system configuration​

Jamba was released as a language-model architecture combining Transformer and Mamba-family layers. In the public description, the key elements are a mixture of attention and Mamba layers, along with MoE for adjusting model capacity and inference cost. This example shows that division of labor can be a choice in an actual model configuration, not merely a theoretical metaphor.

Jamba can be interpreted as assigning attention to portions that need direct access to all information, while Mamba layers handle long sequences through a linear state path. However, the optimal layer ratio and the exact way bottlenecks shift for each task depend on model size and implementation. Results from a paper or product page should not be read as guarantees for a different deployment environment.

The question posed by Mamba-3: is theoretical linearity enough?​

Mamba-3 improves sequence modeling using state-space principles and proposes exponential-trapezoidal discretization, complex-valued state updates, and MIMO SSMs. The provided paper summary explains that MIMO is designed to increase decoding FLOPs under a memory bottleneck and thereby improve hardware utilization. In other words, it emphasizes that real latency is not determined by the notation “linear complexity” alone.

Numbers reported by the source: The evidence-pack summary states that at the 1.5B scale, base Mamba-3 achieved an average downstream improvement of 0.6 percentage points over a strong linear layer, while MIMO added approximately 1.2 percentage points. It also reports up to four times more decoding FLOPs than Mamba-2 at the same state size, with similar actual latency.

Limitation: These numbers come from specific experimental conditions. Without checking the hardware model, kernel version, batch size, and exact evaluation suite, we cannot conclude that “SSMs are always faster.” Mamba-3 is evidence that substantial design space remains inside SSMs, not evidence that Transformers should universally be discarded.

So how should the work be divided?​

In practice, three arrangements are possible.

  1. SSM-centered: Consider this when the input is extremely long, streaming or low-memory inference matters, and the relevant past can be maintained as a summary state.
  2. Attention-centered: Consider this when precisely revisiting arbitrary past tokens or directly combining sparse relationships within the context is the priority.
  3. Hybrid: Assign nearby relationships to local attention, long-range accumulation to recurrence/SSM, and global reference at selected layers to attention. Jamba and Griffin are concrete examples of this direction.

This choice is not only about model architecture. Training parallelism, the KV cache, kernel support, batch processing, quality evaluation, and state recovery after failures must also be evaluated. SSMs have state. Consequently, how state is isolated and stored between requests becomes an operational question that differs from the Transformer case.

Conclusion and limitations​

An SSM is less a “successor” to the Transformer than another axis along which sequence computation can be decomposed. Mamba shows a selective state path, Griffin a combination of recurrence and local attention, Jamba a hybrid language model, and Mamba-3 the potential for quality and hardware improvements within SSMs. The conservative conclusion supported jointly by this evidence is not a “single winner,” but a division of labor according to workload.

This article, however, is limited to the abstracts and public summaries of each paper. Full experiment tables, implementation-specific memory use, proprietary optimizations in commercial models, and task-specific failure cases were not verified and remain unverified. Before adoption, attention-only, SSM-only, and hybrid systems should be compared directly under identical data, hardware, and batch conditions.

Sources​

// COMMENTS

Comments