Skip to main content

Griffin's Compromise: Combining Gated Linear Recurrence with Local Attention

· 7 min read
p4r4d0xb0x
p4r4d0xb0x
Rustacean, AI, OSS Enthusiast

Sequence-model architecture where linear recurrence and local attention alternate

The Transformer became the standard for language modeling through its ability to compare every position directly, but generation requires continually storing past keys and values. Traditional recurrent neural networks (RNNs), by contrast, update one state per token and therefore have low inference cost, but are difficult to parallelize along the time axis and to train at scale. The Griffin paper does not treat this as a choice between two alternatives. It proposes Hawk, a gated linear recurrence, and Griffin, which mixes local attention into Hawk, assigning different time ranges of memory to each component.

The abstract reports that Hawk surpasses reported Mamba performance and that Griffin reaches performance competitive with Llama-2 using more than six times fewer training tokens. It also claims hardware efficiency during training similar to the Transformer, low latency and high throughput during inference, and extrapolation to sequences longer than the training length. This article uses the abstract and the original experimental conditions as evidence; it does not add external reproduction results or new comparisons.

Hawk: Redesigning RNN State with Gates​

A linear recurrence can be understood as a linear combination of the current input and the previous state. In simplified form, it looks like h_t = a_t ⊙ h_{t-1} + b_t ⊙ x_t. The gates control how much of the past to erase and how much of the new input to write. Hawk puts this idea inside a language-model block, using input-dependent gates and stable normalization to improve trainability.

The two sides of this design are clear. During generation, it reads and writes only the current token and a fixed-size state, so it need not revisit the entire sequence. On the other hand, time-axis recurrence is inherently order-dependent and can make training parallelization difficult. The paper expresses gated linear recurrence in a form computable with a parallel scan, and designs both the implementation and the architecture so the actual model can achieve training hardware efficiency similar to the Transformer's.

Term explanation: Gated linear recurrence

Simple definition: A recurrent computation in which learned gates control the relative contributions of the old state and the new input when they are combined.

Example: When recording leftover food in a refrigerator, do not discard or preserve every old note equally: erase expired items and give newly purchased items more weight.

Term explanation: Parallel scan

Simple definition: An algorithm that splits an ordered cumulative computation into chunks, processes them in parallel, and then connects the summary states between chunks.

Example: Instead of having one person add a long list from beginning to end, several people calculate subtotals for ranges and combine them at the end. Each range's result still has to be passed into the next range's computation.

Griffin: Attention for Recent Memory, Recurrence for Long-Range Flow​

Griffin is not simply a stack of Hawk blocks. The paper describes alternating recurrent blocks with local multi-query attention (MQA) blocks. Specifically, it places a residual block using local attention after every two recurrent blocks and gives a default local-attention window of 1024 tokens. Attention directly observes detailed relationships within the latest 1024 tokens, while the recurrent state compresses and carries information from beyond the window.

This combination is not a simple average. Local attention stores a smaller range than a global KV cache, while recurrence passes information outside the window through a fixed state. That creates a design space between “inspect the entire long context with attention” and “compress everything into one vector.” However, which information remains in the window and which is summarized into the state depends on training. Systems that must quote an arbitrary position in a long document precisely need separate evaluation.

The paper says that the residual pattern and MLP block are shared with the Transformer baseline. This matters when interpreting comparisons. Even if the performance difference comes from recurrence and local attention, the tables and experimental section must be checked to determine whether the full block, training recipe, and data composition were identical.

How to Read the Reported Experiments​

The paper varies model size from 100M to 7B and adds a 14B Griffin to present scaling curves. It describes using the MassiveText dataset, 2048-token sequences, and AdamW. For the relationship between validation loss and training FLOPs, it reports lower loss for Griffin than its Transformer baseline across several budget ranges, while Hawk trails relatively but narrows the gap at larger budgets.

The downstream comparison uses the authors' MQA Transformer baseline, Mamba-3B, and Llama-2. The paper's main point is that Hawk-3B performs more strongly than the reported Mamba-3B, while Griffin-7B and 14B achieve averages competitive with Llama-2. However, the original paper explicitly cautions that the external baselines were trained on different data and token counts. In particular, Llama-2 and Mamba saw substantially more tokens than Griffin, and that fact must accompany any reading of the result.

A Systems View: One Fast Token Determines the Cost of the Whole Service​

For a Transformer, inference cost grows with the context and KV cache. Griffin's recurrent blocks use a fixed-size state and its local attention uses a bounded window, so it may reduce memory and latency on long sequences. The paper reports low latency and high throughput, but the actual values and favorable conditions depend on hardware, batch size, sequence length, and kernels. In production, measure not only average latency but also time to first token, inter-token latency, throughput at each concurrency level, and state memory.

During training, both parallel scan and local attention can use GPU parallelism. For distributed training, the fact that the paper scales to a 14B model and describes sharding methods is of practical interest. It does not mean that an existing Transformer training stack can be reused unchanged. Checkpointing of recurrent states, sequence-parallel communication, variable-length batch handling, and fused-kernel support must be validated during implementation.

Paper Structure Analysis​

The paper contrasts the inference advantages and training difficulties of RNNs with the representational power and long-sequence cost of Transformers in the Introduction. It explains the components of linear recurrence and local attention in the Background, then builds Hawk and defines Griffin by mixing the two blocks. It proceeds through scaling, downstream evaluation, long-sequence extrapolation, and speed and memory evaluation: problem → design → validation. The conclusion emphasizes competitive performance, but the interpretation of which tasks favor which structural choice must be read together with the experimental conditions.

Limitations and a Pre-Adoption Checklist​

First, the preference of a local window for recent information is distinct from the ability to retrieve an exact long-range item. Second, although the recurrent state is small, it has a path for information loss through compression. Third, some of the paper's strong baseline comparisons do not use the same training-token count or dataset, so they should not be translated into absolute superiority. Fourth, the speed advantage reported in the paper does not automatically appear in every inference engine.

If you run experiments, compare the following under identical conditions:

  • Prefill and decode latency by context length and batch size
  • Peak state and KV memory as the number of users increases
  • Accuracy separating recent-window questions from long-range associative recall
  • Perplexity and downstream scores under the same token, data, and tuning budgets
  • Recurrent-state consistency during checkpoint recovery and streaming input

Griffin's message is not that we should return to RNNs. It proposes placing recurrence, which summarizes long-range information, and attention, which performs precise comparisons over a nearby range, in one model to adjust the boundary between algorithmic efficiency and representational power. The reported performance is promising within the provided materials, but the original paper, code, and benchmarks should be checked before adoption. The first question is whether a service's context-access pattern fits the 1024-token window and state-compression assumptions.

Sources​

// COMMENTS

Comments