RWKV-5 Eagle and RWKV-6 Finch: Expanding the State into a Matrix and Making Recurrence Dynamic
The Transformer's approach to long context is powerful, but generation incurs the cost of storing and reading past keys and values. RWKV computes an attention-like weighted average as a recurrence, aiming to retain time-axis parallelism during training while using a fixed-size state per token during generation. The paper “Eagle and Finch” extends RWKV-4 in two stages. Eagle (RWKV-5) expands a vector state into a multi-head matrix state, while Finch (RWKV-6) makes decay and token shift input-dependent.
The abstract reports competitive results across several benchmarks after training four Eagle models (0.46B–7.5B) and two Finch models (1.6B and 3.1B). It also introduces a multilingual corpus with 1.12 trillion tokens, a tokenizer based on greedy matching, and models and code released under Apache 2.0. This article is based on the structure and abstract of arXiv 2404.05892v4 and does not expand “competitive” into a claim of universal victory for any particular model.
RWKV-4's Starting Point: Turning Attention into a Decaying Recurrence
RWKV's time mixing accumulates past key-value information together with decay. It gives the current input a separate bonus, while receptance controls how much of the accumulated value is read. Comparing this with the k^T v state update in linear attention makes the intuition clear. At each step, the model computes only the accumulated state and the receptance corresponding to the current query, rather than revisiting the entire past.
The system-level advantage is that there are two computation modes. When training on a complete sequence, a time-parallel implementation provides parallelism; when generating one token at a time, the model updates its state. The original paper's table describes RWKV-4/5/6 as a family with O(1) state and time cost with respect to sequence length during inference. However, O(1) ignores model dimension and head count, and actual speed is determined by matrix multiplications, memory access, and kernel implementation.
Simple definition: A cyclic computation that reuses part of the current result when processing the next input.
Example: Instead of recalculating a ledger's balance from every transaction each day, apply today's deposits and withdrawals to yesterday's balance. It is fast, but the previous records must be summarized into one balance.
Simple definition: A learnable attenuation factor that reduces the influence of a past state over time.
Example: Gradually erase old notes on a classroom board so recent material is easier to see. If the notes fade too quickly, long-term information disappears.
Eagle: Expanding Vector Memory into Matrix Memory
Eagle's largest change is that each head maintains a k^T v state in matrix form rather than as a vector. The paper describes multi-headed matrix-valued states, per-head LayerNorm, SiLU attention gating, and removal of the receptance sigmoid as key changes. The state update can be read approximately as follows:
s' = diag(w) · s + k^T · v
wkv' = s + diag(u) · k^T · v
The paper explains that w is parameterized as exp(-exp(ω)), placing each channel's decay value between 0 and 1. The current token's contribution is multiplied by u, giving it a weight different from the accumulated past. Receptance reads the state like a query, while the SiLU gate controls the flow of the output. This design lets the state preserve richer relationships between keys and values instead of merely storing a channel-wise sum.
Here, saying that “expressive power increased” is a structural interpretation: the state has more degrees of freedom and more varied input-output transformations. It is not a guarantee that accuracy will rise on every task. The paper evaluates Eagle and Finch across several benchmarks, but the fit between state size and data distribution can differ by application.
Simple definition: An internal state that accumulates past information as a matrix with relationships between rows and columns, rather than as a single number or vector.
Example: Instead of keeping only a list for each person, maintain a “person–role” table. It can express richer connections, but storage and computation increase.
Finch: Making Recurrence Respond to the Input
Building on Eagle, Finch makes token shift and decay data-dependent. Eagle's token shift mixes the current and previous inputs with a learned linear interpolation; Finch adds an additional correction computed from the input. The paper also presents a method that uses a LoRA-like form to adjust the decay vector according to context. Thus, even for the same channel, how long the past is retained can vary with the input situation.
This change targets the limitations of fixed decay. Some information should be remembered briefly, while other information must be carried across a long context. Dynamic recurrence delegates that choice to the data. At the same time, it may increase implementation complexity, instability risk, and the cost of parallel training. The paper gives concrete formulas, but that does not mean the dynamic path is as efficient as the fixed path on every kind of hardware.
Multilingual Data and the Tokenizer Are Also Part of the Architecture
The paper covers more than block equations. It introduces RWKV World v2, a public multilingual dataset with 1.12 trillion tokens, and the RWKV World Tokenizer. The tokenizer is described as using Trie-based greedy matching, with the authors aiming to handle underrepresented languages and code more effectively. This is why model performance should not be read as an architecture-only result: data composition, tokenization efficiency, and the number of training tokens all contribute.
The paper trains four Eagle models and two Finch models, evaluating English and multilingual text, associative recall, music, and vision-language tasks. Benchmark scores in the tables must be read together with model size, training data, and evaluation protocol. For a multilingual service, measure tokens per unit of information, long-sentence retrieval, and code-mixed input separately in Korean and the target languages.
Paper Structure and Practical Interpretation
The paper contrasts quadratic attention in Transformers with RNN inference efficiency in the Introduction, then explains the evolution of linear attention, AFT, and RWKV-4 in the Background. It proceeds through the Eagle/Finch architecture, detailed methods, tokenizer and dataset, pretrained models, and experiments in language modeling, speed, and multimodality. The flow defines the bottleneck in existing structures, presents the staged Eagle → Finch changes, and then validates the data, model, and application scope.
Operationally, RWKV's key point is not that it stores an unlimited context, but that it maintains a fixed-size state that is convenient for streaming. For agents handling long-running input or local inference, a state-memory model may be simpler than a KV cache. However, reusing or branching a state differs from the Transformer's ability to read a specific past token again. Product design should make checkpoint-to-state compatibility, different lengths within a batch, and state-initialization policy explicit.
Limitations and What to Check
First, the paper's “competitive performance” means broad evaluation, not that it surpasses Transformers on every model, language, and task. Second, releasing models under Apache 2.0 improves reproducibility, but the dataset's actual composition and usage conditions should be checked again in the paper and release repository. Third, O(1) inference notation describes complexity with respect to sequence length; it does not guarantee low absolute memory or compute when the matrix state becomes large. Fourth, dynamic decay does not automatically solve long-term memory. The balance between retention and forgetting depends on the evaluation data.
Before adoption, check the following:
- Average tokenizer token count in the target languages and code
- Prefill and decode throughput under identical batch size, precision, and hardware
- Peak memory as state size and concurrent request count change
- Separate accuracy for recent memory, long-range retrieval, and order changes
- Versions and licenses of the public inference/training code and model checkpoints
The direction shown by Eagle and Finch is not a declaration for either RNNs or Transformers. It is an incremental design that enriches the state into a matrix and makes recurrent decay respond to inputs while preserving state-based efficiency during generation. Within the scope supported by the provided materials, it is an attractive alternative, but real performance and cost must be measured together with model size, tokenizer, kernels, and workload context. In particular, it is safer to first verify what a fixed-size state preserves and discards than to talk about “infinite context.”
Sources
- Bo Peng et al., “Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence,” arXiv:2404.05892v4, submitted 2024-04-08, revised 2024-09-26, CC BY 4.0. https://arxiv.org/abs/2404.05892
- HTML full text, arXiv:2404.05892v4, sections 1–9 and appendices, retrieved 2026-08-19. https://arxiv.org/html/2404.05892v4
- RWKV project links listed by the paper (models, training, inference); availability and license should be rechecked before deployment. https://huggingface.co/RWKV