Princeton RLT carries decoder state across 96 blocks

Princeton RLT Fixes 96 Blocks per Token

Recurrent Looped Transformer (RLT) arrives with a deceptively simple premise. Stop treating each token like an island. In nearly every decoder-only large language model today, nothing computed at the final layer for one token feeds the first layer of the next, and positions talk to each other only through attention over cached keys and values. A technical report from Princeton researcher Yifan Zhang proposes closing that loop. It carries the decoder's final hidden state. And it carries its layerwise sliding-window attention cache straight into the following token. That happens across both prompt and response. There's no reset at the boundary.

This work is a design specification. It isn't a results paper. It defines an architecture, execution schedules, and a reinforcement learning replay contract, and while it lays out those structural commitments in careful detail, it reports no measured efficiency, no reasoning quality numbers, and no scaling results at all. That candor matters. It matters as much as the design itself.

What Actually Moves Between Tokens

RLT pairs a causal encoder with a recurrent decoder. The encoder processes tokens in parallel under a causal mask, producing representations from which a key-value memory is projected, and that memory can be grouped so it's shared across decoder layers or kept layer-specific, depending on the design. But the recurrence lives in the decoder. It's all in the decoder.

Its complete state is a pair. That's the final decoder output plus the retained sliding-window attention keys and values at every decoder layer. For each token, a gated merge combines the encoder representation with the previous output, and then each decoder block runs causal sliding-window attention over decoder activations, cross-attention to encoder memory, and a feed-forward network. The window includes the current token. So at most W minus one historical entries per layer are retained. The next-token distribution is read from the final decoder output. Initialization happens once before the beginning-of-sequence token. It uses a learned start state and an empty cache.

In the reference tied configuration, 48 encoder and 48 decoder layers share compatible attention and feed-forward weights. So each token runs 96 logical blocks. The report is careful here. Decoder blocks add cross-attention, so per-block FLOPs aren't equal, because the two block types don't do identical work even though they're sharing weights in this tied setup. Zhang calls this parameter reuse, not activation copying.

Three Principles, One Honest Caveat

The first principle is latent reasoning with unbounded temporal depth. It's simple. After processing t tokens, the state path from the start state traverses t times the decoder depth, or 48t in the reference configuration. And per-token work stays fixed while the structural depth of the path grows with the sequence, which means the path keeps stretching deeper even though the work at each token doesn't change at all. So depth grows.

The report warns that gates and contraction may suppress long paths. Structural depth is not a reasoning guarantee.

The second principle is model-hardware co-design. Encoder features and memory projections for known tokens use token-parallel kernels. Decoder transitions stay sequential within a sequence, but ready updates from independent sequences can share one batched kernel. The report states plainly that no exact parallel scan is assumed for the nonlinear decoder, no reduced-prefill speedup is claimed, and a standard parallel sliding-window attention decoder pass is not equivalent to the recurrence. Batching, kernel fusion, and checkpointing are listed as implementation targets, not completed kernels.

The third principle is model and RL algorithm co-design. Pretraining, supervised fine-tuning, sampling, and RL replay all share one state transition. For reinforcement learning, the sampler records each action's behavior log-probability under its actual sampling distribution, including temperature and truncation. The trainer then rebuilds encoder memory, the recurrent output, and every attention cache from the sequence start under current parameters before scoring each action. Old rollout states are never reused. A proposition formalizes the payoff. It's simple. Moving the prompt-response split leaves the conditional distribution unchanged for a fixed token history.

Training, Serving, and the Snapshot Problem

Pretraining is full-sequence next-token prediction with full backpropagation through time. Supervised fine-tuning masks the loss to assistant targets but never masks state updates. So assistant losses backpropagate through user and tool tokens. An appendix shows why partial detaching is risky, because the state-to-state Jacobian has cross terms through decoder key-value memory, which means that detaching only the final output leaves gradient paths through the cache. It's a trap. Any truncated backpropagation scheme must name every detached tensor.

An exact prefix snapshot includes a lot. Encoder cache and memory, the complete decoder state, position metadata, the window convention, and model version all sit inside it for multi-turn serving. A fixed-weight snapshot can be reused. Why? Because the state is independent of the serving split, so nothing forces a rebuild when that split changes. But weight updates invalidate old states. Editing a prefix forces recomputation from an earlier checkpoint. External tokens in multi-turn RL update the state but get no importance-ratio factors.

Where It Sits in the Lineage

Encoder-derived memory follows earlier work that caches key-value pairs once for a cross-decoder, and systems that project decoder global key-value from final encoder states. RLT keeps the memory but drops prompt-wide decoder skipping. Temporal feedback builds on the Feedback Transformer and Recurrent Transformer; RLT instead feeds the previous final decoder output into the next decoder input and runs recurrence over the prompt too. Depth-wise reuse connects to Universal Transformers and recurrent-depth latent reasoning.

The report's own summary is blunt: reasoning quality, efficiency, and RL scaling remain open validation targets.

What to Watch

• The full decoder state, final output plus layerwise sliding-window cache, crosses every prompt and response token with no boundary reset.
• The reference configuration runs 48 tied encoder and decoder layers, 96 logical blocks per token, and a state path of 48t blocks after t tokens.
• Hardware opportunities center on encoder parallelism and batching across sequences. No parallel scan or reduced-prefill speedup is claimed.
• RL replay rebuilds all states under current parameters while keeping recorded behavior log-probabilities as ratio denominators.

Recurrent Looped Transformer (RLT) is a blueprint with its limitations printed on the cover. No measured wins, no scaling curves, no efficiency tables. What it offers instead is a clean specification of how a model could remember its own trajectory, token by token, and a replay contract that keeps reinforcement learning honest when the state never stops moving.