Back to Dynamics
Designing Sequence Models Through Their State Dynamics

From Recurrences to Differential Equations

Differential equations describe how a state evolves in continuous time. A linear homogeneous system has the form \(\dot S(\tau)=A(\tau)S(\tau)\), where \(A\) is its generator. Holding the generator fixed over an interval of length \(\Delta_t\) gives an exact discrete transition:

Conversely, when \(T_t\) admits a real matrix logarithm, we can construct \(A_t=\log(T_t)/\Delta_t\). The transition’s eigenvalues and invariant subspaces tell us which components decay, persist, or change phase.

Different Architectures, Related Dynamics

Different architectures can produce similar state dynamics. The examples below reach decay and oscillation from two starting points.

Starting from an oscillator

LinOSS (Rusch & Rus, 2025) starts from driven linear oscillators. For a positive scalar frequency \(\omega\), its continuous-time dynamics can be written as

Without the forcing term, the first-order state \((x,\dot x)\) has generator eigenvalues \(\pm i\omega\). Rescaling position and velocity reveals a two-dimensional rotation. D-LinOSS (Boyer et al., 2025) adds learnable damping. In our notation, a mode takes the form

When \(0<a<\omega_0\), the generator eigenvalues are \(-a\pm i\sqrt{\omega_0^2-a^2}\). The state both loses amplitude and advances in phase. This describes the continuous system without forcing. The discrete behavior also depends on the chosen discretization.

Starting from selective decay

Mamba-1 (Gu & Dao, 2024) uses selective state spaces. With its real, negative diagonal parameterization, a homogeneous state mode follows \(z_t=\rho_tz_{t-1}\), \(0<\rho_t<1\). The input controls retention. With the parameters held fixed, the corresponding continuous system is \(\dot z=-a_tz\), where \(a_t=-\log\rho_t/\Delta_t\).

Mamba-2 (Dao & Gu, 2024) reorganizes computation through structured state-space duality and uses a scalar state transition within each head. Without new writes, the state still undergoes selective decay. Mamba-3 (Lahoti et al., 2026) adds phase evolution through complex state updates. In a suitable real coordinate system, a rotary pair follows

Here, \(\rho_t\) sets decay and \(\theta_t\) advances phase. With \(\nu_t=\theta_t/\Delta_t\) and \(J=\left[\begin{smallmatrix}0&-1\\1&0\end{smallmatrix}\right]\),

D-LinOSS adds damping to an oscillator. Mamba-3 adds rotation to a decaying state. In suitable coordinates and with parameters held fixed, both give decay plus rotation. Eliminating one coordinate of a rotating pair gives

For \(a>0\) and \(\nu\ne0\), this is an underdamped oscillator, just as in D-LinOSS above.

Linear Attention: Which Directions Should Forget?

Now consider a matrix state that stores key–value associations. We write its update with the transition acting from the left:

The same \(T_t\) acts on every value column. We can therefore analyze one column \(s_t\in\mathbb R^{d_k}\) through \(s_t=T_ts_{t-1}\). Below, keys have unit norm, and we first consider positive gates with \(0<\beta_t<1\). Each model summary describes one head’s homogeneous transition on a state column, with the current key and gates held fixed.

DeltaNet: forgetting along the current key

DeltaNet (Yang et al., 2024) writes a correction to the association currently stored at the key:

Let \(P_t=k_tk_t^\top\). Since \(\|k_t\|_2=1\), \(P_t\) is an orthogonal projector. The component parallel to the key shrinks by \(1-\beta_t\). The orthogonal component stays unchanged:

The corresponding continuous system is \(\dot s=-b_tP_ts\), with \(b_t=-\log(1-\beta_t)/\Delta_t\). DeltaNet implements relaxation in an input-selected direction.

Gated DeltaNet: adding a common forgetting rate

Gated DeltaNet (Yang et al., 2025) adds a scalar forget gate \(0<\alpha_t<1\), giving \(T_t=\alpha_t(I-\beta_tP_t)\).

To see how this acts in different directions, split a state column into \(s=s_\parallel+s_\perp\), where \(s_\parallel=P_t s\) is parallel to the current key and \(k_t^\top s_\perp=0\). Since the key has unit norm, \(P_t s_\parallel=s_\parallel\), while

Applying the full transition therefore gives

The delta factor leaves every component orthogonal to the current key unchanged, so its retention factor is just \(\alpha_t\). The key component receives both the shared factor \(\alpha_t\) and the extra factor \(1-\beta_t\), giving \(\alpha_t(1-\beta_t)\).

Because \(I\) and \(P_t\) commute, its generator is

This separates shared forgetting from the extra correction along the key.

KDA: different decay rates across coordinates

Kimi Delta Attention (KDA) (Kimi Team, 2025) replaces the scalar gate with a diagonal gate:

The diagonal gate scales each state coordinate by \(\alpha_{t,i}\). The delta correction can then mix coordinates, so we need to analyze their combined effect.

With \(0<\alpha_{t,i}<1\) and \(0<\beta_t<1\), hold the current key and gates fixed over an interval \(\Delta>0\). Drop the time index and write \(H=I-\beta kk^\top\), so \(T=HD\). Rescaling the state by \(D^{1/2}\) gives the transition

This matrix is symmetric. Since \((1-\beta)I\preceq H\preceq I\), its eigenvalues satisfy

Let \(U\) be an orthogonal matrix satisfying \(\widetilde T=U\operatorname{Diag}(\lambda_j)U^\top\). First rescale the state by \(D^{1/2}\), then rotate it by \(U^\top\):

Since \(0<\lambda_j<1\), each transformed coordinate decays. The corresponding continuous dynamics over the interval \(\Delta\) are

Larger \(\gamma_j\) means faster decay. Smaller \(\gamma_j\) means longer retention. These coordinates generally differ from the original state axes, and their retention factors \(\lambda_j\) reflect both the diagonal gate and the delta correction. This coordinate system can change when the key or gates change.

Gated DeltaNet-2: separating erase and write

KDA uses the same scalar \(\beta_t\) to erase an old association and write a new value. Gated DeltaNet-2 (GDN2) (Hatamizadeh et al., 2026) separates these decisions into a key-side erase gate \(b_t\) and a value-side write gate \(w_t\), while retaining KDA’s channel-wise decay:

Tying \(b_t=\beta_t\mathbf 1_{d_k}\) and \(w_t=\beta_t\mathbf 1_{d_v}\) recovers KDA. Making the decay scalar as well recovers Gated DeltaNet. Given the input, \(T_t\) determines how the old state evolves, while \(w_t\) controls the new write. We derive its frozen spectrum below using the same approach as for KDA.

Drop the time index and set \(B=\operatorname{Diag}(b)\). Then \(T=D-kk^\top BD\), which is generally asymmetric. For positive \(b_i\) and \(\alpha_i\), define \(P=(BD)^{1/2}\) and \(u=Pk\). Since these diagonal matrices commute, rescaling by \(P\) gives

The right-hand side is real and symmetric, so every eigenvalue of \(T\) is real. An asymmetric erase operation therefore still admits a basis of real modes.

The official default implementation uses sigmoid erase and write gates, exponential decay, and L2-normalized keys. In the mathematical parameterization, \(0<b_i,w_j<1\) and \(0<\alpha_i<1\). With \(\|k\|_2=1\), let \(r=B^{1/2}k\). Then

Because \(I-rr^\top\) is positive definite and at most \(I\), we obtain

As with KDA, diagonalize \(\widetilde T\) with an orthogonal matrix \(U\) and use \(y=U^\top Ps\). Every frozen mode decays:

This also connects to the second-order equations above. For two frozen modes, \(x=y_i+y_j\) satisfies

Its characteristic roots are \(-\gamma_i\) and \(-\gamma_j\). Distinct rates give an overdamped equation. This second-order description applies to a pair of frozen modes. The full state can have many modes, and their basis changes with the input. In particular, a product of changing transitions need not share the real-spectrum property of each step.

The implementation also offers allow_neg_eigval=True, which expands the erase range to \(0<b_i<2\). The same similarity transform remains valid, although positive definiteness can be lost. Negative real eigenvalues can produce alternating signs. The frozen transition still has no non-real conjugate pair.

So far, the key and gates control which components decay and how quickly. GDN2 also separates erasing from writing. Can a frozen transition support an oscillatory mode while retaining the state?

Complex KDA: Oscillation Through Signed Gates

Complex KDA (CKDA) (Siems et al., 2026) expands KDA’s gate ranges, allowing a state to rotate as well as decay:

We analyze one fixed transition for one head and one state column. Here, \(D\) is real and diagonal, \(k\) is a real unit vector, and \(\beta\) is a real scalar.

For a two-dimensional example with fixed gates and key, choose \(0<\rho<1\) and \(0<\theta<\pi/2\). A signed gate and the delta correction give

Its eigenvalues are \(\rho e^{\pm2i\theta}\), a non-real conjugate pair that oscillates and decays because \(\rho<1\). In general, a non-real pair implies oscillation, not stability. An oscillatory mode is asymptotically stable only when \(|\lambda|<1\).

The necessary conditions follow from symmetry. Write \(T=HD\), where \(H=I-\beta kk^\top\) and \(\|k\|_2=1\). If \(\beta<1\), then \(H\) is positive definite, and

The matrix on the right is symmetric, so all eigenvalues are real. The case \(\beta=1\) follows by continuity. If all \(\alpha_i\ge0\), the determinant identity gives

Since \(D^{1/2}HD^{1/2}\) is symmetric, this also forces a real spectrum. Thus a non-real conjugate pair requires both \(\beta>1\) and \(\alpha_i<0\) for at least one \(i\). These are necessary conditions, not a guarantee of oscillation.

To count the possible oscillatory pairs, first assume the gates are distinct and nonzero, ordered as \(\alpha_1<\cdots<\alpha_d\), and that every key coordinate is nonzero. The matrix determinant lemma gives

Between two adjacent gates of the same sign, the expression in parentheses is continuous and runs from one sign of infinity to the other. It must therefore cross zero, giving a real eigenvalue. At most one interval separates negative and positive gates, so at least \(d-2\) intervals contain a real root. This leaves at most two non-real roots. Because \(T\) is real, they must form at most one non-real conjugate pair. The same bound holds for repeated or zero gates and zero key coordinates by continuity.

In our Mamba-3 blog post (Part I) (Liang & Song, 2026), we observed lower long-context perplexity in models trained with oscillation. To explore long-context behavior in linear attention, we also evaluated the officially released CKDA and KDA models on PG-19 (Rae et al., 2019) and LongBench-E (Bai et al., 2024).

PG-19: lower perplexity beyond the training context

CKDA maintains lower perplexity than KDA from 2K to 64K, including beyond the 4K training context reported by Siems et al. (2026).

LongBench-E: gains vary by task

CKDA averages 13.90 across the 13 tasks, compared with 12.75 for KDA, and scores higher on 8 of them.

CKDA performs better than KDA in these evaluations. It achieves lower PG-19 perplexity at every tested context length from 2K to 64K and a higher average score on LongBench-E, with gains on 8 of the 13 tasks.

Momentum: Another Route to Oscillation

Momentum DeltaNet (MDN) (Huang et al., 2026) adds a momentum state \(M_t\) to the memory update:

We consider one head and one state column, with scalar coefficients and a real unit key held fixed, and set aside new writes. The coupling \(P=kk^\top\) has rank one. Let \(s_n\) and \(m_n\) denote the memory column and its momentum. Their updates are \(m_{n+1}=\mu m_n+\alpha\eta Ps_n\) and \(s_{n+1}=\alpha s_n-\beta m_{n+1}\).

Each direction has two components: memory and momentum. Since \(Pk=k\) and \(Pe=0\) for \(e\perp k\), the key direction has the block \(B\). Each of the \(d_k-1\) perpendicular directions has an independent block \(B_\perp\):

\(B_\perp\) is triangular, so its eigenvalues \(\alpha\) and \(\mu\) are real. For the key block, write \(c=\alpha(1-\beta\eta)+\mu\). Its eigenvalues are roots of

Only the key block can have non-real eigenvalues. Its quadratic has just two roots, so the transition has at most one conjugate pair. The pair appears when \(c^2<4\alpha\mu\). With \(0<\alpha,\mu<1\), its modulus is \(\sqrt{\alpha\mu}<1\), so the oscillation decays over time.

What Are the Limits of a Single Oscillator?

A single oscillator, represented by one non-real conjugate pair, has one oscillation frequency. Its linear readouts can differ in amplitude and phase, but they share that frequency. Suppose two pieces of memory need different rotation speeds: 20° and 50° per step. The demo shows why one fixed oscillator cannot sustain both frequencies.

The targets correspond to two independent state pairs, with \(\phi_A=20^\circ\), \(\phi_B=50^\circ\), and \(\rho=0.98\):

Here \(\oplus\) places the two rotations in separate blocks. Each contributes a distinct non-real conjugate pair. Within one head and state column, the fixed CKDA and MDN transitions above have at most one pair. Therefore they cannot exactly maintain both independent rotations with a fixed linear state encoding.

Let \(\mathsf T\) and \(T_\star\) be fixed real linear transitions. Suppose the time-independent linear map \(E\) is injective and encodes the entire target state \(z\), including both rotating pairs, as \(s=Ez\). Exact state dynamics require

If \(T_\star v=\lambda v\), then \(\mathsf T(Ev)=\lambda(Ev)\), with \(Ev\ne0\). Every target eigenvalue must therefore also appear in the model transition. A wider state with only one oscillatory pair still cannot reproduce both target frequencies exactly.

This argument does not rule out finite-horizon approximation, time-varying encodings, or input-dependent products of transitions. Multiple heads or layers can also combine more frequencies. CyFA, which we consider next, supports several oscillatory pairs within a single transition, with related frequencies.

From Rotation to Memory Transport

CyFA (Chen et al., 2026) moves information through memory slots using cyclic shifts. Each head maintains separate key and value states over \(m\) relative-time slots:

Here, \(K_t\in\mathbb R^{m\times d_k}\), \(V_t\in\mathbb R^{m\times d_v}\), and \(e_0\) selects the first slot. The clock increment \(\delta_t\in(0,1)\) controls CyFA’s Fourier fractional shift \(P(\delta_t)\), rather than an arbitrary transport operator. The gate \(\alpha_t\) controls decay and \(\beta_t\) controls writing, with both gates in \((0,1)\). This shift operator is different from the key projector used earlier. Existing information moves before the new key and value are written into slot zero.

Transport in slot space, oscillation in Fourier space

We restrict the count below to an odd number of slots \(m=2p+1\). Even \(m\) adds a real Nyquist mode and requires a separate count. For odd \(m\), an orthogonal real Fourier basis \(\Phi\) decomposes the shift into a constant mode and \(p\) cosine–sine pairs:

In Fourier coordinates, cyclic transport rotates each cosine–sine pair. With no new writes, each pair follows \(z_{t,j}=\alpha_tR(\omega_j\delta_t)z_{t-1,j}\), with eigenvalues

Because \(0<\omega_j\delta_t<\pi\), all \(p=(m-1)/2\) pairs are non-real. Each gives a local damped oscillator after setting \(a_t=-\log\alpha_t/\Delta_t\) and \(\nu_{t,j}=\omega_j\delta_t/\Delta_t\). The constant mode only decays.

Compare the same memory state with and without shifting. In slot coordinates, a shift moves its signed coefficients between slots. In Fourier coordinates, it rotates each cosine–sine pair. The two rows below show the same states in different coordinates.

Many modes, coupled controls

CyFA has multiple oscillatory pairs, but it cannot set their frequencies independently. Their phase increments obey

All modes share one learned clock increment and one decay coefficient. The Fourier subspaces and frequency ratios are fixed by the transport structure. At \(m=5\), for example, the second pair rotates through twice the angle of the first. The two pairs therefore cannot use angles \(\pi/10\) and \(\pi/3\) in the same update.

CyFA's oscillators have fixed frequency ratios, and their number is set by the slot count \(m\). For odd \(m\), each state column has \((m-1)/2\) oscillatory pairs. Increasing \(m\) adds more pairs whose frequencies remain tied to the same learned clock. This describes the homogeneous slot transition. It does not claim that the full input-driven model output contains only these frequencies.

Dynamics as a Design Dimension

Beyond asking whether a model oscillates, we can ask how many oscillatory modes it supports, which controls they share, whether their subspaces change with the input, and how the model writes and reads in those coordinates.

Writing, readout, input dependence, multiple heads, and composition across layers also shape what a complete model can compute. Studying state dynamics lets us examine one part of that design in detail.

These questions suggest experiments. Does a task benefit from more oscillatory modes? Should each pair have its own phase and decay controls, or should modes share a clock? Does allowing their subspaces to change with the input help? How much memory and computation do these choices require, and can the updates still be trained efficiently in parallel?

Back to dynamics. We do not claim that oscillatory dynamics are always better. We hope to treat state dynamics as one dimension of model design, looking beyond memory decay and erasure.

Memory has a lifetime, but it also has a trajectory.