Temporal Credit Is Free

Dimensions overall score 4.0

• Recurrent networks can adapt online using only immediate derivatives, without needing Jacobian propagation.method 95%
  Evidencepartial

  Recurrent networks do not need Jacobian propagation to adapt online.

  Implicationpartial

  Explicitly stated in the abstract and introduction as the core thesis of the paper.

  Verificationpartial

  partial

• Using immediate derivatives with RMSprop matches or exceeds the performance of full Real-Time Recurrent Learning (RTRL).Source proofresult 90%
  Evidenceverified

  RMSprop match or exceed full RTRL, scaling to n = 1024 at 1000× less memory.

  Implicationpartial

  Directly stated in the abstract and supported by results in Table 1 and Figure 4.

  Verificationverified

  verified

• The method scales to a hidden state size of n=1024 with 1000x less memory than full RTRL.Source prooftechnical 95%
  Evidenceverified

  scaling to n = 1024 at 1000× less memory.

  Implicationpartial

  Explicitly stated with clear numeric comparison in the abstract and Figure 4 caption.

  Verificationverified

  verified

• Second-moment normalization (β2) is required for adaptation when gradients must pass through a nonlinear state update with no output bypass.technical 85%
  Evidencepartial

  β 2 is required when gradients must pass through a nonlinear state update with no output bypass, and unnecessary otherwise.

  Implicationpartial

  Stated as an architectural rule in the abstract and elaborated in the analysis, supported by results across multiple architectures.

  Verificationpartial

  partial

• On cross-session Brain-Computer Interface (BCI) decoding with 7-month electrode drift, the method (λ=0 + RMSprop) outperforms all Jacobian-based methods.result 90%
  Evidencepartial

  Cross-session BCI decoding (7-month electrode drift, 5 seeds). λ=0 + RMSprop (106%) exceeds all Jacobian-based methods.

  Implicationpartial

  Directly stated with a specific performance metric (106% recovery) in Figure 4 caption.

  Verificationpartial

  partial

• In a trained vanilla RNN, recurrent weight gradients are approximately 100 times smaller than output weight gradients, creating a scale mismatch that SGD cannot handle.technical 85%
  Evidencepartial

  The recurrent weight gradients (∂L/∂W hh) are approximately 100 times smaller than the output weight gradients (∂L/∂W out).

  Implicationpartial

  Direct measurement reported with specific numeric factor (100x) and explanation in the analysis.

  Verificationpartial

  partial

• On chaotic dynamics (Lorenz system), RWKV architecture exhibits an inversion where SGD is the only optimizer that survives, while all β2-containing optimizers diverge.result 80%
  Evidencepartial

  on Lorenz, all three β2-containing optimizers... diverge on every seed, while SGD is the only optimizer that survives (293% recovery, zero variance).

  Implicationpartial

  Specific result stated for a particular architecture and task, though it is noted as an exception to the general rule.

  Verificationpartial

  partial

• Standard LSTM's cell state provides a gradient highway for backpropagation through time (BPTT), but this does not apply to online adaptation with immediate derivatives.technical 75%
  Evidencepartial

  Standard LSTM is worth examining separately because its cell state provides a gradient highway for backpropagation

  Implicationpartial

  Implied by the analysis discussing LSTM separately, stating its cell state helps BPTT, which is a contrast to the online context of the paper.

  Verificationpartial

  partial

Competitive landscape

A novel recurrent network architecture that drastically reduces memory requirements for online adaptation, matching state-of-the-art performance.