Training Transformers in Cosine Coefficient Space

Dimensions overall score 3.0

• A novel method for training transformers by parameterizing weights in cosine coefficient space, reducing parameter count while maintaining performance. At each forward pass the full weight matrix is reconstructed via the inverse DCT; gradients propagate through the reconstruction to update the spectral coefficients directly.problem 35%
  Evidencepartial

  A novel method for training transformers by parameterizing weights in cosine coefficient space, reducing parameter count while maintaining performance. At each forward pass the full weight matrix is reconstructed via the inverse DCT; gradients propagate through the reconstruction to update the spectral coefficients directly.

  Implicationpartial

  Abstract-backed fallback claim; anchored extraction has not materialized a public claim row yet.

  Verificationpartial

  partial

• We parameterize the weight matrices of a transformer in the two-dimensional discrete cosine transform (DCT) domain, retaining only the lowest-frequency coefficients. At each forward pass the full weight matrix is reconstructed via the inverse DCT; gradients propagate through the reconstruction to update the spectral coefficients directly.method 35%
  Evidencepartial

  We parameterize the weight matrices of a transformer in the two-dimensional discrete cosine transform (DCT) domain, retaining only the lowest-frequency coefficients. At each forward pass the full weight matrix is reconstructed via the inverse DCT; gradients propagate through the reconstruction to update the spectral coefficients directly.

  Implicationpartial

  Abstract-backed fallback claim; anchored extraction has not materialized a public claim row yet.

  Verificationpartial

  partial

• ScienceToStartup currently rates this 3.0/10 on the public viability pass. It reduces to replacing each \texttt{nn.Linear} with a drop-in spectral layer that stores $K$ DCT coefficients instead of $n \times m$ weights.result 35%
  Evidencepartial

  ScienceToStartup currently rates this 3.0/10 on the public viability pass. It reduces to replacing each \texttt{nn.Linear} with a drop-in spectral layer that stores $K$ DCT coefficients instead of $n \times m$ weights.

  Implicationpartial

  Abstract-backed fallback claim; anchored extraction has not materialized a public claim row yet.

  Verificationpartial

  partial

• LLM Training moved forward this cycle; last verified April 2026. Public score 3.0/10.why now 35%
  Evidencepartial

  LLM Training moved forward this cycle; last verified April 2026. Public score 3.0/10.

  Implicationpartial

  Abstract-backed fallback claim; anchored extraction has not materialized a public claim row yet.

  Verificationpartial

  partial

Competitive landscape

A novel method for training transformers by parameterizing weights in cosine coefficient space, reducing parameter count while maintaining performance.