Spectral Compact Training: Pre-Training Large Language Models via Permanent Truncated SVD and Stiefel QR Retraction

Dimensions overall score 7.0

• Spectral Compact Training (SCT) enables training large language models on consumer hardware by replacing dense weight matrices with compact SVD factors, drastically reducing memory usage. We introduce Spectral Compact Training (SCT), a method that replaces dense weight matrices with permanent truncated SVD factors W = U diag(s) V^T, where the full dense matrix is never materialized during training or inference.problem 35%
  Evidencepartial

  Spectral Compact Training (SCT) enables training large language models on consumer hardware by replacing dense weight matrices with compact SVD factors, drastically reducing memory usage. We introduce Spectral Compact Training (SCT), a method that replaces dense weight matrices with permanent truncated SVD factors W = U diag(s) V^T, where the full dense matrix is never materialized during training or inference.

  Implicationpartial

  Abstract-backed fallback claim; anchored extraction has not materialized a public claim row yet.

  Verificationpartial

  partial

• The memory wall remains the primary bottleneck for training large language models on consumer hardware. We introduce Spectral Compact Training (SCT), a method that replaces dense weight matrices with permanent truncated SVD factors W = U diag(s) V^T, where the full dense matrix is never materialized during training or inference.method 35%
  Evidencepartial

  The memory wall remains the primary bottleneck for training large language models on consumer hardware. We introduce Spectral Compact Training (SCT), a method that replaces dense weight matrices with permanent truncated SVD factors W = U diag(s) V^T, where the full dense matrix is never materialized during training or inference.

  Implicationpartial

  Abstract-backed fallback claim; anchored extraction has not materialized a public claim row yet.

  Verificationpartial

  partial

• ScienceToStartup currently rates this 7.0/10 on the public viability pass. SCT achieves up to 199x memory reduction per MLP layer at rank 32, enabling full training steps of 70B-parameter architectures on a Steam Deck handheld (7.2 GB peak memory vs. A public repository is linked, so build verification can inspect implementation evidence instead of treating the paper as PDF-only.result 35%
  Evidencepartial

  ScienceToStartup currently rates this 7.0/10 on the public viability pass. SCT achieves up to 199x memory reduction per MLP layer at rank 32, enabling full training steps of 70B-parameter architectures on a Steam Deck handheld (7.2 GB peak memory vs. A public repository is linked, so build verification can inspect implementation evidence instead of treating the paper as PDF-only.

  Implicationpartial

  Abstract-backed fallback claim; anchored extraction has not materialized a public claim row yet.

  Verificationpartial

  partial

• LLM Training moved forward this cycle; last verified April 2026. Public score 7.0/10. Implementation evidence is present through a linked repository.why now 35%
  Evidencepartial

  LLM Training moved forward this cycle; last verified April 2026. Public score 7.0/10. Implementation evidence is present through a linked repository.

  Implicationpartial

  Abstract-backed fallback claim; anchored extraction has not materialized a public claim row yet.

  Verificationpartial

  partial

Competitive landscape

Spectral Compact Training (SCT) enables training large language models on consumer hardware by replacing dense weight matrices with compact SVD factors, drastically reducing memory usage.