Spectral Compact Training: Pre-Training Large Language Models via Permanent Truncated SVD and Stiefel QR Retraction

Spectral Compact Training: Pre-Training Large Language Models via Permanent Truncated SVD and Stiefel QR Retraction explores Spectral Compact Training (SCT) enables training large language models on consumer hardware by replacing dense weight matrices with compact SVD factors, drastically reducing memory usage.. Commercial viability score: 7/10 in LLM Training.