효율적인 신경망 학습을 위한 동적 랭크 변경 기술

초록

Low-rank training is a primary strategy for efficient deep learning, but it presents a fundamental challenge: while it reduces computational costs, it permanently limits a model's representational capacity and accelerates rank collapse, which diminishes expressive power during training. We address this with dynamic-rank training, a framework built on the intuition that a model can temporarily escape its low-rank constraints to restore its full learning potential. Our approach strategically interleaves full-rank epochs within a low-rank schedule, with the timing of these restorative phases aligned with the learning rate’s noise regimes to maximize their effect. This enables the model to regain expressive power at critical stages of training by restoring the effective rank of its weights. Our extensive evaluations across various computer vision and natural language processing benchmarks demonstrate that the dynamic-rank method achieves the accuracy of full-rank models while retaining the computational advantages of low-rank training.

키워드