Robust Low-Rank Training via Approximate Orthonormal Constraints

Low-rank weight factorisations can reduce both the training and inference cost of neural networks, but they can also weaken robustness to adversarial perturbations. This work connects that loss of robustness to exploding singular values and poor network conditioning.

The proposed training algorithm keeps weights on a low-rank matrix manifold while enforcing approximate orthonormal constraints. This controls conditioning while retaining the computational benefits of low-rank representations.

Theoretical and empirical results show that the resulting models improve adversarial robustness without compromising predictive accuracy.

Paper: https://papers.nips.cc/paper_files/paper/2023/hash/d073692637b4fb8c4eb4b81f0fa2df7b-Abstract-Conference.html