TY  - EJOUR
AU  - Quercia, Alessio
AU  - Cao, Zhuo
AU  - Bangun, Arya
AU  - Paul, Richard D.
AU  - Morrison, Abigail
AU  - Assent, Ira
AU  - Scharr, Hanno
TI  - 1LoRA: Summation Compression for Very Low-Rank Adaptation
JO  - arXiv
IS  - arXiv:2503.08333
M1  - FZJ-2025-03641
M1  - arXiv:2503.08333
PY  - 2025
AB  - Parameter-Efficient Fine-Tuning (PEFT) methods have transformed the approach to fine-tuning large models for downstream tasks by enabling the adjustment of significantly fewer parameters than those in the original model matrices. In this work, we study the 'very low rank regime', where we fine-tune the lowest amount of parameters per linear layer for each considered PEFT method. We propose 1LoRA (Summation Low-Rank Adaptation), a compute, parameter and memory efficient fine-tuning method which uses the feature sum as fixed compression and a single trainable vector as decompression. Differently from state-of-the-art PEFT methods like LoRA, VeRA, and the recent MoRA, 1LoRA uses fewer parameters per layer, reducing the memory footprint and the computational cost. We extensively evaluate our method against state-of-the-art PEFT methods on multiple fine-tuning tasks, and show that our method not only outperforms them, but is also more parameter, memory and computationally efficient. Moreover, thanks to its memory efficiency, 1LoRA allows to fine-tune more evenly across layers, instead of focusing on specific ones (e.g. attention layers), improving performance further.
LB  - PUB:(DE-HGF)25
DO  - DOI:10.48550/arXiv.2503.08333
UR  - https://juser.fz-juelich.de/record/1045982
ER  -