Hauptseite > Publikationsdatenbank > Summation Compression for Very-Low Rank Adaptation > print

001     1046535
005     20251013202055.0
037 _ _ |a FZJ-2025-03853
100 1 _ |a Quercia, Alessio
|0 P:(DE-Juel1)188471
|b 0
|e Corresponding author
|u fzj
111 2 _ |a Helmholtz AI Conference
|c Karlsruhe
|d 2025-06-03 - 2025-06-05
|w Germany
245 _ _ |a Summation Compression for Very-Low Rank Adaptation
260 _ _ |c 2025
336 7 _ |a Conference Paper
|0 33
|2 EndNote
336 7 _ |a Other
|2 DataCite
336 7 _ |a INPROCEEDINGS
|2 BibTeX
336 7 _ |a conferenceObject
|2 DRIVER
336 7 _ |a LECTURE_SPEECH
|2 ORCID
336 7 _ |a Conference Presentation
|b conf
|m conf
|0 PUB:(DE-HGF)6
|s 1760337620_32146
|2 PUB:(DE-HGF)
|x After Call
502 _ _ |c RWTH Aachen
520 _ _ |a Parameter-Efficient Fine-Tuning (PEFT) methods have transformed the approach to fine-tuning large models for downstream tasks by enabling the adjustment of significantly fewer parameters than those in the original model matrices. In this work, we study the "very low rank regime", where we fine-tune the lowest amount of parameters per linear layer for each considered PEFT method. We propose 1LoRA (Summation Low-Rank Adaptation), a compute, parameter and memory efficient fine-tuning method which uses the feature sum as fixed compression and a single trainable vector as decompression. Differently from state-of-the-art PEFT methods like LoRA, VeRA, and the recent MoRA, 1LoRA uses fewer parameters per layer, reducing the memory footprint and the computational cost. We extensively evaluate our method against state-of-the-art PEFT methods on multiple fine-tuning tasks, and show that our method not only outperforms them, but is also more parameter, memory and computationally efficient. Moreover, thanks to its memory efficiency, 1LoRA allows to fine-tune more evenly across layers, instead of focusing on specific ones (e.g. attention layers), improving performance further.
536 _ _ |a 5112 - Cross-Domain Algorithms, Tools, Methods Labs (ATMLs) and Research Groups (POF4-511)
|0 G:(DE-HGF)POF4-5112
|c POF4-511
|f POF IV
|x 0
700 1 _ |a Cao, Zhuo
|0 P:(DE-Juel1)199019
|b 1
|u fzj
700 1 _ |a Bangun, Arya
|0 P:(DE-Juel1)184644
|b 2
|u fzj
700 1 _ |a Paul, Richard Dominik
|0 P:(DE-Juel1)175101
|b 3
|u fzj
700 1 _ |a Morrison, Abigail
|0 P:(DE-Juel1)151166
|b 4
|u fzj
700 1 _ |a Assent, Ira
|0 P:(DE-Juel1)188313
|b 5
|u fzj
700 1 _ |a Scharr, Hanno
|0 P:(DE-Juel1)129394
|b 6
|u fzj
909 C O |o oai:juser.fz-juelich.de:1046535
|p VDB
910 1 _ |a Forschungszentrum Jülich
|0 I:(DE-588b)5008462-8
|k FZJ
|b 0
|6 P:(DE-Juel1)188471
910 1 _ |a Forschungszentrum Jülich
|0 I:(DE-588b)5008462-8
|k FZJ
|b 1
|6 P:(DE-Juel1)199019
910 1 _ |a Forschungszentrum Jülich
|0 I:(DE-588b)5008462-8
|k FZJ
|b 2
|6 P:(DE-Juel1)184644
910 1 _ |a Forschungszentrum Jülich
|0 I:(DE-588b)5008462-8
|k FZJ
|b 3
|6 P:(DE-Juel1)175101
910 1 _ |a Forschungszentrum Jülich
|0 I:(DE-588b)5008462-8
|k FZJ
|b 4
|6 P:(DE-Juel1)151166
910 1 _ |a Forschungszentrum Jülich
|0 I:(DE-588b)5008462-8
|k FZJ
|b 5
|6 P:(DE-Juel1)188313
910 1 _ |a Forschungszentrum Jülich
|0 I:(DE-588b)5008462-8
|k FZJ
|b 6
|6 P:(DE-Juel1)129394
913 1 _ |a DE-HGF
|b Key Technologies
|l Engineering Digital Futures – Supercomputing, Data Management and Information Security for Knowledge and Action
|1 G:(DE-HGF)POF4-510
|0 G:(DE-HGF)POF4-511
|3 G:(DE-HGF)POF4
|2 G:(DE-HGF)POF4-500
|4 G:(DE-HGF)POF
|v Enabling Computational- & Data-Intensive Science and Engineering
|9 G:(DE-HGF)POF4-5112
|x 0
914 1 _ |y 2025
920 _ _ |l yes
920 1 _ |0 I:(DE-Juel1)IAS-8-20210421
|k IAS-8
|l Datenanalyse und Maschinenlernen
|x 0
920 1 _ |0 I:(DE-Juel1)IAS-6-20130828
|k IAS-6
|l Computational and Systems Neuroscience
|x 1
980 _ _ |a conf
980 _ _ |a VDB
980 _ _ |a I:(DE-Juel1)IAS-8-20210421
980 _ _ |a I:(DE-Juel1)IAS-6-20130828
980 _ _ |a UNRESTRICTED

LibraryCollectionCLSMajorCLSMinorLanguageAuthor
Marc 21