001046535 001__ 1046535
001046535 005__ 20251013202055.0
001046535 037__ $$aFZJ-2025-03853
001046535 1001_ $$0P:(DE-Juel1)188471$$aQuercia, Alessio$$b0$$eCorresponding author$$ufzj
001046535 1112_ $$aHelmholtz AI Conference$$cKarlsruhe$$d2025-06-03 - 2025-06-05$$wGermany
001046535 245__ $$aSummation Compression for Very-Low Rank Adaptation
001046535 260__ $$c2025
001046535 3367_ $$033$$2EndNote$$aConference Paper
001046535 3367_ $$2DataCite$$aOther
001046535 3367_ $$2BibTeX$$aINPROCEEDINGS
001046535 3367_ $$2DRIVER$$aconferenceObject
001046535 3367_ $$2ORCID$$aLECTURE_SPEECH
001046535 3367_ $$0PUB:(DE-HGF)6$$2PUB:(DE-HGF)$$aConference Presentation$$bconf$$mconf$$s1760337620_32146$$xAfter Call
001046535 502__ $$cRWTH Aachen
001046535 520__ $$aParameter-Efficient Fine-Tuning (PEFT) methods have transformed the approach to fine-tuning large models for downstream tasks by enabling the adjustment of significantly fewer parameters than those in the original model matrices. In this work, we study the "very low rank regime", where we fine-tune the lowest amount of parameters per linear layer for each considered PEFT method. We propose 1LoRA (Summation Low-Rank Adaptation), a compute, parameter and memory efficient fine-tuning method which uses the feature sum as fixed compression and a single trainable vector as decompression. Differently from state-of-the-art PEFT methods like LoRA, VeRA, and the recent MoRA, 1LoRA uses fewer parameters per layer, reducing the memory footprint and the computational cost. We extensively evaluate our method against state-of-the-art PEFT methods on multiple fine-tuning tasks, and show that our method not only outperforms them, but is also more parameter, memory and computationally efficient. Moreover, thanks to its memory efficiency, 1LoRA allows to fine-tune more evenly across layers, instead of focusing on specific ones (e.g. attention layers), improving performance further.
001046535 536__ $$0G:(DE-HGF)POF4-5112$$a5112 - Cross-Domain Algorithms, Tools, Methods Labs (ATMLs) and Research Groups (POF4-511)$$cPOF4-511$$fPOF IV$$x0
001046535 7001_ $$0P:(DE-Juel1)199019$$aCao, Zhuo$$b1$$ufzj
001046535 7001_ $$0P:(DE-Juel1)184644$$aBangun, Arya$$b2$$ufzj
001046535 7001_ $$0P:(DE-Juel1)175101$$aPaul, Richard Dominik$$b3$$ufzj
001046535 7001_ $$0P:(DE-Juel1)151166$$aMorrison, Abigail$$b4$$ufzj
001046535 7001_ $$0P:(DE-Juel1)188313$$aAssent, Ira$$b5$$ufzj
001046535 7001_ $$0P:(DE-Juel1)129394$$aScharr, Hanno$$b6$$ufzj
001046535 909CO $$ooai:juser.fz-juelich.de:1046535$$pVDB
001046535 9101_ $$0I:(DE-588b)5008462-8$$6P:(DE-Juel1)188471$$aForschungszentrum Jülich$$b0$$kFZJ
001046535 9101_ $$0I:(DE-588b)5008462-8$$6P:(DE-Juel1)199019$$aForschungszentrum Jülich$$b1$$kFZJ
001046535 9101_ $$0I:(DE-588b)5008462-8$$6P:(DE-Juel1)184644$$aForschungszentrum Jülich$$b2$$kFZJ
001046535 9101_ $$0I:(DE-588b)5008462-8$$6P:(DE-Juel1)175101$$aForschungszentrum Jülich$$b3$$kFZJ
001046535 9101_ $$0I:(DE-588b)5008462-8$$6P:(DE-Juel1)151166$$aForschungszentrum Jülich$$b4$$kFZJ
001046535 9101_ $$0I:(DE-588b)5008462-8$$6P:(DE-Juel1)188313$$aForschungszentrum Jülich$$b5$$kFZJ
001046535 9101_ $$0I:(DE-588b)5008462-8$$6P:(DE-Juel1)129394$$aForschungszentrum Jülich$$b6$$kFZJ
001046535 9131_ $$0G:(DE-HGF)POF4-511$$1G:(DE-HGF)POF4-510$$2G:(DE-HGF)POF4-500$$3G:(DE-HGF)POF4$$4G:(DE-HGF)POF$$9G:(DE-HGF)POF4-5112$$aDE-HGF$$bKey Technologies$$lEngineering Digital Futures – Supercomputing, Data Management and Information Security for Knowledge and Action$$vEnabling Computational- & Data-Intensive Science and Engineering$$x0
001046535 9141_ $$y2025
001046535 920__ $$lyes
001046535 9201_ $$0I:(DE-Juel1)IAS-8-20210421$$kIAS-8$$lDatenanalyse und Maschinenlernen$$x0
001046535 9201_ $$0I:(DE-Juel1)IAS-6-20130828$$kIAS-6$$lComputational and Systems Neuroscience$$x1
001046535 980__ $$aconf
001046535 980__ $$aVDB
001046535 980__ $$aI:(DE-Juel1)IAS-8-20210421
001046535 980__ $$aI:(DE-Juel1)IAS-6-20130828
001046535 980__ $$aUNRESTRICTED