001034067 001__ 1034067
001034067 005__ 20250203103442.0
001034067 0247_ $$2datacite_doi$$a10.34734/FZJ-2024-06888
001034067 037__ $$aFZJ-2024-06888
001034067 1001_ $$0P:(DE-Juel1)192254$$aPenke, Carolin$$b0$$eCorresponding author$$ufzj
001034067 1112_ $$aOpenGPT-X Forum 2024$$cBerlin$$d2024-11-05 - 2024-11-05$$wGermany
001034067 245__ $$aMathematical Techniques to Reduce Memory Requirements in Deep Learning
001034067 260__ $$c2024
001034067 3367_ $$033$$2EndNote$$aConference Paper
001034067 3367_ $$2DataCite$$aOther
001034067 3367_ $$2BibTeX$$aINPROCEEDINGS
001034067 3367_ $$2DRIVER$$aconferenceObject
001034067 3367_ $$2ORCID$$aLECTURE_SPEECH
001034067 3367_ $$0PUB:(DE-HGF)6$$2PUB:(DE-HGF)$$aConference Presentation$$bconf$$mconf$$s1736491132_20972$$xOther
001034067 520__ $$aWe present a method to substantially lower memory requirements during the training of deep neural networks, based on the GaLore (Gradient Low-Rank Projection) training framework. A rapid decay of singular values in gradient matrices permits the use of low-rank bases to encapsulate the relevant subspaces, reducing the memory requirements for storing optimizer states between iterations. A novel, rank-adaptive, GPU-optimized version of the randomized range finder algorithm is employed to exploit this property and future research directions are discussed.
001034067 536__ $$0G:(DE-HGF)POF4-5112$$a5112 - Cross-Domain Algorithms, Tools, Methods Labs (ATMLs) and Research Groups (POF4-511)$$cPOF4-511$$fPOF IV$$x0
001034067 536__ $$0G:(DE-Juel-1)68GX21007F$$aOpenGPT-X - Aufbau eines Gaia-X Knotens für große KI-Sprachmodelle und innovative Sprachapplikations-Services; Teilvorhaben: Optimierung und Skalierung auf großen HPC-Systemen (68GX21007F)$$c68GX21007F$$x1
001034067 8564_ $$uhttps://juser.fz-juelich.de/record/1034067/files/LowRankRepresentationsDL.pdf$$yOpenAccess
001034067 909CO $$ooai:juser.fz-juelich.de:1034067$$pdriver$$pVDB$$popen_access$$popenaire
001034067 9101_ $$0I:(DE-588b)5008462-8$$6P:(DE-Juel1)192254$$aForschungszentrum Jülich$$b0$$kFZJ
001034067 9131_ $$0G:(DE-HGF)POF4-511$$1G:(DE-HGF)POF4-510$$2G:(DE-HGF)POF4-500$$3G:(DE-HGF)POF4$$4G:(DE-HGF)POF$$9G:(DE-HGF)POF4-5112$$aDE-HGF$$bKey Technologies$$lEngineering Digital Futures – Supercomputing, Data Management and Information Security for Knowledge and Action$$vEnabling Computational- & Data-Intensive Science and Engineering$$x0
001034067 9141_ $$y2024
001034067 915__ $$0StatID:(DE-HGF)0510$$2StatID$$aOpenAccess
001034067 9201_ $$0I:(DE-Juel1)JSC-20090406$$kJSC$$lJülich Supercomputing Center$$x0
001034067 9801_ $$aFullTexts
001034067 980__ $$aconf
001034067 980__ $$aVDB
001034067 980__ $$aUNRESTRICTED
001034067 980__ $$aI:(DE-Juel1)JSC-20090406