Conference Presentation (Other)

FZJ-2024-06888

http://join2-wiki.gsi.de/foswiki/pub/Main/Artwork/join2_logo100x88.png Mathematical Techniques to Reduce Memory Requirements in Deep Learning

Penke, C. (Corresponding author)FZJ*

2024

This record in other databases:

Please use a persistent id in citations: doi:10.34734/FZJ-2024-06888

Abstract: We present a method to substantially lower memory requirements during the training of deep neural networks, based on the GaLore (Gradient Low-Rank Projection) training framework. A rapid decay of singular values in gradient matrices permits the use of low-rank bases to encapsulate the relevant subspaces, reducing the memory requirements for storing optimizer states between iterations. A novel, rank-adaptive, GPU-optimized version of the randomized range finder algorithm is employed to exploit this property and future research directions are discussed.

---

Contributing Institute(s):

1. Jülich Supercomputing Center (JSC)

Research Program(s):

1. 5112 - Cross-Domain Algorithms, Tools, Methods Labs (ATMLs) and Research Groups (POF4-511) (POF4-511)
2. OpenGPT-X - Aufbau eines Gaia-X Knotens für große KI-Sprachmodelle und innovative Sprachapplikations-Services; Teilvorhaben: Optimierung und Skalierung auf großen HPC-Systemen (68GX21007F) (68GX21007F)

Appears in the scientific report 2024

---

Database coverage:

---