Poster (After Call)

FZJ-2022-03599

http://join2-wiki.gsi.de/foswiki/pub/Main/Artwork/join2_logo100x88.png OpenGPT-X - Training Large Language Models on HPC Systems

Penke, C. (Corresponding author)FZJ* ; John, C. M.FZJ* ; Herten, A.FZJ* ; Ebert, J.FZJ* ; Kesselheim, S.FZJ* ; Suarez, E.FZJ*

2022

Please use a persistent id in citations: http://hdl.handle.net/2128/32006

Abstract: Artificial neural networks represent an HPC workload with increasing importance. In particular the field of Natural Language Processing (NLP) has been undergoing a revolution in recent years. The training of ever larger language models, such as GPT-3, demands large HPC resources and has the potential to greatly impact everyday technology. The OpenGPT-X project was established in 2022 and aims to not leave this field to large tech companies but to provide an open, publicly funded alternative based on European values. The Jülich Supercomputing Centre is a consortium partner providing HPC infrastructure for the pre-training of the models. We research the optimization potential in the training process for example by using novel accelerator architectures.

---

Contributing Institute(s):

1. Jülich Supercomputing Center (JSC)

Research Program(s):

1. 5112 - Cross-Domain Algorithms, Tools, Methods Labs (ATMLs) and Research Groups (POF4-511) (POF4-511)

Appears in the scientific report 2022

---

Database coverage:

---