Poster (After Call)

FZJ-2023-05810

http://join2-wiki.gsi.de/foswiki/pub/Main/Artwork/join2_logo100x88.png Accelerating massive data processing in Python with Heat

Comito, C. (Corresponding author)FZJ* ; Götz, M. ; Gutiérrez Hermosillo Muriedas, J. P. ; Hagemeier, B.FZJ* ; Hoppe, F. ; Knechtges, P. ; Krajsek, K.FZJ* ; Rüttgers, A. ; Streit, A. ; Tarnawa, M.FZJ*

2023

Abstract: Heat [1, 2] is an open-source Python library designed to address the challenges of working with massive data sets and harnessing the power of machine learning across disciplines. Developed collaboratively by within the Helmholtz Association (FZJ, KIT, and DLR), Heat offers cutting-edge capabilities for high-performance data analytics, machine learning, and deep learning.Heat provides a Numpy-like API that simplifies the development of scalable, GPU-accelerated applications. What sets Heat apart is its underlying data-parallelism, implemented on top of MPI, which significantly enhances efficiency and performance of data processing compared to traditional task-parallel frameworks.By exploring practical use cases in space science (materials engineering, atmospheric modeling, anomaly detection) and its potential as a backend for diverse data processing pipelines, we will illustrate how Heat can accelerate AI research and applications.[1] Götz, M., Debus, C., Coquelin, et al.: "HeAT - a Distributed and GPU-accelerated Tensor Framework for Data Analytics" [2] https://github.com/helmholtz-analytics/heat

---

Contributing Institute(s):

1. Jülich Supercomputing Center (JSC)

Research Program(s):

1. 5111 - Domain-Specific Simulation & Data Life Cycle Labs (SDLs) and Research Groups (POF4-511) (POF4-511)
2. 5112 - Cross-Domain Algorithms, Tools, Methods Labs (ATMLs) and Research Groups (POF4-511) (POF4-511)
3. SLNS - SimLab Neuroscience (Helmholtz-SLNS) (Helmholtz-SLNS)

Appears in the scientific report 2023

---