TY  - CONF
AU  - Finkbeiner, Jan
AU  - Gmeinder, Thomas
AU  - Pupilli, Mark
AU  - Titterton, Alexander
AU  - Neftci, Emre
TI  - Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models
M1  - FZJ-2025-01040
SP  - 11996-12005
PY  - 2024
AB  - Current AI training infrastructure is dominated by single instruction multiple data (SIMD) and systolic array architectures, such as Graphics Processing Units (GPUs) and Tensor Processing Units (TPUs), that excel at accelerating parallel workloads and dense vector matrix multiplications. Potentially more efficient neural network models utilizing sparsity and recurrence cannot leverage the full power of SIMD processor and are thus at a severe disadvantage compared to today’s prominent parallel architectures like Transformers and CNNs, thereby hindering the path towards more sustainable AI. To overcome this limitation, we explore sparse and recurrent model training on a massively parallel multiple instruction multiple data (MIMD) architecture with distributed local memory. We implement a training routine based on backpropagation through time (BPTT) for the brain-inspired class of Spiking Neural Networks (SNNs) that feature binary sparse activations. We observe a massive advantage in using sparse activation tensors with a MIMD processor, the Intelligence Processing Unit (IPU) compared to GPUs. On training workloads, our results demonstrate 5-10× throughput gains compared to A100 GPUs and up to 38× gains for higher levels of activation sparsity, without a significant slowdown in training convergence or reduction in final model performance. Furthermore, our results show highly promising trends for both single and multi IPU configurations as we scale up to larger model sizes. Our work paves the way towards more efficient,non-standard models via AI training hardware beyond GPUs, and competitive large scale SNN models.
T2  - AAAI Conference on Artificial Intelligence
CY  - 27 Feb 2024 - 4 Mar 2024, Vancouver (Canada)
Y2  - 27 Feb 2024 - 4 Mar 2024
M2  - Vancouver, Canada
LB  - PUB:(DE-HGF)8
UR  - https://juser.fz-juelich.de/record/1037902
ER  -