Contribution to a conference proceedings

FZJ-2023-04518

http://join2-wiki.gsi.de/foswiki/pub/Main/Artwork/join2_logo100x88.png Short Paper: Accelerating Hyperparameter Optimization Algorithms with Mixed Precision

Aach, M. (Corresponding author)FZJ* ; Sarma, R.FZJ* ; Inanc, E.FZJ* ; Riedel, M.FZJ* ; Lintermann, A.FZJ*

2023
ACM New York, NY, USA

This record in other databases:

Please use a persistent id in citations: doi:10.1145/3624062.3624259  doi:10.34734/FZJ-2023-04518

Abstract: Hyperparameter Optimization (HPO) of Neural Networks (NNs) is a computationally expensive procedure. On accelerators, such as NVIDIA Graphics Processing Units (GPUs) equipped with Tensor Cores, it is possible to speed-up the NN training by reducing the precision of some of the NN parameters, also referred to as mixed precision training. This paper investigates the performance of three popular HPO algorithms in terms of the achieved speed-up and model accuracy, utilizing early stopping, Bayesian, and genetic optimization approaches, in combination with mixed precision functionalities. The benchmarks are performed on 64 GPUs in parallel on three datasets: two from the vision and one from the Computational Fluid Dynamics domain. The results show that larger speed-ups can be achieved for mixed compared to full precision HPO if the checkpoint frequency is kept low. In addition to the reduced runtime, small gains in generalization performance on the test set are observed.

---

Contributing Institute(s):

1. Jülich Supercomputing Center (JSC)

Research Program(s):

1. 5111 - Domain-Specific Simulation & Data Life Cycle Labs (SDLs) and Research Groups (POF4-511) (POF4-511)
2. RAISE - Research on AI- and Simulation-Based Engineering at Exascale (951733) (951733)

Appears in the scientific report 2023

---

Database coverage:

---