TY  - CONF
AU  - Aach, Marcel
AU  - Sarma, Rakesh
AU  - Inanc, Eray
AU  - Riedel, Morris
AU  - Lintermann, Andreas
TI  - Short Paper: Accelerating Hyperparameter Optimization Algorithms with Mixed Precision
PB  - ACM New York, NY, USA
M1  - FZJ-2023-04518
SP  - 1776–1779
PY  - 2023
AB  - Hyperparameter Optimization (HPO) of Neural Networks (NNs) is a computationally expensive procedure. On accelerators, such as NVIDIA Graphics Processing Units (GPUs) equipped with Tensor Cores, it is possible to speed-up the NN training by reducing the precision of some of the NN parameters, also referred to as mixed precision training. This paper investigates the performance of three popular HPO algorithms in terms of the achieved speed-up and model accuracy, utilizing early stopping, Bayesian, and genetic optimization approaches, in combination with mixed precision functionalities. The benchmarks are performed on 64 GPUs in parallel on three datasets: two from the vision and one from the Computational Fluid Dynamics domain. The results show that larger speed-ups can be achieved for mixed compared to full precision HPO if the checkpoint frequency is kept low. In addition to the reduced runtime, small gains in generalization performance on the test set are observed.
T2  - SC-W 2023: Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis
CY  - 12 Nov 2023 - 17 Nov 2023, Denver, CO (USA)
Y2  - 12 Nov 2023 - 17 Nov 2023
M2  - Denver, CO, USA
LB  - PUB:(DE-HGF)8
DO  - DOI:10.1145/3624062.3624259
UR  - https://juser.fz-juelich.de/record/1018062
ER  -