Lecture (Invited)

FZJ-2019-06539

http://join2-wiki.gsi.de/foswiki/pub/Main/Artwork/join2_logo100x88.png Parallel I/O and Portable Data Formats

Lührs, S. (Corresponding author)FZJ*

2019

Please use a persistent id in citations: http://hdl.handle.net/2128/23592

Abstract: This course will be about parallel I/O with a special focus on portable data formats. It will introduce the use of the HDF5 and NetCDF (NetCDF4 and PnetCDF) library interfaces, and hands-on exercises (in C/C++ or Fortran) will allow to immediately test and understand their usage. Performance hints, optimization potential, and best practices for I/O will be discussed in detail throughout the whole course.Numerical simulations conducted on current HPC systems face an ever growing need for scalability pushing the limitations on size and properties that can be accurately simulated. Therefore, ever larger data sets have to be processed, be it reading input data or writing results. Serial approaches on handling I/O in a parallel application will dominate the performance on massively parallel systems, leaving a lot of computing resources idle during those serial I/O phases.In addition to the need for parallel I/O, input and output data is often processed on different and maybe even heterogeneous platforms. Conversion processes can impose a high level of maintenance when different data representations are needed. Portable, self-describing data formats such as HDF5 and netCDF can help to solve these problems.

---

Contributing Institute(s):

1. Jülich Supercomputing Center (JSC)

Research Program(s):

1. 511 - Computational Science and Mathematical Methods (POF3-511) (POF3-511)

Appears in the scientific report 2019

---

Database coverage:

---