Maestro Project Introduction

Pleiter, Dirk

Conference Presentation (Invited)

FZJ-2020-00355

Maestro Project Introduction

Pleiter, D. (Corresponding author)FZJ*

2019

EuroHPC Summit Week 2019, Poznań, Poland, 13 May 2019 - 18 May 2019

Please use a persistent id in citations: http://hdl.handle.net/2128/23940

Abstract: In this talk, we provide an introduction to the Maestro project.This project was started in September 2018 involving the following partners:Appentra, CEA, Cray, CSCS, ECMWF, Forschungszentrum Juelich, Seagate.The goal is to develop and demonstrate a new middleware that aims at addressingtwo key shortcomings observed for today's supercomputing solutions.First, available solutions typically lack data awareness as they are optimisedfor filling processing pipelines, not for managing data.Furthermore, available solutions lack memory awareness, i.e. available middlewaredo not have information about available memory (or storage) hardware and thusare not able to make data transport decisions.Maestro applies a co-design approach for designing a solution addressing these challenges.The co-design applications, in particular, include data-intensive applications likethe weather prediction workflow from ECMWF.In this case, simulations of the IFS model produce huge amounts of data that are consumedby other applications, which generate products for ECMWF's products member states and customers.This matches well Maestro's concept of providing a pool of data objects, wheredata producers give objects to the pool and data consumers take objects from the pool.Using suitable metadata, the Maestro middleware can manage storing and transport of data objectstaking the hardware architecture into account, on which the applications are being executed.We will conclude this talk with an outlook on the work of the Maestro project.

Contributing Institute(s):