001019553 001__ 1019553
001019553 005__ 20240102203541.0
001019553 037__ $$aFZJ-2023-05493
001019553 1001_ $$0P:(DE-Juel1)129347$$aKrajsek, Kai$$b0$$eCorresponding author$$ufzj
001019553 1112_ $$aISC High Performance 2023$$cHamburg$$d2023-05-21 - 2023-05-25$$wGermany
001019553 245__ $$aAdjoint MPI for Non-Additive Separable Loss Functions
001019553 260__ $$c2023
001019553 3367_ $$033$$2EndNote$$aConference Paper
001019553 3367_ $$2BibTeX$$aINPROCEEDINGS
001019553 3367_ $$2DRIVER$$aconferenceObject
001019553 3367_ $$2ORCID$$aCONFERENCE_POSTER
001019553 3367_ $$2DataCite$$aOutput Types/Conference Poster
001019553 3367_ $$0PUB:(DE-HGF)24$$2PUB:(DE-HGF)$$aPoster$$bposter$$mposter$$s1704204930_26654$$xAfter Call
001019553 520__ $$aMost contemporary frameworks for parallelizing deep learning models follow a fixed design pattern that favours a specific parallelization paradigm. More flexible libraries, such as PyTorch.distributed provides communication primitives but lack the flexibility of the capabilities of the MPI standard. PyTorch.distributed currently supports automatic differentiation for point-to-point communication, but not for collective communication. In contrast, flexible and effective communication patterns are useful for effective non-additive separable loss functions encountered in self-supervised contrastive learning approaches. This poster explores the implementation of the adjoint MPI concept in distributed deep learning models. It begins with an introduction to the fundamental principles of adjoint modelling, followed by an adjoint MPI concept for distributed Deep Learning which enables flexible parallelization in conjunction with existing libraries. The poster also covers implementation details and usage of the approach in contrastive self-supervised learning.
001019553 536__ $$0G:(DE-HGF)POF4-5111$$a5111 - Domain-Specific Simulation & Data Life Cycle Labs (SDLs) and Research Groups (POF4-511)$$cPOF4-511$$fPOF IV$$x0
001019553 536__ $$0G:(DE-Juel1)Helmholtz-SLNS$$aSLNS - SimLab Neuroscience (Helmholtz-SLNS)$$cHelmholtz-SLNS$$x1
001019553 909CO $$ooai:juser.fz-juelich.de:1019553$$pVDB
001019553 9101_ $$0I:(DE-588b)5008462-8$$6P:(DE-Juel1)129347$$aForschungszentrum Jülich$$b0$$kFZJ
001019553 9131_ $$0G:(DE-HGF)POF4-511$$1G:(DE-HGF)POF4-510$$2G:(DE-HGF)POF4-500$$3G:(DE-HGF)POF4$$4G:(DE-HGF)POF$$9G:(DE-HGF)POF4-5111$$aDE-HGF$$bKey Technologies$$lEngineering Digital Futures – Supercomputing, Data Management and Information Security for Knowledge and Action$$vEnabling Computational- & Data-Intensive Science and Engineering$$x0
001019553 9141_ $$y2023
001019553 920__ $$lyes
001019553 9201_ $$0I:(DE-Juel1)JSC-20090406$$kJSC$$lJülich Supercomputing Center$$x0
001019553 980__ $$aposter
001019553 980__ $$aVDB
001019553 980__ $$aI:(DE-Juel1)JSC-20090406
001019553 980__ $$aUNRESTRICTED