Publications Repository - Helmholtz-Zentrum Dresden-Rossendorf


openPMD – F.A.I.R Scientific I/O at the Exascale

Pöschel, F.; Hübl, A.

This talk presents openPMD, an open and F.A.I.R. standard for particle-mesh data, and its impact in heterogeneous scientific workflows.
Particle accelerator codes need to span various time and length scales, leading to data processing pipelines consisting of multiple heterogeneous codes.
Standardization of physical data helps bridging the different models with a commonly-understood markup, creating interoperable and flexible workflows.

The openPMD standard is made accessible to scientific software via the openPMD-api, a library for the description of scientific data.
The backend implementations of the openPMD-api are based on established I/O framworks such as HDF5 and ADIOS2, and also include a scalable streaming backend for HPC workflows, provided by ADIOS2.
The talk gives an insight into the existing ecosystem of openPMD and describes the basic concepts of the data markup.

It shortly illuminates recent trends in large-scale I/O and their impact on scientific compute workflows. While traditional attempts at counteracting such trends, e.g. through compression, remain available in the openPMD-api, we propose loose coupling and online analysis via streaming workflows as a sustainable solution that avoids parallel filesystem bottlenecks.

Keywords: high performance computing; F.A.I.R; particle-mesh data; openPMD; streaming; big data

  • Open Access Logo Invited lecture (Conferences)
    EuroNNAc Special Topics Workshop, 18.-24.09.2022, La Biodola Bay, Isola d'Elba, Italien
  • Open Access Logo Poster
    EuroNNAc Special Topics Workshop, 18.-24.09.2022, La Biodola Bay, Isola d'Elba, Italien
  • Open Access Logo Poster
    8. Annual MT Meeting, 26.-27.09.2022, Hamburg, Deutschland

Downloads

Permalink: https://www.hzdr.de/publications/Publ-35580
Publ.-Id: 35580