openPMD – Open and F.A.I.R I/O for Particle-Mesh Data at the Exascale


openPMD – Open and F.A.I.R I/O for Particle-Mesh Data at the Exascale

Pöschel, F.; E, J.; Godoy, W. F.; Podhorszki, N.; Klasky, S.; Eisenhauer, G.; Davis, P. E.; Wan, L.; Gainaru, A.; Gu, J.; Koller, F.; Widera, R.; Bussmann, M.; Huebl, A.

This talk presents openPMD, an open and F.A.I.R. standard for particle-mesh data, and its impact in Exascale scientific workflows. The openPMD standard is made accessible to scientific software via the openPMD-api, a library for the description of scientific data. It approaches recent challenges posed by hardware heterogeneity by decoupling the data description in domain sciences, such as plasma physics simulations, from concrete implementations in hardware and IO. This concept helps us build a transition path from file-based IO to streaming-based workflows of scientific applications in an HPC environment. The streaming backend is provided by the ADIOS2 framework, developed at Oak Ridge National Laboratory.
This talk discusses two openPMD-based loosely coupled setups to demonstrate flexible applicability and to evaluate performance. In loose coupling, as opposed to tight coupling, two (or more) applications are executed separately, e.g. in individual MPI contexts, yet cooperate by exchanging data. This way, a streaming-based workflow allows for standalone codes instead of tightly-coupled plugins, using a unified streaming-aware API and leveraging high-speed communication infrastructure available in modern compute clusters for massive data exchange.
The presented setups show the potential for a more flexible use of compute resources brought by streaming IO as well as the ability to increase throughput by avoiding filesystem bottlenecks.

Keywords: high performance computing; big data; streaming; RDMA; openPMD; ADIOS

  • Open Access Logo Invited lecture (Conferences) (Online presentation)
    SIAM Conference on Parallel Processing for Scientific Computing (PP22), 23.-26.02.2022, Seattle, USA

Downloads

Permalink: https://www.hzdr.de/publications/Publ-35578
Publ.-Id: 35578