Publications Repository - Helmholtz-Zentrum Dresden-Rossendorf

1 Publication

A scalable pipeline for COVID-19: the case study of Germany, Czechia and Poland

Abdussalam, W.; Mertel, A.; Fan, K.; Schüler, L.; Schlechte-Welnicz, W.; Calabrese, J.


Throughout the coronavirus disease 2019 (COVID-19) pandemic, decision makers have relied on forecasting models to determine and implement non-pharmaceutical interventions (NPI). In building the forecasting models, continuously updated datasets from various stakeholders including developers, analysts, and testers are required to provide precise predictions. Here we report the design of a scalable pipeline which serves as a data synchronization to support inter-country top-down spatiotemporal observations and forecasting models of COVID-19, named the \textit{where2test}, for Germany, Czechia and Poland. We have built an operational data store (ODS) using PostgreSQL to continuously consolidate datasets from multiple data sources, perform collaborative work, facilitate high performance data analysis, and trace changes. The ODS has been built not only to store the COVID-19 data from Germany, Czechia, and Poland but also other areas. Employing the dimensional fact model, a schema of metadata is capable of synchronizing the various structures of data from those regions, and is scalable to the entire world. Next, the ODS is populated using batch Extract, Transfer, and Load (ETL) jobs. The SQL queries are subsequently created to reduce the need for pre-processing data for users. The data can then support not only forecasting using a version-controlled Arima-Holt model and other analyses to support decision making, but also risk calculator and optimisation apps. The data synchronization runs at a daily interval, which is displayed at

Keywords: COVID-19; Database Server; Data warehouse; Data pipeline; Database model; Forecast

  • Open Access Logo Contribution to proceedings
    DEco - First International Workshop on Data Ecosystems in conjunction with very large databases, 05.-09.09.2022, Sydney Hybrid, Australia
    Proceedings of DEco
    DOI: 10.48550/arXiv.2208.12928



Years: 2023 2022 2021 2020 2019 2018 2017 2016 2015