NSFCollaborative Research: Elements: ProDM: Developing A Unified Progressive Data Management Library for Exascale Computational Science

Project description

Collaborative Research: Elements: ProDM: Developing A Unified Progressive Data Management Library for Exascale Computational Science

Effective management of scientific data produced by extreme-scale simulations and instruments is crucial for advancing scientific discoveries. Due to the scale of data and the diverse requirements of scientific analytics, there is a growing need to manage data in a progressive manner, such that users can stream as much data as they need to carry out their data analytics with reduced data movement and computation. However, little effort has been put into creating robust and scalable cyberinfrastructure services that link the recent algorithmic innovations in progressive methods with scientific data analytics, leaving these capabilities inaccessible to scientists. This project aims to develop a sustainable framework ProDM that supports the progressive management of scientific data to facilitate its use in scientific applications. The success of this project will enable new scientific research and novel findings by providing a new way to manage and analyze data. Furthermore, outcomes of this project will be delivered as publicly available software to enhance research cyberinfrastructure, promote education and teaching, and broaden participation in computing.

Publications

HPDC'26

Wenbo Li, Qian Gong, Xuan Wu, Jieyang Chen, Qing Liu, Xubin He, Norbert Podhorszki, Scott Klasky, and Xin Liang.
QProR: An Efficient Framework for Quantity-of-Interest Based Progressive Retrieval with Guaranteed Error Control.
Proceedings of the 35th ACM International Symposium on High-Performance Parallel and Distributed Computing, Cleveland, OH, July 13 - 16, 2026. [DOI]

ICDE'26

Mingze Xia, Yuxiao Li, Pu Jiao, Bei Wang, Xin Liang*, and Hanqi Guo.
Time-varying Vector Field Compression with Preserved Critical Point Trajectories.
Proceedings of the 42nd IEEE International Conference on Data Engineering, Montreal, Canada, May 4 - 8, 2026. (*: Corresponding author) [DOI]

SC'25

Franck Cappello et. al.
What to Support When You’re Compressing: The State of Practice, Gaps, and Opportunities for Scientific Data Compression.
Proceedings of the 2025 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, St. Louis, MO, USA, Nov 16 - 21, 2025. Acceptance Rate: 22.0% (137/623). [DOI]

SC'25

Yanliang Li+, Wenbo Li+, Qian Gong, Qing Liu, Norbert Podhorszki, Scott Klasky, Xin Liang, and Jieyang Chen.
HP-MDR: High-performance and Portable Data Refactoring and Progressive Retrieval with Advanced GPUs.
Proceedings of the 2025 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, St. Louis, MO, USA, Nov 16 - 21, 2025. Acceptance Rate: 22.0% (137/623). (+: Co-first authors) [DOI]

IPDPS'25

Jieyang Chen, Qian Gong, Xin Liang, Qing Liu, Lipeng Wan, Yanliang Li, Norbert Podhorszki, and Scott Klasky.
HPDR: High-Performance Portable Scientific Data Reduction Framework.
Proceedings of the 39th IEEE International Parallel & Distributed Processing Symposium, Milan, Italy, June 3 - June 7, 2025. Acceptance Rate: 24.7% (105/425). [DOI]

IPDPS'25

Pu Jiao, Sheng Di, Mingze Xia, Xuan Wu, Jinyang Liu, Xin Liang*, and Franck Cappello.
Improving the Efficiency of Interpolation-Based Scientific Data Compressors with Adaptive Quantization Index Prediction.
Proceedings of the 39th IEEE International Parallel & Distributed Processing Symposium, Milan, Italy, June 3 - June 7, 2025. Acceptance Rate: 24.7% (105/425). (*: Corresponding author) [DOI]

IPDPS'25

Xuan Wu, Sheng Di, Congrong Ren, Pu Jiao, Mingze Xia, Cheng Wang, Hanqi Guo, Xin Liang*, and Franck Cappello.
Enabling Efficient Error-controlled Lossy Compression for Unstructured Scientific Data (Best Paper Award).
Proceedings of the 39th IEEE International Parallel & Distributed Processing Symposium, Milan, Italy, June 3 - June 7, 2025. Acceptance Rate: 24.7% (105/425). (*: Corresponding author) [DOI]

SC'24

Xuan Wu, Qian Gong, Jieyang Chen, Qing Liu, Norbert Podhorszki, Xin Liang*, and Scott Klasky.
Error-controlled Progressive Retrieval of Scientific Data under Derivable Quantities of Interest.
Proceedings of the 2024 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, Atlanta, GA, USA, Nov 17 - 22, 2024. Acceptance Rate: 21.1% (99/470). (*: Corresponding author) [DOI]

IPDPS'24

Zizhe Jian, Sheng Di, Jinyang Liu, Kai Zhao, Xin Liang, Haiying Xu, Robert Underwood, Shixun Wu, Jiajun Huang, Zizhong Chen, and Franck Cappello.
CliZ: Optimizing Lossy Compression for Climate Datasets with Adaptive Fine-tuned Data Prediction.
Proceed- ings of 38th IEEE International Parallel & Distributed Processing Symposium, San Francisco, California, May 27 - May 31, 2024. Acceptance Rate: 26.1% (88/337). [DOI]

ICDE'24

Mingze Xia, Sheng Di, Franck Cappello, Pu Jiao, Kai Zhao, Jinyang Liu, Xuan Wu, Xin Liang*, and Hanqi Guo.
Preserving Topological Feature with Sign-of-Determinant Predicates in Lossy Compression: A Case Study of Vector Field Critical Points.
Proceedings of the 40th IEEE International Conference on Data Engineering, Utrecht, Netherlands, May 13 - 16, 2024. Acceptance Rate: 25.4% (376/1481). (*: Corresponding author) [DOI]