Filters








133 Hits in 5.3 sec

Optimizing Lossy Compression Rate-Distortion from Automatic Online Selection between SZ and ZFP [article]

Dingwen Tao, Sheng Di, Xin Liang, Zizhong Chen, Franck Cappello
2019 arXiv   pre-print
SZ and ZFP are the two leading lossy compressors available to compress scientific data sets.  ...  To optimize for rate-distortion, we investigate the principles of SZ and ZFP.  ...  platforms, to support the nation's exascale computing imperative.  ... 
arXiv:1806.08901v2 fatcat:wxujbiwfqve23ooedoifk627la

cuSZ: An Efficient GPU-Based Error-Bounded Lossy Compression Framework for Scientific Data [article]

Jiannan Tian, Sheng Di, Kai Zhao, Cody Rivera, Megan Hickman Fulp, Robert Underwood, Sian Jin, Xin Liang, Jon Calhoun, Dingwen Tao, Franck Cappello
2020 arXiv   pre-print
To the best of our knowledge, cuSZ is the first error-bounded lossy compressor on GPUs for scientific data.  ...  Error-bounded lossy compression is a state-of-the-art data reduction technique for HPC applications because it not only significantly reduces storage overhead but also can retain high fidelity for postanalysis  ...  platforms, to support the nations exascale computing imperative.  ... 
arXiv:2007.09625v1 fatcat:f4sq3abcmvehvm4b6wishmudie

State of the Art and Future Trends in Data Reduction for High-Performance Computing

2020 Supercomputing Frontiers and Innovations  
This survey paper provides an overview of leveraging points found in high-performance computing (HPC) systems and suitable mechanisms to reduce data volumes.  ...  After introducing relevant use-cases, an overview of modern lossless and lossy compression algorithms and their respective usage at the application and file system layer is given.  ...  Acknowledgements Parts of this publication were enabled by the following projects: This paper is distributed under the terms of the Creative Commons Attribution-Non Commercial 3.0 License which permits  ... 
doi:10.14529/jsfi200101 fatcat:rcaotdomv5frfpjnmrf2giubla

CEAZ: Accelerating Parallel I/O Via Hardware-Algorithm Co-Designed Adaptive Lossy Compression [article]

Chengming Zhang, Sian Jin, Tong Geng, Jiannan Tian, Ang Li, Dingwen Tao
2021 arXiv   pre-print
To this end, many previous works have studied using error-bounded lossy compressors to reduce the data size and improve the I/O performance.  ...  Moreover, we derive a theoretical analysis to support a precise control of compression ratio under an error-bounded compression mode, enabling accurate offline Huffman codewords generation.  ...  Thus, we need to develop a high-throughput lossy compressor to effectively accelerate parallel I/O for HPC applications.  ... 
arXiv:2106.13306v2 fatcat:42fvquu3trcgxncxwdl5izksra

Data Compression for the Exascale Computing Era — Survey

2014 Supercomputing Frontiers and Innovations  
This paper also discusses how the errors introduced by lossy compressions are controlled and the tradeoffs with the compression ratio.  ...  Traditional lossless compression techniques that look for repeated patterns are ineffective for scientific data in which high-precision data is used and hence common patterns are rare to find.  ...  This research used resources of the National Energy Research Scientific Computing Center, which is supported by the Office of Science of the U.S. Department of Energy under Contract No.  ... 
doi:10.14529/jsfi140205 fatcat:qc6b6fpjg5aknebvkirfly6j24

A Novel Memory-Efficient Deep Learning Training Framework via Error-Bounded Lossy Compression [article]

Sian Jin, Guanpeng Li, Shuaiwen Leon Song, Dingwen Tao
2020 arXiv   pre-print
Different from the state-of-the-art solutions that adopt image-based lossy compressors such as JPEG to compress the activation data, our framework purposely designs error-bounded lossy compression with  ...  In this paper, we propose a novel memory-driven high performance DNN training framework that leverages error-bounded lossy compression to significantly reduce the memory requirement for training in order  ...  Figure 3 : 3 Sample error distribution of activation data compressed by cuSZ lossy compression with error bound 10 −4 .  ... 
arXiv:2011.09017v1 fatcat:qcsstdu7ozblpo6k5tr3x77edy

Optimizing Error-Bounded Lossy Compression for Scientific Data on GPUs [article]

Jiannan Tian, Sheng Di, Xiaodong Yu, Cody Rivera, Kai Zhao, Sian Jin, Yunhe Feng, Xin Liang, Dingwen Tao, Franck Cappello
2021 arXiv   pre-print
Error-bounded lossy compression is a critical technique for significantly reducing scientific data volumes.  ...  We identify that data sparsity and data smoothness are key factors for high compression throughputs.  ...  platforms, to support the nation's exascale computing imperative.  ... 
arXiv:2105.12912v3 fatcat:a5ayaxi4brdpvjzjqofqkmdfpu

Z-checker

Dingwen Tao, Sheng Di, Hanqi Guo, Zizhong Chen, Franck Cappello
2017 The international journal of high performance computing applications  
To the best of our knowledge, Z-checker is the first tool designed to assess lossy compression comprehensively for scientific datasets.  ...  the compression error.  ...  He currently serves as a subject area editor for Elsevier Parallel Computing journal and an associate editor for the IEEE Transactions on Parallel and Distributed Systems.  ... 
doi:10.1177/1094342017737147 fatcat:mmcug266vjdbljhe2qfuwjrwtq

Compression Challenges in Large Scale Partial Differential Equation Solvers

Sebastian Götschel, Martin Weiser
2019 Algorithms  
This paper surveys data compression challenges and discusses examples of corresponding solution approaches for PDE problems, covering all levels of the memory hierarchy from mass storage up to the main  ...  For large problems, they involve huge amounts of data that need to be stored and transmitted on all levels of the memory hierarchy.  ...  Acknowledgments: We thank Florian Wende for implementing mixed-precision preconditioners, Alexander Kammeyer for implementation and testing of checkpoint/restart, and Thomas Steinke for many helpful discussions  ... 
doi:10.3390/a12090197 fatcat:6ijsicazfbgxfgnma5sb4kow4q

Full-state quantum circuit simulation by using data compression

Xin-Chuan Wu, Sheng Di, Emma Maitreyee Dasgupta, Franck Cappello, Hal Finkel, Yuri Alexeev, Frederic T. Chong
2019 Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '19  
Our approach optimizes for compression speed and makes sure that errors due to lossy compression are uncorrelated, an important property for comparing simulation output with physical machines.  ...  Specifically, we develop a hybrid solution by combining the lossless compression and our tailored lossy compression method with adaptive error bounds at each timestep of the simulation.  ...  platforms, to support the nation's exascale computing imperative.  ... 
doi:10.1145/3295500.3356155 dblp:conf/sc/WuDDCFAC19 fatcat:s6rezyizhjaipofjld5hg265mu

COMET: A Novel Memory-Efficient Deep Learning Training Framework by Using Error-Bounded Lossy Compression [article]

Sian Jin, Chengming Zhang, Xintong Jiang, Yunhe Feng, Hui Guan, Guanpeng Li, Shuaiwen Leon Song, Dingwen Tao
2021 arXiv   pre-print
Based on these analyses, we optimize the error-bounded lossy compression and propose an adaptive error-bound control scheme for activation data compression.  ...  Different from the state-of-the-art solutions that adopt image-based lossy compressors (such as JPEG) to compress the activation data, our framework purposely adopts error-bounded lossy compression with  ...  The authors acknowledge the Texas Advanced Computing Center (TACC) at The University of Texas at Austin for providing access to the Longhorn system that has contributed to the research results reported  ... 
arXiv:2111.09562v1 fatcat:3cwwocln65gyhadortqx43vf4y

SDC Resilient Error-bounded Lossy Compressor [article]

Sihuan Li, Sheng Di, Kai Zhao, Xin Liang, Zizhong Chen, Franck Cappello
2020 arXiv   pre-print
Lossy compression is one of the most important strategies to resolve the big science data issue, however, little work was done to make it resilient against silent data corruptions (SDC).  ...  In this paper, we propose an SDC resilient error-bounded lossy compressor upon the SZ compression framework.  ...  Accordingly, error-bounded lossy compression has been thought of as one of the best ways to resolve today's big science data issue.  ... 
arXiv:2010.03144v1 fatcat:35b7j4goc5fm3fbqzes77vmyam

A workflow for seismic imaging with quantified uncertainty [article]

Carlos H. S. Barbosa, Liliane N. O. Kunstmann, Rômulo M. Silva, Charlan D. S. Alves, Bruno S. Silva, Djalma M. S. Filho, Marta Mattoso, Fernando A. Rochinha, Alvaro L.G.A. Coutinho
2020 arXiv   pre-print
High levels of data compression are applied to reduce data transfer among workflow activities and data storage.  ...  We observe excellent weak and strong scalability, and results suggest that the use of lossy data compression does not hamper the seismic imaging uncertainty quantification.  ...  Computer time on Stampede is provided by TACC, the University of Texas at Austin. The US NSF supports Stampede under award ACI-1134872.  ... 
arXiv:2001.06444v2 fatcat:bk7ytxjgybgxjjrtpxifjnn7ki

On the path to sustainable, scalable, and energy-efficient data analytics: Challenges, promises, and future directions

Sriram Lakshminarasimhan, Prabhat Kumar, Wei-keng Liao, Alok Choudhary, Vipin Kumar, Nagiza F. Samatova
2012 2012 International Green Computing Conference (IGCC)  
We propose a number of future directions that could be pursued on the path to sustainable data analytics at scale.  ...  In this paper, we present a number of recently created "secret sauces" that promise to address some of these challenges.  ...  Department of Energy, Office of Science and the U.S. National Science Foundation (Expeditions in Computing). Oak Ridge National Laboratory is managed by UT-Battelle for the LLC U.S.  ... 
doi:10.1109/igcc.2012.6322265 dblp:conf/green/LakshminarasimhanKLCKS12 fatcat:dremvfojkre5vm2n7qhimzxjgm

Assessing the effects of data compression in simulations using physically motivated metrics

Daniel Laney, Steven Langer, Christopher Weber, Peter Lindstrom, Al Wegener
2013 Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '13  
Rather than applying classical error metrics from signal processing, we utilize physics-based metrics appropriate for each code to evaluate the impact of compression.  ...  This paper examines whether lossy compression can be used effectively in physics simulations as a possible strategy to combat the expected data-movement bottleneck in future high performance computing  ...  This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.  ... 
doi:10.1145/2503210.2503283 dblp:conf/sc/LaneyLWLW13 fatcat:yrn5d6im25byfgrx72wosbswri
« Previous Showing results 1 — 15 out of 133 results