File grouping for scientific data management

Shyamala Doraimani, Adriana Iamnitchi
2008 Proceedings of the 17th international symposium on High performance distributed computing - HPDC '08  
The analysis of data usage in a large set of real traces from a highenergy physics collaboration revealed the existence of an emergent grouping of files that we coined "filecules". This paper presents the benefits of using this file grouping for prestaging data and compares it with previously proposed file grouping techniques along a range of performance metrics. Our experiments with real workloads demonstrate that filecule grouping is a reliable and useful abstraction for data management in
more » ... ence Grids; that preserving time locality for data prestaging is highly recommended; that job reordering with respect to data availability has significant impact on throughput; and finally, that a relatively short history of traces is a good predictor for filecule grouping. Our experimental results provide lessons for workload modeling and suggest design guidelines for data management in dataintensive resource-sharing environments.
doi:10.1145/1383422.1383429 dblp:conf/hpdc/DoraimaniI08 fatcat:xvspqs5ijzdf3k3enwomsjnifu