Scale and performance in semantic storage management of data grids

Stergios V. Anastasiadis, Syam Gadde, Jeffrey S. Chase
2005 International Journal on Digital Libraries  
Data grids are middleware systems that offer secure shared storage of massive scientific datasets over wide area networks. Main challenge in their design is to provide reliable storage, search and transfer of numerous or large files over geographically dispersed heterogeneous platforms. The Storage Resource Broker (SRB) is an example of such a system that has been deployed in multiple high-performance scientific projects during the past few years. In the present paper, we take a detailed look
more » ... several of its functional features, and examine its scalability using synthetic and tracebased workloads. Unlike traditional file systems, SRB uses a commodity database to manage both system and user-defined metadata. We quantitatively evaluate this decision, and draw insightful conclusions about its implications to the system architecture and performance characteristics. We find that the bulk transfer facilities of SRB demonstrate good scalability properties, and we identify the bottleneck resources across different data search and transfer tasks. We examine the sensitivity to several configuration parameters, and provide details about how different internal operations contribute to the overall performance.
doi:10.1007/s00799-004-0086-8 fatcat:6snhtfcxn5ekzlrydksjuq5ddy