Using WLCG data management software to support other communities

Alastair Dewhurst, Catalin Condurache, Tom Byrne, Ian Johnson
2019 Zenodo  
When the LHC started data taking in 2009 the data rates were unprecedented for the time and forced the WLCG community develop a range of tools for managing their data across many different sites. A decade later other science communities are finding their data requirements have grown far beyond what they can easily manage and are looking for help. The RAL Tier-1's primary mission has always been to provide resources for the LHC experiments although 10% is set aside for non-LHC experiments. In
more » ... last 2 years the Tier-1 has received additional funding to support other scientific communities and now provides over 5 PB disk and 13PB Tape storage to them. RAL has run an FTS service for the LHC experiments for several years. Smaller scientific communities have used this for moving large volumes of data between sites but have frequently reported difficulties. In the past RAL also provided an LFC service for managing files stored on the Grid. The RAL Tier-1 should provide a complete data management solution for these communities and it was therefore decided to setup a Rucio instance to do this. In April 2018 a Rucio instance was setup at RAL which has so far been used by the AENEAS project. RAL is providing the development effort to allow a single Rucio instance to support multiple experiments. RAL also runs a DynaFed service which is providing an authentication and authorization layer in front of RAL S3 storage service.
doi:10.5281/zenodo.3599647 fatcat:ltgqmfvgi5hlpjzv75imcyg4r4