Scalable full-text search for petascale file systems

Andrew W. Leung, Ethan L. Miller
2008 2008 3rd Petascale Data Storage Workshop  
As file system capacities reach the petascale, it is becoming increasingly difficult for users to organize, find, and manage their data. File system search has the potential to greatly improve how users manage and access files. Unfortunately, existing file system search is designed for smaller scale systems, making it difficult for existing solutions to scale to petascale files systems. In this paper, we motivate the importance of file system search in petascale file systems and present a new
more » ... lltext file system search design for petascale file systems. Unlike existing solutions, our design exploits file system properties. Using a novel index partitioning mechanism that utilizes file system namespace locality, we are able to improve search scalability and performance and we discuss how such a design can potentially improve search security and ranking. We describe how our design can be implemented within the Ceph petascale file system.
doi:10.1109/pdsw.2008.4811884 fatcat:nhgxd5h46nbhlfz55sxrvmmady