NCFS: On the Practicality and Extensibility of a Network-Coding-Based Distributed File System

Yuchong Hu, Chiu-Man Yu, Yan Kit Li, Patrick P. C. Lee, John C. S. Lui
2011 2011 International Symposium on Networking Coding  
An emerging application of network coding is to improve the robustness of distributed storage. Recent theoretical work has shown that a class of regenerating codes, which are based on the concept of network coding, can improve the data repair performance over traditional storage schemes such as erasure coding. However, there remain open issues regarding the feasibility of deploying regenerating codes in practical storage systems. We present NCFS, a distributed file system that realizes
more » ... ing codes under real network settings. NCFS transparently stripes data across multiple storage nodes, without requiring the storage nodes to coordinate among themselves. It adopts a layered design that allows extensibility, such that different storage schemes can be readily included into NCFS. We deploy and evaluate our NCFS prototype in different real network settings. In particular, we use NCFS to conduct an empirical study of different storage schemes, including the traditional erasure codes RAID-5 and RAID-6, and a special family of regenerating codes that are based on E-MBR [16]. Our work provides a practical and extensible platform for realizing theories of regenerating codes in distributed file systems.
doi:10.1109/isnetcod.2011.5978919 fatcat:dj5vjmx6ejga3nmqvpfmvgzzba