Filters








20,390 Hits in 4.0 sec

Optimal Systematic Distributed Storage Codes with Fast Encoding [article]

Preetum Nakkiran, K.V. Rashmi, Kannan Ramchandran
2015 arXiv   pre-print
We also present general ways of transforming existing storage and repair optimal codes to enable fast encoding through sparsity.  ...  Erasure codes are being increasingly used in distributed-storage systems in place of data-replication, since they provide the same level of reliability with much lower storage overhead.  ...  Introduction Erasure codes are being increasingly used in distributed-storage systems instead of replication, since they provide the same level of reliability with much less storage overhead.  ... 
arXiv:1509.01858v1 fatcat:pqbtdvm54zafdcaqae3ce6l22a

Fast Product-Matrix Regenerating Codes [article]

Nicolas Le Scouarnec
2014 arXiv   pre-print
Distributed storage systems support failures of individual devices by the use of replication or erasure correcting codes.  ...  We implement the original Product-Matrix Regenerating codes as well as a new optimization we propose and show that the resulting optimized codes allow achieving 790 MB/s for encoding in typical settings  ...  For systematic codes with k = 8, our optimized product-matrix codes encode at 790 MB/s when Reed-Solomon codes encode at 1640 MB/s.  ... 
arXiv:1412.3022v1 fatcat:k7g67mfnofbwpoiv3376ifpzhe

Performance evaluation of the Mojette erasure code for fault-tolerant distributed hot data storage [article]

Dimitri Pertin, Didier Féron, Alexandre Van Kempen, Benoît Parrein
2015 arXiv   pre-print
Packet erasure codes are today a real alternative to replication in fault tolerant distributed storage systems.  ...  Those very good performances allow to deploy Mojette erasure code for hot data distributed storage and I/O intensive applications.  ...  Indeed, to encode a 8 KB block with a (12, 8) code, the encoder splits 8 KB into 8 packets of one kilobyte and produces 4 encoded packets for systematic codes, while non-systematic codes have to compute  ... 
arXiv:1504.07038v1 fatcat:j4ciypzydveazkggprs3cclnfe

RapidRAID: Pipelined Erasure Codes for Fast Data Archival in Distributed Storage Systems [article]

Lluis Pamies-Juarez, Anwitaman Datta, Frederique Oggier
2012 arXiv   pre-print
We further present RapidRAID codes, an explicit family of pipelined erasure codes which provides fast archival without compromising either data reliability or storage overheads.  ...  In contrast, this paper addresses the migration from replication to encoding: traditionally erasure coding is an atomic operation in that a single node with the whole object encodes and uploads all the  ...  codes, aiming at optimizing the data archival in distributed storage systems.  ... 
arXiv:1207.6744v2 fatcat:wqc3rxi3djcbjo7psmzruknloq

Applied Erasure Coding in Networks and Distributed Storage [article]

Katina Kralevska
2018 arXiv   pre-print
implementation of new erasure codes for large-scale distributed storage systems that provide savings in the storage and network resources compared to state-of-the-art codes; and - Provision of a unified  ...  Methods from both coding and information theory have been applied to network coding, Optical Packet Switching (OPS) networks and distributed storage systems.  ...  Regenerating Codes Classical MDS codes are optimal in terms of the storage-reliability tradeoff, but they still do not give a good answer to the key question: How to encode the data in a distributed way  ... 
arXiv:1803.01358v1 fatcat:ooellwzlkzekxdhm75snjpeggu

Founsure 1.0: An erasure code library with efficient repair and update features

Şuayb Ş. Arslan
2021 SoftwareX  
Its implementation utilizes compiler optimizations and multi-threading to generate the right assembly code for the given multi-core CPU architecture with vector processing capabilities.  ...  Founsure library enables efficient encoding, decoding, repairs/rebuilds, and updates while all the required data storage and computations are distributed across the network nodes.  ...  Unlike systematic codes in which the data is explicit at the encoder output, Founsure comes with the non-systematic format as mentioned before.  ... 
doi:10.1016/j.softx.2021.100662 fatcat:uxjrrrobprdpzat37ahpg2cyue

Founsure 1.0: An Erasure Code Library with Efficient Repair and Update Features [article]

Şuayb Ş. Arslan
2021 arXiv   pre-print
Its implementation utilizes compiler optimizations and multi-threading to generate the right assembly code for the given multi-core CPU architecture with vector processing capabilities.  ...  Founsure library enables efficient encoding, decoding, repairs/rebuilds, and updates while all the required data storage and computations are distributed across the network nodes.  ...  As can be seen, with almost a half code rate, we could achieve super fast encoding and decoding speeds with the current implementation.  ... 
arXiv:1702.07409v3 fatcat:upqk2o3h3jhczh6od3y447jouu

The CORE Storage Primitive: Cross-Object Redundancy for Efficient Data Repair & Access in Erasure Coded Storage [article]

Kyumars Sheykh Esmaili, Lluis Pamies-Juarez, Anwitaman Datta
2013 arXiv   pre-print
Consequently, novel codes optimized to cope with distributed storage system nuances are vigorously being researched.  ...  Erasure codes are an integral part of many distributed storage systems aimed at Big Data, since they provide high fault-tolerance for low overheads.  ...  We encode then each vector x i ∈ P (cross-object encoding) with a (3,2) systematic code (a simple parity check code, or SPC), with generator matrix G g = [I 2 , 1 2 ]), where I 2 is the identity matrix  ... 
arXiv:1302.5192v4 fatcat:rwljkazcc5azfi7khqpsy7vc5i

A Survey on Network Codes for Distributed Storage

A G Dimakis, K Ramchandran, Yunnan Wu, Changho Suh
2011 Proceedings of the IEEE  
In distributed storage systems where reliability is maintained using erasure coding, network codes can be designed to meet specific requirements.  ...  ABSTRACT | Distributed storage systems often introduce redundancy to increase reliability.  ...  Fountain codes [8] and low-density parity-check (LDPC) codes [9] are recent code designs that offer approximate MDS properties and fast encoding-and-decoding complexity.  ... 
doi:10.1109/jproc.2010.2096170 fatcat:bpts3gxghbgyzmad6oextvlqim

A Randomly Expandable Method for Data Layout of RAID Storage Systems [article]

De-zhai Yuan, Xing-yi Peng, Ting Liu, Zhe Cui
2017 arXiv   pre-print
RBEC is a family of systematic code, in which the generator matrix consists of two submatrices with entries over GF(2), an identity matrix on the top, and another submatrix on the bottom.  ...  Compared with the existed approaches, the attractive advantages of our schemes include 1) they are completely implemented based on only simple eXclusive OR (XOR) operations and have systematic code property  ...  And when k out of the n encoded data blocks are identical to the k original data blocks, we call it as systematic code, which is distributed in n nodes of the storage system represented in Fig. 1 The  ... 
arXiv:1709.04708v1 fatcat:2xssi7yrfzbn3njsay5u7znpgi

A Randomly Expandable Method for Data Layout of RAID Storage Systems

Dezhai Yuan, Xingyi Peng, Ting Liu, Zhe Cui
2018 International Journal of Innovative Computing, Information and Control  
RBEC is a systematic code family in which the generator matrix consists of two submatrices with entries over GF(2), an identity matrix on the top, and another submatrix on the bottom.  ...  Compared with existing approaches, the attractive advantages of our scheme include the following: (1) it is completely implemented based on simple eXclusive OR (XOR) operations and has systematic code  ...  HACFS is a novel erasure-coded storage system that uses a fast code to optimize recovery performance and a compact code to reduce the storage overhead instead of using two different erasure codes.  ... 
doi:10.24507/ijicic.14.03.1079 fatcat:d4r4qr7uuzev3fh27kvrbceqae

FNT-based Reed-Solomon Erasure Codes [article]

Alexandre Soro, Jerome Lacan
2009 arXiv   pre-print
Thanks to FNT, these codes support practical coding and decoding algorithms with complexity O(n log n), where n is the number of symbols of a codeword.  ...  An open-source implementation shows that the encoding speed can reach 150Mbps for codes of length up to several 10,000s of symbols.  ...  The optimal erasure recovery capability of MDS codes is necessary in many applications like distributed storage (peer-to-peer distributed storage or RAID-based systems) or multimedia/multicast transmissions  ... 
arXiv:0907.1788v1 fatcat:2r5colc2mjdhvm6zs652x7g3xy

Erasure Codes for Increasing the Availability of Grid Data Storage

M. Pitkanen, R. Moussa, M. Swany, T. Niemi
2006 Advanced Int'l Conference on Telecommunications and Int'l Conference on Internet and Web Applications and Services (AICT-ICIW'06)  
Redundant data is computed using Reed-Solomon (RS) codes. The level of availability can be chosen for each individual file.  ...  Our prototype uses existing Grid data management tools (GridFTP) for communication, and the RS encoding and decoding is embedded in the client software.  ...  The codes they use have properties of the LDPC family of codes. In their solution the storage overhead is slightly larger than with RS codes, but very fast coding is achieved.  ... 
doi:10.1109/aict-iciw.2006.90 dblp:conf/aict/PitkanenMSN06 fatcat:6gwfg65ymzbsjkt2qs7hfeu25a

Homomorphic Self-repairing Codes for Agile Maintenance of Distributed Storage Systems [article]

Frederique Oggier, Anwitaman Datta
2011 arXiv   pre-print
Distributed data storage systems are essential to deal with the need to store massive volumes of data.  ...  We propose as an alternative a new family of erasure codes called self-repairing codes (SRC) taking into account the peculiarities of distributed storage systems, specifically the maintenance process.  ...  This overhead reduction is with respect to not only traditional erasure codes, but under certain regimes, also in comparison to other 'optimal' storage centric codes such as regenerating codes.  ... 
arXiv:1107.3129v1 fatcat:aamq3vanyjamriq3bhoxxgemni

On the Effectiveness of Polynomial Realization of Reed-Solomon Codes for Storage Systems [article]

Kyumars Sheykh Esmaili, Anwitaman Datta
2013 arXiv   pre-print
In this paper we evaluate the performance of an implementation of polynomial realization of Reed-Solomon codes, along with our optimized version of it, against that of a widely-used library (Jerasure)  ...  While in the storage community, using the generator matrices to implement RS codes is more popular, in the coding theory community the generator polynomials are typically used to realize RS codes.  ...  Introduction In the past few years, erasure codes, most prominently Reed Solomon (RS) codes, have been increasingly embraced by distributed storage systems -e.g., Facebook's HDFS-RAID [18] , Microsoft  ... 
arXiv:1312.5155v1 fatcat:532lhk3h7bcldiyh7lovyqj6di
« Previous Showing results 1 — 15 out of 20,390 results