A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
Filters
Optimal Systematic Distributed Storage Codes with Fast Encoding
[article]
2015
arXiv
pre-print
Preserved Fulltext
We also present general ways of transforming existing storage and repair optimal codes to enable fast encoding through sparsity. ...
Erasure codes are being increasingly used in distributed-storage systems in place of data-replication, since they provide the same level of reliability with much lower storage overhead. ...
Introduction Erasure codes are being increasingly used in distributed-storage systems instead of replication, since they provide the same level of reliability with much less storage overhead. ...
arXiv:1509.01858v1
fatcat:pqbtdvm54zafdcaqae3ce6l22a
Fast Product-Matrix Regenerating Codes
[article]
2014
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2019; you can also visit the original URL.
The file type is application/pdf.
Distributed storage systems support failures of individual devices by the use of replication or erasure correcting codes. ...
We implement the original Product-Matrix Regenerating codes as well as a new optimization we propose and show that the resulting optimized codes allow achieving 790 MB/s for encoding in typical settings ...
For systematic codes with k = 8, our optimized product-matrix codes encode at 790 MB/s when Reed-Solomon codes encode at 1640 MB/s. ...
arXiv:1412.3022v1
fatcat:k7g67mfnofbwpoiv3376ifpzhe
Performance evaluation of the Mojette erasure code for fault-tolerant distributed hot data storage
[article]
2015
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
Packet erasure codes are today a real alternative to replication in fault tolerant distributed storage systems. ...
Those very good performances allow to deploy Mojette erasure code for hot data distributed storage and I/O intensive applications. ...
Indeed, to encode a 8 KB block with a (12, 8) code, the encoder splits 8 KB into 8 packets of one kilobyte and produces 4 encoded packets for systematic codes, while non-systematic codes have to compute ...
arXiv:1504.07038v1
fatcat:j4ciypzydveazkggprs3cclnfe
RapidRAID: Pipelined Erasure Codes for Fast Data Archival in Distributed Storage Systems
[article]
2012
arXiv
pre-print
Preserved Fulltext
The Internet Archive has a preservation copy of this work in our general collections.
The file type is application/pdf.
Other Versions
We further present RapidRAID codes, an explicit family of pipelined erasure codes which provides fast archival without compromising either data reliability or storage overheads. ...
In contrast, this paper addresses the migration from replication to encoding: traditionally erasure coding is an atomic operation in that a single node with the whole object encodes and uploads all the ...
codes, aiming at optimizing the data archival in distributed storage systems. ...
arXiv:1207.6744v2
fatcat:wqc3rxi3djcbjo7psmzruknloq
Applied Erasure Coding in Networks and Distributed Storage
[article]
2018
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2019; you can also visit the original URL.
The file type is application/pdf.
implementation of new erasure codes for large-scale distributed storage systems that provide savings in the storage and network resources compared to state-of-the-art codes; and - Provision of a unified ...
Methods from both coding and information theory have been applied to network coding, Optical Packet Switching (OPS) networks and distributed storage systems. ...
Regenerating Codes Classical MDS codes are optimal in terms of the storage-reliability tradeoff, but they still do not give a good answer to the key question: How to encode the data in a distributed way ...
arXiv:1803.01358v1
fatcat:ooellwzlkzekxdhm75snjpeggu
Founsure 1.0: An erasure code library with efficient repair and update features
2021
SoftwareX
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2021; you can also visit the original URL.
The file type is application/pdf.
Its implementation utilizes compiler optimizations and multi-threading to generate the right assembly code for the given multi-core CPU architecture with vector processing capabilities. ...
Founsure library enables efficient encoding, decoding, repairs/rebuilds, and updates while all the required data storage and computations are distributed across the network nodes. ...
Unlike systematic codes in which the data is explicit at the encoder output, Founsure comes with the non-systematic format as mentioned before. ...
doi:10.1016/j.softx.2021.100662
fatcat:uxjrrrobprdpzat37ahpg2cyue
Founsure 1.0: An Erasure Code Library with Efficient Repair and Update Features
[article]
2021
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
Note that this fulltext copy is not of the "primary" version of this work. The version it corresponds to is:
2018 pre-print arXiv:1702.07409v2 fatcat:x6cwkl2o4ra6djly6yevsk7wuuOther Versions
Its implementation utilizes compiler optimizations and multi-threading to generate the right assembly code for the given multi-core CPU architecture with vector processing capabilities. ...
Founsure library enables efficient encoding, decoding, repairs/rebuilds, and updates while all the required data storage and computations are distributed across the network nodes. ...
As can be seen, with almost a half code rate, we could achieve super fast encoding and decoding speeds with the current implementation. ...
arXiv:1702.07409v3
fatcat:upqk2o3h3jhczh6od3y447jouu
The CORE Storage Primitive: Cross-Object Redundancy for Efficient Data Repair & Access in Erasure Coded Storage
[article]
2013
arXiv
pre-print
Preserved Fulltext
The Internet Archive has a preservation copy of this work in our general collections.
The file type is application/pdf.
Other Versions
Consequently, novel codes optimized to cope with distributed storage system nuances are vigorously being researched. ...
Erasure codes are an integral part of many distributed storage systems aimed at Big Data, since they provide high fault-tolerance for low overheads. ...
We encode then each vector x i ∈ P (cross-object encoding) with a (3,2) systematic code (a simple parity check code, or SPC), with generator matrix G g = [I 2 , 1 2 ]), where I 2 is the identity matrix ...
arXiv:1302.5192v4
fatcat:rwljkazcc5azfi7khqpsy7vc5i
A Survey on Network Codes for Distributed Storage
2011
Proceedings of the IEEE
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2017; you can also visit the original URL.
The file type is application/pdf.
In distributed storage systems where reliability is maintained using erasure coding, network codes can be designed to meet specific requirements. ...
ABSTRACT | Distributed storage systems often introduce redundancy to increase reliability. ...
Fountain codes [8] and low-density parity-check (LDPC) codes [9] are recent code designs that offer approximate MDS properties and fast encoding-and-decoding complexity. ...
doi:10.1109/jproc.2010.2096170
fatcat:bpts3gxghbgyzmad6oextvlqim
A Randomly Expandable Method for Data Layout of RAID Storage Systems
[article]
2017
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
RBEC is a family of systematic code, in which the generator matrix consists of two submatrices with entries over GF(2), an identity matrix on the top, and another submatrix on the bottom. ...
Compared with the existed approaches, the attractive advantages of our schemes include 1) they are completely implemented based on only simple eXclusive OR (XOR) operations and have systematic code property ...
And when k out of the n encoded data blocks are identical to the k original data blocks, we call it as systematic code, which is distributed in n nodes of the storage system represented in Fig. 1 The ...
arXiv:1709.04708v1
fatcat:2xssi7yrfzbn3njsay5u7znpgi
A Randomly Expandable Method for Data Layout of RAID Storage Systems
2018
International Journal of Innovative Computing, Information and Control
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
RBEC is a systematic code family in which the generator matrix consists of two submatrices with entries over GF(2), an identity matrix on the top, and another submatrix on the bottom. ...
Compared with existing approaches, the attractive advantages of our scheme include the following: (1) it is completely implemented based on simple eXclusive OR (XOR) operations and has systematic code ...
HACFS is a novel erasure-coded storage system that uses a fast code to optimize recovery performance and a compact code to reduce the storage overhead instead of using two different erasure codes. ...
doi:10.24507/ijicic.14.03.1079
fatcat:d4r4qr7uuzev3fh27kvrbceqae
FNT-based Reed-Solomon Erasure Codes
[article]
2009
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
Thanks to FNT, these codes support practical coding and decoding algorithms with complexity O(n log n), where n is the number of symbols of a codeword. ...
An open-source implementation shows that the encoding speed can reach 150Mbps for codes of length up to several 10,000s of symbols. ...
The optimal erasure recovery capability of MDS codes is necessary in many applications like distributed storage (peer-to-peer distributed storage or RAID-based systems) or multimedia/multicast transmissions ...
arXiv:0907.1788v1
fatcat:2r5colc2mjdhvm6zs652x7g3xy
Erasure Codes for Increasing the Availability of Grid Data Storage
2006
Advanced Int'l Conference on Telecommunications and Int'l Conference on Internet and Web Applications and Services (AICT-ICIW'06)
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2017; you can also visit the original URL.
The file type is application/pdf.
Redundant data is computed using Reed-Solomon (RS) codes. The level of availability can be chosen for each individual file. ...
Our prototype uses existing Grid data management tools (GridFTP) for communication, and the RS encoding and decoding is embedded in the client software. ...
The codes they use have properties of the LDPC family of codes. In their solution the storage overhead is slightly larger than with RS codes, but very fast coding is achieved. ...
doi:10.1109/aict-iciw.2006.90
dblp:conf/aict/PitkanenMSN06
fatcat:6gwfg65ymzbsjkt2qs7hfeu25a
Homomorphic Self-repairing Codes for Agile Maintenance of Distributed Storage Systems
[article]
2011
arXiv
pre-print
Preserved Fulltext
The Internet Archive has a preservation copy of this work in our general collections.
The file type is application/pdf.
Distributed data storage systems are essential to deal with the need to store massive volumes of data. ...
We propose as an alternative a new family of erasure codes called self-repairing codes (SRC) taking into account the peculiarities of distributed storage systems, specifically the maintenance process. ...
This overhead reduction is with respect to not only traditional erasure codes, but under certain regimes, also in comparison to other 'optimal' storage centric codes such as regenerating codes. ...
arXiv:1107.3129v1
fatcat:aamq3vanyjamriq3bhoxxgemni
On the Effectiveness of Polynomial Realization of Reed-Solomon Codes for Storage Systems
[article]
2013
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
In this paper we evaluate the performance of an implementation of polynomial realization of Reed-Solomon codes, along with our optimized version of it, against that of a widely-used library (Jerasure) ...
While in the storage community, using the generator matrices to implement RS codes is more popular, in the coding theory community the generator polynomials are typically used to realize RS codes. ...
Introduction In the past few years, erasure codes, most prominently Reed Solomon (RS) codes, have been increasingly embraced by distributed storage systems -e.g., Facebook's HDFS-RAID [18] , Microsoft ...
arXiv:1312.5155v1
fatcat:532lhk3h7bcldiyh7lovyqj6di
« Previous
Showing results 1 — 15 out of 20,390 results