Filters








1,586 Hits in 6.7 sec

Leveraging Naturally Distributed Data Redundancy to Reduce Collective I/O Replication Overhead

Bogdan Nicolae
2015 2015 IEEE International Parallel and Distributed Processing Symposium  
Leveraging naturally distributed data redundancy to reduce collective I/O replication overhead Bogdan Nicolae IBM Research, Ireland bogdan.nicolae@ie.ibm.com Abstract-Dumping large amounts of related data  ...  Index Terms-data resilience; high availability; data replication; deduplication; collective I/O scalability; redundancy management I.  ... 
doi:10.1109/ipdps.2015.82 dblp:conf/ipps/Nicolae15 fatcat:7pjkibnybbdh3gmoos6npwrbvm

Providing Efficient I/O Redundancy in MPI Environments [chapter]

Willam D. Gropp, Robert Ross, Neill Miller
2004 Lecture Notes in Computer Science  
This approach exploits coarser-grain (but precise) semantics to reduce the overhead of constructing recovery data and makes use of parallel computation (proportional to the data size and independent of  ...  number of processors) to construct data.  ...  An additional optimization would leverage accumulate operations during collective writes to avoid the read I/O step of recovery data calculation.  ... 
doi:10.1007/978-3-540-30218-6_17 fatcat:35w6zg6bdfadzabtpuslumd5py

I/O Deduplication

Ricardo Koller, Raju Rangaswami
2010 ACM Transactions on Storage  
We introduce I/O Deduplication, a storage optimization that utilizes content similarity for improving I/O performance by eliminating I/O operations and reducing the mechanical delays during I/O operations  ...  Evaluation of a prototype implementation using these workloads showed an overall improvement in disk I/O performance of 28 to 47% across these workloads.  ...  Here, we refer to intrinsic (or application/user-generated) data replication as opposed to forced (system-generated) redundancy such as in a RAID-1 storage system.  ... 
doi:10.1145/1837915.1837921 fatcat:snyfxksbyzdgzorutruv2ox3p4

Modeling the aging process of flash storage by leveraging semantic I/O

Yuhui Deng, Lijuan Lu, Qiang Zou, Shuqiang Huang, Jipeng Zhou
2014 Future generations computer systems  
This paper proposes to model the aging process of flash memory based storage systems constructed as a Redundant Array of Independent Disks (RAID) by leveraging the semantic I/O.  ...  Many research efforts have been invested in employing flash memory to build high performance and large-scale storage systems for data-intensive applications.  ...  This work is supported by the National Natural Science ---------------------------------------------------------------------------------------  ... 
doi:10.1016/j.future.2013.09.002 fatcat:l6lidnlbibettjv77ryr6zxdbm

Problems in Modern High Performance Parallel I/O Systems [article]

Robert Louis Cloud
2011 arXiv   pre-print
These three topics, the I/O subsystem, the nature of scientific data output, and future possible optimizations are discussed in this report.  ...  to be applicable as computers continue to scale.  ...  than collective I/O  ... 
arXiv:1109.0742v1 fatcat:vknt7cuqpjfnpmxmpk4lt4rhv4

Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks

Qing Liu, Jeremy Logan, Yuan Tian, Hasan Abbasi, Norbert Podhorszki, Jong Youl Choi, Scott Klasky, Roselyne Tchoua, Jay Lofstead, Ron Oldfield, Manish Parashar, Nagiza Samatova (+5 others)
2013 Concurrency and Computation  
This usually involves two rounds of MPI collective calls among the processors that participates in I/O, that is, MPI_Gather to collect local sizes and an MPI_Scatter to distribute the offsets to individual  ...  Focusing on putting users first with a service oriented architecture, we combined cutting edge research into new I/O techniques with a design effort to create near optimal I/O methods.  ...  Their shared experiences with parallel I/O helped  ... 
doi:10.1002/cpe.3125 fatcat:iieybtpgojdedlmlaes26argzu

A Checkpoint of Research on Parallel I/O for High-Performance Computing

Francieli Zanon Boito, Eduardo C. Inacio, Jean Luca Bez, Philippe O. A. Navaux, Mario A. R. Dantas, Yves Denneulin
2018 ACM Computing Surveys  
pattern extraction techniques, and performance modeling, in addition to general aspects of parallel I/O research.  ...  As the available processing power and amount of data increase, I/O remains a central issue for the scientific community.  ...  [105] also propose a new approach for collective I/O. They break collective I/O calls into multiple iterations to fit the buffer size.  ... 
doi:10.1145/3152891 fatcat:uodqvl422rbctd5ws56o4fzkmy

Toward high-performance key-value stores through GPU encoding and locality-aware encoding

Dongfang Zhao, Ke Wang, Kan Qiao, Tonglin Li, Iman Sadooghi, Ioan Raicu
2016 Journal of Parallel and Distributed Computing  
availability that causes high I/O overhead.  ...  The parity code is usually much smaller than the original file, which saves the extra space required for high availability and reduces the I/O overhead.  ...  This is counter-intuitive to some extent, as it is a well-accepted practice to collect data on a small subset of nodes (e.g., collective I/O can batch small and dispersed I/Os to reduce the number of I  ... 
doi:10.1016/j.jpdc.2016.04.015 fatcat:qg5u7jatfzcm3hhsb45phj5juq

Storage codes: Managing big data with small overheads

Anwitaman Datta, Frederique Oggier
2013 2013 International Symposium on Network Coding (NetCod)  
Erasure coding provides a mechanism to store data redundantly for fault-tolerance in a cost-effective manner.  ...  Very often, these novel techniques exploit the computational resources available 'in the network', i.e., leverage on storage units which are not passive entities supporting only read/write of data, but  ...  Depending on how actively the data is being read, the degree of replication is reduced, as erasure coded redundancy is introduced in the system to achieve comparable fault-tolerance but as significantly  ... 
doi:10.1109/netcod.2013.6570839 dblp:conf/netcod/DattaO13 fatcat:nq3tfok4rzgpxaosjq6e5gpa64

On exploiting data locality for iterative mapreduce applications in hybrid clouds

Francisco J. Clemente-Castelló, Bogdan Nicolae, Rafael Mayo, Juan Carlos Fernández, M. Mustafa Rafique
2016 Proceedings of the 3rd IEEE/ACM International Conference on Big Data Computing, Applications and Technologies - BDCAT '16  
In this context, we study how data locality can be leveraged over the weak link both from the storage layer perspective (when and how to move it off-premise) and from the scheduling perspective (when to  ...  complex computations frequently leads to insufficient local data center capacity.  ...  and off-premise; system-level statistics such as CPU, I/O network, I/O disk and memory utilization.  ... 
doi:10.1145/3006299.3006329 dblp:conf/bdc/Clemente-Castello16 fatcat:3xlkkjny6zajtk54p5j2rqfsbu

A case for tracking and exploiting inter-node and intra-node memory content sharing in virtualized large-scale parallel systems

Lei Xia, Peter A. Dinda
2012 Proceedings of the 6th international workshop on Virtualization Technologies in Distributed Computing Date - VTDC '12  
Finally, a service that improves reliability by introducing memory redundancy could leverage existing content sharing to minimize the memory costs of any particular level of redundancy.  ...  We then describe the high level approach we are taking to design and implement a distributed, VMM-based system that can efficiently and scalably identify and track such sharing with low overhead.  ...  Second, it would reduce the total time and the I/O bandwidth needed to transfer and store the checkpoints.  ... 
doi:10.1145/2287056.2287061 dblp:conf/hpdc/XiaD12 fatcat:r2piar42dvc2jm23i2nnk32tdm

PLR: A Software Approach to Transient Fault Tolerance for Multicore Architectures

A. Shye, J. Blomstedt, T. Moseley, V.J. Reddi, D.A. Connors
2009 IEEE Transactions on Dependable and Secure Computing  
This paper presents process-level redundancy (PLR), a software technique for transient fault tolerance, which leverages multiple cores for low overhead.  ...  Redundancy at the process level allows the operating system to freely schedule the processes across all available hardware resources.  ...  The authors would like to thank the anonymous IEEE TDSC reviewers and the anonymous reviewers at DSN for their comments on the preliminary work presented in this paper.  ... 
doi:10.1109/tdsc.2008.62 fatcat:4yne7yxic5dynho7rd4fcxpr4a

The Little Engine(s) That Could: Scaling Online Social Networks

Josep M. Pujol, Vijay Erramilli, Georgos Siganos, Xiaoyuan Yang, Nikolaos Laoutaris, Parminder Chhabra, Pablo Rodriguez
2012 IEEE/ACM Transactions on Networking  
We design, implement, and evaluate SPAR, a social partitioning and replication middle-ware that transparently leverages the social graph structure to achieve data locality while minimizing replication.  ...  ., develop as they would for a single server; scalability is achieved by adding commodity servers with low memory and network I/O requirements; and redundancy is achieved at a fraction of the cost.  ...  Special thanks to Evan Weaver, Zografoula Vagena and Ravi Sundaram for their early comments and feedback.  ... 
doi:10.1109/tnet.2012.2188815 fatcat:ynoplcpiynh77mxwajd3zvmqj4

The little engine(s) that could

Josep M. Pujol, Vijay Erramilli, Georgos Siganos, Xiaoyuan Yang, Nikos Laoutaris, Parminder Chhabra, Pablo Rodriguez
2010 Proceedings of the ACM SIGCOMM 2010 conference on SIGCOMM - SIGCOMM '10  
We design, implement, and evaluate SPAR, a social partitioning and replication middle-ware that transparently leverages the social graph structure to achieve data locality while minimizing replication.  ...  ., develop as they would for a single server; scalability is achieved by adding commodity servers with low memory and network I/O requirements; and redundancy is achieved at a fraction of the cost.  ...  Special thanks to Evan Weaver, Zografoula Vagena and Ravi Sundaram for their early comments and feedback.  ... 
doi:10.1145/1851182.1851227 dblp:conf/sigcomm/PujolESYLCR10 fatcat:rdq5ptcbajdyrjxcjaxuvuqarq

The little engine(s) that could

Josep M. Pujol, Vijay Erramilli, Georgos Siganos, Xiaoyuan Yang, Nikos Laoutaris, Parminder Chhabra, Pablo Rodriguez
2010 Computer communication review  
We design, implement, and evaluate SPAR, a social partitioning and replication middle-ware that transparently leverages the social graph structure to achieve data locality while minimizing replication.  ...  ., develop as they would for a single server; scalability is achieved by adding commodity servers with low memory and network I/O requirements; and redundancy is achieved at a fraction of the cost.  ...  Special thanks to Evan Weaver, Zografoula Vagena and Ravi Sundaram for their early comments and feedback.  ... 
doi:10.1145/1851275.1851227 fatcat:63x4c5anazbuthdgcazyjqpmxq
« Previous Showing results 1 — 15 out of 1,586 results