Filters








122 Hits in 0.58 sec

IO500 Ranked List ISC19

John Bent, Julian Kunkel, Jay Lofstead, George Markomanolis
<span title="2019-06-18">2019</span> <i title="Zenodo"> Zenodo </i> &nbsp;
The IO500 Benchmark publishes twice annual ranked lists of all submissions. This list was first published at the 2019 iSC conference.
<span class="external-identifiers"> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.5281/zenodo.3354660">doi:10.5281/zenodo.3354660</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/zevmff2tdzfshdpixmxvqtu2vy">fatcat:zevmff2tdzfshdpixmxvqtu2vy</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20220202030552/https://zenodo.org/record/3354660/files/IO-500_ISC19.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/4d/27/4d279c7db6942995a3c9a9b7de0705119e871175.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.5281/zenodo.3354660"> <button class="ui left aligned compact blue labeled icon button serp-button"> <i class="unlock alternate icon" style="background-color: #fb971f;"></i> zenodo.org </button> </a>

IO500 Full Ranked List, Supercomputing 2018 (Corrected) [article]

John Bent, Julian Kunkel, Jay Lofstead, George Markomanolis
<span title="2018-11-14">2018</span> <i title="Zenodo"> Zenodo </i> &nbsp;
The IO500 Benchmark publishes twice annual ranked lists of all submissions. This list was first published at the 2018 SC conference but has been corrected to fix a reporting bug in some of the metadata scores.
<span class="external-identifiers"> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.5281/zenodo.2602025">doi:10.5281/zenodo.2602025</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/m2vtgo3bcnbytcc543uh6fycfa">fatcat:m2vtgo3bcnbytcc543uh6fycfa</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20220219231806/https://zenodo.org/record/2602025/files/sc18_2.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/ca/03/ca036583b14df327aa269b909b30bf282e443016.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.5281/zenodo.2602025"> <button class="ui left aligned compact blue labeled icon button serp-button"> <i class="unlock alternate icon" style="background-color: #fb971f;"></i> zenodo.org </button> </a>

IO500 Full Ranked List, Supercomputing 2018 (Corrected) [article]

John Bent, Julian Kunkel, Jay Lofstead, George Markomanolis
<span title="2018-11-14">2018</span> <i title="Zenodo"> Zenodo </i> &nbsp;
The IO500 Benchmark publishes twice annual ranked lists of all submissions. This list was first published at the 2018 SC conference but has been corrected to fix a reporting bug in some of the metadata scores.
<span class="external-identifiers"> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.5281/zenodo.2601990">doi:10.5281/zenodo.2601990</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/ieej5luwuraldfmnqrgk7ij6me">fatcat:ieej5luwuraldfmnqrgk7ij6me</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20220204170629/https://zenodo.org/record/2601990/files/sc18.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/8c/52/8c52e7a88087a4e0b705d72348a1626ce7918575.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.5281/zenodo.2601990"> <button class="ui left aligned compact blue labeled icon button serp-button"> <i class="unlock alternate icon" style="background-color: #fb971f;"></i> zenodo.org </button> </a>

Building Containerized Environments for Reproducibility and Traceability of Scientific Workflows [article]

Paula Olaya, Jay Lofstead, Michela Taufer
<span title="2020-09-17">2020</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>
Scientists rely on simulations to study natural phenomena. Trusting the simulation results is vital to develop sciences in any field. One approach to build trust is to ensure the reproducibility and traceability of the simulations through the annotation of executions at the system-level; by the generation of record trails of data moving through the simulation workflow. In this work, we present a system-level solution that leverages the intrinsic characteristics of containers (i.e., portability,
more &raquo; ... isolation, encapsulation, and unique identifiers). Our solution consists of a containerized environment capable to annotate workflows, capture provenance metadata, and build record trails. We assess our environment on four different workflows and measure containerization costs in terms of time and space. Our solution, built with a tolerable time and space overhead, enables transparent and automatic provenance metadata collection and access, an easy-to-read record trail, and tight connections between data and metadata.
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2009.08495v1">arXiv:2009.08495v1</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/7byxzxywmncivkph4ngdtcglsu">fatcat:7byxzxywmncivkph4ngdtcglsu</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20200922001012/https://arxiv.org/pdf/2009.08495v1.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/4f/ff/4fff17164beb45f26f24ad6f3e75edca0cb1f4f2.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2009.08495v1" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>

Containerization of Workflows: The SOMOSPIE use case [article]

Paula Olaya, Leobardo Valera, Michela Taufer, Ricardo Llamas, Rodrigo Vargas, Jay Lofstead
<span title="2021-03-05">2021</span> <i title="figshare"> figshare.com </i> &nbsp;
Use-case of road map towards reproducibility of scientific workflows through leveraging containers. We present an automatic fine-grained metadata that provides a full description of the dataflow, ensuring traceability of data and explainability of results. We question the trade-off between scalability and reproducibility.
<span class="external-identifiers"> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.6084/m9.figshare.14143679.v2">doi:10.6084/m9.figshare.14143679.v2</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/no5ivyl4ingobeikpi775dlgya">fatcat:no5ivyl4ingobeikpi775dlgya</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20210306013741/https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/26706875/RobustScience_containers.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/7d/11/7d11423833f3490225bc1bffeefe9d92fd674e65.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.6084/m9.figshare.14143679.v2"> <button class="ui left aligned compact blue labeled icon button serp-button"> <i class="unlock alternate icon" style="background-color: #fb971f;"></i> figshare.com </button> </a>

Data Pallets: Containerizing Storage For Reproducibility and Traceability [article]

Jay Lofstead, Joshua Baker, Andrew Younge
<span title="2018-11-07">2018</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>
Trusting simulation output is crucial for Sandia's mission objectives. We rely on these simulations to perform our high-consequence mission tasks given national treaty obligations. Other science and modeling applications, while they may have high-consequence results, still require the strongest levels of trust to enable using the result as the foundation for both practical applications and future research. To this end, the computing community has developed workflow and provenance systems to aid
more &raquo; ... in both automating simulation and modeling execution as well as determining exactly how was some output was created so that conclusions can be drawn from the data. Current approaches for workflows and provenance systems are all at the user level and have little to no system level support making them fragile, difficult to use, and incomplete solutions. The introduction of container technology is a first step towards encapsulating and tracking artifacts used in creating data and resulting insights, but their current implementation is focused solely on making it easy to deploy an application in an isolated "sandbox" and maintaining a strictly read-only mode to avoid any potential changes to the application. All storage activities are still using the system-level shared storage. This project explores extending the container concept to include storage as a new container type we call data pallets. Data Pallets are potentially writeable, auto generated by the system based on IO activities, and usable as a way to link the contained data back to the application and input deck used to create it.
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1811.04740v1">arXiv:1811.04740v1</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/vfch62i6dvbs5jjxz7yzbvwfau">fatcat:vfch62i6dvbs5jjxz7yzbvwfau</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20200909111221/https://arxiv.org/pdf/1811.04740v1.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/63/b2/63b2107b22cb24c2f3cd626c92adfaf1327e026c.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1811.04740v1" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>

Containerization of Workflows: The SOMOSPIE use case [article]

Paula Olaya, Leobardo Valera, Michela Taufer, Ricardo Llamas, Rodrigo Vargas, Jay Lofstead
<span title="2021-03-03">2021</span> <i title="figshare"> figshare.com </i> &nbsp;
Use-case of road map towards reproducibility of scientific workflows through leveraging containers. We present an automatic fine-grained metadata that provides a full description of the dataflow, ensuring traceability of data and explainability of results. We question the trade-off between scalability and reproducibility.
<span class="external-identifiers"> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.6084/m9.figshare.14143679.v1">doi:10.6084/m9.figshare.14143679.v1</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/ybay2pswwneznizruxri47ht74">fatcat:ybay2pswwneznizruxri47ht74</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20210310061046/https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/26668931/RobustScience_containers.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/07/32/073216b2bb461a04402a3ff2dc795b1880796deb.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.6084/m9.figshare.14143679.v1"> <button class="ui left aligned compact blue labeled icon button serp-button"> <i class="unlock alternate icon" style="background-color: #fb971f;"></i> figshare.com </button> </a>

Distributed Versioned Object Storage -- Alternatives at the OSD layer (Poster Extended Abstract) [article]

Ivo Jimenez, Carlos Maltzahn, Jay Lofstead
<span title="2014-06-14">2014</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>
The ability to store multiple versions of a data item is a powerful primitive that has had a wide variety of uses: relational databases, transactional memory, version control systems, to name a few. However, each implementation uses a very particular form of versioning that is customized to the domain in question and hidden away from the user. In our going project, we are reviewing and analyzing multiple uses of versioning in distinct domains, with the goal of identifying the basic components
more &raquo; ... quired to provide a generic distributed multiversioning object storage service, and define how these can be customized in order to serve distinct needs. With this primitive, new services can leverage multiversioning to ease development and provide specific consistency guarantees that address particular use cases. This work presents early results that quantify the trade-offs in implementing versioning at the local storage layer.
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1406.3699v1">arXiv:1406.3699v1</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/bvzdmep76vhjlezk7qzncmcq3m">fatcat:bvzdmep76vhjlezk7qzncmcq3m</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20191013220050/https://arxiv.org/pdf/1406.3699v1.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/bf/ab/bfabafae5b676e38393376e03cbba850395aca60.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1406.3699v1" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>

Extending MPI to Better Support Multi-application Interaction [chapter]

Jay Lofstead, Jai Dayal
<span title="">2012</span> <i title="Springer Berlin Heidelberg"> <a target="_blank" rel="noopener" href="https://fatcat.wiki/container/2w3awgokqne6te4nvlofavy5a4" style="color: black;">Lecture Notes in Computer Science</a> </i> &nbsp;
Current scientific workflows consist of generally several components either integrated in situ or as completely independent, asynchronous components using centralized storage as an interface. Neither of these approaches are likely to scale well into Exascale. Instead, separate applications and services will be launched using online communication to link these components of the scientific discovery process. Our experiences with coupling multiple, independent MPI applications, each with separate
more &raquo; ... rocessing phases, exposes limitations preventing use of some of the optimized mechanisms within the MPI standard. In this regard, we have identified two shortcomings with current MPI implementations. First, MPI intercommunicators offer a mechanism to communicate across application boundaries, but do not address the impact this operating mode has on possible programming models for each separate application. Second, MPI Probe offers a way to interleave both local messaging and remote messages, but has limitations as MPI Bcast and other collective calls are not supported by MPI Probe thus limiting use of optimize collective calls in this operating mode.
<span class="external-identifiers"> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1007/978-3-642-33518-1_32">doi:10.1007/978-3-642-33518-1_32</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/vfgs523kenf27htorm635jr4z4">fatcat:vfgs523kenf27htorm635jr4z4</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20160423015906/http://lofstead.org/papers/2012-imudi.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/0e/50/0e5058b6c53ba41e00c4d3b0b8ed43920fcd53d4.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1007/978-3-642-33518-1_32"> <button class="ui left aligned compact blue labeled icon button serp-button"> <i class="external alternate icon"></i> springer.com </button> </a>

Exploring Spatial Indexing for Accelerated Feature Retrieval in HPC [article]

Margaret Lawson, William Gropp, Jay Lofstead
<span title="2021-08-18">2021</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>
Despite the critical role that range queries play in analysis and visualization for HPC applications, there has been no comprehensive analysis of indices that are designed to accelerate range queries and the extent to which they are viable in an HPC setting. In this state of the practice paper we present the first such evaluation, examining 20 open-source C and C++ libraries that support range queries. Contributions of this paper include answering the following questions: which of the
more &raquo; ... tions are viable in an HPC setting, how do these libraries compare in terms of build time, query time, memory usage, and scalability, what are other trade-offs between these implementations, is there a single overall best solution, and when does a brute force solution offer the best performance? We also share key insights learned during this process that can assist both HPC application scientists and spatial index developers.
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2106.13972v2">arXiv:2106.13972v2</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/sqzad2jnqvg6xnaxqpd7lzgqiu">fatcat:sqzad2jnqvg6xnaxqpd7lzgqiu</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20210825031204/https://arxiv.org/pdf/2106.13972v2.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/87/38/87389414bbf65351d52e5a8f516c60213ba35294.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2106.13972v2" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>

Fine-grained composition of containerized scientific workflows for data traceability and results explainability [article]

Paula Olaya, Leobardo Valera, Ricardo Llamas, Rodrigo Vargas, Jay Lofstead
<span title="2021-04-26">2021</span> <i title="figshare"> figshare.com </i> &nbsp;
Use-case of road map towards trustworthiness of scientific workflows through leveraging containers and annotating the execution of every data component. We present an automatic fine-grained metadata that provides a full description of the dataflow, ensuring traceability of data and explainability of results. We compare the granularity level of containerization with the information captured in the metadata, concluding that in order to be able to trust in the results, fine-grained containerization is needed.
<span class="external-identifiers"> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.6084/m9.figshare.14485164.v1">doi:10.6084/m9.figshare.14485164.v1</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/gmw4tiveejbxheap5urz5qjfua">fatcat:gmw4tiveejbxheap5urz5qjfua</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20210427031426/https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/27738012/OLAYAUSECASEannotations.pptx.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/6d/c5/6dc5ea23d90bfe91682b6e3ab3ac686ca3e5de63.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.6084/m9.figshare.14485164.v1"> <button class="ui left aligned compact blue labeled icon button serp-button"> <i class="unlock alternate icon" style="background-color: #fb971f;"></i> figshare.com </button> </a>

Birds of a Feather session on Practical Reproducibility by Managing Experiments Like Software

Ivo Jimenez, Carlos Maltzahn, Jay Lofstead, Michael Heroux, Kate Keahey
<span title="2017-11-12">2017</span> <i title="Zenodo"> Zenodo </i> &nbsp;
This BoF is for HPC community members who want to make sharing and re-executing scientific experiments more practical. As a starting point, we will introduce the Popper Protocol (http://falsifiable.us), a set of guidelines for managing scientific explorations as software projects. We will moderate discussions, and conduct a survey among attendants on tools/services that can be used to "Popperize" scientific explorations, i.e. fully automate/document experiments by scripting all aspects of their
more &raquo; ... execution/analysis and version-control their evolution using DevOps tools. Our goal: create a list of tool-chain templates that domain-specific communities can use to incentivize researchers/students to generate easily re-executable/shareable experiments.
<span class="external-identifiers"> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.5281/zenodo.1495244">doi:10.5281/zenodo.1495244</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/ystdwsqiivh5hcoiaut4ivrcsa">fatcat:ystdwsqiivh5hcoiaut4ivrcsa</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20200113095432/https://zenodo.org/record/1495244/files/sc17_bof.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/7d/29/7d297694ea3fc14bed02e5d24f25c55f878a01fd.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.5281/zenodo.1495244"> <button class="ui left aligned compact blue labeled icon button serp-button"> <i class="unlock alternate icon" style="background-color: #fb971f;"></i> zenodo.org </button> </a>

Input/output APIs and data organization for high performance scientific computing

Jay Lofstead, Fang Zheng, Scott Klasky, Karsten Schwan
<span title="">2008</span> <i title="IEEE"> 2008 3rd Petascale Data Storage Workshop </i> &nbsp;
Scientific Data Management has become essential to the productivity of scientists using ever larger machines and running applications that produce ever more data. There are several specific issues when running on petascale (and beyond) machines. One is the need for massively parallel data output, which in part, depends on the data formats and semantics being used. Here, the inhibition of parallelism by file system notions of strict and immediate consistency can be addressed with 'delayed data
more &raquo; ... nsistency' methods. Such methods can also be used to remove the runtime coordination steps required for immediate consistency from machine resources like Bluegene's separate networks for barrier calls and its dedicated IO nodes, thereby freeing them to instead, perform alternate tasks that enhance data output performance and/or richness. Second, once data is generated, it is important to be able to efficiently access it, which implies the need for rapid data characterization and indexing. This can be achieved by adding small amounts of metadata to the output process, thereby permitting scientists to quickly make informed decisions about which files to process from large-scale science runs. Third, failure probabilities increase with an increasing number of nodes, which suggests the need for organizing output data to be resilient to failures in which the output from a single or from a small number of nodes is lost or corrupted. This paper demonstrates the utility of using delayed consistency methods for the process of data output from the compute nodes of petascale machines. It also demonstrates the advantages derived from resilient data organization coupled with lightweight methods for data indexing. An implementation of these techniques is realized in ADIOS, the Adaptable IO System, and its BP intermediate file format. The implementation is designed to be compatible with existing, well-known file formats like HDF-5 and NetCDF, thereby permitting end users to exploit the rich tool chains for these formats. Initial performance evaluations of the approach exhibit substantial performance advantages over using native parallel HDF-5 in the Chimera supernova code.
<span class="external-identifiers"> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1109/pdsw.2008.4811881">doi:10.1109/pdsw.2008.4811881</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/2looqfdqn5goxhzx4xbjr6gboq">fatcat:2looqfdqn5goxhzx4xbjr6gboq</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20160423023625/http://lofstead.org/papers/2008-pdsw.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/c7/64/c7640c8400254c01f48c3b1d733846ff12d8f47b.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1109/pdsw.2008.4811881"> <button class="ui left aligned compact blue labeled icon button serp-button"> <i class="external alternate icon"></i> ieee.com </button> </a>

D2T: Doubly Distributed Transactions for High Performance and Distributed Computing

Jay Lofstead, Jai Dayal, Karsten Schwan, Ron Oldfield
<span title="">2012</span> <i title="IEEE"> <a target="_blank" rel="noopener" href="https://fatcat.wiki/container/2ade5dt4mjgqzod5n6qf33litm" style="color: black;">2012 IEEE International Conference on Cluster Computing</a> </i> &nbsp;
The Doubly Distributed Transaction (D2T) protocol offers a mechanism for a collection of clients and a separate collection of servers to orchestrate an action with semantics inspired by database ACID-transactions. Our previous work showed good potential, but suffered from known limitations due to the client and server side split coordination. The initial performance of this approach was acceptable and showed potential for good scalability. However, the communication bottleneck at scale between
more &raquo; ... he client side coordination and the server side coordination would be unworkable. In this poster, we address these limitations and introduce three new technologies: 1) A client-side coordinator only optimization; 2) Data storage requirements and evaluation for a example transaction aware system; and 3) a metadata system requirements and evaluation for supporting transaction control over entries. Additionally, we were able to find many redundant or unneeded messages in the first version of the protocol that we removed without reducing the offered resilience. Experimental results show that with our refinements, we can attain the same level of transactional guarantees with a measurable decrease in overhead costs.
<span class="external-identifiers"> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1109/cluster.2012.79">doi:10.1109/cluster.2012.79</a> <a target="_blank" rel="external noopener" href="https://dblp.org/rec/conf/cluster/LofsteadDSO12.html">dblp:conf/cluster/LofsteadDSO12</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/vfi66xcw2bekvkifeu6kw66iz4">fatcat:vfi66xcw2bekvkifeu6kw66iz4</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20160422193456/http://lofstead.org/papers/2013-hpdc-poster-abstract.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/d3/34/d3340b66aa16acbc7845e50179ec6e8ebd2ce533.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1109/cluster.2012.79"> <button class="ui left aligned compact blue labeled icon button serp-button"> <i class="external alternate icon"></i> ieee.com </button> </a>

Efficient transactions for parallel data movement

Jay Lofstead, Jai Dayal, Ivo Jimenez, Carlos Maltzahn
<span title="">2013</span> <i title="ACM Press"> <a target="_blank" rel="noopener" href="https://fatcat.wiki/container/zigbcra6rjdivda6lkzknwuo5q" style="color: black;">Proceedings of the 8th Parallel Data Storage Workshop on - PDSW &#39;13</a> </i> &nbsp;
<span class="external-identifiers"> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1145/2538542.2538567">doi:10.1145/2538542.2538567</a> <a target="_blank" rel="external noopener" href="https://dblp.org/rec/conf/sc/LofsteadDJM13.html">dblp:conf/sc/LofsteadDJM13</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/uln3gdy4uveolebqpgtmpxe6x4">fatcat:uln3gdy4uveolebqpgtmpxe6x4</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20170830050029/http://www.pdsw.org/pdsw13/slides/lofstead-pdsw13.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/85/16/8516854b403110bcb5888604886a461c589906d1.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1145/2538542.2538567"> <button class="ui left aligned compact blue labeled icon button serp-button"> <i class="external alternate icon"></i> acm.org </button> </a>
&laquo; Previous Showing results 1 &mdash; 15 out of 122 results