Filters








4 Hits in 3.5 sec

Transitioning from file-based HPC workflows to streaming data pipelines with openPMD and ADIOS2 [article]

Franz Poeschel and Juncheng E and William F. Godoy and Norbert Podhorszki and Scott Klasky and Greg Eisenhauer and Philip E. Davis and Lipeng Wan and Ana Gainaru and Junmin Gu and Fabian Koller and René Widera and Michael Bussmann and Axel Huebl
<span title="2022-01-19">2022</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>
This paper aims to create a transition path from file-based IO to streaming-based workflows for scientific applications in an HPC environment.  ...  This paper surveys two openPMD-based loosely-coupled setups to demonstrate flexible applicability and to evaluate performance.  ...  Summary & Outlook The library openPMD-api has been extended to enable domain scientists a straight-forward transition from file-based to streaming IO.  ... 
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2107.06108v2">arXiv:2107.06108v2</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/mticejvez5atbfzepkax4zcpru">fatcat:mticejvez5atbfzepkax4zcpru</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20220125014349/https://arxiv.org/ftp/arxiv/papers/2107/2107.06108.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/a4/31/a4312de086d87e6beed308d1e355076f99842a20.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2107.06108v2" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>

Plenary: openPMD - A Scientific, Community Meta-Data Standard

Axel Huebl, Rémi Lehe, Jean-Luc Vay, David P. Grote, Ivo F. Sbalzarini, Stephan Kuschel, David Sagan, Christopher Mayes, Frédéric Pérez, Fabian Koller, Franz Poeschel, Carsten Fortmann-Grote (+4 others)
<span title="2022-01-26">2022</span> <i title="Zenodo"> Zenodo </i> &nbsp;
Developed in the open as a FAIR data standard, openPMD is based on portable and scalable file formats such as HDF5 and ADIOS2, among others.  ...  The open standard for particle-mesh data (openPMD) is a community meta-data standard. openPMD adds scientific self-description in a machine-actionable format to data sets, which allows sharing data processing  ...  "Transitioning from file-based HPC workflows to streaming data pipelines with openPMD and ADIOS2," accepted in SMC21, https://arxiv.org/abs/2107.06108 (2021) Right: For each arrow in the left figure, one  ... 
<span class="external-identifiers"> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.5281/zenodo.5905052">doi:10.5281/zenodo.5905052</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/blxunrfe5va4td2eb4wjj2fawa">fatcat:blxunrfe5va4td2eb4wjj2fawa</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20220129210059/https://zenodo.org/record/5905053/files/2022-02-26_openPMD_LPA-Control-Systems-LMU.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/6d/89/6d89daa14c6b1bdc0ac7757221ae284101cbb20d.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.5281/zenodo.5905052"> <button class="ui left aligned compact blue labeled icon button serp-button"> <i class="unlock alternate icon" style="background-color: #fb971f;"></i> zenodo.org </button> </a>

openPMD - A Scientific, Community Meta-Data Standard

Axel Huebl, Franz Poeschel
<span title="2021-11-04">2021</span> <i title="Zenodo"> Zenodo </i> &nbsp;
Developed in the open as a FAIR data standard, openPMD is based on portable and scalable file formats such as HDF5 and ADIOS2, among others.  ...  The open standard for particle-mesh data (openPMD) is a FAIR community standard. openPMD meta-data adds scientific self-description in a machine-actionable format to data sets, which allows sharing data  ...  The authors are thankful for the community contributions to libraries, software eco-system, user support, review and integrations.  ... 
<span class="external-identifiers"> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.5281/zenodo.5645623">doi:10.5281/zenodo.5645623</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/l7duiqhgt5eszjnd5rq4vbzkfm">fatcat:l7duiqhgt5eszjnd5rq4vbzkfm</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20211105160319/https://zenodo.org/record/5645624/files/2021-11-04_EUXfel-SAX-openPMD.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/b8/93/b89310255790ddf70a5299fe71edba3a028334be.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.5281/zenodo.5645623"> <button class="ui left aligned compact blue labeled icon button serp-button"> <i class="unlock alternate icon" style="background-color: #fb971f;"></i> zenodo.org </button> </a>

ADIOS 2: The Adaptable Input Output System. A framework for high-performance data management

William F. Godoy, Norbert Podhorszki, Ruonan Wang, Chuck Atkins, Greg Eisenhauer, Junmin Gu, Philip Davis, Jong Choi, Kai Germaschewski, Kevin Huck, Axel Huebl, Mark Kim (+17 others)
<span title="">2020</span> <i title="Elsevier BV"> <a target="_blank" rel="noopener" href="https://fatcat.wiki/container/xayqhkyjsncwvogxxusqncax6q" style="color: black;">SoftwareX</a> </i> &nbsp;
ADIOS 2 addresses scientific data management needs ranging from scalable I/O in supercomputers, to data analysis in personal computer and cloud systems.  ...  Version 2 introduces a unified application programming interface (API) that enables seamless data movement through files, wide-area-networks, and direct memory access, as well as high-level APIs for data  ...  Application Engine Application Areas File BPFile Checkpoint/restart, analysis data, zero-copy buffer, file-based streaming, code coupling HDF5 HDF5 compliant files Data staging SST Interprocess  ... 
<span class="external-identifiers"> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1016/j.softx.2020.100561">doi:10.1016/j.softx.2020.100561</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/6st23w3gmvaz3c7jz4okdz3l6y">fatcat:6st23w3gmvaz3c7jz4okdz3l6y</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20210717133108/https://escholarship.org/content/qt4zn3r3d6/qt4zn3r3d6.pdf?t=qrv5me" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/5e/50/5e50c102d355ce5d8605d50c97955cc36b114b7b.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1016/j.softx.2020.100561"> <button class="ui left aligned compact blue labeled icon button serp-button"> <i class="unlock alternate icon" style="background-color: #fb971f;"></i> elsevier.com </button> </a>