Filters








887 Hits in 3.6 sec

An overview of the HDF5 technology suite and its applications

Mike Folk, Gerd Heber, Quincey Koziol, Elena Pourmal, Dana Robinson
2011 Proceedings of the EDBT/ICDT 2011 Workshop on Array Databases - AD '11  
In this paper, we give an overview of the HDF5 technology suite and some of its applications.  ...  We discuss the HDF5 data model, the HDF5 software architecture and some of its performance enhancing capabilities.  ...  HDF-OPeNDAP [52] (see Figure 11 ) implements an OPeNDAP server for HDF4 and HDF5 files featuring [54] :  CONCLUSION In this paper, we have given an overview of the HDF5 technology suite and some  ... 
doi:10.1145/1966895.1966900 dblp:conf/edbt/FolkHKPR11 fatcat:qiulz2yan5am7prae22fi5zm6m

H5hut: A high-performance I/O library for particle-based simulations

Mark Howison, Andreas Adelmann, E. Wes Bethel, Achim Gsell, Benedikt Oswald, Prabhat
2010 2010 IEEE International Conference On Cluster Computing Workshops and Posters (CLUSTER WORKSHOPS)  
We present the H5hut library, an implementation of several data models for particle-based simulations that encapsulates the complexity of HDF5 and is simple to use, yet does not compromise performance.  ...  Particle-based simulations running on large highperformance computing systems over many time steps can generate an enormous amount of particle-and field-based data for post-processing and analysis.  ...  It leverages an efficient bitmap indexing technology called "FastBit" [1] that is state-of-theart in the database community.  ... 
doi:10.1109/clusterwksp.2010.5613098 fatcat:zol4d2ltxzbqldooe3s25ew2xm

On the storage, management and analysis of (multi) similarity for large scale protein structure datasets in the grid

Gianluigi Folino, Azhar Ali Shah, Natalio Kransnogor
2009 2009 22nd IEEE International Symposium on Computer-Based Medical Systems  
In particular we investigate and compare the storage/query overhead of two commonly used database technologies such as the Hierarchical Data Format (HDF) (HDF5) and Relational Database Management System  ...  As the technologies explored under this investigation are quite generic in the science and engineering domain, our findings would also be beneficial for other scientific applications having related magnitude  ...  The development of such knowledge base is aimed at providing an easy to use interface, so the biologist can perform scientic queries using a vast variety of criteria and options leading to far better and  ... 
doi:10.1109/cbms.2009.5255328 dblp:conf/cbms/FolinoSK09 fatcat:on7lkqisffcgzotarzp5c6lqs4

Exploring Scientific Application Performance Using Large Scale Object Storage [chapter]

Steven Wei-der Chien, Stefano Markidis, Rami Karim, Erwin Laure, Sai Narasimhamurthy
2018 Lecture Notes in Computer Science  
While object storage is a promising technology, it is still unclear how scientific applications will use object storage and what the main performance benefits will be.  ...  The usage of parallel file systems and consistency requirements of POSIX, that all the traditional HPC parallel I/O interfaces adhere to, pose limitations to the scalability of scientific applications.  ...  Acknowledgments Funding for the work is received from the European Commission H2020 program, Grant Agreement No. 671500 (SAGE).  ... 
doi:10.1007/978-3-030-02465-9_8 fatcat:5cmbibuj3jafzcrbl75ug7pxbi

Experimental Directory Structure (Exdir): An Alternative to HDF5 Without Introducing a New File Format

Svenn-Arne Dragly, Milad Hobbi Mobarhan, Mikkel E. Lepperød, Simen Tennøe, Marianne Fyhn, Torkel Hafting, Anders Malthe-Sørenssen
2018 Frontiers in Neuroinformatics  
Exdir uses the same abstractions as HDF5 and is compatible with the HDF5 Abstract Data Model.  ...  Natural sciences generate an increasing amount of data in a wide range of formats developed by different research groups and commercial companies.  ...  ACKNOWLEDGMENTS The development of Exdir owes a great deal to other standardization efforts in science in general and neuroscience in particular, among them the contributors to HDF5, NumPy, YAML, PyYAML  ... 
doi:10.3389/fninf.2018.00016 pmid:29706879 pmcid:PMC5909058 fatcat:ryywgzh5pnc3ldg6cgiki2jdqu

Experimental Directory Structure (Exdir): An alternative to HDF5 without introducing a new file format [article]

Svenn-Arne Dragly, Milad Hobbi Mobarhan, Mikkel Lepperød, Simen Tennøe, Torkel Hafting, Marianne Fyhn, Anders Malthe-Sørenssen
2018 bioRxiv   pre-print
Exdir uses the same abstractions as HDF5 and is compatible with the HDF5 Abstract Data Model.  ...  Natural sciences generate an increasing amount of data in a wide range of formats developed by different research groups and commercial companies.  ...  ACKNOWLEDGMENTS The development of Exdir owes a great deal to other standardization efforts in science in general and neuroscience in particular, among them the contributors to HDF5, NumPy, YAML, PyYAML  ... 
doi:10.1101/249979 fatcat:iwwq5fefhfhhtdhtz3s54iz6n4

High-Content Digital Microscopy with Python [article]

Fabrice Salvaire
2014 arXiv   pre-print
A digital microscopy platform aims at capturing an image of a cover slip, at storing information on a file server and a database, at visualising the image and analysing its content.  ...  We will discuss how the Python ecosystem can provide such software framework efficiently. Moreover this paper will give an illustration of the data chunking approach to manage the huge amount of data.  ...  Finally we described our slide viewer architecture based on the OpenGL programmable pipeline and a texture patchwork rendering. We gave an overview on the vertex and the fragment shader.  ... 
arXiv:1404.6385v2 fatcat:kf3otfg36fgfpa6hyghe3vgr4i

Evaluating the Benefits of Key-Value Databases for Scientific Applications [chapter]

Pol Santamaria, Lena Oden, Eloy Gil, Yolanda Becerra, Raül Sirvent, Philipp Glock, Jordi Torres
2019 Lecture Notes in Computer Science  
The original code uses HDF5 files on GPFS through an intricate interface, imposing synchronizations.  ...  The convergence of Big Data applications with High -Performance Computing requires new methodologies to store, manage and process large amounts of information.  ...  In this section, we give a short overview of the original cell segmentation application.  ... 
doi:10.1007/978-3-030-22734-0_30 fatcat:vjlt4pqsona4njuxdob64fgg4a

Supporting XML Based High-Level Abstractions on HDF5 Datasets: A Case Study in Automatic Data Virtualization [chapter]

Swarup Kumar Sahoo, Gagan Agrawal
2005 Lecture Notes in Computer Science  
If the resulting value is true, the tuple is retained, otherwise, it is discarded. A return clause is used to create an XML record after processing one iteration of the for loop.  ...  We describe how our compiler can generate efficient code to access and process HDF5 datasets using the above information.  ...  This is because HDF5 reads an entire storage chunk, even if only a part of it is actually required.  ... 
doi:10.1007/11532378_22 fatcat:hp5ngp3onfewrnhnhcvpfnmoyi

mycelyso – high-throughput analysis of Streptomyces mycelium live cell imaging data

Christian Carsten Sachs, Joachim Koepff, Wolfgang Wiechert, Alexander Grünberger, Katharina Nöh
2019 BMC Bioinformatics  
Versatile key growth parameters such as mycelium network structure, its development over time, and tip growth rates are extracted.  ...  The batch-analysis mode facilitates the rapid and reproducible processing of large microfluidic screenings, and easy extraction of morphological parameters.  ...  Acknowledgements Not applicable. 1  ... 
doi:10.1186/s12859-019-3004-1 fatcat:4lvgiqsg7jesfo3gkut7xkmrgq

New Storage Layout for Earth System Data (D4.2)

Jakob Luettgau, Julian Kunkel, Bryan Lawrence, Jens Jensen, GIuseppe Congiu, Huang Hua, Paola Nassisi
2017 Zenodo  
The document isolates the key requirements for an earth system middleware and collects numerous use-case outlining the benefit to existing and anticipated workflows and technologies.  ...  The document is not intended to describe all components completely but provides a high-level overview that is necessary to build a first prototype as it is planned in the next phase of the ESiWACE project  ...  The HDF5 Technology suite includes tools and applications for managing, manipulating, viewing, and analysing data in the HDF5 format.  ... 
doi:10.5281/zenodo.2573896 fatcat:q67wragqkreijczychggxgkd6m

New Storage Layout For Earth System Data (D4.2)

Jakob Luettgau, Julian Kunkel, Bryan Lawrence, Jens Jensen, GIuseppe Congiu, Huang Hua, Paola Nassisi
2017 Zenodo  
The document isolates the key requirements for an earth system middleware and collects numerous use-case outlining the benefit to existing and anticipated workflows and technologies.  ...  The document is not intended to describe all components completely but provides a high-level overview that is necessary to build a first prototype as it is planned in the next phase of the ESiWACE project  ...  The HDF5 Technology suite includes tools and applications for managing, manipulating, viewing, and analyzing data in the HDF5 format.  ... 
doi:10.5281/zenodo.1228763 fatcat:e57r7wzxzzfc5nykde4yntk7ny

A Bit-Encoding Based New Data Structure for Time and Memory Efficient Handling of Spike Times in an Electrophysiological Setup

Bengt Ljungquist, Per Petersson, Anders J. Johansson, Jens Schouenborg, Martin Garwicz
2018 Neuroinformatics  
In addition to managing recordings from very large numbers of neurons in real time, it also has the capacity to handle the extensive periods of recording time necessary in certain scientific and clinical  ...  Furthermore, the bit-encoding proposed has the additional advantage of allowing an extremely fast analysis of spatiotemporal spike patterns in a large number of neurons.  ...  Acknowledgements The authors thank Stephan Gerhard for fruitful discussions about the HDF5 data format and Palmi Thor Thorbergsson for constructive feedback during the verification process of the software  ... 
doi:10.1007/s12021-018-9367-z pmid:29508123 pmcid:PMC5984964 fatcat:mvbqsmzfvreozbnniozrmm3g4a

Extreme-scaling applications en route to exascale

Dirk Brömmel, Wolfgang Frings, Brian J. N. Wylie
2016 Proceedings of the Exascale Applications and Software Conference 2016 on - EASC '16  
Poor performance of a NEST-import module which loaded and connected 1.9 TiB of neuron and synapse data was tracked down to an internal data-structure mismatch with the HDF5 file objects that prevented  ...  Despite more mixed results, we learnt more about application file I/O limitations and inefficiencies which continue to be the primary inhibitor to large-scale simulations.  ...  Acknowledgments We would like to thank the workshop participants from the eight code-teams for openly sharing their knowledge and experience: Mathis Bode, Guido Deissmann, Fabien Delalondre, Abhishek Deshmukh  ... 
doi:10.1145/2938615.2938616 dblp:conf/easc/BrommelFW16 fatcat:n5vv7ip4vrhw7e57bd3risfxwy

Requirements for storing electrophysiology data [article]

Jeff Teeters, Jan Benda, Andrew Davison, Stephen Eglen, Richard C. Gerkin, Jeffrey Grethe, Jan Grewe, Kenneth Harris, Christian Kellner, Yann Le Franc, Roman Mouček, Dimiter Prodanov (+7 others)
2016 arXiv   pre-print
While this document describes the requirements of the standard independent of the actual storage technology, the Task Force has recommended basing a standard on HDF5.  ...  The purpose of this document is to specify the basic data types required for storing electrophysiology and optical imaging data to facilitate computer-based neuroscience studies and data sharing.  ...  The Kwik [5] format used by the KlustaKwik spike sorting suite is an HDF5-based system This work was supported by the International Neuroinformatics Coordinating Facility (INCF) * Working group leader  ... 
arXiv:1605.07673v2 fatcat:lwvrrz45xnfijp7dcyj6yxc74u
« Previous Showing results 1 — 15 out of 887 results