Filters








47 Hits in 0.51 sec

dispel4py

Amrey Krause, Rosa Filgueira, Malcolm Atkinson
2015 Proceedings of the 5th Workshop on Python for High-Performance and Scientific Computing - PyHPC '15  
We will show the dispel4py advantages by walking through an example.  ...  We present dispel4py, a novel data intensive and high performance computing middleware provided as a standard Python library for describing stream-based workflows.  ...  7 https://pypi.python.org/pypi/dispel4py dispel4py is installed via the Python package management system pip using a simple command: pip install dispel4py The documentation is available on the dispel4py  ... 
doi:10.1145/2835857.2835863 dblp:conf/sc/KrauseFA15 fatcat:jj2oq7h5ejewjndq5jkaicvgsu

dispel4py: A Python Framework for Data-Intensive Scientific Computing

Rosa Filguiera, Iraklis Klampanos, Amrey Krause, Mario David, Alexander Moreno, Malcolm Atkinson
2014 2014 International Workshop on Data Intensive Scalable Computing Systems  
In this work we present four dispel4py mappings: Apache Storm, MPI, multi-threading and sequential.  ...  The results show that dispel4py is successful in enacting on different platforms, while also providing scalable performance.  ...  / https://github.com/akrause2014/dispel4py/blob/master/dispel4py/worker mpi.py 9 http://www.mpich.org/ 10 http://www.open-mpi.org/ 11 https://github.com/akrause2014/dispel4py/blob/master/dispel4py  ... 
doi:10.1109/discs.2014.12 dblp:conf/sc/FilguieraKKDMA14 fatcat:rw3x2og2ord4nhidkjyr3mu2za

dispel4py: An Agile Framework for Data-Intensive eScience

Rosa Filgueira, Amrey Krause, Malcolm Atkinson, Iraklis Klampanos, Alessandro Spinuso, Susana Sanchez-Exposito
2015 2015 IEEE 11th International Conference on e-Science  
We present dispel4py a versatile data-intensive kit presented as a standard Python library. It empowers scientists to experiment and test ideas using their familiar rapid-prototyping environment.  ...  Section III defines the dispel4py concepts. Section IV discusses dispel4py features.  ...  The registry implements a data schema able to describe dispel4py entities in a form usable by the dispel4py enactment engine.  ... 
doi:10.1109/escience.2015.40 dblp:conf/eScience/FilgueiraKAKSS15 fatcat:b5j3qwe2pzbrhlx2tlnbia7vsq

Asterism: Pegasus and Dispel4py Hybrid Workflows for Data-Intensive Science

Rosa Filgueira, Rafael Ferreira da Silva, Amrey Krause, Ewa Deelman, Malcolm Atkinson
2016 2016 Seventh International Workshop on Data-Intensive Computing in the Clouds (DataCloud)  
dispel4py The dispel4py system [17] is a parallel stream-based dataflow framework for formulating and executing data-intensive methods.  ...  Figure 3 : 3 Seismic Ambient Noise Cross-Correlation dispel4py workflows.  ... 
doi:10.1109/datacloud.2016.004 dblp:conf/sc/FilgueiraSKDA16 fatcat:efpt66w6tnho3p7a3j6eoqejva

dispel4py: A Python framework for data-intensive scientific computing

Rosa Filguiera, Amrey Krause, Malcolm Atkinson, Iraklis Klampanos, Alexander Moreno
2016 The international journal of high performance computing applications  
In this work we present four dispel4py mappings: Apache Storm, MPI, multi-threading and sequential.  ...  The results show that dispel4py is successful in enacting on different platforms, while also providing scalable performance.  ...  / https://github.com/akrause2014/dispel4py/blob/master/dispel4py/worker mpi.py 9 http://www.mpich.org/ 10 http://www.open-mpi.org/ 11 https://github.com/akrause2014/dispel4py/blob/master/dispel4py  ... 
doi:10.1177/1094342016649766 fatcat:jhtjn2khjzcwzolrrpfaisesou

Adaptive Optimizations for Stream-based Workflows

Liang Liang, Rosa Filgueira, Yan Yan
2020 Zenodo  
, dispel4py.  ...  Abstract—This work presents three new adaptive optimizationtechniques to maximize the performance of dispel4py work-flows. dispel4py is a parallel Python-based stream-orientateddataflow framework that  ...  Each phase has been implemented as a dispel4py workflow.  ... 
doi:10.5281/zenodo.4396295 fatcat:7auo2a4yxnht3bki4uwzm7uphu

DARE Platform: Enabling Easy Data-Intensive Workflow Composition and Deployment

Rosa
2019 Zenodo  
For achieving this, we have combined the strengths of dispel4py and CWL scientific workflows, Docker containers, Kubernetes infrastructure orchestration, Jupyter notebooks, and Cloud platforms.  ...  Once validated the results, we executed the same codes (dispel4py workflows and SPECFEM3D) using the NSF-Chameleon 5 cloud (using a VM with 24 cores), with a larger dataset, and the parallel MPI dispel4py  ...  Fig. 3 : 3 dispel4py pre-processing workflow, which corresponds to the third step of the RA.  ... 
doi:10.5281/zenodo.3357805 fatcat:exlz3cby2vg5xncjxbwn4dl3le

Adaptive Optimizations for Stream-based Workflows

Liang Liang, Rosa Filguiera, Yan Yan
2020 2020 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS)  
This work presents three new adaptive optimization techniques to maximize the performance of dispel4py workflows. dispel4py is a parallel Python-based stream-orientated dataflow framework that acts as  ...  The evaluations show that our proposed techniques have significantly improved the performance of the original dispel4py framework.  ...  Each phase has been implemented as a dispel4py workflow.  ... 
doi:10.1109/works51914.2020.00010 fatcat:jaivbc4qnbbkznernlttfaz2xi

DARE Platform: a Developer-Friendly and Self-Optimising Workflows-as-a-Service Framework for e-Science on the Cloud

Iraklis Klampanos, Chrysoula Themeli, Alessandro Spinuso, Rosa Filgueira, Malcolm Atkinson, André Gemünd, Vangelis Karkaletsis
2020 Journal of Open Source Software  
More information is provided in the dispel4py Registry repository.  ...  The core DARE platform components are the following: dispel4py dispel4py is a Python library for describing abstract stream-based workflows for distributed data-intensive applications.  ... 
doi:10.21105/joss.02664 fatcat:zx3k5fughnellcq5wtfpvgfo5m

DARE Platform: Enabling Easy Data-Intensive Workflow Composition and Deployment

Rosa
2019 Zenodo  
For achieving this, we have combined the strengths of dispel4py and CWL scientific workflows, Docker containers, Kubernetes infrastructure orchestration, Jupyter notebooks, and Cloud platforms.  ...  /dispel4py dispel4py parallel stream-based dataflow system High-level stream-based data pipelinẽ Apache Beam Embarrassing parallel data-instensive applications Rosa Filgueira Amrey Krause Malcolm  ...  Key-features: Automatic mappings to different engines, concurrent & stream-based Each PE is translated into one or more instances in run-time • Each instance runs in a process • dispel4py does it for you  ... 
doi:10.5281/zenodo.3458522 fatcat:5cuuewar5jgv7ln4u4gt7ztktm

DARE Deliverable D6.3: Pilot Tools and Services, Execution and Evaluation Report I

Federica Magnoni, Emanuele Casarotti, Athanasios Davvetas, Iraklis Klampanos
2021 Zenodo  
The Dispel4py Registry provides functionality for registering workflow entities, such as processing elements (PEs), functions and literals.  ...  Each call now starts a dispel4py MPI cluster in Kubernetes, the corresponding PE is executed there producing the output files, and at the end the cluster is destroyed.  ... 
doi:10.5281/zenodo.4739217 fatcat:sujwzafp6zasbgdzeaw4wbibhi

DARE: A Reflective Platform Designed to Enable Agile Data-Driven Research on the Cloud

Iraklis Klampanos, Federica Magnoni, Emanuele Casarotti, Christian Page, Mike Lindner, Andreas Ikonomopoulos, Vangelis Karkaletsis, Athanasios Davvetas, Andre Gemund, Malcolm Atkinson, Antonios Koukourikos, Rosa Filgueira (+3 others)
2019 2019 15th International Conference on eScience (eScience)  
It implements the cataloguing and execution of fine-grained and Python-based dispel4py workflows as services.  ...  On the other hand, on the context of executing dispel4py workflows, CWL is used to describe and dynamically parameterize distributed dispel4py execution on command level.  ...  Workflow-as-a-Service components dispel4py [1] , s-ProvFlow [9] and the dispel4py PEs and workflows library 21 were originally developed as part of the VERCE project 22 [10] .  ... 
doi:10.1109/escience.2019.00079 dblp:conf/eScience/KlampanosMCPLIK19 fatcat:7immlbim2racxa534ovdylysfi

DARE to Perform Seismological Workflows

Federica Magnoni, Emanuele Casarotti, Pietro Artale Harris, Mike Lindner, Andreas Rietbrock, Iraklis Angelos Klampanos, Athanasios Davvetas, Alessandro Spinuso, Rosa Filgueira, Amy Krause, Malcolm Atkinson, André Gemund (+1 others)
2019 Zenodo  
Other API calls allow for remote, distributed execution of dispel4py workflows, used to describe the steps for data analysis and download of seismic recorded data via EIDA Research Infrastructure services  ...  and reusing multiple simulations for the same earthquake strongly benefits from customisable management of metadata and lineage through the DARE platform exploiting the integration of S-ProvFlow with dispel4py  ... 
doi:10.5281/zenodo.4396310 fatcat:osgsxuj2tnfahjzal3nhow5vmy

Dare Architecture and Technology D2.2

Atkinson Malcolm, Klampanos Iraklis, Andries Valentina, Constantin Aurora, Filgueira Rosa, Genünd André, Gottschämmer Ellen, Karkaletsis Vangelis, Koukourikos Antonis, Levray Amélie, Linder Mike, Magnoni Federica (+6 others)
2020 Zenodo  
The main concepts of dispel4py are managed via a dispel4py Information Registry (Registry) , 22 which is part of the DARE platform's knowledge base ( §4.2).  ...  We acquire provenance from different types of systems (CWL and dispel4py).  ... 
doi:10.5281/zenodo.4733801 fatcat:nqgcpnkog5gmzgiefuqspiupqe

DARE Deliverable D6.4: Pilot Tools and Services, Execution and Evaluation Report II

Federica Magnoni, Mike Lindner, Ellen Gottschämmer, Alicia Rohnacher, Aurora Constantin, Valentina Andries, Malcolm Atkinson
2021 Zenodo  
workflows: ○ Register the PE/workflow to the dispel4py PE registry: workspace_id,impl_id=dm.register_d4p_workflow(name=wf_name,code=code) ○ Execute the PE/workflow with the dispel4py API call specifying  ...  workflows, made by several dispel4py PEs and launched through specific API calls with the same structure and different parameter set up.  ... 
doi:10.5281/zenodo.4740026 fatcat:klu4geevffh7fcbd54z2acvzru
« Previous Showing results 1 — 15 out of 47 results