543 Hits in 5.3 sec

Custom Execution Environments with Containers in Pegasus-enabled Scientific Workflows [article]

Karan Vahi, Mats Rynge, George Papadimitriou, Duncan A. Brown, Rajiv Mayani, Rafael Ferreira da Silva, Ewa Deelman, Anirban Mandal, Eric Lyons, Michael Zink
2019 arXiv   pre-print
This paper highlights some unique challenges that may arise when using containers in distributed scientific workflows.  ...  While these steps are often completely described, including the input parameters, datasets, and codes, the environment in which these steps are executed is only described at a higher level with endpoints  ...  Workflows."  ... 
arXiv:1905.08204v1 fatcat:wz377xxoifbl5b5on3xsgu3uya

Integrating existing scientific workflow systems

Nandita Mandal, Ewa Deelman, Gaurang Mehta, Mei-Hui Su, Karan Vahi
2007 Proceedings of the 2nd workshop on Workflows in support of large-scale science - WORKS '07  
Scientific workflows have become an important tool used by scientists to conduct large-scale analysis in distributed environments.  ...  In this paper we examine the possibility of integrating two well-known workflow systems Kepler and Pegasus and examine the opportunities and challenges presented by such an integration.  ...  Catalog Integration As we have shown in Figure 4b , we have integrated the Pegasus Transformation Catalog with the Kepler environment in a way which allows the user to customize the Pegasus actor.  ... 
doi:10.1145/1273360.1273365 dblp:conf/hpdc/MandalDMSV07 fatcat:zqpuaz46mraj5ofavx5cgjdjoy

Enabling large-scale scientific workflows on petascale resources using MPI master/worker

Mats Rynge, Scott Callaghan, Ewa Deelman, Gideon Juve, Gaurang Mehta, Karan Vahi, Philip J. Maechling
2012 Proceedings of the 1st Conference of the Extreme Science and Engineering Discovery Environment on Bridging from the eXtreme to the campus and beyond - XSEDE '12  
We describe how the partitioning and job management has been implemented in the Pegasus Workflow Management System.  ...  In this paper we describe a new approach to executing large, fine-grained workflows on distributed petascale systems.  ...  The work described in this paper used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation grant number OCI-1053575.  ... 
doi:10.1145/2335755.2335846 fatcat:tl3il2esvncnvlf7w2epbrok54

Pegasus, a workflow management system for science automation

Ewa Deelman, Karan Vahi, Gideon Juve, Mats Rynge, Scott Callaghan, Philip J. Maechling, Rajiv Mayani, Weiwei Chen, Rafael Ferreira da Silva, Miron Livny, Kent Wenger
2015 Future generations computer systems  
This paper provides an integrated view of the Pegasus system, showing its capabilities that have been developed over time in response to application needs and to the evolution of the scientific computing  ...  The paper describes how Pegasus achieves reliable, scalable workflow execution across a wide variety of computing infrastructures.  ...  Pegasus has been in development since 2001 and has benefited greatly from the expertise and efforts of people who worked on it over the years.  ... 
doi:10.1016/j.future.2014.10.008 fatcat:u5lbouuekvduhfdvdi7phfyn7i

Experiences using cloud computing for a scientific workflow application

Jens-Sönke Vöckler, Gideon Juve, Ewa Deelman, Mats Rynge, Bruce Berriman
2011 Proceedings of the 2nd international workshop on Scientific cloud computing - ScienceCloud '11  
in a cloud environment.  ...  Clouds are rapidly becoming an important platform for scientific applications. In this paper we describe our experiences running a scientific workflow application in the cloud.  ...  This enables workflows to be automatically adapted to run across different environments.  ... 
doi:10.1145/1996109.1996114 fatcat:xaldzcbzrjdvzob4qjglgufkvq

Bringing Scientific Workflow to the Masses via Pegasus and HUBzero

Michael McLennan, Steven M. Clark, Frank McKenna, Ewa Deelman, Mats Rynge, Karan Vahi, Derrick Kearney, Carol X. Song
2013 International Workshop on Science Gateways  
With the press of a button, these tools can launch Pegasus workflows on national computing infrastructures and bring results back for plotting and visualization.  ...  The Pegasus Workflow Management System can manage workflows comprised of millions of tasks, all the while recording data about the execution and intermediate results so that the provenance of the final  ...  Customized Workflows in Pegasus More experienced users can learn how to create their own customized Pegasus workflows.  ... 
dblp:conf/iwsg/McLennanCMDRVKS13 fatcat:dpkhztkt7fh3vndvv6ubxclqny

Executing cyclic scientific workflows in the cloud

Michel Krämer, Hendrik M. Würz, Christian Altenhofen
2021 Journal of Cloud Computing: Advances, Systems and Applications  
AbstractWe present an algorithm and a software architecture for a cloud-based system that executes cyclic scientific workflows whose structure may change during run time.  ...  The evaluation demonstrates that our algorithm is able to execute dynamically changing workflows with cycles and that design and maintenance of complex workflows is easier than with existing solutions.  ...  Pegasus and Argo require custom actions, sub-workflows/templates, or counter variables to simulate this behaviour.  ... 
doi:10.1186/s13677-021-00229-7 fatcat:ogzdrrdidza6zols2yb5sg3fsq

Evaluating I/O aware network management for scientific workflows on networked clouds

Anirban Mandal, Paul Ruth, Ilya Baldin, Yufeng Xin, Claris Castillo, Mats Rynge, Ewa Deelman
2013 Proceedings of the Third International Workshop on Network-Aware Data Management - NDM '13  
The application was deployed on a virtualized HTCondor environment provisioned dynamically from the ExoGENI networked cloud testbed, and managed by the Pegasus workflow manager.  ...  This paper presents a performance evaluation of scientific workflows on networked cloud systems with particular emphasis on evaluating the effect of provisioned network bandwidth on application I/O performance  ...  The recent advances in enabling on-demand network circuits, coupled with programmable edge technologies create an unprecedented opportunity to enable complex data-intensive scientific applications to run  ... 
doi:10.1145/2534695.2534698 dblp:conf/sc/MandalRBXCRD13 fatcat:b4egdprs6nb7xme4codm2k4qou

Pegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed Systems

Ewa Deelman, Gurmeet Singh, Mei-Hui Su, James Blythe, Yolanda Gil, Carl Kesselman, Gaurang Mehta, Karan Vahi, G. Bruce Berriman, John Good, Anastasia Laity, Joseph C. Jacob (+1 others)
2005 Scientific Programming  
Pegasus enables users to represent the workflows at an abstract level without needing to worry about the particulars of the target execution systems.  ...  This paper describes the Pegasus framework that can be used to map complex scientific workflows onto distributed resources.  ...  Acknowledgments Pegasus is supported by NSF under grants ITR-0086044 (GriPhyN), ITR AST0122449 (NVO) and EAR-0122464 (SCEC/ITR).  ... 
doi:10.1155/2005/128026 fatcat:pld7bbr5fjgtxpmlewtgtzd4km

Enhanced Resource Provisioning Strategies for Scientific Workflows in Cloud Environment: A Survey

S. Sridevi, Jeevaa Katiravan
2018 International Journal of Computer Applications  
The on-demand provisioning and resource availability in cloud computing make it ideal for executing scientific workflow applications. To ensure better performance, there is a need for auto-scaling.  ...  The problem of assigning resources to tasks and orchestrating their execution to preserve the dependencies of workflows is NP-complete. [2] Hence, no optimal solution can be found in polynomial time.  ...  The IWD is a population based algorithm proposed by Shah-Hosseini in 2007 for solving combinatorial problems. The IWD algorithm mimics the dynamics of river system and the  ... 
doi:10.5120/ijca2018917097 fatcat:sbdiqpmbxvd6njvdg6xwyeu7iu

Simplifying construction of complex workflows for non-expert users of the Southern California Earthquake Center Community Modeling Environment

Philip Maechling, Gaurang Mehta, Brian Mendenhall, Thomas Russ, Gurmeet Singh, Marc Spraragen, Garrick Staples, Karan Vahi, Hans Chalupsky, Maureen Dougherty, Ewa Deelman, Yolanda Gil (+4 others)
2005 SIGMOD record  
However, non-expert users are better served with simple interfaces that abstract away system complexities and still enable them to construct and execute complex workflows.  ...  Workflow systems often present the user with rich interfaces that express all the capabilities and complexities of the application programs and the computing environments that they support.  ...  Environment (SCEC/CME): An Information Infrastructure for System-Level Earthquake Research).  ... 
doi:10.1145/1084805.1084811 fatcat:o33zte5zarecxlmve3rjrisq2e

From Data to Knowledge to Discoveries: Artificial Intelligence and Scientific Workflows

Yolanda Gil
2009 Scientific Programming  
A key emerging concept is scientific workflows, which provide a declarative representation of complex scientific applications that can be automatically managed and executed in distributed shared resources  ...  Scientific computing has entered a new era of scale and sharing with the arrival of cyberinfrastructure facilities for computational experimentation.  ...  The author would like to thank Ewa Deelman and Carole Goble for very useful discussions on scientific workflows. Bibliography  ... 
doi:10.1155/2009/167604 fatcat:ydfv3sqmkjdwxj4dxmimkdid2e

Comparing FutureGrid, Amazon EC2, and Open Science Grid for Scientific Workflows

Gideon Juve, Mats Rynge, Ewa Deelman, Jens-S. Vockler, G. Bruce Berriman
2013 Computing in science & engineering (Print)  
In this paper we describe our experiences modeling the periodogram application as a scientific workflow using Pegasus, and deploying it on the FutureGrid scientific cloud testbed, the Amazon EC2 commercial  ...  As a result, workflows enable scientists to easily define multistage computational and data processing pipelines that can be executed in parallel on distributed resources, which automates complex analyses  ...  This research will aid in the development of capabilities for science applications in the cloud that cannot be provided by traditional scientific computing platforms such as grids.  ... 
doi:10.1109/mcse.2013.44 fatcat:loobib3hfbh63l3xuevxdpvyue

Cloud infrastructure provenance collection and management to reproduce scientific workflows execution

Khawar Hasham, Kamran Munir, Richard McClatchey
2018 Future generations computer systems  
Therefore, a mechanism is required that enables capturing of infrastructure information along with the provenance of workflows executing on the Cloud to facilitate the re-creation of execution environment  ...  The emergence of Cloud computing provides a new computing paradigm for scientific workflow execution.  ...  Acknowledgements The authors would like to acknowledge the support of the European Union in funding this work via the neuGRID4You (N4U) project (grant agreement n. 283562, 2011-2015).  ... 
doi:10.1016/j.future.2017.07.015 fatcat:mmzzhovmqbhhtovcvq2ztp2c6i

Parameterized specification, configuration and execution of data-intensive scientific workflows

Vijay S. Kumar, Tahsin Kurc, Varun Ratnakar, Jihie Kim, Gaurang Mehta, Karan Vahi, Yoonju Lee Nelson, P. Sadayappan, Ewa Deelman, Yolanda Gil, Mary Hall, Joel Saltz
2010 Cluster Computing  
Data analysis processes in scientific applications can be expressed as coarse-grain workflows of complex data processing operations with data flow dependencies between them.  ...  are known to affect their execution times.  ...  Acknowledgements This research was supported in part by the National Science Foundation under Grants #CNS-0403342, #CNS-0426241, #CSR-0509517, #CSR-0615412, #ANI-0330612, #CCF-0342615, #CNS-0203846, #ACI-b0130437  ... 
doi:10.1007/s10586-010-0133-8 pmid:22623878 pmcid:PMC3356923 fatcat:otnq42fe2nf7rge64b4j3cvaw4
« Previous Showing results 1 — 15 out of 543 results