Filters








91 Hits in 5.1 sec

Distributing the Sloan Digital Sky Survey Using UDT and Sector

Yunhong Gu, Robert Grossman, Alex Szalay, Ani Thakar
2006 2006 Second IEEE International Conference on e-Science and Grid Computing (e-Science'06)  
We also describe our recent experience using Sector to distribute the Sloan Digital Sky Survey BESTDR4 catalog data.  ...  In this paper, we describe a peer-to-peer storage system called Sector that is designed to access and transport large data sets over wide area high performance networks.  ...  We also describe how Sector has been used recently to distribute data from the Sloan Digital Sky Survey (SDSS) [24] .  ... 
doi:10.1109/e-science.2006.261140 dblp:conf/eScience/GuGST06 fatcat:7ji7brxslfg4lbxscgczdtqe6e

Community Collaboration [chapter]

2013 Encyclopedia of Behavioral Medicine  
… Integrating SDSS & 2MASS Data to Finding Candidate Brown Dwarfs Sloan Digital Sky Survey (SDSS) -82 million stars -Visible spectrum Two Micro All Sky Survey (2MASS) -208 million stars -  ...  and declination used by Sloan Digital Sky Survey Community Earth science -1/2 degree x 1/2 degree latitude x longitude grid used by Community Climate Model 3 (CCM3) Bioinformatics -enzyme EC numbers In  ... 
doi:10.1007/978-1-4419-1005-9_100342 fatcat:bxm3nvvlibcz3awkm263gqntke

Compute and Storage Clouds Using Wide Area High Performance Networks [article]

Robert L. Grossman, Yunhong Gu, Michael Sabala, Wanzhi Zhang
2008 arXiv   pre-print
The infrastructure consists of a storage cloud called Sector and a compute cloud called Sphere. We describe two applications that we have built using the cloud and some experimental studies.  ...  We describe a cloud based infrastructure that we have developed that is optimized for wide area, high performance networks and designed to support data mining applications.  ...  In particular, we have used the Sector Cloud to distribute the Sloan Digital Sky Survey data [9] to astronomers around the world.  ... 
arXiv:0808.1802v1 fatcat:7fkomb4yyvc2dniux6ganh3p2u

Compute and storage clouds using wide area high performance networks

Robert L. Grossman, Yunhong Gu, Michael Sabala, Wanzhi Zhang
2009 Future generations computer systems  
The infrastructure consists of a storage cloud called Sector and a compute cloud called Sphere. We describe two applications that we have built using the cloud and some experimental studies.  ...  We describe a cloud based infrastructure that we have developed that is optimized for wide area, high performance networks and designed to support data mining applications.  ...  In particular, we have used the Sector Cloud to distribute the Sloan Digital Sky Survey data [9] to astronomers around the world.  ... 
doi:10.1016/j.future.2008.07.009 fatcat:miuxeweblzacbnvghbnauzb53q

A Peer-to-Peer Infrastructure for Distributing Large Scientific Data Sets over Wide Area High-Performance Networks: Experimental Studies Using Wide Area Layer 2 Services

Yunhong Gu, Robert L. Grossman, Joe Mambretti
2007 Proceedings of the 1st International ICST Conference on Networks for Grid Applications  
This paper presents Sector, a distributed environment that was created specifically to address the challenges inherent in accessing, exploring, analyzing and transporting extremely large scientific datasets  ...  To date, the effective utilization of such datasets has been limited because accessing and transporting large remote data sets in traditional distributed computing environments is often a challenge.  ...  EXPERIMENTAL STUDIES We have been using Sector to store and distribute Sloan Digital Sky Survey (SDSS) data [SDSS] over Teraflow Network [TFN] .  ... 
doi:10.4108/gridnets.2007.2166 dblp:conf/gridnets/GuGM07 fatcat:agtuorasbfdyblyimknb6qopbq

An overview of the Open Science Data Cloud

Robert L. Grossman, Yunhong Gu, Joe Mambretti, Michal Sabala, Alex Szalay, Kevin White
2010 Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing - HPDC '10  
The Open Science Data Cloud is a distributed cloud based infrastructure for managing, analyzing, archiving and sharing scientific datasets.  ...  We introduce the Open Science Data Cloud, give an overview of its architecture, provide an update on its current status, and briefly describe some research areas of relevance.  ...  Equipment for the Open Science Data Cloud has been donated by Yahoo! Inc. This equipment will be used to host the Cistrack data, Sloan Digital Sky Survey data, and a variety of other datasets.  ... 
doi:10.1145/1851476.1851533 dblp:conf/hpdc/GrossmanGMSSW10 fatcat:ozq3i6bqmjcelmfinxvdxqgoqy

Sector and Sphere: the design and implementation of a high-performance data cloud

Y. Gu, R. L. Grossman
2009 Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences  
We describe some experimental studies comparing Sector/Sphere and Hadoop using the Terasort Benchmark. In these studies, Sector is about twice as fast as Hadoop. Sector/Sphere is open source.  ...  In contrast to existing storage and compute clouds, Sector can manage data not only within a data center, but also across geographically distributed data centers.  ...  SDSS Data Distribution Sector is currently used to distribute the data products from the Sloan Digital Sky Survey (SDSS) over the Teraflow Testbed [12, 21] .  ... 
doi:10.1098/rsta.2009.0053 pmid:19451100 pmcid:PMC3391065 fatcat:wmujs7obvndvfh2e32o46xhdfq

Sector and Sphere: Towards Simplified Storage and Processing of Large Scale Distributed Data [article]

Yunhong Gu, Robert L Grossman
2009 arXiv   pre-print
We describe some experimental studies comparing Sector/Sphere and Hadoop using the Terasort Benchmark. In these studies, Sector is about twice as fast as Hadoop. Sector/Sphere is open source.  ...  In contrast to existing storage and compute clouds, Sector can manage data not only within a data center, but also across geographically distributed data centers.  ...  Sector has been used to support distributing Sloan Digital Sky Survey data releases to astronomers around the world [12] .  ... 
arXiv:0809.1181v2 fatcat:56zdxmd22rgbhl76g4ug5sh6ga

Lessons Learned from the SDSS Catalog Archive Server

Ani R. Thakar
2008 Computing in science & engineering (Print)  
Thakar Johns Hopkins University The Sloan Digital Sky Survey is one of the first very large archives in astronomy and other sciences.  ...  S D S S S c i e n c e A r c h i v e T he Sloan Digital Sky Survey, or SDSS, is a multi-institution, internationally funded project to map about half of the northern sky in unprecedented detail with a dedicated  ... 
doi:10.1109/mcse.2008.151 fatcat:cftfdv45ivcyzimv4dhdmac3wu

Snowmass Computing Frontier: Computing for the Cosmic Frontier, Astrophysics, and Cosmology [article]

Andrew Connolly, Salman Habib, Alex Szalay, Julian Borrill, George Fuller, Nick Gnedin, Katrin Heitmann, Danny Jacobs, Don Lamb, Tony Mezzacappa, Bronson Messer, Steve Myers (+5 others)
2013 arXiv   pre-print
This document presents (off-line) computing requrements and challenges for Cosmic Frontier science, covering the areas of data management, analysis, and simulations.  ...  The spectroscopic counterpart to DES is BOSS (the Baryon Oscillation Spectroscopic Survey) [2] ; a multi-fiber spectroscopic survey that utilizes the Sloan Digital Sky Survey telescope (as part of SDSS-III  ...  sector.  ... 
arXiv:1311.2841v1 fatcat:pqhwoukpezairkrutstjcb3sk4

A New Framework for a Model-Based Data Science Computational Platform [article]

Demitri Muna, Eric Huff
2014 arXiv   pre-print
No one can load this data on their own computer, most cannot even keep it at their institution, and worse, no platform exists that allows one to evaluate their models across the whole of the data.  ...  Simply having an extremely large volume of data available in one place is not sufficient; one must be able to make valid, rigorous, scientific comparisons across very different data sets from very different  ...  The Sloan Digital Sky Survey has a web interface that serves SDSS data covering imaging and spectral data.  ... 
arXiv:1402.5932v1 fatcat:au5feoownvhsnndzn3tnsoiele

Big Data Collection in Large-Scale Wireless Sensor Networks

Asside Djedouboum, Ado Abba Ari, Abdelhak Gueroui, Alidou Mohamadou, Zibouda Aliouat
2018 Sensors  
In view of the characteristics of these data, we believe that very large-scale and heterogeneous WSNs can be very useful for collecting and processing these Big Data.  ...  This paper reviews the background and state of the art of Big Data collection in Large-Scale WSNs (LS-WSNs), compares and discusses on challenges of Big Data collection in LS-WSNs, and proposes possible  ...  This is followed in Section 6 by a discussion on the challenges of Big Data collection in LS-WSNs. Finally, we conclude the paper in Section 7.  ... 
doi:10.3390/s18124474 fatcat:eavks4226baatpnfx4bvyqwexq

Big Data Research Infrastructure Collaboration Toward the SKA (BRICSKA)

RUSS TAYLOR, FABIO PORTO, CHENZHOU CUI, YOGESH WADADEKAR, OLEG MALKOV
2021 Anais da Academia Brasileira de Ciências  
South Africa, Brazil, China and India are involved in the Large Synoptic Survey Telescope (LSST), which represents a complementary mega-data challenge, vastly increasing the current data volume of optical  ...  surveys, and providing critical multi-wavelength data set for SKA analytics.  ...  The challenges are (i) Big Data, how to record, transport and deliver it, (ii) Big Compute, how to process, analyse, and visualise big data, (iii) Big Science; by asking big questions as a global community  ... 
doi:10.1590/0001-3765202120201027 pmid:34076205 fatcat:t4rn4jetyrcm5ocwt2d7sxhsum

CMB-S4 Science Case, Reference Design, and Project Plan [article]

Kevork Abazajian, Graeme Addison, Peter Adshead, Zeeshan Ahmed, Steven W. Allen, David Alonso, Marcelo Alvarez, Adam Anderson, Kam S. Arnold, Carlo Baccigalupi, Kathy Bailey, Denis Barkats (+190 others)
2019 arXiv   pre-print
Spectroscopic Survey (BOSS), and Sloan Digital Sky Survey (SDSS) spectroscopic surveys (LSST and other photometric surveys will also provide useful catalogs for such measurements) and cluster samples  ...  to investigate possibilities for implementing a connection to the US data center with sufficient bandwidth to stream the data.  ...  In addition, the readout electronics under development at SLAC use a fully digital approach that is largely insensitive to low-frequency noise induced by temperature fluctuations and phase drifts.  ... 
arXiv:1907.04473v1 fatcat:7lqitlzxefgizgekpzpccbdc3y

Big data: Issues, challenges, tools and Good practices

Avita Katal, Mohammad Wazid, R. H. Goudar
2013 2013 Sixth International Conference on Contemporary Computing (IC3)  
Due to such large size of data it becomes very difficult to perform effective analysis using the existing traditional techniques.  ...  Big data due to its various properties like volume, velocity, variety, variability, value and complexity put forward many challenges.  ...  The Sloan Digital Sky Survey is a multi-filter imaging and spectroscopic redshift survey using a 2.5-m wide-angle optical telescope at Apache Point Observatory in New Mexico, United States.  ... 
doi:10.1109/ic3.2013.6612229 dblp:conf/ic3/KatalWG13 fatcat:abat3k5hmzgbdm3pnpjylcgc2u
« Previous Showing results 1 — 15 out of 91 results