Internet Archive Scholar

Leading magnetic fusion energy science into the big-and-fast data lane

Ralph Kube, R Churchill, Jong Choi, Ruonan Wang, Scott Klasky, CS Chang, Minjun Choi, Jinseop Park
2020 Proceedings of the 19th Python in Science Conference   unpublished

Preserved Fulltext

fulltext thumbnail
Web Archive Capture PDF (458.5 kB)
https://web.archive.org/web/20201106143404/http://conference.scipy.org/proceedings/scipy2020/pdfs/ralph_kube.pdf

A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL. The file type is application/pdf.

We present Delta, a Python framework that connects magnetic fusion experiments to high-performance computing (HPC) facilities in order leverage advanced data analysis for near real-time decisions. Using the ADIOS I/O framework, Delta streams measurement data with over 300 MByte/sec from a remote experimental site in Korea to Cori, a Cray XC-40 supercomputer at the National Energy Energy Research Scientific Computing Centre in California. There Delta dispatches cython data analysis kernels using
more » ... an mpi4py PoolExecutor in order to perform a spectral data analysis workflow. Internally Delta uses queues and worker threads for data communication. With this approach we perform a common spectral analysis suite on imaging measurements more than 100 times faster than with a single-core implementation.
doi:10.25080/majora-342d178e-013 fatcat:2javvmggiba7hcenuxtzhsdse4
Citation

Ralph Kube, R Churchill, Jong Choi, Ruonan Wang, Scott Klasky, CS Chang, Minjun Choi, Jinseop Park. "Leading magnetic fusion energy science into the big-and-fast data lane." Proceedings of the 19th Python in Science Conference (2020)