MDTM: Optimizing Data Transfer Using Multicore-Aware I/O Scheduling

Liang Zhang, Phil Demar, Bockjoo Kim, Wenji Wu
2017 2017 IEEE 42nd Conference on Local Computer Networks (LCN)  
Bulk data transfer is facing significant challenges in the coming era of big data. There are multiple performance bottlenecks along the end-to-end path from the source to destination storage system. The limitations of current generation data transfer tools themselves can have a significant impact on end-to-end data transfer rates. In this paper, we identify the issues that lead to underperformance of these tools, and present a new data transfer tool with an innovative I/O scheduler called MDTM.
more » ... The MDTM scheduler exploits underlying multicore layouts to optimize throughput by reducing delay and contention for I/O reading and writing operations. With our evaluations, we show how MDTM successfully avoids NUMA-based congestion and significantly improves end-to-end data transfer rates across high-speed wide area networks.
doi:10.1109/lcn.2017.64 dblp:conf/lcn/ZhangDKW17 fatcat:5fw764fqmjckjpl4su7fl43kom