Assessing HPC Failure Detectors for MPI Jobs

Kishor Kharbas, Donghoon Kim, Torsten Hoefler, Frank Mueller
2012 2012 20th Euromicro International Conference on Parallel, Distributed and Network-based Processing  
Reliability is one of the challenges faced by exascale computing. Components are poised to fail during largescale executions given current mean time between failure (MTBF) projections. To cope with failures, resilience methods have been proposed as explicit or transparent techniques. For the latter techniques, this paper studies the challenge of fault detection. This work contributes a study on generic fault detection capabilities at the MPI level and beyond. The objective is to assess
more » ... detectors, which ultimately may or may not be implemented within the application's runtime layer. A first approach utilizes a periodic liveness check while a second method promotes sporadic checks upon communication activities. The contributions of this paper are two-fold: (a) We provide generic interposing of MPI applications for fault detection. (b) We experimentally compare periodic and sporadic methods for liveness checking. We show that the sporadic approach, even though it imposes lower bandwidth requirements and utilizes lower frequency checking, results in equal or worse application performance than a periodic liveness test for larger number of nodes. We further show that performing liveness checks in separation from MPI applications results in lower overhead than interpositioning, as demonstrated by our prototypes. Hence, we promote separate periodic fault detection as the superior approach for fault detection.
doi:10.1109/pdp.2012.11 dblp:conf/pdp/KharbasKHM12 fatcat:mpvstwgbmndt5cbxlo25krwt5m