Exploring Multi-threaded Communication Behavior of a Large-Scale CFD Solver with Vampir

Abstract

Analyzing the communication behavior of distributed parallel applications is a key task of parallel performance tools. Over the years, the MPI standard has extended the possibilities to leverage threads within individual processes. The current MPI standard defines several levels of thread-support, with MPI_THREAD_MULTIPLE being the least restrictive level, at which any thread can issue MPI calls at any time. This feature allows applications maximum flexibility in their communication patterns, is used in production codes, but poses major challenges for performance analysis frameworks. Irrespective of the challenges associated with trace data recording, this work focuses on challenges for a scalable performance analysis: reading MPI events across multiple threads, aggregating information from distributed communication events and deriving correct communication metrics. In this work, we present enhancements to the distributed analysis engine of Vampir - a tool to visualize and analyze MPI communication behavior - to support the investigation of MPI calls on any thread level. We demonstrate the applicability of our work by a performance evaluation of a large-scale CFD solver, especially investigating its MPI_THREAD_MULTIPLE based communication patterns which was not possible before. Furthermore, we also compare the induced overhead to the load-time with previous versions of Vampir.