Rank reordering for MPI communication optimization

Abstract

In this paper we describe a procedure for optimizing the MPI communication of an unstructured CFD code in a parallel multi-core environment. By reordering the MPI ranks, a mapping of MPI processes to CPU cores is established, such that the main communication takes place within the compute nodes. The motivation of this approach is based on the observation that the communication between CPU cores on the same compute node is usually much faster than the communication between CPU cores on different nodes. The generic nature of our approach provides an out-of-the-box optimization tool, which can be easily used with other CFD codes due to the external MPI rank reordering procedure. The optimization tool was successfully tested with the DLR TAU code and the results of the optimization are demonstrated by benchmark computations for different geometries of aircraft configurations.