Parallel scalability analysis of a hierarchical adaptive mesh refinement workflow

Kurzfassung

Computational fluid dynamics underpins aerospace development. However, accurate simulation of full aircraft configurations at in-flight speeds requires large grids and, hence, massive parallelism on high performance computers. For such applications, adaptive mesh refinement (AMR) allows to lower the computational effort by refining the mesh at locations of interest. As this happens during the simulation process, performance and scalability of the overall toolchain is tightly coupled with the scalability of the AMR library, the flow solver and the coupling between them. This contribution employs the performance tools Score-P, Cube and Vampir to investigate and improve the scalability of a simulation toolchain using AMR: A small Python-steered toolchain implemented within the FlowSimulator framework, coupling the flow solver "CFD by ONERA, DLR and Airbus" (CODA) with the hierarchical AMR library FSAdaptationNG which was just extended towards MPI parallelism. With the performance tools, it is demonstrated that CODA scales well even down to a few hundred cells per core, whereas FSAdaptationNG's initial MPI implementation scales less well. Using the performance tools' profiling and tracing, the underlying performance bottlenecks - ranging from unrequired synchronization over serial code parts to load imbalance - are identified and thereafter solutions proposed and first performance gains shown. The study demonstrates how parallel analysis tools can expose and fix scalability bugs not just in academic, but also industrial applications and how a tailored repartitioning strategy optimizes the overall runtime of the coupled workflow. After showcasing the usage of the benefits of the performance tools, effectivity, usability, and ease-of-use of them for such applications is discussed.