Distributed Schur Complement Solvers for Real and Complex Block-Structured CFD Problems

Abstract

At the Institute for Propulsion Technology of the German Aerospace Center (DLR), the parallel simulation system TRACE (Turbo-machinery Research Aerodynamic Computational Environment) has been developed specifically for the calculation of internal turbo-machinery flows. The finite volume approach with block-structured grids requires the parallel, iterative solution of large, sparse real and complex systems of linear equations. For convergence acceleration of the iteration, Distributed Schur Complement (DSC) preconditioners for real and complex matrix problems have been investigated. The DSC method requires adaquate partitioning of the matrix problem since the order of the approximate Schur complement system to be solved depends on the number of couplings between the sub-domains. Graph partitioning with ParMETIS from the University of Minnesota is suitable since a minimization of the number of edges cut in the adjacency graph of the matrix corresponds to a minimization of the number of the coupling variables between the subdomains. The latter determine the order of the approximate Schur complement system used for preconditioning. Since even the matrix pattern is non-symmetric for block-structured TRACE problems it has to be symmetrized so that the corresponding matrix adjacency graph becomes undirected und ParMETIS can be applied. Matrix permutations like Reverse Cuthill-McKee (RCM) and Minimum Degree (MD) are employed per sub-domain in order to reduce fill-in in incomplete LU factorizations which are part of the DSC preconditioner. Numerical and performance results of these methods are discussed for typical TRACE problems on multi-core architectures together with an analysis of the pros and cons of the complex problem formulation, e.g. regarding the ratio of calculation operations to memory accesses. The results show that matrix permutations are crucial for DSC preconditioner as well as iterative solver performance. The DSC preconditioned iterative solvers for the complex problem formulation distinctly outperform the solvers for the real formulation. Reasons are that the complex formulation results in lower problem order, more advantageous matrix structure, has higher data locality and a better ratio of computation to memory access.