Evaluation of AI-based P&ID Digitization using Graph Similarity Learning

Abstract

Piping and Instrumentation Diagrams (P&IDs) are plans that represent hydraulic systems in the mechanical industry. However, P&IDs are often available as image or PDF files. The demand is to assess alterations in the system or prevent technical failures. Therefore, there is an interest in the automated conversion of such plans into simulation models. The End-to-End Digitization pipeline presents a possibility for the reconstruction into a simulation model. During this pipeline, the crucial information displayed by symbols, texts, and lines of the P&IDs is detected, extracted and reconstructed as a graph by the pipelines module Plan Digitization. This pipeline’s evaluation system is based on a separate evaluation of the individual detection methods using object detection metrics precision and recall. However, this evaluation does not capture the accuracy of graph reconstruction. Since such P&ID datasets are not publicly accessible, synthetic plans function as training and validation data for the pipeline. A corresponding groundtruth graph exists for each synthetic plan. Exploiting this allows holistically evaluating the reconstruction of P&IDs using graph similarity. However, the computation of the most common graph similarity metrics, Graph Edit Distance (GED), is at least an NP hard problem. The calculation of this metric is exponentially time-consuming, with a linear increase in graph size. An alternative to estimating this metric in a reasonable time is GSL-GNN (Graph Similarity Learning-Graph Neural Network). These models allow the estimation of GED if trained on the same metric. The main proposal of this work is the creation of a promising holistic evaluation method using GSL-GNN models. Unfortunately, after a thorough and detailed analysis, the chosen models are less suitable for a holistic evaluation. The analysis shows that the performance of GSL-GNN models highly depends on the training datasets and lack of generalization. Another discovery is that Plan Digitization only reconstructs some P&ID as planar graphs. Also, the performance of symbol detection seems less connected to the quality of graph reconstruction.