Transformer for Regression

Abstract

Transformer architectures have achieved remarkable success across natural language processing and computer vision, yet their effectiveness on tabular data remains contested. This thesis presents a systematic comparison of a custom transformer model and the gradient-boosted decision tree framework XGBoost for regression on large-scale tabular data from combustion chamber simulations conducted at the German Aerospace Center. The simulation dataset comprises 15 continuous features and approximately 8.2 million samples after data cleaning and outlier removal. Both model families are evaluated across four nested data partitions (10K, 100K, 1M, 8.2M samples) to assess at which scale the added complexity of transformers becomes justified. Each model is examined in terms of predictive performance as well as training, validation, and inference duration. XGBoost establishes a strong baseline across all scales, achieving R2 = 0.992 already on the Small partition and maintaining near-constant accuracy up to the Full partition (R2 = 0.995). The custom transformer starts lower (R2 = 0.898 on the Small partition) but improves steadily with additional data, surpassing XGBoost on the Full partition with R2 = 0.998. Subsequently, a systematic ablation study is conducted to quantify the impact of individual transformer architectural components on regression performance. The study identifies residual connections as the single most critical component: removing them collapses R2 from 0.984 to 0.097. Furthermore, the results show that two encoder layers suffice for lowdimensional tabular data, dropout rates above 0.03 cause underfitting, and positional encoding provides only marginal benefit. Additionally, the pretrained foundation model TabPFN 2.5 is evaluated against XGBoost and the custom transformer across the Small, Medium, and Large partitions. TabPFN 2.5 achieves consistently high accuracy (R2 = 0.988-0.989) without any task-specific training, outperforming the custom transformer on smaller partitions while requiring substantially less computation. Finally, a proof-of-concept surrogate optimization experiment demonstrates that a trained XGBoost model can be used to search for input configurations that minimize the predicted fitness value. The results suggest that the transformer´s performance advantage on the Full partition comes at higher computational cost compared to XGBoost, highlighting the tradeoff between predictive accuracy and training efficiency.