Disentangling geometry and texture: An investigation into large-scale RGB-DSM pre-training for 3D urban understanding

Kurzfassung

Recent Earth observation foundation models have achieved strong transfer performance using large-scale self-supervised pre-training, but most remain primarily spectral or RGB-based and do not explicitly exploit geometric information from digital surface models (DSMs). This limits their effectiveness in dense urban scenes, where height, building boundaries, and vertical structure are essential for tasks such as building extraction and height estimation. In this work, we present HiRes-FusedMIM, an empirical study of large-scale multi-modal pre-training for high-resolution urban remote sensing. We curate a dataset of 1.2 million paired RGB and DSM image tiles from native source products primarily between 0.1 to 0.5 m spatial resolution, processed to training resolutions of 0.2 to 0.5 m where appropriate, making it, to our knowledge, one of the largest publicly available high-resolution paired RGB–DSM datasets. Specifically, we compare shared and modality-specialized dual encoders, masked image modeling and contrastive alignment, and small- versus large-scale pre-training. Our results show that architecture is critical for geometry-sensitive transfer. A shared encoder suffers from modality interference and drops to 18.0 AP50 on UBCv2 instance segmentation, whereas the dual encoder reaches 31.5 AP50. Similarly, the dual encoder improves LoveDA semantic segmentation from 41.58 to 52.28 mIoU and reduces height-estimation error from 9.26 m to 7.89 m RMSE. We further find that global contrastive alignment benefits some scene classification tasks but degrades dense geometric prediction. Finally, scaling unique training samples from 368k to 1.2M improves UBCv2 AP50 by 3.0 points, indicating that data diversity is more effective than repeated training on a smaller corpus. These findings provide practical guidance for designing geometry-aware foundation models for urban remote sensing.