Multi-modal Building Characterization using Vision Language Models

Kurzfassung

City-scale seismic vulnerability assessment requires characterising the structural properties of tens of thousands of individual buildings, a task traditionally performed through expert field surveys that are too slow and expensive to run at urban scale. This thesis investigates whether vision-language models (VLMs) can automate this process across two objectives. Objective 1 evaluates Global Earthquake Model (GEM) building attribute extraction from street-level imagery (SLI). Four InternVL model configurations are compared against a convolutional neural network (CNN) baseline on a dataset of Santiago de Chile buildings. Zero-shot inference achieves only 37.04% mean overall accuracy (OA), confirming that taskspecific adaptation is necessary. Supervised fine-tuning closes the gap substantially, with InternVL3-2B reaching 84.78% OA and approaching the CNN baseline of 87.30%, while retaining the ability to generate natural language that CNNs fundamentally lack. Objective 2 develops and evaluates an end-to-end pipeline for per-building seismic risk report generation. Using all four input modalities (SLI, optical imagery, normalised Digital Surface Models (nDSM), and Level-of-Detail-1 (LoD1) height data), a fine-tuned Qwen3-VL-8B model produces structured, three-part reports covering building and environmental status, intensity-based risk analysis, and mitigation synthesis. Reports combine VLM-generated visual assessments with deterministic vulnerability and damage computations, and are evaluated using a large language model (LLM)-as-judge paradigm against VLM-generated reference reports. The fine-tuned model achieves near-perfect scores across all report parts, while zero-shot inference fails on rule-based numerical reasoning, producing rule violations in roughly 60% of reports. Together, the results show that fine-tuned VLMs can replace both the structured classification step of a CNN system and the narrative report-writing step of a domain expert, producing complete per-building risk assessments from raw imagery and geospatial data without human intervention.