Structured Pruning and Lightweight Adaptation of Foundation Models for Resource-Efficient Earth Observation

Kurzfassung

Structured pruning has gained attention as a promising solution for the cost-effective deployment of large language, vision-language, and domain-specific foundation models. However, most prior work has focused on general-purpose language models and often lacks a systematic comparison of pruning strategies across model scales, architectures, and real-world multimodal applications. In this paper, we conduct a comprehensive evaluation of structured pruning methods, focusing on depth pruning, width pruning, and post-pruning recovery with parameter-efficient fine-tuning. We evaluate multiple open-source models, including Qwen3-0.6B, Qwen3-4B-Instruct, Qwen3-8B, LLaVA-OneVision-0.5B, and Phi-4-Mini-Instruct, across intrinsic metrics and downstream benchmarks such as BBH, MATH Hard, MuSR, and MMLU-Pro. Our findings reveal that (1) depth pruning generally provides a more stable trade-off between efficiency and performance than width pruning; (2) width pruning can reduce memory usage, but often gives limited throughput gains and less predictable accuracy retention; (3) reasoning-intensive benchmarks, particularly mathematical reasoning tasks, are more sensitive to structured pruning than some other evaluation tasks; (4) LoRA-based post-pruning adaptation can substantially recover performance after moderate pruning; and (5) pruning behaviors observed in general-purpose models transfer to Earth observation foundation models, as demonstrated on TerraMind, where depth pruning reduces latency and GPU memory usage while maintaining competitive EuroSAT and Bigearthnet task accuracy.