Structured Pruning of Large Language Models: Empirical Evaluation of Depth and Width Pruning with Dependency-Aware Design and Failure Analysis

Kurzfassung

Large Language Models (LLMs) and multimodal foundation models have achieved impressive performance across diverse reasoning and language understanding tasks, yet their rapidly expanding parameter counts impose severe constraints on memory, latency, energy consumption, and deployability. These limitations are particularly acute in settings requiring repeated evaluation and ablation under constrained hardware environments. This thesis investigates structured pruning and compression strategies for open-source models, focusing on depth pruning, width/channel pruning, dependency-aware correctness, and post-pruning recovery via parameter-efficient finetuning. We systematically examine multiple model families and scales, including Qwen3 (0.6B, 4B, 8B), LLaVA-OneVision-0.5B, Phi-4-Mini, and the specialized Earth observation foundation models TerraMind-Base and Large. The evaluation employs negative log-likelihood for intrinsic importance scoring and perplexity for performance tracking, framing task-level capability drops across standard downstream tracks—Big-Bench Hard (BBH), MMLU-Pro, MATH Hard, and MuSR—using a fixed-prompt forced continuation protocol to guarantee comparative reproducibility.

Our empirical findings reveal four primary architectural insights: (1) depth pruning consistently provides a superior and more stable trade-off between execution latency and accuracy compared to width pruning; (2) width and channel reduction successfully compresses raw memory footprints but offers minimal real-world throughput gains due to GPU kernel efficiency limits and tight dependency constraints; (3) reasoning-intensive benchmarks, particularly complex mathematical sequences, exhibit extreme structural fragility and collapse faster than standard linguistic tasks under compression; and (4) parameter-efficient recovery via low-rank adaptation (LoRA/QLoRA) successfully restores baseline performance configurations at moderate compression ratios, whereas aggressive layer-dropping approaches saturation boundaries requiring continued pretraining. Crucially, these structured pruning profiles transfer predictably to remote sensing backbones; on the TerraMind pipeline, depth-slicing diminishes latency and VRAM allocation while preserving competitive downstream classification performance on the EuroSAT benchmark. Finally, an extensive failure analysis documents environment incompatibilities, rank-deficit gradient flatlines, and multi-spectral channel shape mismatches, underscoring the vital role of tool support and robust serialization guardrails. Ultimately, this work offers a methodologically faithful, failure-aware roadmap for the resource-constrained deployment of modern neural architectures.