Text-to-3D Scene Generation: A Transformer-Driven Framework for Custom Virtual Environments

Abstract

The rapid evolution of generative AI has transformed 3D content creation, yet current text-to-3D pipelines face a fundamental trade-off between inference speed and creative control. Assetretrieval approaches offer rapid scene assembly but remain constrained by predefined asset libraries, while neural rendering approaches achieve photorealistic quality but suffer from high computational costs and non-editable representations. This thesis bridges this gap by proposing an integrated pipeline that combines LLM-driven spatial planning with a transformer-based mesh generator and a hybrid texturing module. At the core of this system is a modified MeshGPT architecture optimized for fast inference, embedded within an agentic workflow that iteratively validates spatial layouts. The pipeline offers a flexible trade-off between speed and fidelity through a dual-mode texturing module, supporting both rapid UV mapping and diffusion-based synthesis. Experimental evaluation demonstrates that the proposed mesh transformer achieves a 29–39% inference speedup and a 32–44% improvement in perceptual quality compared to the baseline MeshGPT model. Endto-end evaluation confirms the system’s ability to generate valid scenes in 4–12 minutes with over 99% semantic precision. Furthermore, a mobility infrastructure case study validates the system’s practical utility, showing that its modular editing capabilities reduce modification time to 56% of the initial generation cost, thereby facilitating real-time participatory design and rapid prototyping.