Semantic Enrichment of Synthetic Earth Observation Data: Concept and Technical Implementation

Kurzfassung

With the advent of large language models (LLMs) and LLM based agents new opportunities arise for Earth observation (EO) data analysis. Central to leveraging the capabilities of LLMs is the development of semantic interfaces that enrich EO data, data products, and processing pipelines. This presentation explores the integration of semantic descriptions into synthetically generated remote sensing scenes, complementing conventional computer vision labels such as segmentation masks and bounding boxes. The goal is to introduce a semantic interface that facilitates machine learning-ready EO datasets and workflows. The applied synthetic data generation process employs an object-oriented approach, assembling individual object instances into coherent and meaningful remote sensing scenes. A key concept is the graph-based representation of scene elements and their interrelations, which are encoded as explicit property-graphs for each synthetic scene. These graphs not only guide the construction of the scene but also serve as the foundation for generating a detailed semantic description of it. The presentation will emphasize the methodology for deriving semantic descriptions from these property-graphs, including mechanisms for adjusting levels of detail to suit various applications. In addition to the scene-level description, the aggregated key-value pairs from the property graphs can be compiled into a comprehensive metadata database. The database’s schema is another way of communicating the property-graph’s structure and enables efficient querying and summarization of the entire datasets characteristics. By leveraging this structured metadata, users, including LLM-agents, are able to perform queries and select specific samples based on desired metadata configurations. This capability enables the construction of tailored training datasets, supporting advanced experimental setups and facilitating more targeted model development and evaluation processes. By formalizing a structured approach to semantic enrichment, this work enables LLMs to interpret detailed metadata associated with machine learning-ready EO datasets. Moreover, the proposed method supports advanced applications by providing rich semantic descriptions of remote sensing scenes, such as training models capable of describing EO scene content (image-to-text) and generating synthetic training pairs for text-to-image tasks.