DLR-Logo -> http://www.dlr.de
DLR Portal Home | Imprint | Privacy Policy | Contact | Deutsch
Fontsize: [-] Text [+]

Combining Supervised and Unsupervised Learning to Detect and Semantically Aggregate Crisis-Related Twitter Content

Bongard, Jan and Kersten, Jens (2021) Combining Supervised and Unsupervised Learning to Detect and Semantically Aggregate Crisis-Related Twitter Content. EGU 2021, 19.-30. April 2021, Wien, Österreich (online).

[img] PDF

Official URL: https://meetingorganizer.copernicus.org/EGU21/EGU21-8637.html


The Twitter Stream API offers the possibility to develop (near) real-time methods and applications to detect and monitor impacts of crisis events and their changes over time. As demonstrated by various related research, the content of individual tweets or even entire thematic trends can be utilized to support disaster management, fill information gaps and augment results of satellite-based workflows as well as to extend and improve disaster management databases. Considering the sheer volume of incoming tweets, it is necessary to automatically identify the small number of crisis-relevant tweets and present them in a manageable way. Current approaches for identifying crisis-related content focus on the use of supervised models that decide on the relevance of each tweet individually. Although supervised models can efficiently process the high number of incoming tweets, they have to be extensively pre-trained. Furthermore, the models do not capture the history of already processed messages. During a crisis, various and unique sub-events can occur that are likely to be not covered by the respective supervised model and its training data. Unsupervised learning offers both, to take into account tweets from the past, and a higher adaptive capability, which in turn allows a customization to the specific needs of different disasters. From a practical point of view, drawbacks of unsupervised methods are the higher computational costs and the potential need of user interaction for result interpretation. In order to enhance the limited generalization capabilities of pre-trained models as well as to speed up and guide unsupervised learning, we propose a combination of both concepts. A successive clustering of incoming tweets allows to semantically aggregate the stream data, whereas pre-trained models allow to identify potentially crisis-relevant clusters. Besides the identification of potentially crisis-related content based on semantically aggregated clusters, this approach offers a sound foundation for visualizations, and further related tasks, like event detection as well as the extraction of detailed information about the temporal or spatial development of events. Our work focuses on analyzing the entire freely available Twitter stream by combining an interval-based semantic clustering with an supervised machine learning model for identifying crisis-related messages. The stream is divided into intervals, e.g. of one hour, and each tweet is projected into a numerical vector by using state-of-the-art sentence embeddings. The embeddings are then grouped by a parametric Chinese Restaurant Process clustering. At the end of each interval, a pre-trained feed-forward neural network decides whether a cluster contains crisis-related tweets. With a further developed concept of cluster chains and central centroids, crisis-related clusters of different intervals can be linked in a topic- and even subtopic-related manner. Initial results show that the hybrid approach can significantly improve the results of pre-trained supervised methods. This is especially true for categories in which the supervised model could not be sufficiently pre-trained due to missing labels. In addition, the semantic clustering of tweets offers a flexible and customizable procedure, resulting in a practical summary of topic-specific stream content.

Item URL in elib:https://elib.dlr.de/143776/
Document Type:Conference or Workshop Item (Speech, Poster)
Title:Combining Supervised and Unsupervised Learning to Detect and Semantically Aggregate Crisis-Related Twitter Content
AuthorsInstitution or Email of AuthorsAuthor's ORCID iD
Bongard, Janjan.bongard (at) dlr.deUNSPECIFIED
Kersten, Jensjens.kersten (at) dlr.dehttps://orcid.org/0000-0002-4735-7360
Date:April 2021
Refereed publication:Yes
Open Access:Yes
Gold Open Access:No
In ISI Web of Science:No
Keywords:Twitter, Natural Disasters, Supervised and Unsupervised Learning, Information Overload Reduction
Event Title:EGU 2021
Event Location:Wien, Österreich (online)
Event Type:international Conference
Event Dates:19.-30. April 2021
Organizer:European Geoscience Union
HGF - Research field:Aeronautics, Space and Transport
HGF - Program:Space
HGF - Program Themes:Space System Technology
DLR - Research area:Raumfahrt
DLR - Program:R SY - Space System Technology
DLR - Research theme (Project):R - Exploration of citizen science methods, R - QS-Project_04 Big-Data-Plattform
Location: Jena
Institutes and Institutions:Institute of Data Science > Citizen Science
Deposited By: Kersten, Dr.-Ing. Jens
Deposited On:18 Oct 2021 08:24
Last Modified:30 Nov 2021 14:29

Repository Staff Only: item control page

Help & Contact
electronic library is running on EPrints 3.3.12
Copyright © 2008-2017 German Aerospace Center (DLR). All rights reserved.