Classification of Incident-related Tweets: Tackling Imbalanced Training Data using Hybrid CNNs and Translation-based Data Augmentation

Abstract

In this paper, we present our four approaches submitted to the 2018 Text REtrieval Conference (TREC) Incident Streams (IS) track. One of the main challenges in this track is the lack of training data for certain classes defined in the ontology. We therefore took measures to expand the provided data; in a first step, additional tweets were manually selected from CrisisLexT26 and EMTerms for all underrepresented classes ensuring a minimum number of 50 tweets per class. Using this expanded data, we trained three models. The first is a baseline model that uses a logistic regression classifier on word statistics. The second model is a state-of-the-art CNN which considers different frame widths on pre-trained word embeddings. This model was then extended with two identical CNN branches trained on the CrisisLexT26 and CrisisNLP data sets, and a posterior fusion network (third approach). Since all of these models still suffered from a lack of training data, more training examples were generated through a data augmentation technique using automatic round-trip translation. The fourth presented approach is identical to the third one, but was trained on this augmented data set. Finally, we describe our importance ranking procedure for tweets. Our method was implemented by weighting the average importance of the detected class and the tweet's relevance obtained with a classifier trained on the CrisisLexT26 data set.