Context based Text-generation using LSTM networks

Kurzfassung

Long short-term memory(LSTM) units on sequence-based models are being used in translation, question-answering systems, classification tasks due to their capability of learning long-term dependencies. Text generation models, an application of LSTM models are recently popular due to their impressive results. LSTM models applied to natural languages are great in learning grammatically stable syntaxes. But the downside is, the system has no basic idea of the context and it generates text given a set of input words irrespective of the use-case. The proposed system trains the model to generate words given input words along with a context vector. Depending upon the use-case, the context vector is derived for a sentence or for a paragraph. A context vector could be a topic (from topic models) or the word having highest tf-idf weight in the sentence or a vector computed from word clusters. Thus, during the training phase, the same context vector is applied across the whole sentence for each window to predict successive words. Due to this structure, the model learns the relation between the context vector and the target word. During prediction, the user could provide keywords or topics to guide the system to generate words around a certain context. Apart from the syntactic structure in the current text-generation models, this proposed model will also provide semantic consistency. Based on the nature of computing context vectors, the model has been tried out with two variations (tf-idf and word clusters). The proposed system could be applied in question-answering systems to respond with a relevant topic. Also in Text-generation of stories with defined hints. The results should be evaluated manually on how semantically closer the text is generated given the context words.