Gaining more from Tweets: Knowledge, Actions, and Requirements Elicitation by a Hybrid Method of Natural Language Processing
Date
2022-11-28
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
In Decision Support Systems (DSS) one of the most important types of input data for supporting the final decisions is textual data. However, in this era, there are significantly more volumes of textual content generated from various sources than could ever be processed, analyzed, and further used for decision-making. Moreover, most of the textual data sources produce unstructured content with no unified scheme, thus, making it even more difficult to automatically mine them for gold nuggets of information supporting pivotal data-supported decisions. One of the most widely used sources of textual content for mining public opinions and extracting subject-specific requirements is Twitter in which people publish over 500 million tweets on a daily basis. While this makes Twitter a great source of knowledge for public requirements and trends, Tweets are very difficult to be processed for requirements and knowledge elicitation since they are unstructured, written in conversational and imperfect grammar, and often non-informative for supporting decisions without being properly processed and analyzed. Here in the course of this thesis, a semi-automatic methodology pipeline named DeKoReMi (Deep Knowledge and Requirements Miner) is proposed that employs state-of-the-art Deep Learning and Natural Language Processing techniques. The goal is to elicit the hidden and integral requirements, and more importantly, the necessary related knowledge and description to explain the extracted requirements from extremely large corpses of textual unstructured content (specifically Tweets). The retrieved information will further be used as the basis of pivotal data-supported decisions in a wide variety of Decision Support Systems. In this research, DeKoReMi has been developed and proved to be effective using the "Action Research" methodology over the course of three real-life industrial-academic projects conducted in collaboration with the City of Calgary, and Suncor Energy having processed over 10 million tweets combined.
Description
Keywords
Requirements Engineering, Natural Language Processing, Decision Support
Citation
Masahati, M. N. (2022). Gaining more from tweets: knowledge, actions, and requirements elicitation by a hybrid method of natural language processing (Master's thesis, University of Calgary, Calgary, Canada). Retrieved from https://prism.ucalgary.ca.