Utilization of Natural Language Processing for Extracting Smart Cities Requirements from Large Social Media Text
Date
2024-05-14
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Major organizations such as urban centers worldwide face challenges from rapid population growth and evolving demands, requiring innovative approaches to stay responsive to residents' needs. This challenge is exemplified by the city of Calgary, where an automated system for aggregating and categorizing resident feedback could improve city planning. What people find important and useful can be seen in the articles they post on social media. One method for determining the performance of urban services and assets for citizens is paying attention to these data generated by the residents. In this regard, we need to examine datasets wherein writing is the primary form of citizen engagement (direct messages, requests, comments, complaints, etc.). To interpret this data, it is necessary to use appropriate tools and techniques for data processing and analysis of large volumes of unstructured text. Some of the most effective tools used by researchers nowadays falls into the scope of computational linguistics, specifically Natural language processing (NLP). Furthermore, Twitter is one of the primary platforms where individuals freely voice their opinions and concerns. In this study, we develop an automated workflow that can scrape, classify, and display tweets in a simplistic view. With the help of this system, local officials will be able to speed up the decision-making process when considering citizens' current problems. Following our research question, we look into the optimal scraping criteria, explore a variety of methods for topic and emotions analysis, and validate these methods both using automatic evaluation and manual assessment. As a result, we are able to identify issues related to city development, senior citizens, taxes, and unemployment using our best performing models (BERTopic for topic modeling and few-shot learning using Setfit for emotion analysis.) Afterward, we collect city employees' opinion regarding our research to determine the usefulness and applicability of this approach. Overall, we demonstrate how delving into these analyses can complement the current systems in place for urban planning.
Description
Keywords
Citation
Mirshafiee Khoozani, M. S. (2024). Utilization of natural language processing for extracting smart cities requirements from large social media text (Master's thesis, University of Calgary, Calgary, Canada). Retrieved from https://prism.ucalgary.ca.