Semantically Formalized Logging and Advanced Analytics for Enhanced Monitoring and Management of Large-scale Applications

Date
2015-05-01
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Monitoring and management of large scale applications has always been a complex task, especially because execution workflow and log (outcome from real-time execution) are modeled in a syntactic manner. This information is quite limited and requires manual interpretation, and hence makes the monitoring and management process slow, cumbersome and hard. We propose our solution by semantically (i.e., highly structured, formalized and expressive) modeling of the execution workflow and logs, and then we use Social Network Analysis, Classification, Clustering and Association Rule Mining based approaches to process the semantic information, to help in automating the monitoring and management process. There have been several related efforts, but these solutions still could not achieve the goal effectively as described in this thesis. Two main reasons are: (1) they do not consider the correlation between the expressive modeling of execution workflow and logs, (2) the methods for processing (for monitoring) execution workflow and log methods are quite weak and limited. To overcome the weaknesses of the approaches described in the literature, our proposed solution helps in automating the process of monitoring and management of large-scale distributed applications. We have designed and developed our unique hybrid approach of partially using formal semantics for logs description, as well as social network analysis and data mining tasks to be able to automatically interpret and process the highly structured information from the logs generated during the execution; this way our approach combines the best characteristics of both. Therefore, it helps in improving the automated monitoring and management of applications. Since the logs are generated based on the execution workflow, our solution takes into account the correlation among both. Further the impact and usefulness of our solution have been demonstrated by applying it on real-life application scenario which was defined in consultation with our research collaborators from the industry. Our recent research publications and collaboration with industry have already shown promising results.
Description
Keywords
Computer Science
Citation
Shafiq, M. O. (2015). Semantically Formalized Logging and Advanced Analytics for Enhanced Monitoring and Management of Large-scale Applications (Doctoral thesis, University of Calgary, Calgary, Canada). Retrieved from https://prism.ucalgary.ca. doi:10.11575/PRISM/27748