An Approach to Server Log Analysis for Abnormal Behaviour Detection

Date
2021-03-19
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract

As the server logs increase in size, it becomes difficult for human experts to manually examine error log messages, analyze the anomalies, and because of the high volume of log data. If the error message is rare or of low frequency, the system does not categorize it as important and get ignored that may leads to fatal errors. Server log analytics has proven to be optimum for active strategies and excellent performances of the system like the preventive maintenance or complete shut-down. Improvements in analytical strategies are necessary for data analysts in handling the large system. For this analytical process to yield good results, the input data need to be of good quality; therefore, research focuses on cleaning and pre-processing techniques. This research proposes the consecutive logical steps to enhance the analysis of log messages. First, we purpose extracting sequences and patterns from the logs by optimizing window sizes without losing valuable information and combining them with forecasting techniques for predictive analytics. Second, we improve topic modelling for low frequency messages through text analysis and language modelling. The resulting proof of concept is not just visualizing the log data; instead, it provides insight into the logs through topics from the error messages. The experiments illustrate the effectiveness of the proposed steps and the approach for error log analysis.

Description
Keywords
LDA - Latent Dirichlet Allocation, LSTM - Long Short-Term Memory, RMSE - Root Mean Square Error, ARIMA - Auto-Regressive Integrated Moving Average, EAI - Enterprise Application Integration, PCA - Principal Component Analysis, EMS - Enterprise Management System, TF-IDF - Term frequency-inverse document frequency
Citation
Suman, R. (2021). An Approach to Server Log Analysis for Abnormal Behaviour Detection (Master's thesis, University of Calgary, Calgary, Canada). Retrieved from https://prism.ucalgary.ca.