Distributed system anomaly detection using deep learning-based log analysis

Pengfei Han, Huakang Li, Gang Xue*, Chao Zhang

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

2 Citations (Scopus)

Abstract

Anomaly detection is a key step in ensuring the security and reliability of large-scale distributed systems. Analyzing system logs through artificial intelligence methods can quickly detect anomalies and thus help maintenance personnel to maintain system security. Most of the current works only focus on the temporal or spatial features of distributed system logs, and they cannot sufficiently extract the global features of distributed system logs to achieve a good correct rate of anomaly detection. To further address the shortcomings of existing methods, this paper proposes a deep learning model with global spatiotemporal features to detect the presence of anomalies in distributed system logs. First, we extract semi-structured log events from log templates and model them as natural language. In addition, we focus on the temporal characteristics of logs using the bidirectional long short-term memory network and the spatial invocation characteristics of logs using the Transformer. Extensive experimental evaluations show the advantages of our proposed model for distributed system log anomaly detection tasks. The optimal F1-Score on three open-source datasets and our own collected distributed system datasets reach 98.04%, 94.34%, 88.16%, and 97.40%, respectively.

Original languageEnglish
Pages (from-to)433-455
Number of pages23
JournalComputational Intelligence
Volume39
Issue number3
DOIs
Publication statusPublished - Jun 2023

Keywords

  • deep learning
  • distributed system
  • spatiotemporal feature extraction
  • system anomaly detection
  • system logs analysis

Fingerprint

Dive into the research topics of 'Distributed system anomaly detection using deep learning-based log analysis'. Together they form a unique fingerprint.

Cite this