A new distributed anomaly detection approach for log IDS management based on deep learning

Abstract

Today, with the rapid increase of data, the security of big data has become more important than ever for managers. However, traditional infrastructure systems cannot cope with increasingly big data that is created like an avalanche. In addition, as the existing database systems increase licensing costs per transaction, organizations using information technologies are shifting to free and open source solutions. For this reason, we propose an anomaly attack detection model on Apache Hadoop distributed file system (HDFS), which stands out in open source big data analytics, and Apache Spark, which stands out with its speed performance in analysis to reduce the costs of organizations. This model consists of four stages: the first of which is to store instant data on HDFS in a distributed manner. In the second stage, the log data generated in the network traffic are analyzed by taking the data on Apache Spark and including the log data created by HDFS. In the third stage, the data preprocessing stage and with the CUDA parallel programming in the TensorFlow library, we apply our deep learning (cuDNN) method to the distributed anomaly detection with the computational support of GPUs. In the last stage, the generated alarms are recorded on HDFS again. We conducted comparative experiments with the approach we propose to detect cyberattack anomalies in log data management with the classification methods used in machine learning. The results obtained in these experiments appear to provide a promising gain in performance evaluation metrics compared to the other available methods.