Optimized random forest framework for big data-driven network intrusion detection with SparkML

Mohammed Borhan Uddin, Mohammad Shamsul Arefin, Jannatul Ferdaus, Zaid Rehman

Abstract


The rapid growth in scale and complexity of modern network traffic has significantly challenged traditional network intrusion detection systems (NIDS). Traditional NIDS struggle to process high-volume, high-velocity data streams while sustaining high detection accuracy, creating critical security gaps in large-scale environments. To address this limitation, this study proposes a scalable, big data-driven NIDS on Apache Spark’s SparkML library. The system was evaluated on the BigFlow-NIDS dataset, contains 66.9 million network flow records with 55 features with 32 attack categories. Six machine learning models; decision tree (DT), logistic regression (LR), random forest (RF), Naive Bayes (NB), linear support vector machine (LSVM), and gradient boosting tree (GBT) were systematically benchmarked for binary (attack versus benign) classification. Feature engineering reduced the features from 51 to the top 20 features. A threshold-optimized RF framework was then proposed to maximize attack F1-score under a strict attack-recall floor of 0.99. Experimental results show that the optimized RF model improved accuracy from 95.99% to 97.31% with a ROC-AUC of 0.9940. At the optimal threshold of 0.07, it achieved a 99.38% attack detection rate, sustained a throughput of 108,465 flows/second, and recorded a false-positive rate of 17.86%, demonstrating its effectiveness as a high-sensitivity first-stage filter.

Keywords


Apache spark; Big data; Cybersecurity; Machine learning; Network intrusion detection

Full Text:

PDF


DOI: https://doi.org/10.11591/eei.v15i6.13925

Refbacks

  • There are currently no refbacks.


Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Bulletin of EEI Statistics

Bulletin of Electrical Engineering and Informatics (BEEI)
ISSN: 2089-3191 , e-ISSN: 2302-9285
This journal is published by the Institute of Advanced Engineering and Science (IAES) .