Optimized random forest framework for big data-driven network intrusion detection with SparkML
Abstract
The rapid growth in scale and complexity of modern network traffic has significantly challenged traditional network intrusion detection systems (NIDS). Traditional NIDS struggle to process high-volume, high-velocity data streams while sustaining high detection accuracy, creating critical security gaps in large-scale environments. To address this limitation, this study proposes a scalable, big data-driven NIDS on Apache Spark’s SparkML library. The system was evaluated on the BigFlow-NIDS dataset, contains 66.9 million network flow records with 55 features with 32 attack categories. Six machine learning models; decision tree (DT), logistic regression (LR), random forest (RF), Naive Bayes (NB), linear support vector machine (LSVM), and gradient boosting tree (GBT) were systematically benchmarked for binary (attack versus benign) classification. Feature engineering reduced the features from 51 to the top 20 features. A threshold-optimized RF framework was then proposed to maximize attack F1-score under a strict attack-recall floor of 0.99. Experimental results show that the optimized RF model improved accuracy from 95.99% to 97.31% with a ROC-AUC of 0.9940. At the optimal threshold of 0.07, it achieved a 99.38% attack detection rate, sustained a throughput of 108,465 flows/second, and recorded a false-positive rate of 17.86%, demonstrating its effectiveness as a high-sensitivity first-stage filter.
Keywords
Apache spark; Big data; Cybersecurity; Machine learning; Network intrusion detection
Full Text:
PDFDOI: https://doi.org/10.11591/eei.v15i6.13925
Refbacks
- There are currently no refbacks.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Bulletin of Electrical Engineering and Informatics (BEEI)
ISSN: 2089-3191
,
e-ISSN: 2302-9285
This journal is published by the
Institute of Advanced Engineering and Science (IAES)
.