Detecting anomalies in MQTT/MQTT-SN traffic using intelligent learning models

Abstract

The widespread adoption of the internet of things (IoT) has heightened demand for secure, efficient communication across constrained devices. Lightweight protocols such as message queuing telemetry transport (MQTT) and its variant MQTT-sensor networks (SN) are widely used for IoT messaging but lack intrinsic security mechanisms, leaving them vulnerable to denial-of-service, spoofing, and injection attacks. This study presents a machine learning (ML)-based anomaly detection framework designed to enhance the security of MQTT and MQTT-SN traffic. We emulate realistic IoT environments to generate both benign and malicious traffic, extracting protocol-specific features such as packet length, topic length, quality of service (QoS) levels, and publish frequency. Three supervised models—random forest (RF), eXtreme gradient boosting (XGBoost), and long short-term memory (LSTM)—were trained and evaluated using cross-validation and statistical performance metrics. Experimental findings demonstrate that XGBoost achieved the best overall results, with 97.4% accuracy, 95.9% F1-score, and low false-positive and false-negative rates. Furthermore, the framework was successfully deployed on edge devices such as Raspberry Pi Zero W and ESP32, confirming its real-time feasibility and efficiency. The proposed approach highlights the potential of intelligent learning models to deliver lightweight, deployable, and effective intrusion detection for IoT systems utilizing MQTT and MQTT-SN communication protocols.