Optimizing Decision Tree and Random Forest with Grid Search and SMOTE for Malware Classification on IoT Network Traffic
DOI:
https://doi.org/10.30871/jaic.v9i5.10542Keywords:
Decision Tree, Grid Search, Malware Classification, SMOTE, IoTAbstract
The rapid growth of the Internet of Things (IoT) has increased the risk of malware attacks, posing serious threats especially to micro, small, and medium enterprises (MSMEs) that often lack sufficient cybersecurity resources. This study aims to optimize Decision Tree (DT) and Random Forest (RF) classifiers using Grid Search, while addressing the class imbalance problem through the Synthetic Minority Oversampling Technique (SMOTE). The Security Attacks Malware IoT Networks dataset with five classes (Benign, Malware, DDoS, Brute Force, Scanning) was used and divided into training and testing sets with stratified 80:20 split. Experimental results show that DT achieved 67.3% accuracy with a macro F1-score of 42.9%, while RF achieved 70.7% accuracy but a very low macro F1-score of 21.4%, indicating bias toward the majority class despite balancing. Boosting methods provided stronger baselines, with XGBoost reaching 87.0% accuracy and 66.7% F1-score, while LightGBM achieved 85.6% accuracy and 64.4% F1-score. ROC curves and confusion matrices confirmed that boosting methods were more balanced in recognizing minority classes. In terms of efficiency, DT required the shortest training time (8 seconds), while LightGBM provided the best trade-off between accuracy and computational cost (26 seconds). Paired t-tests further confirmed that performance differences between DT and RF were not significant, while boosting methods significantly outperformed RF. Overall, optimizing DT and RF with Grid Search and SMOTE enhances their performance, but boosting methods remain more robust for malware detection in IoT traffic. These findings provide practical insights for MSMEs in balancing accuracy and efficiency when deploying intrusion detection systems.
Downloads
References
[1] M. A. S. Arifin, R. Kurniawan, and A. Wicaksono, “Deteksi Aktivitas Malware pada Internet of Things menggunakan Decision Tree dan Random Forest,” Jurnal KLIK: Kajian Ilmiah Informatika dan Komputer, vol. 5, no. 2, pp. 123–132, Jun. 2024.
[2] A. F. A. Arizal, M. Md-Arshad, A. Abdul-Samad, M. Md Sirat, and S. H. Othman, “Performance Comparative Study on Zero Day Malware Detection Using XGBoost and Random Forest Classifiers,” International Journal of Innovative Computing (IJIC), vol. 14, no. 2, pp. 45–52, Dec. 2024.
[3] “Advanced Malware Detection Framework using Random Forest,” International Journal of Engineering Research & Technology (IJERT), vol. 14, no. 4, pp. 112–118, Apr. 2025.
[4] “Malware Detection Using a Random Forest Method Trained on a Balanced Synthetic Dataset,” Science, Engineering and Technology Journal (SET), vol. 3, no. 1, pp. 65–72, Jan. 2025.
[5] K. A. Ahmad, “Dealing with Imbalanced Classes in Bot-IoT Dataset Using Oversampling Techniques,” 2024.
[6] C. Harake, “Optimizing Android Program Malware Classification using GridSearchCV,” in Proceedings of ICCCNT, Jul. 2024.
[7] F. Alharbi, “A Comparative Study of SMOTE and ADASYN for Multiclass Classification of IoT Anomalies,” International Journal on Information Technologies and Security (IJITS), vol. 17, no. 2, pp. 15–26, Feb. 2025.
[8] I. N. Firdaus, A. Prasetyo, and A. Subekti, “Malware Analysis and Classification Using Grid Search Optimization,” in Proceedings of IEEE ICCCNT, Jul. 2024.
[9] H. Nugroho, S. Prabowo, and M. A. Rahman, “Comparison of Multiple Feature Selection Techniques for IoT Attacks Detection,” in Proceedings of the 2024 International Conference on Availability, Reliability and Security (ARES), Aug. 2024, pp. 112–119.
[10] D. Santoso, A. Widodo, and L. Kurniawan, “Optimal Feature Set Analysis with RFE & XGBoost for IoT Malware Detection,” in Proceedings of the 2024 International Conference on Information and Communication Technology (ICOIACT), May 2024, pp. 78–85.
[11] S. Pramanick, R. Sharma, and B. Patel, “Enhanced Intrusion Detection Using BBA and SMOTE-ENN for Imbalanced Data in Cybersecurity,” SN Comput Sci, vol. 5, no. 2, pp. 223–234, Mar. 2024.
[12] S. Gupta and A. Sharma, “Malware Detection in Internet of Things Using Machine Learning and Boosting Models,” in Lecture Notes in Networks and Systems, Springer, 2025, pp. 155–168.
[13] A. Alve, P. Kumar, and J. Singh, “Smart IoT Security: Lightweight Machine Learning Techniques for Intrusion Detection,” 2025.
[14] L. Zhang, H. Chen, and Q. Li, “A Novel Autoencoder-Based GA Optimized XGBoost Model for IoMT Malware Classification,” Expert Syst Appl, vol. 238, pp. 121–135, Dec. 2023.
[15] A. Akif, M. Khan, and F. Aziz, “Hybrid Machine Learning Models for Intrusion Detection in IoT,” 2025.
[16] M. F. Khan and T. Hussain, “IoT Security Enhancement Using XGBoost and Random Forest,” Journal of Network and Computer Applications, vol. 215, pp. 103–115, Nov. 2023.
[17] M. Imani, A. Beikmohammadi, and H. R. Arabnia, “Comprehensive Analysis of Random Forest and XGBoost Performance with SMOTE, ADASYN, and GNUS Under Varying Imbalance Levels,” Technologies (Basel), vol. 13, no. 3, pp. 88–102, Feb. 2025.
[18] A. Rupanetti and N. Kaabouch, “Leveraging Machine Learning for Botnet Attack Detection in Edge-Computing Assisted IoT Networks,” 2025.
[19] J. Lee, H. Park, and Y. Kim, “Enhancing IoT Security: Effective Botnet Attack Detection through Random Forest and XGBoost,” Procedia Comput Sci, vol. 225, pp. 320–328, 2024.
[20] R. Zaidi, H. Ali, and S. Khan, “Enhancing Android Malware Detection with XGBoost and Convolutional Neural Networks,” Computers, Materials & Continua (CMC), vol. 84, no. 2, pp. 223–240, Feb. 2025.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Muhammad Nurus Siroj, Akhmad Khanif Zyen, Gentur Wahyu Nyipto Wibowo
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License (Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) ) that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
