A Robust Hybrid Approach for Malware Detection: Leveraging CNN and LSTM for Encrypted Traffic Analysis

Arif Mudi Priyatno; Yunia Ningsih; Arnes Yuli Vandika; Muhammadong Muhammadong

doi:10.69693/jesa.v1i2.10

Authors

Arif Mudi Priyatno University of Pahlawan Tuanku Tambusai
Yunia Ningsih Department of informatics engineering, Faculty of industry, Universitas Trisakti, Jakarta, Indonesia
Arnes Yuli Vandika Universitas Bandar Lampung
Muhammadong Universitas Negeri Makassar

DOI:

https://doi.org/10.69693/jesa.v1i2.10

Keywords:

Encrypted Traffic Classification, Hybrid CNN-LSTM Detection, Deep Learning Malware Analysis

Abstract

The rapid growth in Internet usage and advancements in network technologies have escalated the risk of network attacks. As the adoption of encryption protocols increases, so does the difficulty in identifying malware within encrypted traffic. Malware represents a significant danger in cyberspace, as it compromises personal data and harms computer systems. Network attacks involve unauthorized access to networks, often aiming to disrupt or damage them, with potentially severe consequences. To counter these threats, researchers, developers, and security experts are constantly innovating new malware detection techniques. Recently, deep learning has gained traction in network security and intrusion detection systems (IDSs), with models such as Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) showing promise in detecting malicious traffic. Despite these advancements, extracting relevant features from diverse malware types remains a challenge. Current solutions demand substantial computational resources and are often inefficient for large datasets. Additionally, existing image-based feature extraction methods consume significant resources. This study tackles these issues by employing a 1D CNN alongside LSTM for the detection and classification of encrypted malicious traffic. Using the Malware Analysis benchmark dataset, which consists of 42,797 malware and 1,079 goodware API call sequences, the proposed model achieved an accuracy of 99.2%, surpassing other state-of-the-art models

Downloads

Download data is not yet available.

References

S. Soderi, D. Masti, and Y. Z. Lun, “Railway Cyber-Security in the Era of Interconnected Systems: A Survey,” IEEE Trans. Intell. Transp. Syst., vol. 24, no. 7, pp. 6764–6779, Jul. 2023, doi: 10.1109/TITS.2023.3254442.

U. Tariq, I. Ahmed, A. K. Bashir, and K. Shaukat, “A Critical Cybersecurity Analysis and Future Research Directions for the Internet of Things: A Comprehensive Review,” Sensors, vol. 23, no. 8, p. 4117, Apr. 2023, doi: 10.3390/s23084117.

J. Saleem, R. Islam, and M. Z. Islam, “Darknet Traffic Analysis: A Systematic Literature Review,” IEEE Access, vol. 12, pp. 42423–42452, 2024, doi: 10.1109/ACCESS.2024.3373769.

A. K. Tyagi and S. R. Addula, “Artificial Intelligence for Malware Analysis,” in Artificial Intelligence‐Enabled Digital Twin for Smart Manufacturing, Wiley, 2024, pp. 359–390. doi: 10.1002/9781394303601.ch17.

V. Vasani, A. K. Bairwa, S. Joshi, A. Pljonkin, M. Kaur, and M. Amoon, “Comprehensive Analysis of Advanced Techniques and Vital Tools for Detecting Malware Intrusion,” Electronics, vol. 12, no. 20, p. 4299, Oct. 2023, doi: 10.3390/electronics12204299.

J. Ferdous, R. Islam, A. Mahboubi, and M. Z. Islam, “A Review of State-of-the-Art Malware Attack Trends and Defense Mechanisms,” IEEE Access, vol. 11, pp. 121118–121141, 2023, doi: 10.1109/ACCESS.2023.3328351.

G. M. and S. C. Sethuraman, “A comprehensive survey on deep learning based malware detection techniques,” Comput. Sci. Rev., vol. 47, p. 100529, Feb. 2023, doi: 10.1016/j.cosrev.2022.100529.

D. Demirci, N. Sahin, M. Sirlancis, and C. Acarturk, “Static Malware Detection Using Stacked BiLSTM and GPT-2,” IEEE Access, vol. 10, pp. 58488–58502, 2022, doi: 10.1109/ACCESS.2022.3179384.

M. Aljabri et al., “Intelligent Techniques for Detecting Network Attacks: Review and Research Directions,” Sensors, vol. 21, no. 21, p. 7070, Oct. 2021, doi: 10.3390/s21217070.

M. J. Awan et al., “Image-Based Malware Classification Using VGG19 Network and Spatial Convolutional Attention,” Electronics, vol. 10, no. 19, p. 2444, Oct. 2021, doi: 10.3390/electronics10192444.

A. M. Alnajim, S. Habib, M. Islam, R. Albelaihi, and A. Alabdulatif, “Mitigating the Risks of Malware Attacks with Deep Learning Techniques,” Electronics, vol. 12, no. 14, p. 3166, Jul. 2023, doi: 10.3390/electronics12143166.

M. N. Al-Andoli, K. S. Sim, S. C. Tan, P. Y. Goh, and C. P. Lim, “An Ensemble-Based Parallel Deep Learning Classifier With PSO-BP Optimization for Malware Detection,” IEEE Access, vol. 11, pp. 76330–76346, 2023, doi: 10.1109/ACCESS.2023.3296789.

A. Mallik, A. Khetarpal, and S. Kumar, “ConRec: malware classification using convolutional recurrence,” J. Comput. Virol. Hacking Tech., vol. 18, no. 4, pp. 297–313, Feb. 2022, doi: 10.1007/s11416-022-00416-3.

E. U. H. Qazi, M. H. Faheem, and T. Zia, “HDLNIDS: Hybrid Deep-Learning-Based Network Intrusion Detection System,” Appl. Sci., vol. 13, no. 8, p. 4921, Apr. 2023, doi: 10.3390/app13084921.

A. I. A. Alzahrani, M. Ayadi, M. M. Asiri, A. Al-Rasheed, and A. Ksibi, “Detecting the Presence of Malware and Identifying the Type of Cyber Attack Using Deep Learning and VGG-16 Techniques,” Electronics, vol. 11, no. 22, p. 3665, Nov. 2022, doi: 10.3390/electronics11223665.

A. R. Nasser, A. M. Hasan, and A. J. Humaidi, “DL-AMDet: Deep learning-based malware detector for android,” Intell. Syst. with Appl., vol. 21, p. 200318, Mar. 2024, doi: 10.1016/j.iswa.2023.200318.

M. Schmitt, “Securing the digital world: Protecting smart infrastructures and digital industries with artificial intelligence (AI)-enabled malware and intrusion detection,” J. Ind. Inf. Integr., vol. 36, p. 100520, Dec. 2023, doi: 10.1016/j.jii.2023.100520.

T. Muralidharan, A. Cohen, N. Gerson, and N. Nissim, “File Packing from the Malware Perspective: Techniques, Analysis Approaches, and Directions for Enhancements,” ACM Comput. Surv., vol. 55, no. 5, pp. 1–45, May 2023, doi: 10.1145/3530810.

Y. Liu, C. Tantithamthavorn, L. Li, and Y. Liu, “Deep Learning for Android Malware Defenses: A Systematic Literature Review,” ACM Comput. Surv., vol. 55, no. 8, pp. 1–36, Aug. 2023, doi: 10.1145/3544968.

S. Sanjaya, A. M. Priyatno, F. Yanto, and I. Afrianty, “Klasifikasi Diabetik Retinopati Menggunakan Wavelet Haar dan Backpropagation Neural Network,” in Seminar Nasional Teknologi Informasi Komunikasi dan Industri (SNTIKI-10), 2018, pp. 77–84.

A. M. Priyatno and F. I. Firmananda, “N-Gram Feature for Comparison of Machine Learning Methods on Sentiment in Financial News Headlines,” RIGGS J. Artif. Intell. Digit. Bus., vol. 1, no. 1, pp. 01–06, Jul. 2022, doi: 10.31004/riggs.v1i1.4.

A. M. Priyatno, “Spammer Detection Based on Account, Tweet, and Community Activity on Twitter,” J. Ilmu Komput. dan Inf., vol. 13, no. 2, pp. 97–107, Jul. 2020, doi: 10.21609/jiki.v13i2.871.

A. M. Priyatno and L. Ningsih, “TF - IDF Weighting to Detect Spammer Accounts on Twitter based on Tweets and Retweet Representation of Tweets,” Sist. J. Sist. Inf., vol. 11, no. 3, pp. 614–622, 2022, [Online]. Available: http://sistemasi.ftik.unisi.ac.id/index.php/stmsi/issue/view/46

M. R. A. Prasetya and A. M. Priyatno, “Dice Similarity and TF-IDF for New Student Admissions Chatbot,” RIGGS J. Artif. Intell. Digit. Bus., vol. 1, no. 1, pp. 13–18, Jul. 2022, doi: 10.31004/riggs.v1i1.5.