Comparative Analysis of Machine Learning Algorithms with SMOTE for Imbalanced Sentiment Classification of IndiHome on Platform X

Rizky Adisaputra; Muhammad Arifin; Soni Adiyono

doi:10.30871/jaic.v10i4.13271

Authors

Rizky Adisaputra Universitas Muria Kudus
Muhammad Arifin Universitas Muria Kudus
Soni Adiyono Universitas Muria Kudus

DOI:

https://doi.org/10.30871/jaic.v10i4.13271

Keywords:

Imbalanced Data, Machine Learning, Sentiment Analysis, SMOTE, TF-IDF

Abstract

Sentiment analysis of IndiHome users on social media X faces a severe class imbalance, with negative tweets dominating 88.26% of the dataset. This study compares four machine learning algorithms, Support Vector Machine (SVM), Naive Bayes, Decision Tree, and Random Forest, for sentiment classification using SMOTE to address the imbalance. Initially, 20,001 Indonesian tweets were scraped using Tweet Harvest with the keyword "indihome". After duplicate removal and preprocessing, 7,199 tweets were retained. Each tweet was manually annotated into positive, negative, and neutral categories. TF-IDF was applied for feature extraction, and Stratified 5-Fold Cross Validation was used for evaluation. Algorithms were tested under two conditions: without and with SMOTE. Before SMOTE, SVM achieved the highest accuracy (94.55%) and F1-score (94.05%). After SMOTE, Random Forest outperformed others with 94.14% accuracy and 93.83% F1-score, as the only algorithm showing consistent improvement across all metrics, including balanced accuracy and MCC. Although Wilcoxon tests showed no statistically significant differences between algorithms, Random Forest demonstrated the most stable and consistent performance. These findings confirm that Random Forest with SMOTE is the most effective strategy for imbalanced sentiment classification in this context.

Downloads

Download data is not yet available.

References

[1] Ardiyansah and Parjito, “Perbandingan Metode Naïve Bayes dan Support Vector Machine Dalam Analisis Sentimen Terhadap Tokoh Publik,” KLIK: Kajian Ilmiah Informatika dan Komputer, vol. 4, no. 6, pp. 2813–2821, Jun. 2024.

[2] Diana Puspitasari and Tata Sutabri, “Analisis Sentimen Berdasarkan pada Twitter (X) terhadap Layanan Indihome Menggunakan Algoritma Support Vector Machine (SVM),” JUMINTAL: Jurnal Manajemen Informatika dan Bisnis Digital, vol. 3, no. 2, 2024, doi: 10.55123/jumintal.v3i2.4449.

[3] H. H. Limbong and Norhikmah, “Optimasi Analisis Sentimen Ulasan Aplikasi Amikom One Menggunakan SMOTE pada Algoritma Artificial Neural Network,” Sistematis : Jurnal Sistem Informasi, vol. 13, no. 5, 2024.

[4] D. Kurniawan and M. N. D. Satria, “Analisis Sentimen Opini Publik Tentang Gempa Megathrust di Indonesia Menggunakan Metode Support Vector Machine dan Naïve Bayes,” Building of Informatics, Technology and Science (BITS), vol. 6, no. 3, 2024, doi: 10.47065/bits.v6i3.6213.

[5] S. Adiyono and M. Arifin, “AI and Digital Transformation Trends: A Systematic Review with Multi-Criteria Analysis,” Journal of Information Systems and Informatics, vol. 8, no. 1, pp. 1011–1044, Mar. 2026, doi: 10.63158/journalisi.v8i1.1462.

[6] N. Malasari and M. Ramli, “Analisis Sentimen Media Sosial Menggunakan Algoritma BERT dan LSTM,” Journal of Computer Science and Information Technology, vol. 1, no. 3, 2025.

[7] L. Cahyaningrum, A. Luthfiarta, and M. Rahayu, “Sentiment Analysis on the Impact of MBKM on Student Organizations Using Supervised Learning with Smote to Handle Data Imbalance,” Inform : Jurnal Ilmiah Bidang Teknologi Informasi dan Komunikasi, vol. 9, no. 1, 2024, doi: 10.25139/inform.v9i1.7484.

[8] E. Erlin, Y. Desnelita, N. Nasution, L. Suryati, and F. Zoromi, “Dampak SMOTE terhadap Kinerja Random Forest Classifier berdasarkan Data Tidak seimbang,” MATRIK : Jurnal Manajemen, Teknik Informatika dan Rekayasa Komputer, vol. 21, no. 3, 2022, doi: 10.30812/matrik.v21i3.1726.

[9] S. E. Prianto, B. Berlilana, and R. E. Saputro, “Analisis Sentimen Program Makan Bergizi Gratis Menggunakan Random Forest dan Support Vector Machine dengan Penyeimbangan Data Berbasis SMOTE,” Jurnal Pendidikan dan Teknologi Indonesia, vol. 6, no. 2, pp. 353–365, Mar. 2026, doi: 10.52436/1.jpti.1491.

[10] F. Panjaitan et al., “Studi Komparatif Algoritma Machine Learning Pada Analisis Sentimen Media Sosial,” JATI (Jurnal Mahasiswa Teknik Informatika), vol. 9, no. 2, 2025, doi: 10.36040/jati.v9i2.13277.

[11] M. Arifin and S. Adiyono, “Hyperparameter Tuning in Machine Learning to Predicting Student Academic Achievement,” International Journal of Artificial Intelegence Research, vol. 8, no. 1, 2024, doi: https://doi.org/10.29099/ijair.v8i1.1.1214.

[12] N. Fajriyah, N. T. Lapatta, D. W. Nugraha, and R. Laila, “Implementasi Svm Dan Smote Pada Analisis Sentimen Media Sosial X Terhadap Pelantikan Agus Harimurti Yudhoyono,” JIPI (Jurnal Ilmiah Penelitian dan Pembelajaran Informatika), vol. 10, no. 2, 2025, doi: 10.29100/jipi.v10i2.6246.

[13] A. F. Anjani, D. Anggraeni, and I. M. Tirta, “Implementasi Random Forest Menggunakan SMOTE untuk Analisis Sentimen Ulasan Aplikasi Sister for Students UNEJ,” Jurnal Nasional Teknologi dan Sistem Informasi, vol. 9, no. 2, 2023, doi: 10.25077/teknosi.v9i2.2023.163-172.

[14] S. Adiyono and T. Hidayat, “Predictive Health Monitoring of a Marine Propulsion Plant Using Multi-Output Random Forest Regression,” International Journal of Marine Engineering Innovation and Research, vol. 11, no. 1, pp. 248–258, 2026.

[15] M. Arifin, S. Adiyono, and A. Kurniawan, “Advanced Machine Learning Approaches to Combat Rice Challenges: A Comparative Study of SMO and PSO-ACO,” International Journal on Technical and Physical Problems of Engineering, vol. 17, no. 4, pp. 40–50, 2025.

[16] C. Mulia and A. Kurniasih, “Teknik SMOTE untuk Mengatasi Imbalance Class dalam Klasifikasi Bank Customer Churn Menggunakan Algoritma Naïve bayes dan Logistic Regression,” Prosiding Seminar Nasional Mahasiswa Bidang Ilmu Komputer dan Aplikasinya, vol. 4, no. 2, 2023.

[17] P. P. Putra, M. K. Anam, A. S. Chan, A. Hadi, N. Hendri, and A. Masnur, “Optimizing Sentiment Analysis on Imbalanced Hotel Review Data Using SMOTE and Ensemble Machine Learning Techniques,” Journal of Applied Data Sciences, vol. 6, no. 2, 2025, doi: 10.47738/jads.v6i2.618.

[18] A. Annur Rohman, G. Alfa Trisnapradika, and K. Kunci, “Perbandingan Algoritma NBC, SVM, Logistic Regression untuk Analisis Sentimen Terhadap Wacana KaburAjaDulu di Media Sosial X,” Building of Informatics, Technology and Science (BITS), vol. 7, no. 1, 2025.

[19] M. Nur Akbar, N. A. S. Yusuf, N. Nasrullah, and M. Mubarak, “Analisis Sentimen Pengguna Indihome dengan Metode Klasifikasi Support Vector Machine (SVM),” Journal Software, Hardware and Information Technology, vol. 2, no. 1, pp. 13–21, Jan. 2022, doi: 10.24252/shift.v2i1.18.

[20] A. R. Muhammad Fikri, J. Jondri, and W. Astuti, “Sentiment Analysis Against IndiHome and First Media Internet Providers Using Ensemble Stacking Method,” Building of Informatics, Technology and Science (BITS), vol. 4, no. 2, Sep. 2022, doi: 10.47065/bits.v4i2.1969.

[21] A. N. Aini, J. Jondri, and W. Astuti, “Sentiment Analysis on Twitter Against IndiHome Providers Using Chi-Square and Ensemble Bagging Methods,” Building of Informatics, Technology and Science (BITS), vol. 4, no. 2, pp. 516–521, Sep. 2022, doi: 10.47065/bits.v4i2.1967.

[22] Alwin Marcellino, Yosefa Camilia Moniung, and Rusbandi, “Penerapan Teknik SMOTE Pada Analisis Sentimen Bea Cukai Menggunakan Algoritma Naïve Bayes,” Jurnal Algoritme, vol. 4, no. 2, 2022.

[23] S. Adiyono, N. Latifah, and D. Laily Fithri, “Analysis of Digital Readiness in the Social Assistance Distribution System with the Unified Theory of Acceptance and Use of Technology (UTAUT),” Journal of Applied Informatics and Computing, vol. 9, no. 2, pp. 483–489, Mar. 2025, doi: 10.30871/jaic.v9i2.9070.

[24] I. Abdul Ghofur and A. D. Putri, “Analisis Perbandingan Svm, Knn Dan Naïve Bayes Pada Sentiment Analysis Tweet Makan Bergizi Gratis,” Computer Based Information System Journal, vol. 14, no. 1, pp. 100–117, Mar. 2026, doi: 10.33884/cbis.v14i1.10994.

[25] S. Dermawan and A. T. Ayunda, “Sentiment Analysis of Coretax on Social Media X Using Naive Bayes, SVM, and LSTM for Service Improvement,” Journal of Applied Informatics and Computing, vol. 9, no. 6, 2025, doi: 10.30871/jaic.v9i6.11063.

[26] D. Saputra and M. R. Pribadi, “Analisis Sentimen Masyarakat terhadap Layanan Provider Internet di Indonesia menggunakan SVM,” MDP Student Conference, vol. 2, no. 1, pp. 32–41, Apr. 2023, doi: 10.35957/mdp-sc.v2i1.4554.

[27] A. Alim Murtopo, M. Aditdya, P. Septiana Ananda, and G. Gunawan, “Penerapan Computer Vision Untuk Mendeteksi Kelengkapan Atribut Siswa Menggunakan Metode CNN,” PROSISKO: Jurnal Pengembangan Riset dan Observasi Sistem Komputer, vol. 11, no. 2, pp. 247–258, Sep. 2024, doi: 10.30656/prosisko.v11i2.8752.

[28] N. F. Octavia and Berlilana, “Penerapan Algoritma K-Nearest Neighbor untuk Analisis Sentimen Ulasan Produk Elektronik pada Platform E-Commerce,” Jurnal Algoritma, vol. 22, no. 2, Nov. 2025, doi: 10.33364/algoritma/v.22-2.3083.

[29] A. Setiawan and H. Yuliansyah, “Analisis Sentimen Berbasis Aspek Terhadap Ulasan Pengguna pada Game Honkai: Star Rail Menggunakan Naïve Bayes Classifier,” SISTEMASI: Jurnal Sistem Informasi, vol. 13, no. 5, 2024.

[30] D. Septiani and I. Isabela, “Analisis Term Frequency Inverse Document Frequency (Tf-Idf) Dalam Temu Kembali Informasi Pada Dokumen Teks,” Sintesia, vol. 1, 2022.

[31] I. K. Dharmendra, I. M. Agus, W. Putra, and Y. P. Atmojo, “Evaluasi Efektivitas SMOTE dan Random Under Sampling pada Klasifikasi Emosi Tweet,” Informatics for Educators And Professionals : Journal of Informatics, vol. 9, no. 2, 2024.

[32] J. R. Landis and G. G. Koch, “The Measurement of Observer Agreement for Categorical Data,” Biometrics, vol. 33, no. 1, p. 159, Mar. 1977, doi: 10.2307/2529310.