Sentiment Classification of Telecommunication Application Reviews Using Naive Bayes and Support Vector Machine: A Comparison of TF-IDF Unigram and N-Gram Feature Representations

Karin Nisrina Andriyani; Hafiyyan Putra Pratama

doi:10.30871/jaic.v10i5.14044

Authors

Karin Nisrina Andriyani Universitas Pendidikan Indonesia
Hafiyyan Putra Pratama Universitas Pendidikan Indonesia

DOI:

https://doi.org/10.30871/jaic.v10i5.14044

Keywords:

MyTelkomsel, Naive Bayes, N-Gram, Support Vector Machine, TF-IDF

Abstract

Important information regarding user satisfaction and complaints about an application can be obtained from reviews uploaded to the Google Play Store. Telecommunications service applications are among the most widely used applications in Indonesia and receive numerous reviews that can be leveraged to evaluate service quality. However, manually analyzing large volumes of reviews is inefficient. This study aims to classify sentiment in telecommunication application user reviews by comparing TF-IDF Unigram (Baseline) and N-Gram (Unigram-Bigram) feature representations and evaluating Naive Bayes and Support Vector Machine (SVM), with hyperparameter validation using Grid Search applied separately to each feature scenario. The dataset consists of 2,836 user reviews collected from the Google Play Store through web scraping. The data underwent preprocessing, rating-based sentiment labeling with rule-based label correction, data splitting, and TF-IDF feature extraction for both scenarios before classification. Model performance was evaluated using accuracy, precision, recall, F1-score, F1-score macro average, and confusion matrix. The results show that SVM outperforms Naive Bayes in both feature scenarios. The best performance was achieved by SVM using the Baseline TF-IDF Unigram representation, with 92.43% accuracy, 91.98% precision, 88.24% recall, 90.07% F1-score, and 91.98% F1-score macro average. Grid Search produced identical optimal configurations for both scenarios, namely linear kernel, C=1, and gamma=scale, indicating their suitability for both feature representations. These findings indicate that, within the dataset and experimental configuration used, adding N-Gram features does not necessarily improve classification performance, particularly for SVM, while unigram representation alone effectively classifies telecommunication application user reviews.

Downloads

Download data is not yet available.

References

[1] A. I. A. Azkiya and B. Santoso, “Penerapan Machine Learning Pada Analisis Sentimen Aplikasi MyTelkomsel Menggunakan Data Ulasan Google Playstore,” Indones. J. Comput. Sci., vol. 13, no. 1, pp. 1227–1240, 2023, [Online]. Available: http://ijcs.stmikindonesia.ac.id/ijcs/index.php/ijcs/article/view/3135

[2] R. Bagus Adinata and D. Laily Fithri, “Sentiment Classification of MyTelkomsel Reviews Using SVM and Logistic Regression,” IJCCS (Indonesian J. Comput. Cybern. Syst., vol. 20, no. 1, pp. 83–94, 2026, doi: 10.22146/ijccs.110409.

[3] T. S. Ningsih, T. I. Hermanto, and I. M. Nugroho, “Sentiment Analysis of Mobile Provider Application Reviews Using Naive Bayes Algorithm and Support Vector Machine,” Sink. J. dan Penelit. Tek. Inform., vol. 8, no. 2, pp. 824–835, 2024, doi: https://doi.org/10.33395/v8i2.13469.

[4] A. Puji Astuti, S. Alam, and I. Jaelani, “Komparasi Algoritma Support Vector Machine dengan Naive Bayes Untuk Analisis Sentimen Pada Aplikasi BRImo,” J. Bangkit Indones., vol. 11, no. 2, pp. 1–6, 2022, doi: 10.52771/bangkitindonesia.v11i2.196.

[5] A. Nurian, “Analisis Sentimen Ulasan Pengguna Aplikasi Google Play Menggunakan Naïve Bayes,” J. Inform. dan Tek. Elektro Terap., vol. 11, no. 3s1, pp. 829–835, 2023, doi: 10.23960/jitet.v11i3s1.3348.

[6] J. Homepage et al., “Analisis Sentimen Pengguna Aplikasi Jamsostek Mobile Berdasarkan Ulasan Google Play Store Menggunakan Algoritma Support Vector Machine dan Naive Bayes,” MALCOM Indones. J. Mach. Learn. Comput. Sci., vol. 6, no. 1, pp. 373–384, 2026, [Online]. Available: https://journal.irpi.or.id/index.php/malcom/article/view/2526

[7] M. Zainuddin Siregar, A. Marwan Elhanafi, and D. Irwan, “Analisis Sentimen Terhadap Ulasan Aplikasi Media Sosial Di Google Play Menggunakan Algoritma Naive Bayes,” JATI (Jurnal Mhs. Tek. Inform., vol. 9, no. 2, pp. 3373–3381, 2025, doi: 10.36040/jati.v9i2.12841.

[8] J. M. Manampiring, J. Halawa, J. Zai, I. Kurniawati, R. Fahlapi, and W. Bismi, “Analisis Sentimen Ulasan Aplikasi X Pada Google Play Store Menggunakan Komparasi Svm, Knn, Dan Naive Bayes,” Djtechno J. Teknol. Inf., vol. 7, no. 1, pp. 1–13, 2026, doi: 10.46576/djtechno.v7i1.8095.

[9] F. Marzuqi, Y. Purba, T. Syaputra, W. Bismi, I. Kurniawati, and R. Fahlapi, “Perbandingan Algoritma Machine Learning Untuk Sentimen Ulasan Aplikasi Jobstreet,” INFOTECH J., vol. 11, no. 2, pp. 416–423, 2025, doi: 10.31949/infotech.v11i2.16609.

[10] F. Amin Mujtahid, B. Zaman, and G. Guntoro Aji, “Analisis Klasifikasi Sentimen Neobank: Perbandingan Konfigurasi N-Gram pada TF-IDF Menggunakan Naive Bayes dan SVM,” TIN Terap. Inform. Nusant., vol. 6, no. 12, pp. 2375–2383, 2026, doi: 10.47065/tin.v6i12.9702.

[11] F. Lawalata, “Analisis Performa Support Vector Machine (SVM) dengan Hyperparameter Tuning untuk Klasifikasi Data Stress Monitoring Siswa,” JOINTER J. Informatics Eng., vol. 6, no. 02, pp. 9–15, 2025, doi: 10.53682/jointer.v6i02.413.

[12] E. Elgeldawi, A. Sayed, A. R. Galal, and A. M. Zaki, “Hyperparameter tuning for machine learning algorithms used for arabic sentiment analysis,” Informatics, vol. 8, no. 4, pp. 1–21, 2021, doi: 10.3390/informatics8040079.

[13] W. Nugraha and A. Sasongko, “Hyperparameter Tuning on Classification Algorithm with Grid Search,” Sistemasi, vol. 11, no. 2, p. 391, 2022, doi: 10.32520/stmsi.v11i2.1750.

[14] O. Alotaibi, E. Pardede, and S. Tomy, “Cleaning Big Data Streams: A Systematic Literature Review,” Technologies, vol. 11, no. 4, pp. 1–24, 2023, doi: 10.3390/technologies11040101.

[15] K. P. Harmandini and K. M. L, “Analysis of TF-IDF and TF-RF Feature Extraction on Product Review Sentiment,” Sinkron, vol. 8, no. 2, pp. 929–937, 2024, doi: 10.33395/sinkron.v8i2.13376.

[16] M. A. Palomino and F. Aider, “Evaluating the Effectiveness of Text Pre-Processing in Sentiment Analysis,” Appl. Sci., vol. 12, no. 17, 2022, doi: 10.3390/app12178765.

[17] A. Gasparetto, M. Marcuzzo, A. Zangari, and A. Albarelli, “A Survey on Text Classification Algorithms: From Text to Predictions,” Inf., vol. 13, no. 2, pp. 1–39, 2022, doi: 10.3390/info13020083.

[18] A. B. Putra Negara, “The Influence Of Applying Stopword Removal And Smote On Indonesian Sentiment Classification,” Lontar Komput. J. Ilm. Teknol. Inf., vol. 14, no. 3, p. 172, 2023, doi: 10.24843/lkjiti.2023.v14.i03.p05.

[19] J. Jumadi, D. S. Maylawati, L. D. Pratiwi, and M. A. Ramdhani, “Comparison of Nazief-Adriani and Paice-Husk algorithm for Indonesian text stemming process,” IOP Conf. Ser. Mater. Sci. Eng., vol. 1098, no. 3, p. 032044, 2021, doi: 10.1088/1757-899x/1098/3/032044.

[20] L. Lavazza and S. Morasca, “Common Problems With the Usage of F-Measure and Accuracy Metrics in Medical Research,” IEEE Access, vol. 11, no. June, pp. 51515–51526, 2023, doi: 10.1109/ACCESS.2023.3278996.

[21] K. M. Sujon, R. Hassan, K. Choi, and M. A. Samad, “Accuracy, precision, recall, F1-score, or MCC Empirical evidence from advanced statistics, ML, and XAI for evaluating business predictive models,” J. Big Data, vol. 12, 2025, doi: 10.1186/s40537-025-01313-4.

[22] M. C. Hinojosa Lee, J. Braet, and J. Springael, “Performance Metrics for Multilabel Emotion Classification: Comparing Micro, Macro, and Weighted F1-Scores,” Appl. Sci., vol. 14, no. 21, 2024, doi: 10.3390/app14219863.

[23] I. Markoulidakis and G. Markoulidakis, “Probabilistic Confusion Matrix: A Novel Method for Machine Learning Algorithm Generalized Performance Analysis,” Technologies, vol. 12, no. 7, 2024, doi: 10.3390/technologies12070113.