Hybrid Lexicon-Machine Learning Approach for Aspect-Based Sentiment Analysis on Large-Scale Indonesian App Reviews: A Case Study of Telegram

Toto Indriyatmoko; Arief Hermanto

doi:10.30871/jaic.v10i5.14141

Authors

Toto Indriyatmoko Universitas Amikom Yogyakarta
Arief Hermanto Universitas Gadjah Mada

DOI:

https://doi.org/10.30871/jaic.v10i5.14141

Keywords:

Aspect-Based Sentiment Analysis, Binary Classification, Lexicon-Machine Learning, LightGBM, Mobile App Reviews

Abstract

Mobile application reviews provide critical unstructured feedback for developers, yet processing large-scale Indonesian app reviews remains hindered by informal language noise and severe class imbalance. While deep learning offers strong context modeling, its computational burden is impractical for industrial-scale deployment exceeding hundreds of thousands of entries. This study proposes a scalable, computationally efficient Hybrid Lexicon-Machine Learning framework for Aspect-Based Sentiment Analysis (ABSA) on large-scale Indonesian user reviews, using Telegram as a case study. We processed 399,527 reviews from the Google Play Store through a pipeline integrating TF-IDF n-grams, explicit domain-adapted lexicon scoring (incorporating negation and intensifier heuristics), and review length metadata into a LightGBM classifier. To overcome the scarcity and linguistic ambiguity of neutral ratings, a binary classification strategy (Positive vs. Negative/Critical) was implemented. The model achieved Accuracy of 80.4%, Macro-F1 of 76.7%, and AUC-ROC of 0.855 on a strictly held-out test set, peaking at 0.933 for Privacy. Ablation experiments confirm that incorporating explicit lexicon scores consistently improves Macro-F1 by up to 5.1% over standalone TF-IDF. Aspect distribution revealed critical bottlenecks in Privacy (73.0% negative) and Notification (66.1% negative), providing actionable roadmaps for software maintenance.

Downloads

Download data is not yet available.

References

[1] N. I. Lestari, S. M. Taib, W. Wibowo, I. A. Aziz, and M. R. Habibi, “Aspect-Based Sentiment Analysis for Mobile App Review Using Convolutional Neural Network (CNN) and Word2Vec,” in 2024 IEEE 7th International Conference on Electrical, Electronics and System Engineering (ICEESE), IEEE, 2024, pp. 1–6.

[2] A. Maroof, S. Wasi, S. I. Jami, and M. S. Siddiqui, “Aspect-based sentiment analysis for service industry,” IEEE Access, vol. 12, pp. 109702–109713, 2024.

[3] F. Meng, M. Qi, and S. Bao, “Innovative Sentiment Analysis of Mobile App Review Texts Based on MHA-MLP-Mixer,” in 2024 IEEE Smart World Congress (SWC), IEEE, 2024, pp. 1150–1157.

[4] R. Budianoor, S. W. Saputro, F. Abadi, R. A. Nugroho, and A. Farmadi, “Quantifying the Impact of Text Preprocessing on IndoBERT Fine-Tuning for Indonesian Informal Culinary Sentiment Analysis,” Journal of Computing Theories and Applications, vol. 3, no. 4, pp. 564–581, 2026.

[5] R. Obiedat et al., “Sentiment analysis of customers’ reviews using a hybrid evolutionary SVM-based approach in an imbalanced data distribution,” Ieee Access, vol. 10, pp. 22260–22273, 2022.

[6] E. Essa, K. Omar, and A. Alqahtani, “Fake news detection based on a hybrid BERT and LightGBM models,” Complex & Intelligent Systems, vol. 9, no. 6, pp. 6581–6592, 2023.

[7] U. Mangal, S. Mogha, and S. Malik, “Data-driven decision making: maximizing insights through business intelligence, artificial intelligence and big data analytics,” in 2024 International Conference on Advances in Computing Research on Science Engineering and Technology (ACROSET), IEEE, 2024, pp. 1–7.

[8] L. Zhu, M. Xu, Y. Bao, Y. Xu, and X. Kong, “Deep learning for aspect-based sentiment analysis: a review,” PeerJ Comput. Sci., vol. 8, p. e1044, 2022.

[9] D. Tang, B. Qin, and T. Liu, “Document modeling with gated recurrent neural network for sentiment classification,” in Proceedings of the 2015 conference on empirical methods in natural language processing, 2015, pp. 1422–1432.

[10] G. Ke et al., “Lightgbm: A highly efficient gradient boosting decision tree,” Adv. Neural Inf. Process. Syst., vol. 30, 2017.

[11] L. Muthathohirin, “Telegram App Reviews on Playstore (400K),” Kaggle. Accessed: Aug. 26, 2026. [Online]. Available: https://www.kaggle.com/datasets/luthfimuthathohirin/telegramm923ueede/data

[12] B. Wilie et al., “IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding,” in Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, 2020, pp. 843–857.

[13] J. Estdale and E. Georgiadou, “Applying the ISO/IEC 25010 quality models to software product,” in European Conference on Software Process Improvement, Springer, 2018, pp. 492–503.

[14] J. Leskovec, A. Rajaraman, and J. D. Ullman, Mining of massive datasets, vol. 2. Cambridge university press New York, NY, USA, 2014.

[15] M. Taboada, J. Brooke, M. Tofiloski, K. Voll, and M. Stede, “Lexicon-based methods for sentiment analysis,” Computational linguistics, vol. 37, no. 2, pp. 267–307, 2011.

[16] S. Kiritchenko, X. Zhu, and S. M. Mohammad, “Sentiment analysis of short informal texts,” Journal of Artificial Intelligence Research, vol. 50, pp. 723–762, 2014.

[17] M. Altalhan, A. Algarni, and M. T.-H. Alouane, “Imbalanced data problem in machine learning: A review,” IEEE access, vol. 13, pp. 13686–13699, 2025.

[18] A. Qazi, N. Hasan, R. Mao, M. E. M. Abo, S. K. Dey, and G. Hardaker, “Machine learning-based opinion spam detection: A systematic literature review,” IEEE Access, vol. 12, pp. 143485–143499, 2024.

[19] D. L. Shanthi and N. Chethan, “Genetic algorithm based hyper-parameter tuning to improve the performance of machine learning models,” SN Comput. Sci., vol. 4, no. 2, p. 119, 2022.

[20] I. H. Witten, E. Frank, M. A. Hall, C. J. Pal, and M. Data, “Practical machine learning tools and techniques,” in Data mining, Elsevier Amsterdam, The Netherlands, 2005, pp. 403–413.

[21] E. Guzman and B. Bruegge, “Towards emotional awareness in software development teams,” in Proceedings of the 2013 9th joint meeting on foundations of software engineering, 2013, pp. 671–674.

[22] T. Saito and M. Rehmsmeier, “The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets,” PLoS One, vol. 10, no. 3, p. e0118432, 2015.

[23] R. Srinivasan and C. N. Subalalitha, “Sentimental analysis from imbalanced code-mixed data using machine learning approaches,” Distrib. Parallel Databases, vol. 41, no. 1, pp. 37–52, 2023.

[24] Rianto, A. B. Mutiara, E. P. Wibowo, and P. I. Santosa, “Improving the accuracy of text classification using stemming method, a case of non-formal Indonesian conversation,” J. Big Data, vol. 8, no. 1, p. 26, 2021.

[25] D. Tang, F. Wei, N. Yang, M. Zhou, T. Liu, and B. Qin, “Learning sentiment-specific word embedding for twitter sentiment classification,” in Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2014, pp. 1555–1565.

[26] S. Panichella, A. Di Sorbo, E. Guzman, C. A. Visaggio, G. Canfora, and H. C. Gall, “How can i improve my app? classifying user reviews for software maintenance and evolution,” in 2015 IEEE international conference on software maintenance and evolution (ICSME), IEEE, 2015, pp. 281–290.

[27] S. Kiritchenko, X. Zhu, and S. M. Mohammad, “Sentiment analysis of short informal texts,” Journal of Artificial Intelligence Research, vol. 50, pp. 723–762, 2014.

[28] M. Taboada, J. Brooke, M. Tofiloski, K. Voll, and M. Stede, “Lexicon-based methods for sentiment analysis,” Computational linguistics, vol. 37, no. 2, pp. 267–307, 2011.

[29] W. Maalej and H. Nabil, “Bug report, feature request, or simply praise? on automatically classifying app reviews,” in 2015 IEEE 23rd international requirements engineering conference (RE), IEEE, 2015, pp. 116–125.