User Review Analysis: Psychological Impact of Duolingo Gamification Using Machine Learning
DOI:
https://doi.org/10.30871/jaic.v10i2.12476Keywords:
Duolingo, Gamification, Machine Learning, Psychological AnalysisAbstract
Gamification in Duolingo aims to enhance user motivation, yet its internal elements trigger diverse psychological responses. This study aims to analyze these responses through app reviews by comparing the effectiveness of four machine learning models, namely Naïve Bayes, Random Forest, Support Vector Machine (SVM), and Logistic Regression. The research design employs a quantitative comparative approach with review data classified into three labels, including Motivated, Frustrated, and Neutral. The methodology involves text preprocessing, label encoding, and addressing data imbalance using SMOTE and Class Weight techniques. The results indicate that all models achieved high accuracy, ranging from 92% to 93%. However, based on the Macro Average F1-Score metric, which measures the model’s performance balance across all classes including minority ones, the SVM and Logistic Regression models delivered the most optimal performance with a score of 0.44, outperforming Naïve Bayes and Random Forest, which scored 0.40. Feature analysis through Feature Importance reveals that the Penalty feature is the dominant factor triggering frustrated conditions, whereas the Daily Challenge feature serves as a strong predictor despite its low frequency. In conclusion, SVM and Logistic Regression are the most reliable models for mapping user emotional responses, where the findings regarding the Penalty feature provide a critical evaluation for the development of gamification mechanics.
Downloads
References
[1] F. Y. Pratama, V. Y. Erviana, I. Suwartini, and M. N. Antono, "Gamifikasi Dalam Kelas Bahasa Indonesia Bagi Penutur Asing (BIPA) Anak," Scholaria: Jurnal Pendidikan dan Kebudayaan, vol. 15, no. 1, pp. 60-67, Jan. 2025.
[2] F. Gao, Advancing Gamification Research and Practice with Three Underexplored Ideas in Self-Determination Theory, TechTrends, vol. 68, pp. 661–671, Jun. 2024. doi: 10.1007/s11528-024-00968-9.
[3] S. A. Pangestu, P. Setyosari, and D. Kuswandi, "Analisis Persepsi Mobile Learning Readiness dengan Menggunakan Technology Acceptance Model dalam Pembelajaran Bahasa Jawa," JKTP: Jurnal Kajian Teknologi Pendidikan, vol. 6, no. 3, pp. 152-162, Agt. 2023.
[4] W. D. P. F. Dewi and M. H. Santosa, "A Systematic Review: The Implementation of MALL toward Students' Reading Skill," Jurnal Pendidikan Bahasa Inggris Undiksha, vol. 10, no. 2, pp. 219-225, 2022.
[5] M. Shortt, S. Tilak, I. Kuznetcova, B. Martens, and B. Akinkuolie, "Gamification in mobile-assisted language learning: a systematic review of Duolingo literature from public release of 2012 to early 2020," Computer Assisted Language Learning, vol. 36, no. 3, pp. 517-554, 2023.
[6] P. Evans, M. Vansteenkiste, P. D. Parker, A. Kingsford-Smith, dan S. Zhou, Cognitive Load Theory and Its Relationships with Motivation: a Self-Determination Theory Perspective, Educational Psychology Review, vol. 36, art. 7, 2024. doi: 10.1007/s10648-023-09841-2.
[7] T. A. Sandy, A. Ghufron, A. Muhtadi, and Pujiriyanto, "Text Classification of Duolingo Reviews on Google Play: Insights for Enhancing M-Learning Applications," International Journal of Interactive Mobile Technologies (iJIM), vol. 19, no. 7, pp. 210-225, 2025.
[8] A. Serlina, A. Rahim, and Arbansyah, "Comparative Analysis of Naïve Bayes Algorithm Performance in English and Indonesian Text Sentiment Classification on Duolingo Application in Playstore," Informatics Engineering, Science and Technology, vol. 1, no. 1, pp. 165-171, Feb. 2025.
[9] C. Salazar, E. Montoya-Múnera, and J. Aguilar, "Sentiment analysis in learning resources," J. Comput. Educ., vol. 10, no. 4, pp. 637–662, 2023, doi: 10.1007/s40692-022-00237-9.
[10] A. Ligthart, C. Catal, and B. Tekinerdogan, "Systematic reviews in sentiment analysis: a tertiary study," Artif. Intell. Rev., vol. 54, pp. 4997–5053, 2021, doi: 10.1007/s10462-021-09973-3.
[11] W. A. Awadh, R. B. Sulaiman, and M. A. Mahmoud, "Aspect-based sentiment analysis in MOOCs: a systematic literature review introducing the MASC-MEF framework," J. King Saud Univ. - Comput. Inf. Sci., vol. 37, no. 2, 2025, doi: 10.1007/s44443-025-00018-1.
[12] L. Davoodi, J. Mezei, and M. Heikkilä, "Aspect-based sentiment classification of user reviews to understand customer satisfaction of e-commerce platforms," Electron. Commer. Res., 2025, doi: 10.1007/s10660-025-09948-4.
[13] Y. C. Hua, P. Denny, J. Wicker, and K. Taskova, "A systematic review of aspect-based sentiment analysis: domains, methods, and trends," Artif. Intell. Rev., vol. 57, no. 296, 2024, doi: 10.1007/s10462-024-10906-z.
[14] T. Shaik, X. Tao, C. Dann, H. Xie, Y. Li, and L. Galligan, "Sentiment analysis and opinion mining on educational data: A survey," Nat. Lang. Process. J., vol. 2, 100003, 2023, doi: 10.1016/j.nlp.2023.100003.
[15] H. Junianto, P. Arsi, B. A. Kusuma, and D. I. S. Saputra, "Evaluasi Aplikasi Raileo Melalui Analisis Sentimen Ulasan Playstore Dengan Metode Naive Bayes," SINTECH J., vol. 7, no. 1, pp. 1–10, 2024.
[16] M. N. Ramadhani, K. D. Tania, and M. Afrina, "Knowledge discovery in sharia mobile banking reviews using aspect-based sentiment analysis and machine learning," Journal of Applied Informatics and Computing (JAIC), vol. 10, no. 1, pp. 640-650, Feb. 2026.
[17] D. N. Fahria, K. D. Tania, and R. D. Kurnia, "Analisis komparatif algoritma Random Forest, XGBoost, dan CatBoost untuk klasifikasi tingkat stres pengguna media sosial," RABIT: Jurnal Teknologi dan Sistem Informasi Univrab, vol. 11, no. 1, pp. 1843-1853, Jan. 2026, doi: 10.36341/rabit.v11i1.7449.
[18] N. Jalal, A. Mehmood, G. S. Choi, and I. Ashraf, "A novel improved random forest for text classification using feature ranking and optimal number of trees," J. King Saud Univ. - Comput. Inf. Sci., vol. 34, no. 5, pp. 2733–2742, 2022.
[19] S. U. Hassan, J. Ahamed, dan K. Ahmad, "Analytics of machine learning-based algorithms for text classification," Sustainable Operations and Computers, vol. 3, pp. 238–248, 2022. doi: 10.1016/j.susoc.2022.03.001.
[20] E. T. Pardede, K. D. Tania, and M. Afrina, "Knowledge discovery: Analisis sentimen dan emosi WhatsApp Business dengan machine learning dan deep learning," Jurnal Nasional Teknologi dan Sistem Informasi, vol. 11, no. 3, pp. 310-318, Jan. 2026, doi: 10.25077/TEKNOSI.v11i3.2025.310-318.
[21] M. Sahira, K. D. Tania, and M. Afrina, "Comparative analysis of machine learning algorithms for cosmetic sales prediction on Tokopedia," JURTEKSI (Jurnal Teknologi dan Sistem Informasi), vol. 11, no. 4, pp. 701-708, Sep. 2025, doi: 10.33330/jurteksi.v11i4.4187.
[22] R. Saha et al., "The psychological and emotional factors of gamification and customer experience. A systematic literature review and future research agenda," Acta Psychologica, vol. 260, 105659, 2025.
[23] M. C. Untoro and M. Farhan, "Sentiment Analysis of Public Opinion on BAWASLU Using Random Forest and Particle Swarm Optimization," Sci. J. Informatics, vol. 12, no. 1, pp. 182-192, 2025.
[24] V. Pichiyan et al., "Web Scraping using Natural Language Processing: Exploiting Unstructured Text for Data Extraction and Analysis," Procedia Comput. Sci., vol. 230, pp. 193–202, 2023, doi: 10.1016/j.procs.2023.12.075.
[25] K. Maharana, S. Mondal, and B. Nemade, "A review: Data pre-processing and data augmentation techniques," Glob. Transitions Proc., vol. 3, no. 1, pp. 91–99, 2022.
[26] C. C. Olisah, L. Smith, and M. Smith, "Diabetes mellitus prediction and diagnosis from a data preprocessing and machine learning perspective," Comput. Methods Programs Biomed., vol. 220, p. 106773, 2022.
[27] T. Nomoto, "Keyword Extraction: A Modern Perspective," SN Comput. Sci., vol. 4, no. 1, p. 92, 2023, doi: 10.1007/s42979-022-01481-7.
[28] S. Albahra et al., "Artificial intelligence and machine learning overview in pathology & laboratory medicine: A general review of data preprocessing and basic supervised concepts," Semin. Diagn. Pathol., vol. 40, no. 2, pp. 71–87, 2023.
[29] G. P. Satria, K. S. Gunawan, and Z. E. Rasjida, "Aspect-Based Sentiment Analysis of Multilingual Customer Product Reviews: Leveraging Unsupervised Aspect Extraction with ABAE for XLM-ROBERTa-based Classification," Procedia Comput. Sci., vol. 269, pp. 239–248, 2025, doi: 10.1016/j.procs.2025.01.031.
[30] T. K. Thivakaran and M. Ramesh, "Exploratory Data analysis and sales forecasting of bigmart dataset using supervised and ANN algorithms," Meas. Sensors, vol. 23, p. 100388, 2022, doi: 10.1016/j.measen.2022.100388.
[31] M. Z. Abedin, C. Guotai, P. Hajek, and T. Zhang, "Combining weighted SMOTE with ensemble learning for the class-imbalanced prediction of small business credit risk," Complex Intell. Syst., vol. 9, no. 4, pp. 3559–3579, 2023.
[32] A. Arafa, N. El-Fishawy, M. Badawy, and M. Radad, "RN-SMOTE: Reduced Noise SMOTE based on DBSCAN for enhancing imbalanced data classification," J. King Saud Univ. - Comput. Inf. Sci., vol. 34, no. 8, pp. 5059–5074, 2022.
[33] I. Araf, A. Idri, and I. Chairi, "Cost-sensitive learning for imbalanced medical data: a review," Artif. Intell. Rev., vol. 57, no. 4, p. 80, 2024.
[34] T. Ling, J. Luo, J. Adams, K. Osinski, X. Liu, and D. Friedland, "Interpretable machine learning text classification for clinical computed tomography reports: a case study of temporal bone fracture," Comput. Methods Programs Biomed. Update, vol. 3, p. 100104, 2023, doi: 10.1016/j.cmpbup.2023.100104.
[35] S. Gächter, E. J. Johnson, and A. Herrmann, "Individual-level loss aversion in riskless and risky choices," Theory Decis., vol. 92, no. 3, pp. 599–624, 2022, doi: 10.1007/s11238-021-09839-8.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Septi Mulia Putri, Ken Ditha Tania
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License (Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) ) that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
