Comparative Analysis of the Performance of Machine Learning Methods and Text Embedding Techniques in Classifying Toxic Conversations in the Roblox Game
DOI:
https://doi.org/10.30871/jaic.v10i3.12646Keywords:
Bag-Of-Words, Machine Learning, Roblox, Text Classification, TF-IDF, Toxic ChatAbstract
Online games have evolved into digital social spaces where player interactions often include toxic communication, potentially affecting user experience and psychological well-being, especially among younger players. This research is intended to examine and compare the performance of various machine learning algorithms in classifying toxic chat on the Roblox platform and to identify underlying linguistic patterns. The dataset consists of 7,119 Indonesian-language chat data labeled into six categories: identity_hate, insult, obscene, severe_toxic, threat, and toxic. The methodology includes data preprocessing, text representation using Bag-of-Words (BoW) and TF-IDF, and classification using Naive Bayes, Support Vector Machine (SVM), and Random Forest. To assess how well the model performs, several metrics are used, including accuracy, precision, recall, F1-score, and 3-fold cross-validation. The results show that SVM with TF-IDF achieves the best performance with 84.48% accuracy, followed closely by SVM with BoW. The findings indicate that while classical machine learning models remain effective, challenges persist in distinguishing linguistically similar categories.
Downloads
References
[1] Y. L. Setiawan, J. Nasir, and R. hadi Putra, “Komunikasi Virtual melalui Perilaku Trash-Talking Antar Pemain Game Online Mobile Legends,” Ekasakti Jurnal Penelitian Dan Pengabdian, vol. 05, no. 01, pp. 133–141, 2024, doi: 10.31933/ejpp.v5i1.1244.
[2] Muh. Zaad and Arni, “Perilaku Komunikasi Toxic Remaja yang Bermain Game Online Mobile Legends Pulau Barrang Lompo Kecamatan Kepulauan Sangkarrang Kota Makassar,” Jurnal Komunikasi dan Organisasi (J-KO), vol. 7, no. 1, pp. 15–20, 2025, doi: 10.26618/jko.v7i1.17585.
[3] Á. Zsila, R. Shabahang, M. S. Aruguete, and G. Orosz, “Toxic behaviors in online multiplayer games: Prevalence, perception, risk factors of victimization, and psychological consequences,” Aggress. Behav., vol. 48, no. 3, pp. 356–364, 2022, doi: 10.1002/ab.22023.
[4] T. Yuliastika and A. Fitriana Poerana, “Motif Penggunaan Game Online Roblox pada Anak Usia Sekolah,” Jurnal Ilmiah Wahana Pendidikan, vol. 9, no. 9, pp. 364–371, 2023, doi: 10.5281/zenodo.7953027.
[5] J. Huang, “Analysis on the Young Age of Roblox Platform Audience Targeting,” Highlights in Business,Economics and Management, vol. 11, pp. 112–117, 2023, doi: doi:10.54097/hbem.v11i.7954.
[6] H. A. Janati, E. Elvandri, S. V. Vianto, B. Agnes, and B. Cholas, “Perilaku Toxic Dalam Game Moba Dan Dampaknya Terhadap Komunitas Gamer Di Batam,” Simtek : jurnal sistem informasi dan teknik komputer, vol. 10, no. 2, pp. 374–378, 2025, doi: 10.51876/simtek.v10i2.1585.
[7] U. Naseem, S. Shiwakoti, S. B. Shah, S. Thapa, and Q. Zhang, “GameTox: A Comprehensive Dataset and Analysis for Enhanced Toxicity Detection in Online Gaming Communities,” Proceedings of the 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies: Long Papers, NAACL-HLT 2025, vol. 2, pp. 440–447, 2025, doi: 10.18653/v1/2025.naacl-short.37.
[8] M. D. Desriansyah, I. U. Sari, and Z. Zulfahmi, “Analisis Efektivitas Algoritma Machine Learning dalam Deteksi Hoaks: Pada Berita Digital Berbahasa Indonesia,” Jurnal Sistem Informasi Dan Informatika, vol. 3, no. 2, pp. 63–69, 2025, doi: 10.47233/jiska.v3i1.2024.
[9] H. Ismail, A. Khalil, and A. Jasmy, “Enhancing online toxicity detection on gaming networks: a novel embeddings-based valence lexicon approach,” Int. J. Data Sci. Anal., vol. 20, no. 5, pp. 4489–4500, 2025, doi: 10.1007/s41060-025-00730-1.
[10] K. Andariefli, J. Leonard, V. Dewanto, and A. D. Novika, “Comparative analysis of two-class and multi-class toxicity detection using multi-source gaming chat data,” Procedia Comput. Sci., vol. 269, pp. 825–833, 2025, doi: 10.1016/j.procs.2025.09.025.
[11] R. P. Sidiq, B. A. Dermawan, and Y. Umaidah, “Sentimen Analisis Komentar Toxic pada Grup Facebook Game Online Menggunakan Klasifikasi Naïve Bayes,” Jurnal Informatika Universitas Pamulang, vol. 5, no. 3, p. 356, 2020, doi: 10.32493/informatika.v5i3.6571.
[12] T. A. Alghamdi and N. Javaid, “A Survey of Preprocessing Methods Used for Analysis of Big Data Originated from Smart Grids,” IEEE Access, vol. 10, pp. 29149–29171, 2022, doi: 10.1109/ACCESS.2022.3157941.
[13] D. Rifaldi, A. Fadlil, and Herman, “Teknik Preprocessing Pada Text Mining Menggunakan Data Tweet ‘Mental Health,’” Decode: Jurnal Pendidikan Teknologi Informasi, vol. 3, no. 2, pp. 161–171, 2023, doi: 10.51454/decode.v3i2.131.
[14] P. Ayuningtiyas, K. Ditha Tania, and W. Kurnia Sari, “Sentiment-Based Knowledge Discovery pada Aplikasi iPusnas Menggunakan Metode Machine Learning dan Deep Learning,” Journal of Applied Informatics and Computing (JAIC), vol. 9, no. 5, pp. 2486–2497, 2025, [Online]. Available: http://jurnal.polibatam.ac.id/index.php/JAIC
[15] C. Suhaeni, S. A. Kamila, F. Fahira, M. Yusran, and G. A. Dito, “Exploring a Large Language Model on the ChatGPT Platform for Indonesian Text Preprocessing Tasks,” Indonesian Journal of Statistics and Its Applications, vol. 9, no. 1, pp. 100–116, 2025, doi: 10.29244/ijsa.v9i1p100-116.
[16] A. A. Pratiwi and M. Kamayani, “Perbandingan Pelabelan Data dalam Analisis Sentimen Kurikulum Proyek di platform TikTok: Pendekatan Naïve Bayes,” Jurnal Eksplora Informatika, vol. 14, no. 1, pp. 96–107, 2024, doi: 10.30864/eksplora.v14i1.1093.
[17] E. W. Pamungkas, C. S. Wahyuni, I. Amal, D. Purworini, and B. S. Rintyarna, “Decoding hate in memes: multimodal and multitask approaches for low-resource Indonesian social media,” PeerJ Comput. Sci., vol. 12, 2026, doi: 10.7717/peerj-cs.3736.
[18] R. Hayami, S. Mohnica, and Soni, “Klasifikasi multilabel komentar toxic pada sosial media twitter menggunakan convolutional neural network(CNN),” Jurnal CoSciTech (Computer Science and Information Technology), vol. 4, no. 1, pp. 1–6, 2023, doi: 10.37859/coscitech.v4i1.4365.
[19] V. N. Romadina, O. Juwita, and P. Pandunata, “Analisis Komentar Toxic Terhadap Informasi COVID-19 pada YouTube Kementerian Kesehatan Menggunakan Metode Naïve Bayes Classifier,” INFORMAL: Informatics Journal, vol. 9, no. 1, pp. 92–99, 2024, doi: 10.19184/isj.v9i1.48126.
[20] N. M. D. Sikiandani, I. M. A. Dwi Suarjaya, and Y. P. Putra, “Browser-Based Detection of Harmful Content with Deep Learning Model,” Journal of Applied Informatics and Computing, vol. 9, no. 4, pp. 1800–1811, 2025, doi: 10.30871/jaic.v9i4.9804.
[21] R. Oktafiani, A. Hermawan, and D. Avianto, “Pengaruh Komposisi Split data Terhadap Performa Klasifikasi Penyakit Kanker Payudara Menggunakan Algoritma Machine Learning,” Jurnal Sains dan Informatika, vol. 9, no. 1, pp. 19–28, 2023, doi: 10.34128/jsi.v9i1.622.
[22] Sutriawan, S. Mutmainnah, T. Ansyor Lorosae, and S. Ramadhan, “Model Text Embedding dan TF-IDF+Ngram untuk Meningkatkan Kinerja Algoritma Binary Classifier pada Klasifikasi SMS Palsu,” vol. 4, no. 1, pp. 55–64, 2025, [Online]. Available: https://ojs.trigunadharma.ac.id/index.php/jsi
[23] A. D. M. Putri, N. Sulistianingsih, and R. Rismayati, “Pengaruh Teknik Representasi Teks Bag-of-Words dan TF-IDF terhadap Akurasi Klasifikasi Sentimen Teks Multi-Domain,” JTIM : Jurnal Teknologi Informasi dan Multimedia, vol. 7, no. 4, pp. 675–688, 2025, doi: 10.35746/jtim.v7i4.756.
[24] S. D. Prasetyo, S. S. Hilabi, and F. Nurapriani, “Analisis Sentimen Relokasi Ibukota Nusantara Menggunakan Algoritma Naïve Bayes dan KNN,” Jurnal KomtekInfo, vol. 10, no. 1, pp. 1–7, 2023, doi: 10.35134/komtekinfo.v10i1.330.
[25] V. Ayumi, D. Ramayanti, H. Noprisson, A. Ratnasari, and U. Salamah, “Pengaruh Tuning Parameter dan Cross Validation Pada Klasifikasi Teks Komplain Bahasa Indonesia Menggunakan Algoritma Support Vector Machine ,” JSAI : Journal Scientific and Applied Informatics , vol. 6, no. 3, pp. 493–498, Nov. 2023.
[26] A. Hussaina and A. Aslamb, “Hate speech against women and immigrants: A comparative analysis of machine learning and text embedding techniques,” Journal of Applied Research and Technology, vol. 22, no. 4, pp. 548–559, 2024, Accessed: Mar. 06, 2026. [Online]. Available: https://jart.icat.unam.mx/index.php/jart/article/view/2466/1129
[27] P. Ayuningtiyas, K. Ditha Tania, and W. Kurnia Sari, “Sentiment-Based Knowledge Discovery pada Aplikasi iPusnas Menggunakan Metode Machine Learning dan Deep Learning,” Journal of Applied Informatics and Computing (JAIC), vol. 9, no. 5, p. 2486, Oct. 2025, [Online]. Available: http://jurnal.polibatam.ac.id/index.php/JAIC
[28] T. N. Pasaribu, J. P. Tanjung, D. Hutauruk, E. S. Hutagalung, and S. Silitonga, “Study of Public Sentiment Towards Beauty Products Using A Machine Learning Approach: Random Forest Analysis On Social Media,” Sinkron: Jurnal dan Penelitian Teknik Informatika, vol. 8, no. 3, pp. 2088–2098, 2024, doi: 10.33395/sinkron.v8i3.13969.
[29] S. Rosalin and B. F. Supriyanto, “Analisis Sentimen Program Merdeka Belajar dengan Text Analysis Wordcloud & Word Frequency,” Jurnal Minfo Polgan, vol. 12, no. 1, pp. 25–32, 2023, doi: 10.33395/jmp.v12i1.12312.
[30] C. H. Pratama and Y. Findawati, “Klasifikasi Hate Speech dan Emosi Dalam Teks Berbahasa Indonesia Pada Pengguna Twitter Menggunakan Metode Naïve Bayes Classifier,” Indonesian Journal of Applied Technology, vol. 1, no. 3, pp. 1–10, 2024, doi: 10.47134/ijat.v1i3.3105.
[31] Sawidiah and M. Ulfa, “Tindak Insult pada Kasus Bullying Verbal di Media Sosial Online (Kajian Linguistik Forensik),” Stilistika: Jurnal Pendidikan Bahasa dan Sastra, vol. 18, no. 2, pp. 347–362, 2025, doi: 10.30651/st.v18i2.25712.
[32] M. P. D. Sari, I. W. Pastika, and M. S. Satyawati, “Ujaran Kebencian Terhadap Selebgram Azizah Salsha Di Media Sosial Tiktok: Kajian Linguistik Forensik,” Kulturistik: Jurnal Bahasa & Budaya, vol. 9, no. 2, pp. 49–57, 2025, doi: 10.22225/kulturistik.9.2.12631.
[33] Jayus, Sumariyah, A. Abdullah, and Mustafa, “YouTube, Public Discourse, and the ‘Makan Siang Gratis’ Program: An Analysis of Toxicity Comments on the Liputan6 Channel,” Jogjakarta Communication Conference (JCC), vol. 3, no. 1, pp. 367–380, 2025, [Online]. Available: https://jcc-indonesia.id/
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Octa Dama Yanti, Syifa Alfariani, Syifa Naura Milla Celesta, Ken Dhita Tania, Ahmad Rifai
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License (Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) ) that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
