Bot Comment Detection in Indonesian Online News Using IndoBERT

Authors

  • M. Wanda Universitas Malikussaleh
  • Rizal Rizal Universitas Malikussaleh
  • Yesy Afrillia Universitas Malikussaleh

DOI:

https://doi.org/10.30871/jaic.v10i5.14100

Keywords:

BERT, IndoBERT, Natural Language Processing, Bot Comment Detection, Text Classification

Abstract

Bot-generated comments in online news discussions can reduce discussion quality and complicate content moderation, particularly when automated comments resemble natural human language. This study aims to evaluate IndoBERT for classifying Indonesian online-news comments into bot and human classes and to compare its performance with Random Forest as a conventional machine-learning baseline. The dataset consists of 1,000 comments collected through web scraping from six Indonesian news and discussion platforms, namely Detik.com, Kompas.com, Kompasiana.com, Kaskus.co.id, Quora.com, and CNN Indonesia. The dataset contains 500 bot comments and 500 human comments. Comments were labeled using rule-guided manual annotation based on repetitive patterns, external promotional links, textual similarity, templated structures, contextual irrelevance, and posting-frequency patterns. The data were preprocessed and divided into 70% training data and 30% evaluation data. IndoBERT was fine-tuned for binary classification using three training epochs. The evaluation produced 95.67% accuracy, 91.56% precision, 100% recall, 95.59% F1-score, and 0.9164 MCC, while Random Forest achieved an F1-score of 91.84%. The results indicate that IndoBERT achieved higher F1-score than Random Forest on the dataset and evaluation procedure used in this study. However, the findings should be interpreted within the study setting because external, temporal, cross-platform validation, repeated runs, and broader baseline comparisons were not performed. The contribution of the study lies in the empirical evaluation of IndoBERT for Indonesian online-news bot-comment classification, while the web application serves as an implementation artifact.

Downloads

Download data is not yet available.

References

[1] K. Hayawi, S. Saha, M. M. Masud, S. S. Mathew, and M. Kaosar, “Social media bot detection with deep learning methods: a systematic review,” Neural Comput. Appl., vol. 35, no. 12, pp. 8903–8918, 2023, doi: 10.1007/s00521-023-08352-z.

[10] A. M. Priyatno, M. M. Muttaqi, F. Syuhada, and A. Z. Arifin, "Detection of twitter spammer bots based on time interval entropy and global vectors for word representations tweet's hashtag," Regist. J. Ilm. Teknol. Last. Inf., vol. 5, no. 1, pp. 37–46, 2019, doi: 10.26594/register.v5i1.1382.

[11] M. Firdaus and P. Nur Miftahur Rizki, "BIJAKAWEB: A Web-Based Platform for Hate Speech Detection in Indonesian News Comments," J. Technology. Inf. and Computing Science., vol. 11, no. 4, pp. 939–948, 2024, doi: 10.25126/jtiik.1148719.

[12] R. Fatmasari, R. K. Septiani, T. H. Pinem, D. Fabiyanto, and W. Gata, "Implementation of the BERT Algorithm on Comments of Academic and Non-Academic Services of Open Universities on Social Media," Science, Apps. Computing and Technology. Inf., vol. 5, no. 2, p. 96, 2024, doi: 10.30872/jsakti.v5i2.13915.

[2] Z. Ellaky, F. Benabbou, and S. Ouahabi, “Systematic Literature Review of Social Media Bots Detection Systems,” vol. 35, 2023, doi: 10.1016/j.jksuci.2023.04.004.

[3] J. Devlin, M.-W. Chang, K. Lee, K. T. Google, and A. I. Language, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” Naacl-Hlt 2019, no. Mlm, pp. 4171–4186, 2018, [Online]. Available: https://aclanthology.org/N19-1423.pdf

[4] F. Koto and T. Baldwin, “IndoLEM and IndoBERT : A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP,” pp. 757–770, 2020.

[5] D. Duki, D. Keˇ, and D. Stipi, “Are You Human ? Detecting Bots on Twitter Using BERT,” 2020, doi: 10.1109/DSAA49011.2020.00089.

[6] M. Heidari and J. H. J. Jones, “Using BERT to Extract Topic-Independent Sentiment Features for Social Media Bot Detection,” 2020.

[7] Q. Guo, H. Xie, Y. Li, W. Ma, and C. Zhang, “Social bots detection via fusing bert and graph convolutional networks,” Symmetry (Basel)., vol. 14, no. 1, pp. 1–14, 2022, doi: 10.3390/sym14010030.

[8] S. Kumar, S. Garg, Y. Vats, and A. S. Parihar, “Content based bot detection using bot language model and BERT embeddings,” in Proceedings of ICCCSP, 2021, pp. 285–289. doi: 10.1109/ICCCSP52374.2021.9465506.

[9] A. Sallah et al., “Fine-Tuned Understanding: Enhancing Social Bot Detection with Transformer-Based Classification,” IEEE Access, vol. 12, no. August, pp. 118250–118269, 2024, doi: 10.1109/ACCESS.2024.3440657.

[13] S. Imron, E. I. Setiawan, and J. Santoso, "Detection of E-Commerce Review Aspects Using IndoBERT Embedding and CNN," J. Intell. Syst. Comput., vol. 5, no. 1, pp. 10–16, 2023, doi: 10.52985/insyst.v5i1.267.

[14] F. A. Pratama and A. Romadhony, "Identification of toxic comments with BERT," e-Proceeding Eng., vol. 7, no. 2, pp. 7941–7949, 2020, doi: 10.34818/eoe.v7i2.13073.

[15] A. D. Rendragraha, M. A. Bijaksana, and A. Romadhony, "Transformers method approach for the detection of foul language in Indonesian online news commentary," e-Proceeding Eng., vol. 8, no. 2, pp. 3385–3395, 2021.

[16] A. Fardhina et al., "A Natural Language Processing (NLP) Algorithm-Based Hoax Detection System Using Bert," vol. 4, no. 1, pp. 450–461, 2025, doi: 10.70247/jumistik.v4i1.156.

[17] M. Averina, W. B. T. Handaya, and A. J. Santoso, "Prediction of bot accounts on social media X using Random Forest, XGBoost, and SVM," J. Inform. Atma Jogja, vol. 6, no. 2, pp. 57–68, 2025, doi: 10.24002/jiaj.v6i2.12948.

[18] R. Tjut Adek, M. Fikry, and U. Khalil, “News Opinion Classification Application With Support Vector Machine Algorithm Using Framework Codeigniter,” J. Informatics Telecommun. Eng., vol. 5, no. 1, pp. 160–166, 2021, doi: 10.31289/jite.v5i1.5189.

[19] S. Rohman and I. W. P. Agung, "Comparison of IndoBERT, SVM, and Random Forest Algorithms in Sentiment Analysis of the Free Nutritious Meal Program," JATI, vol. 9, no. 6, pp. 9464–9471, 2025.

[20] A. Tharwat, “Classification assessment methods,” Appl. Comput. Informatics, vol. 17, no. 1, pp. 168–192, 2021, doi: 10.1016/j.aci.2018.08.003.

Downloads

Published

2026-10-06

How to Cite

[1]
M. Wanda, R. Rizal, and Y. Afrillia, “Bot Comment Detection in Indonesian Online News Using IndoBERT”, JAIC, vol. 10, no. 5, pp. 4505–4509, Oct. 2026.

Similar Articles

1 2 3 4 5 > >> 

You may also start an advanced similarity search for this article.