Ablation Study of Attention Pooling and Adaptive Focal Loss for Imbalanced Indonesian Sentiment Classification Using IndoBERT

Aldi Rifaldi Saputra; Eka Miranda

doi:10.30871/jaic.v10i5.14173

Authors

Aldi Rifaldi Saputra Bina Nusantara University
Eka Miranda Bina Nusantara University

DOI:

https://doi.org/10.30871/jaic.v10i5.14173

Keywords:

Sentiment Analysis, IndoBert, Focal Loss, Supervised Contrastive Loss, Naive Bayes

Abstract

Sentiment classification of Indonesian YouTube comments remains challenging because informal language, target ambiguity, and severe class imbalance can obscure minority-class performance. This study evaluates the individual and combined contributions of attention pooling, supervised contrastive loss (SCL), and dynamic adaptive focal loss to IndoBERT using a complete 2×2×2 full-factorial ablation design. From an initial 75,453 comments, temporal filtering using 21 August 2026 as the case reference date retained 67,633 comments. After removing 159 irrelevant comments and 2,721 exact duplicates, 64,753 weak-labelled comments remained, with a majority-to-minority ratio of 22.24. A disproportionate stratified sample of 500 comments was independently audited by a human annotator; among 498 valid sentiment items, LLM-to-human agreement reached a Macro-F1 of 0.5389 and Cohen’s kappa of 0.3146, indicating substantial weak-label noise. The dataset was split 80:10:10 using a fixed stratified split, and each IndoBERT variant was trained with three seeds (42, 123, and 3407). Plain IndoBERT achieved a mean Macro-F1 of 0.9566 ± 0.0064. The highest observed mean was obtained by the dynamic-focal-only variant at 0.9595 ± 0.0023, closely followed by the full configuration at 0.9595 ± 0.0011. Paired marginal comparisons showed small, context-dependent effects, with all 95% confidence intervals crossing zero. Overall, most gains over TF-IDF baselines came from IndoBERT’s contextual representations, while the added components produced comparatively small and unstable improvements.

Downloads

Download data is not yet available.

Author Biography

Eka Miranda, Bina Nusantara University

Information Systems Department, School of Information Systems, Bina Nusantara University, Jakarta, Indonesia

References

[1] R. Al Balawi, Y. Hu, and L. Qiu, “Brand crisis and customer relationship management on social media: Evidence from a natural experiment from the airline industry,” Information Systems Research, vol. 34, no. 2, pp. 442–462, 2023.

[2] K. Nandini and M. Rahardi, “Sentiment Analysis of Economic Policy Comments on YouTube Using Ensemble Machine Learning,” Journal of Applied Informatics and Computing, vol. 9, no. 5, pp. 2607–2615, 2025.

[3] I. T. Shandy and U. P. Sanjaya, “Comparative Analysis of Machine Learning and IndoBERT Models for Sentiment Analysis of YouTube Comments on the Free Nutritious Meals Program,” Journal of Applied Informatics and Computing, vol. 10, no. 3, pp. 2391–2399, 2026.

[4] A. P. Widyasari and M. Muljono, “Sentiment Analysis of YouTube Comments on the 2025 DPR RI Demonstration Using Machine Learning,” Journal of Applied Informatics and Computing, vol. 10, no. 2, pp. 1729–1740, 2026.

[5] M. C. Hinojosa Lee, J. Braet, and J. Springael, “Performance metrics for multilabel emotion classification: comparing micro, macro, and weighted f1-scores,” Applied Sciences, vol. 14, no. 21, p. 9863, 2024.

[6] S. Farhadpour, T. A. Warner, and A. E. Maxwell, “Selecting and interpreting multiclass loss and accuracy assessment metrics for classifications with class imbalance: Guidance and best practices,” Remote Sens. (Basel)., vol. 16, no. 3, p. 533, 2024.

[7] H. Jayadianti, W. Kaswidjanti, A. T. Utomo, S. Saifullah, F. A. Dwiyanto, and R. Drezewski, “Sentiment analysis of Indonesian reviews using fine-tuning IndoBERT and R-CNN,” ILKOM Jurnal Ilmiah, vol. 14, no. 3, pp. 348–354, 2022.

[8] E. Yulianti and N. K. Nissa, “ABSA of Indonesian customer reviews using IndoBERT: single-sentence and sentence-pair classification approaches,” Bulletin of Electrical Engineering and Informatics, vol. 13, no. 5, pp. 3579–3589, 2024.

[9] R. I. Perwira, V. A. Permadi, D. I. Purnamasari, and R. P. Agusdin, “Domain-specific fine-tuning of IndoBERT for aspect-based sentiment analysis in Indonesian travel user-generated content,” Journal of Information Systems Engineering and Business Intelligence, vol. 11, no. 1, pp. 30–40, 2025.

[10] M. F. Kono, I. N. Fajri, and Y. Pristyanto, “Public Sentiment Analysis on Corruption Issues in Indonesia Using IndoBERT Fine-Tuning, Logistic Regression, and Linear SVM,” Journal of Applied Informatics and Computing, vol. 9, no. 5, pp. 2616–2628, 2025.

[11] M. Teddy Oswaria, T. Yusnitasaria, S. Wijaya, and J. M. Raya, “Sentiment Analysis of Indonesian YouTube Reviews About Lesbian, Gay, Bisexual, and Transgender (LGBT) using IndoBERT Fine Tuning,” LONTAR KOMPUTER, vol. 15, no. 1, Apr. 2024.

[12] A. S. Wijaya, A. Fahmi, and Y. T. C. Pramudi, “Sentiment Classification of Health Education YouTube Comments Using IndoBERT Embeddings with Logistic Regression and Naïve Bayes,” Journal of Applied Informatics and Computing, vol. 10, no. 3, pp. 3005–3020, 2026.

[13] S. Pradana and E. Kartikadarma, “A Comparison of Classical Machine Learning and IndoBERT on Sentiment Analysis of Danantara Program in X,” Journal of Applied Informatics and Computing, vol. 10, no. 4, pp. 3954–3964, 2026.

[14] C. Shaw, P. LaCasse, and L. Champagne, “Exploring emotion classification of indonesian tweets using large scale transfer learning via IndoBERT,” Soc. Netw. Anal. Min., vol. 15, no. 1, p. 22, 2025.

[15] J. Seo, S. Lee, L. Liu, and W. Choi, “TA-SBERT: Token attention sentence-BERT for improving sentence representation,” IEEE Access, vol. 10, pp. 39119–39128, 2022.

[16] B. Chen, W. Peng, and J. Song, “MAI-CBert: Multidirectional Attention Interaction Construction-Bert Sentiment Sentence Representation,” IEEE Access, vol. 10, pp. 70295–70304, 2022.

[17] Z. Labd, S. Bahassine, and K. Housni, “AFL-BERT: Enhancing Minority Class Detection in Multi-Label Text Classification with Adaptive Focal Loss and BERT.,” International Journal of Advanced Computer Science & Applications, vol. 16, no. 7, p. 496, 2025.

[18] M. Wu, “Label knowledge-guided heterogeneous graph contrastive learning for semi-supervised short text sentiment classification,” J. Big Data, vol. 12, no. 1, p. 227, 2025.

[19] L. Qian, Q. Zuo, D. Li, and H. Zhu, “DGMSCL: A dynamic graph mixed supervised contrastive learning approach for class imbalanced multivariate time series classification,” Neural Networks, vol. 185, p. 107131, 2025.

[20] D. R. Alfinsyah and B. P. Hartato, “Evaluating the Impact of Random Over Sampling on IndoBERT Performance for Indonesian Sentiment Analysis,” Journal of Applied Informatics and Computing, vol. 9, no. 6, pp. 3270–3282, 2025.

[21] T. Aprilah and W. Herowati, “Enhancing aspect-based sentiment analysis via Hugging Face fine-tuned IndoBERT,” Journal of Applied Informatics and Computing, vol. 9, no. 6, pp. 3821–3830, 2025.

[22] M. N. F. Hidayat, D. D. Prasetya, and T. Widiyaningtyas, “Sentiment analysis of emoji and latinized Arabic in Indonesian YouTube comments: A LABERT-LSTM model,” Journal of Applied Engineering and Technological Science, vol. 6, no. 2, pp. 1324–1341, 2025.

[23] W. Chen, F. Lin, G. Li, and B. Liu, “A survey of automatic sarcasm detection: Fundamental theories, formulation, datasets, detection methods, and opportunities,” Neurocomputing, vol. 578, p. 127428, 2024.

[24] A. H. Nasution and A. Onan, “ChatGPT label: Comparing the quality of human-generated and LLM-generated annotations in low-resource language NLP tasks,” Ieee Access, vol. 12, pp. 71876–71900, 2024.

[25] B. Wilie et al., “IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding,” in Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, 2020, pp. 843–857.