Comparative Analysis Of Automatic Labeling With Cohen's Kappa Validation For Cyberbullying On Tiktok Using Pre-Trained Transformers

Abstract

The rapid growth of TikTok users in Indonesia increases the risk of cyberbullying, making manual content moderation inefficient. This study compares two automatic labeling methods, InSet Lexicon and Zero-Shot Classification, to address the problem of limited labeled data in cyberbullying detection. A total of 5,864 comment data were collected through scraping techniques and processed through comprehensive text preprocessing stages, including slang normalization. To evaluate labeling reliability, a validation test using Cohen's Kappa Score metric was conducted against a human annotated Gold Standard of 1,141 comments. The results show that Zero-Shot Classification achieves high reliability with a Kappa score of 0.9132 (almost perfect agreement), outperforming InSet Lexicon which drops to 0.2585 (fair agreement) due to lexical rigidity and high false positives on casual slang. The automatically labeled datasets were balanced using Random Oversampling (for the Zero-Shot Classification labeling scenario) and split via a stratified 80:10:10 ratio to fine-tune IndoBERT and RoBERTa. On independent test data, IndoBERT trained on Zero-Shot labels delivers the best performance, reaching an Accuracy and F1-Score of 91.10%, outperforming RoBERTa under the same scenario (85.83%). Conversely, training on InSet Lexicon labels reduces performance, limiting IndoBERT to an 86.94% F1-Score and RoBERTa to 80.21%. This study concludes that using Zero-Shot Classification and fine-tuning IndoBERT is more optimal for application to informal social media comment moderation systems.