Comparative Analysis of Siamese BiLSTM and IndoBERT for Semantic Textual Similarity Detection in Functional Scientific Papers of Meteorology, Climatology, and Geophysics Officers at BMKG

Abstract

The competency assessment process for Meteorology, Climatology, and Geophysics (PMG) functional officers at BMKG requires manual review of Scientific Paper (KTI) similarity, a task that is time-consuming, prone to subjectivity, and increasingly burdensome as submission volume grows. While Semantic Textual Similarity (STS) research has advanced considerably for high-resource languages, empirical evidence for Indonesian-language technical documents in specialized scientific domains remains limited, and prior work has not established which neural architecture is preferable under such data-constrained conditions. This study addresses that gap by empirically comparing two Siamese Network architectures, Siamese BiLSTM and IndoBERT, for automatic STS detection on 87 PMG KTI documents, yielding 3,741 document pairs automatically labeled using the 90th percentile of TF-IDF cosine similarity scores and validated against manual annotation (Cohen's Kappa κ=0.82). Siamese BiLSTM employs Word2Vec embeddings with Focal Loss, while IndoBERT fine-tunes the pretrained indobert-base-p1 model with Contrastive Loss; both apply class weighting to address the 90:10 class imbalance, with classification thresholds independently calibrated via grid search. Evaluated on 1,123 held-out test pairs, Siamese BiLSTM achieves an F1-Score of 64.84% (Accuracy 91.99%, Precision 58.04%, Recall 73.45%) at threshold 0.60, outperforming IndoBERT's F1-Score of 57.73% (Accuracy 89.05%, Precision 47.19%, Recall 74.34%) at threshold 0.935, a difference confirmed statistically significant by McNemar's test (χ²=7.6992; p=0.0055). This result runs counter to the common assumption that Transformer-based models universally outperform recurrent architectures, suggesting that smaller, domain-tuned embeddings can be more effective under limited-data, domain-specific conditions. The better-performing Siamese BiLSTM model, requiring only 16.77 MB with no GPU dependency, was deployed as a Streamlit web application, enabling the BMKG PMG Assessment Team to perform similarity detection quickly and consistently within their existing workflow.