Two-Stage Maritime Anomaly Detection: Unsupervised Outlier Filtering and Optimized Bidirectional LSTM on Southeast Asian AIS Data

Abstract

This paper presents a two-stage framework for detecting anomalous vessel trajectories in Automatic Identification System (AIS) data from Southeast Asian waters, addressing challenges of high traffic density, diverse vessel behaviors, and severe class imbalance. The primary objective is to minimize missed threats while maintaining manageable false alarm rates in security-critical maritime surveillance systems. The research employs a hybrid approach combining unsupervised and supervised learning methods. In the first stage, DBSCAN and Isolation Forest algorithms filter noise and generate high-confidence outlier labels from 15,542 real-world vessel trajectories. Comparative analysis demonstrates substantial agreement between methods with Cohen's Kappa of 0.688 and 55.3% anomaly overlap, indicating complementary detection capabilities that enhance filtering robustness. In the second stage, a Bidirectional Long Short-Term Memory model is optimized through systematic hyperparameter tuning across 48 configurations, covering sequence length, network architecture, dropout rate, learning rate, and sampling strategies. Comprehensive baseline evaluation validates BiLSTM's suitability for security applications, achieving 15.41% F1-score improvement over unidirectional LSTM and 33% fewer false negatives compared to Bidirectional GRU alternative. The optimized BiLSTM attains F1-score of 0.5709 with precision 0.5444 and recall 0.6000, exhibiting 90.03% specificity for normal vessels and 76.17% sensitivity for anomalies. The model misses only 23.8% of threats while maintaining 9.97% false alarm rate, providing balanced performance suitable for human-verified security-critical maritime surveillance in Southeast Asian waters.