LSTM-Based Hand Gesture Recognition for Indonesian Sign Language System (SIBI) on Affix, Alphabet, Number, and Word
DOI:
https://doi.org/10.30871/jaic.v9i3.9607Keywords:
Deep Learning, Hand Gesture Recognition, Indonesian Sign Language System (SIBI), LSTMAbstract
Sign language plays a critical role in enabling communication for the Deaf and hard-of-hearing community in Indonesia, yet there remains a significant gap in technological support for recognizing the official Indonesian sign language, Sistem Isyarat Bahasa Indonesia (SIBI). This study presents a deep learning-based hand gesture recognition system for SIBI, focusing on four primary gesture categories: affix, alphabet, number, and word. A large and diverse dataset of 21,351 videos was collected, covering 18 affix, 26 alphabet, 35 number, and 29 word classes. Hand keypoints were extracted using MediaPipe Holistic, and a bidirectional long short-term memory (BiLSTM) model was trained using 5-fold stratified cross-validation. The model achieved high recognition performance in the alphabet, number, and word categories, with mean test accuracies of 93.94%, 91.48%, and 92.41%, respectively, and slightly lower performance in the affix category at 68.17%. The affix category posed particular challenges due to subtle hand shape differences and high variability between signers, while the alphabet category consistently showed the highest accuracy due to its distinct and static handshapes. Evaluation metrics, including precision, recall, F1-score, and confusion matrix analysis, provided further insights into model strengths and limitations. Overall, the study demonstrates the effectiveness of LSTM models for sequential hand gesture recognition in SIBI and highlights areas for future improvement, such as handling non-manual features and improving generalization across signers.
Downloads
References
[1] Badan Pusat Statistik, “Jumlah Penduduk Berumur 5 Tahun ke Atas menurut Kelompok Umur, Daerah Perkotaan/Perdesaan, Jenis Kelamin, dan Tingkat Kesulitan Mendengar, di INDONESIA - Dataset - Long Form Sensus Penduduk 2022,” Badan Pusat Statistik. Accessed: May 02, 2025. [Online]. Available: https://sensus.bps.go.id/topik/tabular/sp2022/145/0/0
[2] N. Napsiah and Y. T. Wijayanti, “Indonesian Society is Not Disabled Friendly?,” Jurnal Ilmu Sosial, vol. 22, no. 1, pp. 147–164, Jun. 2023, doi: 10.14710/JIS.22.1.2023.147-164.
[3] M. Nur Iman, “Sign Language and Culture: Understanding Communication in the Deaf Community,” in Proceeding of the International Conference on Social Sciences and Humanities Innovation, Asosiasi Peneliti Dan Pengajar Ilmu Sosial Indonesia, 2024, pp. 156–166. [Online]. Available: https://prosiding.appisi.or.id/index.php/ICSSHI
[4] R. S. Fauzi, B. Irmawati, and N. Agitha, “KADARING SIBI (Indonesian Sign System Online Dictionary): Web-based Indonesian Sign System Learning App,” in Proceedings of the First Mandalika International Multi-Conference on Science and Engineering 2022, MIMSE 2022 (Informatics and Computer Science), Atlantis Press International BV, Dec. 2022, pp. 427–436. doi: 10.2991/978-94-6463-084-8_35.
[5] Y. Arief, “Personal Interview with AUDISI Foundation,” Oct. 12, 2024, Jakarta.
[6] I. Damayanti and S. H. Purnamasari, “Relationship between Communication Barriers and Stress in Parents with Deaf Children in Elementary Level Special Needs School in Pekanbaru,” Indonesian Journal of Disability Studies, vol. 6, no. 1, pp. 14–20, May 2019, doi: 10.21776/UB.IJDS.2019.006.01.2.
[7] S. N. Budiman, S. Lestanti, H. Yuana, and B. N. Awwalin, “SIBI (Sistem Bahasa Isyarat Indonesia) berbasis Machine Learning dan Computer Vision untuk Membantu Komunikasi Tuna Rungu dan Tuna Wicara,” Jurnal Teknologi dan Manajemen Informatika, vol. 9, no. 2, pp. 119–128, 2023, Accessed: May 02, 2025. [Online]. Available: http://jurnal.unmer.ac.id/index.php/jtmi
[8] E. Rakun, A. M. Arymurthy, L. Y. Stefanus, A. F. Wicaksono, and I. W. W. Wisesa, “Recognition of Sign Language System for Indonesian Language Using Long Short-Term Memory Neural Networks,” Adv Sci Lett, vol. 24, no. 2, pp. 999–1004, Mar. 2018, doi: 10.1166/asl.2018.10675.
[9] S. Hidayat, Y. V. Via, and E. P. Mandyartha, “Penerapan Model Hybrid Convolutional Neural Network dan Long Short-Term Memory untuk Pengenalan Real-Time Sistem Isyarat Bahasa Indonesia (SIBI),” JURNAL MEDIA INFORMATIKA BUDIDARMA, vol. 8, no. 3, p. 1586, Jul. 2024, doi: 10.30865/mib.v8i3.7837.
[10] F. X. L. Riberu, “Sistem Deteksi Simbol pada SIBI (SISTEM ISYARAT BAHASA INDONESIA) Secara Real-Time Menggunakan Mediapipe dan LSTM,” Universitas Dinamika, 2023.
[11] I. D. M. B. A. Darmawan et al., “Advancing Total Communication in SIBI: A Proposed Conceptual Framework for Sign Language Translation,” in Proceedings - International Conference on Smart-Green Technology in Electrical and Information Systems, ICSGTEIS, Institute of Electrical and Electronics Engineers Inc., Nov. 2023, pp. 23–28. doi: 10.1109/ICSGTEIS60500.2023.10424020.
[12] I. D. M. B. A. Darmawan, Linawati, G. Sukadarmika, N. M. A. E. D. Wirastuti, and R. Pulungan, “Temporal Action Segmentation in Sign Language System for Bahasa Indonesia (SIBI) Videos Using Optical Flow-Based Approach,” Jurnal Ilmu Komputer dan Informasi, vol. 17, no. 2, pp. 195–202, Jun. 2024, doi: 10.21609/jiki.v17i2.1284.
[13] Lembaga Penelitian dan Pengembangan Sistem Isyarat Bahasa Indonesia, “Kamus SIBI.” Accessed: May 08, 2025. [Online]. Available: https://pmpk.kemdikbud.go.id/sibi/kosakata/imbuhan
[14] I. D. M. B. A. Darmawan, L. Linawati, G. Sukadarmika, N. M. A. E. D. Wirastuti, and R. Pulungan, “Indonesian Sign Language System (SIBI) Dataset,” Mendeley Data, vol. 3, Aug. 2024, doi: 10.17632/44PBRBSNKH.3.
[15] Y. Meng, H. Jiang, N. Duan, and H. Wen, “Real-Time Hand Gesture Monitoring Model Based on MediaPipe’s Registerable System,” Sensors 2024, Vol. 24, Page 6262, vol. 24, no. 19, p. 6262, Sep. 2024, doi: 10.3390/S24196262.
[16] I. Galanakis, R. F. Soldatos, N. Karanikolas, A. Voulodimos, I. Voyiatzis, and M. Samarakou, “A MediaPipe Holistic Behavior Classification Model as a Potential Model for Predicting Aggressive Behavior in Individuals with Dementia,” Applied Sciences (Switzerland), vol. 14, no. 22, p. 10266, Nov. 2024, doi: 10.3390/APP142210266/S1.
[17] M. S. Jayaprada et al., “Real-Time Hand Gestures Recognition System,” International Journal of Innovative Research in Technology, vol. 11, no. 11, pp. 4948–4951, 2025, doi: 10.33168/JSMS.2022.0225.
[18] A. Graves and J. Schmidhuber, “Framewise phoneme classification with bidirectional LSTM and other neural network architectures,” Neural Networks, vol. 18, no. 5–6, pp. 602–610, Jul. 2005, doi: 10.1016/J.NEUNET.2005.06.042.
[19] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” The Journal of Machine Learning Research, vol. 15, pp. 1929–1958, Jan. 2014, doi: 10.5555/2627435.2670313.
[20] H. il Lim, “A Study on Dropout Techniques to Reduce Overfitting in Deep Neural Networks,” in Lecture Notes in Electrical Engineering, Springer, Singapore, Dec. 2020, pp. 133–139. doi: 10.1007/978-981-15-9309-3_20.
[21] PyTorch, “CrossEntropyLoss — PyTorch 2.7 documentation.” Accessed: May 05, 2025. [Online]. Available: https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html
[22] O. Rainio, J. Teuho, and R. Klén, “Evaluation metrics and statistical tests for machine learning,” Sci Rep, vol. 14, no. 1, pp. 1–14, Dec. 2024, doi: 10.1038/S41598-024- 66611-y.
[23] C. Miller, T. Portlock, D. M. Nyaga, and J. M. O’Sullivan, “A review of model evaluation metrics for machine learning in genetics and genomics,” Frontiers in Bioinformatics, vol. 4, p. 1457619, Sep. 2024, doi: 10.3389/FBINF.2024.1457619/XML/NLM.
[24] S. Sathyanarayanan and B. R. Tantri, “Confusion Matrix-Based Performance Evaluation Metrics,” African Journal of Biomedical Research, vol. 27, no. 4S, pp. 4023–4031, Nov. 2024, doi: 10.53555/AJBR.V27I4S.4345.
[25] I. K. Nti, O. Nyarko-Boateng, and J. Aning, “Performance of Machine Learning Algorithms with Different K Values in K-fold Cross-Validation,” International Journal of Information Technology and Computer Science, vol. 13, no. 6, pp. 61–71, Dec. 2021, doi: 10.5815/IJITCS.2021.06.05.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Patricia Ho, Handri Santoso
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License (Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) ) that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
