IoT-Based Sign Language Translation System for Deaf Individuals
DOI:
https://doi.org/10.30871/jaic.v10i4.13742Keywords:
Arduino, Flex Sensor, GY-91, IoT, SIBI, Threshold MethodAbstract
Deaf-mute individuals face significant communication barriers due to limited public familiarity with sign language. In Indonesia, SIBI (Sistem Isyarat Bahasa Indonesia) is the government-standardised one-handed finger-spelling system used as the basis of communication for the hearing-impaired. This paper presents the design, implementation, and evaluation of an IoT-based hand sign language translator glove that recognises all 26 SIBI alphabet letters and displays the result on an Android application. The glove integrates five flex sensors for finger-bending detection, a GY-91 module (MPU-9250 + BMP280) for wrist orientation measurement, and a CD4051 analog multiplexer, all processed by a Wemos D1 Mini (ESP8266) microcontroller. Sensor readings are classified using a threshold-based decision method calibrated across three subjects. Classified letter data are transmitted via MQTT over Wi-Fi to a cloud broker and rendered in real time by an Android application. Experimental evaluation covers flex sensor resistance characterisation for all 26 SIBI letters, GY-91 gyroscope orientation profiling, multi-subject threshold calibration, end-to-end application display accuracy, and voltage measurement error percentage. Results confirm that the combined flex-sensor and gyroscope approach identifies SIBI alphabet letters with 90.00% end-to-end display accuracy and a voltage measurement error below 5%, indicating the preliminary feasibility of a low-cost, single-hand wearable IoT glove as an assistive sign-language communication aid, based on testing with a small number of participants. The system in its current form translates individual static SIBI alphabet letters only; it does not yet recognise dynamic gestures, whole words, or continuous sentence-level sign language.
Downloads
References
[1] World Health Organization, "Deafness and hearing loss," WHO Fact Sheet, 2023. [Online]. Available: https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss
[2] Badan Pusat Statistik, "Hasil Sensus Penduduk 2020," BPS Indonesia, Jakarta, 2021.
[3] A. P. Husain, A. S. Nugraheni, and H. Unayah, "Optimalisasi Penggunaan Bahasa Isyarat Dengan SIBI Dan BISINDO Pada Mahasiswa Difabel Tunarungu Di Prodi PGMI UIN Sunan Kalijaga," HOLISTIKA: Jurnal Ilmiah PGSD, vol. 4, no. 1, 2020.
[4] R. Hartanto, "Prototipe Sistem Pengenalan Bahasa Isyarat Abjad Finger Spelling," Tugas Akhir, Universitas Gadjah Mada, Yogyakarta, 2015.
[5] H. Lestari and P. Lestari, "Sarung Tangan Penerjemah Huruf SIBI (A-Z)," Jurnal Inovasi Teknologi Terapan, vol. 1, no. 1, 2020.
[6] L. A. Dipietro, A. M. Sabatini, and P. Dario, "A survey of glove-based systems and their applications," IEEE Trans. Syst. Man Cybern. C, vol. 38, no. 4, pp. 461-482, Jul. 2008.
[7] J. Kim, S. Park, and J. Lee, "Sign language recognition system using flex sensors and IMU sensor," Sensors, vol. 21, no. 8, p. 2796, 2021.
[8] S. A. Mehdi and Y. N. Khan, "Sign language recognition using sensor gloves," in Proc. 9th Int. Conf. Neural Information Processing (ICONIP), Singapore, 2002, pp. 2204-2206.
[9] P. Kakoty, S. M. Hazarika, and D. Bhagat, "Effect of finger angle on flex sensor output characteristics," in Proc. Int. Conf. Advances in Computing, Communications and Informatics (ICACCI), 2015, pp. 1836-1840.
[10] H. A. Al-Kashoash, H. Kharrufa, Y. Al-Nidawi, and A. H. Kemp, "Congestion control for IoT-based wireless sensor networks: A comparison," in Proc. IEEE Int. Conf. Communications (ICC), 2017, pp. 1-6.
[11] A. Hussain, R. Wenbi, A. L. da Silva, M. Nadher, and M. Mudhish, "Health and emergency-care platform for the elderly and disabled people in smart cities using IoT protocol," J. Syst. Softw., vol. 110, pp. 253-263, 2015.
[12] A. Becker, P. Nickel, and B. Loewe, "Real-time gesture recognition on microcontrollers using threshold-based classification," in Proc. IEEE Sensors, 2019, pp. 1-4.
[13] M. S. Islam, M. T. Islam, A. N. Almutairi, N. A. Beng, N. Misran, and N. Amin, "Monitoring of the human body signal through the IoT based HealthCare system and mobile application," Information, vol. 10, no. 3, p. 97, 2019.
[14] Spectra Symbol, "Flex Sensor 4.5 Datasheet," Spectra Symbol Corp., Salt Lake City, UT, USA, Rev. E, 2014.
[15] InvenSense, "MPU-9250 Product Specification Revision 1.1," TDK InvenSense, San Jose, CA, USA, 2016.
[16] T. Susanto, "Kemampuan Berbicara Anak Penyandang Amnestik Aphasia," Jurnal Teknodik, vol. XVI, no. 1, 2012.
[17] J. Brooke, "SUS: A quick and dirty usability scale," in Usability Evaluation in Industry, P. W. Jordan, B. Thomas, B. A. Weerdmeester, and I. L. McClelland, Eds. London, U.K.: Taylor & Francis, 1996, pp. 189–194.
[18] D. Dinculeană and X. Cheng, "Vulnerabilities and limitations of MQTT protocol used between IoT devices," Appl. Sci., vol. 9, no. 5, p. 848, 2019.
[19] Espressif Systems, "ESP8266EX datasheet, version 6.0," Espressif Systems, Shanghai, China, 2020.
[20] N. C. Camgoz, S. Hadfield, O. Koller, H. Ney, and R. Bowden, "Neural sign language translation," in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2018, pp. 7784–7793.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Nafla Zahira Semry, Iman Fahruzi
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License (Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) ) that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
