Real-Time BISINDO Gesture Detection Using YOLOv8 for a Web-Based Text-to-Speech Prototype

Abstract

Communication challenges persist between the deaf and hard-of-hearing community and the general public, largely due to limited awareness and understanding of Indonesian Sign Language (BISINDO) in the broader population. This study develops a real-time web-based system that identifies BISINDO gestures using the YOLOv8 object detection model and converts the recognized gesture sequence into spoken words via a text-to-speech function. The study is based on the CRISP-ML(Q) framework, which includes stages such as understanding the data, preparing the data, building models, assessing their performance, implementing them in real-world applications, and continuously tracking their effectiveness. A total of 1,550 images were independently collected using a laptop camera and categorized into 31 classes, including 30 BISINDO gesture classes and 1 class for negative samples. The dataset was split using a stratified method, allocating 80% for training, 10% for validation, and 10% for testing. The YOLOv8 model was trained on Google Colaboratory using a T4 GPU runtime. The evaluation results indicate that the model achieved a precision of 0.979, a recall of 0.976, an mAP50 of 0.993, and an mAP50-95 of 0.839, which demonstrates robust performance in detecting BISINDO gestures. The developed model was incorporated into a web prototype, with FastAPI serving as the backend and HTML, CSS, and JavaScript utilized for the frontend. The system can identify hand gestures using a webcam, show bounding boxes with corresponding labels, compile the detected gestures into simple text sequences, and produce speech from that information. Latency testing revealed an average response time of around 210 milliseconds when the model was deployed locally and approximately 670 milliseconds when deployed online via Hugging Face Spaces. The system still faces challenges in identifying similar-looking gestures and is influenced by factors such as lighting, hand placement, and the availability of hosting resources. These results indicate that YOLOv8s is effective for detecting the 30 static BISINDO gesture classes evaluated in this study within a controlled data collection setting, though further validation is required before the system can be considered suitable for broader real-world assistive communication use.