A Real-Time Hand Gesture Control of a Quadcopter Swarm Implemented in the Gazebo Simulation Environment

Authors

  • Ryan Satria Wijaya Politeknik Negeri Batam
  • Senanjung Prayoga Politeknik Negeri Batam
  • Rifqi Amalya Fatekha Politeknik Negeri Batam
  • Muhammad Thoriq Mubarak Politeknik Negeri Batam

DOI:

https://doi.org/10.30871/jaic.v9i3.9578

Keywords:

Hand Gesture Recognition, Quadcopter Swarm, Mediapipe, ROS, Gazebo Simulation

Abstract

With the advancement of technology, human-robot interaction (HRI) is becoming more intuitive, including through hand gesture-based control. This study aims to develop a real-time hand gesture recognition system to control a quadcopter swarm within a simulated environment using ROS and Gazebo. The system utilizes Google's MediaPipe framework for detecting 21 hand landmarks, which are then processed through a custom-trained neural network to classify 13 predefined gestures. Each gesture corresponds to a specific command such as basic motion, rotation, or swarm formation, and is published to the /cmd_vel topic using the ROS communication framework. Simulation tests were performed in Gazebo and covered both individual drone maneuvers and simple swarm formations. The results demonstrated a gesture classification accuracy of 90%, low latency, and stable response across multiple drones. This approach offers a scalable and efficient solution for real-time swarm control based on hand gestures, contributing to future applications in human-drone interaction systems.

Downloads

Download data is not yet available.

References

[1] F. Ren and Y. Bao, A review on human-computer interaction and intelligent robots, vol. 19, no. 1. 2020. doi: 10.1142/S0219622019300052.

[2] R. K. Megalingam, V. S. Naick, M. Motheram, J. Yannam, N. C. Gutlapalli, and V. Sivanantham, “Robot operating system based autonomous navigation platform with human robot interaction,” Telkomnika (Telecommunication Comput. Electron. Control., vol. 21, no. 3, pp. 675–683, 2023, doi: 10.12928/TELKOMNIKA.v21i3.24257.

[3] Indriani, M. Harris, and A. S. Agoes, “Applying Hand Gesture Recognition for User Guide Application Using MediaPipe,” Proc. 2nd Int. Semin. Sci. Appl. Technol. (ISSAT 2021), vol. 207, no. Issat, pp. 101–108, 2021, doi: 10.2991/aer.k.211106.017.

[4] Yaseen, O. J. Kwon, J. Kim, S. Jamil, J. Lee, and F. Ullah, “Next-Gen Dynamic Hand Gesture Recognition: MediaPipe, Inception-v3 and LSTM-Based Enhanced Deep Learning Model,” Electron., vol. 13, no. 16, pp. 1–11, 2024, doi: 10.3390/electronics13163233.

[5] Y. Meng, H. Jiang, N. Duan, and H. Wen, “Real-Time Hand Gesture Monitoring Model Based on MediaPipe’s Registerable System,” Sensors, vol. 24, no. 19, 2024, doi: 10.3390/s24196262.

[6] B. Latif, N. Buckley, and E. L. Secco, “Hand Gesture and Human-Drone Interaction,” Lect. Notes Networks Syst., vol. 544 LNNS, no. October, pp. 299–308, 2023, doi: 10.1007/978-3-031-16075-2_20.

[7] A. S. Andoy, R. T. Tayane, and I. Supriadi, “Controlling the Dji Ryze Tello Drone Using Human Hand Gestures,” vol. 2, no. 2, pp. 123–132, 2024.

[8] H. Lee and S. Park, “Sensing-Aware Deep Reinforcement Learning With HCI-Based Human-in-the-Loop Feedback for Autonomous Nonlinear Drone Mobility Control,” IEEE Access, vol. 12, no. December 2023, pp. 1727–1736, 2024, doi: 10.1109/ACCESS.2023.3346917.

[9] A. Tahir, J. Böling, M. H. Haghbayan, H. T. Toivonen, and J. Plosila, “Swarms of Unmanned Aerial Vehicles — A Survey,” J. Ind. Inf. Integr., vol. 16, no. October, p. 100106, 2019, doi: 10.1016/j.jii.2019.100106.

[10] M. Yoo et al., “Motion Estimation and Hand Gesture Recognition-Based Human–UAV Interaction Approach in Real Time,” Sensors, vol. 22, no. 7, 2022, doi: 10.3390/s22072513.

[11] A. Muhamad, S. D. Panjaitan, and R. R. Yacoub, “Design and Development of Flight Controller for Quadcopter Drone Control,” Telecommun. Comput. Electr. Eng. J., vol. 1, no. 3, p. 279, 2024, doi: 10.26418/telectrical.v1i3.73681.

[12] G. Ostojić, S. Stankovski, B. Tejić, N. Dukić, and S. Tegeltija, “Design, control and application of quadcopter,” Int. J. Ind. Eng. Manag., vol. 6, no. 1, pp. 43–48, 2015, doi: 10.24867/ijiem-2015-1-106.

[13] A. Budiyanto, M. I. Ramadhan, I. Burhanudin, H. H. Triharminto, and B. Santoso, “Navigation control of Drone using Hand Gesture based on Complementary Filter Algorithm,” J. Phys. Conf. Ser., vol. 1912, no. 1, 2021, doi: 10.1088/1742-6596/1912/1/012034.

[14] M. Wameed and A. M. Alkamachi, “International Journal of INTELLIGENT SYSTEMS AND APPLICATIONS IN ENGINEERING Hand Gestures Robotic Control Based on Computer Vision,” Orig. Res. Pap. Int. J. Intell. Syst. Appl. Eng. IJISAE, vol. 2023, no. 2, pp. 1013–1021, 2024, [Online]. Available: www.ijisae.org

[15] D. Marek et al., “Swarm of Drones in a Simulation Environment—Efficiency and Adaptation,” Appl. Sci., vol. 14, no. 9, 2024, doi: 10.3390/app14093703.

[16] A. Alvin, N. H. Shabrina, A. Ryo, and E. Christian, “Hand Gesture Detection for Sign Language using Neural Network with Mediapipe,” Ultim. Comput. J. Sist. Komput., vol. 13, no. 2, pp. 57–62, 2021, doi: 10.31937/sk.v13i2.2109.

[17] M. Arif et al., “Teknik dan Multimedia Sistem Pendeteksi Tangan Berbasis Mediapipe dan OpenCV untuk Pengenalan Gerakan,” Biner J. Ilmu Komput., vol. 2, no. 2, pp. 173–177, 2024, [Online]. Available: https://journal.mediapublikasi.id/index.php/Biner

[18] A. Maarif, W. Rahmaniar, M. A. M. Vera, A. A. Nuryono, R. Majdoubi, and A. Cakan, “Artificial Potential Field Algorithm for Obstacle Avoidance in UAV Quadrotor for Dynamic Environment,” 10th IEEE Int. Conf. Commun. Networks Satell. Comnetsat 2021 - Proc., no. October, pp. 184–189, 2021, doi: 10.1109/COMNETSAT53002.2021.9530803.

[19] S. Nur Budiman, S. Lestanti, S. Marselius Evvandri, and R. Kartika Putri, “Pengenalan Gestur Gerakan Jari Untuk Mengontrol Volume Di Komputer Menggunakan Library Opencv Dan Mediapipe,” Antivirus J. Ilm. Tek. Inform., vol. 16, no. 2, pp. 223–232, 2022, doi: 10.35457/antivirus.v16i2.2508.

[20] O. Mishra, P. Suryawanshi, Y. Singh, and S. Deokar, “A Mediapipe-Based Hand Gesture Recognition Home Automation System,” 2023 2nd Int. Conf. Futur. Technol. INCOFT 2023, no. April 2024, 2023, doi: 10.1109/INCOFT60753.2023.10425411.

[21] A. D. Agustiani, S. M. Putri, P. Hidayatullah, and M. R. Sholahuddin, “Penggunaan MediaPipe untuk Pengenalan Gesture Tangan Real-Time dalam Pengendalian Presentasi,” vol. 16, no. 2, 2024.

[22] U. C. Patkar, V. Mandhalkar, A. Chavan, S. Songire, and H. Kothawade, “Robot Operating System: A Comprehensive Analysis and Evaluation,” Int. J. Intell. Syst. Appl. Eng., vol. 12, no. 7s, pp. 516–520, 2024.

[23] Z. Wei, F. Tian, Z. Qiu, Z. Yang, R. Zhan, and J. Zhan, “Research on Machine Vision-Based Control System for Cold Storage Warehouse Robots,” Actuators, vol. 12, no. 8, 2023, doi: 10.3390/act12080334.

[24] S. Alexovič, M. Lacko, and J. Bačík, “Simulation of Multiple Autonomous Mobile Robots Using a Gazebo Simulator,” Lect. Notes Networks Syst., vol. 596 LNNS, no. January, pp. 339–351, 2023, doi: 10.1007/978-3-031-21435-6_30.

[25] S. Hadri, “Hand gestures for drone control using deep learning,” no. December 2018, 2020, doi: 10.13140/RG.2.2.15939.02089.

Downloads

Published

2025-06-20

How to Cite

[1]
R. S. Wijaya, S. Prayoga, R. A. Fatekha, and M. T. Mubarak, “A Real-Time Hand Gesture Control of a Quadcopter Swarm Implemented in the Gazebo Simulation Environment”, JAIC, vol. 9, no. 3, pp. 979–988, Jun. 2025.

Issue

Vol. 9 No. 3 (2025): June 2025

Section

Articles

License

Copyright (c) 2025 Ryan Satria Wijaya, Senanjung Prayoga, Rifqi Amalya Fatekha, Muhammad Thoriq Mubarak

Creative Commons License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Authors who publish with this journal agree to the following terms:

  • Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License (Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) ) that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
  • Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
  • Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).