Chat GPT Impact Analysis on API Testing: A Controlled Experiment
Abstract
This research examines the impact of ChatGPT as a learning aid for students in API testing. A controlled experiment compared two groups: one utilizing ChatGPT and the other relying on traditional documentation. The findings indicate that participants using ChatGPT scored significantly higher in both exam tests compared to the documentation group, despite taking longer to complete tasks. Statistical analysis using t-tests confirmed these differences as significant. Post-test surveys revealed an increase in participants confidence and effectiveness in understanding and using APIs after interacting with ChatGPT. However, potential downsides, such as over-reliance on ChatGPT and insufficient deep conceptual understanding, were also observed. The results suggest that while ChatGPT can greatly enhance the quality of learning and productivity in API-related tasks, users must balance AI assistance with independent problem-solving skills. This study underscores the potential of ChatGPT as a valuable educational tool, provided it is integrated thoughtfully into the learning process.
Downloads
References
C. M. Gallardo Paredes, C. Machuca, and Y. M. Semblantes Claudio, “ChatGPT API: Brief overview and integration in Software Development,” International Journal of Engineering Insights, vol. 1, no. 1, 2023, doi: 10.61961/injei.v1i1.7.
Z. M. Aljazzaf, M. A. M. Capretz, and M. Perry, “Trust-based Service-Oriented Architecture,” Journal of King Saud University - Computer and Information Sciences, vol. 28, no. 4, 2016, doi: 10.1016/j.jksuci.2015.12.003.
Q. Zhu and J. Luo, “Generative Pre-Trained Transformer for Design Concept Generation: An Exploration,” in Proceedings of the Design Society, 2022. doi: 10.1017/pds.2022.185.
V. Guilherme and A. Vincenzi, “An initial investigation of ChatGPT unit test generation capability,” in ACM International Conference Proceeding Series, 2023. doi: 10.1145/3624032.3624035.
T. Wu et al., “A Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development,” IEEE/CAA Journal of Automatica Sinica, vol. 10, no. 5, 2023, doi: 10.1109/JAS.2023.123618.
F. Y. Wang, Q. Miao, X. Li, X. Wang, and Y. Lin, “What Does ChatGPT Say: The DAO from Algorithmic Intelligence to Linguistic Intelligence,” IEEE/CAA Journal of Automatica Sinica, vol. 10, no. 3. 2023. doi: 10.1109/JAS.2023.123486.
Md. A. Haque and S. Li, “The Potential Use of ChatGPT for Debugging and Bug Fixing,” EAI Endorsed Transactions on AI and Robotics, vol. 2, no. 1, 2023, doi: 10.4108/airo.v2i1.3276.
N. Nathalia, A. Paulo, and C. Donald, “Artificial Intelligence vs. Software Engineers: An Empirical Study on Performance and Efficiency using ChatGPT,” in Proceedings of the 33rd Annual International Conference on Computer Science and Software Engineering, 2023.
M. Shershneu and A. Oskin, “Postman Platform For API Development In The Mobile Application ‘Musicians of Russia,’” Repository of Polotsk State University, 2020.
D. BAİDOO-ANU and L. OWUSU ANSAH, “Education in the Era of Generative Artificial Intelligence (AI): Understanding the Potential Benefits of ChatGPT in Promoting Teaching and Learning,” Journal of AI, vol. 7, no. 1, 2023, doi: 10.61969/jai.1337500.
M. Kim et al., “Enhancing REST API Testing with NLP Techniques,” in ISSTA 2023 - Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2023. doi: 10.1145/3597926.3598131.
T. B. Brown et al., “Language models are few-shot learners,” in Advances in Neural Information Processing Systems, 2020.
V. Kozov, G. Ivanova, and D. Atanasova, “Practical Application of AI and Large Language Models in Software Engineering Education,” International Journal of Advanced Computer Science and Applications, vol. 15, no. 1, 2024, doi: 10.14569/IJACSA.2024.0150168.
C. Dilmegani, “100+ AI Use Cases & Applications: In-Depth Guide for 2023,” AI Mutiple .
F. Sarro, “Automated Optimisation of Modern Software System Properties,” 2023. doi: 10.1145/3578244.3583739.
O. Karnalim, E. D. Handoyo, H. Toba, Y. D. Setiawan, M. C. Johan, and J. A. Luwia, “Plagiarism and AI Assistance Misuse in Web Programming: Unfair Benefits and Characteristics,” in 2023 IEEE International Conference on Teaching, Assessment and Learning for Engineering, TALE 2023 - Conference Proceedings, 2023. doi: 10.1109/TALE56641.2023.10398397.
H. Toba, O. Karnalim, M. C. Johan, T. Tada, Y. M. Djajalaksana, and T. Vivaldy, “Inappropriate Benefits and Identification of ChatGPT Misuse in Programming Tests: A Controlled Experiment,” 2024. doi: 10.1007/978-3-031-51979-6_54.
Copyright (c) 2024 Yehezkiel David Setiawan, Laurentius Gusti Ontoseno Panata Yudha, Yovie Adhisti Mulyono, Veronica Marcella Angela Simalango, Oscar Karnalim
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License (Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) ) that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
