Performance and CI/CD Compatibility Analysis of Selenium, Cypress, and Playwright UI Testing Frameworks Using the Distance to Ideal Alternative Method

M Dirga Purnomo Putra; Rakhmat Prasetyo Agung Nugroho; Ridho Surya Kusuma; Latifah Iriani

doi:10.30871/jaic.v10i5.13680

Authors

M Dirga Purnomo Putra Universitas Siber Muhammadiyah
Rakhmat Prasetyo Agung Nugroho Universitas Siber Muhammadiyah
Ridho Surya Kusuma Universitas Siber Muhammadiyah
Latifah Iriani Universitas Siber Muhammadiyah

DOI:

https://doi.org/10.30871/jaic.v10i5.13680

Keywords:

CI/CD Pipeline, Cypress, Playwright, Selenium, Test automation

Abstract

Prior comparative studies have evaluated Selenium, Cypress, and Playwright in terms of functional features or local execution speed, whereas their integration efficiency within a Continuous Integration/Continuous Deployment (CI/CD) pipeline, a decisive factor in modern DevOps practice, has received limited empirical attention. Selecting the optimal framework for User Interface (UI) test automation is nevertheless a critical challenge in modern software development, especially in the context of deep integration into CI/CD. This study aims to analyze the comparative performance and CI/CD compatibility of three leading frameworks: Selenium, Cypress, and Playwright. Through a quantitative experimental design, three real-world test scenarios covering user authentication, dynamic UI interactions, and end-to-end transactions were executed on the GitHub Actions platform. Performance evaluation was based on three weighted criteria: Average Execution Time (Runtime) (C₁), Execution Validity (C₂), and CI/CD Compatibility (C₃). The test results were analyzed using the Multi-Criteria Decision Making (MCDM) method, specifically Distance to Ideal Alternative (DIA), to generate an objective preference ranking. The DIA analysis results ranked Playwright (Ri = 0,000) as the top-performing framework, followed by Cypress (Ri = 0,066), and Selenium (Ri = 0,406), with the underlying runtime differences confirmed statistically significant (Kruskal-Wallis test, p < 0,001). These findings confirm that the modern architectures of Playwright and Cypress are capable of delivering significantly higher execution speeds and better CI/CD compatibility compared to traditional WebDriver-based architectures. As a practical implication, the adoption of modern frameworks, particularly Playwright, is highly recommended for software developers, software testers, and DevOps teams to optimize the feedback loop and improve the efficiency of the application release process.

Downloads

Download data is not yet available.

References

[1] R. S. Pressman, Software engineering: A practitioner’s approach. 2015.

[2] S. Di Meglio, L. L. L. Starace, V. Pontillo, R. Opdebeeck, C. De Roover, and S. Di Martino, “Investigating the adoption and maintenance of web GUI testing: Insights from GitHub repositories,” Inf. Softw. Technol., vol. 189, p. 107928, Jan. 2026, doi: 10.1016/J.INFSOF.2025.107928.

[3] G. Hyun, J. Oak, D. Kim, and K. Kim, “The impact of an automation system built with Jenkins on the efficiency of container-based system deployment,” Sensors 2024, Vol. 24, Page 6002, vol. 24, no. 18, p. 6002, Sep. 2024, doi: 10.3390/S24186002.

[4] F. Dobslaw, R. Feldt, D. Michaelsson, P. Haar, F. G. De Oliveira Neto, and R. Torkar, “Estimating return on investment for GUI test automation frameworks,” Proceedings - International Symposium on Software Reliability Engineering, ISSRE, vol. 2019-October, pp. 271–282, Oct. 2019, doi: 10.1109/ISSRE.2019.00035.

[5] A. J. Abdulwareth and A. A. Al-Shargabi, “Toward a multi-criteria framework for selecting software testing tools.,” IEEE Access, vol. 9, pp. 158872–158891, 2021, doi: 10.1109/ACCESS.2021.3128071.

[6] V. Bhimanapati, P. Goel, and U. Jain, “Leveraging Selenium and Cypress for comprehensive web application testing.,” Journal of Quantum Science and Technology, vol. 1, no. 1, pp. 66–79, Mar. 2024, doi: 10.36676/JQST.V1.I1.10.

[7] B. García, J. M. del Alamo, M. Leotta, and F. Ricca, “Exploring browser automation: A comparative study of Selenium, Cypress, Puppeteer, and Playwright,” Communications in Computer and Information Science, vol. 2178 CCIS, pp. 142–149, 2024, doi: 10.1007/978-3-031-70245-7_10.

[8] A. Polkhovskaya and J. Salonen, “Improving QA processes and implementing Playwright for API and E2E testing automation,” 2025, Accessed: May 17, 2025. [Online]. Available: http://www.theseus.fi/handle/10024/882858

[9] G. A. Prasetyo, “Analisis perbandingan UI test automation framework dengan metode The Distance to the Ideal Alternative,” 2024, Accessed: May 17, 2025. [Online]. Available: https://dspace.uii.ac.id/handle/123456789/51452

[10] Y. Wang, M. V. Mäntylä, Z. Liu, and J. Markkula, “Test automation maturity improves product quality—Quantitative study of open source projects using continuous integration,” Journal of Systems and Software, vol. 188, p. 111259, Jun. 2022, doi: 10.1016/J.JSS.2022.111259.

[11] A. P. Putra, D. Sartika, and D. Lianda, “Decision support system for the best junior high school using the Distance to the Ideal Alternative (DIA) method at the Education and Culture Office of the City of Bengkulu,” Jurnal Media Computer Science, vol. 3, no. 2, Jul. 2024, doi: 10.37676/JMCS.V3I2.6618.

[12] S. Adiatma and A. Darmayantie, “Implementation and comparative analysis of test automation framework performance for functional testing of web-based applications using the Distance to the Ideal Alternative (DIA) method,” Widya Teknik, vol. 22, no. 2, pp. 36–41, May 2023, doi: 10.33508/WT.V22I1.5027.

[13] R. Fakhrii and N. Setiani, “Analisis perbandingan unit test automation framework dengan metode The Distance to the Ideal Alternatif,” EDUSAINTEK: Jurnal Pendidikan, Sains dan Teknologi, vol. 12, no. 1, pp. 253–272, 2025, doi: 10.47668/EDUSAINTEK.V12I1.1489.

[14] G. A. Prasetyo and N. Setiani, “Perbandingan kinerja framework automation UI testing menggunakan The Distance to the Ideal Alternative,” JIPI (Jurnal Ilmiah Penelitian dan Pembelajaran Informatika), vol. 10, no. 1, pp. 224–237, Jan. 2025, doi: 10.29100/JIPI.V10I1.5760.

[15] F. P. Apriliyanto, Y. Prihati, and C. A. Bakti, “Implementasi CI/CD menggunakan Github Actions untuk pengembangan web landing page UMKM XYZ,” INTECOMS: Journal of Information Technology and Computer Science, vol. 7, no. 4, pp. 1239–1249, Jul. 2024, doi: 10.31539/INTECOMS.V7I4.10961.

[16] M. Moń and B. Pańczyk, “A comparative analysis of web application test automation tools,” Journal of Computer Sciences Institute, vol. 35, pp. 159–165, Jun. 2025, doi: 10.35784/JCSI.7119.

[17] B. Bylina and A. Antończak, “Analysis of end-to-end test automation tools based on the examples of Selenium WebDriver and Playwright,” Position Papers of the 19th Conference on Computer Science and Intelligence Systems, vol. 40, pp. 1–8, Nov. 2024, doi: 10.15439/2024F3747.

[18] A. Tymoshchuk, “The evolution of test automation: From Selenium to Playwright. A comparison of automation tools: Selenium vs. Playwright vs. Cypress,” International Journal of Computer (IJC), vol. 54, no. 1, pp. 37–45, Apr. 2025, Accessed: Jan. 16, 2026. [Online]. Available: https://ijcjournal.org/InternationalJournalOfComputer/article/view/2355

[19] M. A. Rizky and A. Wibowo, “Analysis of automation testing on e-commerce websites and user responses,” Society, vol. 13, no. 1, pp. 788–802, Apr. 2025, doi: 10.33019/society.v13i1.767.