Anti-Data Leakage Pipeline for Differentiated Thyroid Cancer Recurrence Prediction: Integrating SMOTE, Optuna-based Optimization, and Bootstrap BCa Validation

Abstract

Thyroid cancer recurrence prediction remains a critical clinical challenge, as early identification of high-risk patients enables targeted monitoring and intervention. This study presents a comparative evaluation of six machine learning classifiers (XGBoost, LightGBM, CatBoost, Logistic Regression, Random Forest, and Decision Tree) using the UCI Differentiated Thyroid Cancer Recurrence dataset which consists of 383 patient records and 16 clinical features. To prevent performance overestimation, a rigorous anti-data leakage pipeline was implemented, encapsulating SMOTE, Optuna-based hyperparameter optimization, and Isotonic Calibration within the cross-validation process. Furthermore, model stability was assessed using Bias-Corrected and accelerated (BCa) Bootstrap validation with 2,000 iterations. Experimental results demonstrate that XGBoost achieved the best overall performance with an F1-score of 0.9545, an AUC-ROC of 0.9967, and the lowest Brier Score of 0.0183. Bootstrap BCa analysis confirmed XGBoost as the most stable model, with a 95% CI F1-score width of 0.1429 and unbiased estimation. These findings suggest that XGBoost, integrated within a zero-leakage pipeline and validated through Bootstrap BCa, is a promising candidate for post-treatment clinical decision support in differentiated thyroid cancer management.