Towards Transparent AI for Lung Cancer Diagnosis: A Dual-Pipeline Explainable Framework Using Clinical and CT Imaging Data

Abstract

In recent years, artificial intelligence (AI) has shown promising performance in medical diagnosis; however, its clinical adoption remains limited due to a lack of interpretability. In this study, we propose a dual-pipeline explainable framework for lung cancer diagnosis using two independent data modalities: structured clinical data and CT imaging data. For the clinical data, several machine learning models were compared, such as LightGBM, CatBoost, XGBoost, Random Forest, Logistic Regression, K-Nearest Neighbors, and Naïve Bayes. For the imaging data, deep learning models such as VGG16, ResNet50, InceptionV3, MobileNetV2, DenseNet121, and Xception were compared using the IQ-OTH/NCCD dataset. To ensure the reliability of the validation results, a strict patient-level split was used to avoid data leakage. The experimental results showed that LightGBM obtained the best results on the clinical data, achieving an accuracy of 98.39% and an ROC-AUC of 0.99. In the imaging data, MobileNetV2 obtained the best results, achieving an accuracy of 0.97, which is highly computationally efficient. To improve the interpretability of the models, SHAP and LIME were used to analyze the clinical feature importance, while Grad-CAM was used to analyze the discriminative regions in the CT image. The reliability of the explanations was also verified through stability analysis with Spearman rank correlation, agreement analysis with SHAP and LIME, as well as through verification with expert clinicians for the Grad-CAM visualizations. The results demonstrate that various XAI methods provide complementary insights, supporting the creation of transparent, reliable, and meaningful AI systems for lung cancer diagnosis.