Adaptive branch-gated fusion of dual autoencoder latent representations improves head and neck squamous cell carcinoma classification and supports exploratory candidate gene prioritization

Head and neck squamous cell carcinoma (HNSCC) is a molecularly heterogeneous malignancy for which robust transcriptome-based classification remains challenging because of the high dimensionality of RNA sequencing data and the limited number of available normal samples. In this study, we developed an adaptive deep learning framework that integrates complementary latent representations from a denoising undercomplete autoencoder (DUAE) and a variational autoencoder (VAE) for binary classification of tumor and normal transcriptomic profiles in HNSCC. RNA-seq data from the TCGA-HNSC cohort was preprocessed through label harmonization, gene symbol normalization, log1p transformation, and z-score standardization. The proposed Adaptive Gated Fusion Expert Model (AGFEM) used sample-specific pathway-level attention weights to dynamically reweight DUAE and VAE latent embedding before downstream classification. Model performance was assessed using five-fold stratified cross-validation and compared against DUAE-only, VAE-only, and raw-feature multilayer perceptron baselines using accuracy, area under the receiver operating characteristic curve (ROC-AUC), precision, recall, and F1-score. Statistical significance of performance differences was evaluated using Wilcoxon signed-rank testing on fold-wise metrics and DeLong testing on pooled out-of-fold ROC predictions. To assess cross-cohort transferability, an independent GEO cohort (GSE65858) was harmonized through probe-to-gene mapping and projected into the learned latent space. Candidate genes were prioritized using model-derived feature importance from the DUAE encoder and further examined using pathway enrichment analysis. Overall, these findings support adaptive fusion of deterministic and probabilistic transcriptomic representations as a useful strategy for internally validated HNSCC classification, while also enabling exploratory candidate gene prioritization for downstream biological follow-up.