Features fusion or not: harnessing multiple pathological foundation models using Meta-Encoder for downstream tasks fine-tuning

The emergence of diverse pathological foundation models has empowered computational pathology tasks, including tumor classification, biomarker prediction, and RNA expression prediction. However, variations in model architecture and data sources lead to inconsistent downstream performance and complicate centralized training. Specifically, the lack of data sharing makes retraining foundation models with pooled data infeasible. Alternatively, the release of model parameters enables combining multiple models during fine-tuning. Inspired by the meta-analysis method, we propose the Meta-Encoder framework, which integrates features from multiple foundation models to generate a comprehensive representation, improving downstream fine-tuning task performance. Comparative experiments demonstrate that Meta-Encoder is more effective than individual foundation models, with its strengths more pronounced in handling complex tasks. While single models may perform sufficiently well for simple tasks, Meta-Encoder can match or even surpass the best-performing single model, alleviating concerns over model selection. Moderately challenging tasks benefit from Meta-Encoder’s concatenation or self-attention strategies, with the latter demonstrating superior performance in more challenging scenarios. For highly complex tasks, such as high-dimensional gene expression prediction, self-attention proves to be the most effective Meta-Encoder strategy, balancing feature integration and computational efficiency. For three patch-level spatial gene expression prediction tasks (HEST-Benchmark, CRC-inhouse, and Her2ST), the self-attention strategy improved the Pearson correlation by 38.58%, 26.06%, and 20.39%, respectively, compared to the average performance of three patch-level single models. Similarly, for the TCGA-BRCA, TCGA-NSCLC, and TCGA-CRC WSI-level bulk gene expression prediction tasks, the Pearson correlation increased by 14.36%, 9.27%, and 42.55%, respectively, compared to the average performance of two WSI-level single models. By leveraging multiple pathological foundation models using Meta-Encoder, it can further improve molecular characterization in pathology images to advance precision oncology.