Predicting Recurrence Before Surgery

Radical prostatectomy removes the prostate, but for a significant proportion of patients, prostate cancer returns – a condition called biochemical recurrence (BCR). Identifying which patients are at risk before surgery can directly inform treatment decisions and follow-up intensity. Two imaging modalities offer complementary views of the tumor: MRI captures morphology and tissue structure, while PSMA-PET reflects molecular expression at the cellular level. Despite this complementarity, no deep learning model had combined them for BCR prediction.

The reason is straightforward: Almost no patients undergo both scans, so paired training data barely exists.
Our cohort had 84 patients; only 12 of whom developed BCR. Training a multimodal deep learning model end-to-end on this would overfit immediately.

We addressed this with three components:

1. Modality-specific pretraining. Rather than training from scratch on paired data, we pretrained separate 3D Swin-B encoders – one for MRI, one for PSMA-PET – on large single-modality datasets using masked volume reconstruction (SimMIM). The encoders were then frozen, reducing trainable parameters to 7.4M.

2. Tumor-anchored cross-modal fusion. Co-registered MRI and PET token maps interact through cross-modal attention, then are pooled with weights proportional to tumor coverage. Since the tumor occupies only ~3.6% of the gland on average, standard uniform pooling would drown out the relevant signal.

3. Validated saliency maps. We used integrated gradients to trace predictions back to individual voxels in each modality, then validated them with deletion tests and model randomization – not just visualized them.