White　matter　hyperintensities　(WMH)　play　a　significant　role　in　predicting　cognitive　risk　in　the　human　brain.　Both　their　location　and　size　can　affect　normal　cognitive　functions.　Convolutional　Neural　Networks　(CNNs)　are　widely　used　in　risk　prediction,　but　they　are　limited　by　their　inherent　mechanisms.　For　example,　pooling　layers　can　lead　to　a　lack　of　local　and　global　semantic　correlation　in　images,　which　constrains　prediction　and　classification　performance.　In　this　paper,　we　propose　an　Image-Text　Fusion　Network　architecture　that　combines　WMH　images,　patient　physiological　indicators,　and　clinical　symptoms.　This　architecture　extracts　and　semantically　matches　features　from　both　lesion　and　clinical　symptom　modalities　to　generate　a　supervisory　vector　that　can　enhance　the　semantic　expression　of　the　classification　network.　Furthermore,　we　introduce　a　Multi-Branch　Transformer　module　to　improve　the　fusion　of　feature　maps　from　different　branches.　This　module　emphasizes　long-range　dependencies　between　features　from　different　channels　and　supervisory　vectors　at　different　scales.　The　final　experimental　results　achieve　an　accuracy　of　0.956　and　an　F1-Score　of　0.836,　outperforming　existing　classification　networks.　These　results　demonstrate　the　algorithm＇s　potential　to　aid　clinicians　in　diagnosis.　©　2024　IEEE.