Domain　adaptation,　as　an　important　branch　of　transfer　learning,　can　be　applied　to　cope　with　data　insufficiency　and　high　subject　variabilities　in　motor　imagery　electroencephalogram　(MI-EEG)　based　brain-computer　interfaces.　The　existing　methods　generally　focus　on　aligning　data　and　feature　distribution;　however,　aligning　each　source　domain　with　the　informative　samples　of　the　target　domain　and　seeking　the　most　appropriate　source　domains　to　enhance　the　classification　effect　has　not　been　considered.　In　this　paper,　we　propose　a　dual　alignment-based　multi-source　domain　adaptation　framework,　denoted　DAMSDAF.　Based　on　continuous　wavelet　transform,　all　channels　of　MI-EEG　signals　are　converted　respectively　and　the　generated　time-frequency　spectrum　images　are　stitched　to　construct　multi-source　domains　and　target　domain.　Then,　the　informative　samples　close　to　the　decision　boundary　are　found　in　the　target　domain　by　using　entropy,　and　they　are　employed　to　align　and　reassign　each　source　domain　with　normalized　mutual　information.　Furthermore,　a　multi-branch　deep　network　(MBDN)　is　designed,　and　the　maximum　mean　discrepancy　is　embedded　in　each　branch　to　realign　the　specific　feature　distribution.　Each　branch　is　separately　trained　by　an　aligned　source　domain,　and　all　the　single　branch　transfer　accuracies　are　arranged　in　descending　order　and　utilized　for　weighted　prediction　of　MBDN.　Therefore,　the　most　suitable　number　of　source　domains　with　top　weights　can　be　automatically　determined.　Extensive　experiments　are　conducted　based　on　3　public　MI-EEG　datasets.　DAMSDAF　achieves　the　classification　accuracies　of　92.56%,　69.45%　and　89.57%,　and　the　statistical　analysis　is　performed　by　the　kappa　value　and　t-test.　Experimental　results　show　that　DAMSDAF　significantly　improves　the　transfer　effects　compared　to　the　present　methods,　indicating　that　dual　alignment　can　sufficiently　use　the　different　weighted　samples　and　even　source　domains　at　different　levels　as　well　as　realizing　optimal　selection　of　multi-source　domains.