Motion　information　has　been　widely　exploited　for　group　activity　recognition　in　sports　video.　However,　in　order　to　model　and　extract　the　various　motion　information　between　the　adjacent　frames,　existing　algorithms　only　use　the　coarse　video-level　labels　as　supervision　cues.　This　may　lead　to　the　ambiguity　of　extracted　features　and　the　omission　of　changing　rules　of　motion　patterns　that　are　also　important　sports　video　recognition.　In　this　paper,　a　latent　label　mining　strategy　for　group　activity　recognition　in　basketball　videos　is　proposed.　The　authors＇　novel　strategy　allows　them　to　obtain　the　latent　labels　set　for　marking　different　frames　in　an　unsupervised　way,　and　build　the　frame-level　and　video-level　representations　with　two　separate　levels　of　supervision　signal.　Firstly,　the　latent　labels　of　motion　patterns　are　digged　using　the　unsupervised　hierarchical　clustering　technique.　The　generated　latent　labels　are　then　taken　as　the　frame-level　supervision　signal　to　train　a　deep　CNN　for　the　frame-level　features　extraction.　Lastly,　the　frame-level　features　are　fed　into　an　LSTM　network　to　build　the　spatio-temporal　representation　for　group　activity　recognition.　Experimental　results　on　the　public　NCAA　dataset　demonstrate　that　the　proposed　algorithm　achieves　state-of-the-art　performance.