Group　activity　recognition　has　received　a　great　deal　of　interest　because　of　its　broader　applications　in　sports　analysis,　autonomous　vehicles,　CCTV　surveillance　systems　and　video　summarization　systems.　Most　existing　methods　typically　use　appearance　features　and　they　seldom　consider　underlying　interaction　information.　In　this　work,　a　technology　of　novel　group　activity　recognition　is　proposed　based　on　multi-modal　relation　representation　with　temporal-spatial　attention.　First,　we　introduce　an　object　relation　module,　which　processes　all　objects　in　a　scene　simultaneously　through　an　interaction　between　their　appearance　feature　and　geometry,　thus　allowing　the　modeling　of　their　relations.　Second,　to　extract　effective　motion　features,　an　optical　flow　network　is　fine-tuned　by　using　the　action　loss　as　the　supervised　signal.　Then,　we　propose　two　types　of　inference　models,　opt-GRU　and　relation-GRU,　which　are　used　to　encode　the　object　relationship　and　motion　representation　effectively,　and　form　the　discriminative　frame-level　feature　representation.　Finally,　an　attention-based　temporal　aggregation　layer　is　proposed　to　integrate　frame-level　features　with　different　weights　and　form　effective　video-level　representations.　We　have　performed　extensive　experiments　on　two　popular　datasets,　and　both　have　achieved　state-of-the-art　performance.　The　datasets　are　the　Volleyball　dataset　and　the　Collective　Activity　dataset,　respectively.