The　action　recognition　framework　based　on　local　features　was　established　to　improve　the　recognition　ability　of　human　behavior　in　video　sequences.　The　algorithms　related　to　the　framework　were　researched　through　spatial　temporal　features　extracting　and　encoding　and　parameters　optimization　of　SVM　classifier.　Firstly,　the　feature　descriptors　composed　of　Histograms　of　Oriented　Gradients　(HOG)　and　Histograms　of　Optical　Flow　(HOF)　were　used　to　describe　Space　Time　Interest　Points　(STIP)　achieved　by　the　Harris　3D　detector　and　then　encoded　by　Fisher　Vector　(FV).　Due　to　the　generalization　ability　of　Support　Vector　Machine　(SVM)　model　for　action　classification　under　fixed　parameters　was　insufficient,　the　particle　swarm　optimization　algorithm　was　applied　to　the　parameter　optimization　of　each　action　classifier.　According　to　the　characteristics　of　population　diversity　changed　from　generation　to　generation,　the　constructed　particles　aggregation　degree　model　was　used　to　adjust　mutation　probability　for　each　generation　of　particles　dynamically.　Finally,　the　proposed　method　was　verified　by　KTH　and　HMDB51　data　sets.　The　results　show　that　the　Adaptive　Mutation　Particle　Swarm　Optimization　(AMPSO)　algorithm　can　avoid　the　local　optimum　and　has　strong　global　optimization　capability.　The　recognition　accuracies　on　KTH　and　HMDB51　data　sets　are　87.50%　and　26.41%,　respectively,　which　are　better　than　two　recognition　methods.　Experimental　results　indicate　that　the　AMPSO　algorithm　has　good　convergence　performance　and　the　overall　recognition　framework　has　high　practicability　and　accuracy.　©　2017,　Science　Press.　All　right　reserved.