To　optimize　the　application　of　H.264　video　encoder　in　DSP,　on　the　basis　of　the　research　on　the　block　size　prediction　of　intra　&　inter　frame,　grey　histogram　and　autocorrelation　are　adopted　to　improve　the　intra　frame　prediction.　Combining　the　sum　of　absolute　difference,　which　is　provided　as　the　half　stop　rule,　and　on　the　basis　of　the　analysis　of　texture　difference,　a　novel　fast　inter　frame　prediction　algorithm　is　proposed.　The　two　algorithms　have　achieved　significant　improvement　as　compared　to　the　original　encoder.　Finally,　the　optimization　techniques　of　H.264　on　DM642　DSP　platform,　which　can　effectively　enhance　the　video　coding　speed,　are　proposed.　The　experimental　results　show　that　without　the　variety　of　image,　basically　the　encoding　speed　is　15　frame/s　on　average,　some　of　video　sequences　may　achieve　18　frame/s.