The　paper　proposes　a　kind　of　joint　feature　with　the　geometric　parameters　and　the　color　moment　to　represent　the　speaking-mouth　image　frames　for　the　image-based　visual　speech　synthesis　system.　The　experiment　results　show　the　proposed　joint　feature　can　effectively　provide　the　basis　for　classify　the　speaking-mouth　images　to　distinguish　the　different　speaking　state　according　to　the　lip　shape　and　the　tooth　visibility.　In　order　to　derive　the　geometric　feature,　a　kind　of　step-by-step　point　feature　localization　is　proposed　here　is　introduced　to　get　the　feature　points.　Comparing　the　localization　results　by　this　automatic　localization　algorithm　with　that　by　hand,　the　paper　proves　the　localization　result　can　meet　the　need　of　geometric　feature　representation.　©2009　IEEE.