Convolutional　neural　network　(CNN)　has　been　widely　used　in　computer　vision　tasks　recently　and　achieved　remarkable　success.　This　paper　presents　a　novel　video-based　recognition　approach　using　CNN　for　Chinese　Sign　Language　(CSL).　The　proposed　method　extracts　upper　body　images　directly　from　videos,　and　employs　a　pre-training　convolutional　network　model　to　recognize　the　gesture　in　the　image.　The　new　method　simplifies　the　hand-shape　segmentation,　and　avoid　information　loss　in　feature　extraction.　We　evaluate　the　method　on　our　self-built　dataset　includes　40　daily　vocabularies,　and　show　that　the　proposed　approach　has　good　performance　on　sign　language　recognition　task,　with　accuracy　reaching　to　99%.