LeNet5　is　a　kind　of　Convolutional　Neural　Network　(CNN)　and　has　been　used　in　handwritten　digits　recognition.　In　order　to　improve　the　recognition　rate　of　LeNet5　in　handwritten　digits　recognition,　this　article　presents　an　improved　LeNet5　by　replacing　the　last　two　layers　of　the　LeNet5　structure　with　Support　Vector　Machines　(SVM)　classifier.　And　LeNet5　performs　as　a　trainable　feature　extractor　and　SVM　works　as　a　recognizer.　To　accelerate　the　network＇s　convergence　speed,　the　stochastic　diagonal　Levenberg-Marquardt　algorithm　is　introduced　to　train　the　network.　A　series　of　studies　has　been　conducted　on　the　MINST　digit　database　to　test　and　evaluate　the　proposed　method　performance.　The　results　show　that　this　method　can　outperform　both　SVMs　and　LeNet5.　Moreover,　the　improved　method　gets　a　faster　convergence　speed　in　training　process.　©　2015　IEEE.