In　recent　years,　the　increasing　number　of　patients　with　various　mental　sicknesses　has　attracted　the　attention　of　medical　experts　and　scholars　at　home　and　abroad.　In　the　study　of　mental　sickness,　a　patient＇s　facial　expression　is　an　important　criterion　for　a　doctor　to　determine　whether　or　not　the　patient　has　the　disease　and　the　severity　of　the　disease.　Expression　is　a　psychological　appearance,　therefore,　the　facial　expression　recognition　has　the　reference　value　for　disease　diagnosis　that　cannot　be　ignored.　Deep　learning　techniques,　especially　the　convolutional　neural　network　(CNN),　have　made　great　progress　in　the　field　of　facial　expression　recognition.　However,　the　deep　learning　models　need　enough　training　samples,　and　they　have　a　high　demand　for　computing　resources.　In　view　of　the　facial　expression　recognition　on　small　datasets,　we　introduce　a　transfer　learning　method　based　on　CNN　structure.　Specifically,　the　hierarchical　representation　of　the　deep　convolutional　image　recognition　model　is　used　as　the　source　model,　and　then　the　high　level　representation　is　learned　on　the　four　small　public　expression　datasets　to　obtain　the　target　models　applicable　to　the　expression　recognition　tasks.　Four　commonly　public　expression　datasets　(CK+,　JAFFE,　Nim　Stim　and　MMI)　are　used　to　evaluate　our　models.　The　experimental　results　show　that　our　method　achieves　the　mean　classification　accuracy　of　99.68%,　88.36%.　86.52%　and　89.34%　respectively　on　CK+,　JAFFE,　Nim　Stim　and　MMI　dataset　without　much　computing　resources.