Tongue　diagnosis　is　widely　used　in　the　Traditional　Chinese　Medicine　(TCM)　and　tongue　image　classification　based　on　pattern　recognition　plays　an　important　role　in　the　development　of　the　modernization　of　TCM.　However,　due　to　labeled　tongue　samples　are　rare　and　costly　or　time　consuming　to　obtain,　most　of　the　existing　methods　such　as　SVM　utilize　labeled　training　samples　merely.　Therefore　the　classifiers　usually　have　poor　performance.　In　contrast,　Universum　SVM　is　a　promising　method　which　incorporates　a　priori　knowledge　into　the　learning　process　with　labeled　data　and　irrelevant　data　(also　called　universum　data).　In　tongue　image　classification,　the　number　of　irrelevant　instances　could　be　very　large　since　there　are　many　irrelevant　categories　for　a　certain　tongue＇s　type.　But　not　all　the　irrelevant　instances　joined　in　training　can　improve　the　classifier＇s　performance.　So　an　algorithm　of　selecting　the　universum　samples　is　also　introduced　in　this　paper.　Experimental　results　show　that　the　Universum　SVM　classifier　is　improved　and　the　algorithm　of　selecting　universum　samples　is　effective.