Cataract　is　a　chronic　eye　disease　that　causes　irreversible　vision　loss.　Automatic　cataract　detection　can　help　people　prevent　visual　impairment　and　decrease　the　possibility　of　blindness.　To　date,　many　studies　utilize　deep　learning　methods　to　grade　cataract　severity　on　fundus　images.　However,　they　mainly　focus　on　the　classification　performance　and　ignore　the　model　interpretability,　which　may　lead　to　a　semantic　gap　between　networks　and　users.　In　this　paper,　we　present　a　deep　learning　network　to　improve　the　model　interpretability,　which　consists　three　main　modules:　deep　feature　extraction,　visual　saliency　module　and　semantic　description　module.　Visual　and　semantic　interpretation　jointly　employed　to　provide　cataract-grade　oriented　interpretation　for　the　overall　model.　Experimental　results　on　real　clinical　data　set　show　that　our　method　improves　the　interpretability　for　cataract　grading　while　ensuring　the　high　classification　performance.