Automatic　image　annotation　is　a　promising　key　to　semantic-based　image　retrieval　by　keywords.　Most　existing　automatic　image　annotation　approaches　focused　on　exploring　the　relationship　between　images　and　annotation　words　and　neglected　the　semantic　information　of　the　annotated　keywords.　In　this　paper　we　propose　a　semi-automatic　image　annotation　framework.　First　we　annotate　the　training　images　with　our　improved　Markov　model.　Then　the　candidate　annotation　terms　are　clustered　according　to　their　semantic　relationship.　Finally　we　further　optimize　the　annotation　sets　with　relevance　feedback　from　people＇s　cognitive　learning　habits.　The　experimental　results　show　that　the　proposed　approach　provides　a　semi-automatic　optimization　of　the　multi-label　image　annotation　results.　©　2012　IEEE.