This　paper　presents　a　discriminative　stochastic　method　for　image　annotation　and　refinement.　We　first　segmented　the　images　into　regions　and　then　cluster　them　into　visual　blobs　with　a　small　number　than　the　whole　training　image　regions.　Each　visual　blob　is　regarded　as　a　key　visual　word.　Given　the　training　image　set　with　annotations,　we　find　that　annotation　process　is　conditioned　by　the　selection　sequence　of　both　the　semantic　word　and　the　key　visual　word.　The　process　could　be　described　in　a　Markov　Chain　with　the　transition　process　both　between　the　candidate　annotations　and　the　visual　words　set.　Experiments　show　the　performance　of　this　annotation　method　outperforms　the　state　of　art　methods.　©　2014　Springer　International　Publishing　Switzerland.　All　rights　reserved.