Automatic　video　annotation　is　to　bridge　the　semantic　gap　and　facilitate　concept　based　video　retrieval　by　detecting　high　level　concepts　from　video　data.　Recently,　utilizing　context　information　has　emerged　as　an　important　direction　in　such　domain.　In　this　paper,　we　present　a　novel　video　annotation　refinement　approach　by　utilizing　extrinsic　semantic　context　extracted　from　video　subtitles　and　intrinsic　context　among　candidate　annotation　concepts.　The　extrinsic　semantic　context　is　formed　by　identifying　a　set　of　key　terms　from　video　subtitles.　The　semantic　similarity　between　those　key　terms　and　the　candidate　annotation　concepts　is　then　exploited　to　refine　initial　annotation　results,　while　most　existing　approaches　utilize　textual　information　heuristically.　Similarity　measurements　including　Google　distance　and　WordNet　distance　have　been　investigated　for　such　a　refinement　purpose,　which　is　different　with　approaches　deriving　semantic　relationship　among　concepts　from　given　training　datasets.　Visualness　is　also　utilized　to　discriminate　individual　terms　for　further　refinement.　In　addition,　Random　Walk　with　Restarts　(RWR)　technique　is　employed　to　perform　final　refinement　of　the　annotation　results　by　exploring　the　inter-relationship　among　annotation　concepts.　Comprehensive　experiments　on　TRECVID　2005　dataset　have　been　conducted　to　demonstrate　the　effectiveness　of　the　proposed　annotation　approach　and　to　investigate　the　impact　of　various　factors.