Video　annotation　is　a　kind　of　＂high-level　feature　extraction＂　or　＂semantic　concept　detection＂,　which　is　a　promising　approach　to　bridging　semantic　gap　between　low-level　features　and　user　descriptions.　In　this　paper　we　propose　a　two　layer　video　annotation　scheme　based　on　video　structure　and　the　visual　context　information　through　the　video　clips.　To　improve　the　retrieval　performance　based　on　annotation　text　we　combine　the　video　title　and　related　text　information　to　re-rank　the　retrieval　sequences.　In　the　experiment　we　test　our　annotation　result　through　text　based　video　retrieval　experiment　meanwhile　we　presented　our　re-rank　result.　Our　annotation　and　re-ranking　approach　could　be　utilized　in　future　web　video　data　organization.