Pedestrian　detection　for　surveillance　video,　which　is　the　basic　of　person　re-identification,　aims　to　capture　the　pedestrians　in　the　monitors.　However,　the　existing　pedestrian　detection　algorithms　still　have　two　issues:　(1)　The　recall　and　precision　are　not　applicable　for　complicated　scenes;　(2)　It　is　limited　for　processing　the　high-resolution　video　in　real-time.　Therefore,　pedestrian　detection　algorithm　based　on　imbalance　prior　for　surveillance　video　is　proposed　in　this　paper.　Firstly,　the　structure　of　pedestrian　is　described　with　a　color　difference　based　image　edge　detection　algorithm,　namely　color　difference　map　(CDM).　Then,　the　imbalance　prior　is　proposed　and　used　for　coarse　classification.　Finally,　the　refine　classification　is　implemented　by　＂HOG+SVM＂　method.　The　experimental　results　on　FHD-175　dataset　show　that　the　recall　and　precision　of　the　proposed　algorithm　are　96.0%　and　99.0%　respectively.　Furthermore,　the　proposed　algorithm　can　process　the　1920x1080　frame　at　the　speed　of　20.73　fps　on　average,　which　can　satisfy　the　requirement　of　real　time　processing.