With　increased　use　of　H.264/AVC　in　various　applications　including　video　surveillance　systems,　feature　extraction　and　knowledge　representation　in　compressed　domain　are　becoming　attractive.　A　real-time　H.264/AVC　compressed　domain　moving　object　segmentation　and　tracking　algorithm　for　surveillance　videos　is　proposed　in　this　paper.　This　algorithm　consists　of　moving　object　detection,　bounding　box　matching,　spatiotemporal　merge　and　split　reasoning　and　trajectory　smoothing,　with　major　innovation　in　incorporating　the　information　provided　by　the　prediction　modes　into　the　framework　of　motion　detection　and　trajectory　construction.　The　experimental　results　on　both　indoor　and　outdoor　surveillance　videos　demonstrate　that　the　adaptive　use　of　the　information　from　motion　vectors,　DCT　coefficients　and　prediction　modes　can　substantially　improve　the　performance　of　moving　object　segmentation　and　tracking.　©　2011　IEEE.