In　this　work,　we　introduce　a　deep-structured　conditional　random　field　(DS-CRF)　model　for　the　purpose　of　state-based　object　silhouette　tracking.　The　proposed　DS-CRF　model　consists　of　a　series　of　state　layers,　where　each　state　layer　spatially　characterizes　the　object　silhouette　at　a　particular　point　in　time.　The　interactions　between　adjacent　state　layers　are　established　by　inter-layer　connectivity　dynamically　determined　based　on　inter-frame　optical　flow.　By　incorporate　both　spatial　and　temporal　context　in　a　dynamic　fashion　within　such　a　deep-structured　probabilistic　graphical　model,　the　proposed　DS-CRF　model　allows　us　to　develop　a　framework　that　can　accurately　and　efficiently　track　object　silhouettes　that　can　change　greatly　over　time,　as　well　as　under　different　situations　such　as　occlusion　and　multiple　targets　within　the　scene.　Experiment　results　using　video　surveillance　datasets　containing　different　scenarios　such　as　occlusion　and　multiple　targets　showed　that　the　proposed　DS-CRF　approach　provides　strong　object　silhouette　tracking　performance　when　compared　to　baseline　methods　such　as　mean-shift　tracking,　as　well　as　state-of-the-art　methods　such　as　context　tracking　and　boosted　particle　filtering.　©　2015　Shafiee　et　al.This　is　an　open　access　article　distributed　under　the　terms　of　the　Creative　Commons　Attribution　License,　which　permits　unrestricted　use,　distribution,　and　reproduction　in　any　medium,　provided　the　original　author　and　source　are　credited.