Focus　on　the　problem　of　dynamic　human　detection　and　tracking　in　complex　scenes,　a　physical　structure　based　Convolutional　Nerual　Network　is　proposed.　Firstly,　aiming　at　the　modeling　and　analysis　of　the　human　body　and　its　components,　the　human　body　detection　algorithm　adapted　to　complex　scenes　is　proposed,　and　the　convolutional　neural　network　is　designed　to　realize　the　model.　Secondly,　the　human　body　tracking　model　based　on　convolutional　neural　network　and　off-line　training　is　designed,　and　the　human　body　tracking　algorithm　is　optimized　to　realize　fast　and　accurate　tracking　of　human　body.　Using　IOU,　Euclidean　distance　and　other　algorithms,　the　relationship　between　the　targets　detected　by　the　detection　algorithms　in　two　adjacent　frames　is　established.　Multi-modal　fusion　of　multiple　models　using　a　state　machine　or　the　like,　so　that　multiple　models　can　work　effectively　at　the　same　time.　This　experiment　carried　out　simulation　experiments　on　the　bus　video　dataset.　The　experimental　results　show　that　the　algorithm　can　effectively　track　the　passengers　who　are　obscured　by　each　other　on　the　bus,　and　the　accuracy　exceeds　the　current　best　algorithms,　which　proves　the　effectiveness　of　the　algorithm.　2019,　Springer　Nature　Singapore　Pte　Ltd.