Traffic　sign　detection　(TSD)　using　convolutional　neural　networks　(CNN)　is　promising　and　intriguing　for　autonomous　driving.　Especially,　with　sophisticated　large-scale　CNN　models,　TSD　can　be　performed　with　high　accuracy.　However,　the　conventional　CNN　models　suffer　the　drawbacks　of　being　time-consuming　and　resource-hungry,　which　limit　their　application　and　deployments　in　various　platforms　of　limited　resources.　In　this　paper,　we　propose　a　novel　real-time　traffic　sign　detection　system　with　a　lightweight　backbone　network　named　Depth　Separable　DetNet　(DS-DetNet)　and　a　lite　fusion　feature　pyramid　network　(LFFPN)　for　efficient　feature　fusion.　The　new　model　can　achieve　a　performance　trade-off　between　speed　and　accuracy　using　a　depthwise　separable　bottleneck　block,　a　lite　fusion　module,　and　an　improved　SSD　detection　front-end.　The　testing　results　on　the　MS　COCO　and　the　GTSDB　datasets　reveal　that　23.1%　mAP　with　6.39　M　parameters　and　only　1.08B　FLOPs　on　MSCOCO,　81.35%　mAP　with　5.78　M　parameters　on　GTSDB.　With　our　model,　the　run　speed　is　61　frames　per　second　(fps)　on　GTX　1080ti,　12　fps　on　Nvidia　Jetson　Nano　and　16　fps　on　Nvidia　Jetson　Xavier　NX.