In　order　to　improve　spectrum　efficiency　of　communication　system,　a　novel　time-frequency　overlapped　hybrid　division　multiplexing　system　was　researched.　A　rapid　detection　algorithm　based　on　Gabor　transform　and　an　improved　detection　algorithm　using　error　back　propagation　method　was　proposed.　The　simulation　results　show　that　the　proposed　rapid　detection　algorithm　obtains　significant　increase　in　detection　speed　at　the　expense　of　some　performance　compared　with　maximum　likelihood　sequence　estimation　algorithm,　and　that　the　improved　detection　algorithm　is　a　good　compromise　between　system　performance　and　detection　speed　and　its　performance　is　still　better　than　the　high　level　modulation　with　the　same　spectrum　efficiency　at　a　high　signal　to　noise　ratio.　This　shows　that　improving　the　transmission　data　rate　with　overlapped　signals　can　achieve　greater　gains　than　using　high-level　QAM　modulation　with　the　same　spectrum　efficiency.