The　overlapped　time-frequency　two　dimensional　multiplexing　system　(OvHDM,　Overlapped　Hybrid　Division　Multiplexing)　can　significantly　improve　the　spectrum　efficiency　of　communication　system.　The　system　can　be　described　as　a　vector　convolutional　encoding　constraint　model　and　the　optimum　detection　algorithm　is　maximum　likelihood　sequence　detection　(MLSD),　but　the　algorithm　causes　large　detection　delay　due　to　high　complexity.　In　this　paper,　a　mathematic　model　similar　to　MIMO　system　is　derived　for　OvHDM　system　and　sphere　decoding　algorithm　is　applied　to　fast　detect　the　received　signal.　Simulation　results　show　that　the　algorithm　based　on　sphere　decoding　obtains　the　same　performance　of　the　optimum　detection　but　the　computational　complexity　is　significantly　reduced　.The　performance　of　OvHDM　system　is　better　than　that　of　QAM　modulation　with　the　same　spectrum　efficiency　at　a　high　signal　to　noise　ratio.　This　shows　that　improving　the　transmission　rate　with　overlapped　signals　can　achieve　greater　gains　than　using　high-order　QAM　modulation　with　the　same　spectrum　efficiency.　©2010　IEEE.