A　gear　vibration　signal　filtering　method　based　on　fractional　Fourier　Transform　(FRFT)　was　proposed　to　eliminate　background　noise　in　the　signal　and　extract　fault　features　in　gear　up　speed.　With　transform　order　being　the　variable,　two-dimensional　distribution　of　the　signal　energy　in　the　fractional　Fourier　plane　is　formed　through　FRFT　to　gear　vibration　signal.　Then,　peak　points　are　found　through　two-dimensional　search　in　the　plane,　where　the　optimal　order　is　determined　while　the　useful　gear　vibration　signal　is　detected.　A　narrow　band　pass　filter　can　be　used　to　separate　the　useful　components　of　gear　vibration　signal　and　the　background　noise.　Wavelet　packet　decomposition　and　reconstruction　was　applied　in　the　useful　components　extracted　from　the　gear　vibration　signal　to　obtain　the　feature　signal　of　frequency　band.　However,　each　frequency　band　feature　signal　of　the　highest　decomposition　layer　after　wavelet　packet　decomposition　is　lined　up　as　a　gear　fault　feature.　Experimental　results　show　that　the　signal　filtering　method　based　on　FRFT　proposed　in　this　paper　was　able　to　effectively　suppress　background　noise　and　highlight　the　target　component.　The　energy　of　the　wavelet　packet　frequency　band　can　accurately　reflect　the　gear　meshing　transmission　information,　which　can　effectively　extract　the　gear　fault　feature.　©　2017　Taylor　&　Francis　Group,　London.