Current　methods　for　detecting　deep　fakes　concentrate　on　specific　patterns　of　forgery　like　noise　characteristics,　local　textures,　or　frequency　statistics.　These　approaches　assume　training　and　test　sets　exhibit　similar　data　distributions,　which　bring　severe　performance　drops　and　further　limit　broader　applications　when　migrating　unseen　domains.　Existing　works　show　that　reconstruction　learning　is　effective　in　capturing　unseen　forgery　clues.　However,　2D　reconstruction　is　insufficient　and　can　not　handle　non-frontal　face　reconstruction,　while　3D　reconstruction　provides　more　critical　details　of　facial　structure　and　finds　accurate　forgery　regions.　In　this　paper,　we　propose　a　bi-source　reconstruction　based　classification　network　(BRCNet)　to　incorporate　2D　and　3D　reconstruction　as　the　supervisions　and　learn　the　optimal　feature　representation.　In　detail,　we　employ　an　encoder-decoder　architecture　to　facilitate　reconstruction　learning,　enhancing　the　learned　representations　to　detect　forgery　patterns　that　are　unknown.　To　further　capture　forgery　evidence　across　multiple　scales,　instead　of　using　encoder　features　from　the　reconstruction　network　only,　we　build　a　feature　improvement　network　to　combine　feature　details　from　encoder　and　decoder　features　in　a　multi-scale　fashion.　In　addition,　we　use　the　reconstruction　difference　to　supervise　the　feature　aggregation,　which　enables　detecting　the　subtle　and　trivial　discrepancies　between　fake　and　real　video　frames.　Extensive　experiments　are　conducted　to　validate　the　performance　of　our　proposed　method　on　several　deep　fake　benchmarks.　The　results　demonstrate　the　efficacy　of　our　approach,　offering　promising　results　and　showcasing　its　potential　for　practical　applications.　The　source　code　is　available　at　https://github.com/cccvl/BRCNet.