With　the　rapid　prevalence　of　video　application,　shot　boundary　detection　(SBD)　as　a　fundamental　and　essential　technique　in　video　analysis　has　aroused　great　attention.　In　this　paper,　to　detect　the　boundary　frames　of　video　shots　automatically　we　propose　a　joint　learning　framework　over　optical　flow　and　color　histogram　for　shot　boundary　detection.　The　proposed　method　carries　out　motion　estimation　on　key　points　to　obtain　roughly　video　segmen-tation　results,　and　simultaneously　evaluate　visual　differentiation　over　spatial　appearance　to　help　derive　more　accurate　boundary　shot　segmentation.　By　constructing　differential　sequences　of　consecutive　frames,　the　abrupt　shot　and　gradual　shot　are　successfully　distinguished　with　the　variation　in　the　dif-ferential　sequences.　The　experimental　performance　demonstrate　that　the　proposed　method　has　a　precision　of　10%　higher　at　least　than　that　of　the　color　histogram-based　approach,　and　the　F　value　is　achieved　about　10%　higher.　©　2019　IEEE.