Extreme　weather　recognition　using　GoogLeNet　can　achieve　excellent　performance,　which　is　far　superior　to　the　conventional　methods.　However,　the　complexity　of　GoogLeNet　is　relatively　high.　Furthermore,　for　the　small　scale　data,　GoogLeNet　usually　cannot　achieve　the　performance　as　the　large　scale　data　does.　In　this　paper,　a　novel　dual　fine-tuning　strategy　is　proposed　to　train　the　GoogLeNet　model.　Firstly,　ILSVRC-2012　Dataset　is　applied　to　pre-　train　GoogLeNet　and　the　initial　model　can　be　generated.　Then,　the　initial　model　is　fine-tuned　on　WeatherDataset,　which　is　constructed　in　this　paper,　to　obtain　the　model-1　for　extreme　weather　recognition.　Next,　the　structure　of　GoogLeNet　is　optimized　by　truncating　operation　to　reduce　the　model　size　of　GoogLeNet.　And　the　truncated　GoogLeNet　is　further　fine-tuned　on　WeatherDataset　to　obtain　the　final　recognition　model.　The　experimental　results　have　demonstrated　that,　compared　with　the　original　GoogLeNet,　the　recognition　accuracy　of　the　proposed　method　increases　from　94.74%　to　95.46%,　which　is　improved　by　0.72%.　The　model　size　of　the　optimized　GoogLeNet　is　only　31.23%　of　original　GoogLeNet.　On　CPU　and　GPU　implementation,　the　proposed　method　processes　the　images　1.39　times　faster　and　2.44　faster　than　the　original　GoogLeNet　respectively.　In　summary,　the　proposed　method　is　better　than　the　original　GoogLeNet　in　the　three　aspects　of　recognition　accuracy,　recognition　speed　and　model　size.　©　2017　IEEE.