Hand　gesture　is　a　kind　of　natural　interaction　way　and　hand　gesture　recognition　has　recently　become　more　and　more　popular　in　human-computer　interaction.　However,　the　complexity　and　variations　of　hand　gesture　like　various　illuminations,　views,　and　self-structural　characteristics　make　the　hand　gesture　recognition　still　challengeable.　How　to　design　an　appropriate　feature　representation　and　classifier　are　the　core　problems.　To　this　end,　this　paper　develops　an　expressive　deep　hybrid　hand　gesture　recognition　architecture　called　CNN-MVRBM-NN.　The　framework　consists　of　three　submodels.　The　CNN　submodel　automatically　extracts　frame-level　spatial　features,　and　the　MVRBM　submodel　fuses　spatial　information　over　time　for　training　higher　level　semantics　inherent　in　hand　gesture,　while　the　NN　submodel　classifies　hand　gesture,　which　is　initialized　by　MVRBM　for　second　order　data　representation,　and　then　such　NN　pre-trained　by　MVRBM　can　be　fine-tuned　by　back　propagation　so　as　to　be　more　discriminative.　The　experimental　results　on　Cambridge　Hand　Gesture　Data　set　show　the　proposed　hybrid　CNN-MVRBM-NN　has　obtained　the　state-of-the-art　recognition　performance.