Aiming　at　the　complexity,　nonlinearity　and　difficulty　in　modeling　of　nonlinear　system.　In　this　paper,　an　improved　back-propagation(BP)　neural　network　based　on　restricted　boltzmann　machine(RBM-IBPNN)　is　proposed　for　nonlinear　systems　modeling.　First,　the　structure　of　BP　neural　network(BPNN)　is　optimized　by　using　sensitivity　analysis(SA)　and　mutual　information(MI)　of　the　hidden　neurons.　Namely　when　the　SA　value　and　the　MI　value　of　the　hidden　neurons　satisfy　the　set　standard,　the　corresponding　neurons　will　be　pruned,　split　or　merged.　second,　the　restricted　boltzmann　machine(RBM)　is　employed　to　perform　parameters　initialization　of　training　on　the　IBPNN.　Finally,　the　proposed　RBM-IBPNN　is　evaluated　on　nonlinear　system　identification,　lorenz　chaotic　time　series　prediction　and　the　total　phosphorus　prediction　problems.　The　experimental　results　demonstrate　that　the　proposed　RBM-IBPNN　not　only　has　faster　convergence　speed　and　higher　prediction　accuracy,　but　also　realizes　a　more　compact　network　structure.