Convolutional　neural　networks　(CNNs)　have　better　performance　in　feature　extraction　and　classification.　Most　of　the　applications　are　based　on　a　traditional　structure　of　CNNs.　However,　due　to　the　fixed　structure,　it　may　not　be　effective　for　large　dataset　which　will　spend　much　time　for　training.　So,　we　use　a　new　algorithm　to　optimize　CNNs,　called　directly　connected　convolutional　neural　networks　(DCCNNs).　In　DCCNNs,　the　down-sampling　layer　can　directly　connect　the　output　layer　with　three-dimensional　matrix　operation,　without　full　connection　(i.e.,　matrix　vectorization).　Thus,　DCCNNs　have　less　weights　and　neurons　than　CNNs.　We　conduct　the　comparison　experiments　on　five　image　databases:　MNIST,　COIL-20,　AR,　Extended　Yale　B,　and　ORL.　The　experiments　show　that　the　model　has　better　recognition　accuracy　and　faster　convergence　than　CNNs.　Furthermore,　two　applications　(i.e.,　water　quality　evaluation　and　image　classification)　following　the　proposed　concepts　further　confirm　the　generality　and　capability　of　DCCNNs.