With　the　maturity　of　vision-based　vehicle　detection　and　tracking,　vision-based　behavior　analysis　of　on-road　vehicles　has　emerged　as　an　active　research　field,　which　sheds　light　on　the　environmental　perception　of　autonomous　driving　and　intelligent　traffic　monitoring.　In　this　paper,　we　are　committed　to　predicting　vehicle　behavior　by　incorporating　the　structure　information　of　vehicle　behavior　into　the　learning　process.　Inspired　by　structured　learning,　the　structure　information　is　extracted　from　the　detected　vehicle　as　its　corresponding　structured　label,　which　visually　expresses　the　vehicle　behavior　as　contrast　to　the　discrete　numeral　label.　With　the　structured　label,　a　structured　convolutional　neural　networks　(SCNN)　method　is　constructed　to　predict　the　vehicle　behavior.　As　for　performance　evaluation,　recognition　accuracy　and　contour　similarity　are　used.　Experimental　results　demonstrate　that　the　proposed　method　achieves　better　recognition　accuracy　than　traditional　methods　of　discrete　labels,　while　the　learned　structured　labels　have　the　implication　of　semantic　interpretation　of　on-road　vehicle　behavior.