Deep　learning-based　visual　sentiment　analysis　requires　a　large　dataset　for　training.　Dataset　from　social　networks　is　popular　but　noisy　because　some　images　collected　in　this　manner　are　mislabeled.　Therefore,　it　is　necessary　to　refine　the　dataset.　Based　on　observations　to　such　datasets,　we　propose　a　refinement　algorithm　based　on　the　sentiments　of　adjective-noun　pairs　(ANPs)　and　tags.　We　first　determine　the　unreliably　labeled　images　through　the　sentiment　contradiction　between　the　ANPs　and　tags.　These　images　are　removed　if　the　numbers　of　tags　with　positive　and　negative　sentiments　are　equal.　The　remaining　images　are　labeled　again　based　on　the　majority　vote　of　the　tags＇　sentiments.　Furthermore,　we　improve　the　traditional　deep　learning　model　by　combining　the　softmax　and　Euclidean　loss　functions.　Additionally,　the　improved　model　is　trained　using　the　refined　dataset.　Experiments　demonstrate　that　both　the　dataset　refinement　algorithm　and　improved　deep　learning　model　are　beneficial.　The　proposed　algorithms　outperform　the　benchmark　results.　©　2017　IEEE.