How　to　efficiently　retrieve　the　images　while　preserving　the　user＇s　privacy　has　gradually　become　a　key　problem　in　some　applications　such　as　Cloud　storage,　social　networks.　In　this　paper,　a　secure　index　used　for　image　retrieval　is　constructed　to　protect　the　retrieval　results　being　leaked　to　the　malicious　attackers.　At　first,　inverted　index　is　generated　using　visual　words　of　images　and　then　encrypted　dually　by　randomized　binary　encoding　and　a　key-based　Gaussian　random　matrix　respectively,　generating　a　secure　index.　The　similarity　of　the　encrypted　indexes　is　measured　directly　in　the　encrypted　domain　without　decryption　and　the　most　similar　k　images　will　be　returned　as　the　retrieval　results　to　the　users.　Experimental　results　and　security　analysis　show　that　the　secure　index　proposed　in　this　paper　can　achieve　comparable　retrieval　performance　to　the　conventional　large　scale　image　retrieval　method,　while　protecting　user＇s　privacy.