The　semantic　gap,　between　the　low-level　visual　features　and　the　high-level　perceptive　or　semantic　concepts,　is　a　big　hurdle　in　content-based　image　retrieval.　To　bridge　the　semantic　gap,　segmentation,　machine-learning,　clustering　and　classification　techniques　have　been　widely　used　in　the　preprocessing　stages　or　during　the　relevance　feedback.　However,　in　these　techniques,　there　are　some　problems　such　as　long　training　or　learning　time,　high　computational　complexity,　some　bad　singular　results　occurring　after　feedback;　and　relearning　required　in　the　retrieval　process　for　new　queries.　According　to　the　fuzzy　characteristic　of　the　human＇s　semantic　knowledge,　this　paper　presents　a　novel　Fuzzy　Semantic　Relevance　Matrix　(FSRM)　to　bridge　the　gap　between　low-level　features　and　semantic　concepts.　The　updating　of　FSRM　imitates　the　human＇s　brain　to　search　the　similar　images　in　the　knowledge　network　and　improve　retrieval　results　continuously　by　memorizing　the　semantic　concepts　learned　in　previous　relevance　feedback　processes.　Experimental　results　demonstrate　the　effectiveness　of　the　proposed　retrieval　scheme.