Numerous　single　image　super-resolution　(SISR)　algorithms　have　been　proposed　during　the　past　years　to　reconstruct　a　high-resolution　(HR)　image　from　its　low-resolution　(LR)　observation.　However,　how　to　fairly　compare　the　performance　of　different　SISR　algorithms/results　remains　a　challenging　problem.　So　far,　the　lack　of　comprehensive　human　subjective　study　on　large-scale　real-world　SISR　datasets　and　accurate　objective　SISR　quality　assessment　metrics　makes　it　unreliable　to　truly　understand　the　performance　of　different　SISR　algorithms.　We　in　this　paper　make　efforts　to　tackle　these　two　issues.　Firstly,　we　construct　a　real-world　SISR　quality　dataset　(i.e.,　RealSRQ)　and　conduct　human　subjective　studies　to　compare　the　performance　of　the　representative　SISR　algorithms.　Secondly,　we　propose　a　new　objective　metric,　i.e.,　KLTSRQA,　based　on　the　Karhunen-Loeve　Transform　(KLT)　to　evaluate　the　quality　of　SISR　images　in　a　no-reference　(NR)　manner.　Experiments　on　our　constructed　RealSRQ　and　the　latest　synthetic　SISR　quality　dataset　(i.e.,　QADS)　have　demonstrated　the　superiority　of　our　proposed　KLTSRQA　metric,　achieving　higher　consistency　with　human　subjective　scores　than　relevant　existing　NR　image　quality　assessment　(NR-IQA)　metrics.　The　dataset　and　the　code　will　be　made　available　at　https://github.com/Zhentao-Liu/RealSRQ-KLTSRQA.