Reinforcement　learning　(RL)　has　roots　in　dynamic　programming　and　it　is　called　adaptive/approximate　dynamic　programming　(ADP)　within　the　control　community.　This　paper　reviews　recent　developments　in　ADP　along　with　RL　and　its　applications　to　various　advanced　control　fields.　First,　the　background　of　the　development　of　ADP　is　described,　emphasizing　the　significance　of　regulation　and　tracking　control　problems.　Some　effective　offline　and　online　algorithms　for　ADP/adaptive　critic　control　are　displayed,　where　the　main　results　towards　discrete-time　systems　and　continuous-time　systems　are　surveyed,　respectively.　Then,　the　research　progress　on　adaptive　critic　control　based　on　the　event-triggered　framework　and　under　uncertain　environment　is　discussed,　respectively,　where　event-based　design,　robust　stabilization,　and　game　design　are　reviewed.　Moreover,　the　extensions　of　ADP　for　addressing　control　problems　under　complex　environment　attract　enormous　attention.　The　ADP　architecture　is　revisited　under　the　perspective　of　data-driven　and　RL　frameworks,　showing　how　they　promote　ADP　formulation　significantly.　Finally,　several　typical　control　applications　with　respect　to　RL　and　ADP　are　summarized,　particularly　in　the　fields　of　wastewater　treatment　processes　and　power　systems,　followed　by　some　general　prospects　for　future　research.　Overall,　the　comprehensive　survey　on　ADP　and　RL　for　advanced　control　applications　has　demonstrated　its　remarkable　potential　within　the　artificial　intelligence　era.　In　addition,　it　also　plays　a　vital　role　in　promoting　environmental　protection　and　industrial　intelligence.