• Complex
  • Title
  • Keyword
  • Abstract
  • Scholars
  • Journal
  • ISSN
  • Conference
搜索

Author:

Ying, C. (Ying, C..) | Yu, J. (Yu, J..) | He, J.S. (He, J.S..) (Scholars:何泾沙)

Indexed by:

Scopus

Abstract:

Performance optimization especially for fault tolerance optimization has been a significant aspect for in-memory computing computing. When node failure occurs, in-memory computing may lose the data, which increases the execution time without checkpoint. In the traditional Spark strategy, the programmer chooses the checkpoint with the uncertainty and risk. Therefore, we aims at the checkpoint strategy of in memory computing framework Spark in this paper. After the theoretical analysis, the checkpoint selection algorithm which taking into account the length of the RDD lineage, the computational cost, the operation complexity and the size in setting the checkpoint is presented. The greater the weight of RDD, the higher priority it has. The RDD with higher cost will be set as the checkpoint first, which can reduce the recomputation cost of the task. When failure occurs, the recovery algorithm is executed, and the efficiency of the task recovery can be effectively improved. And the experimental results show that the strategy optimizes the fault tolerance mechanism for Spark and improves the efficiency of the job recovery. © 2018, Springer-Verlag GmbH Germany, part of Springer Nature.

Keyword:

Failure recovery; RDD weight; Recovery efficiency checkpoint selection; Spark

Author Community:

  • [ 1 ] [Ying, C.]School of Mechanical and Electrical Engineering, Shaoxing University, Shaoxing, China
  • [ 2 ] [Ying, C.]School of Software, Xinjiang University, Urumqi, China
  • [ 3 ] [Yu, J.]School of Software, Xinjiang University, Urumqi, China
  • [ 4 ] [He, J.S.]School of Software, Beijing University of Technology, Beijing, China

Reprint Author's Address:

  • [Ying, C.]School of Software, Xinjiang UniversityChina

Show more details

Related Keywords:

Related Article:

Source :

Journal of Ambient Intelligence and Humanized Computing

ISSN: 1868-5137

Year: 2018

ESI Discipline: COMPUTER SCIENCE;

ESI HC Threshold:161

JCR Journal Grade:3

Cited Count:

WoS CC Cited Count:

SCOPUS Cited Count:

ESI Highly Cited Papers on the List: 0 Unfold All

WanFang Cited Count:

Chinese Cited Count:

30 Days PV: 1

Affiliated Colleges:

Online/Total:488/5439771
Address:BJUT Library(100 Pingleyuan,Chaoyang District,Beijing 100124, China Post Code:100124) Contact Us:010-67392185
Copyright:BJUT Library Technical Support:Beijing Aegean Software Co., Ltd.