• 综合
  • 标题
  • 关键词
  • 摘要
  • 学者
  • 期刊-刊名
  • 期刊-ISSN
  • 会议名称
搜索

作者:

Ying, C. (Ying, C..) | Yu, J. (Yu, J..) | He, J.S. (He, J.S..) (学者:何泾沙)

收录:

Scopus

摘要:

Performance optimization especially for fault tolerance optimization has been a significant aspect for in-memory computing computing. When node failure occurs, in-memory computing may lose the data, which increases the execution time without checkpoint. In the traditional Spark strategy, the programmer chooses the checkpoint with the uncertainty and risk. Therefore, we aims at the checkpoint strategy of in memory computing framework Spark in this paper. After the theoretical analysis, the checkpoint selection algorithm which taking into account the length of the RDD lineage, the computational cost, the operation complexity and the size in setting the checkpoint is presented. The greater the weight of RDD, the higher priority it has. The RDD with higher cost will be set as the checkpoint first, which can reduce the recomputation cost of the task. When failure occurs, the recovery algorithm is executed, and the efficiency of the task recovery can be effectively improved. And the experimental results show that the strategy optimizes the fault tolerance mechanism for Spark and improves the efficiency of the job recovery. © 2018, Springer-Verlag GmbH Germany, part of Springer Nature.

关键词:

Failure recovery; RDD weight; Recovery efficiency checkpoint selection; Spark

作者机构:

  • [ 1 ] [Ying, C.]School of Mechanical and Electrical Engineering, Shaoxing University, Shaoxing, China
  • [ 2 ] [Ying, C.]School of Software, Xinjiang University, Urumqi, China
  • [ 3 ] [Yu, J.]School of Software, Xinjiang University, Urumqi, China
  • [ 4 ] [He, J.S.]School of Software, Beijing University of Technology, Beijing, China

通讯作者信息:

  • [Ying, C.]School of Software, Xinjiang UniversityChina

电子邮件地址:

查看成果更多字段

相关关键词:

相关文章:

来源 :

Journal of Ambient Intelligence and Humanized Computing

ISSN: 1868-5137

年份: 2018

ESI学科: COMPUTER SCIENCE;

ESI高被引阀值:81

JCR分区:3

被引次数:

WoS核心集被引频次: 0

SCOPUS被引频次: 8

ESI高被引论文在榜: 0 展开所有

万方被引频次:

中文被引频次:

近30日浏览量: 1

归属院系:

在线人数/总访问数:391/2893054
地址:北京工业大学图书馆(北京市朝阳区平乐园100号 邮编:100124) 联系我们:010-67392185
版权所有:北京工业大学图书馆 站点建设与维护:北京爱琴海乐之技术有限公司