• 综合
  • 标题
  • 关键词
  • 摘要
  • 学者
  • 期刊-刊名
  • 期刊-ISSN
  • 会议名称
搜索

作者:

Xu, Jungang (Xu, Jungang.) | Huang, Shanshan (Huang, Shanshan.) | Liu, Renfeng (Liu, Renfeng.) | Li, Pengfei (Li, Pengfei.) (学者:李鹏飞)

收录:

EI Scopus

摘要:

The Shuffle module is one of the core modules in Spark platform, its performance directly influences the performance and throughput of the whole Spark platform. The existing memory scheduling algorithm for the Shuffle process only equitably allocates tasks according to the number of tasks without considering the different memory requirements of different tasks, which causes memory utilization to drop and low running efficiency when data is skewed. To solve this problem, one self-adaptive memory scheduling algorithm for the Shuffle process (SAMSAS) is proposed in this paper, which does not need to set the priority of task processing in advance. Instead, it can adjust memory allocation self-adaptively through constantly monitoring and learning the actual memory requirements of task execution. The experimental results show that SAMSAS algorithm can improve the utilization rate of the entire memory pool and the running efficiency of each Task, and specially it can effectively improve the running efficiency of Spark platform when processing skew data. © 2018 IEEE.

关键词:

Big data Data handling Efficiency Memory architecture

作者机构:

  • [ 1 ] [Xu, Jungang]University of Chinese Academy of Sciences, Beijing, China
  • [ 2 ] [Huang, Shanshan]Beijing University of Technology, Beijing, China
  • [ 3 ] [Liu, Renfeng]University of Chinese Academy of Sciences, Beijing, China
  • [ 4 ] [Li, Pengfei]University of Chinese Academy of Sciences, Beijing, China

通讯作者信息:

电子邮件地址:

查看成果更多字段

相关关键词:

相关文章:

来源 :

年份: 2018

页码: 3938-3946

语种: 英文

被引次数:

WoS核心集被引频次: 0

SCOPUS被引频次: 1

ESI高被引论文在榜: 0 展开所有

万方被引频次:

中文被引频次:

近30日浏览量: 3

归属院系:

在线人数/总访问数:520/2899939
地址:北京工业大学图书馆(北京市朝阳区平乐园100号 邮编:100124) 联系我们:010-67392185
版权所有:北京工业大学图书馆 站点建设与维护:北京爱琴海乐之技术有限公司