• 综合
  • 标题
  • 关键词
  • 摘要
  • 学者
  • 期刊-刊名
  • 期刊-ISSN
  • 会议名称
搜索

作者:

Xu, Jungang (Xu, Jungang.) | Huang, Shanshan (Huang, Shanshan.) | Liu, Renfeng (Liu, Renfeng.) | Li, Pengfei (Li, Pengfei.) (学者:李鹏飞)

收录:

CPCI-S

摘要:

The Shuffle module is one of the core modules in Spark platform, its performance directly influences the performance and throughput of the whole Spark platform. The existing memory scheduling algorithm for the Shuffle process only equitably allocates tasks according to the number of tasks without considering the different memory requirements of different tasks, which causes memory utilization to drop and low running efficiency when data is skewed. To solve this problem, one self-adaptive memory scheduling algorithm for the Shuffle process (SAMSAS) is proposed in this paper, which does not need to set the priority of task processing in advance. Instead, it can adjust memory allocation self -adaptively through constantly monitoring and learning the actual memory requirements of task execution. The experimental results show that SAMSAS algorithm can improve the utilization rate of the entire memory pool and the running efficiency of each Task, and specially it can effectively improve the running efficiency of Spark platform when processing skew data.

关键词:

作者机构:

  • [ 1 ] [Xu, Jungang]Univ Chinese Acad Sci, Beijing, Peoples R China
  • [ 2 ] [Liu, Renfeng]Univ Chinese Acad Sci, Beijing, Peoples R China
  • [ 3 ] [Li, Pengfei]Univ Chinese Acad Sci, Beijing, Peoples R China
  • [ 4 ] [Huang, Shanshan]Beijing Univ Technol, Beijing, Peoples R China

通讯作者信息:

  • [Xu, Jungang]Univ Chinese Acad Sci, Beijing, Peoples R China

查看成果更多字段

相关关键词:

相关文章:

来源 :

2018 IEEE INTERNATIONAL CONFERENCE ON BIG DATA (BIG DATA)

ISSN: 2639-1589

年份: 2018

页码: 3938-3946

语种: 英文

被引次数:

WoS核心集被引频次: 0

SCOPUS被引频次:

ESI高被引论文在榜: 0 展开所有

万方被引频次:

中文被引频次:

近30日浏览量: 3

归属院系:

在线人数/总访问数:369/2899774
地址:北京工业大学图书馆(北京市朝阳区平乐园100号 邮编:100124) 联系我们:010-67392185
版权所有:北京工业大学图书馆 站点建设与维护:北京爱琴海乐之技术有限公司