• 综合
  • 标题
  • 关键词
  • 摘要
  • 学者
  • 期刊-刊名
  • 期刊-ISSN
  • 会议名称
搜索

作者:

Liang, Yi (Liang, Yi.) | Wang, Yufeng (Wang, Yufeng.) | Fan, Minglu (Fan, Minglu.) | Zhang, Chen (Zhang, Chen.) | Zhu, Yuqing (Zhu, Yuqing.)

收录:

EI Scopus

摘要:

Map/Reduce is a popular parallel processing framework for data intensive computing. For overlapping the Map task’s execution phase and the Reduce task’s intermediate data fetching and merging phase, existing Map/Reduce schedulers always pre-launch the Reduce task at the specific threshold where its map tasks have been launched, and this pattern incurs the occupation of the consuming resources of the reduce task during its idle time on waiting for fetching the intermediate data from map tasks. To address this issue, we propose an extension version of Hadoop map/reduce framework, called Predoop, in this paper. The basic idea of Predoop is to preempt the reduce task during its idle time and allocate the released resource to the map tasks on schedule. To achieve this goal, first, we introduce the preemptive mechanism for reduce tasks and map tasks respectively to enable Map/Reduce tasks to be preempted or resumed with correct status; second, we adopt the preempting-resuming model for the reduce task with the consideration of the progress of Reduce task data fetching & merging and the Map task execution so as to determine the timing of Reduce task preemption and resuming; third, we introduce the preemption-aware task scheduling strategy to allocate the released resources to the on-schedule Map tasks with the consideration of data locality. Experimental result demonstrates that Predoop outperforms Hadoop on various workload and the average job turnaround time can be reduced by maximum of 66.57%. © Springer International Publishing Switzerland 2014.

关键词:

Benchmarking Data reduction Merging Multitasking Scheduling algorithms

作者机构:

  • [ 1 ] [Liang, Yi]Department of Computer Science, Beijing University of Technology, Beijing, China
  • [ 2 ] [Wang, Yufeng]Department of Computer Science, Beijing University of Technology, Beijing, China
  • [ 3 ] [Fan, Minglu]Department of Computer Science, Beijing University of Technology, Beijing, China
  • [ 4 ] [Zhang, Chen]Department of Computer Science, Beijing University of Technology, Beijing, China
  • [ 5 ] [Zhu, Yuqing]State Key Laboratory of Computer Architecture, Institute of Computing Technology, Chinese Academy Sciences, Beijing, China

通讯作者信息:

  • [liang, yi]department of computer science, beijing university of technology, beijing, china

电子邮件地址:

查看成果更多字段

相关关键词:

来源 :

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

ISSN: 0302-9743

年份: 2014

卷: 8807

页码: 167-180

被引次数:

WoS核心集被引频次: 0

SCOPUS被引频次: 3

ESI高被引论文在榜: 0 展开所有

万方被引频次:

中文被引频次:

近30日浏览量: 2

在线人数/总访问数:1069/2995047
地址:北京工业大学图书馆(北京市朝阳区平乐园100号 邮编:100124) 联系我们:010-67392185
版权所有:北京工业大学图书馆 站点建设与维护:北京爱琴海乐之技术有限公司