• Complex
  • Title
  • Keyword
  • Abstract
  • Scholars
  • Journal
  • ISSN
  • Conference
搜索

Author:

Liang, Yi (Liang, Yi.) | Li, Guangrui (Li, Guangrui.) | Wang, Lei (Wang, Lei.) | Hu, Yanpeng (Hu, Yanpeng.)

Indexed by:

EI Scopus

Abstract:

Map/reduce is a popular parallel processing framework for massive-scale data-intensive computing. The data-iterative application is composed of a serials of map/reduce jobs and need to repeatedly process some data files among these jobs. The existing implementation of map/reduce framework focus on perform data processing in a single pass with one map/reduce job and do not directly support the data-iterative applications, particularly in term of the explicit specification of the repeatedly processed data among jobs. In this paper, we propose an extended version of Hadoop map/reduce framework called Dacoop. Dacoop extends Map/Reduce programming interface to specify the repeatedly processed data, introduces the shared memorybased data cache mechanism to cache the data since its first access, and adopts the caching-aware task scheduling so that the cached data can be shared among the map/reduce jobs of data-iterative applications. We evaluate Dacoop on two typical data-iterative applications: k-means clustering and the domain rule reasoning in sementic web, with real and synthetic datasets. Experimental results show that the data-iterative applications can gain better performance on Dacoop than that on Hadoop. The turnaround time of a data-iterative application can be reduced by the maximum of 15.1%. © 2011 IEEE.

Keyword:

Multitasking Distributed computer systems Scheduling algorithms K-means clustering Cache memory

Author Community:

  • [ 1 ] [Liang, Yi]Department of Computer Science, Beijing University of Technology, Beijing, China
  • [ 2 ] [Li, Guangrui]Department of Computer Science, Beijing University of Technology, Beijing, China
  • [ 3 ] [Wang, Lei]Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
  • [ 4 ] [Hu, Yanpeng]Hwellzen Software Center, Shanghai, China

Reprint Author's Address:

Show more details

Related Keywords:

Related Article:

Source :

Year: 2011

Page: 207-214

Language: English

Cited Count:

WoS CC Cited Count:

SCOPUS Cited Count: 2

ESI Highly Cited Papers on the List: 0 Unfold All

WanFang Cited Count:

Chinese Cited Count:

30 Days PV: 0

Affiliated Colleges:

Online/Total:647/5318519
Address:BJUT Library(100 Pingleyuan,Chaoyang District,Beijing 100124, China Post Code:100124) Contact Us:010-67392185
Copyright:BJUT Library Technical Support:Beijing Aegean Software Co., Ltd.