• Complex
  • Title
  • Keyword
  • Abstract
  • Scholars
  • Journal
  • ISSN
  • Conference
搜索

Author:

Wang, Guolu (Wang, Guolu.) | Xu, Jungang (Xu, Jungang.) | Liu, Renfeng (Liu, Renfeng.) | Huang, Shanshan (Huang, Shanshan.)

Indexed by:

CPCI-S EI Scopus

Abstract:

Apache Spark is a fast and general engine for large-scale data processing using distributed memory. It provides different deploy modes to meet the needs of different users and Spark on YARN is the most popular deploy mode. Different deploy modes have different scheduling mechanisms. Spark on YARN has three different schedulers, including FIFO Scheduler, Fair Scheduler, and Capacity Scheduler. However, these three schedulers cannot fit hard real-time application scenarios. With the application of Apache Spark more widely, the needs of hard real-time scheduling will increase quickly. In this paper, we proposed a novel hard real-time scheduling algorithm called DVDA (Deadline and Value Density-Aware) in order to meet the requirements of hard real-time scheduling. Compared with traditional EDF (Earliest Deadline First) algorithm which only considers the deadline, the DVDA algorithm considers both the deadline and value density of the application. Furthermore, we implement a DVDA Scheduler for Spark on YARN based on the DVDA algorithm. Finally, the experiments are conducted to verify the effectiveness of the algorithm. Experimental results show that the proposed algorithm can increase the application completed rate by 18% and 6%, Value Income by 78% and 32% compared with default Capacity scheduler and EDF-Capacity scheduler respectively.

Keyword:

value density YARN Spark hard real-time deadline

Author Community:

  • [ 1 ] [Wang, Guolu]Univ Chinese Acad Sci, Sch Comp & Control Engn, Beijing, Peoples R China
  • [ 2 ] [Xu, Jungang]Univ Chinese Acad Sci, Sch Comp & Control Engn, Beijing, Peoples R China
  • [ 3 ] [Liu, Renfeng]Univ Chinese Acad Sci, Sch Comp & Control Engn, Beijing, Peoples R China
  • [ 4 ] [Huang, Shanshan]Beijing Univ Technol, Sch Software, Beijing, Peoples R China

Reprint Author's Address:

  • [Wang, Guolu]Univ Chinese Acad Sci, Sch Comp & Control Engn, Beijing, Peoples R China

Show more details

Related Keywords:

Source :

ACM INTERNATIONAL SYMPOSIUM ON CLUSTER, CLOUD AND GRID COMPUTING (CCGRID)

Year: 2018

Page: 645-652

Language: English

Cited Count:

WoS CC Cited Count: 2

SCOPUS Cited Count: 9

ESI Highly Cited Papers on the List: 0 Unfold All

WanFang Cited Count:

Chinese Cited Count:

30 Days PV: 2

Affiliated Colleges:

Online/Total:591/5651393
Address:BJUT Library(100 Pingleyuan,Chaoyang District,Beijing 100124, China Post Code:100124) Contact Us:010-67392185
Copyright:BJUT Library Technical Support:Beijing Aegean Software Co., Ltd.