收录:
摘要:
Complex data-parallel job contains task dependency information defined as Directed Acyclic Graph (DAG). For convenience, the DAG presented data-parallel jobs are named as DAG jobs. The prevalence of DAG jobs in modern data centers has made the scheduling oriented job characterization a big challenge. This paper proposes a deep graph-temporal clustering framework, i.e., DeGTeC, to efficiently categorize DAG jobs leveraging the graphic and temporal information in DAGs. The categorization result can then be naturally used to characterize the resource consumption pattern of DAG jobs. The DeGTeC framework is constructed mainly based on two autoencoders, i.e., TaskAE and JobAE. TaskAE and JobAE contain spectral graph convolutional network (GCN) layers, temporal convolutional network (TCN) layers, and the adaptive pooling layers to help build task embeddings and job embeddings. An extra embedding sorting step takes in the sequential order information and the depth-bias information for job clustering. To our best knowledge, DeGTeC is the first solution to do resource consumption characterization of DAG jobs fully leveraging the task dependencies defined in DAG. Experimental results demonstrate that the DeGTeC framework outperforms the state-of-the-art job resource consumption characterization methods.(c) 2022 Elsevier B.V. All rights reserved.
关键词:
通讯作者信息:
电子邮件地址:
来源 :
FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE
ISSN: 0167-739X
年份: 2023
卷: 141
页码: 81-95
7 . 5 0 0
JCR@2022
ESI学科: COMPUTER SCIENCE;
ESI高被引阀值:19
归属院系: