A strategy for small files processing in HDFS - 文章详情页

作者：

Bao, Z. (Bao, Z..) | Xu, S. (Xu, S..) (学者：徐硕) | Zhang, W. (Zhang, W..) | Chen, J. (Chen, J..) | Liu, J. (Liu, J..)

收录：

Scopus

摘要：

Hadoop　distributed　file　system　(HDFS)　as　a　popular　cloud　storage　platform,　benefiting　from　its　scalable,　reliable　and　low-cost　storage　capability.　However　it　is　mainly　designed　for　batch　processing　of　large　files,　it’s　mean　that　small　files　cannot　be　efficiently　handled　by　HDFS.　In　this　paper,　we　propose　a　mechanism　to　store　small　files　in　HDFS.　In　our　approach,　file　size　need　to　be　judged　before　uploading　to　HDFS.　If　the　file　size　is　less　than　the　size　of　the　block,　all　correlated　small　files　will　be　merged　into　one　single　file　and　we　will　build　index　for　each　small　file.　Furthermore,　prefetching　and　caching　mechanism　are　used　to　improve　the　reading　efficiency　of　small　files.　Meanwhile,　for　the　new　small　files,　we　can　execute　appending　operation　on　the　basis　of　merged　file.　Contrasting　to　original　HDFS,　experimental　results　show　that　the　storage　efficiency　of　small　files　is　improved.　©　Springer　Science+Business　Media　Singapore　2016.

关键词：

Appending operation; File merging; Hadoop; HDFS; Prefetching and caching; Small file

作者机构：

[ 1 ] [Bao, Z.]College of Computer Science, Beijing University of Technology, Beijing, 100124, China
[ 2 ] [Xu, S.]College of Computer Science, Beijing University of Technology, Beijing, 100124, China
[ 3 ] [Zhang, W.]College of Computer Science, Beijing University of Technology, Beijing, 100124, China
[ 4 ] [Chen, J.]College of Computer Science, Beijing University of Technology, Beijing, 100124, China
[ 5 ] [Liu, J.]College of Computer Science, Beijing University of Technology, Beijing, 100124, China

通讯作者信息：

[Bao, Z.]College of Computer Science, Beijing University of TechnologyChina

电子邮件地址：

baozhenshan@bjut.edu.cn

查看成果更多字段

成果类型
所属机构

所有年份指定年份从至