• 综合
  • 标题
  • 关键词
  • 摘要
  • 学者
  • 期刊-刊名
  • 期刊-ISSN
  • 会议名称
搜索

作者:

Luo, Dan (Luo, Dan.) | Zheng, Kangfeng (Zheng, Kangfeng.) | Wu, Chunhua (Wu, Chunhua.) | Wang, Xiujuan (Wang, Xiujuan.) | Wang, Jvjie (Wang, Jvjie.)

收录:

EI Scopus SCIE

摘要:

Despite their potential, the industrial deployment of large language models (LLMs) is constrained by traditional fine-tuning procedures that are both resource-intensive and time-consuming. Low-Rank Adaptation (LoRA) has emerged as a pioneering methodology for addressing these challenges. By integrating low-rank decomposition matrices into network weights to reduce trainable parameters, LoRA effectively accelerates the adaptation process. While research on LoRA primarily focuses on adjusting low-rank matrices, DyLoRA optimizes the rank-setting mechanism to avoid extensive effort in rank size training and searching. However, DyLoRA rank configuration mechanism has its own limitation. First, DyLoRA sets the same rank size for all the low-rank adaptation layers at each time step. Given that layers with different depth contain distinct information, they should have varying rank values to accurately capture their unique characteristics. Second, the truncated phase selected for ordering representation based on nested dropout regulation is only half dynamic, continuously dropping tail units, thereby limiting its ability to access information. In this work, we propose a novel technique, enhanced range adaptation in time and depth aware dynamic LoRA (ERAT-DLoRA) to address these problems. The ERAT-DLoRA method introduces a dynamic range to the truncated phase that makes the truncated phase fully dynamic. Additionally, we design a time and layer-aware dynamic rank to ensure appropriate adjustments at different time steps and layer levels. We evaluate our solution on natural languages understanding and language generation tasks. Extensive evaluation results demonstrate the effectiveness of the proposed method.

关键词:

LoRA Fine-tuning Parameter-efficient

作者机构:

  • [ 1 ] [Luo, Dan]Beijing Univ Posts & Telecommun, 10 Xitu Cheng Rd, Beijing 100876, Peoples R China
  • [ 2 ] [Zheng, Kangfeng]Beijing Univ Posts & Telecommun, 10 Xitu Cheng Rd, Beijing 100876, Peoples R China
  • [ 3 ] [Wu, Chunhua]Beijing Univ Posts & Telecommun, 10 Xitu Cheng Rd, Beijing 100876, Peoples R China
  • [ 4 ] [Wang, Jvjie]Beijing Univ Posts & Telecommun, 10 Xitu Cheng Rd, Beijing 100876, Peoples R China
  • [ 5 ] [Wang, Xiujuan]Beijing Univ Technol, 100 Pingleyuan, Beijing 100124, Peoples R China

通讯作者信息:

  • [Zheng, Kangfeng]Beijing Univ Posts & Telecommun, 10 Xitu Cheng Rd, Beijing 100876, Peoples R China

查看成果更多字段

相关关键词:

相关文章:

来源 :

NEUROCOMPUTING

ISSN: 0925-2312

年份: 2024

卷: 614

6 . 0 0 0

JCR@2022

被引次数:

WoS核心集被引频次:

SCOPUS被引频次: 1

ESI高被引论文在榜: 0 展开所有

万方被引频次:

中文被引频次:

近30日浏览量: 1

归属院系:

在线人数/总访问数:525/4931832
地址:北京工业大学图书馆(北京市朝阳区平乐园100号 邮编:100124) 联系我们:010-67392185
版权所有:北京工业大学图书馆 站点建设与维护:北京爱琴海乐之技术有限公司