• 综合
  • 标题
  • 关键词
  • 摘要
  • 学者
  • 期刊-刊名
  • 期刊-ISSN
  • 会议名称
搜索

作者:

Fan, Hao (Fan, Hao.) | Ma, Zhaoyang (Ma, Zhaoyang.) | Li, Yong (Li, Yong.) | Tian, Rui (Tian, Rui.) | Chen, Yunli (Chen, Yunli.) | Gao, Chenlong (Gao, Chenlong.)

收录:

CPCI-S EI Scopus

摘要:

Pretrained Vision-Language Models (VLMs) like CLIP have exhibited remarkable capacities across downstream tasks, while their image encoders are vulnerable to adversarial examples. A recently introduced lightweight approach, termed Adversarial Prompt Tuning (AdvPT), utilizes adversarial examples for training learnable prompts, enhancing the adversarial robustness of VLMs solely through manipulation of textual inputs. However, the static prompts learned from AdvPT overfit base classes observed during training, compromising the model's generalizability. In this paper, we propose a conditional Adversarial Prompt Tuning method, which extends AdvPT by further learning a network to generate for each input a specific prompt. The dynamic prompts enhance the generalizability of VLMs on unseen classes. Furthermore, since VLMs are inherently powerful generalizers, we try to incorporate the manual prompts used by VLMs in the testing phase to further improve the generalizability of the model. Extensive experiments on 8 datasets demonstrate that our prompt fusion based method significantly outperforms AdvPT on unseen classes, enhancing the generalizability and adversarial robustness of VLMs simultaneously.

关键词:

Generalizability Prompt tuning Adversarial robustness Vision-Language model

作者机构:

  • [ 1 ] [Fan, Hao]Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China
  • [ 2 ] [Gao, Chenlong]Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China
  • [ 3 ] [Fan, Hao]Beijing Univ Technol, Fac Informat Technol, Beijing, Peoples R China
  • [ 4 ] [Li, Yong]Beijing Univ Technol, Fac Informat Technol, Beijing, Peoples R China
  • [ 5 ] [Tian, Rui]Beijing Univ Technol, Fac Informat Technol, Beijing, Peoples R China
  • [ 6 ] [Chen, Yunli]Beijing Univ Technol, Fac Informat Technol, Beijing, Peoples R China
  • [ 7 ] [Ma, Zhaoyang]Beijing Jiaotong Univ, Sch Comp & Informat Technol, Beijing, Peoples R China

通讯作者信息:

  • [Gao, Chenlong]Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China;;

电子邮件地址:

查看成果更多字段

相关关键词:

相关文章:

来源 :

ADVANCED INTELLIGENT COMPUTING TECHNOLOGY AND APPLICATIONS, PT IX, ICIC 2024

ISSN: 0302-9743

年份: 2024

卷: 14870

页码: 328-339

被引次数:

WoS核心集被引频次:

SCOPUS被引频次:

ESI高被引论文在榜: 0 展开所有

万方被引频次:

中文被引频次:

近30日浏览量: 1

归属院系:

在线人数/总访问数:533/4932145
地址:北京工业大学图书馆(北京市朝阳区平乐园100号 邮编:100124) 联系我们:010-67392185
版权所有:北京工业大学图书馆 站点建设与维护:北京爱琴海乐之技术有限公司