• Complex
  • Title
  • Keyword
  • Abstract
  • Scholars
  • Journal
  • ISSN
  • Conference
搜索

Author:

Fan, Hao (Fan, Hao.) | Ma, Zhaoyang (Ma, Zhaoyang.) | Li, Yong (Li, Yong.) | Tian, Rui (Tian, Rui.) | Chen, Yunli (Chen, Yunli.) | Gao, Chenlong (Gao, Chenlong.)

Indexed by:

CPCI-S EI Scopus

Abstract:

Pretrained Vision-Language Models (VLMs) like CLIP have exhibited remarkable capacities across downstream tasks, while their image encoders are vulnerable to adversarial examples. A recently introduced lightweight approach, termed Adversarial Prompt Tuning (AdvPT), utilizes adversarial examples for training learnable prompts, enhancing the adversarial robustness of VLMs solely through manipulation of textual inputs. However, the static prompts learned from AdvPT overfit base classes observed during training, compromising the model's generalizability. In this paper, we propose a conditional Adversarial Prompt Tuning method, which extends AdvPT by further learning a network to generate for each input a specific prompt. The dynamic prompts enhance the generalizability of VLMs on unseen classes. Furthermore, since VLMs are inherently powerful generalizers, we try to incorporate the manual prompts used by VLMs in the testing phase to further improve the generalizability of the model. Extensive experiments on 8 datasets demonstrate that our prompt fusion based method significantly outperforms AdvPT on unseen classes, enhancing the generalizability and adversarial robustness of VLMs simultaneously.

Keyword:

Generalizability Prompt tuning Adversarial robustness Vision-Language model

Author Community:

  • [ 1 ] [Fan, Hao]Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China
  • [ 2 ] [Gao, Chenlong]Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China
  • [ 3 ] [Fan, Hao]Beijing Univ Technol, Fac Informat Technol, Beijing, Peoples R China
  • [ 4 ] [Li, Yong]Beijing Univ Technol, Fac Informat Technol, Beijing, Peoples R China
  • [ 5 ] [Tian, Rui]Beijing Univ Technol, Fac Informat Technol, Beijing, Peoples R China
  • [ 6 ] [Chen, Yunli]Beijing Univ Technol, Fac Informat Technol, Beijing, Peoples R China
  • [ 7 ] [Ma, Zhaoyang]Beijing Jiaotong Univ, Sch Comp & Informat Technol, Beijing, Peoples R China

Reprint Author's Address:

  • [Gao, Chenlong]Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China;;

Show more details

Related Keywords:

Related Article:

Source :

ADVANCED INTELLIGENT COMPUTING TECHNOLOGY AND APPLICATIONS, PT IX, ICIC 2024

ISSN: 0302-9743

Year: 2024

Volume: 14870

Page: 328-339

Cited Count:

WoS CC Cited Count:

SCOPUS Cited Count:

ESI Highly Cited Papers on the List: 0 Unfold All

WanFang Cited Count:

Chinese Cited Count:

30 Days PV: 2

Affiliated Colleges:

Online/Total:656/5301888
Address:BJUT Library(100 Pingleyuan,Chaoyang District,Beijing 100124, China Post Code:100124) Contact Us:010-67392185
Copyright:BJUT Library Technical Support:Beijing Aegean Software Co., Ltd.