• 综合
  • 标题
  • 关键词
  • 摘要
  • 学者
  • 期刊-刊名
  • 期刊-ISSN
  • 会议名称
搜索

作者:

Deng, Sinuo (Deng, Sinuo.) | Wu, Lifang (Wu, Lifang.) | Shi, Ge (Shi, Ge.) | Xing, Lehao (Xing, Lehao.) | Jian, Meng (Jian, Meng.) | Xiang, Ye (Xiang, Ye.) | Dong, Ruihai (Dong, Ruihai.)

收录:

EI Scopus SCIE

摘要:

Image emotion classification (IEC) aims to extract the abstract emotions evoked in images. Recently, language-supervised methods such as contrastive language-image pretraining (CLIP) have demonstrated superior performance in image understanding. However, the underexplored task of IEC presents three major challenges: a tremendous training objective gap between pretraining and IEC, shared suboptimal prompts, and invariant prompts for all instances. In this study, we propose a general framework that effectively exploits the language-supervised CLIP method for the IEC task. First, a prompt-tuning method that mimics the pretraining objective of CLIP is introduced, to exploit the rich image and text semantics associated with CLIP. Subsequently, instance-specific prompts are automatically composed, conditioning them on the categories and image content of instances, diversifying the prompts, and thus avoiding suboptimal problems. Evaluations on six widely used affective datasets show that the proposed method significantly outperforms state-of-the-art methods (up to 9.29% accuracy gain on the EmotionROI dataset) on IEC tasks with only a few trained parameters. The code is publicly available at https://github.com/dsn0w/PT-DPC/for research purposes.

关键词:

multimodal learning pretraining model prompt tuning image emotion analysis

作者机构:

  • [ 1 ] [Deng, Sinuo]Beijing Univ Technol, Fac Informat Technol, Beijing 100124, Peoples R China
  • [ 2 ] [Wu, Lifang]Beijing Univ Technol, Fac Informat Technol, Beijing 100124, Peoples R China
  • [ 3 ] [Shi, Ge]Beijing Univ Technol, Fac Informat Technol, Beijing 100124, Peoples R China
  • [ 4 ] [Xing, Lehao]Beijing Univ Technol, Fac Informat Technol, Beijing 100124, Peoples R China
  • [ 5 ] [Jian, Meng]Beijing Univ Technol, Fac Informat Technol, Beijing 100124, Peoples R China
  • [ 6 ] [Xiang, Ye]Beijing Univ Technol, Fac Informat Technol, Beijing 100124, Peoples R China
  • [ 7 ] [Dong, Ruihai]Univ Coll Dublin, Insight Ctr Data Analyt, Dublin D04 V1W8, Ireland

通讯作者信息:

  • [Shi, Ge]Beijing Univ Technol, Fac Informat Technol, Beijing 100124, Peoples R China;;

电子邮件地址:

查看成果更多字段

相关关键词:

相关文章:

来源 :

COMPUTATIONAL VISUAL MEDIA

ISSN: 2096-0433

年份: 2024

期: 6

卷: 10

页码: 1169-1183

被引次数:

WoS核心集被引频次: 11

SCOPUS被引频次: 13

ESI高被引论文在榜: 0 展开所有

万方被引频次:

中文被引频次:

近30日浏览量: 1

归属院系:

在线人数/总访问数:464/4912462
地址:北京工业大学图书馆(北京市朝阳区平乐园100号 邮编:100124) 联系我们:010-67392185
版权所有:北京工业大学图书馆 站点建设与维护:北京爱琴海乐之技术有限公司