Multimodality

Papers, Datasets, Codes about Multimodality

Paper

Vision-Language Pre-Training

1. VLMO: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts Wenhui Wang, Hangbo Bao, Li Dong, Furu Wei [pdf]
2. Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts Yan Zeng, Xinsong Zhang, Hang Li [pdf]
3. Masked Autoencoders Are Scalable Vision Learners Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick [pdf]

Vision-and-Language Navigation

Dialogue-System

1. Multi-Modal Open-Domain Dialogue Kurt Shuster, Eric Michael Smith, Da Ju, Jason Weston [pdf]
2. Multimodal Dialogue Response Generation Qingfeng Sun, Yujing Wang, Can Xu, Kai Zheng, Yaming Yang, Huang Hu, Fei Xu, Jessica Zhang, Xiubo Geng, Daxin Jiang [pdf]
3. Reason first, then respond:Modular Generation for Knowledge-infused Dialogue Leonard Adolphs, Kurt Shuster, Jack Urbanek, Arthur Szlam, Jason Weston [pdf]

Prompt

1. CPT: COLORFUL PROMPT TUNING FOR PRE-TRAINED VISION-LANGUAGE MODELS Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, Maosong Sun [pdf]
2. Multimodal Few-Shot Learning with Frozen Language Models Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, Felix Hill [pdf]

TOP Conference Paper About Multi-Modal Dialog

ACL

ID	Paper	Author	Conference
1	Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering	Siddharth Karamcheti, Ranjay Krishna, Li Fei-Fei and Christopher Manning	ACL2021
2	TicketTalk: Toward human-level performance with end-to-end, transaction-based dialog systems	Bill Byrne, Karthik Krishnamoorthi, Saravanan Ganesh and Mihir Kale	ACL2021
3	PhotoChat: A Human-Human Dialogue Dataset With Photo Sharing Behavior For Joint Image-Text Modeling	Xiaoxue Zang, Lijuan Liu, Maria Wang, Yang Song, Hao Zhang and Jindong Chen	ACL2021
4	Maria: A Visual Experience Powered Conversational Agent	Zujie Liang, Huang Hu, Can Xu, Chongyang Tao, Xiubo Geng, Yining Chen, Fan Liang and Daxin Jiang	ACL2021
5	MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation	Jingwen Hu, Yuchen Liu, Jinming Zhao and Qin Jin	ACL2021
6	How do people talk about images?A study on open-domain conversation on images	Anonymous ACL submission	-
7	Zero-Shot Visual Grounding of Referring Utterances in Dialogue	Anonymous ACL submission	-
8	When did you become so smart, oh wise one?! Sarcasm Explanation in Multi-modal Multi-party Dialogues	Anonymous ACL submission	-
9	Tackling Situated Multi-Modal Task-Oriented Dialogs with a Single Transformer Model	Anonymous ACL submission	-
10	Co-VQA : Answering by Interactive Sub Question Sequence	Anonymous ACL submission	-

Datasets

ID	NAME	Description	Paper	Conference
1	LAION-40	Multi-Model
2	IEMOCAP	Multi-Model emotion	IEMOCAP: interactive emotional dyadic motion capture database. Lang Resources & Evaluation
3	MELD	Multi-Model emotion	MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversation
4	CH-SIMS	Multi-Model emotion	CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotations of Modality
5	SEMAINE	Multi-Model emotion	The SEMAINE Database: Annotated Multimodal Records of Emotionally Colored Conversations between a Person and a Limited Agent
6	COCO	Multi-Model Retrieval	Microsoft COCO Captions Data Collection and Evaluation Server
7	IAPR TC-12	Multi-Model Retrieval	The IAPR Benchmark: A New Evaluation Resource for Visual Information Systems	LREC
8	Conceptual Captions Dataset	Multi-Model Retrieval	Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning
9	OpenViDial	Multi-Model Dialogue	OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts

Codes