Our work in multimodal learning includes stepwise story illustration using images, radiology report generation, news image caption generation, multimodal fake news detection, and multimodal event representation learning.

Jun Wang, Runcong Zhao, Wenjia Zhang, Vishwash Batra, Lin Gui, Yulan He

• R. Zhao, W. Zhang, J. Li, L. Zhu, Y. Li, Y. He and L. Gui. NarrativePlay: Interactive Narrative Understanding , The 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2024.
• J. Wang, A. Bhalerao, T. Yin, S. See, Y. He. CAMANet: Class Activation Map Guided Attention Network for Radiology Report Generation, arXiv:2211.01412.
• R. Zhao, W. Zhang, J. Li, L. Zhu, Y. Li, Y. He and L. Gui. NarrativePlay: An Automated System for Crafting Visual Worlds in Novels for Role-Playing , The 38th Annual AAAI Conference on Artificial Intelligence (AAAI), 2024.
• J. Wang, A. Bhalerao, L. Zhu, and Y. He. Can Prompt Learning Benefit Radiology Report Generation? , arXiv:2308.16269.
• J. Wang, A. Bhalerao and Y. He. Cross-modal Prototype Driven Network for Radiology Report Generation, 17th European Conference on Computer Vision (ECCV), Oct. 2022.
• W. Zhang, L. Gui and Y. He. Supervised Contrastive Learning for Multi-modal Unreliable News Detection in COVID-19 Pandemic, The 30th ACM International Conference on Information and Knowledge Management (CIKM), Nov. 2021.
• D. Zhou, K. Sun, M. Hu and Y. He. Image Generation from Text with Entity Information Fusion, Knowledge-Based Systems, Vol. 227, 107200, 2021.
• L. Zhang, D. Zhou, Y. He and Z. Yang. MERL: Multimodal Event Representation Learning in Heterogeneous Embedding Spaces, The 35th AAAI Conference on Artificial Intelligence (AAAI), Feb. 2021.