时间：2023年12月20日15:30

地点：经管楼1205室

主讲人：陈刚 特聘副研究员，浙江大学

题目：When Multimodal Interactions Impair Prediction: A Novel Deep Learning Method

摘要：Multimodal data are proliferating and hence flourishing big data-based decision support, exemplified by short video attractiveness prediction (SVAP), multimodal review sentiment classification (MRSC), and multimodal data-based default risk prediction (DRP). However, when data of various modalities (e.g., text, graph, image, and video) are used jointly, they may mutually interact, adversely affecting prediction performance. To unravel the opaque conflicts in multimodal data, we formally conceptualize multimodal interactions and analytically disentangle positive and negative interactions at the feature, modality, and modality-wise instance levels. To better realize the predictive power of multimodal data, we propose a novel deep learning strategy named NIRMD (for negative interaction-regularized multimodal deep learning), which allows positive (negative) multimodal interactions to be effectively encouraged (mitigated) in a learnable nonlinear representation space. Empirical evaluation in three case studies involving SVAP, MRSC, and DRP, respectively, shows that the prediction performance of state-of-the-art multimodal deep learning methods can be enhanced by incorporating NIRMD. Exploratory (ablation, sensitivity, regularization effect, and economic) analyses reveal multimodal interaction patterns and show how NIRMD can tackle them for better prediction.

个人简介：陈刚，浙江大学管理学院特聘副研究员。2022年6月毕业于复旦大学管理学院，获得管理学博士学位；2016年6月本科毕业于合肥工业大学管理学院。研究兴趣包括金融科技、多模态商业数据分析、可解释人工智能等，聚焦于多模态场景下深度学习的可解释问题，致力于为商业决策搭建“全景式+智能化+可解释”的解决方案。研究成果发表于MIS Quarterly (UTD24), Information Systems Research (UTD24), Decision Support Systems, 《管理科学》, 《管理工程学报》等期刊，获得发明专利授权4项。担任Information Systems Research, Decision Support Systems, Information & Management等期刊，ICIS, PACIS, CSWIM 等国际会议匿名审稿人。