Matteo Stefanini

Matteo Stefanini

Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara
Proceedings of the International Conference on Pattern Recognition (ICPR 2020)
July 2020


The joint understanding of vision and language has been recently gaining a lot of attention in both the Computer Vision and Natural Language Processing communities, with the emergence of tasks such as image captioning, image-text matching, and visual question answering. As both images and text can be encoded as sets or sequences of elements -like regions and words- proper reduction functions are needed to transform a set of encoded elements into a single response, like a classification or similarity score. In this paper, we propose a novel fully-attentive reduction method for vision and language. Specifically, our approach computes a set of scores for each element of each modality employing a novel variant of cross-attention, and performs a learnable and cross-modal reduction, which can be used for both classification and ranking. We test our approach on image-text matching and visual question answering, building fair comparisons with other reduction choices, on both COCO and VQA 2.0 datasets. Experimentally, we demonstrate that our approach leads to a performance increase on both tasks. Further, we conduct ablation studies to validate the role of each component of the approach.

Type: Conference Paper

Publication: International Conference on Pattern Recognition – ICPR 2020


Full Paper: link pdf


Please cite with the following BibTeX:

  title={A Novel Attention-based Aggregation Function to Combine Vision and Language},
  author={Stefanini, Matteo and Cornia, Marcella and Baraldi, Lorenzo and Cucchiara, Rita},
  journal={arXiv preprint arXiv:2004.13073},