Framed Multi30K: um dataset multimodal-multilíngue baseado em semântica de frames
Loading...
Date
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Universidade Federal de Juiz de Fora (UFJF)
Abstract
The combination of dierent communication modalities is one of the defining
characteristics of human expression. However, much research has focused on analyzing the
semantics of text and images separately. In recent decades, systems that process image
and text data in a correlated way have been applied to computational tasks such as data
retrieval, machine translation, and caption generation. In this work, we start from the
premise that the development of such computational applications can benefit from a better
understanding of the meanings that are established from the combination of textual and
visual information. In particular, this dissertation contributes an innovative dataset that
adds to a family of gold standard datesets for multimodal and multilingual NLP - Flickr30k,
Multi30k and Flickr30k Entities - semantic information structured in terms of frames, as
modeled in FrameNet Brasil. The resulting dataset, called Framed Multi30k (FM30K),
contributes the following innovative data: (i) 150,000 conceptual descriptions originally
written in Brazilian Portuguese for each of the 30,000 images in the Flickr30k dataset; (ii)
30,000 translations into Brazilian Portuguese of one of the descriptions originally written
in English for each of the images in the Flickr30K; (iii) automatic annotations for frames
of all the conceptual descriptions in the dataset into Brazilian Portuguese and English,
totaling 330,000 semantically annotated descriptions; (iv) manual annotations for each
of the bounding boxes from the dataset Flickr30k Entities in three dierent annotation
conditions: annotation of entities with presence of description, annotation of entities
without presence of description and annotation of events with presence of description. The
resulting dataset was analized for formal aspects of the descriptions created in Brazilian
Portuguese and for the cosine similarity between the semantic representations derived
from the automatic and manual annotations carried out for the descriptions and images,
respectively. Qualitative analyses were also carried out on the dierent perspectives
encoded in the semantic representations generated for the images in each of the annotation
conditions. The analyses corroborate the hypothesis that dierent annotation conditions
regarding the interaction between modalities lead to dierent semantic representations for
the images, which reinforces the argument in favor of adopting a perspectivist approach
to human curation of datasets.
Description
Citation
Collections
Endorsement
Review
Supplemented By
Referenced By
Creative Commons license
Except where otherwise noted, this item's license is described as Acesso Aberto

