Filters








4,914 Hits in 4.5 sec

End-to-End Entity Classification on Multimodal Knowledge Graphs [article]

W.X. Wilcke , F.A.H. van Harmelen Department of Computer Science Vrije Universiteit Amsterdam The Netherlands
2020 arXiv   pre-print
End-to-end multimodal learning on knowledge graphs has been left largely unaddressed.  ...  We propose a multimodal message passing network which not only learns end-to-end from the structure of graphs, but also from their possibly divers set of multimodal node features.  ...  Acknowledgments We express our gratitude to Wouter Beek from Triply 9 for helping us obtain two of the three knowledge graphs that make up the Dutch Monument Graph.  ... 
arXiv:2003.12383v1 fatcat:z5qqcjqaurftzakgf3lagie25y

A Garbage Detection and Classification Method Based on Visual Scene Understanding in the Home Environment

Yuezhong Wu, Xuehao Shen, Qiang Liu, Falong Xiao, Changyun Li, Xuyun Zhang
2021 Complexity  
This method uses knowledge graphs to store and model items in the scene in the form of images, videos, texts, and other multimodal forms.  ...  The ESA attention mechanism is added to the backbone network part of the YOLOv5 network, aiming to improve the feature extraction ability of the network, combining with the built multimodal knowledge graph  ...  Multimodal Knowledge Graph.  ... 
doi:10.1155/2021/1055604 fatcat:i3uvswukarf7bkfpflt7yoaesa

Fake News Detection via Knowledge-driven Multimodal Graph Convolutional Networks

Youze Wang, Shengsheng Qian, Jun Hu, Quan Fang, Changsheng Xu
2020 Proceedings of the 2020 International Conference on Multimedia Retrieval  
In conclusion, the contributions of our work are as follows: • We propose an end-to-end Knowledge-driven Multimodal Graph Convolutional Network to model the semantic-level representations for fake news  ...  correct entities T in the knowledge graph they refer to.  ... 
doi:10.1145/3372278.3390713 dblp:conf/mir/WangQHFX20 fatcat:bdtdwo3pwbhm5bbipy7w3x6idi

Combining and Curating Automated Metadata from Multiple Sources

Lauri Saarikoski
2020 Zenodo  
This presentation was featured in the DataTV 2020 Webinar on September 14, 2020.  ...  • Legacy metadata provided by INA and Yle, (+1600 hours of media) was processed and consolidated according to the EBUCore/MeMAD ontology and put into the MeMAD Knowledge GraphKnowledge Graph is  ...  Named entity recognition and disambiguation 9. Semantic enrichment 10.  ... 
doi:10.5281/zenodo.4058122 fatcat:hczzvlatonddtlwg7u6rairlnq

Fine-Grained Chemical Entity Typing with Multimodal Knowledge Representation [article]

Chenkai Sun, Weijiang Li, Jinfeng Xiao, Nikolaus Nova Parulian, ChengXiang Zhai, Heng Ji
2021 arXiv   pre-print
How to extract detailed knowledge about chemical reactions from the core chemistry literature is a new emerging challenge that has not been well studied.  ...  We introduce a new benchmark data set (CHEMET) to facilitate the study of the new task and propose a novel multi-modal representation learning framework to solve the problem of fine-grained chemical entity  ...  . • We propose and evaluate a novel method that utilizes multimodal knowledge to enrich entity mention repre- sentation.  ... 
arXiv:2108.12899v1 fatcat:cib35n544vdo5pj34ztrm7jowy

Literature mining for context-specific molecular relations using multimodal representations (COMMODAR)

Jaehyun Lee, Doheon Lee, Kwang Hyung Lee
2020 BMC Bioinformatics  
The models based on multiple modalities outperformed those solely based on the linguistic modality.  ...  We applied COMMODAR to the 14 million PubMed abstracts and extracted 9214 context-specific molecular relations.  ...  About this supplement This article has been published as part of BMC Bioinformatics, Volume 21 Supplement 5, 2020: Proceedings of the 13th International Workshop on Data and Text Mining in Biomedical Informatics  ... 
doi:10.1186/s12859-020-3396-y pmid:33106154 fatcat:s3oha7endfcqhnxifzdckxxi5a

Understanding the Gist of Images - Ranking of Concepts for Multimedia Indexing [article]

Lydia Weiland, Simone Paolo Ponzetto, Wolfgang Effelsberg, Laura Dietz
2018 arXiv   pre-print
Thus, the presented end-to-end setting outperforms DBM and competes with Hashing-based methods.  ...  Our pipeline benefits from external knowledge and two subsequent learning- to-rank (l2r) settings. The first l2r produces a ranking of concepts rep- resenting the respective multimedia instance.  ...  In an end-to-end approach we have demonstrated that multimodal modelling -even if one modality comes from noisy object detectors -performs better than a single modality approach.  ... 
arXiv:1809.08593v1 fatcat:4kqujgxphrgrjc2ukzidnk3pse

Knowledge Extraction And Representation Learning For Music Recommendation And Classification

Sergio Oramas, Xavier Serra
2017 Zenodo  
To this end, we first focus on the problem of linking music-related texts with online knowledge repositories and on the automated construction of music knowledge bases.  ...  We focus on the semantic enrichment of descriptions associated to musical items (e.g., artists biographies, album reviews, metadata), and the exploitation of multimodal data (e.g., text, audio, images)  ...  To this end, we present MuMu, a new large-scale multimodal dataset for multilabel music genre classification.  ... 
doi:10.5281/zenodo.1048497 fatcat:kdh5jhvocbh3riwln6n2f756su

Knowledge Extraction And Representation Learning For Music Recommendation And Classification

Sergio Oramas, Xavier Serra
2017 Zenodo  
To this end, we first focus on the problem of linking music-related texts with online knowledge repositories and on the automated construction of music knowledge bases.  ...  We focus on the semantic enrichment of descriptions associated to musical items (e.g., artists biographies, album reviews, metadata), and the exploitation of multimodal data (e.g., text, audio, images)  ...  To this end, we present MuMu, a new large-scale multimodal dataset for multilabel music genre classification.  ... 
doi:10.5281/zenodo.1100973 fatcat:yfpmc6qxbbakjp6qzvywyoaoci

A Multimodal Translation-Based Approach for Knowledge Graph Representation Learning

Hatem Mousselly Sergieh, Teresa Botschen, Iryna Gurevych, Stefan Roth
2018 Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics  
to the KG entities.  ...  We compared the performance of our approach to other baselines on two standard tasks, namely knowledge graph completion and triple classification, using our as well as the WN9-IMG dataset. 1 The results  ...  (5) to combine the linguistic and visual embeddings into a single one. To that end, we identified three methods for multimodal representation learning and adapted them to KG entities.  ... 
doi:10.18653/v1/s18-2027 dblp:conf/starsem/SergiehBGR18 fatcat:dftmpkrljbecjfoqrksnklwyni

Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graph [article]

Wentian Zhao, Yao Hu, Heda Wang, Xinxiao Wu, Jiebo Luo
2021 arXiv   pre-print
To tackle these challenges, we propose a novel approach that constructs a multi-modal knowledge graph to associate the visual objects with named entities and capture the relationship between entities simultaneously  ...  Entity-aware image captioning aims to describe named entities and events related to the image by utilizing the background knowledge in the associated article.  ...  To obtain rich representations of the entities in the multimodal knowledge graph, we encode the nodes in the multimodal knowledge graph using a two-layer graph attention network (GAT) [25] .  ... 
arXiv:2107.11970v1 fatcat:drxuorjiobfuzeqdv4cvq3upzm

Multimodal Classification of Violent Online Political Extremism Content with Graph Convolutional Networks

Stevan Rudinac, Iva Gornishka, Marcel Worring
2017 Proceedings of the on Thematic Workshops of ACM Multimedia 2017 - Thematic Workshops '17  
were recently proven effective in classification tasks on knowledge graphs.  ...  In this paper we present a multimodal approach to categorizing user posts based on their discussion topic.  ...  The authors would like to thank Stefan Verbruggen and Gijs Koot from TNO for their help with collecting the data as well as Prof.dr.  ... 
doi:10.1145/3126686.3126776 dblp:conf/mm/RudinacGW17 fatcat:kxpmaws5yzeovijqlg57rsd3xe

LingYi: Medical Conversational Question Answering System based on Multi-modal Knowledge Graphs [article]

Fei Xia, Bin Li, Yixuan Weng, Shizhu He, Kang Liu, Bin Sun, Shutao Li, Jun Zhao
2022 arXiv   pre-print
This paper presents a medical conversational question answering (CQA) system based on the multi-modal knowledge graph, namely "LingYi", which is designed as a pipeline framework to maintain high flexibility  ...  To conduct knowledge-grounded dialogues with patients, we first construct a Chinese Medical Multi-Modal Knowledge Graph (CM3KG) and collect a large-scale Chinese Medical CQA (CMCQA) dataset.  ...  First, the CRM maps the medical entities obtained in the disambiguation module to specific attributes on the knowledge graph, and stores the past entites into the dictionary.  ... 
arXiv:2204.09220v1 fatcat:t7ai6qosmzac7nbehowfmy4fg4

Current Approaches and Applications in Natural Language Processing

Arturo Montejo-Ráez, Salud María Jiménez-Zafra
2022 Applied Sciences  
Artificial Intelligence has gained a lot of popularity in recent years thanks to the advent of, mainly, Deep Learning techniques [...]  ...  [10] proposes an approach to entity linking (associate mentions in documents to existing entities in a knowledge graph) that profits from structural information of the graph, so correlation information  ...  Linguistic-based methods have been surpassed by end-to-end architectures, where no prior knowledge on language is needed, although only when a massive amount of data is available.  ... 
doi:10.3390/app12104859 fatcat:yhoyyoqcazflrbx7veksnkrrdq

AR-BERT: Aspect-relation enhanced Aspect-level Sentiment Classification with Multi-modal Explanations [article]

Sk Mainul Islam, Sourangshu Bhattacharya
2022 arXiv   pre-print
Existing models do not incorporate information on aspect-aspect relations in knowledge graphs (KGs), e.g. DBpedia.  ...  Two main challenges stem from inaccurate disambiguation of aspects to KG entities, and the inability to learn aspect representations from the large KGs in joint training with ALSC models.  ...  Hence, we seek to design a postthoc global explanation model for predicting multimodal explanations from both context words, and KG-entities.  ... 
arXiv:2108.11656v2 fatcat:xpymls6pdjbunctvvcvl66jgp4
« Previous Showing results 1 — 15 out of 4,914 results