Filters








9 Hits in 10.3 sec

KAT: A Knowledge Augmented Transformer for Vision-and-Language [article]

Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, Jianfeng Gao
2022 arXiv   pre-print
knowledge used, and how the reasoning processes over implicit and explicit knowledge should be integrated.  ...  Our approach integrates implicit and explicit knowledge in an end to end encoder-decoder architecture, while still jointly reasoning over both knowledge sources during answer generation.  ...  We thank Zhe Gan, Zhengyuan Yang, Lijuan Wang from cognition service team of Microsoft for their work and their generous helps and feedback for the project.  ... 
arXiv:2112.08614v2 fatcat:fvqo2qkodrhdjabphbzrfhljqy

Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answering [article]

Man Luo, Yankai Zeng, Pratyay Banerjee, Chitta Baral
2021 arXiv   pre-print
One dataset that is mostly used in evaluating knowledge-based VQA is OK-VQA, but it lacks a gold standard knowledge corpus for retrieval.  ...  To address this issue, we collect a natural language knowledge base that can be used for any VQA system. Moreover, we propose a Visual Retriever-Reader pipeline to approach knowledge-based VQA.  ...  Acknowledgements The authors acknowledge support from the NSF grant 1816039, DARPA grant W911NF2020006, DARPA grant FA875019C0003, and ONR award N00014-20-1-2332; and thank the reviewers for their feedback  ... 
arXiv:2109.04014v1 fatcat:rnm2ghrosbd4xkctt4jnozfndu

A Thousand Words Are Worth More Than a Picture: Natural Language-Centric Outside-Knowledge Visual Question Answering [article]

Feng Gao, Qing Ping, Govind Thattai, Aishwarya Reganti, Ying Nian Wu, Prem Natarajan
2022 arXiv   pre-print
In this paper, we call for a paradigm shift for the OK-VQA task, which transforms the image into plain text, so that we can enable knowledge passage retrieval, and generative question-answering in the  ...  This paradigm takes advantage of the sheer volume of gigantic knowledge bases and the richness of pre-trained language models.  ...  Open-Domain Question Answering in NLP Opendomain question answering (Open-Domain QA) has been popular in the NLP community in recent years.  ... 
arXiv:2201.05299v1 fatcat:cbr5icvimbh7xfj7m2vxmpjyma

A Review of Some Techniques for Inclusion of Domain-Knowledge into Deep Neural Networks [article]

Tirtharaj Dash, Sharad Chitlangia, Aditya Ahuja, Ashwin Srinivasan
2021 arXiv   pre-print
This paper examines the inclusion of domain-knowledge by means of changes to: the input, the loss-function, and the architecture of deep networks.  ...  In many such instances, machine-based model construction may benefit significantly from being provided with human-knowledge of the domain encoded in a sufficiently precise form.  ...  Krisp: Integrating implicit and symbolic [31] Ellis, K., Morales, L., Meyer, M. S., Solar-Lezama, knowledge for open-domain knowledge-based vqa. In A. & Tenenbaum, J. B.  ... 
arXiv:2107.10295v4 fatcat:ifkgq3cptbapfamr3dld57uruu

Low-resource Learning with Knowledge Graphs: A Comprehensive Survey [article]

Jiaoyan Chen and Yuxia Geng and Zhuo Chen and Jeff Z. Pan and Yuan He and Wen Zhang and Ian Horrocks and Huajun Chen
2021 arXiv   pre-print
Among all the low-resource learning studies, many prefer to utilize some auxiliary information in the form of Knowledge Graph (KG), which is becoming more and more popular for knowledge representation,  ...  dividing them into different paradigms such as the mapping-based, the data augmentation, the propagation-based and the optimization-based.  ...  KRISP: Integrating implicit and symbolic knowledge for open-domain knowledge-based VQA.  ... 
arXiv:2112.10006v3 fatcat:wkz6gjx4r5gvlhh673p3rqsmgi

Towards Knowledge-capable AI: Agents that See, Speak, Act and Know

Kenneth Marino
2022
Then we develop a method on that dataset which combines two types of knowledge: knowledge graphs and implicit knowledge from large language models.  ...  We introduce a benchmark for vision and language that requires models with the capability to bring in and reason about knowledge about the world.  ...  Abhinav has been a dogged advocate during my career, providing advice, support and guidance throught my PhD.  ... 
doi:10.1184/r1/19552225.v1 fatcat:rxhy64wyqfb4rjnizil6hrffge

Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answering

Man Luo, Yankai Zeng, Pratyay Banerjee, Chitta Baral
2021 Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing   unpublished
One dataset that is mostly used in evaluating knowledge-based VQA is OK-VQA, but it lacks a gold standard knowledge corpus for retrieval.  ...  To address this issue, we collect a natural language knowledge base that can be used for any VQA system. Moreover, we propose a Visual Retriever-Reader pipeline to approach knowledge-based VQA.  ...  Acknowledgements The authors acknowledge support from the NSF grant 1816039, DARPA grant W911NF2020006, DARPA grant FA875019C0003, and ONR award N00014-20-1-2332; and thank the reviewers for their feedback  ... 
doi:10.18653/v1/2021.emnlp-main.517 fatcat:eacsuzciajb2npa2fholawrkpu

Integrating common sense knowledge in visual question answering systems [article]

Adrika Mukherjee, Universität Stuttgart
2022
Aditionally, This thesis work comprehensively studies existing approaches for knowledge-based VQA architectures and analyze their shortcomings.  ...  Finally, a novel pipeline-based architecture to integrate common sense knowledge into VQA systems, using an automatic know [...]  ...  Thang Vu for his valuable input during the entire duration of my thesis.  ... 
doi:10.18419/opus-11965 fatcat:gkdhgd25lrdwzosc5dvw2kjexa

Low-resource Learning with Knowledge Graphs: A Comprehensive Survey [article]

Jiaoyan Chen, Yuxia Geng, Zhuo Chen, Jeff Z. Pan, Yuan He, Wen Zhang, Ian Horrocks, Huajun Chen
2021
Among all the low-resource learning studies, many prefer to utilize some auxiliary information in the form of Knowledge Graph (KG), which is becoming more and more popular for knowledge representation,  ...  dividing them into different paradigms such as the mapping-based, the data augmentation, the propagation-based and the optimization-based.  ...  ACKNOWLEDGMENTS This work was supported by the SIRIUS Centre for Scalable Data Access (Research Council of Norway, project 237889), eBay, Samsung Research UK, Siemens AG, and the EPSRC projects OASIS (  ... 
doi:10.48550/arxiv.2112.10006 fatcat:e2pdfnugbfad5nd4s525ge3tru