Enhanced Temporal Knowledge Embeddings with Contextualized Language Representations [article]

Zhen Han, Ruotong Liao, Beiyan Liu, Yao Zhang, Zifeng Ding, Jindong Gu, Heinz Köppl, Hinrich Schütze, Volker Tresp
<span title="2022-05-20">2022</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>
Within the emerging research efforts to combine structured and unstructured knowledge, many approaches incorporate factual knowledge, e.g., available in form of structured knowledge graphs (KGs), into pre-trained language models (PLMs) and then apply the knowledge-enhanced PLMs to downstream NLP tasks. However, (1) they typically only consider static factual knowledge, whereas, e.g., knowledge graphs (KGs) also contain temporal facts or events indicating evolutionary relationships among
more &raquo; ... at different timestamps. (2) PLMs cannot be directly applied to many KG tasks, such as temporal KG completion. In this paper, we focus on enhancing temporal knowledge embeddings with contextualized language representations (ECOLA). We align structured knowledge, contained in temporal knowledge graphs, with their textual descriptions extracted from news articles, and propose a novel knowledge-text prediction task to inject the abundant information from descriptions into temporal knowledge embeddings. ECOLA jointly optimizes the knowledge-text prediction objective and the temporal knowledge embeddings, which can simultaneously take full advantage of textual and knowledge information. The proposed fusion method is model-agnostic and can be combined with potentially any temporal KG model. For training ECOLA, we introduce three temporal KG datasets with aligned textual descriptions. Experimental results on the temporal knowledge graph completion task show that ECOLA outperforms state-of-the-art temporal KG models by a large margin. The proposed datasets can serve as new temporal KG benchmarks and facilitate future research on structured and unstructured knowledge integration.
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2203.09590v3">arXiv:2203.09590v3</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/vovdzstpzrcmlnray7ojmplhtq">fatcat:vovdzstpzrcmlnray7ojmplhtq</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20220525220910/https://arxiv.org/pdf/2203.09590v3.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/96/98/9698c4f6380bb0e9aba28fcdead9103a7fbe337b.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2203.09590v3" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>