A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit <a rel="external noopener" href="https://arxiv.org/pdf/1908.10700v1.pdf">the original URL</a>. The file type is <code>application/pdf</code>.
Explainable Video Action Reasoning via Prior Knowledge and State Transitions
[article]
<span title="2019-08-28">2019</span>
<i >
arXiv
</i>
<span class="release-stage" >pre-print</span>
Preserved Fulltext
Human action analysis and understanding in videos is an important and challenging task. Although substantial progress has been made in past years, the explainability of existing methods is still limited. In this work, we propose a novel action reasoning framework that uses prior knowledge to explain semantic-level observations of video state changes. Our method takes advantage of both classical reasoning and modern deep learning approaches. Specifically, prior knowledge is defined as the
<span class="external-identifiers">
<a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1908.10700v1">arXiv:1908.10700v1</a>
<a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/itbcavk37fgkfmnhn5syzze6iy">fatcat:itbcavk37fgkfmnhn5syzze6iy</a>
</span>
more »
... tion of a target video domain, including a set of objects, attributes and relationships in the target video domain, as well as relevant actions defined by the temporal attribute and relationship changes (i.e. state transitions). Given a video sequence, we first generate a scene graph on each frame to represent concerned objects, attributes and relationships. Then those scene graphs are linked by tracking objects across frames to form a spatio-temporal graph (also called video graph), which represents semantic-level video states. Finally, by sequentially examining each state transition in the video graph, our method can detect and explain how those actions are executed with prior knowledge, just like the logical manner of thinking by humans. Compared to previous works, the action reasoning results of our method can be explained by both logical rules and semantic-level observations of video content changes. Besides, the proposed method can be used to detect multiple concurrent actions with detailed information, such as who (particular objects), when (time), where (object locations) and how (what kind of changes). Experiments on a re-annotated dataset CAD-120 show the effectiveness of our method.
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20200823152004/https://arxiv.org/pdf/1908.10700v1.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext">
<button class="ui simple right pointing dropdown compact black labeled icon button serp-button">
<i class="icon ia-icon"></i>
Web Archive
[PDF]
<div class="menu fulltext-thumbnail">
<img src="https://blobs.fatcat.wiki/thumbnail/pdf/25/30/253043ffa3c298817b1e8bc032e80fe0f32dba8a.180px.jpg" alt="fulltext thumbnail" loading="lazy">
</div>
</button>
</a>
<a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1908.10700v1" title="arxiv.org access">
<button class="ui compact blue labeled icon button serp-button">
<i class="file alternate outline icon"></i>
arxiv.org
</button>
</a>