Filters








6,387 Hits in 3.8 sec

Graph Representation Learning via Multi-task Knowledge Distillation [article]

Jiaqi Ma, Qiaozhu Mei
2019 arXiv   pre-print
Specifically, we proposed a multi-task knowledge distillation method.  ...  By incorporating network-theory-based graph metrics as auxiliary tasks, we show on both synthetic and real datasets that the proposed multi-task learning method can improve the prediction performance of  ...  We then use these graph metrics as auxiliary tasks and distill the knowledge of network theory into the learned graph representations via multi-task learning.  ... 
arXiv:1911.05700v1 fatcat:afxxmfxj6remjl3ilirfm2gy6u

Better and Faster: Knowledge Transfer from Multiple Self-supervised Learning Tasks via Graph Distillation for Video Classification [article]

Chenrui Zhang, Yuxin Peng
2018 arXiv   pre-print
In this paper, we propose a graph-based distillation framework to address these problems: (1) We propose logits graph and representation graph to transfer knowledge from multiple self-supervised tasks,  ...  where the former distills classifier-level knowledge by solving a multi-distribution joint matching problem, and the latter distills internal feature knowledge from pairwise ensembled representations  ...  We first train the teacher models for privileged information learning on auxiliary tasks, then transfer the knowledge from teachers to student via the proposed graph distillation framework.  ... 
arXiv:1804.10069v1 fatcat:27xmwll4vzec3aatscu6zmwriu

Better and Faster: Knowledge Transfer from Multiple Self-supervised Learning Tasks via Graph Distillation for Video Classification

Chenrui Zhang, Yuxin Peng
2018 Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence  
In this paper, we propose a graph-based distillation framework to address these problems: (1) We propose logits graph and representation graph to transfer knowledge from multiple self-supervised tasks,  ...  where the former distills classifier-level knowledge by solving a multi-distribution joint matching problem, and the latter distills internal feature knowledge from pairwise ensembled representations  ...  We first train the teacher models for privileged information learning on auxiliary tasks, then transfer the knowledge from teachers to student via the proposed graph distillation framework.  ... 
doi:10.24963/ijcai.2018/158 dblp:conf/ijcai/ZhangP18a fatcat:56vujzi3ajhfdhmnxubck7relu

On Representation Knowledge Distillation for Graph Neural Networks [article]

Chaitanya K. Joshi, Fayao Liu, Xu Xun, Jie Lin, Chuan-Sheng Foo
2021 arXiv   pre-print
Knowledge distillation is a promising learning paradigm for boosting the performance and reliability of resource-efficient graph neural networks (GNNs) using more expressive yet cumbersome teacher models  ...  Distillation (G-CRD), which uses contrastive learning to align the student node embeddings to those of the teacher in a shared representation space.  ...  Graph Representation Learning Graph Neural Networks (GNNs) take as input an unordered set of nodes and the graph connectivity among them, and learn latent node representations for them via iterative feature  ... 
arXiv:2111.04964v1 fatcat:xowf5oylhnhtpk6lai23cd7aii

Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks [article]

Lin Wang, Kuk-Jin Yoon
2021 arXiv   pre-print
To achieve faster speeds and to handle the problems caused by the lack of data, knowledge distillation (KD) has been proposed to transfer information learned from one model to another.  ...  KD is often characterized by the so-called 'Student-Teacher' (S-T) learning framework and has been broadly applied in model compression and knowledge transfer.  ...  [17] considers the instance features and instance relationships as instance graphs, and [213] builds an input graph representation for multi-task knowledge distillation.  ... 
arXiv:2004.05937v6 fatcat:yqzo7nylzbbn7pfhzpfc2qaxea

Graphonomy: Universal Human Parsing via Graph Transfer Learning

Ke Gong, Yiming Gao, Xiaodan Liang, Xiaohui Shen, Meng Wang, Liang Lin
2019 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)  
By distilling universal semantic graph representation to each specific task, Graphonomy is able to predict all levels of parsing labels in one system without piling up the complexity.  ...  In particular, Graphonomy first learns and propagates compact high-level graph representation among the labels within one dataset via Intra-Graph Reasoning, and then transfers semantic information across  ...  Superior to multi-task learning, our Graphonomy is able to distill universal semantic graph representation and enhance individualized representation for each label graph. # Basic network [3] Adjacency  ... 
doi:10.1109/cvpr.2019.00763 dblp:conf/cvpr/Gong0LS0L19 fatcat:rv5xgqag4jcghmzffi3s67fp2u

Graphonomy: Universal Human Parsing via Graph Transfer Learning [article]

Ke Gong, Yiming Gao, Xiaodan Liang, Xiaohui Shen, Meng Wang, Liang Lin
2019 arXiv   pre-print
By distilling universal semantic graph representation to each specific task, Graphonomy is able to predict all levels of parsing labels in one system without piling up the complexity.  ...  In particular, Graphonomy first learns and propagates compact high-level graph representation among the labels within one dataset via Intra-Graph Reasoning, and then transfers semantic information across  ...  Superior to multi-task learning, our Graphonomy is able to distill universal semantic graph representation and enhance individualized representation for each label graph. # Basic network [3] Adjacency  ... 
arXiv:1904.04536v1 fatcat:di2yce3ytbhadml5lljt7yn66m

Iterative Graph Self-Distillation [article]

Hanlin Zhang, Shuai Lin, Weiyang Liu, Pan Zhou, Jian Tang, Xiaodan Liang, Eric P. Xing
2021 arXiv   pre-print
Inspired by the recent success of unsupervised contrastive learning, we aim to learn graph-level representation in an unsupervised manner.  ...  Specifically, we propose a novel unsupervised graph learning paradigm called Iterative Graph Self-Distillation (IGSD) which iteratively performs the teacher-student distillation with graph augmentations  ...  Table 3 : Effects of projectors on Graph classification accuracies (%). CONCLUSIONS In this paper, we propose IGSD, a novel graph-level representation learning framework via self-distillation.  ... 
arXiv:2010.12609v2 fatcat:h5csmfxatbg4jcliukikimsgnm

Knowledge Distillation: A Survey [article]

Jianping Gou, Baosheng Yu, Stephen John Maybank, Dacheng Tao
2021 arXiv   pre-print
As a representative type of model compression and acceleration, knowledge distillation effectively learns a small student model from a large teacher model.  ...  This paper provides a comprehensive survey of knowledge distillation from the perspectives of knowledge categories, training schemes, teacher-student architecture, distillation algorithms, performance  ...  Multi-head graph-based knowledge distillation was proposed by Lee and Song (2019) . The graph knowledge is the intra-data relations between any two feature maps via multi-head attention network.  ... 
arXiv:2006.05525v6 fatcat:aedzaeln5zf3jgjsgsn5kvjrri

Joint Multiple Intent Detection and Slot Filling via Self-distillation [article]

Lisong Chen, Peilin Zhou, Yuexian Zou
2021 arXiv   pre-print
In this paper, we propose a novel Self-Distillation Joint NLU model (SDJN) for multi-intent NLU.  ...  Then, we design an auxiliary loop via self-distillation with three orderly arranged decoders: Initial Slot Decoder, MIL Intent Decoder, and Final Slot Decoder.  ...  Second, inspired by self-distillation network [22] , we proposed a self-distillation method for joint NLU modeling by taking advantage of multi-task.  ... 
arXiv:2108.08042v1 fatcat:6twwj42w2neh3axwxiffu5sh2a

Multi-Scale Contrastive Siamese Networks for Self-Supervised Graph Representation Learning [article]

Ming Jin, Yizhen Zheng, Yuan-Fang Li, Chen Gong, Chuan Zhou, Shirui Pan
2021 arXiv   pre-print
to learn node representations by enhancing Siamese self-distillation with multi-scale contrastive learning.  ...  Graph representation learning plays a vital role in processing graph-structured data. However, prior arts on graph representation learning heavily rely on labeling information.  ...  By leveraging the backbone Siamese GNNs, we design a cross-network contrastiveness to distill the knowledge from historical representations to guide and stabilize the training of online graph encoder.  ... 
arXiv:2105.05682v2 fatcat:57t7mylspbbs3hb4wf3h5jsgoy

CCGL: Contrastive Cascade Graph Learning [article]

Xovee Xu, Fan Zhou, Kunpeng Zhang, Siyuan Liu
2021 arXiv   pre-print
In this work, we present Contrastive Cascade Graph Learning (CCGL), a novel framework for cascade graph representation learning in a contrastive, self-supervised, and task-agnostic way.  ...  Second, it learns a generic model for graph cascade tasks via self-supervised contrastive pre-training using both unlabeled and labeled data.  ...  This might be because the model learns the generic knowledge in pre-training and task-specific knowledge in fine-tuning and distillation.  ... 
arXiv:2107.12576v1 fatcat:kb3si37j65gntar64i63aq7hzi

Knowledge Integration Networks for Action Recognition [article]

Shiwen Zhang and Sheng Guo and Limin Wang and Weilin Huang and Matthew R. Scott
2020 arXiv   pre-print
We explore two pre-trained models as teacher networks to distill the knowledge of human and scene for training the auxiliary tasks of KINet.  ...  an Action Knowledge Graph (AKG) for effectively fusing high-level context information.  ...  three tasks via knowledge distillation, yet without applying CBI module or Action Knowledge Graph here.  ... 
arXiv:2002.07471v1 fatcat:mzg6xcv7nbewbb6ungcvinfgwi

Knowledge Integration Networks for Action Recognition

Shiwen Zhang, Sheng Guo, Limin Wang, Weilin Huang, Matthew Scott
2020 PROCEEDINGS OF THE THIRTIETH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE AND THE TWENTY-EIGHTH INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE CONFERENCE  
We explore two pre-trained models as teacher networks to distill the knowledge of human and scene for training the auxiliary tasks of KINet.  ...  an Action Knowledge Graph (AKG) for effectively fusing high-level context information.  ...  three tasks via knowledge distillation, yet without applying CBI module or Action Knowledge Graph here.  ... 
doi:10.1609/aaai.v34i07.6983 fatcat:mb2yttp3xrbppm2ty652ppnbb4

Graphonomy: Universal Image Parsing via Graph Reasoning and Transfer [article]

Liang Lin and Yiming Gao and Ke Gong and Meng Wang and Xiaodan Liang
2021 arXiv   pre-print
To address these challenges, we propose a graph reasoning and transfer learning framework, named "Graphonomy", which incorporates human knowledge and label taxonomy into the intermediate graph representation  ...  The former extracts the semantic graph in each domain to improve the feature representation learning by propagating information with the graph; the latter exploits the dependencies among the graphs from  ...  domains via graph transfer learning to achieve multiple levels of human parsing tasks.  ... 
arXiv:2101.10620v1 fatcat:hnbuqiugsfhvbc7phn5htmsvcy
« Previous Showing results 1 — 15 out of 6,387 results