Filters








1,972 Hits in 3.1 sec

Compressing GANs using Knowledge Distillation [article]

Angeline Aguinaldo, Ping-Yeh Chiang, Alex Gain, Ameya Patil, Kolten Pearson, Soheil Feizi
2019 arXiv   pre-print
Therefore, we propose a method to compress GANs using knowledge distillation techniques, in which a smaller "student" GAN learns to mimic a larger "teacher" GAN.  ...  We show that the distillation methods used on MNIST, CIFAR-10, and Celeb-A datasets can compress teacher GANs at ratios of 1669:1, 58:1, and 87:1, respectively, while retaining the quality of the generated  ...  Similarly, we see that the student GANs consistently outper-Compressing GANs using Knowledge Distillation Figure 6 .  ... 
arXiv:1902.00159v1 fatcat:35ul76wtbjguzmx74vx3rpupc4

PPCD-GAN: Progressive Pruning and Class-Aware Distillation for Large-Scale Conditional GANs Compression [article]

Duc Minh Vo, Akihiro Sugimoto, Hideki Nakayama
2022 arXiv   pre-print
We push forward neural network compression research by exploiting a novel challenging task of large-scale conditional generative adversarial networks (GANs) compression.  ...  To this end, we propose a gradually shrinking GAN (PPCD-GAN) by introducing progressive pruning residual block (PP-Res) and class-aware distillation.  ...  Distill+NAS [6] indicates the model (SA-GAN [2] ) is first trained with knowledge distillation (i.e., learnable remapping [6] ) and then compressed using NAS [17] .  ... 
arXiv:2203.08456v1 fatcat:oe2dyjbzdjd57jsoersxalloqa

MFAGAN: A Compression Framework for Memory-Efficient On-Device Super-Resolution GAN [article]

Wenlong Cheng and Mingbo Zhao and Zhiling Ye and Shuhang Gu
2021 arXiv   pre-print
Third, to balance the student discriminator and the compressed generator, we distill both the generator and the discriminator.  ...  In this paper, we propose a novel compression framework Multi-scale Feature Aggregation Net based GAN (MFAGAN) for reducing the memory access cost of the generator.  ...  Besides the generator, the discriminator stores useful knowledge of a learned GAN-based SR. It is useful to distill the teacher discriminator to stabilize the compressed generator training.  ... 
arXiv:2107.12679v1 fatcat:66jm5m2q5vaprhirf7weg74u5i

GAN Compression: Efficient Architectures for Interactive Conditional GANs [article]

Muyang Li, Ji Lin, Yaoyao Ding, Zhijian Liu, Jun-Yan Zhu, Song Han
2020 arXiv   pre-print
First, to stabilize the GAN training, we transfer knowledge of multiple intermediate representations of the original model to its compressed model, and unify unpaired and paired learning.  ...  Directly applying existing CNNs compression methods yields poor performance due to the difficulty of GAN training and the differences in generator architectures.  ...  A widely-used method for CNN model compression is knowledge distillation [25, 48, 10, 72, 36, 53, 12] .  ... 
arXiv:2003.08936v3 fatcat:ng36z3k2hzbfbob5gjrvh62kiq

A survey on GAN acceleration using memory compression techniques

Dina Tantawy, Mohamed Zahran, Amr Wassal
2021 Journal of Engineering and Applied Science (Cairo) (Online)  
Lossy compression techniques are further classified into (a) pruning, (b) knowledge distillation, (c) low-rank factorization, (d) lowering numeric precision, and (e) encoding.  ...  Our findings showed the superiority of knowledge distillation over pruning alone and the gaps in the research field that needs to be explored like encoding and different combination of compression techniques  ...  Although this work "uses" GAN to perform distillation, it does not consider GAN themselves for compression.  ... 
doi:10.1186/s44147-021-00045-5 fatcat:hy3oxa4fvzavhophwiekph4rum

GAN Slimming: All-in-One GAN Compression by A Unified Optimization Framework [article]

Haotao Wang, Shupeng Gui, Haichuan Yang, Ji Liu, Zhangyang Wang
2020 arXiv   pre-print
GS seamlessly integrates three mainstream compression techniques: model distillation, channel pruning and quantization, together with the GAN minimax objective, into one unified optimization form, that  ...  To this end, we propose the first unified optimization framework combining multiple compression means for GAN compression, dubbed GAN Slimming (GS).  ...  Knowledge distillation was first developed in [22] to transfer the knowledge in an ensemble of models to a single model, using a soft target distribution produced by the former models.  ... 
arXiv:2008.11062v1 fatcat:567bndbj5re2himu2kwwyiz7y4

P-KDGAN: Progressive Knowledge Distillation with GANs for One-class Novelty Detection [article]

Zhiwei Zhang, Shifeng Chen, Lei Sun
2021 arXiv   pre-print
Therefore, Progressive Knowledge Distillation with GANs (PKDGAN) is proposed to learn compact and fast novelty detection networks.  ...  The P-KDGAN is a novel attempt to connect two standard GANs by the designed distillation loss for transferring knowledge from the teacher to the student.  ...  To compress the model, the progressive knowledge distillation with GANs is proposed, which is a novel exploration that applies the knowledge distillation on two standard GANs.  ... 
arXiv:2007.06963v2 fatcat:b5xddnfusbat5ci3k2jyuolho4

A Survey on GAN Acceleration Using Memory Compression Technique [article]

Dina Tantawy, Mohamed Zahran, Amr Wassal
2021 arXiv   pre-print
Because data transfer is the main source of energy usage, memory compression leads to the most savings. Thus, in this paper, we survey memory compression techniques for CNN-Based GANs.  ...  Hence, accelerating GANs is pivotal. Accelerating GANs can be classified into three main tracks: (1) Memory compression, (2) Computation optimization, and (3) Data-flow optimization.  ...  Although this work "uses" GAN to perform distillation, it does not consider GAN themselves for compression.  ... 
arXiv:2108.06626v1 fatcat:b4imro6ap5fkvoceewn3qbazgy

Online Multi-Granularity Distillation for GAN Compression [article]

Yuxi Ren, Jie Wu, Xuefeng Xiao, Jianchao Yang
2021 arXiv   pre-print
We offer the first attempt to popularize single-stage online distillation for GAN-oriented compression, where the progressively promoted teacher generator helps to refine the discriminator-free based student  ...  Although recent efforts on compressing GANs have acquired remarkable results, they still exist potential model redundancies and can be further compressed.  ...  Knowledge Distillation Knowledge Distillation (KD) [19] is a fundamental compression technique, where a smaller student model is optimized under the effective information transfer and supervision of  ... 
arXiv:2108.06908v2 fatcat:56ctz3tbofatjaqafmvnxx6lnq

Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks

Jinghui Chen, Dongruo Zhou, Yiqi Tang, Ziyan Yang, Yuan Cao, Quanquan Gu
2020 Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence  
To compress the model, the progressive knowledge distillation with GANs is proposed, which is a novel exploration that applies the knowledge distillation on two standard GANs.  ...  Moreover, our proposed method can be used to compress other GANs-based applications, such as image generation.  ... 
doi:10.24963/ijcai.2020/448 dblp:conf/ijcai/ZhangCS20 fatcat:lo7lunpacnbppirpuu3l2zp3ha

Distilling Portable Generative Adversarial Networks for Image Translation

Hanting Chen, Yunhe Wang, Han Shu, Changyuan Wen, Chunjing Xu, Boxin Shi, Chao Xu, Chang Xu
2020 PROCEEDINGS OF THE THIRTIETH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE AND THE TWENTY-EIGHTH INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE CONFERENCE  
Inspired by knowledge distillation, a student generator of fewer parameters is trained by inheriting the low-level and high-level information from the original heavy teacher generator.  ...  Despite Generative Adversarial Networks (GANs) have been widely used in various image-to-image translation tasks, they can be hardly applied on mobile devices due to their heavy computation and storage  ...  Moreover, they do not distill knowledge to the discriminator, which takes an important part in GANs' training.  ... 
doi:10.1609/aaai.v34i04.5765 fatcat:v3stk3d36jdxpn4xcu4ly2y65u

Distilling portable Generative Adversarial Networks for Image Translation [article]

Hanting Chen, Yunhe Wang, Han Shu, Changyuan Wen, Chunjing Xu, Boxin Shi, Chao Xu, Chang Xu
2020 arXiv   pre-print
Inspired by knowledge distillation, a student generator of fewer parameters is trained by inheriting the low-level and high-level information from the original heavy teacher generator.  ...  Despite Generative Adversarial Networks (GANs) have been widely used in various image-to-image translation tasks, they can be hardly applied on mobile devices due to their heavy computation and storage  ...  Moreover, they do not distill knowledge to the discriminator, which takes an important part in GANs' training.  ... 
arXiv:2003.03519v1 fatcat:zch5iumcvbbofo6eneoiwv75hm

Semantic Relation Preserving Knowledge Distillation for Image-to-Image Translation [article]

Zeqi Li, Ruowei Jiang, Parham Aarabi
2021 arXiv   pre-print
In this work, we propose a novel method to address this problem by applying knowledge distillation together with distillation of a semantic relation preserving matrix.  ...  In contrast to existing compression methods designed for classification tasks, our proposed method adapts well to the image-to-image translation task on GANs.  ...  Conclusions We approach model compression of GANs via a novel proposed method extended on traditional knowledge distillation.  ... 
arXiv:2104.15082v2 fatcat:dbcz2zs3xbcjthh63kztcpuexm

Region-aware Knowledge Distillation for Efficient Image-to-Image Translation [article]

Linfeng Zhang, Xin Chen, Runpei Dong, Kaisheng Ma
2022 arXiv   pre-print
In this paper, we propose Region-aware Knowledge Distillation ReKo to compress image-to-image translation models.  ...  To address this issue, knowledge distillation is proposed to transfer the knowledge from a cumbersome teacher model to an efficient student model.  ...  GAN Knowledge Distillation In the last several years, there has been some research proposed to apply knowledge distillation to the compression of GANs. 3 Methodology Formulation Patch-wise Contrastive  ... 
arXiv:2205.12451v1 fatcat:gzifgo4zobfovf4lqtu4i7m5ue

Teachers Do More Than Teach: Compressing Image-to-Image Models [article]

Qing Jin, Jian Ren, Oliver J. Woodford, Jiazhuo Wang, Geng Yuan, Yanzhi Wang, Sergey Tulyakov
2021 arXiv   pre-print
Finally, we propose to distill knowledge through maximizing feature similarity between teacher and student via an index named Global Kernel Alignment (GKA).  ...  Recent efforts on compression GANs show noticeable progress in obtaining smaller generators by sacrificing image quality or involving a time-consuming searching process.  ...  Most GAN compression methods [1, 9, 20] use response-based distillation, enforcing the synthesized images from the teacher and student networks to be the same. Li et al.  ... 
arXiv:2103.03467v2 fatcat:d3rjuwhsdbbsvfna53i3vikmbi
« Previous Showing results 1 — 15 out of 1,972 results