651 Hits in 2.3 sec

Towards Evolutional Compression [article]

Yunhe Wang, Chang Xu, Jiayan Qiu, Chao Xu, Dacheng Tao
2017 arXiv   pre-print
In the DCT frequency domain, Wang et al. [24] excavated redundancy on all weights and their underlying connections to deliver higher compression and speed-up ratios.  ... 
arXiv:1707.08005v1 fatcat:3ay6vkbe6be2zcauedfum2xwae

Adversarially Robust Neural Architectures [article]

Minjing Dong, Yanxi Li, Yunhe Wang, Chang Xu
2020 arXiv   pre-print
Deep Neural Network (DNN) are vulnerable to adversarial attack. Existing methods are devoted to developing various robust training strategies or regularizations to update the weights of the neural network. But beyond the weights, the overall structure and information flow in the network are explicitly determined by the neural architecture, which remains unexplored. This paper thus aims to improve the adversarial robustness of the network from the architecture perspective with NAS framework. We
more » ... xplore the relationship among adversarial robustness, Lipschitz constant, and architecture parameters and show that an appropriate constraint on architecture parameters could reduce the Lipschitz constant to further improve the robustness. For NAS framework, all the architecture parameters are equally treated when the discrete architecture is sampled from supernet. However, the importance of architecture parameters could vary from operation to operation or connection to connection, which is not explored and might reduce the confidence of robust architecture sampling. Thus, we propose to sample architecture parameters from trainable multivariate log-normal distributions, with which the Lipschitz constant of entire network can be approximated using a univariate log-normal distribution with mean and variance related to architecture parameters. Compared with adversarially trained neural architectures searched by various NAS algorithms as well as efficient human-designed models, our algorithm empirically achieves the best performance among all the models under various attacks on different datasets.
arXiv:2009.00902v1 fatcat:st4c7uv5cjbt7cum6u5mfis754

Multimodal Token Fusion for Vision Transformers [article]

Yikai Wang, Xinghao Chen, Lele Cao, Wenbing Huang, Fuchun Sun, Yunhe Wang
2022 arXiv   pre-print
Many adaptations of transformers have emerged to address the single-modal vision tasks, where self-attention modules are stacked to handle input sources like images. Intuitively, feeding multiple modalities of data to vision transformers could improve the performance, yet the inner-modal attentive weights may also be diluted, which could thus undermine the final performance. In this paper, we propose a multimodal token fusion method (TokenFusion), tailored for transformer-based vision tasks. To
more » ... effectively fuse multiple modalities, TokenFusion dynamically detects uninformative tokens and substitutes these tokens with projected and aggregated inter-modal features. Residual positional alignment is also adopted to enable explicit utilization of the inter-modal alignments after fusion. The design of TokenFusion allows the transformer to learn correlations among multimodal features, while the single-modal transformer architecture remains largely intact. Extensive experiments are conducted on a variety of homogeneous and heterogeneous modalities and demonstrate that TokenFusion surpasses state-of-the-art methods in three typical vision tasks: multimodal image-to-image translation, RGB-depth semantic segmentation, and 3D object detection with point cloud and images.
arXiv:2204.08721v1 fatcat:z7hgwsvvjneyhkiw4bybl3ne5q

Privileged Multi-label Learning [article]

Shan You, Chang Xu, Yunhe Wang, Chao Xu, Dacheng Tao
2017 arXiv   pre-print
harvested from the training data, and it has been widely-used in many applications, such as document categorization Yang et al. (2009) ; Li et al. (2015) , image/videos classification/annotation ; Wang  ... 
arXiv:1701.07194v1 fatcat:s2npmi5morapjf36aghmfzlm6e

Nijenhuis operators on pre-Lie algebras [article]

Qi Wang, Chengming Bai, Jiefeng Liu, Yunhe Sheng
2017 arXiv   pre-print
First we use a new approach to give a graded Lie algebra whose Maurer-Cartan elements characterize pre-Lie algebra structures. Then using this graded Lie bracket we define the notion of a Nijenhuis operator on a pre-Lie algebra which generates a trivial deformation of this pre-Lie algebra. There are close relationships between O-operators, Rota-Baxter operators and Nijenhuis operators on a pre-Lie algebra. In particular, a Nijenhuis operator "connects" two O-operators on a pre-Lie algebra whose
more » ... any linear combination is still an O-operator in certain sense and hence compatible L-dendriform algebras appear naturally as the induced algebraic structures. For the case of the dual representation of the regular representation of a pre-Lie algebra, there is a geometric interpretation by introducing the notion of a pseudo-Hessian-Nijenhuis structure which gives rise to a sequence of pseudo-Hessian and pseudo-Hessian-Nijenhuis structures. Another application of Nijenhuis operators on pre-Lie algebras in geometry is illustrated by introducing the notion of a para-complex structure on a pre-Lie algebra and then studying paracomplex quadratic pre-Lie algebras and paracomplex pseudo-Hessian pre-Lie algebras in detail. Finally, we give some examples of Nijenhuis operators on pre-Lie algebras.
arXiv:1710.03749v1 fatcat:fwqhu3u7hbd77musk42ivzdfaq

Neural Architecture Dilation for Adversarial Robustness [article]

Yanxi Li, Zhaohui Yang, Yunhe Wang, Chang Xu
2021 arXiv   pre-print
Wang 2 , Chang Xu 1 1 School of Computer Science, University of Sydney, Australia  ...  Architecture Dilation for Adversarial Robustness Yanxi Li 1 , Zhaohui Yang 2,3 , Yunhe  ... 
arXiv:2108.06885v1 fatcat:zk2s2jiv6vdvtln7sg4irhbntq

Automatically Searching for U-Net Image Translator Architecture [article]

Han Shu, Yunhe Wang
2020 arXiv   pre-print
Probabilities of s 1 = 0.2, s 2 = 0.7, s 3 = 0.1 are adopted for selection, cross over and mutation in each generation as suggested in [Wang et al., 2018b] .  ...  [Wang et al., 2018a] presented a novel adversarial loss as well as a coarse-to-fine generator and a multi-scale discriminator to address the high-resolution image-to-image translation problem.  ... 
arXiv:2002.11581v1 fatcat:vsfgfinzk5fd7cb22oibuqc4ya

Coarse-to-Fine Searching for Efficient Generative Adversarial Networks [article]

Jiahao Wang, Han Shu, Weihao Xia, Yujiu Yang, Yunhe Wang
2021 arXiv   pre-print
Wang et al. [71] further tackled this problem from the perspective of DCT frequency domain to achieve higher compression ratios.  ...  Wang et al. [65] combined pruning and quantization strategy to establish a unified compression method. Chen et al. [9] compressed GANs by the lottery ticket hypothesis.  ... 
arXiv:2104.09223v1 fatcat:u3t62uvt7nhopipjeo2vfu24ty

Winograd Algorithm for AdderNet [article]

Wenshuo Li, Hanting Chen, Mingqiang Huang, Xinghao Chen, Chunjing Xu, Yunhe Wang
2021 arXiv   pre-print
Wenshuo Li <>, Yunhe Wang <>. Proceedings of the 38 th International Conference on Machine Learning, PMLR 139, 2021.  ... 
arXiv:2105.05530v1 fatcat:nfngehf4cbez3hrbjxld66cea4

Learning Student Networks via Feature Embedding [article]

Hanting Chen, Yunhe Wang, Chang Xu, Chao Xu, Dacheng Tao
2018 arXiv   pre-print
In addition, Wang  ...  Wang [35] factorized the convolutional layer by considering spatial convolution.  ... 
arXiv:1812.06597v1 fatcat:wnn2lukhunfvxb7xhvvowfjd4m

Post-Training Quantization for Vision Transformer [article]

Zhenhua Liu, Yunhe Wang, Kai Han, Siwei Ma, Wen Gao
2021 arXiv   pre-print
Wang et al.  ... 
arXiv:2106.14156v1 fatcat:a7ieqevmyveahfutbtetjhb2jm

DC-NAS: Divide-and-Conquer Neural Architecture Search [article]

Yunhe Wang, Yixing Xu, Dacheng Tao
2020 arXiv   pre-print
Wang [39] proposed an evolutionary method to automatically identify redundant filters in pre-trained deep neural networks. Pham  ... 
arXiv:2005.14456v1 fatcat:4pledk4txjefhg4tu6gz2h5v7i

Searching for Accurate Binary Neural Architectures [article]

Mingzhu Shen and Kai Han and Chunjing Xu and Yunhe Wang
2019 arXiv   pre-print
Binary neural networks have attracted tremendous attention due to the efficiency for deploying them on mobile devices. Since the weak expression ability of binary weights and features, their accuracy is usually much lower than that of full-precision (i.e. 32-bit) models. Here we present a new frame work for automatically searching for compact but accurate binary neural networks. In practice, number of channels in each layer will be encoded into the search space and optimized using the
more » ... ry algorithm. Experiments conducted on benchmark datasets and neural architectures demonstrate that our searched binary networks can achieve the performance of full-precision models with acceptable increments on model sizes and calculations.
arXiv:1909.07378v1 fatcat:rswjrttx4vd3xhgcggw3zcslga

A Data-Driven Uncertainty Quantification Method for Stochastic Economic Dispatch [article]

Xiaoting Wang, Rong-Peng Liu, Xiaozhe Wang, Yunhe Hou, François Bouffard
2021 arXiv   pre-print
This letter proposes a data-driven sparse polynomial chaos expansion-based surrogate model for the stochastic economic dispatch problem considering uncertainty from wind power. The proposed method can provide accurate estimations for the statistical information (e.g., mean, variance, probability density function, and cumulative distribution function) for the stochastic economic dispatch solution efficiently without requiring the probability distributions of random inputs. Simulation studies on
more » ... n integrated electricity and gas system (IEEE 118-bus system integrated with a 20-node gas system are presented, demonstrating the efficiency and accuracy of the proposed method compared to the Monte Carlo simulations.
arXiv:2109.08195v1 fatcat:3jmbq5ao3nhwzfyv4dg5mi7lmq

Augmented Shortcuts for Vision Transformers [article]

Yehui Tang, Kai Han, Chang Xu, An Xiao, Yiping Deng, Chao Xu, Yunhe Wang
2021 arXiv   pre-print
Transformer models have achieved great progress on computer vision tasks recently. The rapid development of vision transformers is mainly contributed by their high representation ability for extracting informative features from input images. However, the mainstream transformer models are designed with deep architectures, and the feature diversity will be continuously reduced as the depth increases, i.e., feature collapse. In this paper, we theoretically analyze the feature collapse phenomenon
more » ... d study the relationship between shortcuts and feature diversity in these transformer models. Then, we present an augmented shortcut scheme, which inserts additional paths with learnable parameters in parallel on the original shortcuts. To save the computational costs, we further explore an efficient approach that uses the block-circulant projection to implement augmented shortcuts. Extensive experiments conducted on benchmark datasets demonstrate the effectiveness of the proposed method, which brings about 1% accuracy increase of the state-of-the-art visual transformers without obviously increasing their parameters and FLOPs.
arXiv:2106.15941v1 fatcat:de5jbvo3izfexblz7m6nsjxqnq
« Previous Showing results 1 — 15 out of 651 results