58,768 Hits in 3.8 sec

Memory-Efficient Differentiable Transformer Architecture Search [article]

Yuekai Zhao, Li Dong, Yelong Shen, Zhihua Zhang, Furu Wei, Weizhu Chen
2021 arXiv   pre-print
Differentiable architecture search (DARTS) is successfully applied in many vision tasks.  ...  However, directly using DARTS for Transformers is memory-intensive, which renders the search process infeasible. To this end, we propose a multi-split reversible network and combine it with DARTS.  ...  Reversible networks Conclusion We have proposed a memory-efficient differentiable architecture search (DARTS) method on sequence-to-sequence tasks.  ... 
arXiv:2105.14669v1 fatcat:iqvywprvm5dpvmpwviypy7qhti

TND-NAS: Towards Non-Differentiable Objectives in Differentiable Neural Architecture Search [article]

Bo Lyu, Shiping Wen, Zheng Yan, Kaibo Shi, Ke Li, Tingwen Huang
2022 arXiv   pre-print
Differentiable architecture search has gradually become the mainstream research topic in the field of Neural Architecture Search (NAS) for its high efficiency compared with the early NAS (EA-based, RL-based  ...  Recent differentiable NAS also aims at further improving the search performance and reducing the GPU-memory consumption.  ...  Even with high search efficiency, differentiable NAS researches rarely involves the non-differentiable objectives, e.g., energy, latency, or memory consumption, and multi-objective are merely jointly considered  ... 
arXiv:2111.03892v2 fatcat:wgn45wznjvcntp6evov4so25gm

ReconfigISP: Reconfigurable Camera Image Processing Pipeline [article]

Ke Yu, Zexian Li, Yue Peng, Chen Change Loy, Jinwei Gu
2021 arXiv   pre-print
search and effectively search for the optimal ISP architecture.  ...  In particular, we implement several ISP modules, and enable backpropagation for each module by training a differentiable proxy, hence allowing us to leverage the popular differentiable neural architecture  ...  With all modules differentiable, the ISP architecture can be efficiently explored with the help of neural architecture search.  ... 
arXiv:2109.04760v1 fatcat:7khy3l2s5bdfbnp2cxzwd4uukq

Evolved Speech-Transformer: Applying Neural Architecture Search to End-to-End Automatic Speech Recognition

Jihwan Kim, Jisung Wang, Sangki Kim, Yeha Lee
2020 Interspeech 2020  
With a combination of carefully designed search space and Progressive dynamic hurdles, a genetic algorithm based, our algorithm finds a memory-efficient architecture which outperforms vanilla Transformer  ...  Neural architecture search (NAS) has been successfully applied to finding efficient, high-performance deep neural network architectures in a task-adaptive manner without extensive human intervention.  ...  In this work, we designed a search space apt for speech recognition and applied PDH to find a memory-efficient architecture which performs well across standard benchmark datasets with reduced training  ... 
doi:10.21437/interspeech.2020-1233 dblp:conf/interspeech/KimWKL20 fatcat:pseebobwqfgkpgztxmnqnsjvwq

Joint Search of Data Augmentation Policies and Network Architectures [article]

Taiga Kashima, Yoshihiro Yamada, Shunta Saito
2021 arXiv   pre-print
The proposed method combines differentiable methods for augmentation policy search and network architecture search to jointly optimize them in the end-to-end manner.  ...  The core idea of our approach is to make the whole part differentiable.  ...  memory and power efficiency at inference time by model compression.  ... 
arXiv:2012.09407v2 fatcat:otozvh2rz5btfn5hd7cacgei4y

Memory-efficient Embedding for Recommendations [article]

Xiangyu Zhao, Haochen Liu, Hui Liu, Jiliang Tang, Weiwei Guo, Jun Shi, Sida Wang, Huiji Gao, Bo Long
2020 arXiv   pre-print
Most of them empirically allocate a unified dimension to all feature fields, which is memory inefficient.  ...  Specifically, we first proposed an end-to-end differentiable framework that can calculate the weights over various dimensions for feature fields in a soft and continuous manner with an AutoML based optimization  ...  Inspired by the differentiable architecture search (DARTS) techniques [24] , W and are alternately optimized through gradient descent.  ... 
arXiv:2006.14827v2 fatcat:2ujor3sqf5aild7r7rkknwxvtm

Mutually-aware Sub-Graphs Differentiable Architecture Search [article]

Haoxian Tan, Sheng Guo, Yujie Zhong, Matthew R. Scott, Weilin Huang
2021 arXiv   pre-print
Differentiable architecture search is prevalent in the field of NAS because of its simplicity and efficiency, where two paradigms, multi-path algorithms and single-path methods, are dominated.  ...  In this paper, we propose a conceptually simple yet efficient method to bridge these two paradigms, referred as Mutually-aware Sub-Graphs Differentiable Architecture Search (MSG-DAS).  ...  PC-DARTS [48] reduces the memory usage by performing architecture search in a randomly sampled subset of operations.  ... 
arXiv:2107.04324v3 fatcat:xi47dor6ljdp3iygqhvgcvucwq

SparseMask: Differentiable Connectivity Learning for Dense Image Prediction

Huikai Wu, Junge Zhang, Kaiqi Huang
2019 2019 IEEE/CVF International Conference on Computer Vision (ICCV)  
In this paper, we aim at automatically searching an efficient network architecture for dense image prediction.  ...  We then employ gradient descent to search the optimal connectivity from the dense connections.  ...  After training, we prune FDN to obtain the final architecture with sparse connectivity (Figure 1c ), which is time and memory efficient.  ... 
doi:10.1109/iccv.2019.00687 dblp:conf/iccv/WuZH19 fatcat:wy3x2fdqkzgdtbeesln2jggjje

Searching for A Robust Neural Architecture in Four GPU Hours [article]

Xuanyi Dong, Yi Yang
2019 arXiv   pre-print
In this way, our approach can be trained in an end-to-end fashion by gradient descent, named Gradient-based search using Differentiable Architecture Sampler (GDAS).  ...  We propose an efficient NAS approach learning to search by gradient descent. Our approach represents the search space as a directed acyclic graph (DAG).  ...  Conclusion In this paper, we propose a Gradient-based neural architecture search approach using Differentiable Architecture Sampler (GDAS).  ... 
arXiv:1910.04465v2 fatcat:fsuntwbkbngbhl5aopl5rfxkym

An auto-tuning framework for parallel multicore stencil computations

Shoaib Kamil, Cy Chan, Leonid Oliker, John Shalf, Samuel Williams
2010 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPS)  
Overall we demonstrate that such domain-specific auto-tuners hold enormous promise for architectural efficiency, programmer productivity, performance portability, and algorithmic adaptability on existing  ...  Although stencil auto-tuning has shown tremendous potential in effectively utilizing architectural resources, it has hitherto been limited to single kernel instantiations; in addition, the large variety  ...  most power efficient architecture evaluated in this study.  ... 
doi:10.1109/ipdps.2010.5470421 dblp:conf/ipps/KamilCOSW10 fatcat:endkmhvhjjfuvmnpmuxp7elpm4

Searching for a Robust Neural Architecture in Four GPU Hours

Xuanyi Dong, Yi Yang
2019 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)  
In this way, our approach can be trained in an end-to-end fashion by gradient descent, named Gradient-based search using Differentiable Architecture Sampler (GDAS).  ...  We propose an efficient NAS approach learning to search by gradient descent. Our approach represents the search space as a directed acyclic graph (DAG).  ...  Conclusion In this paper, we propose a Gradient-based neural architecture search approach using Differentiable Architecture Sampler (GDAS).  ... 
doi:10.1109/cvpr.2019.00186 dblp:conf/cvpr/DongY19 fatcat:lz6dsbsqazddbhqnd2ims27rxu

Representation Sharing for Fast Object Detector Search and Beyond [article]

Yujie Zhong, Zelu Deng, Sheng Guo, Matthew R. Scott, Weilin Huang
2020 arXiv   pre-print
FAD consists of a designed search space and an efficient architecture search algorithm. The search space contains a rich set of diverse transformations designed specifically for object detection.  ...  To enhance such capability, we propose an extremely efficient neural architecture search method, named Fast And Diverse (FAD), to better explore the optimal configuration of receptive fields and convolution  ...  A stream of efficient NAS methods is the differentiable NAS [26, 23] .  ... 
arXiv:2007.12075v4 fatcat:wj2hhhnyrfgivg2wr5sx3msgsm

Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications

Han Cai, Ji Lin, Yujun Lin, Zhijian Liu, Haotian Tang, Hanrui Wang, Ligeng Zhu, Song Han
2022 ACM Transactions on Design Automation of Electronic Systems  
To reduce the large design cost of these manual solutions, we discuss the AutoML framework for each of them, such as neural architecture search (NAS) and automated pruning and quantization.  ...  This article provides an overview of efficient deep learning methods, systems, and applications.  ...  It then employs differentiable NAS to search for models with low memory usage and low op count. Rusci et al.  ... 
doi:10.1145/3486618 fatcat:h6xwv2slo5eklift2fl24usine

Learning Algorithmic Solutions to Symbolic Planning Tasks with a Neural Computer Architecture [article]

Daniel Tanneberg, Elmar Rueckert, Jan Peters
2020 arXiv   pre-print
Therefore, we present a novel architecture, based on memory-augmented networks, that is inspired by the von Neumann and Harvard architectures of modern computers.  ...  This architecture enables the learning of abstract algorithmic solutions via Evolution Strategies in a reinforcement learning setting.  ...  Due to the design of the interface and the attention mechanisms, the DNC is independent of the memory size and fully differentiable, allowing gradient-based end-to-end learning. Our architecture.  ... 
arXiv:1911.00926v2 fatcat:h23q6v2lkvdgrht7ecbnfchviq

DSNAS: Direct Neural Architecture Search without Parameter Retraining [article]

Shoukang Hu, Sirui Xie, Hehui Zheng, Chunxiao Liu, Jianping Shi, Xunying Liu, Dahua Lin
2020 arXiv   pre-print
Seeing that most existing methods do not solve this problem directly, we propose DSNAS, an efficient differentiable NAS framework that simultaneously optimizes architecture and parameters with a low-biased  ...  However, performance of the same architecture in the two stages correlates poorly. In this work, we propose a new problem definition for NAS, task-specific end-to-end, based on this observation.  ...  Our proposal is an efficient differentiable NAS framework, Discrete Stochastic Neural Architecture Search (DSNAS).  ... 
arXiv:2002.09128v2 fatcat:lmv7yx6wxfc63ilye5vfjtj6ii
« Previous Showing results 1 — 15 out of 58,768 results