Filters








22,852 Hits in 4.3 sec

On the information bottleneck theory of deep learning

Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, David D Cox
2019 Journal of Statistical Mechanics: Theory and Experiment  
Finally, we show that when an input domain consists of a subset of task-relevant and task-irrelevant information, hidden representations do compress the task-irrelevant information, although the overall  ...  information about the input may monotonically increase with training time, and that this compression happens concurrently with the fitting process rather than during a subsequent compression period.  ...  On the information bottleneck theory of deep learning As shown in figures 2(C) and (D), as a function of the weight w 1 , mutual information with the input first increases and then decreases for the tanh  ... 
doi:10.1088/1742-5468/ab3985 fatcat:t5b5mmcy2vhwxj5smvqiloexki

Guest Editorial

Richard Baraniuk, Alex Dimakis, Negar Kiyavash, Sewoong Oh, Rebecca Willett
2020 IEEE Journal on Selected Areas in Information Theory  
of information theory and deep learning: the information bottleneck, deep learning for communication systems, and deep learning for inverse problems.  ...  of deep learning fundamentals and new applications in important areas.  ... 
doi:10.1109/jsait.2020.2991703 fatcat:xedinamg35ggxga65xn3esf6vu

Variational Information Bottleneck Model for Accurate Indoor Position Recognition [article]

Weizhu Qian, Franck Gechter
2021 arXiv   pre-print
Based on these two approaches, we propose a Variational Information Bottleneck model for accurate indoor positioning. The proposed model consists of an encoder structure and a predictor structure.  ...  In this work, we solve this issue by combining the Information Bottleneck method and Variational Inference.  ...  ACKNOWLEDGMENT The authors would like to thank the China Scholarship Council for the financial support.  ... 
arXiv:2101.10655v1 fatcat:zbk7wtobpvbynpejq55ydugo7e

Information Bottleneck Theory on Convolutional Neural Networks [article]

Junjie Li, Ding Liu
2021 arXiv   pre-print
Among them, Information Bottleneck (IB) theory claims that there are two distinct phases consisting of fitting phase and compression phase in the course of training.  ...  Recent years, many researches attempt to open the black box of deep neural networks and propose a various of theories to understand it.  ...  DISCUSSION Information bottleneck theory provides a interesting analytic tool to explore the inner behaviour of deep neural network, and based on this, people try to understand why deep learning works  ... 
arXiv:1911.03722v2 fatcat:pnb5mwnudfad7fma2z27syg33m

Information flows of diverse autoencoders [article]

Sungyeop Lee, Junghyo Jo
2021 arXiv   pre-print
The outstanding performance of deep learning in various fields has been a fundamental query, which can be potentially examined using information theory that interprets the learning process as the transmission  ...  Information plane analyses of the mutual information between the input-hidden-output layers demonstrated two distinct learning phases of fitting and compression.  ...  The information bottleneck (IB) theory interprets the learning process of neural networks as the transmission and compression of information [6] .  ... 
arXiv:2102.07402v2 fatcat:bhfae3zfm5fwbgpdflz7nucl74

Information Flows of Diverse Autoencoders

Sungyeop Lee, Junghyo Jo
2021 Entropy  
Information theory provides a potential answer by interpreting the learning process as the information transmission and compression of data.  ...  The information flows can be visualized on the information plane of the mutual information among the input, hidden, and output layers.  ...  The information bottleneck (IB) theory interprets the learning process of neural networks as the transmission and compression of information [6] .  ... 
doi:10.3390/e23070862 fatcat:nm4ls6ym35hffh3wuqg6gxwn44

Neural Network Activation Quantization with Bitwise Information Bottlenecks [article]

Xichuan Zhou, Kui Liu, Cong Shi, Haijun Liu, Ji Liu
2020 arXiv   pre-print
Recent researches on information bottleneck shed new light on the continuous attempts to open the black box of neural signal encoding.  ...  Based on the rate-distortion theory, the Bitwise Information Bottleneck attempts to determine the most significant bits in activation representation by assigning and approximating the sparse coefficient  ...  Recently, the concept of information bottleneck is attracting attention for its potential to bring a better understanding of the deep learning optimization process [27, 25] .  ... 
arXiv:2006.05210v1 fatcat:ncl4xrd7evgprdhx3peqzf5zrm

Improving Generalization of Deep Networks for Inverse Reconstruction of Image Sequences [article]

Sandesh Ghimire, Prashnna Kumar Gyawali, Jwala Dhamala, John L Sapp, Milan Horacek, Linwei Wang
2019 arXiv   pre-print
Deep learning networks have shown state-of-the-art performance in many image reconstruction problems.  ...  First, drawing from analytical learning theory, we theoretically show that a stochastic latent space will improve the ability of a network to generalize to test data outside the training distribution.  ...  In this paper, we take an information theoretic perspective -along with analytical learning theory -to investigate and improve the generalization ability of deep image reconstruction networks.  ... 
arXiv:1903.02948v1 fatcat:c4tvttdgyfhh3i7sz7tlslu7ke

The HSIC Bottleneck: Deep Learning without Back-Propagation

Wan-Duo Kurt Ma, J. P. Lewis, W. Bastiaan Kleijn
2020 PROCEEDINGS OF THE THIRTIETH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE AND THE TWENTY-EIGHTH INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE CONFERENCE  
The HSIC bottleneck is an alternative to the conventional cross-entropy loss and backpropagation that has a number of distinct advantages.  ...  We introduce the HSIC (Hilbert-Schmidt independence criterion) bottleneck for training deep neural networks.  ...  Information theory (Cover and Thomas 2006) underlies much research on learning theory (Belghazi et al. 2018; Kwak and Chong-Ho Choi 2002; Brakel and Bengio 2018) as well as thinking in neuroscience  ... 
doi:10.1609/aaai.v34i04.5950 fatcat:gro36egngzfmnfnkrt4qhpqaja

The Effect of Evidence Transfer on Latent Feature Relevance for Clustering

Athanasios Davvetas, Iraklis A. Klampanos, Spiros Skiadopoulos, Vangelis Karkaletsis
2019 Informatics  
We interpret the effects of evidence transfer on the latent representation of an autoencoder by comparing our method to the information bottleneck method.  ...  Evidence transfer for clustering is a deep learning method that manipulates the latent representations of an autoencoder according to external categorical evidence with the effect of improving a clustering  ...  Information bottleneck recently received a lot of attention in the domain of deep learning.  ... 
doi:10.3390/informatics6020017 fatcat:t7ytk4kg2vgflcjqkodlmou26e

The HSIC Bottleneck: Deep Learning without Back-Propagation [article]

Wan-Duo Kurt Ma, J.P. Lewis, W. Bastiaan Kleijn
2019 arXiv   pre-print
The HSIC bottleneck is an alternative to the conventional cross-entropy loss and backpropagation that has a number of distinct advantages.  ...  We introduce the HSIC (Hilbert-Schmidt independence criterion) bottleneck for training deep neural networks.  ...  Information theory (Cover and Thomas 2006) underlies much research on learning theory (Belghazi et al. 2018; Kwak and Chong-Ho Choi 2002; Brakel and Bengio 2018) as well as thinking in neuroscience  ... 
arXiv:1908.01580v3 fatcat:6yjb3azvrzaqffpsgcuyah3oye

Sentiment Analysis via Deep Multichannel Neural Networks with Variational Information Bottleneck

Tong Gu, Guoliang Xu, Jiangtao Luo
2020 IEEE Access  
ACKNOWLEDGMENT The authors thank the anonymous reviewers for their helpful comments. Also, scholars such as Tan who provided support of experimental data should be appreciated.  ...  Based on the information theory, an information bottleneck theory was proposed [32] and used to analyze deep neural networks. , the information of X is compressed by Z as much as possible, while retaining  ...  In 2006, a fast learning algorithm for deep belief nets was proposed [8] , which is the beginning of deep learning.  ... 
doi:10.1109/access.2020.3006569 fatcat:ssdkrt6annhvtnlnxx3bijo2cy

Self-Supervised Graph Representation Learning via Information Bottleneck

Junhua Gu, Zichen Zheng, Wenmiao Zhou, Yajuan Zhang, Zhengjun Lu, Liang Yang
2022 Symmetry  
Therefore, the self-supervised graph information bottleneck (SGIB) proposed in this paper uses the symmetry and asymmetry of graphs to establish comparative learning and introduces the information bottleneck  ...  Therefore, we can infer from different network analysis experiments that it would be an effective improvement of the performance of downstream tasks through introducing information bottleneck theory to  ...  Informed Consent Statement: Not applicable. Data Availability Statement: Not applicable. Conflicts of Interest: The authors declare no conflict of interest.  ... 
doi:10.3390/sym14040657 fatcat:oosvggxjerfc3hvbtlreeibu7i

The Distributed Information Bottleneck reveals the explanatory structure of complex systems [article]

Kieran A. Murphy, Dani S. Bassett
2022 arXiv   pre-print
Here we show that a crucial modification -- distributing bottlenecks across multiple components of the input -- opens fundamentally new avenues for interpretable deep learning in science.  ...  By way of a principled scheme of approximations, the Distributed IB brings much-needed interpretability to deep learning and enables unprecedented analysis of information flow through a system.  ...  Kim for helpful discussions and comments on the manuscript, and Dr. Sylvain Patinet for the amorphous plasticity data.  ... 
arXiv:2204.07576v1 fatcat:xmsn7kr5wbcb7fp567bmxsufpy

Aggregated Learning: A Deep Learning Framework Based on Information-Bottleneck Vector Quantization [article]

Hongyu Guo, Yongyi Mao, Ali Al-Bashabsheh, Richong Zhang
2019 arXiv   pre-print
Based on the notion of information bottleneck (IB), we formulate a quantization problem called "IB quantization". We show that IB quantization is equivalent to learning based on the IB principle.  ...  We also empirically show that AgrLearn can reduce up to 80% of the training samples needed for ResNet training.  ...  It builds on an equivalence between IB learning and IB quantization and exploits the power of vector quantization, well known in information theory.  ... 
arXiv:1807.10251v3 fatcat:7opuatzfknfh5ld2qat5y4qzuq
« Previous Showing results 1 — 15 out of 22,852 results