Filters








3,145 Hits in 6.6 sec

CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information [article]

Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, Lawrence Carin
2020 arXiv   pre-print
In this paper, we propose a novel Contrastive Log-ratio Upper Bound (CLUB) of mutual information. We provide a theoretical analysis of the properties of CLUB and its variational approximation.  ...  Based on this upper bound, we introduce a MI minimization training scheme and further accelerate it with a negative sampling strategy.  ...  The portion of this work performed at Duke University was supported in part by DARPA, DOE, NIH, NSF and ONR.  ... 
arXiv:2006.12013v6 fatcat:ilsrj3qxsjerfgkqr5c7sq4nuq

Computation is concentrated in rich clubs 1 of local cortical networks

Samantha P. Faber, Nicholas M. Timme, John M. Beggs, Ehren L. Newman
2018 Network Neuroscience  
Rich clubs, highly interconnected sets of neurons, are known to propagate a disproportionate amount of information within cortical circuits.  ...  Here, we test the hypothesis that rich clubs also perform a disproportionate amount of computation.  ...  (C) Histogram of computation ratio values for all receivers in all networks. (D) Histogram of log-scaled computation ratio values for all receivers in all networks.  ... 
doi:10.1162/netn_a_00069 pmid:30793088 pmcid:PMC6370472 fatcat:55ru6nbi3jc7pmboxfxfwgb7qi

Computation is concentrated in rich clubs of local cortical neurons [article]

Samantha P Faber, Nicholas M Timme, John M Beggs, Ehren L Newman
2018 bioRxiv   pre-print
Rich-clubs, highly interconnected sets of neurons, are known to propagate a disproportionate amount of information within cortical circuits.  ...  Here, we test the hypothesis that rich-clubs also perform a disproportionate amount of computation.  ...  This is in contrast to information propagation which simply passes (unmodified) information from a source to a receiver.  ... 
doi:10.1101/290981 fatcat:j3s7exyj3fg4jirq2y64r5w6ky

Estimating Total Correlation with Mutual Information Bounds [article]

Pengyu Cheng, Weituo Hao, Lawrence Carin
2020 arXiv   pre-print
Total correlation (TC) is a fundamental concept in information theory to measure the statistical dependency of multiple random variables.  ...  Specifically, we connect the TC with mutual information (MI) and constructed two calculation paths to decompose TC into MI terms.  ...  The MI contrastive log-ratio upper bound (CLUB) estimator [8] is based on a parameterized distribution q θ (y|x): I(x; y) ≤ E[ 1 N N i=1 [log p(x i |y i ) − 1 N N j=1 log p(x j |y i )]]. (7) All the  ... 
arXiv:2011.04794v1 fatcat:frlhdpea6neltoxyvbml2dozay

Club Goods and Group Identity: Evidence from Islamic Resurgence during the Indonesian Financial Crisis

Daniel L. Chen
2010 Journal of Political Economy  
This paper tests a model in which group identity in the form of religious intensity functions as ex post insurance.  ...  ) that is weaker for other forms of group identity.  ...  Consistent with an increasing demand, according to a 1992 Library of Congress report, an upper bound of 17 percent of the Muslim schoolage population attended Islamic schools in 1992. 5 According to the  ... 
doi:10.1086/652462 fatcat:d3g3ao2flrbfjlvrghdiocp6lq

Robust Speech Representation Learning via Flow-based Embedding Regularization [article]

Woo Hyun Kang, Jahangir Alam, Abderrahim Fathan
2021 arXiv   pre-print
To achieve this, our proposed method directly incorporates the information bottleneck scheme into the training process, where the mutual information is estimated using the main task classifier and an auxiliary  ...  In order to alleviate this problem, we propose a novel training strategy that regularizes the embedding network to have minimum information about the nuisance attributes.  ...  Contrastive Log-ratio Upper-Bound Mutual Information CLUB [Cheng et al., 2020] is a mutual information estimation method that is trained via contrastive learning.  ... 
arXiv:2112.03454v1 fatcat:ym3ywfcjbnhxzi4cwsbgz7aswa

Hindsight Value Function for Variance Reduction in Stochastic Dynamic Environment [article]

Jiaming Guo, Rui Zhang, Xishan Zhang, Shaohui Peng, Qi Yi, Zidong Du, Xing Hu, Qi Guo, Yunji Chen
2021 arXiv   pre-print
In this paper, we propose to replace the state value function with a novel hindsight value function, which leverages the information from the future to reduce the variance of the gradient estimate for  ...  Particularly, to obtain an ideally unbiased gradient estimate, we propose an information-theoretic approach, which optimizes the embeddings of the future to be independent of previous actions.  ...  To minimize the mutual information, we employ the contrastive logratio upper bound(CLUB) [Cheng et al., 2020] , which is a method to estimate the upper bound of mutual information.  ... 
arXiv:2107.12216v2 fatcat:vygbpu2ybbhwrdf22l4jkm27ti

Adaptive Information Bottleneck Guided Joint Source-Channel Coding [article]

Lunan Sun, Caili Guo, Yang Yang
2022 arXiv   pre-print
In this paper, we propose an adaptive Information Bottleneck (IB) guided JSCC (AIB-JSCC), which aims at achieving the theoretically maximal compression ratio for a given reconstruction quality.  ...  In particular, we first derive a mathematically tractable form of loss function for AIB-JSCC.  ...  In particular, we first derive a mathematically tractable form of IB objective for AIB-JSCC via variational lower bound and contrastive log-ratio upper bound (CLUB) of mutual information.  ... 
arXiv:2203.06492v1 fatcat:73h4qjpnbndnhbwbxrkmpxvv44

Multimodal Representations Learning Based on Mutual Information Maximization and Minimization and Identity Embedding for Multimodal Sentiment Analysis [article]

Jiahao Zheng, Sen Zhang, Xiaoping Wang, Zhigang Zeng
2022 arXiv   pre-print
In this work, we propose a multimodal representation model based on Mutual information Maximization and Minimization and Identity Embedding (MMMIE).  ...  We combine mutual information maximization between modal pairs, and mutual information minimization between input data and corresponding features to mine the modal-invariant and task-related information  ...  Acknowledgments The work was supported by the National Natural Science Foundation of China under Grant no. 61876209 and the Na-  ... 
arXiv:2201.03969v1 fatcat:vmsrtedszbgovbhvakfhljpgye

Task-Oriented Semantic Communication with Semantic Reconstruction: An Extended Rate-Distortion Theory Based Scheme [article]

Fangfang Liu, Wanjie Tong, Zhengfen Sun, Yang Yang, Caili Guo
2022 arXiv   pre-print
In the TOSC-SR scheme which is feasible in practice, a relaxed version of loss function is derived based on variational approximation of mutual information.  ...  We formulate the TOSC-SR scheme as a rate-distortion optimization problem, where a novel semantic distortion measurement is defined by mutual information of source, the semantic-reconstructed images, and  ...  𝐼 MINE := E 𝑝 ( 𝑥,𝑦) [𝑇 𝜃 (𝑥, 𝑦)] − log E 𝑝 ( 𝑥) 𝑝 ( 𝑦) 𝑒 𝑇 𝜃 ( 𝑥,𝑦) , Recently, Cheng et al. [29] introduce a Contrastive Logratio Upper Bound (CLUB).  ... 
arXiv:2201.10929v2 fatcat:wv5ii7ei75fnjfpobljzi6u7uy

Robust Inference of Risks of Large Portfolios

Jianqing Fan, Fang Han, Han Liu, Byron Vickers
2015 Social Science Research Network  
We propose a bootstrap-based robust high-confidence level upper bound (Robust H-CLUB) for assessing the risks of large portfolios.  ...  The proposed approach exploits rank-based and quantile-based estimators, and can be viewed as a robust extension of the H-CLUB procedure (Fan et al., 2015) .  ...  In this table we present coverage proportions, means of RE 1 and RE 2 , as well as the mean of the ratio between the 95% bound and the value it is upper bounding, with this ratio given by U (0.05)/∆.  ... 
doi:10.2139/ssrn.2547986 fatcat:6xst5utrkzbr7lbwxmy4hklwzy

Robust inference of risks of large portfolios

Jianqing Fan, Fang Han, Han Liu, Byron Vickers
2016 Journal of Econometrics  
We propose a bootstrap-based robust high-confidence level upper bound (Robust H-CLUB) for assessing the risks of large portfolios.  ...  The proposed approach exploits rank-based and quantile-based estimators, and can be viewed as a robust extension of the H-CLUB procedure (Fan et al., 2015) .  ...  In this table we present coverage proportions, means of RE 1 and RE 2 , as well as the mean of the ratio between the 95% bound and the value it is upper bounding, with this ratio given by U (0.05)/∆.  ... 
doi:10.1016/j.jeconom.2016.05.008 pmid:27818569 pmcid:PMC5091326 fatcat:c5za5vqphvgevgvnep7yb4r5gm

A Novel Estimator of Mutual Information for Learning to Disentangle Textual Representations [article]

Pierre Colombo and Chloe Clavel and Pablo Piantanida
2021 arXiv   pre-print
This paper introduces a novel variational upper bound to the mutual information between an attribute and the latent code of an encoder.  ...  on minimising variational bounds of the mutual information between latent code and the value attribute.  ...  The work of Prof. Pablo Piantanida was supported by the European Commission's Marie Sklodowska-Curie Actions (MSCA), through the Marie Sklodowska-Curie IF (H2020-MSCAIF-2017-EF-797805).  ... 
arXiv:2105.02685v1 fatcat:yqda2gawn5gkpas7yesm3ddhwu

Asymptotic resolution bounds of generalized modularity and statistically significant community detection [article]

Xiaoyan Lu, Boleslaw K. Szymanski
2019 arXiv   pre-print
We establish the asymptotic theoretical bounds on the resolution parameter of generalized modularity using the random graph properties.  ...  From this new perspective on random graph model, we find the resolution limit of modularity maximization can be explained in a surprisingly simple and straightforward way.  ...  theoretical upper and lower bounds on the resolution parameter of generalized modularity.  ... 
arXiv:1902.04243v1 fatcat:4a3uju47p5d5nk6o3tep6jpu5y

A unified data representation theory for network visualization, ordering and coarse-graining

István A. Kovács, Réka Mizsei, Péter Csermely
2015 Scientific Reports  
Here we show an elegant, information theoretic data representation approach as a unified solution of network visualization, data ordering and coarse-graining.  ...  Representation of large data sets became a key question of many scientific disciplines in the last decade.  ...  Acknowledgements We are grateful to the members of the LINK-group (www.linkgroup.hu) and E. Güney for useful discussions.  ... 
doi:10.1038/srep13786 pmid:26348923 pmcid:PMC4642569 fatcat:utj4ckqgl5hatleydy7dqo4aby
« Previous Showing results 1 — 15 out of 3,145 results