100 Hits in 3.9 sec

Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization [article]

Qing Guo, Junya Chen, Dong Wang, Yuewei Yang, Xinwei Deng, Lawrence Carin, Fan Li, Chenyang Tao
2022 arXiv   pre-print
Successful applications of InfoNCE and its variants have popularized the use of contrastive variational mutual information (MI) estimators in machine learning.  ...  Theoretically, we show that the FLO estimator is tight, and it provably converges under stochastic gradient descent.  ...  Fenchel-Legendre Optimization for mutual information estimation This section presents the main result of this paper.  ... 
arXiv:2107.01131v2 fatcat:o5ynrffksrdjxddo4dolnbn3oe

Learning with Fenchel-Young Losses [article]

Mathieu Blondel, André F. T. Martins, Vlad Niculae
2020 arXiv   pre-print
We show that Fenchel-Young losses unify many well-known loss functions and allow to create useful new ones easily.  ...  Finally, we derive efficient predictive and training algorithms, making Fenchel-Young losses appealing both in theory and practice.  ...  H, of which mutual information is a special case.  ... 
arXiv:1901.02324v2 fatcat:ejo24qz6y5dqdmoupqulrlhyzy

Compensated Convex Based Transforms for Image Processing and Shape Interrogation [article]

Antonio Orlando, Elaine Crooks, Kewei Zhang
2020 arXiv   pre-print
This paper reviews some recent applications of the theory of the compensated convex transforms or of the proximity hull as developed by the authors to image processing and shape interrogation with special  ...  the discrete Legendre-Fenchel transform of a function.  ...  However, crucially they are not 'tight approximations' of f , in contrast with our lower and upper transforms C l λ (f )(x) and C u λ (f )(x) [116] .  ... 
arXiv:2010.04720v1 fatcat:yuved5tfb5dkfisgtxorubsgjy

Asymptotic Results for Decentralized Detection in Power Constrained Wireless Sensor Networks

J.-F. Chamberland, V.V. Veeravalli
2004 IEEE Journal on Selected Areas in Communications  
We supplement the theory with examples to illustrate how the results derived in this paper apply to the design of practical sensing systems.  ...  Large deviation theory is used to show that having identical sensor nodes, i.e., each node using the same transmission scheme, is asymptotically optimal.  ...  The large deviations associated with the empirical mean of independent and identically distributed random variables is characterized by the Fenchel-Legendre transform of Λ 0 (λ), which is defined as Λ  ... 
doi:10.1109/jsac.2004.830894 fatcat:nb3zkm54wbdnzio246ay37ioye

Signal processing for application-specific ad hoc networks

Youngchul Sung, S. Misra, Lang Tong, A. Ephremides
2006 IEEE Signal Processing Magazine  
In contrast to the design of the Internet, serving individual nodes is not always the ultimate objective.  ...  If an application-specific metric is to be optimized, signal processing may have a role in defining network architectures  ...  The bound that we use to derive optimized routing is tight as the number of nodes in the route increases.  ... 
doi:10.1109/msp.2006.1708414 fatcat:w25j2imdrzai7bbdjczjrxgxni

Bottleneck Problems: An Information and Estimation-Theoretic View

Shahab Asoodeh, Flavio P. Calmon
2020 Entropy  
In the second part, we introduce a general family of optimization problems, termed "bottleneck problems", by replacing mutual information in IB and PF with other notions of mutual information, namely f-information  ...  and Arimoto's mutual information.  ...  This is in contrast with mutual information for which the similar upper bound scales as O(log n/ √ n) as shown in [33] .  ... 
doi:10.3390/e22111325 pmid:33287090 pmcid:PMC7712227 fatcat:c4jyies3hncpdaipvayqkzlxma

Moreau-Yosida f-divergences [article]

Dávid Terjék
2021 arXiv   pre-print
On the practical side, we propose an algorithm to calculate the tight convex conjugate of f-divergences compatible with automatic differentiation frameworks.  ...  Jensen-Shannon, Jeffreys, triangular discrimination and total variation divergences as GANs trained on CIFAR-10, leading to competitive results and a simple solution to the problem of uniqueness of the optimal  ...  ., 2016) , mutual information estimation (Belghazi et al., 2018) and maximization (Hjelm et al., 2019) , and energybased models (Arbel et al., 2021) .  ... 
arXiv:2102.13416v2 fatcat:qn3y27yxirdihbeulitinlbkfe

Bottleneck Problems: Information and Estimation-Theoretic View [article]

Shahab Asoodeh, Flavio Calmon
2020 arXiv   pre-print
In the second part, we introduce a general family of optimization problems, termed as bottleneck problems, by replacing mutual information in IB and PF with other notions of mutual information, namely  ...  f-information and Arimoto's mutual information.  ...  This is in contrast with mutual information for which the similar upper bound scales as O(log n/ √ n) as shown in [31] .  ... 
arXiv:2011.06208v1 fatcat:okqpkfgmkba3tgf54tsl37bkde

Outage Exponent: A Unified Performance Metric for Parallel Fading Channels

Bo Bai, Wei Chen, Khaled B. Letaief, Zhigang Cao
2013 IEEE Transactions on Information Theory  
It is shown that the proposed outage exponent is not only an accurate estimation of the outage probability for any number of subchannels, any SNR, and any target transmission rate, but also provides an  ...  The Legendre-Fenchel transform is then defined as Λ * n (s) = sup ξ∈R {ξs − Λ n (ξ)} . Define the Legendre duality as Ξ (s) = arg sup ξ∈R {ξs − Λ n (ξ)} .  ...  Hence, the mutual information between the transmitter and the receiver, denoted by I (H), is then given by I (H) = L l=1 ln 1 + |h l | 2 γ , where γ is the received SNR.  ... 
doi:10.1109/tit.2012.2227454 fatcat:5jupwcr43velzesr63gjjahmam

Large Deviations Performance of Consensus+Innovations Distributed Detection With Non-Gaussian Observations

D. Bajovic, D. Jakovetic, J. M. F. Moura, J. Xavier, B. Sinopoli
2012 IEEE Transactions on Signal Processing  
Above a threshold, distributed is as good as centralized, with the same optimal asymptotic detection performance, but, below the threshold, distributed detection is suboptimal with respect to centralized  ...  At each time instant, sensors 1) combine theirs with the decision variables of their neighbors (consensus) and 2) assimilate their new observations (innovations).  ...  He has published more than 450 technical Journal and Conference papers, filed 11 patents with eight issued by the U.S. Patent Office.  ... 
doi:10.1109/tsp.2012.2210885 fatcat:et5anvl2ujc7lokru4diombyjy

Sample-Path Large Deviations in Credit Risk

V. J. G. Leijdekker, M. R. H. Mandjes, P. J. C. Spreij
2011 Journal of Applied Mathematics  
The Fenchel-Legendre transform Λ X of Λ X is then defined by Λ X x : sup θ θx − Λ X θ . 2.7 We sometimes say that Λ X is the Fenchel-Legendre transform of X.  ...  sup θ θ Δx i ϕ i − Λ U θ . 3.50 The rate functions 3.7 and 3.8 are of comparable complexity, as both correspond to an N-dimensional optimization where 3.8 also involves the evaluation of the Fenchel-Legendre  ...  Define Then the sequence of laws {μ n • p −1 J } satisfies the LDP in R d |j| with the good rate function where Λ is the Fenchel-Legendre transform of X 1 . Proof.  ... 
doi:10.1155/2011/354171 fatcat:fd3rjweqc5ay7klf3tigrau3qe

Generalization Error Bounds Via Rényi-, f-Divergences and Maximal Leakage [article]

Amedeo Roberto Esposito, Michael Gastpar, Ibrahim Issa
2020 arXiv   pre-print
Bounds are given in terms of Sibson's Mutual Information, α-Divergences, Hellinger Divergences, and f-Divergences.  ...  A case of particular interest is the Maximal Leakage (or Sibson's Mutual Information of order infinity), since this measure is robust to post-processing and composes adaptively.  ...  respect to y), and ϕ ⋆ and ψ ⋆ are, respectively, the Legendre-Fenchel duals of ϕ and ψ.  ... 
arXiv:1912.01439v3 fatcat:kwmqnsaqbrbpbbnwndbg6xy6b4

Multi-User Guesswork and Brute Force Security

Mark M. Christiansen, Ken R. Duffy, Flavio du Pin Calmon, Muriel Medard
2015 IEEE Transactions on Information Theory  
class of strategies; the number of guesses required in an asymptotically optimal strategy satisfies a large deviation principle with a rate function, which is not necessarily convex, that can be determined  ...  The inquisitor knows the selection probabilities of each user and is equipped with a method that enables the testing of each (user, string) pair, one at a time, for whether that string had been selected  ...  ACKNOWLEDGMENTS: The authors thank Erdal Arikan (Bilkent University) for informative feedback and for pointing out the relationship between multi-user guesswork and sequential decoding.  ... 
doi:10.1109/tit.2015.2482972 fatcat:gh7mfxlhrbdhhntvbm5vtxvjaa

Survey of deterministic and stochastic service curve models in the network calculus

Markus Fidler
2010 IEEE Communications Surveys and Tutorials  
We reported recent measurement techniques for service curve estimation and identification.  ...  Moreover, the measurement-based approach can provide service curve estimates for systems that are not solved today.  ...  In the min-plus systems theory convex and concave Fenchel conjugates, also referred to as Legendre transform, are known to take the place of the Fourier transform.  ... 
doi:10.1109/surv.2010.020110.00019 fatcat:sjvumzp3erhvrnsvefbd6nwjoi

Support Recovery With Sparsely Sampled Free Random Matrices

Antonia M. Tulino, Giuseppe Caire, Sergio Verdu, Shlomo Shamai
2013 IEEE Transactions on Information Theory  
MUTUAL INFORMATION RATE In this section we are concerned with the mutual information rate and the right-most equality in (7) A.  ...  Using Cramér's theorem, we have that the rate function I (u) (Q; h) for the measure (301) is given by the Legendre-Fenchel transform µ (u) n (dQ; h) = b0,...  ... 
doi:10.1109/tit.2013.2250578 fatcat:yiq2sh7sz5fehlt35pnrzkzgq4
« Previous Showing results 1 — 15 out of 100 results