Filters








11,815 Hits in 4.3 sec

Upper Bound on Normalized Maximum Likelihood Codes for Gaussian Mixture Models [article]

So Hirai, Kenji Yamanishi
2018 arXiv   pre-print
This paper shows that the normalized maximum likelihood~(NML) code-length calculated in [1] is an upper bound on the NML code-length strictly calculated for the Gaussian Mixture Model.  ...  When we use this upper bound on the NML code-length, we must change the scale of the data sequence to satisfy the restricted domain.  ...  Here, we use the Gaussian Model Class N (µ, Σ), µ ∈ R m , Σ ∈ R m×m , and calculate the normalized maximum likelihood (NML) code-length for the Gaussian Model.  ... 
arXiv:1709.00925v2 fatcat:u6fdh6v5rbdzdigthflcnqrlka

Rate Distortion Behavior of Sparse Sources

Claudio Weidmann, Martin Vetterli
2012 IEEE Transactions on Information Theory  
The former lead to low-and high-rate upper bounds on mean squared error , while the latter yields lower and upper bounds on source entropy, thereby characterizing asymptotic behavior.  ...  These bounding techniques are applied to two source models: Gaussian mixtures and power laws matching the approximately scale-invariant decay of wavelet coefficients.  ...  Telatar for helpful discussions, as well as the anonymous reviewers and the associate editor for their valuable suggestions and comments, which greatly helped improving the presentation of this material  ... 
doi:10.1109/tit.2012.2201335 fatcat:i7x3mjprujgdvhtb6o4bx6a4q4

Technical report: Training Mixture Density Networks with full covariance matrices [article]

Jakob Kruse
2020 arXiv   pre-print
In the standard formulation, an MDN takes some input and outputs parameters for a Gaussian mixture model with restrictions on the mixture components' covariance.  ...  Mixture Density Networks are a tried and tested tool for modelling conditional probability distributions. As such, they constitute a great baseline for novel approaches to this problem.  ...  Then the latter network can be trained via back-propagation through the GMM block, using the negative log-likelihood loss function which is also supplied.  ... 
arXiv:2003.05739v1 fatcat:wdylwz7flrbefbavxbigearvz4

Bit error rate estimation for turbo decoding

N. Letzepis, A. Grant
2003 IEEE International Symposium on Information Theory, 2003. Proceedings.  
We model the log-likelihood ratios as a mixture of two Gaussian random variables and derive estimators for the mean and variance of these distributions, which can be used to estimate BER.  ...  We propose a method for on-line estimation of Bit Error Rate during turbo decoding.  ...  We have derived estimators for the parameters of the LLR distribution by modelling LLRs as a Gaussian mixture.  ... 
doi:10.1109/isit.2003.1228454 fatcat:26uqgm5r5zam3jbus6rorlwua4

Bit error rate estimation for turbo decoding

N. Letzepis, A. Grant
2009 IEEE Transactions on Communications  
We model the log-likelihood ratios as a mixture of two Gaussian random variables and derive estimators for the mean and variance of these distributions, which can be used to estimate BER.  ...  We propose a method for on-line estimation of Bit Error Rate during turbo decoding.  ...  We have derived estimators for the parameters of the LLR distribution by modelling LLRs as a Gaussian mixture.  ... 
doi:10.1109/tcomm.2009.03.060410 fatcat:q5ye7ehgpfcd3oo74v6kvo7yn4

The minimum description length principle in coding and modeling

A. Barron, J. Rissanen, Bin Yu
1998 IEEE Transactions on Information Theory  
The normalized maximized likelihood, mixture, and predictive codings are each shown to achieve the stochastic complexity to within asymptotically vanishing terms.  ...  Context tree modeling, density estimation, and model selection in Gaussian linear regression serve as examples.  ...  the desired asymptotic equivalence of the Jeffreys mixture and normalized maximum likelihood.  ... 
doi:10.1109/18.720554 fatcat:s7ylg53uvzhufabbucrzq4m26q

The Minimum Description Length Principle in Coding and Modeling [chapter]

2009 Information Theory  
The normalized maximized likelihood, mixture, and predictive codings are each shown to achieve the stochastic complexity to within asymptotically vanishing terms.  ...  Context tree modeling, density estimation, and model selection in Gaussian linear regression serve as examples.  ...  the desired asymptotic equivalence of the Jeffreys mixture and normalized maximum likelihood.  ... 
doi:10.1109/9780470544907.ch25 fatcat:b5bnutcdg5cyvetrcohul7omvm

Complexity of simple nonlogarithmic loss functions

J. Rissanen
2003 IEEE Transactions on Information Theory  
Index Terms--loss functions, complexity, maximum entropy, min-max bounds, prediction bound.  ...  The loss complexity for nonlogarithmic loss functions is defined analogously to the stochastic complexity for logarithmic loss functions such that its mean provides an achievable lower bound for estimation  ...  If the integral is finite, we can define the normalized maximum-likelihood (NML) model, [2] , [10] (9) where (11) We derive next a few important properties of the models in the class for a simple  ... 
doi:10.1109/tit.2002.807281 fatcat:uv5p7pk2njarzbywzp2t4bhlha

A Tutorial on VAEs: From Bayes' Rule to Lossless Compression [article]

Ronald Yu
2020 arXiv   pre-print
The Variational Auto-Encoder (VAE) is a simple, efficient, and popular deep maximum likelihood model. Though usage of VAEs is widespread, the derivation of the VAE is not as widely understood.  ...  Finally, we will visualize the capabilities and limitations of VAEs using a code example (with an accompanying Jupyter notebook) on toy 2D data.  ...  VLAE [7] shows that the negative ELBO −L(x) is an upper bound on this number by constructing a code to describe p gt (x) that on average uses −L(x) bits.  ... 
arXiv:2006.10273v2 fatcat:qevuyc6vrrhg7lcvlkkerowixi

PARAFAC-Based Blind Identification of Underdetermined Mixtures Using Gaussian Mixture Model

Fanglin Gu, Hang Zhang, Wenwu Wang, Desheng Zhu
2013 Circuits, systems, and signal processing  
The GMM-PARAFAC algorithm uses Gaussian mixture model (GMM) to model non-Gaussianity of the independent sources.  ...  In order to reduce the computation complexity, the mixing matrix is estimated by maximizing a tight upper bound of the likelihood instead of the log-likelihood itself.  ...  Acknowledgements The authors would like to thank Lieven De Lathauwer for sharing with us the Matlab codes of the FOOBI algorithm.  ... 
doi:10.1007/s00034-013-9719-8 fatcat:k443mbfxafaofd6ckyohbt52nq

Density-Preserving Hierarchical EM Algorithm: Simplifying Gaussian Mixture Models for Approximate Inference

Lei Yu, Tianyu Yang, Antoni B. Chan
2018 IEEE Transactions on Pattern Analysis and Machine Intelligence  
We propose an algorithm for simplifying a finite mixture model into a reduced mixture model with fewer mixture components.  ...  For recursive Bayesian filtering, we propose an efficient algorithm for approximating an arbitrary likelihood function as a sum of scaled Gaussian.  ...  and codes in [66] ; K Zhang, L Zhang and MH Yang for the codes in [64] ; MA Brubaker, A Geiger and R Urtasun for the dataset and codes for selflocalization in [21] .  ... 
doi:10.1109/tpami.2018.2845371 pmid:29994194 fatcat:5qm3jc4j7fesrg7j4wjixvtqne

Learning deep kernels for exponential family densities [article]

Li Wenliang, Danica J. Sutherland, Heiko Strathmann, Arthur Gretton
2021 arXiv   pre-print
Compared to deep density models fit via maximum likelihood, our approach provides a complementary set of strengths and tradeoffs: in empirical studies, the former can yield higher likelihoods, whereas  ...  This gives a very rich class of density models, capable of fitting complex structures on moderate-dimensional problems.  ...  For each model fit on each distribution, we report the normalized log likelihood (left) and Fisher divergence (right).  ... 
arXiv:1811.08357v4 fatcat:f66ynommebdwdc7zradtnewfiu

Independent Component Analysis by Entropy Bound Minimization

Xi-Lin Li, Tülay Adali
2010 IEEE Transactions on Signal Processing  
We show that such an estimator exists for a wide class of measuring functions, and provide a number of design examples to demonstrate its flexible nature.  ...  A novel (differential) entropy estimator is introduced where the maximum entropy bound is used to approximate the entropy given the observations, and is computed using a numerical procedure thus resulting  ...  to the upper bound of , i.e., for , we have By solving the above inequality, we obtain On the other hand, there is an such that for , the upper bound of is no greater than the lower bound of , i.e.  ... 
doi:10.1109/tsp.2010.2055859 fatcat:4rg3si2k7jb6dm3djtfblwmaoe

Iterated logarithmic expansions of the pathwise code lengths for exponential families

Bin Yu, Lei Li
2000 IEEE Transactions on Information Theory  
For exponential families we obtain pathwise expansions, to the constant order, of the predictive and mixture code lengths used in MDL. The results are useful for understanding di erent MDL forms.  ...  Rissanen's Minimum Description Length (MDL) principle is a statistical modeling principle motivated by coding theory.  ...  This modi ed two-stage code is called Normalized Maximum Likelihood (NML) code.  ... 
doi:10.1109/18.887882 fatcat:saxo7ntikbhf7apafzxyqqizqa

Likelihood Inference in the Errors-in-Variables Model

S.A. Murphy, A.W. Van Der Vaart
1996 Journal of Multivariate Analysis  
We consider estimation and con dence regions for the parameters and based on the observations X 1 ; Y 1 ; : : : ; X n ; Y n in the errors-in-variables model X i = Z i + e i and Y i = + Z i +f i for normal  ...  Similarly we study the con dence regions obtained from the likelihood ratio statistic for the mixture model and show that these are asymptotically consistent both in the mixture case and in the case that  ...  Theorem 1.1 and its proof shows that the latter is true for the maximum likelihood estimator for the mixture model.  ... 
doi:10.1006/jmva.1996.0055 fatcat:vbm72woeqzg7nlgrdwsgxc56l4
« Previous Showing results 1 — 15 out of 11,815 results