Filters








458 Hits in 3.2 sec

Learning Discrete Distributions by Dequantization [article]

Emiel Hoogeboom, Taco S. Cohen, Jakub M. Tomczak
2020 arXiv   pre-print
Media is generally stored digitally and is therefore discrete. Many successful deep distribution models in deep learning learn a density, i.e., the distribution of a continuous random variable.  ...  In addition, we introduce autoregressive dequantization (ARD) for more flexible dequantization distributions.  ...  LG] 30 Jan 2020 Learning Discrete Distributions by Dequantization When a continuous density model is trained on discrete data using maximum likelihood, the solution may achieve arbitrarily high likelihoods  ... 
arXiv:2001.11235v1 fatcat:acmyv3fjfjbevojdoendmbiviq

Closing the Dequantization Gap: PixelCNN as a Single-Layer Flow [article]

Didrik Nielsen, Ole Winther
2020 arXiv   pre-print
Due to the continuous nature of flow models, dequantization is typically applied when using them for such discrete data, resulting in lower bound estimates of the likelihood.  ...  Flow models have recently made great progress at modeling ordinal discrete data such as images and audio.  ...  These models can be applied to for example unsupervised learning on images and audio.  ... 
arXiv:2002.02547v3 fatcat:42rjr5zmp5brpfx2ynd4ewvy6u

Semi-Discrete Normalizing Flows through Differentiable Tessellation [article]

Ricky T. Q. Chen, Brandon Amos, Maximilian Nickel
2022 arXiv   pre-print
Firstly, a Voronoi dequantization allows automatically learning quantization boundaries in a multidimensional space.  ...  Mapping between discrete and continuous distributions is a difficult task and many have had to resort to heuristical approaches.  ...  Firstly, the learned tessellation naturally allows defining a dequantization method for discrete variables.  ... 
arXiv:2203.06832v2 fatcat:fgl5pvhkrncobnttcpykdlzh4y

Audio Dequantization for High Fidelity Audio Generation in Flow-based Neural Vocoder [article]

Hyun-Wook Yoon, Sang-Hoon Lee, Hyeong-Rae Noh, Seong-Whan Lee
2020 arXiv   pre-print
However, training a continuous density model on discrete audio data can degrade model performance due to the topological difference between latent and actual distribution.  ...  The sequence of invertible flow operations allows the model to convert samples from simple distribution to audio samples.  ...  Gaussian Dequantization In flow-based neural vocoder, discrete data distribution is transformed into a spherical Gaussian distribution.  ... 
arXiv:2008.06867v1 fatcat:q4rqr63hhbektp2ps7pqbfcgty

Audio Dequantization for High Fidelity Audio Generation in Flow-Based Neural Vocoder

Hyun-Wook Yoon, Sang-Hoon Lee, Hyeong-Rae Noh, Seong-Whan Lee
2020 Interspeech 2020  
However, training a continuous density model on discrete audio data can degrade model performance due to the topological difference between latent and actual distribution.  ...  The sequence of invertible flow operations allows the model to convert samples from simple distribution to audio samples.  ...  Gaussian Dequantization In flow-based neural vocoder, discrete data distribution is transformed into a spherical Gaussian distribution.  ... 
doi:10.21437/interspeech.2020-1226 dblp:conf/interspeech/YoonLNL20 fatcat:sys53xm7djgytklxh27faccgcu

MaCow: Masked Convolutional Generative Flow [article]

Xuezhe Ma, Xiang Kong, Shanghang Zhang, Eduard Hovy
2019 arXiv   pre-print
By restricting the local connectivity in a small kernel, MaCow enjoys the properties of fast and stable training, and efficient sampling, while achieving significant improvements over Glow for density  ...  A common solution to this problem is "dequantization" that converts the discrete data distribution into a continuous one.  ...  of the dequantized variable Y under the dequantization noise distribution q(u|X).  ... 
arXiv:1902.04208v5 fatcat:u4djxn3hwjf4ljwd63j7akyrwm

Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions [article]

Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, Max Welling
2021 arXiv   pre-print
Argmax Flows are defined by a composition of a continuous distribution (such as a normalizing flow), and an argmax function.  ...  We demonstrate that our method outperforms existing dequantization approaches on text modelling and modelling on image segmentation maps in log-likelihood.  ...  Normalizing Flows typically learn a continuous distribution and dequantization is required to train these methods on ordinal data such as images.  ... 
arXiv:2102.05379v3 fatcat:yqwifulvjrcqhetyqazow5kore

Learning Likelihoods with Conditional Normalizing Flows [article]

Christina Winkler, Daniel Worrall, Emiel Hoogeboom, Max Welling
2019 arXiv   pre-print
Normalizing Flows (NFs) are able to model complicated distributions p(y) with strong inter-dimensional correlations and high multimodality by transforming a simple base density p(z) through an invertible  ...  Variational dequantization When modeling discrete data, Theis et al. (2016) introduced the concept of dequantization.  ...  In practice with deep learning models, the unknown distribution is often learned by a factored model: p(y|x) = D d=1 p(y d |x), (6) where y d represents the dth dimension of y.  ... 
arXiv:1912.00042v1 fatcat:5bpuvv5govet5ej7lr2j75qgmm

Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design [article]

Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, Pieter Abbeel
2019 arXiv   pre-print
In this paper, we investigate and improve upon three limiting design choices employed by flow-based models in prior work: the use of uniform noise for dequantization, the use of inexpressive affine flows  ...  This work was funded in part by ONR PECASE N000141612723, Huawei, Amazon AWS, and Google Cloud.  ...  UNIFORM DEQUANTIZATION Dequantization is usually performed in prior work by adding uniform noise to the discrete data over the width of each discrete bin: if each of the D components of the discrete data  ... 
arXiv:1902.00275v2 fatcat:hykcghhwcfhuxcecj6njl5jkde

Differentially Private Normalizing Flows for Synthetic Tabular Data Generation

Jaewoo Lee, Minjung Kim, Yonghyun Jeong, Youngmin Ro
2022 PROCEEDINGS OF THE THIRTIETH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE AND THE TWENTY-EIGHTH INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE CONFERENCE  
Acknowledgements This research was supported by the National Science Foundation under Grant No. 1943046.  ...  The dequantized (continuous) data is obtained by adding real-valued noise to the discrete data: y (c) = x (c) + u.  ...  When training a continuous density model, such as normalizing flows, on discrete data using MLE, the model may end up learning a degenerate distribution in which arbitrarily high likelihoods are assigned  ... 
doi:10.1609/aaai.v36i7.20697 fatcat:emuygkwakzdn3kvb7zftuhhbzy

Event Generation and Density Estimation with Surjective Normalizing Flows [article]

Rob Verheyen
2022 arXiv   pre-print
framework of Nielsen et al. (2020), we introduce several surjective and stochastic transform layers to a baseline normalizing flow to improve modelling of permutation symmetry, varying dimensionality and discrete  ...  A likely explanation is that the mixture models benefit from the fact that the marginalized discrete distributions are correct by construction, while all other models need to learn them implicitly.  ...  However, scattering events are often not only characterized by their energy-momentum distributions, but also by a variety of discrete features which are related to the quantum numbers of the particles  ... 
arXiv:2205.01697v2 fatcat:eofojkgoj5dnvd3lzci6s7bnje

Manifold Density Estimation via Generalized Dequantization [article]

James A. Brofos, Marcus A. Brubaker, Roy R. Lederman
2021 arXiv   pre-print
We propose a method, inspired by the literature on "dequantization," which we interpret through the lens of a coordinate transformation of an ambient Euclidean space and a smooth manifold of interest.  ...  Density estimation is an important technique for characterizing distributions given observations.  ...  We use algorithm 1 to learn the parameters of the ambient and dequantization distributions.  ... 
arXiv:2102.07143v2 fatcat:6ddmnouquncajhijwuyjidkbdy

RBM-Flow and D-Flow: Invertible Flows with Discrete Energy Base Spaces [article]

Daniel O'Connor, Walter Vinci
2021 arXiv   pre-print
Efficient sampling of complex data distributions can be achieved using trained invertible flows (IF), where the model distribution is generated by pushing a simple base distribution through multiple non-linear  ...  Furthermore, we also obtain D-Flow, an IF model with uncorrelated discrete latent variables.  ...  Government is authorized to reproduce and distribute reprints for Governmental purpose notwithstanding any copyright annotation thereon.  ... 
arXiv:2012.13196v3 fatcat:4losmcinsfcb3kbsbw2autkruy

Bitwidth Heterogeneous Federated Learning with Progressive Weight Dequantization [article]

Jaehong Yoon, Geon Park, Wonyong Jeong, Sung Ju Hwang
2022 arXiv   pre-print
In practical federated learning scenarios, the participating devices may have different bitwidths for computation and memory storage by design.  ...  To tackle this problem, we propose ProWD framework, which has a trainable weight dequantizer at the central server that progressively reconstructs the low-bitwidth weights into higher bitwidth weights,  ...  In generative flow models, Nielsen & Winther (2020) dequantizes the discrete-valued data by adding a uniform noise to guarantee that the data is able to have any value in the continuous domain.  ... 
arXiv:2202.11453v4 fatcat:5xygzbuw4nainbin3gq7mj4fcy

Categorical Normalizing Flows via Continuous Transformations [article]

Phillip Lippe, Efstratios Gavves
2021 arXiv   pre-print
The current practice of using dequantization to map discrete data to a continuous space is inapplicable as categorical data has no intrinsic order.  ...  By casting the encoding of categorical data in continuous space as a variational inference problem, we jointly optimize the continuous representation and the model likelihood.  ...  In contrast to dequantization, the continuous encoding z is not bounded by the domain of the encoding distribution. Instead, the partitioning is jointly learned with the model likelihood.  ... 
arXiv:2006.09790v3 fatcat:dwecz4ikjzhppirfzay57xy2cq
« Previous Showing results 1 — 15 out of 458 results