Filters








44,692 Hits in 4.0 sec

On Biased Compression for Distributed Learning [article]

Aleksandr Beznosikov and Samuel Horváth and Peter Richtárik and Mher Safaryan
2021 arXiv   pre-print
We show for the first time that biased compressors can lead to linear convergence rates both in the single node and distributed settings.  ...  In the last few years, various communication compression techniques have emerged as an indispensable tool helping to alleviate the communication bottleneck in distributed learning.  ...  On Biased Compression for Distributed Learning Aleksandr Beznosikov∗ Samuel Horváth  ... 
arXiv:2002.12410v2 fatcat:n45gzaj5z5fy5mbwa5ux66mxku

Using Natural Language and Program Abstractions to Instill Human Inductive Biases in Machines [article]

Sreejan Kumar, Carlos G. Correa, Ishita Dasgupta, Raja Marjieh, Michael Y. Hu, Robert D. Hawkins, Nathaniel D. Daw, Jonathan D. Cohen, Karthik Narasimhan, Thomas L. Griffiths
2022 arXiv   pre-print
Although meta-learning has emerged as an approach for endowing neural networks with useful inductive biases, agents trained by meta-learning may acquire very different strategies from humans.  ...  We show that co-training these agents on predicting representations from natural language task descriptions and from programs induced to generate such tasks guides them toward human-like inductive biases  ...  By analogy, library-learned concepts allow for compression of programs (Fig. 4B ).  ... 
arXiv:2205.11558v1 fatcat:wqa4h3ogabbhpn4y26gcfaqdxq

Semantic Compression of Episodic Memories [article]

David G. Nagy, Balázs Török, Gergő Orbán
2018 arXiv   pre-print
We formalise the compression of episodes in the normative framework of information theory and argue that semantic memory provides the distortion function for compression of experiences.  ...  Recent advances and insights from machine learning allow us to approximate semantic compression in naturalistic domains and contrast the resulting deviations in compressed episodes with memory errors observed  ...  Acknowledgements The authors thank the anonymous reviewers for useful comments and Ferenc Huszár for discussions.  ... 
arXiv:1806.07990v1 fatcat:3xuv6pk3dbh57gfsoqeqpvv56q

CosSGD: Communication-Efficient Federated Learning with a Simple Cosine-Based Quantization [article]

Yang He and Hui-Po Wang and Maximilian Zenk and Mario Fritz
2022 arXiv   pre-print
Further, our approach is highly suitable for federated learning problems since it has low computational complexity and requires only a little additional data to recover the compressed information.  ...  Federated learning is a promising framework to mitigate data privacy and computation concerns.  ...  The biased linear quantization has an error bound of bv 2 s −1 = cos(b θ ) 2 s −1 • v 2 for all the gradients on [−b v , b v ].  ... 
arXiv:2012.08241v2 fatcat:pes6bfrkxveohcouipxa4uwdga

Debiasing Methods in Natural Language Understanding Make Bias More Accessible [article]

Michael Mendelson, Yonatan Belinkov
2021 arXiv   pre-print
Model robustness to bias is often determined by the generalization on carefully designed out-of-distribution datasets.  ...  We propose a general probing-based framework that allows for post-hoc interpretation of biases in language models, and use an information-theoretic approach to measure the extractability of certain biases  ...  We thank Victor Sanh and Prasetya Ajie Utama for their feedback and help reproducing the models. We also thank the anonymous reviewers for their useful suggestions.  ... 
arXiv:2109.04095v1 fatcat:lrt56s2e55gzrjacb3plloiqei

Robust Simulation-Based Inference in Cosmology with Bayesian Neural Networks [article]

Pablo Lemos, Miles Cranmer, Muntazir Abidi, ChangHoon Hahn, Michael Eickenberg, Elena Massara, David Yallup, Shirley Ho
2022 arXiv   pre-print
Simulation-based inference (SBI) is rapidly establishing itself as a standard machine learning technique for analyzing data in cosmological surveys.  ...  outside the training distribution, which is mostly unconstrained.  ...  The reason for this hyperparameter is that the covariance matrix estimated by SWA will depend on the learning rate.  ... 
arXiv:2207.08435v2 fatcat:a4dk3wqzhrgdriwaialxtgn43a

Generalization of prior information for rapid Bayesian time estimation

Neil W. Roach, Paul V. McGraw, David J. Whitaker, James Heron
2016 Proceedings of the National Academy of Sciences of the United States of America  
For example, a variety of biases in visual perception have been shown to be consistent with reliance on prior knowledge regarding statistical regularities in the environment, such as the distribution of  ...  Similarly, Gekas et al. (24) showed that distinct priors for motion direction can be learned for sets of dot stimuli of different colors.  ...  We thank Thomas Veale for assisting with data collection. This work was supported by Wellcome Trust Research Fellowship WT097387 (to N.W.R.).  ... 
doi:10.1073/pnas.1610706114 pmid:28007982 pmcid:PMC5240697 fatcat:egonw364j5autac2whrm4pi7ii

De-biasing facial detection system using VAE [article]

Vedant V. Kandge, Siddhant V. Kandge, Kajal Kumbharkar, Prof. Tanuja Pattanshetti
2022 arXiv   pre-print
The proposed approach uses generative models which are best suited for learning underlying features(latent variables) from the dataset and by using these learned features models try to reduce the threats  ...  And then we train models on two datasets and compare the results.  ...  Ultimately gives an algorithm for debiasing against learned biases based on the unsupervised latent space.  ... 
arXiv:2204.09556v1 fatcat:oh4t7ianwvdixjyktw2sdk754e

DESP: Deep Enhanced Sampling of Proteins' Conformation Spaces Using AI-Inspired Biasing Forces

Emmanuel Oluwatobi Salawu
2021 Frontiers in Molecular Biosciences  
and deep neural networks (DNNs), in which biasing potentials for guiding the MD simulations are derived from the KL divergence between the DNN-learned latent space vectors of [a] the most recently sampled  ...  In this work, we present Deep Enhanced Sampling of Proteins' Conformation Spaces Using AI-Inspired Biasing Forces (DESP), a technique for enhanced sampling that combines molecular dynamics (MD) simulations  ...  on the distribution of the molecule's conformations in that compressed representation space.  ... 
doi:10.3389/fmolb.2021.587151 pmid:34026817 pmcid:PMC8132871 fatcat:4bmn2eyeyrdhrehzvvuxoutipm

FEDZIP: A Compression Framework for Communication-Efficient Federated Learning [article]

Amirhossein Malekijoo, Mohammad Javad Fadaeieslam, Hanieh Malekijou, Morteza Homayounfar, Farshid Alizadeh-Shabdiz, Reza Rawassizadeh
2021 arXiv   pre-print
It assigns the learning process independently to each client. First, clients locally train a machine learning model based on local data.  ...  FedZip outperforms state-of-the-art compression frameworks and reaches compression rates up to 1085x, and preserves up to 99% of bandwidth and 99% of energy for clients during communication.  ...  Table 1 : 1 Comparison of state-of-the-art distributed learning methods with compression and FedZip.  ... 
arXiv:2102.01593v1 fatcat:24gyfd4etzbwxjm2psqry4mace

Language as a Latent Variable: Discrete Generative Models for Sentence Compression [article]

Yishu Miao, Phil Blunsom
2016 arXiv   pre-print
We formulate a variational auto-encoder for inference in this model and apply it to the task of compressing sentences.  ...  In our empirical evaluation we show that generative formulations of both abstractive and extractive compression yield state-of-the-art results when trained on a large amount of supervised data.  ...  Although we introduce a biased estimator by using pointer networks, it is still very difficult for the compression model to generate reasonable natural language sentences at the early stage of learning  ... 
arXiv:1609.07317v2 fatcat:fppddfdenfgkhjaadassg4dxcy

Language as a Latent Variable: Discrete Generative Models for Sentence Compression

Yishu Miao, Phil Blunsom
2016 Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing  
We formulate a variational auto-encoder for inference in this model and apply it to the task of compressing sentences.  ...  In our empirical evaluation we show that generative formulations of both abstractive and extractive compression yield state-of-the-art results when trained on a large amount of supervised data.  ...  Although we introduce a biased estimator by using pointer networks, it is still very difficult for the compression model to generate reasonable natural language sentences at the early stage of learning  ... 
doi:10.18653/v1/d16-1031 dblp:conf/emnlp/MiaoB16 fatcat:hdvrqfdv6rhddg3cniw5e7ft7m

Meta-Learning Sparse Compression Networks [article]

Jonathan Richard Schwarz, Yee Whye Teh
2022 arXiv   pre-print
Secondly, introduce the first method allowing for sparsification to be employed in the inner-loop of commonly used Meta-Learning algorithms, drastically improving both compression and the computational  ...  Recent work on such Implicit Neural Representations (INRs) has shown that - following careful architecture search - INRs can outperform established compression methods such as JPEG (e.g.  ...  Their cooperation sets a high standard for scientific collaboration and should be applauded.  ... 
arXiv:2205.08957v2 fatcat:zgioj3pwijgztjr3c577cklucy

Autoencoders, Kernels, and Multilayer Perceptrons for Electron Micrograph Restoration and Compression [article]

Jeffrey M. Ede
2018 arXiv   pre-print
TEM autoencoders have been trained for 1×, 4×, 16× and 64× compression, STEM autoencoders for 1×, 4× and 16× compression and TEM+STEM autoencoders for 1×, 2×, 4×, 8×, 16×, 32× and 64× compression.  ...  We present 14 autoencoders, 15 kernels and 14 multilayer perceptrons for electron micrograph restoration and compression.  ...  Minima subtraction was applied to increase the similarity of input intensity distributions, making it easier for the autoencoders to learn.  ... 
arXiv:1808.09916v1 fatcat:canyqapil5aonkdvedth6ia2zu

Trends and Advancements in Deep Neural Network Communication [article]

Felix Sattler, Thomas Wiegand, Wojciech Samek
2020 arXiv   pre-print
This paper gives an overview over the recent advancements and challenges in this new field of research at the intersection of machine learning and communications.  ...  To address the challenges of these distributed environments, a wide range of training and evaluation schemes have been developed, which require the communication of neural network parametrizations.  ...  An established technique to reduce the impact of biased compression on the convergence speed is error accumulation.  ... 
arXiv:2003.03320v1 fatcat:tgs7b6nbovflngyuxhxwkmxhv4
« Previous Showing results 1 — 15 out of 44,692 results