Learning Rich Nearest Neighbor Representations from Self-supervised Ensembles [article]

Bram Wallace, Devansh Arpit, Huan Wang, Caiming Xiong
<span title="2021-10-19">2021</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>
Pretraining convolutional neural networks via self-supervision, and applying them in transfer learning, is an incredibly fast-growing field that is rapidly and iteratively improving performance across practically all image domains. Meanwhile, model ensembling is one of the most universally applicable techniques in supervised learning literature and practice, offering a simple solution to reliably improve performance. But how to optimally combine self-supervised models to maximize representation
more &raquo; ... quality has largely remained unaddressed. In this work, we provide a framework to perform self-supervised model ensembling via a novel method of learning representations directly through gradient descent at inference time. This technique improves representation quality, as measured by k-nearest neighbors, both on the in-domain dataset and in the transfer setting, with models transferable from the former setting to the latter. Additionally, this direct learning of feature through backpropagation improves representations from even a single model, echoing the improvements found in self-distillation.
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2110.10293v1">arXiv:2110.10293v1</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/kuqlalsw6nbylnnik57eh7f4sm">fatcat:kuqlalsw6nbylnnik57eh7f4sm</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20211025153937/https://arxiv.org/pdf/2110.10293v1.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/65/ef/65eff4b45a8a49b75e584fe4d54feea4e7c43d81.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2110.10293v1" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>