Do ImageNet Classifiers Generalize to ImageNet? [article]

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, Vaishaal Shankar
<span title="2019-06-12">2019</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>
We build new test sets for the CIFAR-10 and ImageNet datasets. Both benchmarks have been the focus of intense research for almost a decade, raising the danger of overfitting to excessively re-used test sets. By closely following the original dataset creation processes, we test to what extent current classification models generalize to new data. We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet. However, accuracy gains on the original
more &raquo; ... st sets translate to larger gains on the new test sets. Our results suggest that the accuracy drops are not caused by adaptivity, but by the models' inability to generalize to slightly "harder" images than those found in the original test sets.
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1902.10811v2">arXiv:1902.10811v2</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/d32xzqv56fbxfh26omlfy6g4bi">fatcat:d32xzqv56fbxfh26omlfy6g4bi</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20200912085642/https://arxiv.org/pdf/1902.10811v1.pdf" title="fulltext PDF download [not primary version]" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <span style="color: #f43e3e;">&#10033;</span> <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/5c/1d/5c1dd1967e30f5e07653031732bc037f0e4b5a46.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1902.10811v2" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>