A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2021; you can also visit <a rel="external noopener" href="https://arxiv.org/pdf/2106.15941v1.pdf">the original URL</a>. The file type is <code>application/pdf</code>.
Augmented Shortcuts for Vision Transformers
[article]
<span title="2021-06-30">2021</span>
<i >
arXiv
</i>
<span class="release-stage" >pre-print</span>
Preserved Fulltext
Transformer models have achieved great progress on computer vision tasks recently. The rapid development of vision transformers is mainly contributed by their high representation ability for extracting informative features from input images. However, the mainstream transformer models are designed with deep architectures, and the feature diversity will be continuously reduced as the depth increases, i.e., feature collapse. In this paper, we theoretically analyze the feature collapse phenomenon
<span class="external-identifiers">
<a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2106.15941v1">arXiv:2106.15941v1</a>
<a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/de5jbvo3izfexblz7m6nsjxqnq">fatcat:de5jbvo3izfexblz7m6nsjxqnq</a>
</span>
more »
... d study the relationship between shortcuts and feature diversity in these transformer models. Then, we present an augmented shortcut scheme, which inserts additional paths with learnable parameters in parallel on the original shortcuts. To save the computational costs, we further explore an efficient approach that uses the block-circulant projection to implement augmented shortcuts. Extensive experiments conducted on benchmark datasets demonstrate the effectiveness of the proposed method, which brings about 1% accuracy increase of the state-of-the-art visual transformers without obviously increasing their parameters and FLOPs.
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20210702013642/https://arxiv.org/pdf/2106.15941v1.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext">
<button class="ui simple right pointing dropdown compact black labeled icon button serp-button">
<i class="icon ia-icon"></i>
Web Archive
[PDF]
<div class="menu fulltext-thumbnail">
<img src="https://blobs.fatcat.wiki/thumbnail/pdf/bb/6e/bb6eba6d1c7f09255a9649eb2ed55bbd4f048091.180px.jpg" alt="fulltext thumbnail" loading="lazy">
</div>
</button>
</a>
<a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2106.15941v1" title="arxiv.org access">
<button class="ui compact blue labeled icon button serp-button">
<i class="file alternate outline icon"></i>
arxiv.org
</button>
</a>