A Studious Approach to Semi-Supervised Learning [article]

Sahil Khose, Shruti Jain, V Manushree
<span title="2021-09-18">2021</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>
The problem of learning from few labeled examples while using large amounts of unlabeled data has been approached by various semi-supervised methods. Although these methods can achieve superior performance, the models are often not deployable due to the large number of parameters. This paper is an ablation study of distillation in a semi-supervised setting, which not just reduces the number of parameters of the model but can achieve this while improving the performance over the baseline
more &raquo; ... ed model and making it better at generalizing. After the supervised pretraining, the network is used as a teacher model, and a student network is trained over the soft labels that the teacher model generates over the entire unlabeled data. We find that the fewer the labels, the more this approach benefits from a smaller student network. This brings forward the potential of distillation as an effective solution to enhance performance in semi-supervised computer vision tasks while maintaining deployability.
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2109.08924v1">arXiv:2109.08924v1</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/qzyigrlvlrdzrnbttdrnvt4oo4">fatcat:qzyigrlvlrdzrnbttdrnvt4oo4</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20210925213131/https://arxiv.org/pdf/2109.08924v1.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/6d/7f/6d7fa1f3a05f3baaef0f05ea07d46370167f6c1e.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2109.08924v1" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>