A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit <a rel="external noopener" href="https://arxiv.org/pdf/2003.00138v1.pdf">the original URL</a>. The file type is <code>application/pdf</code>.
A Note on Latency Variability of Deep Neural Networks for Mobile Inference
[article]
<span title="2020-02-29">2020</span>
<i >
arXiv
</i>
<span class="release-stage" >pre-print</span>
Preserved Fulltext
Running deep neural network (DNN) inference on mobile devices, i.e., mobile inference, has become a growing trend, making inference less dependent on network connections and keeping private data locally. The prior studies on optimizing DNNs for mobile inference typically focus on the metric of average inference latency, thus implicitly assuming that mobile inference exhibits little latency variability. In this note, we conduct a preliminary measurement study on the latency variability of DNNs
<span class="external-identifiers">
<a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2003.00138v1">arXiv:2003.00138v1</a>
<a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/va2e5zlubvf57favc2h4u7g77q">fatcat:va2e5zlubvf57favc2h4u7g77q</a>
</span>
more »
... r mobile inference. We show that the inference latency variability can become quite significant in the presence of CPU resource contention. More interestingly, unlike the common belief that the relative performance superiority of DNNs on one device can carry over to another device and/or another level of resource contention, we highlight that a DNN model with a better latency performance than another model can become outperformed by the other model when resource contention be more severe or running on another device. Thus, when optimizing DNN models for mobile inference, only measuring the average latency may not be adequate; instead, latency variability under various conditions should be accounted for, including but not limited to different devices and different levels of CPU resource contention considered in this note.
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20200321221738/https://arxiv.org/pdf/2003.00138v1.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext">
<button class="ui simple right pointing dropdown compact black labeled icon button serp-button">
<i class="icon ia-icon"></i>
Web Archive
[PDF]
</button>
</a>
<a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2003.00138v1" title="arxiv.org access">
<button class="ui compact blue labeled icon button serp-button">
<i class="file alternate outline icon"></i>
arxiv.org
</button>
</a>