RoboBench: Towards sustainable robotics system benchmarking

Jonathan Weisz, Yipeng Huang, Florian Lier, Simha Sethumadhavan, Peter Allen
<span title="">2016</span> <i title="IEEE"> <a target="_blank" rel="noopener" href="https://fatcat.wiki/container/nytnrt3gtzbixld5r6a4talbky" style="color: black;">2016 IEEE International Conference on Robotics and Automation (ICRA)</a> </i> &nbsp;
We present RoboBench, a novel platform for sharing robot full-system simulations for benchmarking. The creation of this platform and benchmark suite is motivated by a need for reproducible research. A challenge in creating a full-system benchmarks are incompatibilities in software created by different groups and the difficulty of reproducing software environments. We solve this problem by using software containers, an emerging virtualization technology. RoboBench enables sharing robot software
more &raquo; ... n a runnable state, capturing the software behavior of robots carrying out missions. These simulations make clear the performance impact and resource usage of programs and algorithms relative to other software involved in the mission. These containers are integrated with the CITK platform for reproducible research, which automates generation and publishing of the containers. We present an overview of the system, a description of our prototype set of benchmark missions, along with a validation study comparing the computational load profile of a mission performed on a real and simulated robot. Additionally, we present preliminary results of an overall analysis of the benchmarks in the RoboBench suite, showing where computational work is expended in robotics common robotics tasks. RoboBench is extensible, and is the first step toward a robust, quantitative approach to engineering computationally-efficient robots.
<span class="external-identifiers"> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1109/icra.2016.7487514">doi:10.1109/icra.2016.7487514</a> <a target="_blank" rel="external noopener" href="https://dblp.org/rec/conf/icra/WeiszHLSA16.html">dblp:conf/icra/WeiszHLSA16</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/oyzorut3s5agbkgqc3tn42ktey">fatcat:oyzorut3s5agbkgqc3tn42ktey</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20160418075139/http://www.cs.columbia.edu:80/~allen/PAPERS/robobench_icra_v6.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/08/d0/08d0d203df25c1cd97ca74e68e5dd42328042eef.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1109/icra.2016.7487514"> <button class="ui left aligned compact blue labeled icon button serp-button"> <i class="external alternate icon"></i> ieee.com </button> </a>