Filters








3,565 Hits in 6.1 sec

Integrated Benchmarking and Design for Reproducible and Accessible Evaluation of Robotic Agents [article]

Jacopo Tani and Andrea F. Daniele and Gianmarco Bernasconi and Amaury Camus and Aleksandar Petrov and Anthony Courchesne and Bhairav Mehta and Rohit Suri and Tomasz Zaluska and Matthew R. Walter and Emilio Frazzoli and Liam Paull and Andrea Censi
2020 arXiv   pre-print
In this paper, we describe a new concept for reproducible robotics research that integrates development and benchmarking, so that reproducibility is obtained "by design" from the beginning of the research  ...  One of the central components of this setup is the Duckietown Autolab, a remotely accessible standardized setup that is itself also relatively low-cost and reproducible.  ...  at ETH's Institute for Dynamic Systems and Control (IDSC) who supported us.  ... 
arXiv:2009.04362v1 fatcat:z5mz7cvnv5cp5ehz5nciroqcwu

rl_reach: Reproducible reinforcement learning experiments for robotic reaching tasks

Pierre Aumjaud, David McAuliffe, Francisco J. Rodriguez-Lera, Philip Cardiff
2021 Software Impacts  
Acknowledgements This Career-FIT project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 713654.  ...  necessary elements for producing a robust performance benchmark of RL solutions for simple robotics reaching tasks.  ...  The evaluation metrics are averaged across all the seed runs to promote reproducible, reliable and robust experiments. • Straightforward benchmark : When a trained policy is evaluated, the evaluation metrics  ... 
doi:10.1016/j.simpa.2021.100061 fatcat:gxfzu56trjg5lhd5o7iemjckne

OPEB: Open Physical Environment Benchmark for Artificial Intelligence [article]

Hamid Mirzaei, Mona Fathollahi, Tony Givargis
2017 arXiv   pre-print
To involve a large number of research groups, standard benchmarks are needed to evaluate and compare proposed algorithms.  ...  In this paper, we propose a physical environment benchmark framework to facilitate collaborative research in this area by enabling different research groups to integrate their designed benchmarks in a  ...  Access to these physical environments (hardwares/robots) is not feasible for a lot of research groups. This hinders partnerships and cooperation between academia and industry.  ... 
arXiv:1707.00790v1 fatcat:vtsgef6fxncmhikpvw3lvfo5ny

ROBEL: Robotics Benchmarks for Learning with Low-Cost Robots [article]

Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, Vikash Kumar
2019 arXiv   pre-print
ROBEL is an open-source platform of cost-effective robots designed for reinforcement learning in the real world.  ...  To leverage this platform, we propose an extensible set of continuous control benchmark tasks for each robot.  ...  Acknowledgments We thank Aravind Rajeswaran, Emo Todorov, Vincent Vanhoucke, Matt Neiss, Chad Richards, Thinh Nguyen, Byron David, Garrett Peake, Krista Reymann, and the rest of Robotics at Google for  ... 
arXiv:1909.11639v3 fatcat:gzivfkxwfre7lnghtkvkwinqze

RoSmEEry: Robotic Simulated Environment for Evaluation and Benchmarking of Semantic Mapping Algorithms [article]

Sara Kaszuba, Sandeep Reddy Sabbella, Vincenzo Suriani, Francesco Riccio, Daniele Nardi
2021 arXiv   pre-print
To tackle these issues we design RoSmEEry, a novel framework based on the Gazebo simulator, where we introduce an accessible and ready-to-use methodology for a systematic evaluation of semantic mapping  ...  In fact, it is difficult to obtain a rich and accurate semantic map of the environment and of the objects therein.  ...  Hence, to overcome these limitations, we contribute to the state-of-the-art with an evaluation benchmarking environment specifically designed for robot semantic mapping algorithms.  ... 
arXiv:2105.07938v1 fatcat:r2tp57pjjffq7ncsrhbvhnfwya

OffWorld Gym: open-access physical robotics environment for real-world reinforcement learning benchmark and research [article]

Ashish Kumar, Toby Buckley, John B. Lanier, Qiaozhi Wang, Alicia Kavelaars, Ilya Kuzovkin
2020 arXiv   pre-print
In this work, we present a prototype real-world environment from OffWorld Gym -- a collection of real-world environments for reinforcement learning in robotics with free public remote access.  ...  Close integration into existing ecosystem allows the community to start using OffWorld Gym without any prior experience in robotics and takes away the burden of managing a physical robotics system, abstracting  ...  Acknowledgments Special thanks to Eric Tola, Matt Tomlinson, Matthew Schwab and Piyush Patil for help with the mechanical and electrical design and implementation.  ... 
arXiv:1910.08639v4 fatcat:zhrino7uzfbfrihslpiqepb6g4

robosuite: A Modular Simulation Framework and Benchmark for Robot Learning [article]

Yuke Zhu and Josiah Wong and Ajay Mandlekar and Roberto Martín-Martín
2020 arXiv   pre-print
It offers a modular design for creating robotic tasks as well as a suite of benchmark environments for reproducible research.  ...  This paper discusses the key system modules and the benchmark environments of our new release robosuite v1.0.  ...  Acknowledgement We would like to thank members of the Stanford People, AI, & Robots (PAIR) Group for their support and feedback to robosuite.  ... 
arXiv:2009.12293v1 fatcat:2b2fxy2lqfesxoinxtbp7irkwa

The AI Driving Olympics at NeurIPS 2018 [article]

Julian Zilly, Jacopo Tani, Breandan Considine, Bhairav Mehta, Andrea F. Daniele, Manfred Diaz, Gianmarco Bernasconi, Claudio Ruch, Jan Hakenberg, Florian Golemo, A. Kirsten Bowser, Matthew R. Walter, Ruslan Hristov (+4 others)
2019 arXiv   pre-print
For each task, we provide tools for competitors to use in the form of simulators, logs, code templates, baseline implementations and low-cost access to robotic hardware.  ...  The results of AI-DO 1 highlight the need for better benchmarks, which are lacking in robotics, as well as improved mechanisms to bridge the gap between simulation and reality.  ...  We are grateful to Amazon AWS and Aptiv for their sponsorship and hands-on help that went into this competition.  ... 
arXiv:1903.02503v1 fatcat:udpe2g7ypjckppq65qgmcnb7ru

SwarmRob: A Toolkit for Reproducibility and Sharing of Experimental Artifacts in Robotics Research [article]

Aljoscha Pörtner, Martin Hoffmann, Matthias König
2018 arXiv   pre-print
Due to the complexity of robotics, the reproducibility of results and experiments is one of the fundamental problems in robotics research.  ...  As a result, the proposed solution supports researchers in executing and reproducing experimental evaluations.  ...  This feature would follow up on the goal of an independent an easily accessible toolkit for reproducibility in experimental robotics research.  ... 
arXiv:1801.04199v2 fatcat:kx53mlnrnnd67nhk3s7rzbtcye

Benchmarking Wearable Robots: Challenges and Recommendations From Functional, User Experience, and Methodological Perspectives

Diego Torricelli, Carlos Rodriguez-Guerrero, Jan F. Veneman, Simona Crea, Kristin Briem, Bigna Lenggenhager, Philipp Beckerle
2020 Frontiers in Robotics and AI  
The third one proposes a strategic path for a global benchmarking methodology, composed by reproducible experimental procedures representing real-life conditions.  ...  In order for WRs to be effectively and widely adopted by end-users, a common benchmarking framework needs to be established.  ...  For systematic consideration, existing human-oriented design approaches evaluate user experience and integrate it into design processes, e.g., ISO 9241 (Jokela et al., 2003) or humanmachine-centered  ... 
doi:10.3389/frobt.2020.561774 pmid:33501326 pmcid:PMC7805816 fatcat:nbfmrjxcljeujcm2ajpngjkpse

RB2: Robotic Manipulation Benchmarking with a Twist [article]

Sudeep Dasari, Jianren Wang, Joyce Hong, Shikhar Bahl, Yixin Lin, Austin Wang, Abitha Thankaraj, Karanbir Chahal, Berk Calli, Saurabh Gupta, David Held, Lerrel Pinto (+3 others)
2022 arXiv   pre-print
In robotic manipulation research, there is a trade-off between reproducibility and broad accessibility.  ...  In this paper, we re-imagine benchmarking for robotic manipulation as state-of-the-art algorithmic implementations, alongside the usual set of tasks and experimental protocols.  ...  In particular, we'd like to thank Aditya Prakash, Kenneth Marino, and Zisu Dong.  ... 
arXiv:2203.08098v1 fatcat:n5igsh6mkrcgpkw37dhta55wja

Crowd against the machine: A simulation-based benchmark tool to evaluate and compare robot capabilities to navigate a human crowd [article]

Fabien Grzeskowiak
2021 arXiv   pre-print
to test robots, to establish standard scenarios and metrics to evaluate navigation techniques in terms of safety and efficiency, and thus, to install new methods to benchmarking robots' crowd navigation  ...  The evaluation of robot capabilities to navigate human crowds is essential to conceive new robots intended to operate in public spaces.  ...  ACKNOWLEDGMENT This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 779942, CrowdBot.  ... 
arXiv:2104.14177v1 fatcat:mtvh6qedgbawrpfwls6c7b6duy

RoboCup@Home : Analysis and results of evolving competitions for domestic and service robots

Luca Iocchi, Dirk Holz, Javier Ruiz-del-Solar, Komei Sugiura, Tijn van der Zant
2015 Artificial Intelligence  
The RoCKIn project also aims for better integration between competitions and scientific benchmarking [3] by collecting benchmarking data during tests that combine the logged internal states of robots and  ...  A good test and scoring system design allows for suitable development of corresponding solutions from participants.  ...  A set of benchmark tests is designed and used to evaluate robot abilities and performances in a realistic non-standardized home environment.  ... 
doi:10.1016/j.artint.2015.08.002 fatcat:tfzepm2urza5vmnsjumqmnoz3y

Communication in Human-Robot Interaction

Andrea Bonarini
2020 Current Robotics Reports  
Communication is just one of the functionalities of an interactive robot and, as all the others, will need to be benchmarked to support the possibility for social robots to reach a real market.  ...  To present the multi-faceted aspects of communication between robot and humans (HRI), putting in evidence that it is not limited to language-based interaction, but it includes all aspects that are relevant  ...  Benchmarking: the "Real" Performance "Robot benchmarking can be defined as an objective performance evaluation of a robot system/subsystem under controlled, reproducible conditions. [...]  ... 
doi:10.1007/s43154-020-00026-1 pmid:34977589 pmcid:PMC7449787 fatcat:mk347sj7mbazjn3m3pn3i37esa

Gym-Ignition: Reproducible Robotic Simulations for Reinforcement Learning [article]

Diego Ferigo, Silvio Traversaro, Giorgio Metta, Daniele Pucci
2019 arXiv   pre-print
This paper presents Gym-Ignition, a new framework to create reproducible robotic environments for reinforcement learning research.  ...  The core of Gym-Ignition is a component that contains the Ignition Gazebo simulator and exposes a simple interface for its configuration and execution.  ...  The European Commission or its services cannot be held responsible for any use that may be made of the information it contains.  ... 
arXiv:1911.01715v2 fatcat:a2j6ma2eezanzfhwnnrgcjzux4
« Previous Showing results 1 — 15 out of 3,565 results