Filters








2,447 Hits in 5.9 sec

Inferring Compact Representations for Efficient Natural Language Understanding of Robot Instructions [article]

Siddharth Patki and Andrea F. Daniele and Matthew R. Walter and Thomas M. Howard
2019 arXiv   pre-print
An open problem is then to develop methods capable of producing the most compact environment model sufficient for accurate and efficient natural language understanding.  ...  A great deal of attention has been paid to developing models and approximate inference algorithms that improve the efficiency of language understanding.  ...  environment representation sufficient for planning and natural language understanding.  ... 
arXiv:1903.09243v1 fatcat:xxl5ora4zvd3pku7smactjqdxy

Language-guided Semantic Mapping and Mobile Manipulation in Partially Observable Environments [article]

Siddharth Patki, Ethan Fahnestock, Thomas M. Howard, Matthew R. Walter
2019 arXiv   pre-print
Recent advances in data-driven models for grounded language understanding have enabled robots to interpret increasingly complex instructions.  ...  Recent semantic mapping methods address partial observability by exploiting language as a sensor to infer a distribution over topological, metric and semantic properties of the environment.  ...  The natural language understanding models were trained on a data-augmented corpus of approximately 115 instructions annotated separately for perception, behavior, and map inferences.  ... 
arXiv:1910.10034v1 fatcat:xvtsqctqu5aq3jl4gl6kkbpyoi

Language-guided Adaptive Perception with Hierarchical Symbolic Representations for Mobile Manipulators [article]

Ethan Fahnestock, Siddharth Patki, Thomas M. Howard
2019 arXiv   pre-print
Language is an effective medium for bi-directional communication in human-robot teams.  ...  To infer the meaning of many instructions, robots need to construct a model of their surroundings that describe the spatial, semantic, and metric properties of objects from observations and prior information  ...  For language to be an effective tool for human-robot teams, robots must understand instructions in the context of its surround- Figure 1 : An example of objects with natural hierarchies.  ... 
arXiv:1909.09880v1 fatcat:p6ak36k56ra4xmpvpdxpgwmlxi

Language Understanding for Field and Service Robots in a Priori Unknown Environments

Matthew Walter, Siddharth Patki, Andrea Daniele, Ethan Fahnestock, Felix Duvallet, Sachithra Hemachandra, Jean Oh, Anthony Stentz, Nicholas Roy, Thomas Howard
2022 Field Robotics  
Natural language provides one such medium, and through significant progress in statistical methods for natural-language understanding, robots are now able to interpret a diverse array of free-form navigation  ...  that the algorithm can follow natural-language instructions without prior knowledge of the environment.  ...  Natural-Language Understanding The approach to natural-language understanding of robot instructions in this paper relies on variations of the Distributed Correspondence Graph (DCG) (Howard et al., 2014b  ... 
doi:10.55417/fr.2022040 fatcat:2zcvlzqehfghtlispn3m7dhwy4

Learning models for following natural language directions in unknown environments

Sachithra Hemachandra, Felix Duvallet, Thomas M. Howard, Nicholas Roy, Anthony Stentz, Matthew R. Walter
2015 2015 IEEE International Conference on Robotics and Automation (ICRA)  
Natural language offers an intuitive and flexible means for humans to communicate with the robots that we will increasingly work alongside in our homes and workplaces.  ...  Our method uses this distribution in place of the latent world model and interprets the natural language instruction as a distribution over the intended behavior.  ...  ACKNOWLEDGMENTS This work was supported in part by the Robotics Consortium of the U.S.  ... 
doi:10.1109/icra.2015.7139984 dblp:conf/icra/HemachandraDHRS15 fatcat:fgmwhttnv5e7ffrqqmcvdx24ui

Learning Models for Following Natural Language Directions in Unknown Environments [article]

Sachithra Hemachandra, Felix Duvallet, Thomas M. Howard, Nicholas Roy, Anthony Stentz, Matthew R. Walter
2015 arXiv   pre-print
Natural language offers an intuitive and flexible means for humans to communicate with the robots that we will increasingly work alongside in our homes and workplaces.  ...  Our method uses this distribution in place of the latent world model and interprets the natural language instruction as a distribution over the intended behavior.  ...  ACKNOWLEDGMENTS This work was supported in part by the Robotics Consortium of the U.S.  ... 
arXiv:1503.05079v1 fatcat:vqfp5di7ejea7nk5obibvqdgyy

Cloud-Based Probabilistic Knowledge Services for Instruction Interpretation [chapter]

Daniel Nyga, Michael Beetz
2017 Springer Proceedings in Advanced Robotics  
As the tasks of autonomous manipulation robots get more complex, the tasking of the robots using natural-language instructions becomes more important.  ...  These joint probability distributions are then used to compute the plan instantiation that has the highest probability of producing the intended action given the natural language instruction.  ...  In other words, understanding a natural-language instruction for robot execution requires appropriate interpretation and completion.  ... 
doi:10.1007/978-3-319-60916-4_37 dblp:conf/isrr/NygaB15 fatcat:myzydcewivdbfeyl3nnnauqbqq

What Matters in Language Conditioned Robotic Imitation Learning [article]

Oier Mees, Lukas Hermann, Wolfram Burgard
2022 arXiv   pre-print
A long-standing goal in robotics is to build robots that can perform a wide range of daily tasks from perceptions obtained with their onboard sensors and specified only via natural language.  ...  plans and a self-supervised contrastive loss that aligns video and language representations.  ...  However, we expect our model to use the time-dependent representation of the sequence visual observations in order to truly understand the meaning of a language instruction.  ... 
arXiv:2204.06252v1 fatcat:xnq35sp7xvelpfubinjpraec7a

Approaches for Action Sequence Representation in Robotics: A Review

Hirenkumar Nakawala, Paulo J. S. Goncalves, Paolo Fiorini, Giancarlo Ferringo, Elena De Momi
2018 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)  
Robust representation of actions and its sequences for complex robotic tasks would transform robot's understanding to execute robotic tasks efficiently.  ...  In this manuscript, we present a review of literature dealing with representation of action and action sequences for robot task planning and execution.  ...  Different combinations of keywords were used: "action sequences", "knowledge representation", "logic", "robot", "natural language processing", "open-source frameworks", "ROS", "visual representation languages  ... 
doi:10.1109/iros.2018.8594256 dblp:conf/iros/NakawalaGFFM18 fatcat:aamljxaqyvbnvjpoqulgmefc7a

Deep Multimodal Embedding: Manipulating Novel Objects with Point-clouds, Language and Trajectories [article]

Jaeyong Sung, Ian Lenz, Ashutosh Saxena
2017 arXiv   pre-print
A robot operating in a real-world environment needs to perform reasoning over a variety of sensor modalities such as vision, language and motion trajectories.  ...  On a large dataset, we achieve significant improvements in both accuracy and inference time over the previous state of the art.  ...  The robot must use the combination of its observations of the world and natural language instructions to infer how to manipulate objects.  ... 
arXiv:1509.07831v2 fatcat:4f5dl4wze5gapeajo5zpvfwlnu

Deep multimodal embedding: Manipulating novel objects with point-clouds, language and trajectories

Jaeyong Sung, Ian Lenz, Ashutosh Saxena
2017 2017 IEEE International Conference on Robotics and Automation (ICRA)  
A robot operating in a real-world environment needs to perform reasoning over a variety of sensor modalities such as vision, language and motion trajectories.  ...  On a large dataset, we achieve significant improvements in both accuracy and inference time over the previous state of the art.  ...  The robot must use the combination of its observations of the world and natural language instructions to infer how to manipulate objects.  ... 
doi:10.1109/icra.2017.7989325 dblp:conf/icra/SungLS17 fatcat:cht7lkjt5zdvxm4jlxi6bbvay4

Sequence-to-Sequence Language Grounding of Non-Markovian Task Specifications

Nakul Gopalan, Dilip Arumugam, Lawson Wong, Stefanie Tellex
2018 Robotics: Science and Systems XIV  
This class of behaviors poses a serious obstacle to existing language understanding for robotics approaches that map to either action sequences or goal state representations.  ...  One example of this could be instructing a wheeled robot to "go to the living room but avoid the kitchen," in order to avoid scuffing the floor.  ...  APPROACH In order to fully specify our system for converting natural language to robot behavior, we begin by describing our problem setting and clarify its connection to the GLTL semantic representation  ... 
doi:10.15607/rss.2018.xiv.067 dblp:conf/rss/GopalanAWT18 fatcat:mlhrku6zrzfbbjbco62wbmnsva

Object Graph Networks for Spatial Language Grounding

Philip Hawkins, Frederic Maire, Simon Denman, Mahsa Baktashmotlagh
2019 2019 Digital Image Computing: Techniques and Applications (DICTA)  
Consider a domestic robot being asked to pick up "the cup nearest to the plate". Natural language is an intuitive way for humans to interact with robots.  ...  However, enabling robots to comprehend natural language, and correctly interpret spatial references, is challenging for two reasons.  ...  The third bypasses intermediate representations and infers actions directly from natural language instruction and observations of the environment using neural networks [25] , [34] .  ... 
doi:10.1109/dicta47822.2019.8946101 dblp:conf/dicta/HawkinsMDB19 fatcat:pkh22hrgnzdgtmgzufupz5gbdq

Multimodal Binding of Parameters for Task-based Robot Programming Based on Semantic Descriptions of Modalities and Parameter Types

Alexander Clifford Perzylo, Nikhil Somani, Stefan Profanter, Markus Rickert, Alois C. Knoll
2015 IEEE/RJS International Conference on Intelligent RObots and Systems  
In this paper, we describe our ongoing efforts to design a cognition-enabled industrial robotic workcell, which significantly increases the efficiency of teaching and adapting robot tasks.  ...  We have designed a formalism to match task parameter and input modality types, in order to infer suitable means for binding values to those parameters.  ...  Some of the missing parameters can be automatically inferred. One aspect of naturally teaching tasks to a robot system is an adequate selection of communication modalities.  ... 
dblp:conf/iros/PerzyloSPRK15 fatcat:go3qtacxybdojpvcy2vzy5ko5m

Unnatural Language Processing: Bridging the Gap Between Synthetic and Natural Language Data [article]

Alana Marzoev, Samuel Madden, M. Frans Kaashoek, Michael Cafarella, Jacob Andreas
2020 arXiv   pre-print
We address this problem by introducing a general purpose technique for "simulation-to-real" transfer in language understanding problems with a delimited set of target behaviors, making it possible to develop  ...  To generalize to natural utterances, we automatically find projections of natural language utterances onto the support of the synthetic language, using learned sentence embeddings to define a distance  ...  The semantic originally described by Wang et al. for this task was equipped with a "canonical grammar"-a compact set of rules describing a mapping from logical forms to (somewhat stilted) natural language  ... 
arXiv:2004.13645v1 fatcat:vee34acnwzdhte6e5djherpt6q
« Previous Showing results 1 — 15 out of 2,447 results