Filters








16 Hits in 2.3 sec

VirtualHome: Simulating Household Activities via Programs [article]

Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, Antonio Torralba
2018 arXiv   pre-print
Our VirtualHome simulator allows us to create a large activity video dataset with rich ground-truth, enabling training and testing of video understanding models.  ...  Towards this goal, we first crowd-source programs for a variety of activities that happen in people's homes, via a game-like interface used for teaching kids how to code.  ...  Acknowledgements: We acknowledge partial support from "La Caixa" Fellowship, NSERC COHESA NETGP485577-15, Samsung, DARPA Explainable AI (XAI) program and IARPA D17PC00341. The U.S.  ... 
arXiv:1806.07011v1 fatcat:hvl5buwjwvh4hh6vhokbqdf4wm

VirtualHome: Simulating Household Activities Via Programs

Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, Antonio Torralba
2018 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition  
Our VirtualHome simulator allows us to create a large activity video dataset with rich ground-truth, enabling training and testing of video understanding models.  ...  Towards this goal, we first crowd-source programs for a variety of activities that happen in people's homes, via a game-like interface used for teaching kids how to code.  ...  Acknowledgements: We acknowledge partial support from "La Caixa" Fellowship, NSERC COHESA NETGP485577-15, Samsung, DARPA Explainable AI (XAI) program and IARPA D17PC00341. The U.S.  ... 
doi:10.1109/cvpr.2018.00886 dblp:conf/cvpr/PuigRBLWF018 fatcat:ex4gg4ggd5c4llu5f3khzr4poa

Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents [article]

Wenlong Huang, Pieter Abbeel, Deepak Pathak, Igor Mordatch
2022 arXiv   pre-print
Our evaluation in the recent VirtualHome environment shows that the resulting method substantially improves executability over the LLM baseline.  ...  It can simulate a large variety of realistic human activities in a household environment and supports the ability to perform them via embodied actions defined with a verb-object syntax.  ...  Evaluated Environment: VirtualHome Preliminaries In VirtualHome, activities are expressed as programs.  ... 
arXiv:2201.07207v2 fatcat:2ighvy7jsfaxfllziu4yngv3n4

Synthesizing Environment-Aware Activities via Activity Sketches

Yuan-Hong Liao, Xavier Puig, Marko Boben, Antonio Torralba, Sanja Fidler
2019 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)  
To this end, we build upon VirtualHome [15] to create a new dataset VirtualHome-Env, where we collect program sketches to represent activities and match programs with environments that can afford them.  ...  Given a visual demonstration or a description of an activity, we generate program sketches representing the essential instructions and propose a model to transform these into full programs representing  ...  VirtualHome is a dataset and simulator to represent human household activities. The dataset was collected by asking annotators to come up with various activities and provide a description for them.  ... 
doi:10.1109/cvpr.2019.00645 dblp:conf/cvpr/LiaoPB0F19 fatcat:dcfwnczzjrhonbfdx5w5d65gli

Environments for Lifelong Reinforcement Learning [article]

Khimya Khetarpal, Shagun Sodhani, Sarath Chandar, Doina Precup
2018 arXiv   pre-print
VirtualHome: Simulating Household Activities via Programs VirtualHome crowdsourced a dataset of "programs" for performing different activities in a house.  ...  The programs and the simulated environment can be used to train an agent to perform household tasks based on language instructions.  ... 
arXiv:1811.10732v2 fatcat:wvjiaywe7vde5gvs5eopnihnfy

Mapping Language to Programs using Multiple Reward Components with Inverse Reinforcement Learning [article]

Sayan Ghosh, Shashank Srivastava
2021 arXiv   pre-print
On the VirtualHome framework, we get improvements of up to 9.0% on the Longest Common Subsequence metric and 14.7% on recall-based metrics over previous work on this framework (Puig et al., 2018).  ...  In this paper, we pose program generation from language as Inverse Reinforcement Learning.  ...  , we obtain the percentage of generated programs which are executable in the graph-based simulator of VirtualHome.  ... 
arXiv:2110.00842v1 fatcat:cmgwzzkkk5eudm6astqqftwe24

UnrealROX+: An Improved Tool for Acquiring Synthetic Data from Virtual 3D Environments [article]

Pablo Martinez-Gonzalez, Sergiu Oprea, John Alejandro Castro-Vargas, Alberto Garcia-Garcia, Sergio Orts-Escolano, Jose Garcia-Rodriguez, Markus Vincze
2021 arXiv   pre-print
Few more recent generators or environments are VirtualHome, Habitat or ElderSim. VirtualHome [15] is a multi-agent platform to simulate activities in a household.  ...  ElderSim [17] is a synthetic action simulation platform that can generate synthetic data on elders' daily activities.  ... 
arXiv:2104.11776v1 fatcat:wttdiq7oebcibbclf74cl6tx4e

A Survey of Embodied AI: From Simulators to Research Tasks [article]

Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, Cheston Tan
2022 arXiv   pre-print
By evaluating nine current embodied AI simulators with our proposed seven features, this paper aims to understand the simulators in their provision for use in embodied AI research and their limitations  ...  This paper aims to provide an encyclopedic survey for the field of embodied AI, from its simulators to its research.  ...  The VirtualHome simulator requires a program script from the annotators before it can animate the corresponding interaction or tasks that can be performed within its virtual environment.  ... 
arXiv:2103.04918v8 fatcat:2zu4klcchbhnvmjej5ry3emu4u

Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents [article]

Wenlong Huang, Pieter Abbeel, Deepak Pathak, Igor Mordatch
2022
Our evaluation in the recent VirtualHome environment shows that the resulting method substantially improves executability over the LLM baseline.  ...  It can simulate a large variety of realistic human activities in a household environment and supports the ability to perform them via embodied actions defined with a verb-object syntax.  ...  Evaluated Environment: VirtualHome Preliminaries In VirtualHome, activities are expressed as programs.  ... 
doi:10.48550/arxiv.2201.07207 fatcat:vldc5ikmtjfznp5wwhopz34y3u

BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments [article]

Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martín-Martín, Fei Xia, Kent Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, C. Karen Liu, Silvio Savarese, Hyowon Gweon (+2 others)
2021 arXiv   pre-print
We introduce BEHAVIOR, a benchmark for embodied AI with 100 activities in simulation, spanning a range of everyday household chores such as cleaning, maintenance, and food preparation.  ...  Building such a benchmark poses three fundamental difficulties for each activity: definition (it can differ by time, place, or person), instantiation in a simulator, and evaluation.  ...  A 4 Defining Realistic, Diverse, and Complex Household Activities with BDDL BEHAVIOR challenges embodied AI agents to achieve a diverse set of complex long-horizon household activities through physical  ... 
arXiv:2108.03332v1 fatcat:jviakd3qwvabderhycrilezxoe

On the Synergy between Virtual Reality and Multi-Agent Systems

Alejandra Ospina-Bohórquez, Sara Rodríguez-González, Diego Vergara-Rodríguez
2021 Sustainability  
On the other hand, virtual reality applications are accepted as viable techniques in different areas such as visualization, simulation, design, and research.  ...  : Simulating Household Activities Via Programs 2018 Conference paper Conceptual, Experimental N/A Face Interactions 2017 Journal Conceptual, Experimental Model Ranjbartabar et al. [55] A virtual  ...  [52] is to model activities that occur at home, so it proposes to use programs as a high-level representation of complex tasks. In the paper of Andelfinger et al.  ... 
doi:10.3390/su13084326 fatcat:24m7twqsm5duxau4clemtuqwzm

ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks [article]

Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, Dieter Fox
2020 arXiv   pre-print
We present ALFRED (Action Learning From Realistic Environments and Directives), a benchmark for learning a mapping from natural language instructions and egocentric vision to sequences of actions for household  ...  In VirtualHome [42] , programs are generated from video demonstration and natural language instructions, but inference does not involve egocentric visual and action feedback or partial observability.  ...  Expert Demonstrations When sampling task parameters, we employ an active strategy to maximize data heterogeneity.  ... 
arXiv:1912.01734v2 fatcat:bencnpeqzjcfxjw7qnzupsz6ce

Core Challenges in Embodied Vision-Language Planning [article]

Jonathan Francis, Nariaki Kitamura, Felix Labelle, Xiaopeng Lu, Ingrid Navarro, Jean Oh
2022 arXiv   pre-print
We propose a taxonomy to unify these tasks and provide an in-depth analysis and comparison of the new and current algorithmic approaches, metrics, simulated environments, as well as the datasets used for  ...  VirtualHome (Puig et al., 2018) is an environment designed for simulating household activities by generating programs containing sequences of symbolic instructions representing atomic actions.  ...  ., via visual or auditory sensing modalities.  ... 
arXiv:2106.13948v3 fatcat:tk32nr4jtjekboh33zutellvnm

Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense

Yixin Zhu, Tao Gao, Lifeng Fan, Siyuan Huang, Mark Edmonds, Hangxin Liu, Feng Gao, Chi Zhang, Siyuan Qi, Ying Nian Wu, Joshua B. Tenenbaum, Song-Chun Zhu
2020 Engineering  
: Simulating household activ- 160.  ...  via perspective points, in: NeurIPS, 2019.  ... 
doi:10.1016/j.eng.2020.01.011 fatcat:dltxealx3zgk5eyi2hlszbjhmu

proScript: Partially Ordered Scripts Generation

Keisuke Sakaguchi, Chandra Bhagavatula, Ronan Le Bras, Niket Tandon, Peter Clark, Yejin Choi
2021 Findings of the Association for Computational Linguistics: EMNLP 2021   unpublished
Virtualhome: Simulating household activities nian, and Nathanael Chambers. 2018. Hierarchi- via programs.  ...  account (1 min) clothes (1 hour) (1 month) learn activities interactively in a household in a  ... 
doi:10.18653/v1/2021.findings-emnlp.184 fatcat:6eknga2cjrbnbo5lkf6pgimv3y
« Previous Showing results 1 — 15 out of 16 results