Autonomous systems that efficiently utilize tools can assist humans in completing many common tasks such as cooking and cleaning. However, current systems fall short of matching human-level of intelligence in terms of adapting to novel tools. Prior works based on affordance often make strong assumptions about the environments and cannot scale to more complex, contact-rich tasks. In this work, we tackle this challenge and explore how agents can learn to use previously unseen tools to manipulate deformable objects. We propose to learn a generative model of the tool-use trajectories as a sequence of tool point clouds, which generalizes to different tool shapes. Given any novel tool, we first generate a tool-use trajectory and then optimize the sequence of tool poses to align with the generated trajectory. We train a single model on four different challenging deformable object manipulation tasks, using demonstration data from only one tool per task. The model generalizes to various novel tools, significantly outperforming baselines. We further test our trained policy in the real world with unseen tools, where it achieves the performance comparable to human.
@inproceedings{qitooluse2024,title={Learning Generalizable Tool-use Skills through Trajectory Generation},author={Qi, Carl and Wu, Yilin and Yu, Lifan and Liu, Haoyue and Jiang, Bowen and Lin, Xingyu and Held, David},booktitle={IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},year={2024},}
RSS
HACMan++: Spatially-Grounded Motion Primitives for Manipulation
In this work, we introduce spatially-grounded parameterized motion primitives to improve policy generalization for robotic manipulation tasks. By grounding the primitives on a spatial location in the environment, our proposed method is able to effectively generalize across object shape and pose variations.
@inproceedings{jiang2024hacman++,title={HACMan++: Spatially-Grounded Motion Primitives for Manipulation},author={Jiang, Bowen and Wu, Yilin and Zhou, Wenxuan and Paxton, Chris and Held, David},booktitle={Robotics: Science and Systems (RSS)},year={2024},}
RSS
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
In this work, we introduce DROID (Distributed Robot Interaction Dataset), a diverse robot manipulation dataset with 76k demonstration trajectories or 350h of interaction data, collected across 564 scenes and 86 tasks by 50 data collectors in North America, Asia, and Europe over the course of 12 months. We demonstrate that training with DROID leads to policies with higher performance, greater robustness, and improved generalization ability.
@inproceedings{khazatsky2024droid,title={DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset},author={Team, DROID Dataset},booktitle={Robotics: Science and Systems (RSS)},year={2024},}
ICRA
Open X-Embodiment: Robotic Learning Datasets and RT-X Models
Open X-Embodiment Team
In IEEE International Conference on Robotics and Automation (ICRA), 2024
@inproceedings{open_x_embodiment_rt_x_2023,title={Open {X-E}mbodiment: Robotic Learning Datasets and {RT-X} Models},author={Team, Open X-Embodiment},booktitle={IEEE International Conference on Robotics and Automation (ICRA)},year={2024},}
2023
CoRL
Stabilize to Act: Learning to Coordinate for Bimanual Manipulation
We present a system for bimanual manipulation that coordinates by assigning roles to arms: a stabilizing arm holds an object stationary while an acting arm acts in this simplified environment.
@inproceedings{grannen2023stabilize,title={Stabilize to Act: Learning to Coordinate for Bimanual Manipulation},author={Grannen, Jennifer and Wu, Yilin and Vu, Brandon and Sadigh, Dorsa},booktitle={Proceedings of the 7th Conference on Robotic Learning (CoRL)},year={2023},}
ICRA
In-Mouth Robotic Bite Transfer with Visual and Haptic Sensing
We build a semi autonomous robotic system to do in-mouth transfer of food safely and comfortably for disabled people. The system is composed of a force-reactive controller to safely accommodate the motions of the user throughout the transfer, a novel dexterous wrist-like end effector to reduce the discomfort and a visual sensor to identify the user mouth.
@inproceedings{10160467,author={Shaikewitz, Lorenzo and Wu, Yilin and Belkhale, Suneel and Grannen, Jennifer and Sundaresan, Priya and Sadigh, Dorsa},booktitle={2023 IEEE International Conference on Robotics and Automation (ICRA)},title={In-Mouth Robotic Bite Transfer with Visual and Haptic Sensing},year={2023},}
2022
CoRL
Learning Bimanual Scooping Policies for Food Acquisition
We propose a general bimanual scooping primitive and an adaptive stabilization strategy that enables successful acquisition of a diverse set of food geometries and physical properties with close-loop visual feedback.
@inproceedings{grannen2022learning,title={Learning Bimanual Scooping Policies for Food Acquisition},author={Grannen, Jennifer and Wu, Yilin and Belkhale, Suneel and Sadigh, Dorsa},booktitle={Proceedings of the 6th Conference on Robot Learning (CoRL)},year={2022},}
2021
ICLR
Solving Compositional Reinforcement Learning Problems via Task Reduction
Our work is to train a RL agent to acquire rope-spreading and cloth-spreading skills without any human demonstrations and the method applies to real robots after domain adaptation.
@inproceedings{li2021solving,title={Solving Compositional Reinforcement Learning Problems via Task Reduction},author={Li, Yunfei and Wu, Yilin and Xu, Huazhe and Wang, Xiaolong and Wu, Yi},booktitle={International Conference on Learning Representations (ICLR)},year={2021},}
2020
RSS
Learning to Manipulate Deformable Objects without Demonstrations
In this work, we introduce spatially-grounded parameterized motion primitives to improve policy generalization for robotic manipulation tasks. By grounding the primitives on a spatial location in the environment, our proposed method is able to effectively generalize across object shape and pose variations.
@inproceedings{wu2020learning,title={Learning to Manipulate Deformable Objects without Demonstrations},author={Wu, Yilin and Yan, Wilson and Kurutach, Thanard and Pinto, Lerrel and Abbeel, Pieter},booktitle={Robotics: Science and Systems, (RSS)},year={2020},organization={MIT Press Journals},}
