RoboPapers

How can we do in-context learning for robots? Watch this episode to find out. This work was an ICLR 2025 oral paper, and winner of Best Paper Award at the ICLR 2025 Robot Learning Workshop. Following the impressive capabilities of in-context learning with large transformers, In-Context Imitation Learning (ICIL) is a promising opportunity for robotics. We introduce Instant Policy , which learns new...

How can world models be used for training autonomous driving? Learn by watching this episode with Florent Bartoccioni! We explores the potential of large-scale generative video models to enhance autonomous driving capabilities, introducing an open-source autoregressive video model (VaViM) and a companion video-action model (VaVAM). VaViM is a simple autoregressive model that predicts frames using...

Two papers in one episode! Learn about how we can use small amounts of data to train transformers capable of doing truly impressive stuff. Original Post on X Dom, cars don’t fly!—Or do they? In-Air Vehicle Maneuver for High-Speed Off-Road Navigation When pushing the speed limit for aggressive off-road navigation on uneven terrain, it is inevitable that vehicles may become airborne from time to tim...

Large real-world robot datasets hold great potential to train generalist robot models, but scaling real-world human data collection is time-consuming and resource-intensive. Simulation has great potential in supplementing large-scale data, especially with recent advances in generative AI and automated data generation tools that enable scalable creation of robot behavior datasets. However, training...

It’s hard to collect data for humanoid robots at sufficient scale for generalization. The authors of “Humanoid Policy ~ Human Policy” have the answer: collect human data at scale, and retarget it to humanoid robots. This acts as a multiplier, letting you get away with using far less robot data to accomplish challenging robot tasks. Watch or listen to learn more. Abstract: Training manipulation pol...

Evaluating robot policies is challenging, because everyone has a slightly different testing setup and environment. Paul joined us to talk about his work AutoEval, which is a continuously-operating benchmark which allows you to test robot policies remotely, in any environment. Abstract: Scalable and reproducible policy evaluation has been a long-standing challenge in robot learning. Evaluations are...

3D spatial information provides a really strong signal for robotics policies, something we’ve discussed in previous episodes . But computing this 3D structure is hard, and often relies on imperfect, low-quality depth sensors. It would be great if we could reconstruct this information from cameras alone, with little prior information. Well, that’s exactly what VGGT does! We present VGGT , a feed-fo...

Hao-Shu talks to us about how we can learn a contact-centric grasp representation which works across many different robots. We introduce an efficient approach for learning dexterous grasping with minimal data, advancing robotic manipulation capabilities across different robotic hands. Unlike traditional methods that require millions of grasp labels for each robotic hand, our method achieves high p...

3D features are uniquely better at generalizing than 2D image features, given less data. Learn about how we can use them to train generalizable robot manipulation policies with Yang Gao. Following its success in natural language processing and computer vision, foundation models that are pre-trained on large-scale multi-task datasets have also shown great potential in robotics. However, most existi...

Human data is much more plentiful than robot data, and humans already know how to perform so many tasks. Teaching robots from human videos, then, has a ton of potential. We present R+X, a framework which enables robots to learn skills from long, unlabelled, first-person videos of humans performing everyday tasks. Given a language command from a human, R+X first retrieves short video clips containi...

Value prediction is an important robotics problem, wherein we can determine how useful a state is to the successful execution of a task in the future. The key insight that Jason and his co-authors showed in this paper was simple: * Large vision-language models already encode a ton of information about task completion from being trained on a wide range of human image and video data * We can use the...

Sim-to-real training is an important part of the future of robot skill learning. But a lot of sim-to-real work focuses on navigation , grasping, or, more recently, non-interactive robot behaviors like dancing . Training dexterous policies for humanoids is very different, because manipulation is a very hard problem and multi-finger dexterous hands are even more difficult. Enter this cool work from...

One of the dreams of robotics research is being able to download and test out a model and have it *just work*. Mahi talks to us about “robot utility models,” which are essentially just this: models that you can download and test out to do useful things like opening a cabinet or picking up a can. Robot models, particularly those trained with large amounts of data, have recently shown a plethora of...

Robotic manipulation systems operating in diverse, dynamic environments must exhibit three critical abilities: multitask interaction, generalization to unseen scenarios, and spatial memory. While significant progress has been made in robotic manipulation, existing approaches often fall short in generalization to complex environmental variations and addressing memory-dependent tasks. To bridge this...