The field of robotics has been experiencing a shift towards using end-to-end machine learning models to create generalist policies that can perform challenging robotic manipulation tasks. However, these models lack persistent memory, resulting in inconsistencies when predicting longer sequences. Our research will be focused on tackling those issues by incorporating a persistent state into the model. We will investigate the use of structured spatio-temporal representations to accurately represent both static and dynamic aspects of the scene. We will also examine the effectiveness of learnable latent representations, evaluating how they compare to the additional structure of explicit representations in achieving long-term consistency and predictive capability.

Through this SuperUROP, I hope to deepen my knowledge in world models and spatio-temporal memory representations, exploring the intersection of machine learning and classical methods, like SLAM and 3D reconstruction, and their application towards consistent long-term planning. I’m excited to explore both areas and work towards bridging the gap between them, while gaining valuable research experience in the field.