Developing a Reward Metric in Reinforcement Learning through Text and Image Subgoals
Reinforcement learning provides a neurally inspired perspective of how an agent learns to interact with its environment. This notion of interaction is formalized by considering policies that map states to actions which may or may not lead to a reward. Despite successes in limited domains learning from such rewards has proven challenging when the environmental feedback signal is particularly sparse. To that effect we will explore a means of augmenting the environmental reward signal with a reward function learned by the agent itself. We will train an agent to use text directions associated with a representative image of the task to aid in subgoal completion. The advantage to this approach is that it will be possible to learn tasks for which the environment offers no explicit reward.

I began doing machine learning work in Josh Tenenbaum’s Computational Cognitive Science group. After getting some experience with vision projects I became interested in exploring another domain: language. This year I’m excited to learn more about and bring together ideas from the natural language processing computer vision and reinforcement learning communities.