Building on a theory of predictive coding which suggests large language models (like ChatGPT) straighten their representation of language to more easily predict the next word in a sentence, I will explore the internal representations which characterize straightening and next word prediction. I will apply both mechanistic interpretability and computational cognitive science techniques in an effort to understand how and why models straighten language trajectories. This builds on a new but quickly growing geometric/dynamics analysis paradigm for modeling human/artificial neural networks. Characterizing straightening will inform our understanding of how large language models produce the output they do, and it may shed some light on the computation underlying temporal prediction in general.

I am participating in SuperUROP to continue my exploration of machine learning / computational cognitive science research with more independence. Not only is this an opportunity to explore a field that greatly excites me, but I hope to gain much more experience bringing a cohesive research project into fruition. I am most excited about the opportunity to explore a question which has implications for intelligence in both minds and machines.