Jack G. King

Jack G. King

Scholar Title

MIT EECS Undergraduate Research and Innovation Scholar

Research Title

Statistical Features Govern Temporal Straightening in Large Language Models

Cohort

2024–2025

Department

Electrical Engineering and Computer Science

Research Areas
  • Brain and Cognitive Science
Supervisor

Evelina Fedorenko

Abstract

Building on a theory of predictive coding which suggests large language models (like ChatGPT) straighten their representation of language to more easily predict the next word in a sentence, I will explore the internal representations which characterize straightening and next word prediction. I will apply both mechanistic interpretability and computational cognitive science techniques in an effort to understand how and why models straighten language trajectories. This builds on a new but quickly growing geometric/dynamics analysis paradigm for modeling human/artificial neural networks. Characterizing straightening will inform our understanding of how large language models produce the output they do, and it may shed some light on the computation underlying temporal prediction in general.

Quote

I am participating in SuperUROP to continue my exploration of machine learning / computational cognitive science research with more independence. Not only is this an opportunity to explore a field that greatly excites me, but I hope to gain much more experience bringing a cohesive research project into fruition. I am most excited about the opportunity to explore a question which has implications for intelligence in both minds and machines.

Back to Scholars