Attention-based large language models attend to all tokens in a context window, which includes information across varying time scales. In particular, long-context settings are expensive and wasteful as attention is stretched across many tokens, not all of which are useful. The question then becomes instead of treating all tokens as “live memory,” how can we best represent general information that may be helpful across a broad range of specific behaviors and queries? The answer can be architectural-for instance, where the split between long-term memory and short-term attention is explicitly delineated by hierarchical models-or a matter of better representation within the bounds of attention-KV-cache augmentation as a way to store compact representations of long-term dependencies or embedding generalizable information into the model scaffolding system itself.

I am participating in the SuperUROP program to further develop my research skills under the support of a structured program. My hope for the upcoming school year is to go deeper into research projects I have started outside of the UROP program.