Reasoning LLMs generate chains of thought spanning tens of thousands of tokens, making the KV cache a primary serving bottleneck: every token’s KV states persist in memory and incur attention cost long after they are last needed. Recent work (Memento) fine-tunes a model to compress each completed block of reasoning into a short textual summary, called a “memento.” The block is then evicted from the KV cache, and the model continues from the accumulated summaries. This cuts peak KV cache memory 2-3x on single-turn math, code, and science tasks. However, each memento is generated while its block is still in context, meaning its KV states are computed with the full block visible. When the cache is instead discarded at each block boundary and recomputed from the prompt and summary text, AIME’24 accuracy drops by 15 points. It remains open whether retaining these states offers a genuine advantage over reasoning from the summaries alone, or whether the drop reflects evaluation outside the model’s training regime. Whether this implicit memory channel can be controlled is similarly unexplored. We aim to characterize this channel, develop methods to control it under a fixed KV budget, and evaluate learned compaction in long-horizon, multi-turn coding environments.

I’m participating in SuperUROP to build on my experience in long-context LLM research. Previously, I worked on improving harnesses that help LLMs handle very large contexts. I think large contexts and long-horizon tasks are some of the most exciting challenges in the field today, and this project will let me approach them from an angle I haven’t tried before.