Research Project Title:
Exploring the potential of Model Stitching
abstract:We are going to explore and expand the uses of Model stitching, particularly for comparing representations. In model stitching, we cut two pre-trained neural networks somewhere down the middle and then connect the bottom of one with the top of the other through a simple (i.e. linear) "stitching" layer. We then optimize this stitching layer and note the resulting model's performance. Stitching similar representations tends to yield accurate models. Stitching fits within a space of representation similarity measures that rely on behavior, more so than (their much less understood) structure. We want to expand the current repertoire of tools in this space, if nothing else, to better understand neural networks' internals.
I am doing SuperUROP because I'm curious to explore machine learning research, especially as a way to understand our own inductive biases.