We expand model stitching (Lenc & Vedaldi 2015) as a methodology to compare neural networks. Previously, Bansal, Nakkiran & Barak used it to compare the representations learned by differently seeded and/or trained neural networks.

We use it to compare the representations learned by neural networks with different architectures.
This gives us insight into the properties of stitching. Namely, we find that stitching can reach unintuitively
high accuracy for intuitively different layers if those layers come earlier in the first (sender) network than in
the second (receiver). This leads us to hypothesize that stitches are not in fact learning to match the
representations expected by receiver layers, but instead to find representations which yield similar results.

I am doing SuperUROP because I’m curious to explore machine learning research, especially as a way to understand our own inductive biases.