Machine learning models have benefited greatly from scale, which has enabled emergent capabilities such as reasoning and in-context learning. However, due to neural scaling laws, further improvements will require exponentially more data. For language models, natural language data also introduces undesirable human biases and requires intensive curation and cleaning, and prior work has shown that synthetic data is a promising alternative that can make pretraining more efficient. Our research will extend this work to investigate whether the structural properties of synthetic training data transfer to diffusion models, with a focus on robotics, a field that also faces a severe data bottleneck: high-quality, properly labeled visual data is not only rare but also extremely expensive and tedious to generate. Our goal is to show that a short pre-pretraining phase on synthetic data, in place of random initialization, increases the overall efficiency of pretraining. A large portion of the research will be to understand which properties of the data the models benefit from and how to generate data with those properties synthetically. One of the largest open questions is whether synthetic data for diffusion models should aim to resemble real-world images, using methods such as fractals, or should instead focus on abstract structure, such as nested brackets.

Whether it is a math riddle or poker, I have always loved to reason through hard problems. In this current world where ML is moving so quickly, there are many bottlenecks that I see for AI to truly impact the world. One of these bottlenecks that I have seen is the scarcity of data. I have also realized that the greatest insights and discoveries are only as good as you can present the information to others. Therefore, I believe that the SuperUROP program could be a great opportunity for me to combine my love of reasoning with the learned skill of conveying information as I work on a problem that I genuinely believe could make a large impact.