Mitchell Wortsman

I am a PhD student at the University of Washington, advised by Ali Farhadi and Ludwig Schmidt. I am broadly interested in large scale machine learning, and have been fortunate to be a part of the model soups and OpenCLIP projects.

Email  /  Google Scholar  /  Twitter

• (May, 2023) New preprint on DataComp: In search of the next generation of multimodal datasets.
• (May, 2023) New preprint on stable and low-precision CLIP training.
• (April, 2023) We introduce OpenFlamingo (led by Anas Awadalla and Irena Gao).
• (January, 2023) We've reached >80% zero-shot ImageNet accuracy with OpenCLIP ViT-G/14.
• (October, 2022) New preprint: lo-fi: distributed fine-tuning without communication.
• (October, 2022) New preprint: LAION-5B: An open large-scale dataset for training next generation image-text models.
• (October, 2022) New preprint: Exploring The Landscape of Robustness in QA (led by Anas Awadalla).
• (August, 2022) New preprint: Patching models by interpolating weights.
• (August, 2022) New preprint: Quality not quantity: on robustness and dataset design (led by Thao Nguyen).
• (May, 2022) New preprint: Data determines distributional robustness (led by Alex Fang).
• (March, 2022) New preprint: CLIP on Wheels (CoW) (led by Samir Gadre).
• (March, 2022) New preprint: Model soups.

DataComp: In search of the next generation of multimodal datasets
Samir Yitzhak Gadre*, Gabriel Ilharco*, Alex Fang*, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, et al., Yair Carmon, Vaishaal Shankar, Ludwig Schmidt
arXiv, 2023
arXiv

Stable and low-precision training for large-scale vision-language models
Mitchell Wortsman*, Tim Dettmers*, Luke Zettlemoyer, Ari S. Morcos, Ali Farhadi, Ludwig Schmidt
ArXiv, 2023
arxiv

OpenFlamingo: an open-source framework for training large multimodal models
Anas Awadalla, Irena Goa, Joshua Gardner, Jack Hessel, et al., Mitchell Wortsman, Ludwig Schmidt
GitHub, 2023

lo-fi: distributed fine-tuning without communication
Mitchell Wortsman, Suchin Gururangan, Shen Li, Ali Farhadi, Ludwig Schmidt, Micheal Rabbat, Ari S. Morcos
ArXiv, 2022
arxiv

LAION-5B: An open large-scale dataset for training next generation image-text models
Christoph Schuhmann*, Romain Beaumont*, Richard Vencu*, Cade Gordon*, Ross Wightman*, Mehdi Cherti*, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt**, Robert Kaczmarczyk**, Jenia Jitsev**
NeurIPS, 2022 (outstanding paper award)
arxiv

Exploring The Landscape of Distributional Robustness for Question Answering Models
Anas Awadalla, Mitchell Wortsman, Gabriel Ilharco, Sewon Min, Ian Magnusson, Hannaneh Hajishirzi, Ludwig Schmidt
EMNLP Findings, 2022
arxiv

Patching open-vocabulary models by interpolating weights
Gabriel Ilharco*, Mitchell Wortsman*, Samir Yitzhak Gadre*, Shuran Song Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, Ludwig Schmidt
NeurIPS, 2022
arxiv

Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP
Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, Ludwig Schmidt
NeurIPS, 2022 (oral)
arxiv

CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration
Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt Shuran Song
ArXiv, 2022
arxiv

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon**, Simon Kornblith**, Ludwig Schmidt**
ICML, 2022
arxiv

Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)
Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, Ludwig Schmidt
ICML, 2022
arxiv

Robust fine-tuning of zero-shot models
Mitchell Wortsman*, Gabriel Ilharco*, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo-Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, Ludwig Schmidt
CVPR, 2022 (oral, best paper finalist)
arxiv / code

OpenCLIP: An open source implementation of CLIP
Gabriel Ilharco*, Mitchell Wortsman*, Ross Wightman*, Cade Gordon*, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, Ludwig Schmidt
GitHub, 2021

Learning Neural Network Subspaces
Mitchell Wortsman, Maxwell Horton, Carlos Guestrin, Ali Farhadi, Mohammad Rastegari
ICML, 2021
arxiv / code

Supermasks in Superposition
Mitchell Wortsman*, Vivek Ramanujan*, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, Ali Farhadi
NeurIPS, 2020
arxiv / code

Soft Threshold Weight Reparameterization for Learnable Sparsity
Aditya Kusupati, Raghav Somani*, Vivek Ramanujan*, Mitchell Wortsman*, Prateek Jain, Sham Kakade, Ali Farhadi
ICML, 2020
arxiv / code

What's Hidden in a Randomly Weighted Neural Network?
Vivek Ramanujan*, Mitchell Wortsman*, Aniruddha Kembhavi, Ali Farhadi, Mohammad Rastegari
CVPR, 2020
arxiv / code

Discovering Neural Wirings
Mitchell Wortsman, Ali Farhadi, Mohammad Rastegari
NeurIPS, 2019
arxiv / code

Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learning
Mitchell Wortsman, Kiana Ehsani, Mohammad Rastegari Ali Farhadi, Roozbeh Mottaghi
CVPR, 2019 (oral)
arxiv / code

• I am part of team which works on an open source implementation of CLIP.

• I am involved with ML Collective which aims to make ML research opportunities accessible.

• Previously, I interned at FAIR with Ari S. Morcos, at Apple with Mohammad Rastegari, and spent a great year at the Allen Institute for AI with Roozbeh Mottaghi.

• I completed my undergraduate thesis in Applied Mathematics at Brown University with senior thesis advisor Professor Kavita Ramanan. Thesis: Interacting Particles Systems and Efficient Approximations for Large Sparse Graphs. This was my first research experience, for which I am grateful.

• I am from Toronto, Canada and enjoy music, skiing, hiking, camping, reading, and climbing.