Differences

This shows you the differences between two versions of the page.

@@ Line 1: / Line 1: @@
+===== Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization =====
+//Abstract//: Foundation models are redefining how AI systems are built. Practitioners now follow a standard procedure to build their machine learning
+solutions: from a pre-trained foundation model,
+they fine-tune the weights on the target task of
+interest. So, the Internet is swarmed by a handful
+of foundation models fine-tuned on many diverse
+tasks: these individual fine-tunings exist in isolation without benefiting from each other. In our
+opinion, this is a missed opportunity, as these specialized models contain rich and diverse features.
+In this paper, we thus propose model ratatouille,
+a new strategy to recycle the multiple fine-tunings
+of the same foundation model on diverse auxiliary tasks. Specifically, we repurpose these auxiliary weights as initializations for multiple parallel
+fine-tunings on the target task; then, we average
+all fine-tuned weights to obtain the final model.
+This recycling strategy aims at maximizing the
+diversity in weights by leveraging the diversity in
+auxiliary tasks. Empirically, it improves the state
+of the art on the reference DomainBed benchmark
+for out-of-distribution generalization. Looking
+forward, this work contributes to the emerging
+paradigm of updatable machine learning where,
+akin to open-source software development, the
+community collaborates to reliably update machine learning models. Our code is released [[https://github.com/facebookresearch/ModelRatatouille|here]].
+{{ ratatouille.png?600 }}
+<box 99% orange>
+Alexandre Rame, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Leon Bottou and David Lopez-Paz:  **Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization**,  //Proceedings of the 40th International Conference on Machine Learning//, 202:28656--28679, Edited by Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato and Jonathan Scarlett, Proceedings of Machine Learning Research, PMLR, 23--29 Jul 2023.
+[[http://leon.bottou.org/publications/djvu/pmlr-rame-2023.djvu|pmlr-rame-2023.djvu]]
+[[http://leon.bottou.org/publications/pdf/pmlr-rame-2023.pdf|pmlr-rame-2023.pdf]]
+[[http://leon.bottou.org/publications/psgz/pmlr-rame-2023.ps.gz|pmlr-rame-2023.ps.gz]]
+</box>
+  @inproceedings{rame-2023,
+    title = {Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization},
+    author = {Rame, Alexandre and Ahuja, Kartik and Zhang, Jianyu and Cord, Matthieu and Bottou, Leon and Lopez-Paz, David},
+    booktitle = {Proceedings of the 40th International Conference on Machine Learning},
+    pages = {28656--28679},
+    year = {2023},
+    editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan},
+    volume = {202},
+    series = {Proceedings of Machine Learning Research},
+    month = {23--29 Jul},
+    publisher = {PMLR},
+    url = {http://leon.bottou.org/papers/rame-2023},
+  }

User Tools

Site Tools

Differences

Page Tools