Differences

This shows you the differences between two versions of the page.

@@ Line 1: / Line 1: @@
+===== Rich feature construction for the optimization-generalization dilemma =====
+//Abstract//: There often is a dilemma between ease of optimization and robust out-of-distribution (OoD)
+generalization. For instance, many OoD methods
+rely on penalty terms whose optimization is challenging. They are either too strong to optimize
+reliably or too weak to achieve their goals.
+In order to escape this dilemma, we propose to
+first construct a rich representation (RFC) containing a palette of potentially useful features, ready
+to be used by even simple models. On the one
+hand, a rich representation provides a good initialization for the optimizer. On the other hand,
+it also provides an inductive bias that helps OoD
+generalization. RFC is constructed in a succession of training episodes. During each step of
+the discovery phase, we craft a multi-objective
+optimization criterion and its associated datasets
+in a manner that prevents the network from using
+the features constructed in the previous iterations.
+During the synthesis phase, we use knowledge
+distillation to force the network to simultaneously
+develop all the features identified during the discovery phase.
+RFC consistently helps six OoD methods achieve
+top performance on challenging invariant training benchmarks, ColoredMNIST (Arjovsky et al.,
+). Furthermore, on the realistic Camelyon17
+task, our method helps both OoD and ERM
+methods outperform earlier compatable results
+by at least 5%, reduce standard deviation by at
+least 4.1%, and makes hyperparameter tuning and
+model selection more reliable.
+{{ rfc.png?500 }}
+<box 99% orange>
+Jianyu Zhang, David Lopez-Paz and Léon Bottou:  **Rich feature construction for the optimization-generalization dilemma**,  //International Conference on Machine Learning//, 26397--26411, PMLR, 2022.
+[[http://leon.bottou.org/publications/djvu/zhang-2022.djvu|zhang-2022.djvu]]
+[[http://leon.bottou.org/publications/pdf/zhang-2022.pdf|zhang-2022.pdf]]
+[[http://leon.bottou.org/publications/psgz/zhang-2022.ps.gz|zhang-2022.ps.gz]]
+</box>
+  @inproceedings{zhang-2022,
+    title = {Rich feature construction for the optimization-generalization dilemma},
+    author = {Zhang, Jianyu and Lopez-Paz, David and Bottou, L{\'e}on},
+    booktitle = {International Conference on Machine Learning},
+    pages = {26397--26411},
+    year = {2022},
+    organization = {PMLR},
+    url = {http://leon.bottou.org/papers/zhang-2022},

User Tools

Site Tools

Differences

Page Tools