Differences

This shows you the differences between two versions of the page.

@@ Line 1: / Line 1: @@
+===== The Effects of Regularization and Data Augmentation are Class Dependent =====
+{{ randalls.png?300}}
+//Abstract//: Regularization is a fundamental technique to prevent over-fitting and to improve generalization
+performances by constraining a model’s complexity. Current Deep Networks heavily rely on regularizers such as Data-Augmentation (DA) or weight-decay, and employ structural risk minimization,
+i.e. cross-validation, to select the optimal regularization hyper-parameters. In this study, we demonstrate that techniques such as DA or weight decay produce a model with a reduced complexity that
+is unfair across classes. The optimal amount of DA or weight decay found from cross-validation leads
+to disastrous model performances on some classes e.g. on Imagenet with a resnet50, the “barn spider” classification test accuracy falls from 68% to 46% only by introducing random crop DA during
+training. Even more surprising, such performance drop also appears when introducing uninformative
+regularization techniques such as weight decay. Those results demonstrate that our search for ever
+increasing generalization performance -averaged over all classes and samples- has left us with models
+and regularizers that silently sacrifice performances on some classes. This scenario can become dangerous when deploying a model on downstream tasks e.g. an Imagenet pre-trained resnet50 deployed
+on INaturalist sees its performances fall from 70% to 30% on class #8889 when introducing random
+crop DA during the Imagenet pre-training phase. Those results demonstrate that designing novel
+regularizers without class-dependent bias remains an open research question.
+<box 99% orange>
+Randall Balestriero, Leon Bottou and Yann LeCun:  **The Effects of Regularization and Data Augmentation are Class Dependent**,  //Advances in Neural Information Processing Systems//, 35:37878--37891, Edited by S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho and A. Oh, Curran Associates, Inc., 2022.
+[[http://leon.bottou.org/publications/djvu/balestriero-2022.djvu|balestriero-2022.djvu]]
+[[http://leon.bottou.org/publications/pdf/balestriero-2022.pdf|balestriero-2022.pdf]]
+[[http://leon.bottou.org/publications/psgz/balestriero-2022.ps.gz|balestriero-2022.ps.gz]]
+</box>
+  @inproceedings{balestriero-2022,
+    author = {Balestriero, Randall and Bottou, Leon and LeCun, Yann},
+    booktitle = {Advances in Neural Information Processing Systems},
+    editor = {Koyejo, S. and Mohamed, S. and Agarwal, A. and Belgrave, D. and Cho, K. and Oh, A.},
+    pages = {37878--37891},
+    publisher = {Curran Associates, Inc.},
+    title = {The Effects of Regularization and Data Augmentation are Class Dependent},
+    volume = {35},
+    year = {2022},
+    url = {http://leon.bottou.org/papers/balestriero-2022},
+  }

User Tools

Site Tools

Differences

Page Tools