====== Stochastic Algorithms for One Pass Learning ======

This talk was first given in the [[http://www.nyas.org/Events/Detail.aspx?cid=750f1a55-1e71-4d43-bacd-c5453f1dc3d5|Sixth Annual Machine Learning Symposium]] of the [[http://www.nyas.org|New York Academy of Sciences]] and in the [[http://sites.google.com/site/costnips/|NIPS 2011 Workshop on Computational Trade-offs in Statistical Learning]].

===== Summary =====

{{asgd.png?130 }}
The goal of the presentation is to describe practical stochastic gradient algorithms that process each training example only once, yet asymptotically match the performance of the true empirical optimum. This statement needs, of course, to be made more precise. To achieve this, we'll review the works of Nevel'son and Has'minskij (1972), Fabian (1973, 1978), Murata & Amari (1998), Bottou & LeCun (2004), Polyak & Juditsky (1992), Wei Xu (2010), and Bach & Moulines (2011). We will then show how these ideas lead to practical algorithms and new challenges.

===== Links =====

  * The slides [[http://leon.bottou.org/slides/onepass/onepass.djvu|(djvu, 199k)]] [[http://leon.bottou.org/slides/onepass/onepass.pdf|(pdf, 292k)]]
  * [[:projects:sgd|Stochastic Gradient source code for SVM and CRF]].
  * [[:research/largescale|Learning with Approximative Optimization]].
  * [[:research/stochastic|Learning with Stochastic Gradient]].


===== Papers =====

<box 99% orange>
Léon Bottou and Yann LeCun:  **Large Scale Online Learning**,  //Advances in Neural Information Processing Systems 16 (NIPS 2003)//, Edited by Sebastian Thrun, Lawrence Saul and Bernhard Schölkopf, MIT Press, Cambridge, MA, 2004.

[[papers/bottou-lecun-2004|more...]]
</box>

<box 99% orange>
Léon Bottou and Yann LeCun:  **On-line Learning for Very Large Datasets**,  //Applied Stochastic Models in Business and Industry//, 21(2):137-151, 2005.

[[:papers/bottou-lecun-2004a|more...]]
</box>