===== Efficient Backprop =====

//Abstract//:
The convergence of back-propagation learning is analyzed
so as to explain common phenomenon observed by practitioners. Many
undesirable behaviors of backprop can be avoided with tricks that are
rarely exposed in serious technical publications. This paper gives some
of those tricks, and offers explanations of why they work.
Many authors have suggested that second-order optimization methods
are advantageous for neural net training. It is shown that most "classical"
second-order methods are impractical for large neural networks. A few
methods are proposed that do not have these limitations.

//Note//: This paper follows the presentation
[[:talks#The BackPropagation CookBook|The BackPropagation CookBook]]
given during the NIPS 1996 Workshop 
[[http://www.willamette.edu/~gorr/nipsws.htm|Tricks of the Trade]]
organized by [[http://www.willamette.edu/~gorr|Jenny Orr]]
and [[http://ida.first.fraunhofer.de/~klaus|Klaus-Robert Müller]].

<box 99% orange>
Yann Le Cun, Léon Bottou, Genevieve B. Orr and Klaus-Robert Müller:  **Efficient Backprop**,  //Neural Networks, Tricks of the Trade//, Lecture Notes in Computer Science LNCS 1524, Springer Verlag, 1998.

[[http://leon.bottou.org/publications/djvu/tricks-1998.djvu|tricks-1998.djvu]]
[[http://leon.bottou.org/publications/pdf/tricks-1998.pdf|tricks-1998.pdf]]
[[http://leon.bottou.org/publications/psgz/tricks-1998.ps.gz|tricks-1998.ps.gz]]
</box>

  @incollection{lecun-98x,
    author = {{Le Cun}, Yann and Bottou, L\'{e}on and Orr, Genevieve B. and M{\"{u}}ller, Klaus-Robert},
    title = {Efficient Backprop},
    booktitle = {Neural Networks, Tricks of the Trade},
    series = {Lecture Notes in Computer Science LNCS~1524},
    publisher = {Springer Verlag},
    year = {1998},
    url = {http://leon.bottou.org/papers/lecun-98x},
  }