*Excerpt*:
The analysis of online algorithms is much more difficult than that of
ordinary optimization algorithms. Practical successes in signal
processing (Widrow and Stearns, 1985) motivated the creation of
sophisticated mathematical tools known as {\em stochastic
approximations} (Ljung and Soderstrom, 1983; Benveniste, Metivier and Priouret, 1990)
[…]
The first section describes and illustrates a general framework for
neural network learning algorithms based on stochastic gradient
descent. The second section presents stochastic approximation results
describing the *final phase*. The third section discusses the
conceptual aspects of the *search phase* and comments some of the
newest results.

Léon Bottou and Noboru Murata: **Stochastic Approximations and Efficient Learning**, *The Handbook of Brain Theory and Neural Networks, Second edition,*, Edited by M. A. Arbib, The MIT Press, Cambridge, MA, 2002.

@incollection{bottou-murata-2002, author = {Bottou, L\'{e}on and Murata, Noboru}, title = {Stochastic Approximations and Efficient Learning}, booktitle = {The Handbook of Brain Theory and Neural Networks, Second edition,}, editor = {Arbib, M. A.}, publisher = {The MIT Press}, address = {Cambridge, MA}, year = {2002}, url = {http://leon.bottou.org/papers/bottou-murata-2002}, }