====== Convolutional Networks ======

===== Time Delay Neural Networks =====

{{tdnn-simple.png?220 }}
During the first years of my thesis, my main thema
was the construction of speech recognition systems using neural networks.
Kevin Lang and Geoff Hinton had published a tech report describing Time-Delay Neural Networks (TDNN).
Alex Waibel and his team then demonstrated their efficiency at discriminating
the japanese phonems /b/, /d/, /g/.  But their approach was very costly. Training took weeks
on their Alliant super-computer.

Using [[stochastic|Stochastic Gradient Descent]],  
I proposed a new and computationally efficient variant of Time-Delay Neural Networks.
I was able to run speaker-independent word recognition systems on a regular workstation
instead of a super-computer. This was later extended to continuous speech recognition 
system by combining a time-delay neural network and a Viterbi decoder.  
The combination was trained globally using a discriminant algorithm. 



===== Convolutional Networks for Computer Vision =====


{{ invar.png?220}}
This work led to a long collaboration with [[http://yann.lecun.com|Yann LeCun]]
on convolutional networks applied to a broad variety of problems in image recognition
and signal processing, using increasingly sophisticated 
[[structured|Structured Learning]] techniques. See also Yann's pages about 
[[http://www.cs.nyu.edu/~yann/research|convolutional networks]].

\\

\\

===== Publications =====


<box 99% orange>
Léon Bottou:  **Reconnaissance de la parole par reseaux connexionnistes**,  //Proceedings of Neuro Nimes 88//, 197-218, Nimes, France, 1988.

[[:papers:bottou-88b|more...]]
</box>

<box 99% orange>
Léon Bottou, Françoise. Fogelman Soulié, Pascal Blanchet and Jean Sylvain Lienard:  **Experiments with Time Delay Networks and Dynamic Time Warping for Speaker Independent Isolated Digit Recognition**,  //Proceedings of EuroSpeech 89//, 2:537-540, Paris, France, 1989.

[[:papers:bottou-89|more...]]
</box>

<box 99% orange>
M.D. Bedworth, L. Bottou, J. S. Bridle, F.  Fallside, L. Flynn, F. Fogelman Soulié, K.M.  Ponting and R.W. Prager:  **Comparison of neural and conventional classifiers on a speech recognition problem**,  //Proceedings of IEE 1st International Conference on Artificial Neural Networks//, London, 1989.

[[:papers:bedworth-89|more...]]
</box>

<box 99% orange>
Xavier Driancourt and Léon Bottou:  **TDNN-Extracted features**,  //Proceedings of Neuro Nimes 90//, EC2, Nimes, France, 1990.

[[:papers:driancourt-bottou-90|more...]]
</box>

<box 99% orange>
Léon Bottou, Françoise Fogelman Soulié, Pascal Blanchet and Jean Sylvain Lienard:  **Speaker independent isolated digit recognition: Multilayer perceptron vs Dynamic Time Warping**,  //Neural Networks//, 3:453-465, 1990.

[[:papers/bottou-90|more...]]
</box>

<box 99% blue>
Léon Bottou:  //**Une Approche théorique de l'Apprentissage Connexionniste: Applications à la Reconnaissance de la Parole**//, Orsay, France, 1991.

[[:papers/bottou-91a|more...]]
</box>

<box 99% orange>
Xavier Driancourt, Léon Bottou and Patrick Gallinari:  **Learning Vector Quantization, Multi Layer Perceptron and Dynamic Time Warping: Comparison and Cooperation**,  //Proceedings of the International Joint Conference on Neural Networks//, Seattle, 1991.

[[:papers/driancourt-bottou-91|more...]]
</box>

<box 99% orange>
Jame Bromley, Jim W. Bentz, Léon Bottou, Isabelle Guyon, Yann Le Cun, C. Moore, Eduard Säckinger and Roopak Shah:  **Signature Verification using a Siamese Time Delay Neural Network**,  //International Journal of Pattern Recognition and Artificial Intelligence//, 7(4), 1993.

[[papers/bromley-bentz-93|more...]]
</box>

<box 99% orange>
Léon Bottou, Corinna Cortes, John S. Denker, Harris Drucker, Isabelle Guyon, Lawrence D. Jackel, Yann Le Cun, Urs A. Muller, Eduard Säckinger, Patrice Simard and Vladimir Vapnik:  **Comparison of classifier methods: a case study in handwritten digit recognition**,  //Proceedings of the 12th IAPR International Conference on Pattern Recognition, Conference B: Computer Vision & Image Processing.//, 2:77-82, IEEE, Jerusalem, October 1994.

[[papers/bottou-cortes-94|more...]]
</box>

<box 99% orange>
Yann Le Cun, Léon Bottou, Yoshua Bengio and Patrick Haffner:  **Gradient Based Learning Applied to Document Recognition**,  //Proceedings of IEEE//, 86(11):2278-2324, 1998.

[[papers/lecun-98h|more...]]
</box>

<box 99% orange>
Yann LeCun, Léon Bottou and Jie HuangFu:  **Learning Methods for Generic Object Recognition with Invariance to Pose and Lighting**,  //Proc. of Computer Vision and Pattern Recognition//, IEEE, Washington, D.C., 2004.

[[papers/lecun-bottou-huangfu-2004|more...]]
</box>