Table of Contents

Convolutional Networks

Time Delay Neural Networks

During the first years of my thesis, my main thema was the construction of speech recognition systems using neural networks. Kevin Lang and Geoff Hinton had published a tech report describing Time-Delay Neural Networks (TDNN). Alex Waibel and his team then demonstrated their efficiency at discriminating the japanese phonems /b/, /d/, /g/. But their approach was very costly. Training took weeks on their Alliant super-computer.

Using Stochastic Gradient Descent, I proposed a new and computationally efficient variant of Time-Delay Neural Networks. I was able to run speaker-independent word recognition systems on a regular workstation instead of a super-computer. This was later extended to continuous speech recognition system by combining a time-delay neural network and a Viterbi decoder. The combination was trained globally using a discriminant algorithm.

Convolutional Networks for Computer Vision

This work led to a long collaboration with Yann LeCun on convolutional networks applied to a broad variety of problems in image recognition and signal processing, using increasingly sophisticated Structured Learning techniques. See also Yann's pages about convolutional networks.



Publications

Léon Bottou: Reconnaissance de la parole par reseaux connexionnistes, Proceedings of Neuro Nimes 88, 197-218, Nimes, France, 1988.

more...

Léon Bottou, Françoise. Fogelman Soulié, Pascal Blanchet and Jean Sylvain Lienard: Experiments with Time Delay Networks and Dynamic Time Warping for Speaker Independent Isolated Digit Recognition, Proceedings of EuroSpeech 89, 2:537-540, Paris, France, 1989.

more...

M.D. Bedworth, L. Bottou, J. S. Bridle, F. Fallside, L. Flynn, F. Fogelman Soulié, K.M. Ponting and R.W. Prager: Comparison of neural and conventional classifiers on a speech recognition problem, Proceedings of IEE 1st International Conference on Artificial Neural Networks, London, 1989.

more...

Xavier Driancourt and Léon Bottou: TDNN-Extracted features, Proceedings of Neuro Nimes 90, EC2, Nimes, France, 1990.

more...

Léon Bottou, Françoise Fogelman Soulié, Pascal Blanchet and Jean Sylvain Lienard: Speaker independent isolated digit recognition: Multilayer perceptron vs Dynamic Time Warping, Neural Networks, 3:453-465, 1990.

more...

Léon Bottou: Une Approche théorique de l'Apprentissage Connexionniste: Applications à la Reconnaissance de la Parole, Orsay, France, 1991.

more...

Xavier Driancourt, Léon Bottou and Patrick Gallinari: Learning Vector Quantization, Multi Layer Perceptron and Dynamic Time Warping: Comparison and Cooperation, Proceedings of the International Joint Conference on Neural Networks, Seattle, 1991.

more...

Jame Bromley, Jim W. Bentz, Léon Bottou, Isabelle Guyon, Yann Le Cun, C. Moore, Eduard Säckinger and Roopak Shah: Signature Verification using a Siamese Time Delay Neural Network, International Journal of Pattern Recognition and Artificial Intelligence, 7(4), 1993.

more...

Léon Bottou, Corinna Cortes, John S. Denker, Harris Drucker, Isabelle Guyon, Lawrence D. Jackel, Yann Le Cun, Urs A. Muller, Eduard Säckinger, Patrice Simard and Vladimir Vapnik: Comparison of classifier methods: a case study in handwritten digit recognition, Proceedings of the 12th IAPR International Conference on Pattern Recognition, Conference B: Computer Vision & Image Processing., 2:77-82, IEEE, Jerusalem, October 1994.

more...

Yann Le Cun, Léon Bottou, Yoshua Bengio and Patrick Haffner: Gradient Based Learning Applied to Document Recognition, Proceedings of IEEE, 86(11):2278-2324, 1998.

more...

Yann LeCun, Léon Bottou and Jie HuangFu: Learning Methods for Generic Object Recognition with Invariance to Pose and Lighting, Proc. of Computer Vision and Pattern Recognition, IEEE, Washington, D.C., 2004.

more...