http://leon.bottou.org/ 2010-07-13T22:26:58-04:00 http://leon.bottou.org/ http://leon.bottou.org/lib/images/favicon.ico text/html 2007-07-24T19:11:49-04:00 http://leon.bottou.org/papers/bakir-bottou-weston-2005?rev=1185318709&do=diff Breaking SVM Complexity with Cross-Training Abstract: We propose to selectively remove examples from the training set using probabilistic estimates related to editing algorithms (Devijver and Kittler, 1982). This heuristic procedure aims at creating a separable distribution of training examples with minimal impact on the position of the decision boundary. It breaks the linear dependency between the number of SVs and the number of training examples, and sharply reduces the complexity of SVMs … text/html 2006-12-13T13:57:14-04:00 http://leon.bottou.org/papers/bedworth-89?rev=1166036234&do=diff Comparison of neural and conventional classifiers on a speech recognition problem Comparison of neural and conventional classifiers on a speech recognition problemProceedings of IEE 1st International Conference on Artificial Neural Networks iee-1989.djvu iee-1989.pdf iee-1989.ps.gz text/html 2008-03-12T11:51:29-04:00 http://leon.bottou.org/papers/bordes-2007?rev=1205337089&do=diff Solving MultiClass Support Vector Machines with LaRank Abstract: Optimization algorithms for large margin multiclass recognizers are often too costly to handle ambitious problems with structured outputs and exponential numbers of classes. Optimization algorithms that rely on the full gradient are not effective because, unlike the solution, the gradient is not sparse and is very large. The LaRank algorithm sidesteps this difficulty by relying on a randomized exploration inspired by the perceptr… text/html 2006-12-14T14:12:59-04:00 http://leon.bottou.org/papers/bordes-bottou-2005?rev=1166123579&do=diff The Huller: a simple and efficient online SVM We propose a novel online kernel classifier algorithm that converges to the Hard Margin SVM solution. The same update rule is used to both add and remove support vectors from the current classifier. Experiments suggest that this algorithm matches the SVM accuracies after a single pass over the training examples. This algorithm is attractive when one seeks a competitive classifier with large datasets and limited computing resources. text/html 2010-01-26T10:51:13-04:00 http://leon.bottou.org/papers/bordes-bottou-gallinari-2009?rev=1264521073&do=diff SGD-QN: Careful Quasi-Newton Stochastic Gradient Descent Abstract: The SGDQN algorithm is a stochastic gradient descent algorithm that makes careful use of second-order information and splits the parameter update into independently scheduled components. Thanks to this design, SGDQN iterates nearly as fast as a first-order stochastic gradient descent but requires less iterations to achieve the same accuracy. This algorithm won the “Wild Track” of the first PASCAL Large Scale Learning Cha… text/html 2009-08-21T16:36:10-04:00 http://leon.bottou.org/papers/bordes-ertekin-weston-bottou-2005?rev=1250886970&do=diff Fast Kernel Classifiers with Online and Active Learning Abstract: Very high dimensional learning systems become theoretically possible when training examples are abundant. The computing cost then becomes the limiting factor. Any efficient learning algorithm should at least pay a brief look at each example. But should all examples be given equal attention? This contribution proposes an empirical answer. We first presents an online SVM algorithm based on this premise. LASVM yields competitive… text/html 2008-08-18T14:57:25-04:00 http://leon.bottou.org/papers/bordes-usunier-bottou-2008?rev=1219085845&do=diff Sequence Labelling SVMs Trained in One Pass Abstract: This paper proposes an online solver of the dual formulation of support vector machines for structured output spaces. We apply it to sequence labelling using the exact and greedy inference schemes. In both cases, the per-sequence training time is the same as a perceptron based on the same inference procedure, up to a small multiplicative constant. Comparing the two inference schemes, the greedy version is much faster. It is also amenable to… text/html 2006-12-13T13:58:07-04:00 http://leon.bottou.org/papers/bottou-2000?rev=1166036287&do=diff DjVu: Un Système de Compression d'Images pour la Distribution Réticulaire de Documents Numérisés DjVu: Un Système de Compression d'Images pour la Distribution Réticulaire de Documents NumérisésActes de la Conférence Internationale Francophone sur l'Ecrit et le Document text/html 2007-10-03T17:59:40-04:00 http://leon.bottou.org/papers/bottou-2001?rev=1191448780&do=diff Efficient Conversion of Digital Documents to Multilayer Raster Formats Abstract: How can we turn the description of a digital (i.e., electronically produced) document into something efficient for multilayer raster formats? It is first shown that a foreground/background segmentation without overlapping foreground components can be more efficient for viewing or printing. Then, a new algorithm that prevents overlaps between foreground components while optimizing both the document quality and comp… text/html 2006-04-20T10:57:44-04:00 http://leon.bottou.org/papers/bottou-88b?rev=1145545064&do=diff Reconnaissance de la parole par reseaux connexionnistes Reconnaissance de la parole par reseaux connexionnistesProceedings of Neuro Nimes 88 @inproceedings{bottou-88b, author = {Bottou, {L\'eon}}, title = {Reconnaissance de la parole par reseaux connexionnistes}, pages = {197-218}, booktitle = {Proceedings of Neuro Nimes 88}, address = {Nimes, France}, year = {1988}, url = {http://leon.bottou.org/papers/bottou-88b}, } text/html 2006-04-20T10:57:44-04:00 http://leon.bottou.org/papers/bottou-89?rev=1145545064&do=diff Experiments with Time Delay Networks and Dynamic Time Warping for Speaker Independent Isolated Digit Recognition Experiments with Time Delay Networks and Dynamic Time Warping for Speaker Independent Isolated Digit RecognitionProceedings of EuroSpeech 89 text/html 2006-04-20T10:57:44-04:00 http://leon.bottou.org/papers/bottou-90?rev=1145545064&do=diff Speaker independent isolated digit recognition: Multilayer perceptron vs Dynamic Time Warping Speaker independent isolated digit recognition: Multilayer perceptron vs Dynamic Time WarpingNeural Networks @article{bottou-90, author = {Bottou, {L\'eon} and Fogelman Souli\'e, Fran\c{c}oise and Blanchet, Pascal and Lienard, {Jean Sylvain}}, title = {Speaker independent isolated digit recognition: Multilayer perceptron vs Dynamic Time Warping}, journal = {Neural Networks}, year = {1990}, … text/html 2009-08-09T16:30:48-04:00 http://leon.bottou.org/papers/bottou-91a?rev=1249849848&do=diff Ph. D. Dissertation The long title translates as A theoretical approach to connectionist learning, with applications for speech recognition. This is a relatively big document with a number of relatively new ideas for the time. * Connectionist learning algorithms can be studied as wp>stochastic approximations and more specifically wp>stochastic gradient descent algorithms, possibly involving surrogate loss functions (chapters 2 and 3). * The performance of these algorithms can be studie… text/html 2006-04-20T10:57:44-04:00 http://leon.bottou.org/papers/bottou-91c?rev=1145545064&do=diff Stochastic Gradient Learning in Neural Networks Stochastic Gradient Learning in Neural NetworksProceedings of Neuro-Nîmes 91 nimes-1991.djvu nimes-1991.pdf nimes-1991.ps.gz @inproceedings{bottou-91c, author = {Bottou, {L\'eon}}, title = {Stochastic Gradient Learning in Neural Networks}, booktitle = {Proceedings of Neuro-N\^imes 91}, year = {1991}, address = {Nimes, France}, publisher = {EC2}, url = {http://leon.bottou.org/papers/bottou-91c}, } text/html 2007-10-02T16:52:58-04:00 http://leon.bottou.org/papers/bottou-97?rev=1191358378&do=diff Global Training of Document Processing Systems using Graph Transformer Networks Abstract: We propose a new machine learning paradigm called Graph Transformer Networks that extends the applicability of gradient-based learning algorithms to systems composed of modules that take graphs as inputs and produce graphs as output. Training is performed by computing gradients of a global objective function with respect to all the parameters in the system using a kind of back-propagation procedure. A com… text/html 2006-04-20T10:57:44-04:00 http://leon.bottou.org/papers/bottou-97b?rev=1145545064&do=diff La mise en oeuvre des idées de V. N. Vapnik La mise en oeuvre des idées de V. N. VapnikStatistique et méthodes neuronales modulad-1997.djvu modulad-1997.pdf modulad-1997.ps.gz @incollection{bottou-97b, author = {Bottou, L\'{e}on}, title = {La mise en oeuvre des id\'{e}es de {V.} {N.} {Vapnik}}, booktitle = {Statistique et m\'{e}thodes neuronales}, publisher = {Dunod}, address = {Paris}, year = {1997}, url = {http://leon.bottou.org/papers/bottou-97b}, } text/html 2006-12-13T13:59:02-04:00 http://leon.bottou.org/papers/bottou-98?rev=1166036342&do=diff High Quality Document Image Compression with DjVu High Quality Document Image Compression with DjVuJournal of Electronic Imaging jei-1998.djvu jei-1998.pdf jei-1998.ps.gz @article{bottou-98, author = {Bottou, L\'{e}on and Haffner, Patrick and Howard, Paul G. and Simard, Patrice and Bengio, Yoshua and {Le Cun}, Yann}, title = {High Quality Document Image Compression with {DjVu}}, journal = {Journal of Electronic Imaging}, year = {1998}, volume = {7}, number = {3}, pages = {410… text/html 2006-12-13T13:59:05-04:00 http://leon.bottou.org/papers/bottou-98b?rev=1166036345&do=diff Browsing through High Quality Document Images with DjVu Browsing through High Quality Document Images with DjVuProceedings of IEEE Advances in Digital Libraries'98 adl-1998.djvu adl-1998.pdf adl-1998.ps.gz @inproceedings{bottou-98b, author = {Haffner, Patrick and Bottou, L\'{e}on and Howard, Paul G. and Simard, Patrice and Bengio, Yoshua and {Le Cun}, Yann}, title = {Browsing through High Quality Document Images with DjVu}, booktitle = {Proceedings of IEEE Advances in Digital Librari… text/html 2006-04-20T10:57:44-04:00 http://leon.bottou.org/papers/bottou-98x?rev=1145545064&do=diff Online Algorithms and Stochastic Approximations Online Algorithms and Stochastic ApproximationsOnline Learning and Neural Networks online-1998.djvu online-1998.pdf online-1998.ps.gz @incollection{bottou-98x, author = {Bottou, L\'{e}on}, title = {Online Algorithms and Stochastic Approximations}, booktitle = {Online Learning and Neural Networks}, editor = {Saad, David}, publisher = {Cambridge University Press}, address = {Cambridge, UK}, year = {1998}, url = {http://leon.bott… text/html 2007-07-24T19:14:04-04:00 http://leon.bottou.org/papers/bottou-bengio-95?rev=1185318844&do=diff Convergence Properties of the KMeans Algorithm Convergence Properties of the KMeans AlgorithmAdvances in Neural Information Processing SystemsMIT nips-1995.djvu nips-1995.pdf nips-1995.ps.gz @incollection{bottou-bengio-95, author = {Bottou, L\'eon and Bengio, Yoshua}, title = {Convergence Properties of the KMeans Algorithm}, booktitle = {Advances in Neural Information Processing Systems}, address = {Denver}, volume = {7}, publisher = {MIT Press}, year = {1995}, url = {http:… text/html 2008-12-15T12:04:19-04:00 http://leon.bottou.org/papers/bottou-bousquet-2008?rev=1229360659&do=diff The Tradeoffs of Large Scale Learning Abstract: This contribution develops a theoretical framework that takes into account the effect of approximate optimization on learning algorithms. The analysis shows distinct tradeoffs for the case of small-scale and large-scale learning problems. Small-scale learning problems are subject to the usual approximation--estimation tradeoff. Large-scale learning problems are subject to a qualitatively different tradeoff involving the computational complexity o… text/html 2008-08-18T10:38:39-04:00 http://leon.bottou.org/papers/bottou-bousquet-2008b?rev=1219070319&do=diff Learning Using Large Datasets Abstract: This contribution develops a theoretical framework that takes into account the effect of approximate optimization on learning algorithms. The analysis shows distinct tradeoffs for the case of small-scale and large-scale learning problems. Small-scale learning problems are subject to the usual approximation–estimation tradeoff. Large-scale learning problems are subject to a qualitatively different tradeoff involving the computational complexity of the … text/html 2006-12-13T13:59:54-04:00 http://leon.bottou.org/papers/bottou-cortes-94?rev=1166036394&do=diff Comparison of classifier methods: a case study in handwritten digit recognition Comparison of classifier methods: a case study in handwritten digit recognitionProceedings of the 12th IAPR International Conference on Pattern Recognition, Conference B: Computer Vision & Image Processing. text/html 2006-12-13T14:01:36-04:00 http://leon.bottou.org/papers/bottou-cortes-vapnik-94?rev=1166036496&do=diff On the Effective VC Dimension. Abstract: The very idea of an effective Vapnik Chervonenkis (VC) dimension (Vapnik et al, 1994) relies on the hypothesis that the relation between the generalization error and the number of training examples can be expressed by a formula algebraically similar to the VC bound. This hypothesis calls for a serious discussion since the traditional VC bound widely overestimates the generalization error. In this paper we describe an algorithm and data dependent meas… text/html 2007-07-24T19:14:49-04:00 http://leon.bottou.org/papers/bottou-gallinari-90?rev=1185318889&do=diff A Framework for the Cooperation of Learning Algorithms A Framework for the Cooperation of Learning AlgorithmsAdvances in Neural Information Processing Systems nips-1990.djvu nips-1990.pdf nips-1990.ps.gz @incollection{bottou-gallinari-90, author = {Bottou, {L\'eon} and Gallinari, Patrick}, title = {A Framework for the Cooperation of Learning Algorithms}, booktitle = {Advances in Neural Information Processing Systems}, address = {Denver}, publisher = {Morgan Kaufmann}, editor = … text/html 2006-04-20T10:57:44-04:00 http://leon.bottou.org/papers/bottou-gallinari-92?rev=1145545064&do=diff A unified formalism for neural net training algorithms A unified formalism for neural net training algorithmsProceedings of the International Joint Conference on Neural Networks ijcnn-1992.djvu ijcnn-1992.pdf ijcnn-1992.ps.gz @inproceedings{bottou-gallinari-92, author = {Bottou, {L\'{e}on} and Gallinari, Patrick}, title = {A unified formalism for neural net training algorithms}, booktitle = {Proceedings of the International Joint Conference on Neural Networks}, pages = {7-12}, ye… text/html 2007-12-12T16:55:23-04:00 http://leon.bottou.org/papers/bottou-howard-bengio-98?rev=1197496523&do=diff The Z-Coder Adaptive Binary Coder Abstract: We present the Z-Coder, a new adaptive data compression coder for coding binary data. The Z-Coder is derived from the Golomb run-length coder, and retains most of the speed and simplicity of the earlier coder. The Z-Coder can also be thought of as a multiplication-free approximate arithmetic coder, showing the close relationship between run-length coding and arithmetic cod- ing. The Z-Coder improves upon existing arithmetic coders by its speed and it… text/html 2007-07-24T19:12:32-04:00 http://leon.bottou.org/papers/bottou-lecun-2004?rev=1185318752&do=diff Large Scale Online Learning Abstract: We consider situations where training data is abundant and computing resources are comparatively scarce. We argue that suitably designed online learning algorithms asymptotically outperform any batch learning algorithm. Both theoretical and experimental evidences are presented. text/html 2006-04-20T11:02:45-04:00 http://leon.bottou.org/papers/bottou-lecun-2004a?rev=1145545365&do=diff On-line Learning for Very Large Datasets Excerpt: The main point of this paper is to show that, in situations where the supply of training samples is essentially unlimited, a well designed on-line algorithm converges toward the minimum of the expected cost just as fast as any batch algorithm. In those situations, the convergence speed is mainly limited by the fact that some informative examples have not yet been seen rather than by the fact that the examples already seen have not been fully ex… text/html 2006-08-09T12:47:54-04:00 http://leon.bottou.org/papers/bottou-lecun-2005?rev=1155142074&do=diff Graph Transformer Networks for Image Recognition Abstract: This contribution takes the example of a check reading system to discuss the modeling and estimation issues associated with large scale pattern recognition systems. Graph Transformer Networks for Image RecognitionBulletin of the International Statistical Institute (ISI) text/html 2006-04-20T10:57:44-04:00 http://leon.bottou.org/papers/bottou-lecun-88?rev=1145545064&do=diff SN: A Simulator for Connectionist Models SN: A Simulator for Connectionist ModelsProceedings of NeuroNimes 88 @inproceedings{bottou-lecun-88, author = {Bottou, {L\'eon} and {Le Cun}, Yann}, title = {SN: A Simulator for Connectionist Models}, pages = {371-382}, booktitle = {Proceedings of NeuroNimes 88}, address = {Nimes, France}, year = {1988}, url = {http://leon.bottou.org/papers/bottou-lecun-88}, } text/html 2006-12-13T14:03:07-04:00 http://leon.bottou.org/papers/bottou-lecun-vapnik-1999?rev=1166036587&do=diff Report: Predicting Learning Curves without the Ground Truth Hypothesis Abstract: Upper bounds for the deviation between test error and training error of a learning machine are derived in the case where no probability distribution that generates the examples is assumed to exist. The bounds are data-dependent and algorithm dependent. The result justifies the concept of data-dependent and algorithm dependent VC-dimension. text/html 2007-10-04T13:58:53-04:00 http://leon.bottou.org/papers/bottou-lin-2006?rev=1191520733&do=diff Support Vector Machine Solvers Abstract:Considerable efforts have been devoted to the implementation of efficient optimization method for solving the Support Vector Machine dual problem. This document proposes an historical perspective and an in depth review of the algorithmic and computational issues associated with this problem. text/html 2008-08-18T14:59:41-04:00 http://leon.bottou.org/papers/bottou-mlss-2004?rev=1219085981&do=diff Stochastic Learning This paper summarizes my lecture at the Machine Learning Summer School 2003, Tübingen. Abstract: This contribution presents an overview of the theoretical and practical aspects of the broad family of learning algorithms based on Stochastic Gradient Descent, including Perceptrons, Adalines, K-Means, LVQ, Multi-Layer Networks, and Graph Transformer Networks. text/html 2006-04-20T11:04:07-04:00 http://leon.bottou.org/papers/bottou-murata-2002?rev=1145545447&do=diff Stochastic Approximations and Efficient Learning Excerpt: The analysis of online algorithms is much more difficult than that of ordinary optimization algorithms. Practical successes in signal processing (Widrow and Stearns, 1985) motivated the creation of sophisticated mathematical tools known as {\em stochastic approximations} (Ljung and Soderstrom, 1983; Benveniste, Metivier and Priouret, 1990) [...] The first section describes and illustrates a general framework for neural network learning… text/html 2006-12-13T14:03:46-04:00 http://leon.bottou.org/papers/bottou-pigeon-98?rev=1166036626&do=diff Lossy Compression of Partially Masked Still Images Lossy Compression of Partially Masked Still ImagesProceedings of IEEE Data Compression Conference dcc-mask-1998.djvu dcc-mask-1998.pdf dcc-mask-1998.ps.gz @inproceedings{bottou-pigeon-98, author = {Bottou, L\'{e}on and Pigeon, Steven}, title = {Lossy Compression of Partially Masked Still Images}, booktitle = {Proceedings of IEEE Data Compression Conference}, month = {April}, address = {Snowbird, UT}, year = {1998}, note = {Ex… text/html 2006-04-20T11:03:07-04:00 http://leon.bottou.org/papers/bottou-still-2004?rev=1145545387&do=diff Geometric Clustering Using the Information Bottleneck Method Abstract: We argue that K--means and deterministic annealing algorithms for geometric clustering can be derived from the more general Information Bottleneck approach. If we cluster the identities of data points to preserve information about their location, the set of optimal solutions is massively degenerate. But if we treat the equations that define the optimal solution as an iterative algorithm, then a set of “smooth” initial … text/html 2006-04-20T10:57:44-04:00 http://leon.bottou.org/papers/bottou-vapnik-92?rev=1145545064&do=diff Local Learning Algorithms Local Learning AlgorithmsNeural Computation nc-1992.djvu nc-1992.pdf nc-1992.ps.gz @article{bottou-vapnik-92, author = {Bottou, {L\'eon} and Vapnik, Vladimir N.}, title = {Local Learning Algorithms}, journal = {Neural Computation}, year = {1992}, volume = {4(6)}, pages = {888-900}, url = {http://leon.bottou.org/papers/bottou-vapnik-92}, } text/html 2006-04-20T10:57:44-04:00 http://leon.bottou.org/papers/bromley-bentz-93?rev=1145545064&do=diff Signature Verification using a Siamese Time Delay Neural Network Signature Verification using a Siamese Time Delay Neural NetworkInternational Journal of Pattern Recognition and Artificial Intelligence ijprai-1993.djvu ijprai-1993.pdf ijprai-1993.ps.gz text/html 2007-07-24T19:13:09-04:00 http://leon.bottou.org/papers/chapelle-2000?rev=1185318789&do=diff Vicinal Risk Minimization Vicinal Risk MinimizationAdvances in Neural Information Processing SystemsMIT nips-2000.djvu nips-2000.pdf nips-2000.ps.gz @incollection{chapelle-2000, author = {Chapelle, Olivier and Weston, Jason and Bottou , L\'{e}on and Vapnik, Vladimir}, title = {Vicinal Risk Minimization}, booktitle = {Advances in Neural Information Processing Systems}, address = {Denver}, volume = {12}, publisher = {MIT Press}, year = {2000}, url = {http://leon.bottou.org/pa… text/html 2006-04-20T10:59:56-04:00 http://leon.bottou.org/papers/collobert-sinz-weston-bottou-2005?rev=1145545196&do=diff Report: Large Scale Transductive SVM Abstract: We show how the Concave-Convex Procedure can be applied to Transductive SVMs, which traditionally requires solving a combinatorial search problem. This provides for the first time a highly scalable algorithm in the nonlinear case. Detailed experiments verify the utility of our approach. text/html 2009-08-07T10:18:43-04:00 http://leon.bottou.org/papers/collobert-sinz-weston-bottou-2006?rev=1249654723&do=diff Large Scale Transductive SVMs Abstract: We show how the Concave-Convex Procedure can be applied to Transductive SVMs, which traditionally require solving a combinatorial search problem. This provides for the first time a highly scalable algorithm in the nonlinear case. Detailed experiments verify the utility of our approach. text/html 2007-01-31T14:45:08-04:00 http://leon.bottou.org/papers/collobert-weston-bottou-2006?rev=1170272708&do=diff Trading Convexity for Scalability Abstract: Convex learning algorithms, such as Support Vector Machines (SVMs), are often seen as highly desirable because they offer strong practical properties and are amenable to theoretical analysis. However, in this work we show how non-convexity can provide scalability advantages over convexity. We show how concave-convex programming can be applied to produce (i) faster SVMs where training errors are no longer support vectors, and (ii) much faster Transd… text/html 2006-04-20T10:57:44-04:00 http://leon.bottou.org/papers/driancourt-bottou-90?rev=1145545064&do=diff TDNN-Extracted features TDNN-Extracted featuresProceedings of Neuro Nimes 90 @inproceedings{driancourt-bottou-90, author = {Driancourt, Xavier and Bottou, {L\'eon}}, title = {{TDNN}-Extracted features}, booktitle = {Proceedings of Neuro Nimes 90}, publisher = {EC2}, address = {Nimes, France}, year = {1990}, url = {http://leon.bottou.org/papers/driancourt-bottou-90}, } text/html 2006-04-20T10:57:44-04:00 http://leon.bottou.org/papers/driancourt-bottou-91?rev=1145545064&do=diff Learning Vector Quantization, Multi Layer Perceptron and Dynamic Time Warping: Comparison and Cooperation Learning Vector Quantization, Multi Layer Perceptron and Dynamic Time Warping: Comparison and CooperationProceedings of the International Joint Conference on Neural Networks text/html 2006-04-20T10:57:44-04:00 http://leon.bottou.org/papers/driancourt-bottou-91b?rev=1145545064&do=diff Comparison and Cooperation of Several Classifiers Comparison and Cooperation of Several ClassifiersProceedings of the International Conference on Artificial Neural Networks @inproceedings{driancourt-bottou-91b, author = {Driancourt, Xavier and Bottou, {L\'eon} and Gallinari, Patrick}, title = {Comparison and Cooperation of Several Classifiers}, booktitle = {Proceedings of the International Conference on Artificial Neural Networks}, address = {Helsinki}, year = {1991}, url = {htt… text/html 2007-10-04T13:55:19-04:00 http://leon.bottou.org/papers/ertekin-2007?rev=1191520519&do=diff Learning on the Border: Active Learning in Imbalanced Data Classification Abstract: This paper is concerned with the class imbalance problem which has been known to hinder the learning performance of classification algorithms. The problem occurs when there are significantly less number of observations of the target concept. Various real-world classification tasks, such as medical diagnosis, text categorization and fraud detection suffer from this phenomenon. The standard machine learning algorith… text/html 2007-07-24T19:21:40-04:00 http://leon.bottou.org/papers/graf-cosatto-2005?rev=1185319300&do=diff Parallel Support Vector Machines: The Cascade SVM Abstract: We describe an algorithm for support vector machines (SVM) that can be parallelized efficiently and scales to very large problems with hundreds of thousands of training vectors. Instead of searching the whole training set for support vectors in one optimization step, the data are split into subsets and analyzed separately with multiple SVMs. The partial results are combined and filtered again in a Cascade of SVMs until the global opti… text/html 2007-07-24T19:14:17-04:00 http://leon.bottou.org/papers/guyon-92?rev=1185318857&do=diff Structural Risk Minimization for Character Recognition Structural Risk Minimization for Character RecognitionAdvances in Neural Information Processing Systems nips-1992.djvu nips-1992.pdf nips-1992.ps.gz @incollection{guyon-92, author = {Guyon, Isabelle and Vapnik, Vladimir N. and Boser, Bernhardt E. and Bottou, {L\'eon} and Solla, Sara A.}, title = {Structural Risk Minimization for Character Recognition}, booktitle = {Advances in Neural Information Processing Systems}, volume = {4… text/html 2006-12-13T14:04:50-04:00 http://leon.bottou.org/papers/guyon-vapnik-92?rev=1166036690&do=diff Capacity control in linear classifiers for pattern recognition Capacity control in linear classifiers for pattern recognitionProceedings of the 11th IAPR International Conference on Pattern Recognition, Conference B: Pattern Recognition Methodology and Systems text/html 2006-12-13T14:05:41-04:00 http://leon.bottou.org/papers/haffner-2002?rev=1166036741&do=diff A General Segmentation Scheme for DjVu Document Compression Abstract: We describe the “DjVu” technology: an efficient document image compression methodology, a file format, and a delivery platform that together, enable instant access to high quality documents from essentially any platform, over any connection. Originally developed for scanned color documents, it was recently expanded to electronic documents, so DjVu has now truly become a universal document interchange format. With DjVu, … text/html 2006-04-20T10:57:44-04:00 http://leon.bottou.org/papers/haffner-99?rev=1145545064&do=diff DjVu : Analyzing and Compressing Scanned Documents for Internet Distribution. DjVu : Analyzing and Compressing Scanned Documents for Internet Distribution. Proceedings of the International Conference on Document Analysis and Recognition.-- icdar-1999.djvu icdar-1999.pdf icdar-1999.ps.gz text/html 2006-12-13T14:06:03-04:00 http://leon.bottou.org/papers/haffner-99k?rev=1166036763&do=diff Color Documents on the Web with DjVu Color Documents on the Web with DjVuProceedings of the International Conference on Image Processing icip-1999.djvu icip-1999.pdf icip-1999.ps.gz @inproceedings{haffner-99k, author = {Haffner, Patrick and {Le Cun}, Yann and Bottou, L\'{e}on and Howard, Paul and Vincent, Pascal}, title = {Color Documents on the Web with {DjVu}}, booktitle = {Proceedings of the International Conference on Image Processing}, volume = {1}, pages = {239-243}, addr… text/html 2006-12-13T12:12:48-04:00 http://leon.bottou.org/papers/haskell-98?rev=1166029968&do=diff Image and video coding - Emerging standards and beyond Image and video coding - Emerging standards and beyondIEEE Transaction on Circuits and Systems for Video Technology cstv-1998.djvu cstv-1998.pdf cstv-1998.ps.gz @article{haskell-98, author = {Haskell, Barry and Howard, Paul and{Le Cun}, Yann and Puri , Atul and Ostermann, Joern and Civanlar, M. Reha and Rabiner , Larry and Bottou, L\'{e}on and Haffner, Patrick}, title = {Image and video coding - Emerging standards and beyond}, jo… text/html 2009-07-24T16:42:44-04:00 http://leon.bottou.org/papers/icml-2009?rev=1248468164&do=diff Proceedings of the 26th International Conference on Machine Learning (ICML 2009) Léon Bottou and Michael Littman This volume contains the papers accepted to the 26th International Conference on Machine Learning (ICML 2009). ICML is the annual conference of the International Machine Learning Society (IMLS) and provides a venue for the presentation and discussion of current research in the field of machine learning. ICML 2009 was held June 14–18 on the downtown campus of McGill University in… text/html 2006-12-13T14:07:18-04:00 http://leon.bottou.org/papers/lecun-2001?rev=1166036838&do=diff DjVu document browsing with on-demand loading and rendering of image components Abstract: Image-based digital documents are composed of multiple pages, each of which may be composed of multiple components such as the text, pictures, background, and annotations. We describe the image structure and software architecture that allows the DjVu system to load and render the required components on demand while minimizing the bandwidth requirements, and the memory requirements in the client. DjVu d… text/html 2006-04-20T10:57:45-04:00 http://leon.bottou.org/papers/lecun-2001b?rev=1145545065&do=diff Overview of the DjVu Document Compression Technology Overview of the DjVu Document Compression TechnologyProceedings of the Symposium on Document Image Understanding Technologies (SDIUT'01)-- sdiut-2001.djvu sdiut-2001.pdf sdiut-2001.ps.gz @inproceedings{lecun-2001b, author = {{Le Cun}, Yann and Bottou, L\'{e}on and Haffner, Patrick and Triggs, Jeffery and Riemers, Bill and Vincent, Luc}, title = {Overview of the {DjVu} Document Compression Technology}, booktitle = {Proceedings of t… text/html 2006-04-20T10:57:45-04:00 http://leon.bottou.org/papers/lecun-95a?rev=1145545065&do=diff Learning Algorithms For Classification: A Comparison On Handwritten Digit Recognition Learning Algorithms For Classification: A Comparison On Handwritten Digit RecognitionNeural Networks: The Statistical Mechanics Perspective nnsmp-1995.djvu nnsmp-1995.pdf nnsmp-1995.ps.gz text/html 2006-12-13T14:07:59-04:00 http://leon.bottou.org/papers/lecun-97?rev=1166036879&do=diff Reading Checks with graph transformer networks Reading Checks with graph transformer networksInternational Conference on Acoustics, Speech, and Signal Processing icassp-1997.djvu icassp-1997.pdf icassp-1997.ps.gz @inproceedings{lecun-97, author = {{Le Cun}, Yann and Bottou, L\'{e}on and Bengio, Yoshua}, title = {Reading Checks with graph transformer networks}, booktitle = {International Conference on Acoustics, Speech, and Signal Processing}, volume = {1}, publisher = {IEEE}, a… text/html 2006-12-13T14:08:22-04:00 http://leon.bottou.org/papers/lecun-98c?rev=1166036902&do=diff DjVu: a compression method for distributing scanned documents in color over the internet DjVu: a compression method for distributing scanned documents in color over the internetSixth Color Imaging Conference: Color Science, Systems and Applications text/html 2006-12-13T12:00:20-04:00 http://leon.bottou.org/papers/lecun-98h?rev=1166029220&do=diff Gradient Based Learning Applied to Document Recognition Gradient Based Learning Applied to Document RecognitionProceedings of IEEE ieee-1998.djvu ieee-1998.pdf ieee-1998.ps.gz @article{lecun-98h, author = {{Le Cun}, Yann and Bottou, L\'{e}on and Bengio, Yoshua and Haffner, Patrick}, title = {Gradient Based Learning Applied to Document Recognition}, journal = {Proceedings of IEEE}, volume = {86}, number = {11}, pages = {2278-2324}, year = {1998}, url = {http://leon.bottou.or… text/html 2007-08-17T14:39:25-04:00 http://leon.bottou.org/papers/lecun-98x?rev=1187375965&do=diff Efficient Backprop Abstract: The convergence of back-propagation learning is analyzed so as to explain common phenomenon observed by practitioners. Many undesirable behaviors of backprop can be avoided with tricks that are rarely exposed in serious technical publications. This paper gives some of those tricks, and offers explanations of why they work. Many authors have suggested that second-order optimization methods are advantageous for neural net training. It is shown that most “classical” s… text/html 2006-12-13T14:08:48-04:00 http://leon.bottou.org/papers/lecun-99f?rev=1166036928&do=diff Object Recognition with Gradient-Based Learning Object Recognition with Gradient-Based LearningFeature Grouping features-1999.djvu features-1999.pdf features-1999.ps.gz @incollection{lecun-99f, author = {{Le Cun}, Yann and Haffner, Patrick and Bottou, L\'{e}on and Bengio, Yoshua}, title = {Object Recognition with Gradient-Based Learning}, booktitle = {Feature Grouping}, editor = {Forsyth, David}, publisher = {Springer Verlag}, year = {1999}, url = {http://leon.bottou.org/pape… text/html 2006-04-20T11:02:34-04:00 http://leon.bottou.org/papers/lecun-bottou-huangfu-2004?rev=1145545354&do=diff Learning Methods for Generic Object Recognition with Invariance to Pose and Lighting Abstract: We assess the applicability of several popular learning methods for the problem of recognizing generic visual categories with invariance to pose, lighting, and surrounding clutter. A large dataset comprising stereo image pairs of 50 uniform-colored toys under 36 angles, 9 azimuths, and 6 lighting conditions was collected (for a total of 194,400 individual images). The objects were 10 instances … text/html 2008-04-23T10:53:42-04:00 http://leon.bottou.org/papers/loosli-canu-bottou-2006?rev=1208962422&do=diff Training Invariant Support Vector Machines using Selective Sampling Abstract: Bordes et al (2005) describe the efficient online LASVM algorithm using selective sampling. On the other hand, Loosli et al. (2005) propose a strategy for handling invariance in SVMs, also using selective sampling. This paper combines the two approaches to build a very large SVM. We present state-of-the-art results obtained on a handwritten digit recognition problem with 8 millions examples on a single processor.… text/html 2007-09-26T14:34:12-04:00 http://leon.bottou.org/papers/lskm-2007?rev=1190831652&do=diff Large Scale Kernel Machines Léon Bottou, Olivier Chapelle, Dennis DeCoste, Jason Weston. Pervasive and networked computers have dramatically reduced the cost of collecting and distributing large datasets. In this context, machine learning algorithms that scale poorly could simply become irrelevant. We need learning algorithms that scale linearly with the volume of the data while maintaining enough statistical efficiency to outperform algorithms that simply process a random subset of the dat… text/html 2006-04-20T10:57:45-04:00 http://leon.bottou.org/papers/matic-guyon-92?rev=1145545065&do=diff Computer Aided Cleaning of Large Databases for Character Recognition Computer Aided Cleaning of Large Databases for Character RecognitionProceedings of the 11th IAPR International Conference on Pattern Recognition, Conference B: Pattern Recognition Methodology and Systems text/html 2006-04-20T10:57:45-04:00 http://leon.bottou.org/papers/mejia-90?rev=1145545065&do=diff Galatea: A library for connectionist applications Galatea: A library for connectionist applicationsProceedings of the International Neural Networks Conference, INNC'90 @inproceedings{mejia-90, author = {Mejia, Carlos and Bottou, {L\'eon} and Fogelman Souli\'e, Fran\c{c}oise}, title = {Galatea: A library for connectionist applications}, booktitle = {Proceedings of the International Neural Networks Conference, INNC'90}, address = {Paris}, volume = {1}, pages = {9-13}, year = {19… text/html 2006-04-20T11:06:21-04:00 http://leon.bottou.org/papers/mikheev-2002?rev=1145545581&do=diff Electronic Document Publishing using DjVu Abstract: Online access to complex compound documents with client side search and browsing capability is one of the key requirements of effective content management. “DjVu” is a highly efficient document image compression methodology, a file format, and a delivery platform that, when considered together, has shown to effectively address these issues. Originally developed for scanned color documents, the DjVu technology was recently expanded to e… text/html 2006-04-20T11:01:06-04:00 http://leon.bottou.org/papers/ning-delhomme-2005?rev=1145545266&do=diff Toward Automatic Phenotyping of Developing Embryos from Videos Abstract: We describe a trainable system for analyzing videos of developing C. elegans embryos. The system automatically detects, segments, and locates cells and nuclei in microscopic images. The system was designed as the central component of a fully-automated phenotyping system. The system contains three modules (1) a convolutional network trained classify each pixel into five categories: cell wall, cytoplasm, nucleus membrane, … text/html 2007-07-24T18:26:02-04:00 http://leon.bottou.org/papers/nips-2005?rev=1185315962&do=diff Advances in Neural Information Processing Systems 17 Lawrence Saul, Yair Weiss and Léon Bottou. This volume contains the papers presented at the eighteenth annual conference on Neural Information Processing Systems (NIPS), held in British Columbia, Canada from December 13 through 16, 2004. NIPS is a unique, interdisciplinary, conference that brings together researchers from a wide variety of fields including computer science, neuroscience, physics, cognitive science, psychology, engineerin… text/html 2009-07-29T08:07:09-04:00 http://leon.bottou.org/papers/nips-2009?rev=1248869229&do=diff Advances in Neural Information Processing Systems 21 Daphne Koller, Yoshua Bengio, Léon Bottou, Aron Culotta This volume contains the papers presented at the twenty-second annual conference on Neural Information Processing Systems (NIPS), held in British Columbia, Canada from December 8th through 11th, 2008. In agreement with the computational nature of NIPS, NIPS’0 was the first conference, and this book is entitled Advances in Neural Information Processing Systems 21. NIPS is a premier… text/html 2006-12-13T14:10:28-04:00 http://leon.bottou.org/papers/pan-2003?rev=1166037028&do=diff An Iterative Algorithm for Accurate Motion Estimation in Very Low Bit Rate Video Coding Abstract: In video coding, accurate motion estimation is very important since good temporal prediction can significantly eliminate temporal redundancy and save bits in coding the motion. In block-based motion estimation systems, we can increase the estimation accuracy by using smaller block sizes. However, more bits are required to code the motion information due to increased number of blocks. It is desirab… text/html 2006-04-20T10:57:45-04:00 http://leon.bottou.org/papers/reibman-2001?rev=1145545065&do=diff DCT-based scalable video coding with drift DCT-based scalable video coding with driftProceedings of International Conference on Image Processing 2001 icip-2001.djvu icip-2001.pdf icip-2001.ps.gz @inproceedings{reibman-2001, author = {Reibman, Amy R. and Bottou, L\'{e}on and Basso, Andrea}, title = {{DCT}-based scalable video coding with drift}, booktitle = {Proceedings of International Conference on Image Processing 2001}, address = {Thessaloniki, Greece}, month = {October}, pu… text/html 2006-04-20T11:03:38-04:00 http://leon.bottou.org/papers/reibman-2003?rev=1145545418&do=diff Scalable video coding with managed drift Abstract: Traditional scalable video encoders sacrifice coding efficiency to reduce error propagation because they have avoided using enhancement-layer (EL) information to predict the base layer (BL) to prevent the error propagation termed “drift”. Drift can produce very poor video quality if left unchecked. In this paper, we propose a video coder with significantly better compression efficiency because it intentionally allows the drift produced by… text/html 2006-04-20T11:12:10-04:00 http://leon.bottou.org/papers/reibman-bottou-2001?rev=1145545930&do=diff Managing drift in DCT-based scalable video coding Abstract: When compressed video is transmitted over erasure-prone channels, errors will propagate whenever temporal or spatial prediction is used. Typical tools to combat this error propagation are packetization, re-synchronizing codewords, intra-coding, and scalability. In recent years, the concern over so- called “drift” has sent researchers toward structures for scalability that do not use enhancement-layer information to predict base-l… text/html 2007-07-24T19:13:53-04:00 http://leon.bottou.org/papers/simard-99?rev=1185318833&do=diff Boxlets: a Fast Convolution Algorithm for Neural Networks and Signal Processing Abstract: Signal processing and pattern recognition algorithms make extensive use of convolution. In many cases, computational accuracy is not as important as computational speed. In feature extraction, for instance, the features of interest in a signal are usually quite distorted. This form of noise justifies some level of quantization in order to achieve faster feature extraction. Our approach consists of approx… text/html 2008-08-18T10:43:29-04:00 http://leon.bottou.org/papers/sonnenburg-2007?rev=1219070609&do=diff The Need for Open Source Software in Machine Learning Abstract: Open source tools have recently reached a level of maturity which makes them suitable for building large-scale real-world systems. At the same time, the field of machine learning has developed a large body of powerful learning algorithms for diverse applications. However, the true potential of these methods is not used, since existing implementations are not openly shared, resulting in software with low usability, and weak interop… text/html 2006-04-20T10:57:45-04:00 http://leon.bottou.org/papers/vapnik-bottou-93?rev=1145545065&do=diff Local Algorithms for Pattern Recognition and Dependencies Estimation Local Algorithms for Pattern Recognition and Dependencies EstimationNeural Computation nc-1993.djvu nc-1993.pdf nc-1993.ps.gz @article{vapnik-bottou-93, author = {Vapnik, Vladimir N. and Bottou, {L\'eon}}, title = {Local Algorithms for Pattern Recognition and Dependencies Estimation}, journal = {Neural Computation}, year = {1993}, volume = {5(6)}, pages = {893-909}, url = {http://leon.bottou.org/papers/vapni… text/html 2006-04-20T11:01:30-04:00 http://leon.bottou.org/papers/weston-bordes-bottou-2005?rev=1145545290&do=diff Online (and Offline) on an Even Tighter Budget Abstract: We develop a fast online kernel algorithm for classification which can be viewed as an improvement over the one suggested by (Crammer et al., 2003). In that previous work, the authors introduced an on-the-fly compression of the number of examples used in the prediction function using the size of the margin as a quality measure. Although displaying impressive results on relatively noise-free data we show how their algorithm is susceptib… text/html 2006-12-13T14:11:40-04:00 http://leon.bottou.org/papers/weston-collobert-sinz-bottou-vapnik-2006?rev=1166037100&do=diff Inference with the Universum Abstract: In this paper we study a new framework introduced by Vapnik (1998; 2006) that is an alternative capacity concept to the large margin approach. In the particular case of binary classification, we are given a set of labeled examples, and a collection of “non-examples” that do not belong to either class of interest. This collection, called the Universum, allows one to encode prior knowledge by representing meaningful concepts in the same domain as the pro…