http://leon.bottou.org/ 2008-07-05T06:53:10-04:00 http://leon.bottou.org/ http://leon.bottou.org/lib/images/favicon.ico text/html 2008-04-23T10:53:42-04:00 Leon Bottou http://leon.bottou.org/papers/loosli-canu-bottou-2006?rev=1208962422&do=diff1208962422 Training Invariant Support Vector Machines using Selective Sampling Abstract: Bordes et al (2005) describe the efficient online LASVM algorithm using selective sampling. On the other hand, Loosli et al. (2005) propose a strategy for handling invariance in SVMs, also using selective sampling. This paper combines the two approaches to build a very large SVM. We present state-of-the-art results obtained on a handwritten digit recognition problem with 8 millions examples on a single processor.… text/html 2008-04-07T18:41:40-04:00 Leon Bottou http://leon.bottou.org/projects/sgd?rev=1207608100&do=diff1207608100 Stochastic Gradient Descent (SGD) has been historically associated with back-propagation algorithms in multilayer neural networks. These nonlinear nonconvex problems can be very difficult. Therefore it is useful to see how Stochastic Gradient Descent performs on simple linear and convex problems such as linear Support Vector Machines (SVMs) or Conditional Random Fields (CRFs). text/html 2008-03-12T11:51:29-04:00 Leon Bottou http://leon.bottou.org/papers/bordes-2007?rev=1205337089&do=diff1205337089 Solving MultiClass Support Vector Machines with LaRank Abstract: Optimization algorithms for large margin multiclass recognizers are often too costly to handle ambitious problems with structured outputs and exponential numbers of classes. Optimization algorithms that rely on the full gradient are not effective because, unlike the solution, the gradient is not sparse and is very large. The LaRank algorithm sidesteps this difficulty by relying on a randomized exploration inspired by the perceptr… text/html 2008-03-12T11:51:03-04:00 Leon Bottou http://leon.bottou.org/news/larank_code?rev=1205337063&do=diff1205337063 Antoine Bordes provides an implementation of the LaRank algorithm, together with the datasets. This new implementation runs slightly faster than the code we have used for the LaRank paper. In addition there is a special version for the case of linear kernels. text/html 2008-02-19T00:20:43-04:00 Leon Bottou http://leon.bottou.org/start?rev=1203398443&do=diff1203398443 Léon Bottou NEC Laboratories America 4 Independence Way, Suite 200 Princeton, NJ 08540, USA Email: <myfirstname@bottou.org> Phone: 609-951-2732 I am a research scientist with NEC Labs America's Machine Learning group in Princeton. Although my main main interest is practical and theoretical machine learning, my best known work is probably the DjVu document compression system. text/html 2008-01-15T16:23:16-04:00 Leon Bottou http://leon.bottou.org/talks?rev=1200432196&do=diff1200432196 This page contains pointers to my most significant lectures. All the slides are available under both the PDF and DjVu formats. * The BackPropagation CookBook * Graph Transformer Networks * DjVu: Scanned Documents on the Web * Online Learning and Stochastic Approximations * The Tradeoffs of Large Scale Learning text/html 2008-01-15T16:13:08-04:00 Leon Bottou http://leon.bottou.org/talks/largescale?rev=1200431588&do=diff1200431588 This lecture was prepared for the NIPS 2007 tutorial. Variants of this lecture were give at IPAM Google, Microsoft Research, and the International Conference of Nonconvex Programming, NCP07. Summary Pervasive and networked computers have reduced the cost of collecting and distributing large-scale datasets. Since usual machine learning algorithms demand computing times that grow faster than the volume of the data, computing time is now the bottleneck in real life applications. text/html 2008-01-15T16:12:04-04:00 Leon Bottou http://leon.bottou.org/research/stochastic?rev=1200431524&do=diff1200431524 This is a subtopic of Large-Scale Learning. Basics Many numerical learning algorithms amount to optimizing a cost function that can be expressed as an average over the training examples. The loss function measures how well (or how poorly) the learning system performs on each example. The cost function is then the average of the loss function measures on all training examples, possibly augmented with capacity control terms. text/html 2008-01-15T16:06:15-04:00 Leon Bottou http://leon.bottou.org/news/nips07_tutorial?rev=1200431175&do=diff1200431175 A page has been allocated for my segment of the NIPS 2007 Tutorials. The second part of the tutorial Learning with Large Datasets was given by Alex Gray. Alex had to replace Andrew Moore on short notice because airplane delays conspired against our initial plans. The page contains the slides and a video recording a the lecture I gave at Microsoft Research a few days after NIPS. text/html 2008-01-15T16:02:14-04:00 Leon Bottou http://leon.bottou.org/research/approxalgo?rev=1200430934&do=diff1200430934 This is a subtopic of Large-Scale Learning. Basics This work develops a theoretical framework that takes into account the effect of approximate optimization on learning algorithms. The analysis shows distinct tradeoffs for the case of small-scale and large-scale learning problems. Small-scale learning problems are subject to the usual approximation--estimation tradeoff. Large-scale learning problems are subject to a qualitatively different tradeoff involving the computational complexity o…