Neuristique s.a.

Neuristique was founded in 1988 by a dozen friends with big dreams. The mission statement was very long sentence that mentions the application of artificial neural networks, the development of artificial brains, and the exploration of space. We were very young and inexperienced.

This adventure had positive developments. Neuristique was in business from 1988 to 2003. The company was closed “in the black” because it structure was becoming too heavy for its activities. The technology developed by Neuristique is still used by leading companies around the world. The human side was more painful. The contrast between our inexperience and our ambitions has caused many disappointments. Friendships have been severed…

Neuristique's first product was the SN neural network simulation software. Yann LeCun and I initially wrote SN to support our research. We wanted a prototyping platform able to support ambitious applications of machine learning such as optical character recognition or speech recognition.

SN was basically a Lisp interpreter with highly optimized numerical routines for simulating and training neural networks. Lisp was chosen because we could write a compact and yet powerful interpreter without compromising the speed of the system. The very first version of SN already had a very fast stochastic gradient descent and convolutionnal neural networks. Thanks to many people, successive versions of SN acquired many features and libraries.

Leading R&D organizations around the world bought SN licenses. AT&T Bell Laboratories had its own internal version. SN programs were industrially deployed and are still running.

Around 1997, Neuristique released a version of the Lisp interpreter named TL3 under the GPL. In 2001, AT&T Labs released most of its modifications under the GPL. Yann and I then gathered all these incompatible freely available pieces and produced the latest descendant of SN: Lush. Almost twenty years after the initial version of SN, Lush is still going strong!

After writing my thesis and working with Vapnik in AT&T_Bell_Laboratories, it was clear that Machine Learning was a scientific revolution in the making. We had practical successes and we were quickly acquiring a theoretical understanding. What were the business aspects of this revolution?

A few years earlier in Ecole Poytechnique, Claude Henry gave a wonderful class about micro-economical models of public economy. For instance, when you build a toll bridge, the cost must be paid before a single car passes the bridge. Once the bridge is build, allowing an extra car costs nearly nothing. It seems reasonable to allow any extra car provided that the driver pays more than this small marginal cost. On the other hand, if everybody pays a toll price equal to this marginal costs, that will never pay for the fixed cost of building the bridge. The best way to recover the fixed cost would be to guess how much each driver is willing to pay for the benefit of passing the bridge and charging that. Therefore one designs complicated toll price structure based on a priori discrimination: commercial vehicles pay more, peak hours are more expensive, etc. In mathematical terms, the market pricing is undermined by the nonconvex relation between the number of billable units and the total cost of the infrastructure. This applies to much more than building bridges.

The nonconvexity appears whenever an activity benefits from economy of scale. Airline pricing is an extreme example. Yet things as different as software pricing, telecommunication pricing obey similar rules. Internet advertisement is a minor variant of that phenomenon: advertisers name their price and internet companies decide the level of service, that is, they decide where and when to show the advertisement. All these businesses have in common that markets cannot determine a simple relation between asked price and level of services. In order to deploy other pricing mechanisms, these businesses must rely on statistical data collected during their operation. I am still convinced that this is the fundamental driver of the Data Mining industry.

The TL/Prevision project took shape when I realized that this was an powerful business case for machine learning. The idea was to develop an automated machine learning package able to collect data from the businesses'SQL databases and perform various predictive tasks. The internal algorithms were based on Locally Regularized Learning Systems and optimized using Hoeffding races (in fact TL/Prevision performs Bennet races) The software package was implemented by Neuristique using seed funds from ANVAR and was applied to a number of problems such as call center optimization, water demand prediction, and traffic prediction.

See some slides about the story of TL/Prevision.

Despite our recognized technological leadership, the operation of Neuristique was dependent on the goodwill of its scattered founders. No decision could be taken without the consensus of its active founders. Every small disagreement took existential proportions. That was exhausting. I left Neuristique in 2000. Xavier Driancourt (http://www.orlogia.com) ran it profitably until 2003 when we decided that its business activity no longer required the relatively costly legal structure of the company. Xavier still runs Orlogia, a consulting business.

Despite these disappointements, the legacy of Neuristique is very positive:

• Numerous machine learning algorithms have been developed with SN, including convolutionnal neural networks for image recognition or signal processing, and early implementations of Support Vector Machines.

• A SN based handwriting recognition system was used by many banks across the world to read checks. Some ATM machines made by NCR use compiled SN code running on embedded DSP boards. A version of this check reader was embedded in large check reading engines sold by NCR and OrboGraph. According to some estimates, this system processes 10 to 20 percent of all the checks written in the US.

• The first prototype of the DjVu image and document compression system, including the first decoder and the first foreground/background segmenter.

• The TL/Prevision technology was licensed to KXEN Inc. and helped defining their initial view of the business.

Here is a non exhaustive list of the Neuristique¹⁾ collaborators.

• Jean Bourrely (thesis)
• Michel Diricq
• Xavier Driancourt (http://www.orlogia.com)
• Christian and Marie-Pia Ignace
• Yann LeCun (http://yann.lecun.com)
• Steve Suddarth (pubs)

See also http://www.neuristique.fr.