Skip to content
AI Digested
History2006 to 20143 min read

AlexNet: the 2012 breakthrough that started the deep learning boom

How a neural network trained on two gaming graphics cards in a Toronto bedroom won the 2012 ImageNet competition by a huge margin, and set off the deep learning boom behind today's AI.

First published · AI Digested

Every modern AI story, from ChatGPT to self-driving cars, traces back to a single result announced in September 2012. A neural network called AlexNet won an image recognition competition, not narrowly, but by a margin so large it changed the direction of an entire field.

The problem: getting computers to see

For decades, getting a computer to recognise what's in a photo was painfully hard. Researchers hand-designed features, like edges, corners and colour patterns, and fed them into learning algorithms. Progress was slow and incremental.

Neural networks, which could in theory learn their own features, had been out of fashion since the AI winters. A small group of researchers kept faith, most famously Geoffrey Hinton in Toronto, Yann LeCun in New York and Yoshua Bengio in Montreal.

Ingredient one: ImageNet

In 2006, a young professor named Fei-Fei Li began building something unfashionable: not a better algorithm, but a better dataset. Her team collected millions of images from the internet and paid thousands of online workers through Amazon Mechanical Turk to label them.

ImageNet, presented in 2009, grew to more than 14 million labelled images. From 2010, an annual contest challenged teams to classify images into 1,000 categories, from "container ship" to "Siamese cat".

Ingredient two: graphics cards

Training neural networks involves vast amounts of simple arithmetic, most of which can be done in parallel. GPUs, graphics processing units built by companies like Nvidia to render video games, happen to be brilliant at exactly that. In 2007, Nvidia released CUDA, software that made it practical to use GPUs for general calculations.

Ingredient three: better training tricks

Hinton's group and others had found ways to train deeper networks: new activation functions (ReLU) that sped up learning, and a technique called dropout that stopped networks memorising their training data.

AlexNet

Hinton's student Alex Krizhevsky built a deep convolutional neural network with eight learning layers and about 60 million parameters. He trained it on two Nvidia GeForce GTX 580 graphics cards, gaming cards that cost around $500 each, running for about a week. Fellow student Ilya Sutskever pushed the project forward, and Hinton supervised.

The results in the 2012 ImageNet contest:

Entry Top-5 error rate
AlexNet (Toronto) 15.3%
Next best entry 26.2%

("Top-5 error" means the right answer wasn't among the model's five best guesses.) In a field used to improvements of a percentage point or two, it was a thunderclap.

The gold rush

Things moved fast:

  • 2013: Hinton, Krizhevsky and Sutskever's newly founded company, DNNresearch, was auctioned. Google won, reportedly for $44 million, beating Baidu, Microsoft and DeepMind. Hinton split his time between Google and Toronto for the next decade.
  • 2014: Google bought DeepMind. Facebook had opened an AI lab led by Yann LeCun the year before.
  • 2015: A deep network from Microsoft Research beat estimated human performance on ImageNet.
  • 2015: Ilya Sutskever left Google to co-found OpenAI.

Deep learning soon took over speech recognition, translation, recommendation systems and more. Nvidia, the maker of those gaming cards, transformed into the world's most valuable chipmaker.

Recognition

In 2018, Hinton, LeCun and Bengio received the Turing Award, computing's highest honour, as the "fathers of the deep learning revolution". In 2024, Hinton shared the Nobel Prize in Physics with John Hopfield.

Hinton has since become one of the most prominent voices warning about AI's risks. He left Google in 2023, saying he wanted to speak freely about the dangers of the technology he helped create.

Why it matters

AlexNet proved a simple, powerful recipe: big data, plus big computing power, plus deep neural networks, produces results nobody could hand-engineer. That same recipe, scaled up enormously and combined with the transformer, produced the chatbots we use today.

Read on

Common questions

What is ImageNet?

A huge database of labelled photos created by Fei-Fei Li and colleagues, first presented in 2009. Its annual competition asked programs to sort about 1.2 million training images into 1,000 categories, then label new ones.

Why were graphics cards so important?

Training a neural network involves enormous numbers of simple calculations that can be done at the same time. Graphics cards, built to colour millions of pixels at once for video games, turned out to be perfect for it, and far cheaper than supercomputers.

What happened to the AlexNet team?

Their company, DNNresearch, was bought by Google in 2013. Ilya Sutskever went on to co-found OpenAI in 2015. Geoffrey Hinton shared the 2018 Turing Award and the 2024 Nobel Prize in Physics.

Sources and further reading

  1. Krizhevsky, Sutskever & Hinton (2012). ImageNet Classification with Deep Convolutional Neural Networks
  2. Deng et al. (2009). ImageNet: A Large-Scale Hierarchical Image Database
  3. ACM: Fathers of the Deep Learning Revolution Receive 2018 Turing Award

Spotted a mistake or something out of date? Tell us and we will fix it. Read how we research and write our articles.