Skip to content
AI Digested
History20173 min read

'Attention Is All You Need': the 2017 paper behind ChatGPT

How eight Google researchers invented the transformer in 2017 to improve translation, and accidentally created the technology behind ChatGPT, Claude, Gemini and nearly all modern AI.

First published · AI Digested

In June 2017, a research paper with a cheeky title, a nod to the Beatles' "All You Need Is Love", appeared online. "Attention Is All You Need" was written by eight researchers at Google, and it was about improving machine translation. It became one of the most cited scientific papers of the century, because the design it described, the transformer, turned out to power almost every modern AI system.

The problem with reading in order

Before 2017, the best language AI used recurrent neural networks. They read text the way you might read a sentence through a keyhole: one word at a time, carrying a summary of everything so far.

That had two big problems:

  1. They were slow to train. Each word had to wait for the one before it, so you couldn't make full use of GPUs, which are brilliant at doing many calculations at once.
  2. They forgot. By the end of a long paragraph, details from the beginning had faded.

Researchers had added a helper technique called attention, letting the network glance back at earlier words when it needed to. The Google team's radical idea was to throw out the recurrent part entirely and build a model from attention alone.

How attention works, simply

Read this sentence:

"The animal didn't cross the street because it was too tired."

What does "it" refer to? You know instantly: the animal. Now change one word:

"The animal didn't cross the street because it was too wide."

Now "it" is the street. To understand a word, you need to look at the other words around it, and decide which ones matter.

Attention does exactly this. For every word, the model calculates how much it should "pay attention" to every other word in the passage, then blends their information together. It does this for all the words at the same time, and in many different ways at once, called "multi-head attention", so it can track grammar, meaning and references simultaneously.

Stack many layers of this, and the model builds up an increasingly rich understanding of the text.

Why it changed everything

Speed. Because every word is processed in parallel, transformers train dramatically faster on GPUs. The original model reached state-of-the-art translation results after training for 3.5 days on eight GPUs, a fraction of the cost of previous best models.

Scale. Faster training meant you could train much bigger models on much more data. And it turned out that transformers kept getting better as they got bigger, with no obvious ceiling in sight.

Generality. The same design worked for almost anything that could be turned into a sequence: text, code, images cut into patches, audio, DNA.

From translation to ChatGPT

  • 2018: OpenAI released GPT, a "generative pre-trained transformer" that learned by predicting the next word in books. Google released BERT, a transformer that transformed Google Search.
  • 2019: GPT-2 wrote such convincing text that OpenAI initially withheld the full model.
  • 2020: GPT-3, with 175 billion parameters, could perform tasks from just a few examples.
  • 2022: ChatGPT put a transformer in front of the public. Read the story.

Today, ChatGPT, Claude, Gemini, Llama and almost every other major model are transformers. Learn how they work.

The eight authors

The paper lists eight authors, with a note that the order was random and all contributed equally: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser and Illia Polosukhin. Gomez was a 20-year-old intern at the time.

All eight eventually left Google. Between them, they went on to found or help lead companies including Cohere (Gomez), Character.AI (Shazeer, who later returned to Google), Sakana AI (Jones), Essential AI (Vaswani and Parmar), Inceptive (Uszkoreit) and NEAR (Polosukhin). Kaiser joined OpenAI.

Google invented the transformer and published it openly. Other companies, most famously OpenAI, were quickest to see where it could lead.

Read on

Common questions

Why is it called a transformer?

Because it transforms one sequence, such as a sentence in English, into another, such as the same sentence in German. One of the authors, Jakob Uszkoreit, has said the name was chosen partly because it sounded cool.

What came before transformers?

Recurrent neural networks, including a type called LSTM, read text one word at a time, in order. That made them slow to train and bad at remembering things from much earlier in a passage.

Are transformers only used for language?

No. They are now used for images, audio, video, protein structures, weather forecasting and robotics.

Sources and further reading

  1. Vaswani et al. (2017). Attention Is All You Need. arXiv:1706.03762
  2. Google Research blog (2017): Transformer, a novel neural network architecture for language understanding

Spotted a mistake or something out of date? Tell us and we will fix it. Read how we research and write our articles.