'Attention Is All You Need': the 2017 paper behind ChatGPT
How eight Google researchers invented the transformer in 2017 to improve translation, and accidentally created the technology behind ChatGPT, Claude, Gemini and nearly all modern AI.
First published · AI Digested
In June 2017, a research paper with a cheeky title, a nod to the Beatles' "All You Need Is Love", appeared online. "Attention Is All You Need" was written by eight researchers at Google, and it was about improving machine translation. It became one of the most cited scientific papers of the century, because the design it described, the transformer, turned out to power almost every modern AI system.
The problem with reading in order
Before 2017, the best language AI used recurrent neural networks. They read text the way you might read a sentence through a keyhole: one word at a time, carrying a summary of everything so far.
That had two big problems:
- They were slow to train. Each word had to wait for the one before it, so you couldn't make full use of GPUs, which are brilliant at doing many calculations at once.
- They forgot. By the end of a long paragraph, details from the beginning had faded.
Researchers had added a helper technique called attention, letting the network glance back at earlier words when it needed to. The Google team's radical idea was to throw out the recurrent part entirely and build a model from attention alone.
How attention works, simply
Read this sentence:
"The animal didn't cross the street because it was too tired."
What does "it" refer to? You know instantly: the animal. Now change one word:
"The animal didn't cross the street because it was too wide."
Now "it" is the street. To understand a word, you need to look at the other words around it, and decide which ones matter.
Attention does exactly this. For every word, the model calculates how much it should "pay attention" to every other word in the passage, then blends their information together. It does this for all the words at the same time, and in many different ways at once, called "multi-head attention", so it can track grammar, meaning and references simultaneously.
Stack many layers of this, and the model builds up an increasingly rich understanding of the text.
Why it changed everything
Speed. Because every word is processed in parallel, transformers train dramatically faster on GPUs. The original model reached state-of-the-art translation results after training for 3.5 days on eight GPUs, a fraction of the cost of previous best models.
Scale. Faster training meant you could train much bigger models on much more data. And it turned out that transformers kept getting better as they got bigger, with no obvious ceiling in sight.
Generality. The same design worked for almost anything that could be turned into a sequence: text, code, images cut into patches, audio, DNA.
From translation to ChatGPT
- 2018: OpenAI released GPT, a "generative pre-trained transformer" that learned by predicting the next word in books. Google released BERT, a transformer that transformed Google Search.
- 2019: GPT-2 wrote such convincing text that OpenAI initially withheld the full model.
- 2020: GPT-3, with 175 billion parameters, could perform tasks from just a few examples.
- 2022: ChatGPT put a transformer in front of the public. Read the story.
Today, ChatGPT, Claude, Gemini, Llama and almost every other major model are transformers. Learn how they work.
The eight authors
The paper lists eight authors, with a note that the order was random and all contributed equally: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser and Illia Polosukhin. Gomez was a 20-year-old intern at the time.
All eight eventually left Google. Between them, they went on to found or help lead companies including Cohere (Gomez), Character.AI (Shazeer, who later returned to Google), Sakana AI (Jones), Essential AI (Vaswani and Parmar), Inceptive (Uszkoreit) and NEAR (Polosukhin). Kaiser joined OpenAI.
Google invented the transformer and published it openly. Other companies, most famously OpenAI, were quickest to see where it could lead.
Read on
Common questions
Why is it called a transformer?
Because it transforms one sequence, such as a sentence in English, into another, such as the same sentence in German. One of the authors, Jakob Uszkoreit, has said the name was chosen partly because it sounded cool.
What came before transformers?
Recurrent neural networks, including a type called LSTM, read text one word at a time, in order. That made them slow to train and bad at remembering things from much earlier in a passage.
Are transformers only used for language?
No. They are now used for images, audio, video, protein structures, weather forecasting and robotics.
Sources and further reading
Spotted a mistake or something out of date? Tell us and we will fix it. Read how we research and write our articles.
Read next
Alan Turing and the Turing test: the man who asked 'Can machines think?'
The story of Alan Turing, the British mathematician and codebreaker who imagined thinking machines in 1950, invented the Turing test, and laid the foundations of computing and AI.
Updated 22 November 2026
Dartmouth 1956: the summer artificial intelligence got its name
How a small 1956 summer workshop at Dartmouth College named the field of artificial intelligence, who was there, what they hoped to achieve, and why their optimism was so wrong about timing.
Updated 22 November 2026
The AI winters: when the hype collapsed and the money ran out
Twice, in the 1970s and late 1980s, artificial intelligence failed to live up to its promises and funding collapsed. What happened, why, and what the AI winters can teach us about today's boom.
Updated 22 November 2026