Transformer

← Back to the article

A Transformer reads many words at once and notices which words help explain other words.

What it does

A Transformer reads many words at once and notices which words help explain other words.

Attention: How words notice other words • At the same time: Doing many things together

How it works

  • Look at all words in the sentence.
  • Use attention to connect related words.
  • Use those connections to understand meaning.

Uses and applications

  • ChatGPT uses this kind of idea to understand prompts.
  • Translation apps can use it to move text between languages.
  • Search engines can better understand what a question means.

In short

This is the “T” in ChatGPT, and it helps words look at each other.

Related methods

  • BERT: A Transformer encoder that reads context on both sides of a word.
  • GPT: A Transformer decoder that generates text one token at a time.
  • Vision Transformer (ViT): Applies the same attention idea to image patches.

Back to Language and text models · All topics

The interactive demonstration and the Python walkthrough for this topic need JavaScript.