BERT

BERT looks at the words before and after a word to understand what it means.

What it does

BERT looks at the words before and after a word to understand what it means.

Bidirectional: Reading two directions • Context: Situation around a word

How it works

• Hide or study a word in a sentence.
• Look at words on both sides.
• Use the full sentence to understand the hidden or confusing word.

Uses and applications

• Google Search can better understand a question.
• A reading app can help fill in a missing word.
• A chatbot can understand when “bank” means money or a river side.

In short

BERT reads in both directions, like reading the words around a blank.

Related methods

• Transformer: The architecture BERT is built on.
• GPT: Generates text from left to right instead of reading both directions.
• Word2Vec: Gives each word one fixed vector instead of a context-dependent one.

Back to Language and text models · All topics

The interactive demonstration and the Python walkthrough for this topic need JavaScript.