CLIP

CLIP learns how pictures and words match each other.

What it does

CLIP learns how pictures and words match each other.

Embedding: Math code for data • Alignment: Matching word to photo

How it works

• Look at many image-and-text pairs.
• Put matching pictures and words close together in number space.
• Use that space to compare new pictures and words.

Uses and applications

• A photo search can find dog pictures when you type “dog.”
• A safety tool can check whether an image matches a label.
• A robot can connect written instructions to what it sees.

In short

CLIP helps AI recognize that a word can match a photo.

Related methods

• Vision Transformer (ViT): Often serves as CLIP's image encoder.
• Stable Diffusion: Uses a text encoder from CLIP to guide image generation.
• Word2Vec: Embeds words alone rather than matching words with images.

Back to Computer vision · All topics

The interactive demonstration and the Python walkthrough for this topic need JavaScript.