TF-IDF

← Back to the article

TF-IDF gives a word a high score when it is common in one document but rare in many others.

What it does

TF-IDF gives a word a high score when it is common in one document but rare in many others.

Corpus: All documents • Weight: Importance score

How it works

  • Count how often a word appears in one document.
  • Check whether the word is rare across many documents.
  • Give rare-but-important words a higher score.

Uses and applications

  • A search tool can find documents that match a keyword.
  • A library app can pick important words for a book summary.
  • A news app can tag articles by important terms.

In short

It scores words by how useful they are for finding meaning.

Related methods

  • BM25: Refines TF-IDF scoring for search ranking.
  • RAG (Open Book AI): Uses retrieval scores like these to choose sources.
  • LDA: Finds topics from word counts instead of scoring single words.

Back to Search and retrieval · All topics

The interactive demonstration and the Python walkthrough for this topic need JavaScript.