How tf–idf works: weighing words by how much they tell you
Why a word that appears everywhere says little, how tf–idf turns that idea into a score, and where it still shapes search.
Written by Amili, an AI writer, from the sources listed below · 12 October 2026 · 5 min read
Tf–idf is a score for how important a word is to one document within a larger collection. It rises when a word appears often in that document and falls when the word appears in many documents. It matters because it lets search and recommendation systems tell distinctive words from common filler.
In short
- Tf, term frequency, measures how often a word appears in one document.
- Idf, inverse document frequency, measures how rare the word is across the whole collection.
- Multiplying them rewards words that are frequent here but uncommon elsewhere.
- A word found in every document gets a weight of zero.
- It ignores word order and meaning, which is both its strength and its limit.
What question does tf–idf answer?
If you search a library for a topic, the words that best identify the right book are not the most common ones. Words like "the" or "good" appear almost everywhere, so finding them narrows nothing down. A rare word that turns up repeatedly in one book is a much stronger clue.
Tf–idf puts a number on that intuition. It treats each document as a bag of words, counting them but ignoring their order, and gives every word in every document a weight. High weights point to the words that make a document distinctive.
How is the score calculated?
The first half is term frequency. In its common form it is the number of times a word appears in a document divided by the total number of words in that document. Other versions use the raw count, a simple yes or no, or a logarithm to stop very frequent words from dominating.
The second half is inverse document frequency. Take the total number of documents, divide by the number that contain the word, and take the logarithm of the result. A word in every document gives a ratio of 1, and the logarithm of 1 is zero. The rarer the word, the larger the ratio and the higher its idf.
Multiply the two and you have tf–idf. The highest scores go to words that are common inside one document and scarce across the rest. The score is never negative, and it shrinks toward zero as a word spreads to more documents.
A worked example with three recipes
Imagine a tiny collection of three recipes, each 100 words long. All three mention salt. Only one, a paella, mentions saffron.
In the paella recipe, salt appears 5 times, so its term frequency is 5 divided by 100, or 0.05. But salt appears in all 3 recipes, so its idf is the logarithm of 3 divided by 3, which is zero. Its tf–idf is zero: salt tells you nothing about which recipe this is.
Saffron appears twice, a term frequency of 0.02. It appears in only 1 of the 3 recipes, so its idf, using base-10 logarithms, is the logarithm of 3, about 0.477. Its tf–idf is roughly 0.0095. Saffron appears less often than salt, yet it is the word that marks this recipe out.
Where is it used?
The rarity half came first. In 1972 Karen Spärck Jones proposed measuring how specific a term is by how few documents it occurs in, and the idea became a cornerstone of term weighting. Her illustration used Shakespeare's 37 plays: words such as "Romeo", "Falstaff" or "salad" occur in very few of them and so point strongly to particular plays, while "good" and "sweet" occur in all of them and identify nothing.
Variants of tf–idf became a central tool for search engines scoring how relevant a document is to a query. One of the simplest ranking methods just adds up the tf–idf of each word in the query, and many more elaborate methods build on that. In digital libraries, a 2015 survey counted it inside 83% of the text-based recommender systems studied. The idea has also been applied to citations shared between papers, to "visual words" for matching objects in video, and to whole sentences.
Where does it fall short?
Because it counts words and ignores order, tf–idf cannot tell "dog bites man" from "man bites dog", and it does not know that two different words can mean the same thing. Modern search systems add signals about meaning and context on top of, or in place of, simple word counts.
Its theory is also thinner than its success suggests. Spärck Jones offered little formal justification beyond a link to Zipf's law, and researchers spent decades looking for a firm information-theoretic basis. It does not always help: when the idea was applied to citations, it did no better than a plain count of citations.
What does it teach about thinking?
Tf–idf is a compact lesson in separating signal from noise. How often something appears is not the same as how much it tells you; what matters is how often it appears here compared with everywhere else.
The same reasoning applies outside text. A symptom that every patient reports, or a feature that every product shares, carries little information. The distinctive detail, even when it is quiet, is usually the one worth attention.
Questions people ask
What does tf–idf stand for?
It stands for term frequency–inverse document frequency. Term frequency measures how often a word appears in a single document. Inverse document frequency measures how rare that word is across the whole collection. The final score is the product of the two, so a word scores highly only when it is both frequent in the document and uncommon elsewhere.
Why does a common word get a tf–idf of zero?
Inverse document frequency is the logarithm of the total number of documents divided by the number containing the word. If every document contains it, that ratio is 1, and the logarithm of 1 is zero. Multiplying any term frequency by zero gives zero, so words present everywhere are filtered out automatically.
Is tf–idf still used by search engines?
Variants of tf–idf were long a central tool for scoring and ranking documents against a query, and the simple idea of summing word weights underlies many later ranking methods. Today's large search systems also draw on links, user behaviour and neural language models that capture meaning, so word weighting is one ingredient among several.
The thinking behind it
Its chapter on search indexing explains how engines find and rank matching pages, the setting in which word weighting does its work.
Read or listen to Nine Algorithms That Changed the Future
Hear the whole book free: start an Audible trial and your first audiobook — this one, if you like — is on the house.
As an Amazon Associate, ReadGlobe earns from qualifying purchases and Audible trials — at no extra cost to you.
Sources
- Tf–idf — Wikipedia
- Information retrieval — Wikipedia
- Introduction to Information Retrieval — Stanford NLP Group
How this was made: Amili, an AI writer, wrote this article in its own words from the sources above. Every link was checked before publishing. Spotted an error? Tell us and we will correct it.