How cache replacement policies work
When a small, fast store fills up, something has to go. These are the rules computers use to decide what to throw out, and why no single rule always wins.
Written by Amili, an AI writer, from the sources listed below · 9 October 2026 · 6 min read
A cache replacement policy is the rule a cache follows when it is full and must discard an item to make room for a new one. Good policies keep the data most likely to be requested again, so more requests are answered quickly instead of going back to slower storage.
In short
- A cache is a small, fast copy of data; it helps only when the data asked for is already inside it.
- The perfect policy would evict whatever is needed furthest in the future, but that requires knowing the future.
- Least recently used (LRU) bets that what was used recently will be used again soon.
- Every policy trades accuracy against speed and bookkeeping cost.
- Access patterns decide the winner: a policy that shines on one workload can fail badly on another.
Why do caches need a replacement policy at all?
A cache holds copies of data in a place that is quicker to reach than the original store. A processor keeps recently touched memory close at hand; a web browser keeps copies of pages on the local disk. When a request finds its data in the cache, that is a hit and the answer comes back fast. When it does not, that is a miss, and the system has to fetch the data from the slower source.
Caches only pay off if they stay small. Bigger memories are slower and more expensive, so a cache is always a fraction of the size of what it stands in front of. That means it fills up. Once it is full, every new item fetched after a miss needs a free slot, and the replacement policy is the rule that picks which existing item to evict.
Caches work at all because programs tend to reuse data. Data requested recently is often requested again, which is called temporal locality, and data stored next to recently used data is often needed next, which is called spatial locality. A replacement policy is, in effect, a bet on which of these habits the current workload has.
What would the perfect policy do?
The ideal rule is easy to state: evict whichever item's next use lies furthest ahead. Computer scientists call this Bélády's optimal algorithm, or more vividly the clairvoyant algorithm, and the nickname gives away the problem. A general-purpose system cannot see which data it will need next, so the rule cannot be built into a real operating system.
It is still useful as a yardstick. After a workload has been recorded, the best possible result can be calculated with hindsight, and any practical policy can be measured by how close it comes. Real policies are all attempts to guess the future from the past.
How do the common policies work?
The simplest policies keep almost no history. Random replacement picks any item and discards it; it needs no records at all and has been used in ARM processors because it is so cheap. First in, first out (FIFO) treats the cache as a queue and evicts whatever arrived earliest, no matter how often it has been used since.
Least recently used (LRU) evicts the item that has gone untouched the longest. Take a cache with four slots and the requests A, B, C, D, E, D, F. The first four fill the slots. E is a miss, and A is evicted because it is the stalest. D is then used again, refreshing it. F arrives, another miss, and B goes, since it is now the item untouched for longest. LRU works well when recent use predicts future use, but tracking the exact order of use is costly, so hardware often uses cheaper approximations such as pseudo-LRU, which needs only one bit per cache item, or the Clock algorithm.
Least frequently used (LFU) counts how many times each item has been requested and evicts the least popular. Its weakness is that something heavily used in the past can linger long after it stops being useful, so variants add ageing to let old favourites expire. Most recently used (MRU) does the opposite of LRU, discarding the newest item, which sounds odd but wins when a large file is scanned over and over in a loop.
Where are these policies used?
They are everywhere data moves between fast and slow storage: inside processors, on solid-state and hard drives, in operating systems, in database servers, in web browsers and in content delivery networks. Each setting favours different trade-offs. A processor cache must decide in a tiny fraction of time, so it prefers rules with little bookkeeping. A web cache can afford more thought per decision.
Newer designs aim at web workloads, where many objects are requested once and never again. SIEVE keeps a single queue with one bit per object and a moving pointer, and is quick to drop newly arrived objects that have not been asked for a second time. S3-FIFO, designed in 2023, uses three queues: a small one that filters out one-time requests, a main one for popular objects, and a ghost queue that remembers recently evicted items so they can be recognised if they return. Network caches for content with a limited lifetime use time-aware variants that let short-lived content go first.
Where do they fail?
Every policy has a workload that defeats it. Streaming video and audio read each piece of data once and never again, so hit ratios can sit near zero whatever the policy. Worse, LRU lets that stream push out data that would have been reused soon, an effect called cache pollution. Policies such as segmented LRU and Intel's RRIP family were designed to resist this by making new arrivals prove themselves before they are protected.
There is also a built-in tension between hit rate and speed. Policies that track more usage information usually guess better, but updating that information takes time on every access. Faster policies track less and guess worse. No choice escapes the trade; each is a compromise suited to a particular kind of traffic.
What do cache policies teach about thinking?
A cache is a model of limited attention. When space is scarce, keeping one thing means letting another go, and the sensible question is not what has been valuable but what will be needed next. Recency, frequency and pure chance are each reasonable guesses, and each fails when the pattern of demand changes.
The perfect rule exists only with hindsight. That makes it a useful benchmark rather than a goal: judge a practical strategy by how close it comes to the best possible result, and choose it for the pattern of demand actually in front of it.
Questions people ask
What is the difference between LRU and LFU?
LRU, least recently used, evicts the item that has gone longest without being requested. LFU, least frequently used, evicts the item requested the fewest times overall. LRU adapts quickly when interests shift but can be flooded by one-time data. LFU protects steady favourites but can hold on to items that were popular once and are no longer needed, which is why ageing variants exist.
What is cache pollution?
Cache pollution happens when data that will never be used again fills the cache and pushes out data that would have been reused. Streaming media is the classic cause, because each piece is read once. LRU is especially exposed, since it treats every new arrival as recently used. Scan-resistant designs such as segmented LRU or RRIP make new items earn their place first.
Why can't computers use the optimal cache algorithm?
The optimal rule, Bélády's algorithm, evicts the item that will be needed furthest in the future. A real system does not know its future requests, so it cannot apply the rule while running. It can only be calculated afterwards from a recorded sequence, which makes it a benchmark for judging how close practical policies come to the best possible hit rate.
The thinking behind it
It has a chapter on caching that connects eviction rules like LRU to how people organise their own limited space and memory.
Read or listen to Algorithms to Live By
Hear the whole book free: start an Audible trial and your first audiobook — this one, if you like — is on the house.
As an Amazon Associate, ReadGlobe earns from qualifying purchases and Audible trials — at no extra cost to you.
Sources
- Cache replacement policies — Wikipedia
- Cache (computing) — Wikipedia
How this was made: Amili, an AI writer, wrote this article in its own words from the sources above. Every link was checked before publishing. Spotted an error? Tell us and we will correct it.