How Q-learning works
How a program with no map of its world learns, by trial and error, which action is worth taking in each situation.
Written by Amili, an AI writer, from the sources listed below · 23 October 2026 · 6 min read
Q-learning is a reinforcement learning method that teaches an agent the value of each action in each situation purely from experience. After every move it nudges its estimate toward the reward it just got plus the best value it expects next. Given enough exploration, it learns the best policy without ever needing a model of its world.
In short
- Q stands for quality: the expected long-run reward of taking a given action in a given state.
- It is model-free: the agent never needs to know how the world works, only what happened when it acted.
- Two dials shape it: the learning rate (how much new experience overrides old) and the discount factor (how much the future counts).
- It must balance exploring new actions against exploiting the best one it knows.
What is Q-learning trying to learn?
Picture an agent, a program, that sits in some situation, called a state, and can choose among a few actions. Each action moves it to a new state and hands it a reward, a number that can be positive or negative. The agent's goal is not the biggest reward on the next step but the biggest total over everything that follows.
Q-learning keeps a score for every pairing of state and action. That score, the Q value, is the agent's current estimate of how much total reward it will collect if it takes that action now and plays well afterwards. Once the scores are accurate, behaving well is easy: in each state, pick the action with the highest score. The algorithm was introduced by Chris Watkins in 1989, and Watkins and Peter Dayan published a proof in 1992 that it converges.
How does a single update work?
After each move the agent has three pieces of information: the reward it just received, the state it landed in, and its existing scores for the actions available there. It forms a target by adding the reward to the best score in the new state, with that future part shrunk by a discount factor between 0 and 1. Then it moves its old estimate part of the way toward that target. How far it moves is the learning rate.
A toy example shows the arithmetic. Imagine a short corridor of three rooms with an exit in the last one worth 10 points, a learning rate of 0.5 and a discount factor of 0.9. All scores start at zero. The first time the agent steps from room 2 into the exit, the target is 10, so its score for moving right in room 2 goes halfway, to 5. Later, when it steps from room 1 into room 2, the target is 0 plus 0.9 times 5, which is 4.5, so its score for moving right in room 1 becomes 2.25. Value seeps backwards, one step at a time, from the reward to the moves that lead to it. With repetition the scores settle, and rooms closer to the exit carry higher values.
This kind of update is a form of the Bellman equation, the recursive relationship at the heart of dynamic programming. The difference is that Q-learning learns it from samples instead of computing it from a known model.
Why do the learning rate and discount factor matter?
The learning rate decides how much new experience overwrites old belief. At 0 the agent never learns; at 1 it trusts only the latest result and forgets everything else. In a perfectly predictable world a high rate works well. When outcomes are noisy, convergence in theory requires the rate to shrink over time, though in practice a small constant is common.
The discount factor decides how much the future counts. At 0 the agent is short-sighted and cares only about the immediate reward. Close to 1 it plans for the long run. Set it at or above 1 and the values can grow without limit, which is why it is kept below 1.
Why does it need to explore?
An agent that always picks its current best action may never discover that another action is better. A common fix, called epsilon-greedy, has the agent take its best-known action most of the time and a random one a small fraction of the time. Another trick is to start every score optimistically high, so that each tried action is disappointed into a lower value and the untried ones look attractive by comparison.
The guarantee that Q-learning finds the optimal policy for any finite Markov decision process assumes unlimited exploration time and a policy with some randomness in it. Cut exploration short and the agent can lock in a merely good habit.
Where is it used, and where does it struggle?
The simplest version stores every score in a table, which breaks down when there are too many states for the agent to visit each one often enough. Grouping similar states into buckets helps a little. The bigger leap is to replace the table with a function that can generalise, such as a neural network. In 2014 Google DeepMind patented deep Q-learning, which played Atari 2600 games at the level of expert humans.
Neural networks make plain Q-learning unstable, because consecutive experiences are correlated and small changes in the scores can shift behaviour sharply. Deep Q-learning steadied it with experience replay, which trains on a random sample of past moves, and with targets that are updated only now and then. A separate problem is overestimation: using the same scores to choose and to judge the best next action tends to inflate them in noisy settings. Double Q-learning fixes that by keeping two sets of scores, one to pick the action and one to evaluate it. The standard method also assumes discrete states and actions, which makes continuous problems inefficient.
What does it teach about thinking?
Judge a move by where it leads, not only by how it feels right now: a choice that costs something today can be the start of the best path. And keep a little room for trying things, because a strategy you never test is a strategy you can never learn is wrong.
Questions people ask
What does the Q in Q-learning stand for?
Quality. The Q function is the algorithm's estimate of how good it is to take a particular action in a particular state, measured as the total discounted reward the agent can expect from that point on if it then acts well. Learning these quality scores is the whole job; choosing actions afterwards is just picking the highest score.
What does model-free mean in Q-learning?
It means the agent does not need to know the rules of its environment, such as which state each action leads to or how likely each outcome is. It learns only from what actually happens when it acts. That makes Q-learning usable where the environment is unknown or too complex to write down, at the cost of needing a lot of experience.
What is the difference between Q-learning and deep Q-learning?
Plain Q-learning keeps a table with one score for every state and action, which only works when there are few of them. Deep Q-learning replaces the table with a neural network that estimates the scores, so it can handle huge or continuous state spaces such as video game screens. It adds techniques such as experience replay to keep learning stable.
The thinking behind it
It covers reinforcement learning and the difficulty of specifying rewards that produce the behaviour you actually want.
Read or listen to The Alignment Problem
Hear the whole book free: start an Audible trial and your first audiobook — this one, if you like — is on the house.
As an Amazon Associate, ReadGlobe earns from qualifying purchases and Audible trials — at no extra cost to you.
Sources
- Q-learning — Wikipedia
- Reinforcement learning — Wikipedia
How this was made: Amili, an AI writer, wrote this article in its own words from the sources above. Every link was checked before publishing. Spotted an error? Tell us and we will correct it.