What algorithmic bias is, and how it gets into software
How unfair outcomes creep into automated decisions through data, design and use, with real cases from medicine, healthcare, policing and face recognition.
Written by Amili, an AI writer, from the sources listed below · 12 October 2026 · 5 min read
Algorithmic bias is a repeatable pattern in which a computer system treats one group of people worse than another for reasons unrelated to the task. It matters because such systems now help decide loans, hiring, policing and medical care, and their errors repeat at scale while looking neutral.
In short
- Bias means a systematic, repeatable unfairness, not a one-off mistake.
- It can enter through the data, the choice of what to predict, the design, or the way a tool is used.
- A proxy target, such as cost standing in for need, can quietly build in unfairness.
- Feedback loops let a biased output produce the data that confirms it.
- There is no single agreed definition of fairness, and some definitions conflict.
What counts as bias in an algorithm?
An algorithm is a set of instructions that turns data into a decision or a ranking. Being strict is not the same as being biased. A credit scoring program that weighs only relevant financial facts, and weighs them the same way every time, may refuse a loan without doing anything unfair.
The problem starts when two applicants who are nearly identical on everything that matters get different answers because of something irrelevant, and when that difference shows up again and again. That repeatability is the defining feature. A single odd result is noise; a pattern that keeps favouring one group over another is bias, whether or not anyone intended it.
How does bias get into a system?
The first route is the data. Before a program learns anything, people decide what gets collected, how it is categorised and what is thrown away. If past decisions were unfair, a system trained to imitate them inherits that unfairness. A program that copies the judgments of the person who used to do the job will also copy that person's blind spots.
The second route is the choice of target. Researchers call one version label choice bias: the system is trained to predict an easy-to-measure stand-in rather than the thing people actually care about. The third route is design, such as rules about what to include in results. The fourth is use, when a tool built for one setting is applied to people its designers never considered.
Size of the sample matters too. A system tends to give more confident answers where it has more data, which pulls its results toward large groups and can leave smaller, underrepresented groups poorly served.
What do real cases look like?
One early case came from St George's Hospital Medical School. Between 1982 and 1986 a computer screening system, built on historical admissions patterns, turned away up to 60 women and ethnic minority applicants a year, including men with names that sounded foreign. Other schools had similar biases; this one stood out because it automated them.
Healthcare offers a clearer view of the proxy problem. A widely used algorithm estimated future healthcare costs and treated that as a measure of need when deciding who should receive extra support. Because Black patients tended to generate lower costs even when they were just as sick, the tool underrated their needs. When the target was changed from cost to need, the number of Black patients selected for the programme almost doubled.
Face recognition shows the data problem. In 2018, researchers Joy Buolamwini and Timnit Gebru reported that commercial systems were wrong on as many as 35% of darker-skinned women, against under 1% for lighter-skinned men. Such gaps, traced to unbalanced training images, have been linked to several wrongful arrests.
A worked example: how a feedback loop locks bias in
Imagine a city with two neighbourhoods where crime is actually the same, but one has historically been patrolled more heavily. More patrols mean more recorded arrests there, simply because more officers are present to make them.
An algorithm that sends patrols wherever arrests are high will send even more officers to that neighbourhood. They record still more arrests, which the algorithm reads as confirmation. After a few rounds the data look like overwhelming proof of a difference that began as an accident of where people were looking. Nothing in the code is malicious; the loop does the work.
Why is it so hard to detect and fix?
Many algorithms are trade secrets, so outsiders cannot inspect them. Even when the code is open, some models are too complex to follow, and they may change as new data arrive. On a single website there is often no one algorithm to examine, but many linked programs that behave differently for different users.
Fixing it also requires choosing what fair means. Fairness research grew sharply after 2016, when ProPublica reported that the COMPAS risk tool used in US courts wrongly labelled Black defendants as high risk almost twice as often as white defendants. The maker, Northpointe, disputed the analysis. Part of the dispute is that several reasonable definitions of fairness exist, and some cannot all be met at once.
Law has only recently caught up, through two European laws: the General Data Protection Regulation, in force since 2018, and the AI Act, adopted in 2024.
What does it teach about thinking?
In 1976 the artificial intelligence pioneer Joseph Weizenbaum warned against trusting decisions from programs we do not understand. He compared it to a tourist who reaches the hotel by flipping a coin at each corner: arriving says nothing about whether the method is sound.
Two human habits make things worse. One is treating a machine's output as more authoritative than a person's judgment. The other, called selective adherence, is following the algorithm only when it agrees with what we already believed. The lesson is to ask what a system was trained to predict, on whose past decisions, and who checks its results.
Questions people ask
Can an algorithm be biased if nobody programmed it to discriminate?
Yes. Most cases are unintended. Bias can come from historical data that reflect past unfair decisions, from predicting a convenient proxy instead of the real goal, or from using a tool on people it was not designed for. The program then repeats the pattern faithfully, which is exactly what makes it hard to notice.
Are algorithms fairer than people?
Sometimes. Well designed algorithms can reach outcomes as equitable as human decisions, or more so, and they are consistent. But that consistency cuts both ways: when a system is biased, it applies the same error to every case it sees, and its apparent neutrality can make people less likely to question it.
What is label choice bias?
It is the bias that appears when a system is trained to predict a measurable stand-in rather than the outcome people actually want. Predicting healthcare costs instead of healthcare needs is the best known example. The fix is to bring the predicted target closer to the real goal, which in that case greatly increased the share of Black patients offered extra care.
The thinking behind it
A readable account of how opaque scoring models in credit, policing and education can deepen existing inequality while appearing objective.
Read or listen to Weapons of Math Destruction
Hear the whole book free: start an Audible trial and your first audiobook — this one, if you like — is on the house.
As an Amazon Associate, ReadGlobe earns from qualifying purchases and Audible trials — at no extra cost to you.
Sources
- Algorithmic bias — Wikipedia
- Fairness (machine learning) — Wikipedia
How this was made: Amili, an AI writer, wrote this article in its own words from the sources above. Every link was checked before publishing. Spotted an error? Tell us and we will correct it.