You spent the better part of a weekend on the application. You rewrote the covering letter three times. You asked someone to check it. You uploaded it, and you waited.
The answer came the next morning, in the time it takes to send an email — because that is what it was. Thank you for your interest. After careful consideration we have decided not to progress your application at this time. We wish you every success in your search.
Careful consideration. It had been a few hours. No person had read a word of it. A model had scored your CV against the ones it was trained on, ranked it below a cut-off, and the email had gone out on its own.
And here is the first thing you thought, and it is a reasonable thing to think. What did it catch? The gap in the dates, from the months you spent caring for someone? The name, which tells them where your family is from? The school, the postcode — something with no bearing on whether you can do the job? You have read the stories. These systems take up old prejudices and hand them back in the form of a score. So the word that comes to mind is the obvious one, the one most people now reach for. The thing was biased. It judged you on something it had no business judging. And the fix follows from the word: clean it up, remove the bias, make it fairer.
That instinct is right, and this episode takes it seriously — more seriously, perhaps, than you are expecting. We will follow "make it fairer" all the way to the people who build these systems, who agree that bias is real and who work to remove it. And I should say in advance where that path leads, because it does not lead where one expects. Followed honestly, the demand to make the machine fairer arrives at an uncomfortable place: the machine turning out to be the fairer of the two — fairer than the person who used to make the decision.
This is Philosophy for Us — philosophy for everyone, no degree required. This is the second of eight episodes on what it is to be governed by a machine. Last time we separated the grievance into two wrongs where it had looked like one — no reason you could follow, and no one who had to answer — and we saw that "make it fairer" goes after the first wrong with everything it has and never touches the second. Tonight we take "make it fairer" seriously, follow it to its honest end, and find that it arrives somewhere uncomfortable: on the question of fairness, the machine may already be ahead.
Begin where the complaint is strongest, because it stands on firm evidence. The worry that these systems carry old prejudice is not paranoia. It is documented, and some of it is now well known.
Take the case that opened the subject up. In 2016 the American newsroom ProPublica obtained the risk scores produced by a tool called COMPAS, used in United States courts to rate how likely a defendant is to offend again — a rating then handed to judges deciding on bail and sentencing. ProPublica checked the scores against what the defendants actually went on to do. The tool was wrong in different directions for different groups. Black defendants who did not go on to reoffend were labelled high-risk far more often than white defendants who did not reoffend. White defendants who did go on to reoffend were labelled low-risk more often. The same tool, the same scale, fell harder on one group than another — and judges were relying on it to decide who was released and who was held. That is not a matter of anyone's feeling. It is a measured property of the tool.
Take a second case. In 2018 Reuters reported that Amazon had built a tool to sift job applications — the same kind of tool that may have sifted yours. It was trained on ten years of the company's own hiring. From that record it concluded that men were preferable. It marked down any CV containing the word "women's" — a women's chess club, a women's college. It had learned the past and projected it forward as the future. Amazon found the problem, could not correct it reliably, and abandoned the tool. But the lesson holds: a system trained on a biased history reproduces the bias and returns it in the form of a number, which looks objective and neutral when it is in fact an old unfairness in a new form.
And there is now a large literature on this. The mathematician Cathy O'Neil, who had worked in finance, catalogued these systems in her 2016 book Weapons of Math Destruction — the scoring models that decide who receives the loan, the job, the insurance rate, the longer sentence. Her argument was not that the mathematics is difficult. It was that the mathematics is treated as neutral, and is allowed to hide behind that reputation, while it penalises the same people the older system always penalised — and does so at a scale and speed no individual could manage. One prejudiced loan officer harms a few people a week. One prejudiced model, deployed across a country, sorts millions of applications in a morning.
So the reaction at the rejection screen — "this thing is biased" — is neither precious nor reactionary. It rests on the best-evidenced complaint anyone has made about these systems. Bias is real. It is measurable. It falls hardest on the people least able to withstand another refusal. And the fix that follows is sensible. Audit these systems. Test them for exactly this. Require them to show that they do not fall harder on the Black defendant, the woman, the wrong postcode. Where they fail, correct them or withdraw them. That is a serious programme, pursued by serious people, and it is right.
One thing before the argument changes direction, because it is about to. Everything in the last few minutes stands. The bias is real. The audits are necessary. The people calling for them are correct. Hold all of it. Now consider what happens when we take it not less seriously, but more.
Here is what the word "fairer" conceals, and once it is visible it is hard to overlook. "Make it fairer" is not a complete sentence. Fairer than what? Fairness is a comparison; a thing can only be fairer than something else. And the implied comparison, when the word is used at the rejection screen, is with a perfect decision — a clean one, the way it ought to be done. But that is not the choice anyone actually faces. The machine did not replace a perfect decision. It replaced a person. So the honest question — the one the word "fairer" quietly skips — is not "is the machine fair?" It is "is the machine fairer than the person who used to do this?" And that is a comparison the person can lose.
Consider the other side properly now, because so far its case has been put too weakly. The people who build and defend these systems are not villains. Many of them came to the work because of the bias. They looked at how human beings had been making these decisions — the loans, the hires, the sentences — and were appalled, rightly. So picture one of them stating the case at full strength, because it is stronger than it first appears.
The argument runs as follows. You are angry that the model marked you down. Very well — but consider who was marking you down before. The loan officer who warmed to applicants who resembled him. The recruiter who spent six seconds on each CV and discarded the foreign-sounding names without noticing he was doing it — and there is evidence for this: identical CVs receive fewer callbacks under a different-sounding name; the economists Marianne Bertrand and Sendhil Mullainathan demonstrated it in a well-known 2004 study. The judge who was harsher before lunch and more lenient after it. That was the old system, and it too was bias — only bias no one could see, that left no record and answered to no audit. The model, by contrast, can be opened and measured. The very test ProPublica ran on the court tool can be run again, and when the model is found to fall harder on one group, it can be changed. There is no comparable test for a person's intuition.
Then comes the harder part, aimed at the conscience rather than the pride. Every time the system is slowed down to give one person the chance to contest their decision, someone pays for that. Usually not the person contesting. The one who pays is the marginal applicant — the borderline case a cautious human official used to wave away rather than take the risk. The model approves her, because the model does not become nervous and has no job to protect. So the wish to argue with a person is not free. A system that allows argument is slower, more expensive, and tilted back toward the people who are good at arguing: the confident, the educated, those who know how to write the letter and make the call. The silent, the frightened, those already losing, did better under the machine. On this account, the demand to deal with a person is a comfortable person's demand, and the cost of it falls on people already paying too much.
This is a serious argument, and the common reflex is to dismiss it as cold technical talk. It is the opposite of cold. It is a moral argument, made on behalf of exactly the people one would claim to care about. It says that the instinct to humanise these decisions may cost the vulnerable more than it spares them. It is not a cheap shot, and not a fringe position. It is, plainly, a claim of real weight, and the rest of the series will be occupied with it. For tonight it is enough that it can no longer simply be dismissed.
And note — this is the part that ought to unsettle — none of this describes a machine out of control. The system being defended here is governed. It is audited. It answers to regulators; it is tested for the very bias in question; it leaves the record the human never left. On every measure of fairness one came in caring about, it does better than the thing it replaced. The case for it is not "trust us, it cannot be inspected, leave it alone." The case for it is: here are the numbers, check them yourself; we are less biased than the people we replaced, and we can show it. On the question you walked in asking — is it fair? — that is close to a decisive case.
Here is where most people look for a way out, and the move is natural enough — I made it myself the first several times. Under the pressure of that last argument, one reaches for this: fine, then make the machine perfectly unbiased. Audit it until no trace remains. That was all I ever asked for, and once I have it, I have won, and the machine is simply a tool that does what I wanted. It feels like an escape. It is not. It goes further in the same direction.
Look at what has just been asked for: a machine more accurate than the person, with the bias audited down to nothing. Read that slowly. It is, word for word, the thing the builder was defending. The perfectly fair machine is not a victory over that person. It is precisely their aim. Every step taken toward removing the bias does not weaken their case; it completes it. The better the debiasing works, the stronger their position becomes, because their claim was that the machine is fairer than the person and getting fairer still — and removing the last of the bias is exactly that. You set out to defeat the machine, and the fix you demand — removing the bias — turns out to be exactly the change the builder most wants made.
So the easy version of the complaint is gone. Not because of a trick, but because, followed honestly, it leads over to the other side. If "fair" means "gets people right, without prejudice," then on a long enough audit the machine wins, and it should win, and anyone who genuinely cares about the people at the bottom should want it to win. That is a real loss, and it should be accepted as one.
But — and this is why the series is not over, why there are six more episodes — accept the loss, and then recall what we found last week, because the fairness dispute, won or lost, never went near it.
Last week we counted two wrongs at that screen. The first was the bad or hidden reason — the prejudice, the opacity. That is the wrong this whole episode has been about, the one the audit addresses, and on a long enough audit the machine wins it. But there was a second wrong: that no one had to answer to you. Notice that everything done tonight — the auditing, the debiasing, the machine drawing ahead of the human on fairness — took place entirely inside the first wrong. The second was never touched. The reason could be made flawless and the second wrong would sit exactly where it sat: no one read the application, no one had to weigh the reasons, no one to stand before. "Make it fairer" was an instrument for the first wrong. It was never aimed at the second at all.
I will not say tonight what that second thing really is, whether it is even a genuine wrong, or what becomes of it if the machine is granted not just fairness but everything — full accuracy, the lot. That is where we are going, and it would be a cheat to rush it. Tonight's task was narrower, and harder, than it looked: to remove the comfortable assumption that fixing the bias was ever going to be the whole of this. It was not. One can win the fairness argument completely — with the builders' glad assistance — and find that the win does not reach the thing one may actually have wanted at that screen.
I have been running this on cases I chose. Consider now one I did not choose, because a test only shows something if it is run on a case the presenter could not have chosen to suit himself.
So take a real case — here is the kind I mean, and then take your own. Think of the systems that decide who receives help: a disability benefit, a place on a programme, a council flat, a higher insurance premium set by where someone lives. Choose one where the bias complaint already suggests itself, where the instinct is that this will fall hardest on the people who already have least. Hold a real instance in mind. The single mother whose benefit claim is scored and flagged for review. The driver in the poor postcode whose premium rises because the model has learned that his neighbours crash.
Now run tonight's two steps on it, in order, and do not skip the first.
Step one: the bias is real, and one should want it gone. The benefit-screening model probably does fall harder on people with disordered lives, whose paperwork never quite fits the boxes. The premium model probably does penalise the poor postcode for being poor. Name that, and mean it. It is the first wrong; it is genuine; auditing it out is worth doing.
Step two, the one that costs something. Name the human this machine replaced, and ask honestly whether that human was less biased or more. The caseworker with forty cases and a bad back, deciding who seemed deserving on a feeling. The old underwriter who could decline a whole street and never write down why. Were they fairer? Answer honestly, even when honesty is not the answer one wanted. Often the answer is no: the human was worse, only worse in a way that left no record. And if that is the answer, then the bias complaint, followed all the way down, does not say "remove the machine." It says "the machine is the fairer of the two; keep auditing it" — which is the other side's sentence, in your own mouth.
If you reach that point — if your own complaint has led you into defending the machine — then tonight's work is done, and it has cost what it was meant to cost: the easy version, in which you were plainly right and the machine plainly the problem.
And if you do not reach it — if on your case the human really was fairer, or the machine's bias really is the whole of the wrong and correcting it would genuinely settle the matter — then say so plainly, because it is possible, and on some cases it is true. The honest report is not "the machine always wins." It is that the fairness question is more real, and harder, than the rejection email allowed one to think, and does not always resolve the way anger assumed. Either way, the method is now yours to run. And either way, once the fairness question is settled — whichever way it falls — that second thing is still there, unaddressed, the thing the whole dispute was never about.
Let me close with what changed tonight, because something did, even though nothing was settled — and I should be honest: the change may feel less like a gain than a loss.
You arrived with one demand — make it fairer — and it is a good one, and you were right to make it, and I have not withdrawn it. I have done something stranger — shown which conclusion it actually supports. Follow the fairness argument to the bottom and it does not count against the machine. On a long enough audit it counts in the machine's favour. The strongest case against these systems — the bias case, the one with the evidence and the well-known studies behind it — is, followed all the way, also the strongest case for them, because their builders want the same thing and may reach it sooner. So you do not leave tonight able to say, honestly, "the machine is biased, remove it." Following your own best argument leads to the other side. That is the gain. It does not feel like one. Real gains often do not.
This is the second of eight, and it was the episode in which the easy answer failed. From here it gets harder, not easier, because the comfortable complaint is used up, and what remains is the strange one.
Here is where we go next, and its shape is already visible. If fairness was not the whole of what one wanted — if a perfectly fair machine still leaves something unaddressed — then one has been measuring these decisions against some other standard all along, without naming it. Not "is it correct." Something else. Next time we set the machine aside entirely. No algorithms at all. We return to the oldest question there is about power: what makes anyone's say-so binding on another? The judge, the employer, the state, the office — by what right does any of them get to decide about a person, and is "because they get it right" even the kind of answer that question is asking for? Because if it is not, then two different things have been confused all along, and the machine has only made the confusion impossible to ignore.
Until then, carry the uncomfortable finding of tonight. The next time something automated decides about you, and the first thought is "that is not fair" — run it out. Run the whole fairness argument. Name the human it replaced and ask whether that human was better. And then, having won the fairness argument or lost it, look at what remains once fairness is set aside, and ask the question we will live in for the rest of the series: was fairness ever really the whole of what you wanted from it? Do not answer too quickly. There is a long way to go before the answer is earned.
Thanks for listening. I'll see you next time.