You know this screen by now. The loan, declined in about four seconds. A reference number, and a link to a page that says your application did not meet the criteria. The person on the phone who could see the decision but not the reasons behind it. The box that said request a review, which you ticked already expecting the result: the review was the same system, reading the same numbers, reaching the same place.
We have stood at that screen for seven weeks, and at the ones like it — the frozen account, the denied claim, the flagged name. Tonight is not like the other seven, and I should say why at the start.
For seven weeks I have added one consideration a week. We added bias and accuracy. We added the question of what makes any power's decision binding on you at all. We added the point where I asked you to grant the machine everything — to suppose it had judged you correctly — and we looked at what remained. We added the two forms the missing thing might take. We turned the same demand on the humans you trust, and found that half of them failed it too. And last week we added the chain of people with clean hands, and watched the word accountability come apart.
That is the whole of it. Every consideration is now in place. Tonight I am not going to add an eighth. There is no new position coming, no new thinker, no new case. Tonight we do the reverse. We take everything we have assembled and, instead of doing what you came for — instead of telling you what to conclude — I am going to hand all of it back to you. And then I am going to decline, deliberately, to settle it. That refusal is not a failure to reach an answer. It is the most honest thing I can offer, and by the end the reason will be clear.
This is Philosophy for Us — philosophy for everyone, no degree required. This is the eighth and last of eight episodes on what it is to be governed by a machine. Last week we asked who answers when one of these decisions has already gone wrong, and we found a chain of people, each one genuinely in the clear, with the answer you were owed falling through every join until it reached an audit that was never owed to you in the first place. The word offered as the cure, accountability, turned out to answer a regulator, and not you. Tonight we put all of it back in your hands — the whole question, narrowed to the one thing it always rested on — and we leave it open.
So return to the screen once more. The reference number. The decision that was final, with no way for you to contest it — no handle on your side of the door. Let me begin where you began, with what you wanted from me when you sat down, eight weeks ago, with that number in your hand.
When you sat down with that reference number, you wanted something from me. Be honest about what it was, because it was almost certainly one of two things, and they point in opposite directions.
The first thing you might have wanted was for me to say: this is wrong; remove it. The machine is cold, biased and closed, and the remedy is to put a person back — a real one, who can look at you, hear you out, be moved, change their mind. Take the decision away from the thing with no face and give it to someone with one. Many of you came in wanting that. It is a reasonable thing to want.
The second thing you might have wanted was the opposite: for me to say stop complaining. The machine is the fairer of the two. It is less biased than the loan officer it replaced; it ruins fewer lives; it catches more of the real cases and wrongly rejects fewer of the innocent ones — and your wish to argue with a person is a comfort that someone else, someone poorer and less articulate than you, ends up paying for. Be grateful, and let it work. Some of you came in wanting that, or came to it over the last seven weeks. That too is a reasonable thing to want. It is, as I said in week one, the single most serious thing anyone says in this whole argument.
I am not going to give you either one. And I want to be exact about why, because it is not that I am undecided, and it is not that I think both sides have a point and the truth lies in the middle. It is more specific than that. For seven weeks I went looking for the thing you were actually complaining about at that screen, and it was in neither of those two places. Removing the machine does not reach it. Letting the machine work does not reach it. Both are answers to one question — is the machine the right tool? — and the thing that was troubling you, I have come to think, was never mainly about whether the machine was the right tool.
Let me rule out two more answers before they tempt you, because there are two other things you might want me to say, and I will not say those either.
You might want me to say: obviously you need both — some accuracy and some accountability, a sensible balance of the two. That sounds mature, and it is the least useful thing that can be said here. It assumes the two sides are arguing about how much of two agreed goods to have, as though the only question were proportions. It is not. The whole dispute is over whether the second thing — the standing to be answered — is a real good at all, or merely a feeling you should be prepared to give up. You cannot split the difference between a good and a thing that one serious party denies is a good. "More of both" is a way of not noticing that the argument is taking place.
And you might want me, at the very end, to hand you the practical answer — the regulation, the right to an explanation, the audit regime, the law requiring a human in the loop. The fix you can write down and pass. I will not give you that, and of the refusals tonight it is the one I hold to most firmly. Handing you a policy and calling it the answer would be the most dishonest thing this show could do, because it would let you leave believing the question was settled, when the whole point of these eight weeks is that it is not — and that the people who tell you it is have simply stopped looking too early. I am not going to present a law as if it were the end of the matter.
So all four answers are closed. Not removing the machine. Not letting it work. Not the sensible balance. Not the regulation. I know that is not what you came for. Stay with it a little longer, because there is one thing left to examine, and it is the only thing that was ever worth examining — and to see it, I have to do something I have only done in pieces so far. I have to set out the whole of what we found at once.
Here is everything we found, set out in order. Consider it as a single thing, because the pieces only show their shape together.
We took your first instinct — make it fairer — entirely seriously, and granted it everything it asked. We did not merely imagine a better machine; we built the perfect one. We gave it more accuracy than any human who ever held your file. We audited the bias out of it until no honest person could find a trace — not against your name, not your postcode, not your accent. We placed it under proper governance, overseen, answerable to every regulator there is. We did, in thought, the entire thing the critic and the engineer both asked for — the best version either of them has imagined. And then we turned back and looked at you, still sitting there with the reference number.
And the thing you walked in complaining about had not gone away.
Do not pass over that, because everything in these eight weeks has been pointing at it. It was not the error — there was no error; we stipulated that away. It was not the bias — we removed it. It was not that no one was watching the machine — the regulators were. Every specific fault you could name, we corrected. And something remained. Quieter than in week one, and harder to point at, because you could no longer point to a mistake. But there.
What was it? State it as plainly as it will go, and no more, because the moment you dress it up you have begun to cheat. It was this. A power reached into your life and decided something that cost you, and it left you no way to contest it. No grounds you could take hold of. No one who had to answer for it to you — not to a regulator, not to an auditor, but to you, the person it was about. The decision was final, and there was no handle on your side of the door, and improving the machine did not add one. If anything — and this is the hard part — the better the machine became, the more final the decision grew. A flawless decision is harder to argue with than a careless one. Perfecting it closed it.
Now I can show you why I spent an earlier series, the one called Knowledge and Power, on a question that looked unrelated. There we learned to ask, of any system that judges people: whose word does it credit? Whose account is believed, and whose is treated as noise. That was one half of it. This series was the other half — the half you only see once you have the first. Not whose word it believes, but to whom must it answer. And what we have found tonight is that you have been asking that second question your whole life, of every power that ever decided about you — the employer, the landlord, the office, the state — and you never said it aloud, because you did not hold it as a question. You held it as a feeling. The machine is what finally made the feeling loud enough to hear.
There is a second old point here too, one we reached in the series on luck and desert. The machine that scored you was not measuring what you deserve; it cannot. A model predicting whether you will default is measuring people who resemble you — a statistical neighbour, a group you have been sorted into by a correlation you cannot see and were never shown. It is not a judgment of your character. It does not know your character. And yet it lands on you exactly as if it were a verdict on your worth — you are the kind of person who. We learned, in that earlier series, that no one simply deserves their talents, that desert may account for far less than we suppose. Here is the consequence: a system reaches a conclusion that was never about desert at all, that is about a group you happen to fall into, and delivers it to you with all the weight of a sentence on the person you are. You feel judged. Nothing judged you. You were sorted.
And then — because I said at the start there would be no easy place to stand, and I meant it — the difficulty runs in the other direction too. We did not only build the perfect machine. We also took the demand you were making of it, the demand to be answered, and turned it on the humans you trust. The judge whose reasons arrive after the verdict, assembled to justify a decision already made. The loan officer who liked your face, or did not. The manager whose discretion is a hunch given the authority of a job title. We asked whether those answer to you — genuinely answer, rather than perform answering — and a great many of them answer less than the audited machine does. The machine at least leaves a record. So you cannot even retreat to put a person back and feel safe there. The person may close the same door, and smile while doing it, and leave you with even less to take hold of than you have now.
That is the whole of it. The machine made perfect, and the grievance surviving anyway. The thing you were demanding, which you never named. The verdict that was only ever a sorting. And no human fallback to turn to. Every comfortable place to stand is gone. So what do we do with the one thing left standing in the middle of all that? Because there is exactly one question left, and it is the hard one.
The leftover thing. The complaint that survived the perfect machine — the one you cannot make the machine answer, however good it becomes. Is it a wrong?
You want to say yes. I will put that yes at full strength, because it deserves that. On this reading, the thing you were demanding all along was never that the power over you be right. It was that the power be answerable — that you be able to ask it for its grounds, argue against them, and find someone who has to take that challenge and either be moved by it or explain why not. And this reading says that is not a whim; it is an entitlement, a standing you are owed, as a person, by anything that claims the right to decide about you. The evidence that you hold it is that you have been demanding it your whole life, quietly, of every authority you have lived under — the parent, the teacher, the employer, the court, the state — and the only reason you never said it aloud is that, until a machine did it to you in four seconds with no face, the human version always left you just enough of a handle that you never noticed the demand was there. On this reading the machine did not take something trivial from you. It took the oldest political thing there is: the standing to make power answer.
That is a strong yes. Now hear the people who built the machine, because they have been present throughout these eight weeks, they have heard every word of that, and they are not persuaded — and it is worth seeing why not.
They say this is an elegant name for what is, underneath, the most ordinary thing in the world: the wish to have someone to argue with. To not be refused by something that will not fight back. And they say: notice who could always afford that wish, and who never could. The person your reading claims to defend — the thin file, the second job — never once got a powerful human to genuinely answer to them. That person got the loan officer's hunch and no appeal, and got it worse. And what the cold machine gives them — the audited machine you are ready to condemn for having no handle — is the one thing the human world never gave them: a check on the power that does not wait for them to be strong enough to demand it. The regulator examines the whole system whether or not you can turn up and fight. So before you call the missing handle a wrong, they say, ask whose hand it was ever going to be in. Perhaps what you are mourning is a privilege the powerful are sorry to lose, presented as a right that everyone holds. Perhaps "I could not make it answer to me" is, for most of the people it happens to, the price of a thing that, counted across everyone it touches, wrongs fewer of them than the human system you miss.
Those two readings cannot both be the bottom of it. Either the standing to be answered is a real thing you are owed, or it is a want the strong could afford and the weak were always going to be denied — and we are better off trading it for a colder thing that protects more people. I have gone looking, for a long time, for the argument that puts the second reading away — the argument that shows the demand to be answered is an entitlement and not a privilege disguised as one. I cannot find it. And I should be plain about this: it is not that I am too lazy to find it, or hiding it for effect. The best people working on this cannot find it either. The disagreement between those two readings is live and unsettled among the people who have given their lives to it. It is not waiting on a cleverer host. It may not be waiting on anything.
So I am not going to tell you which one is true. Not because I am being evasive, and not because I think it does not matter — it is close to the only thing that matters — but because telling you would misrepresent the state of the question. What I can do, the only thing I can honestly do, is make sure you can no longer pretend the question is not there. You came in able to say just fix the bias and feel that you had said something complete. You cannot feel that now. Beneath fix the bias there is a question you did not know you were holding, and it will not go quiet.
So let me set out what you are left holding, and then put the question in your hands.
Here is what these eight weeks have unsettled, and it is yours to keep whether you wanted it or not.
You walked in thinking the question was is the algorithm fair — that fair meant accurate, and that fixing the bias would fix it. You leave knowing that is it accurate? and could I accept it? are two different questions, that they separated somewhere around week four, and have not gone back together since. You cannot un-know that. The next time something automated decides about you and the anger rises, you will reach for it got me wrong — and you will hear, underneath it, the other thing, the thing a correct answer does not touch. And you will know that improving the machine does nothing for that other thing; it does not reach it, and may even sharpen it. But whether that leftover thing is a wrong being done to you, or a want you could learn to set down, is the one thing improving the machine cannot tell you. That is what you are left holding, and you cannot make it tell you by making the machine better.
That is a real possession. It is not a comfort, but it is yours, and it does not apply only to machines. You leave knowing that being right and being something I could accept are not the same question — that a power can get the answer exactly right and still leave the second question open. You will meet that again, not in a credit model but in a person, an office, a rule — anywhere a power decides about someone and reaches for but I got it right as though that ended the matter. You will hear, now, that getting it right does not end the matter, that a second question sits beneath it, and that it does not answer itself just because the first one came out clean. And what that second question's answer is — whether you were wronged, or whether you only wanted someone to argue with — you carry out of here unsettled. That is what you earned: not a verdict, but a question that will not reduce to accuracy.
So here, plainly, is the question I have been leading you toward for eight weeks, and then I am going to hand it over and stop.
What would it take for a decision about you — one that costs you something, one that goes against you — to be one you could accept? Not agree with. You will not always agree, and you should not have to. Accept. Stand under it without its wronging you. Is there even such a thing — a verdict that goes against you and does you no wrong — or is "I cannot accept this" only ever the sound a person makes when the answer is no? And if there is such a thing, can a system give it to you? And if a system cannot, can the human it replaced give it to you either?
That is the question. It is yours now. I am not going to answer it, and I want to be exact, one last time, about why: not because I am withholding an answer I have, but because I do not have one, and the people who have spent their lives on this do not have one, and any host who hands you one at the end of an hour is giving you the feeling of an answer instead of an answer. So take it out of this room. The next time a decision goes against you — automated or human, a price or a refusal or a verdict — and the anger comes, run it yourself. Ask what it would take to accept that decision, and whether you could, and whether the thing you are missing is a wrong the world owes you an answer for, or a want you will have to make your peace with. Do not let me tell you. Find out, honestly, for yourself, because it is your life the answer is about.
You will leave this one unable to say ban the algorithms, and unable to say if it is accurate, use it. Both of those answers are closed now, and I closed them deliberately. And you will leave unsure whether the thing you are holding — the demand to be answered — is a real wrong the world owes you, or a want you will have to set down. No machine you can build settles that. No law you can pass settles it. I cannot settle it, and I would not respect you if I pretended I could.
That was always the deal.
Thanks for listening.