For two episodes we have set the machine aside. Tonight we return to it, and we take the hardest step this series asks. Let me say in advance exactly what I am going to do, because it may sound as though I am changing sides.
Recall the screen from the first episode. You remember all of it — the four seconds, the word declined, the sympathetic voice on the phone that could see the decision but not the reasons behind it, the review box you completed knowing it would change nothing. Two episodes ago we found that there were two complaints there, not one — that you were given no reason you could follow, and that there was no one who had to answer to you — and that your first instinct, to make the decision fairer, addressed only the first of them.
Here is what I am going to do tonight. I am going to grant the people who build these systems everything they have ever asked for, all at once, and more than they have now. Then I am going to put you back in front of the screen and ask a simple question. When the machine is perfect — when it is more accurate than any human who ever held your file, when there is no trace of bias left in it that anyone alive can find, when it is governed and audited and overseen — is the thing you were complaining about gone, or is it still there?
I am not going to tell you the answer; that is not mine to give. You will have to find it for yourself. And whatever you find, you will have to say honestly what it is, because there is more than one honest answer, and the whole of tonight is learning to tell them apart.
This is Philosophy for Us — philosophy for everyone, no degree required. This is the fourth of eight episodes on what it is to be governed by a machine. Last time there was no machine at all: we set it aside completely and built a tool. We separated three things you had been treating as one — whether a decision is correct, whether the body making it is entitled to make it, and whether that entitlement was come by in a way you could accept — and we watched "correct" and "could I accept it" turn out to be different questions, on a referee, on a wise friend who is not your boss, on a case from your own life. Tonight we bring the machine back and apply that tool to it, by granting it the one thing the tool says is only one of the three. We grant that it is correct. And then we look for what remains.
So let me make the grant, and let me make it completely, because a grudging grant is no grant at all, and the whole experiment is worthless if I leave myself an escape.
Consider the best system anyone could honestly build. Not the one we have. The one the people who believe in these systems are working toward, and some of them are not far off. It has been trained on more cases than any human loan officer could see in a thousand careers, and it has learned which facts predict whether a loan comes good and which only appeared to. It is more accurate than the best human who ever held your file — not slightly, but considerably. When it judges you a risk, it is right more often than any person would be, about more people, including the people the old human discretion used to get catastrophically wrong.
And the bias — the thing we spent the last two episodes on — is gone. Audited out. Every protected group, every proxy for one, every hidden correlation that let the postcode stand in for something the law forbids: found and removed, the audit repeated every quarter by people whose whole job is to find what remains, and the results published. There is no trace of the old unfairness in it that anyone alive can find. By every measure we know how to build, it is fairer than the human it replaced. And recall where last week left us: the human it replaced was often worse, more swayed by your accent and your name, and the people who paid for that human's bad days were the ones who could least afford to. So this is not a cold corporate convenience. On the numbers, deploying this machine spares people real harm. Fewer wrecked lives, not more. That is the case, and it is a serious one, made by serious people who mean it.
And it is governed. This is the part it is easiest to get wrong, so hold it exactly. The machine does not answer to nobody. It sits inside a system of oversight — regulators who can inspect it, rules it must meet, auditors it must satisfy, a documented record, a regulator with real power to act. If it drifts, someone is meant to catch it. If it breaks the law, someone can be made to answer for that, to the regulator. It is, in the fullest institutional sense, accountable. I want to be clear about this, because the easy version of tonight's argument is to call this machine "unaccountable" and dismiss it, and that would be a cheat. It is not unaccountable. It is watched, checked, and answerable upward, to the bodies whose job is to watch it.
So apply our tool to it. Three questions. Is it correct? Yes — more reliably than any human, and we have granted that completely and we are not taking it back. Is it the body entitled to decide? Grant that too: the bank is allowed to decide whom it lends to, the law permits it, no one seized a power that was not theirs. And is its power governed, overseen, answerable to the institutions that license it? Yes again. On the first of our three questions it fully succeeds, and permanently. No better answer is coming. This is as correct as a decision about you has ever been, or perhaps ever could be.
Now stand in front of the screen with all of that true — every word of it, granted and meant. The machine got you right. The machine was fair. The machine was watched. And look honestly at what remains, because something will be there, and the rest of the hour depends on whether you will be honest about it.
Here is what I think you find. I am going to describe it carefully, and then I am going to stop short of telling you what it is, because that last step is yours and not mine.
The complaint did not go away. You sit there with the reference number, and the decision is correct — we proved that, you can no longer say they got me wrong, that complaint is gone. And the decision is unbiased — you can no longer say it treated me worse because of who I am, we closed that one too. And the decision is watched, governed, lawful — you can no longer say this thing answers to nobody, because it answers to a great many people. Every complaint you walked in with has been answered. And something is still there. Not loudly. But it has not gone.
So let us be exact about what it is, by the only method that works: saying what it is not. It is not that the machine erred, because it did not. It is not that the machine was unfair, because it was not. It is not even that no one is watching the machine, because someone is. Set all of those aside — genuinely set them aside, and do not let one of them return in disguise — and look at what is left underneath.
What is left is this. A decision landed on your life — and it landed, as we said in week one, on your house, your year, your sense of whether the system is built for someone like you — and there was no one you could address. No grounds you could take up and argue against. No one whose job it was to hear you, in particular, and to answer to you — not a regulator, not an auditor, but you, the person the decision fell on. The reasons existed. The oversight existed. But between you and the thing that decided about you there was no point of contact — no one obliged to take your account and respond to it. The decision was made, and then it was final, and you had no part in it. The machine was perfect, and you were still left with no one to turn to.
Now, I am not going to tell you what that is. I would like to, and I am not going to, because the moment I name it I have decided the whole question for you, and that is exactly the thing I promised this show would never do. But I can tell you what has just happened, which is enough to be going on with. Two episodes ago, standing at this same screen, you would have said the problem was that the machine might get you wrong, and the fix was to make it more accurate. You have now watched the machine get you right — completely, permanently, fairly, lawfully — and the problem did not go with the inaccuracy. It remained. Which means it was never the inaccuracy. The thing you were actually holding, the whole time, was something else, and you have just felt it separate from accuracy and stand on its own, with nothing left to attribute it to.
That is the belief beginning to fail. You came in thinking is it fair? and did it get me right? were the same question. You have just watched a decision be flawless and still leave a complaint standing — a complaint that getting it right did not touch and could not touch. So they were two questions all along, and you have spent your life running them together. Was it correct is one thing. Could I accept it is another. And tonight, in front of a machine that answered the first one outright, you felt the second one refuse to be answered by it.
Now there is a quick way out of the position I have just put you in, and it is so reasonable-sounding that most people take it without noticing it is a way out. So let me block it, because it is wrong, and seeing why it is wrong is most of tonight's work.
The reply goes like this. So the problem is that it would not tell me why. The machine's workings are hidden. Open them up. Make it explain itself. Give me the reasons, in plain language, and then I will have what I need, and the complaint will be gone. Transparency. That is the fix people reach for second, once "make it more accurate" has failed them. And it sounds like exactly the missing thing.
It is not. And I can show you it is not without any cleverness at all, simply by supplying the fix in full and watching it fall short. Suppose the machine does explain itself, perfectly. It sets out for you — or a clean, honest page does — every reason, in order, in words you can follow. The thin credit file counted this much. The gap in your employment counted that much. Two addresses in three years, a little. Here is the threshold; here is the line; here is exactly where you fell on the wrong side of it, and why, and by how much. Nothing hidden, nothing in jargon. You asked why, and you have been told why, completely.
And you still have nowhere to take it. Read it back. It changes almost nothing. You now know the reasons — and there is still no one whose job it is to take your answer to those reasons and be moved by it. You can see exactly why the gap in your employment counted against you; you cannot say to anyone that gap was the year I spent nursing my mother and it will never happen again, and have that reach someone, and matter. The page explained. The page cannot listen. You have the reasons, and you have, still, no one to give your reasons to.
We saw this two episodes ago and could not yet say why it cut so deep. There were two complaints at that screen, not one: no reason you could follow, and no one who had to answer to you. Transparency is the cure for the first complaint, and it is a real cure, and I am not disparaging it. Give me the reasons — yes, good, I want them. But look at what it does to the second complaint: nothing. It does not touch it. You can set out every reason fully and publicly, in the plainest possible terms, and the second complaint stands exactly where it stood before, because the problem was never a lack of information. The missing thing was never information.
So whatever is left over once we have made the machine perfect, it is not a reason you are owed and have not been given. We can give you the reason. It is something that a reason on a page does not reach. What it is, exactly, I am not going to say yet — naming it is choosing what it is, and that is the one thing I keep refusing to do for you. But I can rule out the easy mistakes. It is not accuracy. It is not bias. It is not secrecy. It is not the absence of oversight. It is the thing that is still there when all four of those are fixed.
I have been running this on the loan screen, which I chose, and a case you did not choose will show more than one I did. So take a fresh one, and run the whole thing yourself, because a step like this only counts when you take it on a case I never touched.
Here is one that is not the loan. You apply to rent a flat. You need it; the old lease is ending; you have the deposit ready. And the letting agent runs your name through a tenant-screening service — these are everywhere now — and a score comes back, and the score says no. Not "let us talk." No. And the agent is apologetic and completely powerless, and tells you, in effect, that the system flagged you and they cannot let to a flagged applicant, and no, they do not know exactly what flagged you, and no, there is no one you can call. You have been refused a home by a number, and the next viewing is already booked for someone else.
Now do to it what we did to the loan. Grant the screening machine everything. Suppose it really is more accurate than any landlord's instinct — and instinct, remember, is where much of the worst discrimination in housing has always operated, so this is no small thing. Suppose it predicts who will fall behind on rent better than any human could, with the bias stripped out and the whole thing audited and overseen. Suppose that, in cold fact, you are a worse bet than you feel like, and the machine simply saw it. Grant all of that, and mean it. Now ask yourself, honestly: is the thing you are feeling gone?
You already know it is not gone. And you can locate exactly where it now sits, because you have the tool. It is not on correct — you granted correct. It sits in the same place it sat at the loan screen: that a decision refused you a home and there was no one you could turn to, no one obliged to take your account of yourself and answer it. And here — this is the part I will not do for you, the part that is the whole point — ask what that leftover thing actually is. Because there are two honest answers, and I am not going to choose between them tonight, and neither, if you are honest, can you.
The first answer says: that is a real wrong. It says the thing you have been demanding, at the loan screen and the letting agent and quietly your whole life, was never only that the power get it right — it was that the power be one you could answer, that there be someone who has to face you and account for what was done to you, and that this is something you are genuinely owed, as a person, that getting the decision correct was never going to supply. On that reading, the machine wronged you even though it got everything right, because it left out the one thing a person is owed that has nothing to do with the verdict being correct.
And the second answer — put to your face by the most serious people in this argument, who are neither fools nor cold — says: no. That leftover thing is the ordinary reaction of a person when a decision goes against them. It is the wish to argue with someone, to have someone to appeal to, to feel heard — and it is a comfort, a real and human one, but a comfort, and not a wrong done to you. And here is the hard part of their case: indulging that wish has a price, and other people pay it. Every requirement that a human sit and hear out every applicant is slower and more expensive, and lets more of the old bias back in through the fatigue and lapses of a tired human — and the people who lose by that are not you, they are the ones the machine was protecting, the ones a landlord's instinct always turned away. So your sense of being refused a hearing, they say, is a luxury, and you are asking the worst-off to pay for it.
Both of those fit the thing you have just felt. Read them again and you cannot tell, from the feeling alone, which one it is — whether you have found a wrong the world owes you an answer for, or a want you are going to have to make your peace with. That is not me being evasive. That is the actual state of the question, held by the best people working on it, and it does not become easier the more accurate the machine gets. If anything it becomes harder, because the better the machine works, the more clearly the only thing left to complain about is this — the thing that has nothing to do with whether it was right.
Let me close with what changed tonight, and I want to be honest that it changed by taking something away from you, not by handing you something.
You came in still half-able to believe that fixing the machine would fix your complaint — that is it fair and did it get me right were, underneath, the same question, and that accuracy was the route to both. Tonight you made the machine perfect with your own hands, granted it everything, and watched your complaint refuse to leave along with the inaccuracy. So you can no longer believe that. Was it correct and could I accept it separated in front of you, on a machine that answered the first one outright — and you saw that improving the machine does nothing at all for the second. It may even make it worse: the more flawless the decision, the more final it is, and the less there is to say except the one thing that was never about whether it was right. That is the belief failing in your hands. And once it has failed, you cannot put it back together.
But I have to be just as careful about what I have not shown you, because the easy mistake from here is to think I have proved the leftover thing is a real wrong — that I have quietly handed you the verdict that being able to answer the power is what you were owed all along. I have not, and I will not, and you should not let me. The serious people on the other side are still here and their case is still standing: that what survived the perfect machine is the wish to argue with someone, a human comfort that costs other people real harm, and that a grown-up may have to give it up. I cannot find the argument that answers them. I have looked, and the best people working on this have looked, and it is not there. So you leave tonight holding both — a complaint that survived a flawless decision, and two honest readings of what that complaint is, and no way to settle, from the feeling alone, which it is. I am not going to make this tidier than it is. A tidy answer here would be the lie.
So here is where we go next, and you can see the shape of it. We have found that there is something the perfect machine leaves out — something that is not accuracy, not bias, not secrecy, not the absence of oversight. We have ruled out the easy mistakes. What we have not done is say what it is. And it turns out there is more than one serious answer to that — more than one account of what the missing thing could be, each held by people who have thought about it harder than either of us. Next time we go and meet them, and we find out whether the thing you felt separate tonight has a name, and whether the name changes anything. I am not going to tell you now which account is right, partly because I am not sure, and partly because you have to arrive at them holding the question, not the answer, or they will mean nothing.
Until then, carry the experiment. The next time something automated decides about you — your rate, your limit, a yes, a no — do the whole of tonight in your head. Grant that it got you right. Grant it was fair. Grant that someone is watching it. And then look at what is still standing once all of that is true, and try to say, out loud, which of the two it is — a wrong you are owed an answer for, or a want you will have to make your peace with. You will not be able to. And learning that you cannot — that you have been carrying a question you never once stated and cannot yet settle — is the whole of what tonight was for.
Thanks for listening. I'll see you next time.