You applied for the loan online, and the answer came back in about four seconds: declined. Not a request to talk it over, not a note on what was missing. Declined, with a reference number and a link to a page saying your application did not meet the criteria.
So you call. The person on the phone is kind, and completely powerless. What she tells you, more or less, is that she can see the decision but not the reasoning behind it. The model scored you. She does not know which factors counted, or by how much. She does not know whether the late payment from four years ago is the whole story or none of it. She cannot override it. She can run it again, but it will return the same result, because nothing has changed. There is a box you can tick that says "request a review." You tick it, already fairly sure that the review is the same system, reading the same numbers, reaching the same place.
Now consider the exact shape of what is troubling you, because it is easy to describe wrongly. It is not, mainly, that you think the machine made a mistake. Perhaps it did not. Perhaps you are, by some cold measure, a worse risk than you feel. What troubles you is something else, and it is harder to state. A decision was made about you — about your house, your year, your sense of whether the system is built for people like you — and no one stood behind it. There was no reason you could follow, and no one you could hold to account. The decision was made, and then it was final, with no way back into it.
Here is the question the next several episodes turn on. Suppose it could be shown that the machine is more accurate than the loan officer it replaced. Suppose it is the fairer of the two — less swayed by mood, by your accent, by your name, by what you were wearing the day you walked in. Would that settle what is wrong? Or would the decision still be one you had no way to answer?
Hold that question in view. We will return to it.
This is Philosophy for Us — philosophy for everyone, no degree required. This is the first of eight episodes on what it is to be governed by a machine. Tonight: one screen, one four-second decision, and the question underneath the anger it produces.
I should say what this series is not, because it would be easy to mistake. It is not a survey of what is wrong with algorithms — the bias, the opacity, the data. That account is available everywhere, and most of it is true. I am after something more basic: the thing you actually wanted as you sat there with the reference number, which turns out to be older and stranger than "make it more accurate." I do not have a verdict to hand you at the end. I live inside these systems too, scored and sorted like everyone else. What I can do is make sure that, by the time we are through, you cannot stop seeing the question. That is the whole of the offer.
Go back to the screen. The first task is to describe what went wrong, slowly and precisely, because the rest of the series depends on getting this right — and it is very easy to get wrong.
Count what actually happened, because there is more than one thing here, and they are not the same.
The first is this. You could not find out why. There was a reason somewhere. The model had its reasons — a weighting, a threshold, a line you fell on the wrong side of. But no reason was given to you: nothing in words you could examine, or contest, or even understand. The decision arrived without a reason you could follow. Call that the first wrong: the missing reason.
The second is harder to see, because the first one hides it. Suppose the fix for the first wrong actually happens. Someone sits you down and reads out the reasons, line by line: the late payment counted this much, the thin credit file that much, the postcode a little. Now you have the reasons, and you can follow them. And notice that you still have nothing to do with them. There is no one whose job is to take your objection and be moved by it. The voice on the phone was kind and powerless. The box led back to the same system. No one on the other side of the decision had to answer to you. Call that the second wrong: not the missing reason, but the missing person.
Two wrongs. Stated as plainly as they will go: no reason you could follow, and no one who had to answer. Sitting there with the reference number, they feel like one thing, a single sense that this is not right. But examine them and they separate, and almost everything in this series concerns the difference between the two.
Here is why the gap matters. Faced with that screen, most of us reach for the same response, and it is a reasonable one. The response is: make it fairer. Fix the machine. Make it more accurate. Remove the bias. Make it show its working, so the reasons are not hidden. If it scored you wrongly, get a better score. If it scored you without explanation, make the reasoning visible.
Every part of that is worth wanting. But look closely at what it addresses. "Make it fairer" aims at the first wrong — the bad reason, the hidden reason. It amounts to saying: get the reasons right, and let me see them. Now look at what it leaves untouched. It does not address the second wrong at all. You can make the reasons perfect. You can make them accurate, visible, audited, published in full. And there is still no one who has to answer to you. A better score does not provide a person. Making the reasoning visible does not put anyone there to hear you. Fixing the reason leaves the missing person exactly where it was.
That is the shape of the whole matter, and it is worth fixing in mind before we go further. There are two wrongs at that screen. Your first and best response is aimed entirely at one of them. The other it does not even notice.
Before we go on, two responses to set aside for now. Both are tempting; you can feel the pull of each already. And both are too quick — not wrong, exactly, but premature. We will return and examine each one properly later. For tonight they only need to be off the table, so that we do not settle on either before we have looked at the problem.
The first response is the angry one. It says: so the machines are the problem — remove them, and put a human back. A real person, who can look you in the eye, hear you out, change their mind. On this view the whole grievance is simply the coldness of the thing, and the remedy is to restore the warmth that used to be there.
I understand the pull of it. But notice how quickly it skips a question. It assumes a human would not have reached the same decision — and often that is simply not true. Consider the loan officer the machine replaced: the one who could be charming or curt depending on his morning, who liked your face or did not, whose reasons, if you ever heard them, were the ones he reached for after he had already decided. Did he answer to you? Often less than the machine does. He could not even be audited; the model at least leaves a record. So "put a human back" is not yet an answer. It is a hope presented as an answer — the hope that somewhere behind the decision there is someone who will finally be moved by you. We will test that hope, and hard, later in the series. Tonight, simply note that it is a hope, not a finding. Some people refuse you more firmly than any machine, and are pleasant while they do.
The second response is the opposite, and it is the more serious of the two, so give it room. It says: if the machine is accurate, what exactly is your complaint? It scores better than the people it replaced. It is less biased. It identifies more of the real cases and ruins fewer of the innocent ones. Be grateful, and stop idealising the human who used to wreck lives on a whim. Call this one the shrug.
And here is the important thing about the shrug. It is neither stupid nor heartless. Serious people hold it — people who build these systems, and people who study them — and many hold it on behalf of exactly those who used to suffer most under human judgment: the poor applicant, the one with the wrong name, the one the old system waved through on a hunch or turned away on a mood. The shrug says: your wish to argue with someone is a comfort, and that comfort is not free — someone else ends up bearing its cost. That is a real argument. It is, I will say now, the strongest thing anyone says in this whole debate. Exactly how that cost is borne, and by whom, is next time's business, not tonight's. I am only naming the position here, so you know it is coming.
So I will not dismiss it. I will do the opposite. Next time I will state it at full strength and put it to you in terms you already accept. But not here, and not as a reflex — because the shrug, too, skips a step. It tells you the machine got the decision right. It does not tell you why getting it right is the whole of what you wanted. And whether it is the whole of what you wanted is precisely what we have not yet examined.
So set both responses down: not the removal, not the shrug. Stay in the gap between the two wrongs a little longer, committed to neither answer.
I think we have all been making the same small mistake for years without noticing, and this is the centre of tonight's episode.
When a machine decides something about you and you become angry, you reach for one word to explain the anger — and the word is wrong. You say: it got me wrong. The credit score is wrong about me. The fraud flag is wrong. The filter that discarded my application is wrong. The anger feels like the anger of being mismeasured, of being added up wrong by something that does not know you. And so the remedy seems obvious: measure me correctly.
But watch what happens when I remove that remedy — not by refusing it, but simply by granting it. Treat this as an experiment on yourself, not a claim I am proving to you. Grant, for a moment, that the machine got you exactly right. The score is correct. You really are, by every honest measure anyone could name, the risk it says you are. No error in it, no bias in it. The number is true.
Is the anger gone?
Be honest about the actual answer, not the one that sounds better. For most people, told sincerely that the machine had them right, the anger does not leave. It changes. It grows quieter, and harder to say aloud, because now there is no mistake to point at. But something remains. And that remaining part — the part that survives being told, "no, it had you exactly right" — is what this whole series is pursuing.
Consider what that survival shows. If your complaint had really been "it got me wrong," then granting that it got you right should have dissolved the complaint entirely. Nothing should remain. That something remains means the complaint was never only about being got wrong. You described it that way because "wrong" was the word ready to hand — and because it is the part you can argue about, the part with a number attached, the part you can take to a regulator. But underneath that part, you were angry about the other wrong, the second one, the one a correct score does nothing about. You were angry that there was no one to answer to. You called the whole thing by the name of the first wrong, because the first wrong is the one that comes with evidence.
And there is a reason you always reach for the first wrong, a reason that is not your fault. Look at what the system actually offers you. The page says your application did not meet the criteria. The box says "request a review." The letter, if you receive one, lists factors. Every channel it provides is a channel for one kind of complaint — the complaint that a fact is wrong. Wrong income. Wrong address. A payment that was in fact on time. You can dispute a fact; there is a form for it, and somewhere to send it.
But there is no box that says "I want someone to answer to me." There is no form for the second wrong. The system is built to receive the first complaint and has no place at all for the second. So your whole grievance is directed into the one form the system will accept — the dispute-a-fact form — because that is the only opening available. You did not choose to make it all about accuracy. You were channelled into making it about accuracy. The shape of your complaint was determined by the shape of the only opening you were allowed to use.
That is worth slowing down for, because it is a deeper problem beneath the first. It is not only that you misnamed the wrong. It is that the system rewards the misnaming and quietly discards whatever does not fit it. The part of you that wanted a person to answer gets no acknowledgement, no reference number, no channel. It is treated as though it were not a complaint at all. And a complaint the system will not even let you file begins, after a while, to feel unreal — as though it were never a complaint, only a mood. Hold that thought; we will need it.
If you were here for an earlier series of this show — Knowledge and Power — you will feel an old question turning into a new one. There, we kept asking whose word the system believes: when you and the machine disagree, whose account is treated as the truth, and whose as noise? Whose testimony counts? That question is still alive. But tonight it is the same machine seen from the other side. Not whom it believes, but to whom it must answer. And the two come apart. The system can believe you completely — it can get every fact about you right, accept your whole account — and still be something you cannot make answer for what it then does with you. Being believed and being answered to are not the same. You can secure the first and lose the second, and the loss is what remains once the accuracy is gone.
Now, I am being careful not to name that remaining thing too quickly, and the care is deliberate, because the naming is exactly where the real dispute lies, and we have not earned it yet. I could give it a grand word now — a term from political philosophy that would make it sound settled and handled. I will not, because naming it would let you file it away and feel finished, and you are not finished; none of us is. The honest report, for tonight, is smaller and stranger than a grand word. When you take accuracy off the table, the grievance does not leave. It remains, with nothing to point at. We will spend the rest of these eight episodes finding out what it is — and, harder, whether it is a real wrong the world owes you an answer for, or only the noise a person makes when a decision goes against them. Tonight, only this much: it is there. And it is not the same thing as "the machine got me wrong."
I have been running this on my example. Now take yours, because the test only shows something if you run it on a case I did not build for you. A case I set up, I can rig; one of your own, I cannot.
So set the loan aside. Here is a different one.
On a Tuesday morning your bank account is frozen. No warning. Your card is declined at the petrol station, with people waiting behind you, and when you finally get a signal and log in there is a message: unusual activity, account locked, please contact us. So you contact them. And it is the loan call again, in a different setting. A person who is sorry. Who cannot tell you what triggered the flag. Who cannot lift it, because the system raised it and the system has to clear it, and it clears on its own schedule, not yours. Three days, they think. Your rent goes out on Friday.
Now do the two things we just did, but do them yourself, about this.
First, the reflex. The flag is wrong. There was no fraud. The machine made a mistake. And you are very probably right. So grant yourself the fix. Picture the fraud system improving — catching the real thieves, freezing fewer innocent people, learning the difference. Good. Genuinely worth having. Keep it.
Then do the harder half, the one that costs something. Suppose the flag was not a mistake. Suppose that on the numbers your account that morning really did look exactly like a stolen one — the same pattern, the same odds, the same signature the thieves leave — and that freezing it was, coldly, the correct call, the one that protects the most people, you included. Grant the machine all of that. It was right.
And now ask yourself, plainly: is there nothing left? Or is there still something — that for three days a process locked you out of your own money and stopped your week, and there was no one you could stand in front of? No one who had to hear "my rent is due Friday" and answer you. Not a person who chose to be unmoved. No person at all. A process, clearing on its own schedule, while you waited.
If you felt something survive that second supposition, then you have found, in yourself and not from me, what these eight episodes are about. You have found that "was the machine right?" and "could I accept what it did to me?" are two different questions — because you have just answered yes to the first, and you are still unsure of the second.
And if you felt nothing survive — if, the moment you granted that the freeze was genuinely the right call, your complaint simply went away — then acknowledge that just as honestly, because it is a real position, and a serious one, and the people who hold it are not fools. The series owes you an argument, not reassurance. So do not manufacture a leftover feeling to keep me company. The whole point of the experiment is to find out what is actually there once the accuracy question is closed. For many of us, something is there, with nothing to point at. For some of us, perhaps not. Either way, you now know how to look — and you know the answer is yours, not mine.
Let me close with what changed tonight, and what remains open. Nothing here was settled. We took something apart instead.
Here is what you have now that you did not have an hour ago. You came in with one word for your anger at the machine — it got me wrong — and one remedy for it: make it more accurate. You leave able to see that the one word was covering two different wrongs, and that the remedy reaches only one of them. No reason you could follow. No one who had to answer. "Make it fairer" is aimed entirely at the missing reason, and rightly so. The missing person it cannot reach. That is not a comfortable thing to know. If anything, it takes an easy answer away from you. From tonight on, you will not be able to hear "just fix the bias" without also hearing the other half of the complaint, the half that remains — and you will not be able to stop hearing it.
And you have a question you did not walk in with, the one the whole series is built around. Not "is it accurate?" — we are nearly finished with that one. The question is this: could a decision that went against you, that cost you something, that hurt, be one you could accept anyway? Not agree with. Accept. Stand under it without its wronging you. What would that even require? And could a machine provide it — and here is the part that should leave you a little uneasy, could the human it replaced provide it either?
This is the first of eight, and the lightest step of the eight. All we have done is open the question and take apart, carefully, what is troubling you.
Next time we take your own reflex completely seriously: "make it fairer." We follow it all the way to the people who build these systems and mean every word of it, because they have an answer, and it is better than you expect. They will tell you, to your face, with the worst-off standing behind them, that the machine is the fairer of the two choices — less biased than the humans it replaced, fewer lives wrecked — and that your wish to argue with a person is a luxury other people pay for. That is not a strawman. It carries real force, and next time it is put directly to the easy version of your own position. Come ready to lose that easy version. You will get it back later, if you can earn it, but harder.
Until then, carry the small thing you found tonight. The next time something automated decides about you — a price, a flag, a yes, a no — try the experiment. Grant that it got you right. And then look hard at what still remains once the accuracy is gone, and ask yourself what it is. Do not let me name it for you. Find out, for yourself, whether you think it is a wrong the world owes you an answer for, or only the sound a person makes when a verdict goes against them. You will not settle it tonight. You are not meant to.
Thanks for listening. I'll see you next time.