You have been driving for one of the ride-hailing apps for two years. It is not a great living, but it is a living, it is yours, and you are good at it. Your rating sits near five stars, which took work to keep there.
Then one Tuesday morning the app will not let you start. A grey screen. Your account has been deactivated. No shift, no income. You tap through, and there is one line: there has been activity on your account inconsistent with the terms. That is all of it. You did nothing you can think of. You tap the help link, it opens a form, you fill in the form, and a day later a message comes back: your case has been reviewed and the decision stands. Reviewed by whom, it does not say. Somewhere there is a flag, a score, a model that decided you had become a risk, and between you and that model there is a web form and a sentence with no name attached to it.
You recognise this pattern by now. You have been examining it with me for three weeks. A decision fell on your life quickly, then it closed, and there was no one on the other side of it. And you know by now not to reach first for the reply it got me wrong, because last week I removed that reply.
Last week we built the machine you cannot fault. We granted it everything the people who build these systems have ever asked for: more accurate than any human who ever held your file, its bias audited out until no trace remained, governed and overseen and answerable to every regulator in the country. It settled the question did it get me right, completely and permanently. And then we put you back in front of the screen, and the thing you had walked in complaining about was still there. So we went looking for what that thing was. We ruled out the easy answers — it was not the error, not the bias, not the secrecy, not the missing oversight — and once we had isolated it, I declined to name it. I said there was more than one honest account of what it might be, and that I would not choose between them for you.
This is Philosophy for Us — philosophy for everyone, no degree required. This is the fifth of eight episodes on what it is to be governed by a machine. Last week we made the machine perfect, watched the complaint survive, and stopped short of naming what was left. Tonight we do the opposite. We go and meet the people who will name it.
And here is what makes the question worth the hour. There is not one serious account of what the leftover is. There are two. Two careful, well-defended accounts of what you were owed that the perfect machine did not provide — and they do not agree. They point to two different missing things. By the end you will be holding both, and finding that they cannot be combined into one.
One warning before we start, which I will repeat at the end because it is the easiest thing to lose tonight. We have not shown that the leftover is a wrong at all. The position from last week still stands: that the thing you are missing is a wish for control under a more respectable description. Tonight two serious thinkers will give that thing a name — two names, in fact — and a name carries weight. It can make a feeling sound like a right. So hold the warning in mind. Naming the leftover twice does not make it real. Whether it is real is still open, and giving it good names is not the same as settling it.
Start with the word we kept reaching for last week, because the whole of tonight is contained in it.
When we had the leftover isolated, recall how it was described. There was no one who had to answer to you. No grounds you could take hold of and push against. The decision was made, then it closed, and you had no way to challenge it. That is the word: answer. You wanted a power you could answer, or a power that had to answer you. And that word is doing two entirely different jobs at once, and the difference between them stays hidden until you look for it.
Here is the difference. To want a power you can answer can mean one of two things, and they are not the same, and most of us have never had to tell them apart.
The first is this. You wanted a way to challenge the decision — to put your side, to make your case to something that could actually change its mind. Not to be told the verdict and shown the exit, but to contest it, and to have the contest count. On this reading, what was missing is a channel: a way in. The decision came at you in one direction, and you wanted it to run in both. Call this the sense in which you can answer the decision — you can push back, be heard, and perhaps change the outcome. Call it a handle: something on your side of the door you can take hold of.
The second is different, and quieter, and it does not disappear even if you are given the first. You wanted there to be someone there. Not a channel — a person. Someone who had to face you, who had to take account of you, this particular person with this particular life, rather than sort you by a rule and read off the result. On this reading the wrong is not that the decision ran in one direction. The wrong is that there was no one on the other end at all — no one who owed you, as a person, the account that one person owes another. Call this the sense in which someone is answerable to you: a who, not a how. Call that the someone.
Notice how far apart those two are, because the rest of the hour is about the gap between them. You can have the first without the second. Consider a system with a genuine appeals process — a real channel, a review that can overturn the result — run entirely by software, with no human involved. A way to challenge it: yes. Someone there: no. And you can have the second without the first. Consider an official who looks you in the eye, hears your whole story, and sympathises — and then tells you the rules give you no way to challenge the outcome, and that is that. Someone there: yes. A way to challenge it: no. Two different missing things. Last week you felt one of them and could not tell which, because the single word covered both.
So here is what we will do tonight. There are two serious accounts — two traditions, really — that have each taken one of those two things and worked it out fully. One says the missing thing is the channel to contest, and builds an account of legitimacy on it. The other says the channel is a consolation, and that the missing thing was always the person, and builds a different account. We will take each at full strength. And we will find that they cannot both be the last word, and that you cannot split the difference between them.
One more thing before we begin, and it is the warning from a moment ago, made precise, because everything tonight depends on it. I am going to spend this hour speaking as if the leftover is a real wrong — working out what shape such a wrong would take. That is the only way to find out what we would even be claiming. It is not a concession that it is a wrong. Working out the two possible forms of a thing does not prove the thing is there. The position from last week — that the whole leftover is a wish for control you ought to be willing to trade away — is not refuted by anything in the next forty minutes. It returns at the end. We are going to give the leftover two names. Keep asking, throughout, whether a thing with a good name is thereby a real thing.
The first account takes that channel to contest as the whole of the matter, and for many listeners it will feel, at first, like someone finally stating plainly what they had been trying to say.
It comes out of the oldest idea we have about fair power, and that idea is not about getting the answer right. It is the idea of due process. For centuries, when we have worried about a power deciding against a person — a court, a magistrate, the state coming for someone's liberty or land — the first demand was never be correct. It was: tell the person what is alleged. Let them be heard before you decide. Give them a way to challenge it. Have someone with the authority to overturn the decision look at it again. We built all of that not because it guarantees the right answer — it does not; courts get it wrong constantly — but because a decision reached that way is one a person can stand under even when it goes against them, and a decision reached the other way is not. Due process is the name for the channel to contest. It is what turns a verdict that falls on you into a verdict you were a party to.
The legal scholar Danielle Citron gave the missing thing a name. In a 2008 article, "Technological Due Process", and then with Frank Pasquale in a 2014 paper on what they called the scored society — the world in which a model rates you and the rating decides — she made a plain argument. We did not abolish due process when we automated the decision. We quietly stopped providing it. The automated decision gives you no notice you can use, no real hearing, no channel to challenge that can move the result, no one with the authority to look again. By the standard we have applied to serious power for eight hundred years, this is not a hard case. It is a plain failure of process. The wrong is not that the machine might be mistaken. The wrong is that it decided about you in a way no legitimate power is permitted to — without giving you any way to challenge it.
There is a deeper version of this underneath, because it explains why the ability to challenge is not merely a convenience. There is a tradition in political philosophy — the republican tradition, given its modern form by the philosopher Philip Pettit in his 1997 book Republicanism — that defines freedom not as being left alone, and not as receiving good outcomes, but as not being dominated: not living at the mercy of a power that can act on you as it pleases without having to answer for it. On that view a kind master is still a master. You are not free because he treats you well; you are unfree because he could treat you however he liked and you would have no recourse. What makes you free is not his good behaviour — it is that he cannot act against you without facing a challenge he must answer. Apply this to the automated system, and the implication is direct. A benevolent, accurate, well-audited system that you cannot contest is precisely the kind master. It is being good to you. It is not answerable to you. And the difference between those two — between a power that happens to treat you well and a power that must answer if it does not — is the whole of the matter. The ability to challenge is what makes you a free person under the decision rather than a well-treated subject of it.
Now consider what this account says about last week, because it is confident about it. Of course the perfect machine did not resolve the complaint. You were never complaining about accuracy — we established that. You were complaining about the missing ability to contest, and making the machine more accurate does not supply that. Indeed — and this is the most striking implication of the view — a system that is less accurate but that you can genuinely contest is, on this account, more legitimate than a flawless one you cannot. That is worth restating, because it sounds wrong and is meant to. A clumsy human tribunal that hears you out and can be overruled has more legitimacy, on this view, than an infallible model that is right every time but cannot be challenged. Because legitimacy was never accuracy. It was the ability to contest. And the ability to contest a slightly worse decision is worth more than a flawless decision you cannot question.
If you came into tonight assuming the answer would obviously be make a human do it or open up the model so we can see inside, notice that this account offers neither. It does not care whether a human or a machine decides. It does not even mainly care whether you can see how the decision was reached. It cares whether you can challenge it and have the challenge count. A system you cannot see into but can genuinely appeal is, on this view, better than a transparent one you cannot appeal. The missing thing was the ability to act, not the ability to see how the decision was made. Not show me why, but let me challenge this, to someone who can change it.
That is the first account at full strength, and it is serious, and it names something real. For many listeners it will feel like an exact description of what was missing. But it has a weak point. And then there is a second account that says the whole thing is aimed at the wrong target.
Here is the weak point. A way to challenge a decision is worth only as much as what happens when you use it. And what happens when you use it depends on something the channel itself does not supply: whether, when you give your account, you are believed.
We devoted an earlier series to exactly this — Knowledge and Power. If you heard it, you will see this coming. If you did not, here it is in a sentence: the world does not distribute credibility evenly. Some people, when they say that is not what happened, let me explain, are heard as a person with a case. Others, saying the same words, are heard as noise — as excuse-making, as exactly what someone in their position would be expected to say. Whose word counts as testimony and whose is discounted is not distributed fairly, and never has been.
Now set that beside the ability to contest. A right to challenge a decision is a right to give your account and have it weighed. But weighed by whom, and how? Consider who is involved. The people a scoring system is most likely to flag wrongly — the thin file, the unusual pattern, the life that does not sit neatly in the data — are very often the same people whose word, when they come to explain, has always been doubted first. So the channel is there, real, written into the rules. But it is weighted. The person who most needs the challenge to succeed is the person whose account is least likely to be believed when they make it. You can build the channel perfectly and the challenge will still go one way, because the challenge was never only a channel. It was a channel plus a question — do we believe you — and the channel answers the first and is silent on the second.
That is the mild version of the difficulty. The blunt version is what has happened each time someone has tried to write the right to challenge into law. The European Union's data-protection law, the GDPR, contains a provision — Article 22 — that says, roughly, you have a right not to have a decision with real stakes made about you by a machine alone, and a right to contest such a decision and obtain human review. A right to challenge, written down and guaranteed. And in practice it has been nearly ineffective. The human review turns out, again and again, to be a person who approves what the model produced. The contest turns out to be a form that routes back to the same logic. The right is written into the rules, and you invoke it, and nothing moves, because no one with the standing and the will to overturn the decision was ever actually there. Writing you may contest this into the law is easy. Placing a person there who has the power to be moved by your challenge, the duty to weigh it, and the willingness to credit your account — that is expensive and slow, and it is the first thing cut.
In fairness to the account, this is not a refutation, and I will not present it as one. A defender of the first account says, correctly, that none of this shows the ability to contest is the wrong thing to want. It shows it is hard to build and easy to fake. So build it properly. Make the review a person who can overrule the model. Address the credibility problem: train the reviewers, shift the burden of proof, stop treating those the model flags as already guilty. All true. The ability to contest can be made real; mostly we have simply not paid for it. The account survives.
But notice what the difficulty has exposed, because it is the opening the second account uses. Make the ability to contest as real as you like — a genuine channel, a reviewer with real authority, your account fully credited — and you have given the person something. You have not yet settled what. There are two quite different things you might say you have given them. One: a working process, a fair contest, a challenge that counts — and that, says the first account, is the whole of legitimacy, and the job is done. But the other is stranger, and the second account will insist on it: that what made even the perfected process worth anything was never the process itself. It was that, somewhere in it, at last, there was a person — someone who had to sit there, take your account, and answer to you for what they did with it. On that reading the process was never the point. The process was only the least costly way of ensuring a person was involved. And if that is right, then a process with no person in it — the flawless automated appeal, the review with no one behind it — gives you the form of the thing and not the thing. Which is exactly where the second account begins.
The second account says the ability to contest was never the heart of the matter. The heart of it was the person who must answer. And it builds the wrong on a different foundation — not on process, but on what a person is owed as a person, before any process begins.
Start with what a scoring model actually does, described honestly. It does not decide about you. It decides about people like you. It finds the pattern that the thin file, the gap in employment, and the two addresses in three years tend to fall into, and it puts you where the pattern says you belong. Even when it is right — even when you really do belong in that group, and the group really does behave as the model says — notice that it has not, at any point, met you. It has located you. You arrived as a person with a particular life and a particular reason for each item on the file, and it processed you as an instance of a class. The computer scientist and legal scholar Reuben Binns, in a 2022 paper on individual justice in automated decisions, drew exactly this distinction: between being judged as yourself and being judged as a member of a set. And his claim is that something is owed to a person that being judged as a member of a set can never provide, however accurate the set.
You can hear the older version of this in Kant, in the Groundwork of the Metaphysics of Morals of 1785 — the line most people half-remember and slightly misstate. Never treat a person merely as a means. The word that matters is merely. We may use people; we use one another all day — the driver, the clerk, the stranger who holds the lift. What we may not do is treat a person as only an instrument, an item in our machinery, with no standing of their own to be addressed. And on this account that is exactly what the scoring system does, by its very structure: it processes you rather than addressing you. The wrong is not in the verdict. The wrong is in the kind of regard: that you were processed rather than addressed, sorted rather than faced. And here is why this account cannot be answered the way the first one can. It does not improve when the machine becomes more accurate. A more accurate sorting is still a sorting. The wrong is relational. It lies in how you were treated, not in what was concluded — so improving the conclusion leaves it exactly where it was.
And — this is the part aimed directly at the first account — it does not improve when you add a perfect ability to contest either. Suppose you build the flawless automated appeal from a moment ago: a real channel, a review with real authority, your account fully credited, the whole of it run by software with no human involved. The first account says: legitimacy delivered, you have your contest. The second account says: you have been processed by one machine and then processed by a second machine that calls itself a review, and at no point has one person sat across from you and had to answer to you for what was done. The process gave you a fair contest against a thing. It did not give you what you were actually short of, which was a person. On this account, what you are owed is a human decision — and hear this carefully, because it is where the account is at once strongest and weakest — not because a human decides more accurately. The whole point is that a human might decide worse. You are owed a human because only a person can stand in the relation a person is owed: can be addressed, can be answerable to you, can be the one who has to look at you and give an account. A system cannot stand in that relation. It is the wrong kind of thing to be answerable, in the way an object is the wrong kind of thing to make a promise.
So now the two accounts stand squarely against each other, and notice how each exposes what the other lacks, because neither is left intact. The second account says to the first: your perfected process is a fair contest with no one in it — form without substance. And the first account says straight back: and your perfected person is someone answerable to you who can change nothing — substance with no remedy. Consider the sympathetic official again. He faces you. He hears your whole story. He is genuinely and fully answerable to you as a person — and the rules give him nothing he can do, and you lose your home anyway. The second account says you were not wronged: a person addressed you. But were you not wronged? You wanted to keep your home. You received sympathy. A power that can regard you with real sympathy and still act against you without recourse is the kind master again — and the first account is right that this is domination softened by kindness, not freedom. So the process is hollow without a person, and the person is hollow without the power to act. Each is the other's missing half. And yet they are not two halves of one thing you can simply add together, because they disagree about what the thing is. One says the wrong is a missing process. The other says the wrong is a missing kind of regard. You cannot average a process and a kind of regard. There is no midpoint between them.
Which rules out, before you reach for it, the comfortable exit. Surely the answer is some of each — a good appeals process and a human in the loop. No. That sounds like wisdom and it is evasion, because it assumes the two accounts point at the same missing thing and you simply need more of it. They do not. They are two different diagnoses of what went wrong, held by serious people who each think the other is aimed at the wrong target. Some of each does not resolve that disagreement. It only declines to face it.
And — the warning, on schedule — the position from last week has been present the whole time, and the second account is the one most exposed to it. Because I am owed a person who has to face me is the easiest claim tonight to read as a want rather than a wrong. The ability to contest you can at least describe as a working mechanism. But I needed there to be someone there — the people from last week will say, quietly, that this is the clearest case of the wish for a face to appeal to. Of everything we have named tonight, the person who must answer is the one that most requires you to ask, honestly, whether you have found a wrong the world owes you, or named, very precisely, a comfort.
I have been running this on the case I chose — the loan refusal, the frozen account. So take one I did not choose, and run both accounts through it yourself, because a distinction like this only counts when you apply it to a case I never chose.
Here is one. You are trying to bring someone into the country — a partner, say, from abroad — and the application goes in, and it is scored. There are risk models now, in many places, that rate an application before any person reads it: this nationality, this age gap, this pattern of visits, this bank balance, flagged as the kind of application that tends to go wrong. Yours comes back refused, or sent to the bottom of a queue that moves in years, and the caseworker you eventually reach is sympathetic and entirely powerless, and tells you, more or less, that the system rated the application high-risk, that there is a process you can follow, and that, if she is honest, she would not hold out much hope.
Now do not ask whether it was accurate. Grant that it was — grant that the model really has seen a thousand applications like yours go wrong. Ask the two questions instead. First: do you have a way to challenge it? Is there a real channel here — someone you can take your case to who can actually overturn it, and who will credit your account when you give it, rather than hear your explanation as exactly what a high-risk applicant would say? You already know how that goes. And second: even if they built you that channel, even if the appeal were real and your word were believed — is that what is wronging you here? Or is the wrong that the life you are trying to live, the person you are trying to be with, was rated as a category and refused as a category, and at no point did one human being have to look at the two of you and give an account of keeping you apart? Ask both. And notice — this is the whole of tonight — that they do not arrive at the same place, and that you cannot honestly say which of the two is the actual wrong. Perhaps it is the missing channel. Perhaps it is the missing person. Perhaps — and do not look away from this — it is neither; perhaps it is the ordinary devastation of wanting something and being refused, and you are looking for a wrong to attach it to. You cannot tell from the feeling alone. Nor can I.
You came in with one leftover and no name for it. You leave with two names — and with the discovery that they are not two words for one thing. The single word you reached for last week, answerable, has come apart into two: a way to challenge the decision, and a person who has to face you. We took each to the people who have thought about it longest, and found that each, once perfected, exposes what the other lacks — the process is a fair contest with no one in it; the person is someone answerable who can change nothing — and that you cannot split the difference between them, because they do not even agree on what went wrong. That is what you have that you did not have an hour ago. Not the answer to what you are owed. A precise, two-part account of what you would have to be owed, if you are owed anything at all.
And there is the difficulty I set out at the start and promised to return to. We spent the hour as if the leftover were a real wrong, working out its two forms. We have not, anywhere, shown that it is one. The position from last week has not moved. It has watched us give the leftover two careful names, and it is unconvinced, because a name is not a proof. And it would point out, fairly, that the second name — the person who must answer, the one that probably moved you most — is also the one that most resembles simply wanting a face to argue with. So you do not leave able to say it turns out the machine owes me a way to contest it, or it turns out I was owed a human. You leave knowing exactly what the two claims are, exactly where they conflict, and exactly how much is still unsettled — which is all of it.
This is the fifth of eight, and we have done something specific tonight. We have worked out what answerability would have to be, on the assumption that it is owed. Next time we stop granting the easier half of that assumption. Because all night you have been leaning on a belief you have not examined — that whatever the machine cannot give you, a person could; that the person who must answer is available, if only we put a human back into the decision. Next time we take the standard you have just built, both parts of it, and turn it on the humans you actually trust — the judge, the officer, the manager with the sympathetic face — and ask whether they pass it. You have spent three weeks learning not to let the machine off easily. Next week you find out whether you have been letting something else off easily the whole time.
Until then, carry the two questions. The next time something automated decides about you — a flag, a score, a closed account, a refusal — ask them in order. Is what I am missing a way to challenge it, a way to push back that would actually count? Or is it a person, someone who has to face me and answer? See whether you can tell them apart in your own case. See whether you can tell whether either is a wrong, or whether you are naming a want. You will not settle it. But you will know, now, precisely what you are failing to settle — and that is a different thing from the vague sense of grievance you arrived with three weeks ago.
Thanks for listening. I'll see you next time.