AI Governance · Episode 6 · 28-min read

Would a human pass?

Transcript · audio coming

You have been on hold for thirty-five minutes. The menu sent you round in a circle twice. The music has looped so many times you could hum it in your sleep. And then the music stops, and there is a small human cough, and a voice — a real one, tired and a little flat, but unmistakably a person — says hello and asks how they can help.

And you feel relief. After the form that went nowhere, and the email with no reply, and the message that said your case has been reviewed and the decision stands, here at last is a person. You could almost laugh. Finally, you think. Now we can sort this out.

Hold on to that relief, because that relief is the whole of tonight. Not the machine this time — the relief you feel when the machine hands you off to a human. That is the thing we are going to examine.

Because you know how the call ends. It has ended this way for you before. The person is kind. The person is sorry. The person listens to your whole story, makes a sympathetic sound in the right places, and then explains that the system made the decision, that they cannot override it, and that the most they can do is record that you called. You hang up. You are exactly where you were thirty-five minutes ago, except that now you have also lost thirty-five minutes. The person you were so relieved to reach turned out to be the visible front of the same decision that would not change.

This shape is familiar by now. Five weeks of it. A decision fell on your life, and it would not reopen, and you went looking for exactly what was wrong with that. Last week we made real progress. We found that the thing you were missing has two possible shapes, gave each a name, and held them apart.

This is Philosophy for Us — philosophy for everyone, no degree required. This is the sixth of eight episodes on what it is to be governed by a machine. Last week we took the thing you were complaining about — the thing that survives even a perfect machine — and found that it divides in two. One part we called a handle: a real way to push back, to put your side to someone who can actually overturn the decision. The other we called a someone: not a channel but a person, somebody who has to face you and answer to you, this particular person with this particular life. We took each to full strength. And we found that neither one settles the matter on its own — a handle with no person behind it is a fair procedure with no one across the table, and a person with no power is a kind face that changes nothing. You left unable to say which of the two you were actually short of, or even — and we left this open on purpose — whether you were short of anything you are genuinely owed at all.

So you have a standard now: a two-part test for whether a power that decides about you is one you can accept. Does it give you a handle? Is there a someone? You built that test to hold up against the machine. And all through this series, while you built it, you have been holding something in reserve — a fallback, the unspoken second half of every complaint you have made about the loan screen and the frozen account and the deactivated app. Under every the machine shouldn't get to do this to me there has been a quiet a person wouldn't. If only a human had looked at it. If only there were someone there. All this time you have been making the case against the machine, and you have kept one argument for human judgment in reserve, and you have never once examined it.

Tonight we examine it. We take the standard you just built — the handle and the someone, both parts of it — and we turn it around. We stop applying it to the machine and apply it to the humans. The judge. The loan officer. The doctor. The manager. The very people you have been quietly wishing would come back and decide these things instead. And we ask, with exactly the hardness we have used all series: do they pass? When a human decides your fate, would they give you a handle? Are they really a someone who has to answer — or only a face?

And let me give you the warning I give every week, because this one is the easiest to mistake. If we find tonight that the humans do not clearly pass — and we will — that is not me telling you the machine wins. Losing your confidence in the human is not the same as gaining confidence in the machine. We are not going to spend tonight declaring the machine the winner. We are going to spend it removing the one comfortable position you had left — at least a person would answer me — and asking whether you ever had the right to stand there. You came into this series certain the machine was the villain. Tonight you find out whether you have been letting something else off lightly the whole time.

Start with the relief, because the relief is where the assumption hides.

When the human came on the line and you felt that relief, what exactly were you relieved about? Say it slowly. You were relieved because you thought: now there is someone here who can hear me, weigh what I say, and answer to me for what happens next. You did not put it in words. You felt it all at once, as a kind of safety — the safety of being in front of a person rather than a wall. But take the safety apart and that is what is inside it. The arrival of a human felt like the arrival of answerability. A someone appeared, and you treated the someone as proof that the rest had appeared too.

Now look at what you actually got. You got a face. You got a voice that was sorry. And that is genuinely something — it is not nothing to be spoken to like a person instead of processed like a form. But measure the call against the standard you built last week. Did the face give you a handle? Could you push, have the push count, and reach someone who could overturn the decision? No. The voice told you plainly it could not override the system. Did the face give you a someone in the full sense — not just a person present, but a person who had to answer to you, who owed you an account and could be held to it? Also no. They recorded that you called. They were answerable for nothing. They were a someone in the thin sense — a human was there — and not a someone in the sense the standard actually demands, which is a person who has to answer.

So watch what the relief did. It took the thin thing — a person is present — and quietly treated it as the thick thing — therefore I will be answered. It ran the two together so fast you never saw the join. And that join is the assumption we have never tested. Every time in this series you have reached for a human would be different, you have leaned on it. Not a human would be more accurate — we are long past accuracy, and in any case you do not really believe humans are more accurate; we spent a whole week on how the machine reduced the bias the humans carried. What you believe is subtler and more stubborn: that a human would be more answerable. That a person, simply by being a person, is the kind of thing that has to account to you in a way the machine cannot.

And here is what should stop you. That belief is doing enormous work in everything you feel about these systems — it is the foundation under your whole sense that the machine is the special problem — and you have never once tested whether it holds. You inferred it from the relief. You took the warmth of being faced as evidence of the substance of being answered. But those two come apart, exactly as the handle and the someone came apart last week. A person can face you warmly and owe you nothing. A person can be fully, visibly present and completely beyond your reach. The kindly official from last week — the one who hears your whole story and then tells you the rules give you nothing — remember who that was. We used him last week as a hypothetical. He is not a hypothetical. He is the human on the phone. He is most of the humans.

If you have been with this show a while, you recognise this, because you have done it before, on yourself, on other questions. You caught the standpoint hiding under I earned it — the theory of desert you never actually chose. You caught the one hiding under I'm just being open-minded. This is the same thing you did then, turned on the most comfortable belief you have left. There is a standpoint hiding under a human would answer me — a whole theory, that human address is answerable by its nature — and you are about to find that it is not a fact you checked. It is a hope you have been carrying, dressed as a fact, because the alternative is to have nowhere comfortable to retreat to.

So let us check it. Let us do the thing you have never made yourself do. Take three humans you genuinely trust to decide things — not cartoon villains, but real authorities you have accepted your whole life — and run them through your own two-part standard, the handle and the someone, without mercy. The same hardness you spent on the machine. And let us see what passes.

Take the judge first, because the judge is the human authority we trust most to be answerable. A judge gives reasons. A judge sits in the open, in a gown, and explains the decision in a published judgment you can read. If anyone passes the someone test, surely it is the figure whose entire job is to account for the verdict in writing.

So test it. There is something that happens in the mind of anyone who decides — honest judges will admit it, and you can catch it in your own decisions if you watch yourself closely — where the verdict arrives first, quickly, as a kind of read on the case, and the reasons are assembled afterwards to support it. The conclusion comes first, and then the reasoning is built to reach it. The published judgment is real, and it is careful, and it may also be, in part, an account written backwards from a verdict that came from somewhere the account never mentions — the mood that morning, the order of the cases, what the defendant's face did, how like or unlike the judge the defendant seemed. You get to read the reasons. You do not get to question the thing that actually produced them, because that thing does not appear in the reasons, and quite possibly the judge cannot see it either. So: a handle? A narrow, slow, expensive one — you can appeal, to another human who decides much the same way, and the appeal tests the reasons as written, not the intuition that produced them. A someone who answers? There is a face, and a document. There is not a person who can be made to give you the true account of why, because the true account is partly hidden even from them. The judge is more answerable than almost any human authority there is. And the judge is still not as answerable as you assumed when you wished for one.

Now the loan officer — the human you have been wishing back into the loan screen all series. Picture the real one, the one the machine replaced. A person at a desk, with your file and a quota and a long day behind them, and a feeling about you that forms in the first thirty seconds — from your name, your accent, your clothes, from whether you remind them of someone who repaid them or someone who did not. We know this person was there. And you do not need a study to know it — it runs through the whole of the last century — that the human at the desk decided differently on the same file depending on what the applicant looked like and sounded like. That is the human you are nostalgic for. Run the standard on them. A handle? You could ask why, and get a sentence — you didn't meet our criteria — as sealed as anything a machine ever gave you, except that now there is no log, no record, no audit, nothing written down at all: just a person's word about a feeling they may not even know they had. A someone who answers? They have a face. They answer for nothing. And here is the part that should genuinely unsettle you, because it is the opposite of what you came in believing: the bias in that human was worse, and harder to catch, than the bias in the machine. The machine — the audited one from last week — at least leaves a trail: a logged decision, a documented basis, a process someone can inspect and challenge and find the skew in. The loan officer's hunch leaves nothing. You cannot audit a hunch. You cannot appeal a feeling. The human you wanted back was, by your own standard, less answerable than the machine, not more — better at appearing answerable, worse at being it.

And the manager — the one who passed you over, who said you were not quite ready, whose discretion set the shape of years of your life. Discretion is a flattering word for it. Take the word away and discretion is simply a decision with the reasons kept private and the bias behind a face. Could you make them account for it? You got a meeting, perhaps, and some warm coaching language, and nothing you could actually challenge, and no one above them who would overturn a judgment call — because the whole point of calling it judgment is to place it beyond challenge. A handle? A meeting is not a handle. A someone who answers? A face that says I just had a sense is not a someone who has to answer. It is the kindly official again. It is always, when you look closely, the kindly official again.

So look at what we have, because it is not what you expected. You ran your own test — the handle, the someone — on the three human authorities you trust most, and they came back weak on both parts; and on at least one count the most trusted of them came back worse than the audited machine: less logged, less inspectable, more sealed, the reasons more thoroughly hidden even from the person giving them. The face was real every time. The answerability was thin every time. The someone you were so relieved to reach on the phone turns out, held to your own standard, to be mostly the feeling of being faced, sitting on top of a power as closed as the machine's — and sometimes more closed.

Now, I have been doing this carefully, and I want to be honest about where it leaves us — before someone steps forward to put it more sharply than I have. Because someone is about to.

The people who built the machine have had a reply waiting for six weeks, and now it arrives — because you have just made their case for them.

Here is what they say, and I will put it in their voice, at full strength, because it deserves that. You ran your standard on the humans. Good. Now finish the thought instead of flinching from it. Your standard — a real handle, a someone who must answer — was just failed by the judge, the loan officer, and the manager. It is failed, if you are honest, by the head teacher, the immigration officer, the consultant, the panel, the board, the committee — by almost every human power that has ever decided anything that mattered in your life. So you have two options, and there is no third. Either you accept the conclusion in full: you say yes, my standard is right, and almost none of it ever cleared the bar; the whole structure of human authority I have lived under my entire life was illegitimate, and I am only now seeing it. That is a real position. People have held it. But be clear about what it costs, and be sure you mean it. Or you admit the thing we have been telling you since the second week: the machine is not the special villain. It never was. It is the newest authority to decide about you without fully answering for it, the latest in a succession that goes back as far as authority itself; and the only reason it looks uniquely sinister to you is that it is new, and has no face to make you feel better while it does to you exactly what the loan officer always did.

And they are not finished, and the next part is aimed at your own ground. You have spent this series quietly mourning the human we replaced. Then mourn accurately. The human at the desk decided worse — more arbitrarily, more swayed by your name and your skin and the hour of the day — and decided it with no log, no record, nothing you could ever inspect. The people who paid for that arbitrariness were not the comfortable. They were the ones already easiest to dismiss: the thin file, the wrong accent, the unfamiliar name. The machine — audited, documented, inspectable — took some of that caprice away from exactly those people. So if you are going to be hard on a power that decides about you without fully answering, be even-handed, and notice that on that very test the machine you resent may actually come out better than the human you miss. Your wish for a person to answer to is a real wish. We are only asking you to see whose suffering you would be trading for the comfort of it.

Take the measure of how strong that is, because I am not going to pretend I can refute it, and you should not let me pretend it. It is not a trick. Every step of it is something we have either established together or have good reason to believe. The humans really do fail your test. The machine really is, on at least one part of the standard, more inspectable than the hunch it replaced. The cost of accepting the first option really is enormous. That is a serious position, held by serious people, and it leaves you genuinely stuck, and the difficulty is real.

But watch the one step it takes that it has not earned, because the whole hour turns on catching it. Listen to where they want you to end up. They want you to take the second option — so the machine is not the special villain — and then keep going, past the point where that sentence stops, all the way to a different sentence they never quite said aloud: so the machine is fine. So use it. Case closed. And that does not follow. Look hard at the second option and see exactly how far it reaches. It reaches to: the machine is not uniquely bad. It does not reach one inch further, to: the machine is good, or legitimate, or owed your acceptance. Finding out that the human authorities also fail your standard does not tell you the standard was satisfied by anyone. It tells you it was satisfied by no one. You have not discovered that the machine passes. You have discovered that nothing you have ever stood under passed — which is a more disturbing thing, and a more honest one, and it leaves the machine exactly as unjudged as it was an hour ago.

So before you take either option, there is one more thing to look at — the thing you came here holding without knowing you held it.

Do this one yourself. I am not going to do it for you, because it only works if it is your own sentence you catch.

Take the thing you have been saying all series, under your breath — the fallback. Say it in full: whatever the machine cannot give me, a human could. Or the warmer version, the one from the phone: at least, with a person, there would be someone there. Now do to it what you have done before in this show to beliefs you were sure of. Do not ask whether the sentence is true. Ask what it rests on. What has to be the case about human beings for that sentence to be the comfort you have been treating it as?

It rests on this: that a human, simply by being human, is answerable to you in a way a system cannot be. Not present — answerable. Able to be made to give the true account, reachable, and movable. That is the claim underneath the comfort. And now the only question that matters: where did you get it? Did you ever test it? Or did you feel it — in the relief on the phone, in the warmth of a face turned toward you, in being spoken to instead of processed? Be honest, because you already know. You did not test it. You inferred the whole substance of being answered from the warmth of being faced, and you have stood on that inference, as if on solid ground, under every complaint you have made for six weeks. The foundation under the machine is the special problem is a belief about human answerability that you have never once put weight on to see whether it holds. That is the standpoint. It was hiding exactly where standpoints always hide — in the thing so obvious you never thought to check it.

And here is what catching it costs you, and I am not going to soften it, because this whole arc has refused to soften things. It costs you the comfortable retreat. From tonight, you cannot say at least a human would answer me and have it hold you up, because you will hear the question underneath it — would they, though? have you ever made one answer? — and you will know the honest reply is that you mostly never have; and that the times you tried, you got a meeting, and a sentence, and a face, and a note recording that you called. The fallback is gone. The argument you kept in reserve was finally examined, and it could not stand.

But now hold steady, because this is the exact place the whole hour can go wrong — and it will go wrong if you let the relief of certainty return by another route. Losing the human fallback is not the machine winning. That is the one sentence tonight must not lose. You have not discovered that the machine is legitimate, or fair, or owed your acceptance. You have discovered that the position you were going to argue from, to condemn it, does not hold up either. Those are completely different findings. The first would settle something. The second settles nothing — it only removes a cheap way you had of not having to think. The real question, the one this whole series is built on, is exactly where it was an hour ago: is the thing you are missing — the handle, the someone, whatever it is — a wrong that some power owes you an answer for, or is it, as the machine's defenders have said since week four, the wish for control in better clothes? They are still saying it. And tonight they gained a new and troubling piece of evidence, because if every authority you have ever known fails your test, the simplest explanation is not that all of human history was illegitimate. It is that your test was asking for something no one was ever owed. You cannot answer them tonight. You could not answer them last week. Turning the standard on the humans did not resolve whether the missing thing is a real wrong or a mere wish. It made the question harder.

And one more thing it did not do — so you do not evade it. Running your two parts on the human did not even tell you which part was the one that mattered. The judge was thin on the handle and thin on the someone. The loan officer failed both. You did not learn, from any of this, that the missing thing was really the handle all along, or really the someone all along. The choice you left with last week is still open. You have simply learned that the humans have not settled it for you; they face the same uncertainty you do.

I have been running this on the judge and the loan officer, the examples I chose. So take one I did not choose, and run it yourself, on a human you trust about as much as you trust anyone with power over your life. Because the exercise only counts when you do it on ground I never touched.

Here it is. You are in a consulting room — you, or someone you love. Across the desk is a doctor: senior, experienced, kind, the real thing, the someone you have wanted on the end of the phone all series. And the doctor tells you, gently, that you do not meet the threshold. For the scan. For the referral. For the drug that costs more than you could pay. For the place on the list. It is, they say, a clinical judgment. They have weighed it, with all their training, and the answer is no.

Now do not ask whether they got it right — you cannot judge that, and it was never our question. Run your two parts on them, the same two, the same hardness. First, the handle. Is there a real channel here to have this reconsidered — someone above them who can actually overturn it, who will hear your account and credit it rather than close ranks? Or is clinical judgment exactly the kind of sealed reason we met in the judge — a verdict that arrived before the explanation and cannot be examined, with a complaints process that goes in a circle, and a second opinion you cannot get, or cannot afford, or are told would say the same? You already sense which way that goes. And second, the someone. They have a face — the most sympathetic and competent face we have met all night. But do they have to answer to you for the decision? Can you make them sit with you and give the true account of why you and not the next patient, why this year's threshold and not last year's? Or is in my clinical judgment the most respectable version we have yet seen of the loan officer's I just had a sense — the same closed decision, with a much better manner?

Run it honestly and you will find the consultant comes back weak on both parts — and on the handle, quite possibly more sealed than a triage algorithm would be, because at least the algorithm's threshold is written down somewhere a person could in principle inspect, while the clinician's is in a head you cannot open. By your own standard, the trusted human is not clearly passing.

And now the difficult point, the one that should keep you awake. Knowing all that — having just watched the consultant fail your test — do you want them replaced by a model? Do you want the scan rationed by a clean, audited, inspectable algorithm instead of by this flawed, sealed, sometimes-arbitrary human? Be honest. Most of you recoil from it hard. Something in you says no, not this, not this decision; I want the person here, even though the person just failed every test I built. Look hard at that reaction, because it is the most interesting thing in the room, and I cannot tell you what it is, and neither can you — and that is the point. Perhaps the reaction is only the old wish for a face, the thing the machine's defenders have been naming since week four: sentiment, with no entitlement under it, which you should override the way you would override any comfort that costs other people their fair share of the scanner. Or perhaps the reaction is tracking something your two-part standard never managed to catch — some third thing you are owed by a person, that is neither the handle nor the answering, that we have not named and perhaps cannot. You cannot tell which from inside the reaction. Nor can I. We have run out of test, and we are left holding a response we can neither certify as a wrong nor dismiss as a wish.

You came in with the machine as the villain and a human held quietly in reserve as the answer — the someone who would finally answer you. You leave with the reserve spent. You held your own standard up to the humans you trust most, and they came back weak; and the most trusted of them came back, on at least one count, worse than the machine you resent — better at seeming answerable, no better at being it. The fallback is gone. And — hold on to this — the machine did not win when the fallback fell. It is exactly where it was. You have lost a cheap reason to condemn it and gained no reason at all to accept it. The machine's defenders have a new piece of evidence, and it is a real one: if every authority you have ever stood under fails this test, perhaps the test was asking for something no one was ever owed. They may be right. They were not refuted tonight. They are never refuted by anything that only takes a comfort away.

This is the sixth of eight, and we did something specific. We took the standard you built to judge the machine and turned it on the humans, and found that the humans do not pass it cleanly either — and that this tells you almost nothing about the machine, and almost everything about how little you had ever tested the ground you were standing on.

There is one thing we have carefully not asked yet, and it comes next. All night we have looked at the person who decides — and found them sealed, faced, hard to make answer. But there is a harder question waiting, and it is not who decides. It is who answers when the decision goes wrong. When the score ruins someone, and it turns out to have been built on bad data — who, in the long chain from the firm that made it to the clerk who ran it, actually has to stand up and own it? All night we have asked whether a human would answer better. Next week we ask the harder thing underneath: when real damage has been done, and a decision has passed through many hands, whether anyone in that chain has to answer for it at all.

Until then, carry the hard thing you found tonight. The next time a person decides something that lands on your life — a doctor, a manager, a judge, a clerk with a kind face — do not reach, as you always have, for the relief of at least there's a human here. Run the two questions on the human instead. Is there a handle — a real way to make them reconsider, through someone who can overturn it? Is there a someone — not a face, but a person who has to account to you and can be held to it? Notice how seldom the human you trust actually passes. And then watch yourself still not want the machine instead — and ask what that refusal is made of, because you do not know yet, and not knowing is the truest thing you have.

Thanks for listening. I'll see you next time.

← Back to AI Governance Back to top ↑