AI Governance · Episode 7 · 30-min read

When it goes wrong, who answers?

Transcript · audio coming

The email arrives on an ordinary morning. Your payment account — the account your small business runs through, the one your customers pay into, and the one that in turn pays your suppliers, your rent, and, last of all, you — has been suspended. Effective immediately, pending review, for suspected fraudulent activity. There is a case number. There is no name.

You have done nothing wrong. That changes nothing, because the money is already frozen — your money, and money your customers paid you in good faith — and the suppliers still expect payment, the rent is still due, and every day the account stays shut is a day your business loses value. So you do the one thing available to you. You go looking for the person who did this, to make them reverse it and, beneath that, to make them answer for it — to stand in front of you and take responsibility.

And here is what you find, and it is the whole of tonight's episode. There is no such person. The support line runs the system but cannot see into it and cannot reverse it; the agent can only record that you called. The system itself was built by a company three steps removed, and if you reached them they would tell you, correctly, that they only supplied the software; how it was configured and what it was set to flag was the client's affair. The client — your payment provider — would tell you, correctly, that they rely on the vendor's model and follow the regulator's anti-fraud rules exactly. The regulator would confirm, correctly, that the provider followed an approved process and filed every report on time. Each party followed something. Each, by its own account, has done nothing wrong. The harm fell on one identifiable person, you. Responsibility for it falls on no one.

Notice how strange that is — not how unfair; we will come to fair — how strange. Something was done. It fell on one specific person. It was done by no one. Every party that passed it along had acted correctly by the time it left their hands. You are left with a wrong that nobody committed.

This is Philosophy for Us — philosophy for everyone, no degree required. This is the seventh of eight episodes on what it is to be governed by a machine. Tonight the question changes.

For six weeks we have stood at a single moment: the moment a decision is made. We have asked whether you could follow its reasons, whether you could challenge it, and whether there was anyone behind it. Last week we took the standard we had built and applied it to the humans you trust — the judge, the loan officer, the doctor — and found that they fail it too, sometimes more badly than the audited machine. You ended last week with your human alternative spent and the machine no more legitimate for it. Tonight we move one step forward in time, past the decision to what comes after it. The decision has been made, it was wrong, and the damage is done — and now you go looking for who has to answer for it. Not who decides. Who answers when it has gone wrong.

There is a second matter here, quieter than the sequence of parties, and it concerns a single word. It is the word everyone reaches for as the remedy for exactly this: the word in every policy paper and every press release, the word offered as the resolution of the whole problem. Accountability. We are told these systems are becoming accountable, that they are audited now, that this is the fix. By the end of tonight that word should no longer be able to reassure you — not because I have a better word to offer, I do not, but so that you can hear what it quietly stopped meaning, and stopped meaning at precisely the moment it began to be sold back to you as the answer.

Let me trace the chain slowly, because the slowness is the point. When you go looking for who must answer, you do not meet a single obstacle you could confront and name. You meet a line of people, and each one, when you reach them, passes you on to the next.

Start with the person in front of you: the operator, the clerk, the support agent — the human who ran the system, or who works for the people who ran it. Ask them to answer for the decision, and they tell you, truthfully, that they did not make it. They entered your details, or read the outcome off a screen; the model produced the flag, and they are required to act on it. Had they overridden it because you seemed honest, they would be the one breaking the rules. So they pass you up the line. This is not cowardice. On their own terms they are exactly right: they did not decide, and they were not permitted to un-decide.

Go up a step, to the institution that deployed the system — the payment provider, the bank, the agency, the platform. Ask them to answer, and they tell you, truthfully, that they did not build the model. They bought it, or licensed it, from a firm whose entire business is fraud detection and which does it better than they could in-house. They configured it within the vendor's parameters and ran it under the regulator's rules. They followed the guidance. On their own terms they are exactly right: they are not the authors of the thing that flagged you, only its careful users.

Go up again, to the vendor — the company, three steps back, that actually wrote the system. Ask them, and they tell you, truthfully, that they supply a tool. How it is deployed, what thresholds the client sets, which population it is pointed at — that is the client's decision, not theirs. They built to specification and tested to specification. On their own terms they are exactly right: the maker of a tool does not answer for every use it is put to.

And above all of them, the regulator — the body whose job is to make the whole arrangement accountable. Ask them to answer for what happened to you, and they tell you, truthfully, that they do not adjudicate individual cases. They check that the provider has a process, that the process meets the standard, that the reports arrive on time and the audits are clean. On their own terms they are exactly right: they oversee the system, not your file.

Now stand back and consider what you have walked through. Four answers, every one of them true, every one of them given by an honest person who is neither lying nor dodging. And at the far end of all that truth, there is nothing — no one. The harm landed on one person with a name; the answering scattered across a structure with none. You did not find a villain at the end of the chain. You found that the chain has no end: it simply keeps passing you on, and each party has done nothing wrong by the time it lets go.

People who study these systems have begun to call this agency laundering, and the term is worth pausing on, because of what laundering does. Laundering does not destroy the thing it works on. It takes something whose origin you could trace — a decision a person made, which you could follow back to them — and runs it through enough hands and enough process that by the time it reaches you it is untraceable, no one's. The decision still happened; the harm still happened. What was removed, somewhere along the way, was the person responsible at the end of it. Run a judgment through a long enough process — and I mean the whole apparatus: the software, the firms, and the rules, not just the algorithm — and you can produce a real outcome that no human being has to own. This is not a side effect you could correct with a tidier organisational chart. For many of these arrangements it is the most useful thing they do. A decision that no one has to answer for is, from the institution's point of view, an advantage. The discomfort has been designed out.

Notice that this is a different finding from last week's, and a sharper one. Last week we stood at the decision and found that the human who made it was sealed off — the judge whose reasons came after the verdict, the officer whose hunch left no record. That was about the opacity of a single decider. This is worse, because here you are not even asking the thing that decided to explain itself; you have given that up. You are asking only this: something went wrong and it wrecked my year — will one human being stand up and say, that was mine, I answer for it. And the structure is built so that the honest answer, from every person in it, is: not me. Not because they are hiding, but because there is genuinely no position in the structure from which that sentence is true.

Hold on to that — a wrong for which there is no position that must answer — because we are about to find that it is much older than the machine, and that it is concealed inside the very word everyone offers to put it right.

Take the word apart: accountable. You use it every day and have probably never looked inside it, and inside it is something small, hard, and surprising.

At its root is account, and an account, before it meant anything abstract, was a count — a reckoning, a tally of what was owed. The word comes down to us from counting, from computation: a person sitting with a ledger, working out the sum that one party owes another. To give an account of yourself was, first and most literally, to render that reckoning — to lay out the books — to someone. And that someone was not incidental. You do not render a reckoning into the air. A count of what is owed is owed to a particular party: the one with the standing to demand it, the one you are answerable to. To be accountable, close to its root, was to be in that position — there is someone who can call you, make you stand before them, and require the account of what you did, and you must comply. The relation is not a decoration on the word; the relation is most of what the word was for. Remove the person to whom the account is owed, and you do not have a weaker form of accountability. You have a tally that no one can call in. You have bookkeeping.

Keep one thing in view from the start, because there is a direction this can slide in, and I do not want it to slide there yet. I am about to show you that the word moved, and that much is history, which I will stand on. But whether what moved away from you was something you were owed, or only something you happen to miss, is a different and heavier question, and the history I am about to set out does not answer it. Watch the word change, but do not let me decide for you whether the change was a theft.

Now follow the word upward. From that root it grows into something we already have a name for, from a few weeks ago: answerability. When we stepped away from the technology entirely, in week three, and asked what makes any power's say-so binding on you, this was underneath it — a power is answerable when those it governs can call it to give an account of itself, before them, in terms they can grasp. The judge answers to the public in open court. The minister answers to parliament. The old idea is that power renders its reckoning downward, to the governed, and the governed have the standing to demand it. The account runs from the one who holds the power to the one it was used on.

Now consider where the word has actually gone. Consider what accountable means when a press release uses it. It means audited. It means there is a process, and the process is checked, on a schedule, by a body, against a standard, and a report is filed. Every word of that describes real work, and I am not disparaging it. But note the direction in which the reckoning now runs: it runs upward. The account is rendered up — to a regulator, to a board, to an oversight body — in a form you will never see and could not read if you did. The one party reliably not present when the modern account is rendered is you, the person it is rendered about. The reckoning still happens; it has simply stopped being rendered to you. The party with the standing to call it in used to be, at least in principle, the governed. Now it is another office.

Let me be exact about what I have and have not shown, because this is the point at which it would be easy to cheat. I have shown that the word moved. That is a fact about the word, and I will stand on it: accountable once named a relation — answerable, to someone, who could demand it — and in common modern use it names a property of a system, audited, checked, certified. That much is simply the history. What I have not shown — and I will not pretend I have — is that the old meaning was the right one and the new meaning a fraud. That is a different and heavier claim, and it is exactly the claim this series refuses to settle for you. The word thinned; whether thinned is even the right description — whether something was lost that you were owed, or whether the word simply became realistic about how power works at scale — is the live question, and I will not steal it from you by smuggling a verdict into the history.

But I will say why the history matters, because it does. If you have followed this show for a while, you have seen the same operation performed before, on a different word. We did it to merit. We took I earned it apart and found that the word deserve was doing quiet work — converting luck into virtue, taking the raw fact of who happened to receive what and presenting it as a property of the person, something they are, so that the question of who was owed what could quietly close. This is the same shape, and it is one of the oldest operations philosophy knows: you take a word that names a relation — who owes what to whom — and watch it harden, over time, into a property of a thing, a quality it simply has, so that the relation drops out of view and stops being a question anyone asks. Accountable has done exactly that. It has gone from answerable to you to certified, elsewhere. And the reason that is worth noticing is not that the new meaning is wicked. It is that once you have heard the older meaning underneath it, you can no longer let the newer one close the conversation. When someone tells you the system is accountable now — it is audited — a question opens under the sentence that you cannot un-hear: accountable to whom? Audited for whom? Who is the party with the standing to call this in — and is it, has it ever been, you?

So set the history back against the chain we traced at the start, because now the two fit together, and the fit is the heart of the matter.

You went looking for who must answer for the wrong, and you found a sequence of parties each in the clear, with an audit at the top. And you were told — this is the point — that the audit is the answer; that the system's being audited is what its being accountable means. But you have now heard what that word once carried, and so you can hear what that reassurance quietly leaves out, in a way you could not an hour ago. The audit answers a regulator. It renders the reckoning upward. The one party it does not answer is the one with your name in the inbox. They did not fail to make the system accountable to you. They built something called accountability that was never directed at you in the first place — that runs the account in the other direction, away from you, to an office, on a schedule. You were not left out of the loop by accident. In the modern sense of the word, you were never in it.

There is a view in political philosophy that the whole series has been moving towards, and this is the point where it belongs; it is the most serious challenge we have reached. We met it briefly in week three, and I said we would return to it. It says: stop asking whether the decision was accurate, and stop even asking whether you could contest this particular output; those are questions about the decision. The real question was never about any single decision. It is about the standing of the thing that gets to decide at all. On this view — its sharpest modern statement is Rainer Forst's, that a power is legitimate only when it can justify itself to you in terms you could reasonably accept, though the root of the idea is much older — authority is legitimate only if the power to wield it was authorized by the people it governs, and remains answerable to them. Not answerable about this output. Answerable as such. The source of the authority, and its continuing answerability to the governed — that, and not the quality of any individual decision, is what makes its verdicts binding on you rather than merely things that happen to you.

Consider what that view does to your frozen account. It says the deepest thing wrong here was never the false fraud flag; the false flag is only where you noticed. The deeper thing is that a power acquired the standing to decide your livelihood — to reach into your account and close it — without ever having been authorized by you, and with no live channel through which it must answer to you for how it uses that standing. You never agreed to be scored by the fraud model. No one asked you. It was simply switched on over your life, as most of the powers that sort you were simply switched on, and the word that was supposed to keep it answerable to you turned out, when we opened it, to answer somewhere else. On this reading the missing answer at the end of the chain is not a fault in an otherwise legitimate arrangement; it is the truth of the arrangement, finally visible.

And here is the part of the view you must not miss, because it is what keeps tonight honest and keeps the machine from being cast as the villain. On this reading, the machine did not create the gap. The machine entered a gap that was already present and widening. Power had been thinning its answerability into audit for generations before anyone trained a model — this is what large bureaucracies do, public and private alike. They grow until the party that decides about you is a department, and a department has no face; it renders its account upward, to a minister or a board, never down to you. The algorithm did not invent faceless deciding power. It is that power's newest and most efficient tool for doing so. What the machine did was make the gap impossible to keep ignoring: it took the slow administrative draining-away of answerability and carried it out in four seconds, with a case number and no name, so plainly that you finally looked up and saw the shape of something that had been forming across your whole life. That is the most this view claims about the machine, and it is careful: not that the machine is uniquely sinister, but that it exposed a condition that was there all along.

I will say where the view runs out, because it has a far edge, and honesty means walking you to it. If legitimacy really requires that a power be authorized by you and answerable to you, in terms you could genuinely accept, then push it hard and it does not obviously stop at the machine. The tax office was never authorized by you in any sense you would honestly recognise; you were born under it. The employer's authority over your hours, the state's over your liberty — apply the same full-strength test, and much of the authority you live under every day begins to look as though it never met the standard either. Which leaves the view with a hard consequence of its own, one we will have to hold rather than dodge: perhaps its standard is so demanding that almost no real power passes it, and the machine is merely where you happened to notice that you have been governed, your whole life, by authorities you never authorized and cannot make answer. That is a real possibility, and a disorienting one, and I am not going to close it tonight. I am raising it now so that when this view attracts you — and it will; it is the most satisfying thing said all evening — you remember that it does not stop where you would want it to, and that its own logic reaches far beyond the machine.

Now pause, because that view is attractive, and the more compelling something sounds, the harder you must look at what it passed over. The people who design and defend these systems have a reply; they have heard every word of the history, and they are not persuaded by it — and you should hear exactly why, because their reply is strong.

Here is what they say. It is a fine piece of history. The word accountable used to mean answerable-to-a-person, and now it mostly means audited. Granted, every word of it. Now tell us what follows, because so far nothing does. You have shown that a word changed. You have not shown that the change was a loss. You are mourning the old meaning as though the fact that it came first makes it the true one, and that is nostalgia, not an argument. So let us actually ask the question the history is avoiding: was personal answerability — a power required to stand in front of each subject and render the account to them, face to face — ever a good idea at scale? Or was it a feature of small communities that simply does not survive contact with millions of decisions a day?

And then they make the point on your own ground, as they always do, and it holds. Picture the world in which the account is owed to you personally, in which every fraud flag, every claim, every score comes with a human being who must sit across from you and answer for it to your satisfaction. Who can afford to demand that account? The confident. The articulate. Those with time, a lawyer, a good accent, and a free afternoon to sit in the office and not back down. The person with the thin file, the second job, and the imperfect English — the very person the history claims to be defending — never gets to call that account in, and never did, in the small world you are romanticising. They got the loan officer's hunch and no record of it. What the audit gives them — the cold, upward-rendered, faceless audit you are disparaging — is the one thing personal answerability never gave them: a check on the power that does not depend on their being able to fight for themselves. The regulator answers on their behalf whether or not they can show up. So before you insist that the reckoning should run back down to the individual, ask which individual. The audit may be exactly the right form for the account, precisely because it does not wait for you to be strong enough to demand it.

Take the full weight of that, because I cannot refute it and I will not pretend to. It is not a dodge. The history is real, and the history settles nothing, and both of those are true at once. Showing that accountable once named a relation to a person tells you what the word used to do. It does not tell you that you are owed that relation now, or that a system that renders its account upward instead of down to you has wronged you, rather than simply governed you at a scale where the old closeness was never going to hold. That is the question — the same question this series keeps refusing to close — and the history did not answer it. It sharpened it. It gave you a vivid and accurate picture of what dropped out of the word, and left entirely open the thing that actually matters: whether what dropped out was something you are owed, or something the powerful could always afford to give the strong and never gave the weak, and are well rid of.

So notice that inference before you draw it, because it is the mirror image of the one we caught last week. Last week the danger was moving from humans fail the test too to therefore the machine is fine. Tonight the danger runs the other way: moving from the word that was supposed to make it answerable to you actually answers somewhere else to therefore you have been wronged, the system is illegitimate, the older meaning was the real one. Do not make that move. The honest finding is colder and more useful than either comfort. You have learned that the thing offered to you as the remedy — do not worry, it is accountable now, it is audited — does not even point in your direction; that it answers a question you were not asking, and is owed to a party that is not you. That is worth knowing; it strips away the easy reassurance. But it does not tell you that you are owed the account personally. The defenders have not been refuted. They are standing exactly where they stood in week four, saying that the remaining wish — that someone must answer to me — may be the oldest comfort there is, the wish to have a person to blame, dressed up tonight in an old etymology. They may be right. Nothing in the history of a word can prove them wrong.

So you leave this with the same two things in your hands you have carried all series, only heavier and closer together. First: the word that was supposed to settle this — accountability — turns out not to settle it, because it answers somewhere other than to you. Second: whether being answered, personally, by a power that decides about you is something you are genuinely owed, or something you merely badly want, is exactly as unsettled tonight as it was the day you sat at the loan screen. The machine is no more legitimate for any of this, and no less. You have lost another comfortable place to stand — at least there is accountability now — and gained, again, no verdict at all.

I have been running this on the frozen account, which I handed to you. So take a case I did not hand you, and work it through yourself, because it becomes yours only when you apply it to a case I never chose.

Your house floods. Not a leak — a flood, the kind that takes the floors, the wiring, the boiler, and the lower half of everything you own. You are insured. You have paid the premium every month for eleven years, precisely so that this moment would not also be the end of you. You file the claim. And it comes back denied, quickly, by a model: your policy was classified one way, your address fell into a flood zone coded another, one rule met another, and the answer was no. You are not now at a four-second loan screen. You are standing in a wet house with a denial and a year ahead of you and nothing to fall back on.

So do what this episode has taught you. Go looking for who answers — and this time trace the chain yourself, because you already know its shape. Start with the adjuster who read you the refusal, and ask them to take responsibility for it; then follow wherever they send you, and wherever the next person sends you, up through the insurer, the firm that built the model, and the regulator that approved the process. I will not narrate it for you — you saw how each of them answers in the first half of tonight, and you can hear it coming before they speak. Trace it, and at each party ask the two questions you have earned tonight. First: when they tell you the whole arrangement is accountable — audited, top to bottom — ask the question that now opens under the word. Audited to whom? Is there a single reckoning, anywhere in that chain, that is owed to you, the one standing in the water? Or does every account in it run upward, to an office, on a schedule, in a language you will never be shown? Second, and harder: is there one human being in that entire structure who has to stand in front of you and take responsibility for the thing that wrecked your year — or has the answering been dispersed, party by party, until no one in the whole structure has to give it?

Follow it to the end yourself, honestly, and see where the reckoning lands, and whether anywhere in that chain you found a party from whom the answer was owed to you. Eleven years of premiums, a flooded house, and a wrong with a precise address.

And now the question I cannot resolve for you and will not pretend to. Standing in that house — is the thing you are missing a wrong? Is there a reckoning the world genuinely owes you, owed to you by name, that this whole audited and certified arrangement has quietly arranged never to pay? You want to say yes, and after tonight you can even say why — you can name the thing the word lost, you can point to the missing answer. But the defenders are standing in the same water with you, and they say, quietly, that perhaps what you want, standing there, is simply someone to blame — someone to hold responsible for the year. And perhaps the same cold, upward-running audit that left you with no one is, at this moment, catching ten thousand wrongful denials you will never hear about, and paying ten thousand claims that a human adjuster's bad morning would have refused, for ten thousand people who could never have fought the way you are about to fight. You cannot see them from inside your own flooded house; they cannot see you. And you cannot tell, from where you stand, whether the missing answer is an injustice done to you or the necessary price of an arrangement that, in an aggregate you will never witness, wrongs fewer people than the human system it replaced. You have run out of test again. You are holding a reaction you can neither certify as a wrong nor dismiss as a mere want.

Let me close with what changed tonight, and keep faith with how I have closed every week — by being honest that what changed is something taken apart, not something handed over.

You came in tonight with one comfortable thought still standing: that whatever else is wrong with these systems, at least we are making them accountable now, at least they are audited. You leave unable to let that sentence end the conversation, because you have heard what is inside the word. Accountable, near its root, named answerability — being answerable to a person who could call you to render the account — and the modern version renders it the other way, upward, to an office; and you, the person the old word would have answered to, are simply not who the new one answers to at all. That comfort is gone. And — hold on to both points, as we have every week — the loss of that comfort does not hand you its opposite. You have not proven the system illegitimate, or that you are owed a personal reckoning, or that the old meaning was the true one. The people who built the machine have a real answer, the same one they have had since week four, and tonight it can draw on the etymology too: perhaps the someone-who-must-answer-to-me was always a comfort the strong could afford and the weak never received, and we are not wrong to have traded it for a colder arrangement that protects more people. They were not refuted tonight. A history of a word cannot refute them. It can only ensure that you can no longer reach for at least it is accountable and rely on it to settle the matter.

This is the seventh of eight, and we did one specific thing. We stepped past the decision to the wreckage after it, and found that when these systems go wrong the answer scatters across a sequence of parties, each in the clear, and ends at an audit that was never owed to you — and that the word we were handed to fix precisely this turns out, when you open it, to point away from you. The machine did not create that gap. It found it, already old, already widening, and showed it to you in four seconds, under a case number nobody signed.

Next week is the last one. And next week we add nothing new — no new view, no new thinker, no new case. We have set all of it out now: the bias and the accuracy, the contesting and the someone behind the decision, the human who fails the test too, and tonight the answer that scatters and the word that points the other way. Next week we take the whole of it, every piece on the table, and instead of resolving it for you, I am going to hand it back — and then refuse, deliberately, to settle it. Come back for the last one. It is the part where I give you nothing, and where we find out whether nothing, honestly, is what there has been to give all along.

Thanks for listening. I'll see you next time.

← Back to AI Governance Back to top ↑