Philosophy for Us
Part Two · The Shared World & Technology · Series 31

AI Governance

The computer said no, and there's no one to ask why.

8 episodes ↓ 24 sources ↓ Political & SocialEthics

The question

You applied for the loan online, and the answer came back in about four seconds. Declined. Not "let's talk about it," not "here's what's missing" — just declined, with a reference number and a link to a page that says your application did not meet the criteria. So you call. And the person on the phone is kind, and completely powerless, and what she tells you, more or less, is that she can see the decision but she can't see into it. The model scored you. She doesn't know which things counted, or how much, or whether the late payment from four years ago is the whole story or none of it. She can't override it. She can re-run it, but it'll say the same thing, because nothing's changed. There's a box you can tick to "request a review," and you tick it, and you already know — the way you know things at 2 a.m. — that the review is the same machine, looking at the same numbers, reaching the same place.

Now look hard at the exact shape of what's bothering you, because it's easy to name it wrong. It is not, mainly, that you think the machine got it wrong — maybe it didn't; maybe you are, by some cold measure, a worse bet than you feel like. It's something else, and it's harder to say. It's that a decision got made about you — about your house, your year, your sense of whether the system is for people like you — and there was no one standing behind it. No reason you could follow. No one you could make answer. The thing decided, and then it closed, like a door with no handle on your side. And here's the part the next half hour is going to be about: suppose I could prove to you that the machine is more accurate than the loan officer it replaced — fairer, less moody, less swayed by your accent or your name. Would that fix the thing that's wrong? Or would the door still have no handle?

What you practise

"It got me wrong. I want a person to look at it." You say some version of that when a machine turns you down. Here you test the complaint. You grant the machine everything — perfect accuracy, no bias, proper oversight — and look at what is left. Something is left: nobody had to answer to you. You work out whether that is a real wrong or only the wish to argue with someone, then put the same test to the judge, the manager, the doctor. Nothing here gets settled for you. You come out able to say what it would take to accept a decision that goes against you.

The episodes

Each episode builds on the one before, and the finale hands the question back to you sharper, not settled.

  1. 01

    The computer says no

    A loan declined in four seconds, with no reason you can follow and no one who has to answer. The first episode fixes the exact shape of the complaint — not that the machine erred, but that a decision fell on a life and no one stood behind it — and asks whether proving the machine accurate would settle it.

    ▶ Listen Read transcript →
  2. 02

    Make it fairer

    The instinct to audit the bias out is taken completely seriously, and so is the builders' reply — the machine can be less biased than the humans it replaced, and insisting on arguing your case with a person may cost the worst-off more than it spares them. The episode ends with a suspicion that fairness was never the whole of what was wanted.

    ▶ Listen Read transcript →
  3. 03

    What makes any authority legitimate?

    No machines this week. The episode returns to the oldest question about power — what makes anyone's say-so binding? — with Joseph Raz on authority, and separates three things usually blurred together — getting it right, being entitled to decide, and being a standing one could accept. Correct and binding turn out to be different questions.

    ▶ Listen Read transcript →
  4. 04

    Suppose it's right

    Everything is granted — a system more accurate than any human, audited until no trace of bias remains, governed and overseen. The episode then puts the listener back in front of the screen and asks whether the complaint is gone, and finds it still standing — now detached from accuracy, and not yet named.

    ▶ Listen Read transcript →
  5. 05

    The right to contest

    Two accounts of the missing thing. Danielle Citron and Frank Pasquale argue that automating the decision quietly abolished due process — what is owed is notice, a hearing, a channel that can actually move the result. Reuben Binns presses the deeper reading — a scoring model never meets you; it locates you as an instance of a class. Both are built seriously, and neither is declared the winner.

    ▶ Listen Read transcript →
  6. 06

    Would a human pass?

    The relief of finally reaching a human is examined and does not survive. The standard built for the machine — a real way to contest, a someone who must answer — is turned on the humans we trust, and they often fail it too. What remains is the refusal to want the machine instead, which the episode declines to explain away.

    ▶ Listen Read transcript →
  7. 07

    When it goes wrong, who answers?

    A frozen account, and a chain of parties — vendor, deployer, regulator — each of whom acted correctly by the time it left their hands, leaving a harm that fell on one person and a responsibility that falls on no one. The word accountability is traced to its root, a reckoning owed to someone who could demand it, and its algorithmic use is found to have quietly deleted that someone.

    ▶ Listen Read transcript →
  8. 08

    Could you accept it?

    The final episode adds nothing new. It assembles the seven considerations and deliberately declines to settle them; both easy answers — ban the algorithms, and if it is accurate, use it — are closed. The question handed back is what would make a decision that goes against you one you could accept, and whether the demand to be answered is a real wrong or a want.

    ▶ Listen Read transcript →

Thinkers

The thinkers and traditions this series draws on:

Cathy O'NeilJoseph RazDavid EstlundRainer ForstDanielle CitronPhilip PettitReuben BinnsImmanuel Kant

Related series

Your turn

One summer the exams are cancelled, and every student gets a grade from a computer model. It takes each teacher's ranking of the class and fits it to how the school has done in past years. Your daughter's teacher expected an A. The model gives her a C, because nobody at her school has got above a B in three years. She loses her university place. On the board's figures, the model was more accurate than the teachers. Their guesses ran high, and highest at private schools. You can appeal, but only if the board got a fact wrong. Nobody got a fact wrong.

  1. Grant all of that. Is anything still wrong? Name it without the words "inaccurate" or "unfair".
  2. The appeal only takes wrong facts. What does she need that it has no room for: a way to challenge the grade, someone who has to answer her, or a say in whether she should be graded this way at all?
  3. If her teacher had given her the C himself, would that be easier to accept? Say what he would have to give her that the model cannot — and then whether he would actually have given it.

A good answer keeps two questions apart — whether the grade was right, and whether she could accept it — and says what the second one would take. Calling the grade unfair collapses them back together, and so does pointing out that the model beat the teachers.

Sources

Written by AI under human direction. This is the reading the series was built from. checked against source means the passage was found in the cited work and a second, independent model checked the episode's use of it; supports in part means that check judged the use a little stronger than the passage alone establishes; passage located means a reader found the passage but it has not had the second check. Unmarked entries are background reading.

Where to start