Language, LLMs & Machine Understanding

The episodes

Each episode builds on the one before, and the finale hands the question back to you sharper, not settled.

1. 01

   It explained it back to me

   A language model explains your own work back to you, with the one example you needed — and the immediate reaction, that it does not actually understand any of this, turns out hard to state. Every test reached for to keep the machine out disqualifies some of your own words too, and a question comes into view that has never actually been asked — what is it for a word to mean something?

   ▶ Listen Read transcript →
2. 02

   The most elaborate parrot ever built

   The strongest case that the machine is a parrot — John Searle's Chinese room, where flawless answers issue from a sealed room with no understanding inside, and Emily Bender's octopus and "stochastic parrots" arguments, on which the fluency is projected onto pure form. The position is built at full strength by the people who hold it most seriously, and left standing.

   ▶ Listen Read transcript →
3. 03

   Know a word by the company it keeps

   Firth's dictum — you shall know a word by the company it keeps — becomes a rival theory of meaning. If a word's meaning is the pattern of the other words it runs with, the machine has exactly that, and there was never a second thing hidden behind the pattern. The two accounts of meaning are left pulling directly against each other, with no way yet to choose.

   ▶ Listen Read transcript →
4. 04

   What's the missing extra?

   Ludwig Wittgenstein's question, put to the listener's own case — point to the grasp, the understanding beneath your use of a word that is something more than everything you can already do with it. The search comes back empty-handed, and the test of continuing correctly into a case never shown turns out to be one the machine does not fail.

   ▶ Listen Read transcript →
5. 05

   What it was never built to do

   The tests that do hold. Stevan Harnad on symbol grounding, Ruth Millikan on words shaped by a history of tracking the world, and Grice on the intention to be understood — real differences between speaker and machine — set beside the view that the yes-or-no question was the wrong shape from the start. Neither side is allowed to win.

   ▶ Listen Read transcript →
6. 06

   Say what understanding is

   No new theory this time. The episode asks the listener to state, in plain words, the rule they have been applying all along — the one thing the machine lacks that makes its words noise and yours meaning — and shows that stating it means adopting one contested account of meaning, resting on two assumptions never checked.

   ▶ Listen Read transcript →
7. 07

   It runs through you — or does it?

   The rule stated last week is now applied — turned around and run over the listener's own vocabulary. Some rules place half of one's own words on the machine's side of the line; others hold and name something real. The episode does not decide which — the rule you chose decides, and nothing available says which rule is right.

   ▶ Listen Read transcript →
8. 08

   Whose words is it using, anyway?

   Two histories. The idea that meaning is a word's connection to a thing in the world turns out to be one side of a hundred-year-old dispute, with Frege and Russell in its ancestry, rather than a plain fact; and the machine's fluency turns out to be fluency in a particular corpus — someone's language, not the world's.

   ▶ Listen Read transcript →
9. 09

   Are you sure you do more?

   The final episode sets every position side by side — the parrot, the company of words, the grounding, history and intention tests, and the view that the question was malformed — and tests only one thing, the quiet certainty that you do more than the machine does. No verdict on the machine is offered; the question is handed back sharper than it arrived.

   ▶ Listen Read transcript →

Thinkers

The thinkers and traditions this series draws on:

Related series

• Sets up 05 · Mind & Machine
• Sets up 28 · Knowledge & Power
• Sets up 29 · Are You Responsible for What You Believe?

Sources

Written by AI under human direction. This is the reading the series was built from. checked against source means the passage was found in the cited work and a second, independent model checked the episode's use of it; supports in part means that check judged the use a little stronger than the passage alone establishes; passage located means a reader found the passage but it has not had the second check. Unmarked entries are background reading.

• John R. Searle — Minds, Brains, and Programs (1980) The paper behind the locked room: a person follows a rulebook to hand out perfect Chinese answers without understanding a word, and Searle argues that a computer running a program is in the same position. It also carries the two best-known replies and his answers to them. The link goes to the journal's page for the article. supports in part
• Emily M. Bender and Alexander Koller — Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data (2020) Two linguists tell the story of an octopus that taps into a telegraph cable between two islands and learns to fake one side of the conversation, until a bear attack shows it never knew what any of the words were about. Their point is that a system trained only on the shape of language cannot learn its meaning. Full paper, free to read. checked against source
• Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell — On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? (2021) The 2021 paper that gave the view its name: a language model strings together word-shapes it has seen, by the odds, with no hold on what they mean, and people reading its output supply the meaning themselves. It also argues that the text such models learn from over-represents some voices and leaves others out. Full paper, free to read. supports in part
• J. R. Firth — A Synopsis of Linguistic Theory, 1930–1955 (1957) The linguist's essay that gave us 'you shall know a word by the company it keeps', the slogan behind the idea that a word's meaning shows in the words around it. The link goes to an encyclopedia summary of that idea, not to Firth's essay, which is not free online. checked against source
• Zellig S. Harris — Distributional Structure (1954) Zellig Harris's 1954 paper set out a method: describe a word by tracking every setting it appears in. Episode 3 credits him with the method, then separates it from the later, stronger claim that meaning is only that pattern. Links to a journal record.
• Ludwig Wittgenstein — Philosophical Investigations (1953) Wittgenstein's late book argues that for many words the meaning is the way the word is used, and that understanding shows in being able to carry on correctly rather than in some private grasp behind the words. The link goes to the Stanford Encyclopedia's overview of Wittgenstein, not to the book. supports in part
• Saul A. Kripke — Wittgenstein on Rules and Private Language (1982) Kripke's reading of Wittgenstein asks what makes it true that you are following one rule rather than another that has agreed with it so far, and argues that nothing inside your head can settle it. The link goes to the publisher's page, not the book. checked against source
• Ruth Garrett Millikan — Language, Thought, and Other Biological Categories: New Foundations for Realism (1984) Millikan argues that a sign means what it was shaped by a long history of selection to track, so meaning depends on what something is for, not only on what it does now. The link goes to the publisher's page, not the book. supports in part
• Fred Dretske — Explaining Behavior: Reasons in a World of Causes (1988) Dretske's book argues that an inner state means what it was recruited to signal, a close cousin of Millikan's view that meaning comes from function. The link goes to the Stanford Encyclopedia's summary of these theories, not to the book.
• Fred Dretske — Misrepresentation (1986) Dretske's essay asks how a signal can ever be wrong about the world, and answers that a signal can misfire only because it has a job it is supposed to do. The link goes to an encyclopedia summary, not the essay.
• Stevan Harnad — The Symbol Grounding Problem (1990) Harnad argues that words defined only by other words never reach the world, like a dictionary in a language you cannot read, and that some words must be tied to things through seeing and handling them. Full paper, free to read as the author's preprint. checked against source
• Francisco J. Varela, Evan Thompson, and Eleanor Rosch — The Embodied Mind: Cognitive Science and Human Experience (1991) Argues thinking is something a living body does by coping with the world, not pictures in a head. Used here as one answer to what grounds meaning — a body's contact with the world. The link goes to the publisher's page, not the book.
• H. P. Grice — Meaning (1957) Grice's short paper argues that to mean something is to say it intending someone to respond, and intending them to recognise that very intention. The link goes to the journal archive's page for the paper. checked against source
• Luc Steels — The Symbol Grounding Problem Has Been Solved. So What's Next? (2008) Steels, who builds robots that make up and share words for what their cameras see, argues that tying symbols to the world is an engineering problem already solved in principle. The link goes to the publisher's page for the book, not the chapter.
• Melanie Mitchell and David C. Krakauer — The debate over understanding in AI's large language models (2023) A short survey of the split among AI researchers over whether large language models understand anything, which argues the field may need more than one idea of understanding. Full paper, free to read. checked against source
• Ellie Pavlick — Semantic Structure in Deep Learning (2022) A review of experiments that look inside neural language models to see what structure of meaning, if any, they have picked up. The link goes to the journal's page for the article.
• Daniel C. Dennett — The Intentional Stance (1987) Dennett argues that to have beliefs and to mean things is to be something whose behaviour is best predicted by treating it as having them, with no further hidden fact to check. The link goes to the publisher's page, not the book.
• Donald Davidson — Radical Interpretation (1973) Davidson asks how you could work out what someone means starting from nothing, and answers that meaning is whatever a careful, charitable interpreter is entitled to credit them with. The link goes to the journal's page for the paper.
• Robert Brandom — Making It Explicit: Reasoning, Representing, and Discursive Commitment (1994) Brandom argues that to mean something is to take a place in the practice of giving and asking for reasons: to be committed to what follows from your words and answerable when someone challenges them. The link goes to the publisher's page, not the book.
• Ned Block — Advertisement for a Semantics for Psychology (1986) Block defends the view that what a thought is about depends on the job it does among a system's other thoughts, a view that leaves room for machines. The link goes to the journal's page for the paper.
• Hilary Putnam — The Meaning of 'Meaning' (1975) Putnam's paper imagines a second Earth where the stuff called 'water' is a different substance, and argues that what your words mean is fixed partly by the world and by other speakers, not only by what is in your head. The link goes to the University of Minnesota's free archive of the volume the paper appeared in. checked against source
• Saul A. Kripke — Naming and Necessity (1980) Kripke argues a name reaches its bearer through a chain of speakers passing it on, back to someone who first fixed it, not through a description in the speaker's head. The link goes to the publisher's page, not the book. supports in part
• Augustine of Hippo — Confessions, Book I (ch. 8) (c. 400) Augustine's memory of learning to speak: the grown-ups named a thing and pointed at it, and he gathered which sound went with which thing. It is the picture of words as names for things that the later episodes trace and question. Full text, free to read. supports in part
• John Locke — An Essay Concerning Human Understanding, Book III (Of Words) (1690) In Book III Locke argues that words stand first for ideas in the speaker's mind, not for things in the world — which moved meaning inward. Full text, free to read. checked against source
• Gottlob Frege — On Sense and Reference (Über Sinn und Bedeutung) (1892) Frege's paper argues that a word has both a reference, the thing it picks out, and a sense, the way it presents that thing, which is why "the morning star" and "the evening star" differ in meaning though both name Venus. Full text, free to read in translation. checked against source
• Bertrand Russell — The Philosophy of Logical Atomism (1918) Russell's lectures sketch an ideal language in which each simple thing in the world gets exactly one name. The link goes to the Stanford Encyclopedia's summary of logical atomism, not the lectures.
• Ludwig Wittgenstein — Tractatus Logico-Philosophicus (1921) Wittgenstein's early book pictures language as mirroring the world, with names standing for simple objects, the view he later took apart. Full text, free to read.

Where to start

• Emily M. Bender and Alexander Koller — Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data (2020) This is the octopus story from Episodes 2 and 3, told by the two linguists who made it up. It is free and short, and the story itself needs no technical background. Read it to meet the strongest case that a machine trained only on text cannot mean anything, in its authors' own words.
• Melanie Mitchell and David C. Krakauer — The debate over understanding in AI's large language models (2023) A short, plain, free overview of the exact argument the series works through: whether these systems understand what they say, with the best case on each side put fairly. It also suggests that understanding may come in more than one kind, which is where the finale leaves you.
• Murray Shanahan — Talking About Large Language Models (2022) A clear, free essay by an AI researcher about the words we use for these systems, such as 'knows', 'believes' and 'understands', and when they fit. It puts the series' question to work on how you talk about the machine every day.