Latest / Collective Intelligence: Why groups get smart, strange and stuck / The ox, and the man who wanted you to be mediocre
Transcript
- 0:00Hello, I'm Niccolò Pescetelli and this is Collective Intelligence. Why groups get smart,
- 0:04strange and stuck. Somewhere in Plymouth, in the last month of 1906, there is an ox standing in a
- 0:10pen. It is a fat ox. This is the whole point of it. This is the West of England Fat Stock and
- 0:16Poultry Exhibition. An annual event where farmers bring animals they have spent a year feeding.
- 0:23And other farmers walk around and look at them and there are rosettes involved.
- 0:27The ox has been selected for a competition, not a beauty competition, a guessing competition.
- 0:33Here is how it works. You pay six pens. In exchange, you get a card stamped and numbered
- 0:38and on that card you write three things. Your name, your address and a number. The number is
- 0:44your estimate of what this animal will weigh after it has been slaughtered and dressed. Not what it
- 0:51weighs now, standing there breathing, but what will be left of it on the hook once the
- 0:57parts nobody buys have been taken away. Think about what a strange thing that is to be
- 1:03asked. You are looking at a living animal and you are being asked to perform an act of imagination
- 1:08and arithmetic at the same time. You have to see through the hide to the carcass and then you
- 1:16have to write a number down in public with your name attached in front of butchers. About 800 people
- 1:23paid their sixpence. Some of them knew exactly what they were doing. There were butchers in that queue and
- 1:30farmers, people who had spent 30 years putting a hand on an animal's flank and gnawing. And some of
- 1:36them had no idea at all. They had overheard somebody else's guess or they liked the look of the beast or
- 1:44they had a lucky number. The sixpence in 1906 is not nothing. It is enough to make you take it seriously and cheap enough that you would risk it on a hunch.
- 1:55Then the show ended. The prizes went to whoever came closest. And the cards, having done their job, were boxed up and posted to an 84-year-old man in London who had asked for them.
- 2:08He arrayed them in order, smallest to largest, all 787 of the ones that were legible. He converted them all into pounds because, you know, people had written them in hundredweights and quarters and whatever else.
- 2:25He worked out where the middle was. And in March 1907, he published two pages in Nature about it.
- 2:33Those two pages are the reason you have heard of this. Not the Ox, not the exhibition, not Plymouth. You have heard some version of this story.
- 2:43And the version you have heard goes roughly like this. A crowd of ordinary people guessed the weight of a cow. And their average guess was almost exactly right. Better than the experts.
- 2:55And isn't that remarkable? And doesn't that tell you something hopeful about democracy and markets and the internet?
- 3:04That version is doing well for itself. As I record this, BBC One is running a Saturday night game show called Wisdom of the Crowd,
- 3:11in which contestants try to beat the average answer of the studio audience for £10,000 a round.
- 3:18And the publicity for it credits Galton and the Ox by name.
- 3:23120 years on, An Afternoon in Plymouth is a light entertainment format.
- 3:27I want to tell you a different version. Three things about that different version.
- 3:32The first is who wrote those two pages?
- 3:35Because the man who published the founding document of the Wisdom of Crowds
- 3:38also invented the word eugenics and spent his life trying to prove that ordinary people were,
- 3:46in a quite specific technical sense, mediocre.
- 3:50The second is that his results have nothing to do with wisdom and nothing to do with crowds.
- 3:58Nobody in that queue spoke to anybody.
- 4:00There was no discussion, no deliberation, no group.
- 4:03What happened at Plymouth is a piece of arithmetic and the number that proves it is sitting in his own paper,
- 4:10in a table where it has been for over a century.
- 4:15And the third thing is the condition.
- 4:17The arithmetic works, but only if something very specific is true about the people doing the guessing.
- 4:23Galton knew what the condition was, he wrote it down.
- 4:26And then everyone who retold the story for the next hundred years quietly dropped it.
- 4:32Which is a problem because the condition is the thing that a real society almost never has.
- 4:39Well, let's start with the man.
- 4:41Francis Galton was Charles Darwin's half-cousin and that fact organized his entire life.
- 4:47When On the Origin of Species came out in 1859, Galton read it the way a certain kind of person reads a new tool.
- 4:57He did not read it as a book about finches.
- 4:59He read it as a book about inheritance.
- 5:03And he immediately asked the question his cousin had been careful not to ask out loud.
- 5:09If physical traits are inherited and selected, what about mental ones?
- 5:15What about talent?
- 5:16What about character?
- 5:18Ten years later, he published his answer.
- 5:21Hereditary Genius, 1869.
- 5:24The method is simple and once you see it, it's quite mad.
- 5:28Galton took lists of eminent men, judges, statesmen, commanders, scientists, poets, wrestlers.
- 5:35Then he traced their families and counted how many relatives were also eminent.
- 5:40He found more eminence in those families than chance would predict.
- 5:45And he concluded that ability is inherited.
- 5:50Now, you can see the whole in these from, you know, where you're sitting.
- 5:55The son of a judge in 1869 has a few advantages that are not in his blood.
- 6:00He has a library, for once.
- 6:03He has dinner guests who are also judges.
- 6:05He has a surname that opens doors, money, and no need to earn a living at 15.
- 6:12Galton considered this objection and dismissed it because he had already decided what the answer was.
- 6:20And that is the pattern.
- 6:21That is the thing to hold on to for the rest of this episode.
- 6:26Galton did not gather data and then form a view.
- 6:29He had the view and he built instruments to demonstrate it.
- 6:33By 1883, he had a name for the project.
- 6:36He called it eugenics, from the Greek meaning well-born.
- 6:41The idea was straightforward and, to him, obviously humane.
- 6:44If ability is inherited, then a society could improve itself by encouraging the right people
- 6:51to have more children and discouraging the wrong ones.
- 6:56He wanted marriage guided by measurement.
- 6:59He imagined certificates.
- 7:00He thought the state should have a view on who reproduced.
- 7:06In 1884, he set up an anthropometric laboratory at the International Health Exhibition in South Kensington.
- 7:14He charged the public threepence to be measured.
- 7:19Height, weight, strength, hearing, the keenness of your eyesight, your sense of color.
- 7:26You paid your threepence, you were quantified, and you went home with a copy of your own dimensions,
- 7:33while he kept the original for his files.
- 7:36Nearly 10,000 people did it, out of the 4 million who came through the exhibition.
- 7:41He was assembling a census of human variation, one body at a time,
- 7:47in order to prove that the variation was hereditary.
- 7:51And here is where the story turns technical, and where it gets genuinely interesting.
- 7:56To make his argument work, Galton needed to measure the relationship between parents and children.
- 8:03Not just know that tall fathers have tall sons,
- 8:07but put a number on how strongly the one follows the other.
- 8:12There was no existing mathematics for this, so he made some.
- 8:16He looked at the height of parents and the height of their grown children,
- 8:20and he found something he had not expected.
- 8:23Very tall parents had tall children, but not as tall as themselves.
- 8:29Very short parents had short children, but not as short.
- 8:33Each generation drifted back towards the middle of the population.
- 8:39This is a real statistical phenomenon, and you will meet it everywhere once you know it.
- 8:44The best performing fund this year is probably not the best next year.
- 8:49The student who scored highest on the mock will probably score slightly lower on the real thing.
- 8:55Extremes are partly real and partly luck.
- 8:58And the luck does not repeat.
- 9:01Galton described this phenomenon in a paper in 1886, and I want you to hear what he called it.
- 9:08He called it regression towards mediocrity.
- 9:12Say that out loud and notice what had happened.
- 9:15There is a mathematical fact here, which is that offspring of extreme parents fall back towards the population average.
- 9:22And then there is the world he chose for where they land, mediocrity.
- 9:27In his sentence, the average has a moral temperature and falling towards it is a decline, a disappointment, a leak in the aristocracy of talent.
- 9:37He is describing the arithmetic of normal distribution in the vocabulary of a man watching something valuable being diluted.
- 9:48Later he softened it, regression towards the mean.
- 9:51That is probably the phrase you've learned, if you've learned it.
- 9:55The moral judgment got sanded off and the mechanism kept the name.
- 10:02And regression was not the end of it.
- 10:05Out of the same program in the following years, Galton and then his disciple, Carl Pearson, built correlation.
- 10:13The correlation coefficient, the correlation coefficient, that number between minus one and one that you have used or seen used in every field there is,
- 10:23you know, from drug trials to school league tables to whether coffee is bad for you this week.
- 10:29It came out of an attempt to quantify heredity in service of selective breeding.
- 10:37There is a book about this, Aubrey Clayton, Bernoulli's Fallacy from 2021.
- 10:43They voted two chapters to Galton, Pearson and Ronald Fisher.
- 10:47And the argument is roughly this.
- 10:50The standard statistical toolkit that every science now runs on, correlation, regression, significance testing, the chi-square test, the t-test, etc.
- 11:00were largely built by three men who were committed eugenicists in the course of that commitment.
- 11:08And the insistence that these methods were purely objective, free of judgment, a machine you feed data into, was useful to them.
- 11:17If your politics comes out of the machine, it is not politics anymore.
- 11:21It is just what the numbers say.
- 11:24Now, I want to be careful here because there is a very easy mistake available and I'm not going to make it.
- 11:29The easy mistake is to conclude that the statistics are therefore tainted, that correlation is a racist idea, that regression carries its origins around with it, that we should not use these tools.
- 11:45Jerry Cohen wrote a sharp piece against Clayton in 2022, making exactly this objection.
- 11:52And on the central point is his right.
- 11:55He concedes the racism immediately.
- 11:58Nobody's arguing about the racism.
- 11:59And then he says, so what?
- 12:03Whether a method works is a question about the methods.
- 12:06If the chi-square test is valid, it is valid.
- 12:09And who invented it and why is a matter for historians?
- 12:13Reasoning from the origin of an idea to the truth of it has a name.
- 12:19And it is the genetic fallacy.
- 12:22And it is a fallacy.
- 12:24And to be fair to Clayton, his book's actual argument against frequentist statistics is technical, not moral.
- 12:30He thinks the logic is wrong and that we should be Bayesian.
- 12:35The eugenics is context.
- 12:38So hold both of these at once because you will need both.
- 12:41The history is real and it is not decoration.
- 12:44Galton built these tools for a purpose.
- 12:47And the purpose shaped which questions seemed worth asking.
- 12:52And the tools work anyway.
- 12:55Those two statements do not fight each other, right?
- 12:57Like they only seem to fight if you think the value of an idea is inherited from its parents.
- 13:04Which, and I enjoy this more than I should, like is exactly what Galton believed about people.
- 13:10Which brings us back to the ox.
- 13:13Because of everything Francis Galton published in a very long and very productive life, the thing that made him famous outside statistics, the thing you have heard of, the thing that gets quoted in business books and TED talks and BBC programs,
- 13:29And what those two pages appear to show is that 800 ordinary people, the mediocrity, the mass, the ones he wanted measured and certificated and guided into appropriate marriages,
- 13:50Collectively knew something that almost none of them knew individually.
- 13:56He noticed.
- 13:58He says in the paper, more or less directly, that the average competitor at the cattle show is about as well equipped to judge the weight of an ox as the average voter is to judge a political question.
- 14:12That is not a throwaway line.
- 14:14He wrote a separate letter to Nature the week before this one, arguing about voting.
- 14:20He was thinking about democracy.
- 14:23And his verdict on his own result, the last line of the argument, is one of the most wonderfully grudging sentences in the history of science.
- 14:33He says,
- 14:33More than might have been expected, but by whom, Francis?
- 14:45So how did the man who thought the mass was mediocre ended up proving the opposite?
- 14:50Well, he didn't.
- 14:51And that is the second thing.
- 14:53Let's go back to the queue in Plymouth and look at what actually happened physically, right, in the world.
- 15:00A man pays sixpence, he takes a card, he looks at the animal, he thinks, he writes, say, 1150.
- 15:07He hands the card in, he walks away.
- 15:10A woman behind him pays sixpence, takes a card, looks at the animal, writes, say, 1260, hands in it, hands the card in, walks away.
- 15:20Now, that is it.
- 15:21That is the entire social structure of the most famous experiment in collective intelligence.
- 15:27787 people, one after another, each writing a number on a piece of paper and leaving.
- 15:35Nobody held a meeting.
- 15:37Nobody argued.
- 15:39Nobody was persuaded of anything.
- 15:42There was no discussion, no deliberation, no consensus, no group in any meaningful sense.
- 15:50These people were a queue, not a community.
- 15:53If you took the same 787 cards and shuffled them or mailed them in from 787 different countries on 787 different days,
- 16:04you would get precisely the same result.
- 16:09I want to sit on this for a moment because it is routinely skipped and it changes what the story means.
- 16:15The wisdom of crowds, in its founding case, contains nothing social at all.
- 16:22Whatever happened at Plymouth, it is not a fact about human cooperation.
- 16:27It is not about how groups think together or how conversations sharpens judgment or how diverse perspectives combine.
- 16:35None of those things occurred.
- 16:37What occurred was that a large number of people independently produced a number and then one man added them up.
- 16:46So what is going on?
- 16:47Here is the mechanism and you already have everything you need to understand it.
- 16:52When that first man writes 1150, his number is made of two things.
- 16:58There is the real weight of the ox, which is a fixed fact about the world.
- 17:03And there is his error, which is however far off he is from the true value.
- 17:11Every card in the box is the truth plus an error.
- 17:14Some truth, different errors.
- 17:18Now, think about the errors.
- 17:19Some people guess too high.
- 17:21Some guess too low.
- 17:23The butcher who has handled 10,000 animals was out by 20 pounds in one direction.
- 17:30The man who liked the look of the beast was out by 200 in the other.
- 17:35And crucially, there is no reason for those two mistakes to agree with each other.
- 17:41They are made for different reasons, by different people, with different experience, standing in front of the animal at different moments.
- 17:51When you average them, the errors meet each other.
- 17:54The too high and the too low push in opposite directions and partly cancel.
- 18:00The truth is in every single guess.
- 18:03So the truth does not cancel.
- 18:05It survives.
- 18:06And everything else quietly sort of eats itself.
- 18:12That is the whole trick, not wisdom, right?
- 18:14It's just noise cancellation.
- 18:16And here is the part I find genuinely delicious.
- 18:20Because it means Galton had the answer in his hands and did not quite see what it explained.
- 18:27In that paper, in the table, Galton reports how spread out the individual guesses were.
- 18:33It gives what was then called the probable error of a single estimate.
- 18:40And the number it gives is 37 pounds.
- 18:44In plain terms, pick one person's card at random.
- 18:48And it is a coin flip whether they are within 37 pounds of the middle or further out than that.
- 18:57That is his own measurement of how bad the typical guesser was.
- 19:03And they were bad.
- 19:0437 pounds on an animal of about 1,200 is 3%.
- 19:09In a competition where the winner needed to be right to the pound.
- 19:14Now, apply the cancellation.
- 19:17When you average independent errors, the spread of the average shrinks.
- 19:22It does not shrink in proportion to the number of people.
- 19:26It shrinks in proportion to the square root of the number of people.
- 19:31That square root is the single most important thing in these episodes.
- 19:36So, let me make, you know, concrete rather than algebraic.
- 19:41Imagine 4 people.
- 19:43The square root of 4 is 2.
- 19:46Your error halves.
- 19:4816 people.
- 19:49Square root is 4.
- 19:53Your error quarters.
- 19:5564 people.
- 19:568 times better.
- 19:58256 people.
- 20:0016 times better.
- 20:02Notice the shape of that.
- 20:03To halve your error, you do not add a few more people.
- 20:07You quadruple the crowd.
- 20:11The first handful of people do almost all of the work.
- 20:14And after that, you are buying improvement at a punishing exchange rate.
- 20:19So, 787 people.
- 20:24The square root of 787 is just over 28.
- 20:29Take Galton's own 37 pounds.
- 20:32Divide by 28.
- 20:34And the arithmetic predicts that the average of those cards should land within about one and a third pounds of the truth.
- 20:45The average of those cards was 1,197 pounds.
- 20:50The ox dressed out at 1,198.
- 20:56So, that was one pound difference.
- 20:59Which makes it a prediction rather than a miracle.
- 21:02A prediction made from a number Galton printed himself coming true to within a third of a pound.
- 21:10If the average had been out by 40 pounds, that would have been surprising and would have needed explaining.
- 21:17Landing where he landed is the least surprising thing in the paper.
- 21:21The crowd was numerous.
- 21:23That is all it had to be.
- 21:25And while we are in the neighbourhoods, a small piece of history that statisticians enjoy.
- 21:31And that I will give you in 30 seconds.
- 21:33Because it is a good story.
- 21:35And not, you know, the whole point.
- 21:37But Galton did not use the average.
- 21:40He used the middle value.
- 21:41The one where half the guesses are higher, half are lower.
- 21:45He had a democratic reason for it.
- 21:47And he had made the argument in print the week before.
- 21:53The middle value is the only one that the majority does not vote against.
- 21:58Everything above it, most voters say it's too high.
- 22:01Everything below it, most say it's too low.
- 22:05So, the middle is the ballot box verdict.
- 22:10And that middle value was 1,207.
- 22:15Which is nine pounds out, not one.
- 22:19And two weeks after the paper appeared, a man called Hooker wrote in to Nature and pointed out politely that the plain average looked like it would do better than Galton's middle value.
- 22:34And Galton replied a week later, he calculated the average of all 787 figures, gave it as 1,197, made a satisfied remark about the usefulness of his table, and did not really address Hooker's point at all.
- 22:50So, the number in every retelling of the story, the one pound, the near-perfect crowds, comes out of a letter Galton wrote to fend off a critic.
- 23:01It is not the famous paper.
- 23:05And if you go to his archive, as the economist Ken Wallis did in 2014, you find that the true weight of the ox was not 1,198 either.
- 23:16It was 1,197.
- 23:20Which means the average of the crowd was not one pound out, it was exactly right.
- 23:27Anyway, fun fact.
- 23:29Back to the argument, because there is something much more important sitting underneath all of this.
- 23:34And it is the thing that makes averaging fail.
- 23:38Cancellation works on noise.
- 23:40It does absolutely nothing to bias.
- 23:44Noise is error that points in random directions.
- 23:48Bias is error that points the same way for everyone.
- 23:54And if every single person in that queue was making the same mistake in the same direction,
- 24:01then averaging them, you know, does not help at all.
- 24:04You can put a million people in the queue.
- 24:07The million will be wrong by exactly the same amount the one was.
- 24:12Because there is nothing for the errors to cancel against.
- 24:16They are all essentially pushing the same way.
- 24:21Imagine the exhibition organizers had hung a sign over the pen reading, you know, last year's ox was 1,400 pounds.
- 24:32Now, everybody's guess is dragged upwards.
- 24:36The spread might look the same.
- 24:37The average is ruined, you know, permanently because of this sign.
- 24:42And there is a hint of something like this in Galton's own data, which he noticed, reported honestly and could not explain.
- 24:54When he looked at the shape of the distribution, it was not symmetrical.
- 24:59The guesses below the middle were spread out more widely than the guesses above it.
- 25:04He fitted the lower half to one curve and the upper half to a different one.
- 25:10And he says plainly, plainly, that he does not know why.
- 25:15That he does not have enough understanding of how people judge weights to offer a useful opinion.
- 25:22And he gestures vaguely at psychology.
- 25:25But there are symmetries sitting there in the founding paper of collective intelligence, unexplained.
- 25:30And it is precisely the footprint you would expect a shared bias to leave.
- 25:36Nobody in the retelling mentions this.
- 25:39It is not a hopeful fact, so it fell out of the story.
- 25:44Which raises the question of what that sixpence was really buying.
- 25:49Everything I have just described rests on one word.
- 25:53And I have been using it without stopping on it, which is independence.
- 25:59The error cancel if they are independent, if they are unrelated, if the reason you guess is too high has nothing to do with the reason my guess is too high.
- 26:10Most people, when they hear that, you know, file it as a technicality, like a footnote.
- 26:16A condition kind of you check off before getting on with the interesting part.
- 26:21And I want you to persuade you that it is the whole thing in a way that will probably annoy you.
- 26:27Because once you see how fragile it is, you know, you will start noticing it being violated everywhere.
- 26:34And there is no way to stop noticing it.
- 26:36So let's break it deliberately and watch what happens.
- 26:40Now go back to the queue.
- 26:41Same ox, same card, same sixpence.
- 26:43But this time, one man near the front of the queue is, let's say, a big personality.
- 26:48And as he hands in his card, he turns around and announces to the 40 people behind him,
- 26:551100.
- 26:56I've been doing this for 30 years.
- 26:581100.
- 26:59Now, what happens to those 40 cards?
- 27:02Some people ignore him.
- 27:04Some people, being contrary, deliberately go the other way.
- 27:07But most people do the thing that people do, which is that the number 1100 is now in their head.
- 27:15And it was not there 10 seconds ago.
- 27:17And their own hazy sense of the animal drifts towards it.
- 27:22They were going to write, say, I don't know, 1250.
- 27:28And now they write 1180.
- 27:30Because 1100 sounded confident and they are not.
- 27:36You still have 40 cards.
- 27:38You still count 40 entries.
- 27:40Galton would still write 787 at the top of his table.
- 27:44But you no longer have 40 independent opinions.
- 27:49You have something closer to one loud opinion wearing, you know, 40 coats.
- 27:56And that is the failure mode.
- 27:58And it has a precise mathematical shape, which I'm going to give you in the form of a single terrible number.
- 28:06Statisticians measure how much people's errors move together with a correlation.
- 28:10Zero means completely unrelated.
- 28:12Everyone wrong in their own private way.
- 28:15One means perfectly locked together.
- 28:18Everyone making identical mistakes.
- 28:21Real human beings living in the same society, reading the same news, using the same rules of thumb,
- 28:27are somewhere in between and usually much closer to zero than to one.
- 28:34So, you know, it can be generous.
- 28:36Let's say the correlation between people's error is 1%.
- 28:39Well, one in a hundred.
- 28:43You know, it's almost nothing.
- 28:44A society is so weakly coupled that it barely deserves the name.
- 28:50At 1% correlation, 787 people are worth about 89.
- 28:59You have paid for 787 opinions.
- 29:02You have collected 787 cards.
- 29:07You have, in terms of what the cancellation can actually do for you, 89 people.
- 29:137a, 7-8 of your crowd has basically evaporated.
- 29:19And the cause is a correlation so small, you would struggle to detect it.
- 29:23Push it a little, 5%, which is still a weak coupling by any standard.
- 29:28You would recognize, you know, from the real world, 787 people become now 19, 10%.
- 29:35You have an exhibition full of guesses and you're getting the benefit of a dinner party.
- 29:50And it gets worse, or rather it gets stranger, because there is a second consequence that is genuinely counterintuitive.
- 29:57Once the errors are correlated at all, there is a floor.
- 30:00A level of error that the average cannot get below, no matter how many people you add.
- 30:07And think about why.
- 30:08The cancellation only ever ate the part of each error that was private.
- 30:13If some slice of everyone's error is shared, that slice is in every single guess.
- 30:20Exactly like the truth is in every single guess.
- 30:23And so it survives averaging exactly like the truth does.
- 30:29The averaging cannot tell the difference between a fact that everybody knows and a mistake that everybody makes.
- 30:36It keeps both, right?
- 30:38So at 10% correlation, with that £37 spread, the average's error settles at about £12 and stops.
- 30:51Not £12 at 700 people and better, you know, 7,000.
- 30:56Like £12 at 700, £12 at 7 million, £12 at, you know, any number.
- 31:05You have hit the floor and more people, adding more people is not a lever anymore.
- 31:11Now, I have to be honest with you about something, because this is the point in an episode where it would be easy to slide from mathematics into assertion.
- 31:19I do not know what the correlation was between the errors of the people in that queue at Plymouth.
- 31:27Nobody does.
- 31:28Galton did not measure it and could not have.
- 31:31The figures I gave you are what the model says happens at those levels of correlation.
- 31:37They are not a measurement of his crowd.
- 31:40They tell you how sensitive the mechanism is.
- 31:43They do not tell you where his crowds are on that scale.
- 31:48But here is what makes me think about it.
- 31:51And it is Galton himself, and it is the part of the paper that everybody drops.
- 31:58When he describes his data, he pauses to explain why he thinks these particular guesses are worth analysing.
- 32:05And his reasons are not about the people.
- 32:07His reasons are about the conditions.
- 32:09He says the judgments were unbiased by passion and uninfluenced by oratory.
- 32:17He says the six penny fee, the third practical joking.
- 32:21He says the hope of a prize and the joy of competition made people try.
- 32:27Read that list again as a piece of engineering.
- 32:30No passion, no oratory.
- 32:32Nobody standing on a box telling the crowd what to think.
- 32:35A fee, which filters out anyone who is messing about.
- 32:40And a prize, which makes you concentrate.
- 32:43And a small stamped card filled in alone, handed in, gone.
- 32:47Those are the specifications for an isolation chamber.
- 32:52Francis Galton in 1907 wrote down the operating conditions for his own result.
- 32:58And the conditions are, make sure these people cannot influence each other.
- 33:03The sixpence was not just a filter for seriousness.
- 33:07Along with the card and the cue and the individual act of writing,
- 33:11it was the apparatus that kept 787 people from becoming one person with 787 voices.
- 33:22And this is where the whole thing turns for me, from a charming anecdote into something closer to a warning.
- 33:29Because every single thing that makes a society a society pushes that correlation up.
- 33:36Conversation pushes it up.
- 33:38That is what conversation is for, right?
- 33:40News push it up.
- 33:43And shared news pushes it up even harder.
- 33:47Prestige pushes it up.
- 33:49Because when someone we respect commits to a number, the rest of us move.
- 33:54Platforms push it up by design.
- 33:59Since a feed that shows you what everyone else is looking at is a machine for correlating errors.
- 34:05Optimized and at scale.
- 34:08Even expertise pushes it up because experts are trained on the same material and taught the same heuristics
- 34:15and will therefore be wrong together in ways that amateurs would not be.
- 34:20The founding results of collective intelligence require people not to talk to each other.
- 34:26And we have spent the subsequent century building an infrastructure whose entire purpose is to
- 34:31make people talk to each other more, faster, and about the same things at the same time.
- 34:36So when you see a poll or a market prize or a trending topic or
- 34:40a hundred thousand people agreeing about something on the internet,
- 34:44the question to ask is not how many.
- 34:46The question is how many independent ones?
- 34:50And the honest answer is almost always a far smaller number than one printed at the top.
- 34:58Which sounds like I am about, you know, to tell you the whole idea is just a curiosity
- 35:05that only works when nobody is listening.
- 35:08No, I'm not.
- 35:09Because there are people who took this seriously.
- 35:12Built the isolation on purpose and used it to find cancer.
- 35:18A mammogram is a grey image of a breast and somewhere in that grey there may or may not be a cancer.
- 35:26This is a hard problem and not hard in the way that, you know, a hard sum is hard.
- 35:32It is hard because the signal is faint and buried in texture.
- 35:36And because the two ways of being wrong are both terrible in different directions.
- 35:42Miss a cancer and a woman goes home reassured with a tumor in her.
- 35:47Call a cancer that is not there and a healthy woman gets a biopsy, a fortnight of terror and a scar.
- 35:57Radiologists are trained and careful and they disagree with each other constantly.
- 36:04Show the same image to two of them and you will sometimes get two answers.
- 36:09Show it to the same one twice and you will occasionally get two answers.
- 36:15In 2016, a group led by Ralf Kurvers with Stefan Herzog, Ralph Hertwig, Jens Krause and Max Wolf and other colleagues published a paper in the Proceedings of the National Academy of Sciences that does to mammography exactly what Galton did to the ox.
- 36:34They had one of the largest data sets of its kind, a 101 radiologists, a 182 mammograms and 16,813 separate interpretations.
- 36:50They run the whole thing again on skin cancer as well with 40 dermatologists, 108 lesions, 4,320 diagnoses.
- 37:02Individually, the radiologists were about as good as you would fear.
- 37:08On average, they caught roughly 77% of the cancers and correctly cleared about 67% of the healthy cases.
- 37:18Which means roughly a quarter of cancers missed and a third of healthy women recalled by trained specialists on their own.
- 37:28Then the researchers took those independent readings, which had been made by people who never met, never discussed the case, and combined them into groups of two, three, five, etc.
- 37:42Not real groups, constructed ones, right?
- 37:45Assembled after the fact from readings that already existed.
- 37:51They tried different ways of combining.
- 37:54For example, a majority rule, where you go with whatever most of them said.
- 37:59A confidence rule, where you take the call of whoever was most sure of themselves.
- 38:05And the pooled verdicts beat the individuals.
- 38:08Not on one measure at the cost of another, which is the usual way these things go.
- 38:13They caught more cancers and they recalled fewer healthy women at the same time.
- 38:25Both directions at once.
- 38:28More than that, the pooled verdicts beat the best single radiologist in the group.
- 38:33Not the average member, like the best one.
- 38:36The people you would have wanted reading your scan, if you had known who they were, and you never do, they beat those ones.
- 38:45Now there is Plymouth in a hospital, 109 years later.
- 38:49Same mechanism, same absence of conversation, and the isolation is not an accident of a cattle show, but a deliberate design choice.
- 38:59Now the refinement, and this is the part to hold on to, because it is where the story stops being simply encouraging.
- 39:06It does not always work.
- 39:10The pooling beat the best individual when the members of the group were of roughly similar accuracy.
- 39:17When they were not, when you put a genuinely strong diagnostician together with a weak one, the strong one on their own did better than the combination.
- 39:30Think about what that means.
- 39:32Averaging is not a free upgrade.
- 39:34If you already know who the best person in the room is, and they are substantially better than the others,
- 39:40then combining opinions actively destroys information.
- 39:45You are diluting a good signal with a bad one, and kind of calling it wisdom.
- 39:50The crowd only helps when nobody in it is decisively better than the rest.
- 39:56And notice that in the real world, you usually cannot tell which situation you are in,
- 40:00because working out who the best diagnostician is requires exactly the kind of large-scale outcome data that these researchers had,
- 40:10and that a hospital on a Tuesday does not.
- 40:14The authors are careful about the boundary of their claim, and I want to be careful with it, too.
- 40:21They combined independent judgments.
- 40:24They did not study doctors talking to each other.
- 40:26They say so explicitly, and they say they do not know how far their findings would carry over to a group that discusses a case.
- 40:35Which is, of course, how medicine actually works.
- 40:37Real radiologists confer.
- 40:40There are multidisciplinary meetings.
- 40:44Juniors ask seniors, and seniors have reputations, and the room has a hierarchy in it, etc., etc.
- 40:50The moment they talk, the correlation goes up.
- 40:53And you are back in the problem from the last section.
- 40:58So, the honest summary is narrower and more useful than the one you will see quoted.
- 41:03Aggregating independent judgments of similar quality in a domain with a checkable answer reliably beats individuals,
- 41:11including the best individual.
- 41:13And that is a real robust, valuable finding.
- 41:17It is also a much smaller claim than the wisdom of the crowds.
- 41:23And it comes with a list of conditions that a cattle show in 1906 happened to satisfy by accident,
- 41:30and that most human institutions violate by default.
- 41:37So, let's go back and stand in front of the animal one more time.
- 41:43The ox is real.
- 41:44The sixpence is real.
- 41:45The 787 cards in the box posted to an old man in London,
- 41:50who had spent 40 years trying to prove that people like the ones in that queue were the mediocrity that talent regresses towards.
- 41:59All real.
- 42:01And what came out of it is real too, but it is not what it has been sold as.
- 42:08It is not evidence that people are collectively wise.
- 42:11It is evidence that independent errors cancel,
- 42:14which is a fact about arithmetic that would be equally true of 787 thermometers
- 42:21or 787 slightly miscalibrated bathroom scales.
- 42:27There is no reason to feel warm about it.
- 42:29A crowd that cancels its own noise is not displaying a virtue.
- 42:33It is displaying a property of addition.
- 42:38The genuinely interesting thing Galton found, and did not know he had found,
- 42:42is the list of conditions.
- 42:45No passion, no oratory, a fee, a prize, a private card.
- 42:50He thought he was explaining why his data was clean.
- 42:54He was, in fact, writing down the specification for a machine.
- 42:58And the specification says, keep people apart.
- 43:02So here is the so what, and it's not very comfortable.
- 43:08Every time you see a large number of people agreeing,
- 43:11your instinct is to treat the number as the evidence.
- 43:15100,000 people can't all be wrong.
- 43:1810,000 five-star reviews, 60% in the polling.
- 43:22The market has priced it in.
- 43:25The instinct is that scale is reassuring.
- 43:29And the instinct comes from this story, like from the ox,
- 43:33from the idea that many eyes see better than one pair.
- 43:37The arithmetic says the number is almost never the number.
- 43:40What matters is how many of those opinions were formed without reference to each other.
- 43:47And in a society with newspapers and phones and feeds,
- 43:51the answer is a small fraction of the total.
- 43:54And sometimes the answer is close to one.
- 43:59100,000 people who all read the same post that morning
- 44:02are not 100,000 pieces of evidence.
- 44:06They are one piece of evidence, repeated,
- 44:10and the repetition feels like a corroboration
- 44:12because our instincts were built for a world
- 44:14where hearing something twice usually meant two people had seen it.
- 44:20And the correction is not to distrust crowds.
- 44:23That is kind of a lazy move.
- 44:25It's wrong.
- 44:27The correction is to ask a harder question,
- 44:29which is what the mammogram people ask.
- 44:33Are these judgments independent?
- 44:35Are they of comparable quality?
- 44:38And is there a checkable answer at the end?
- 44:43When those three things hold,
- 44:45pooling works.
- 44:46It works better than the best experts you could find,
- 44:49and you should use it on purpose.
- 44:53When they do not hold,
- 44:54you have a cue in which one loud man announces 1100
- 44:58and everyone else is now wearing his opinion.
- 45:01Which leaves me with the question I actually want to spend this series on.
- 45:07And I will tell you now that I do not think it has a clean answer.
- 45:13If independence is the thing that makes aggregation work,
- 45:17then talking to each other is the thing that breaks it.
- 45:20Every conversation, every meeting, every deliberative assembly,
- 45:24every attempt to reason together is,
- 45:26from the point of view of the mathematics in this episode,
- 45:29a machine for correlating errors and destroying the very property that made the crowd smart.
- 45:37And yet, we do not believe that.
- 45:40Nobody really believes the best way to run a society
- 45:43is to keep everybody in separate rooms,
- 45:47writing numbers on cards.
- 45:50We think discussion improves things.
- 45:53We have built parliaments and juries and committees and peer review
- 45:57on the assumption that it does.
- 46:00So, either we are wrong about deliberation,
- 46:03or there is something that talking ads,
- 46:06which more than pays for the independence it destroys.
- 46:12That is the next few episodes.
- 46:14There are two very good papers,
- 46:16published in the same journal, six years apart,
- 46:20that ask exactly this question and reach opposite conclusions.
- 46:24One finds that letting people see each other's estimates wrecks the crowd.
- 46:29The other finds it reliably improves it.
- 46:33The thing that separates them is not the question,
- 46:36and it is not the quality of the work.
- 46:38It is a single choice about how the people
- 46:40were connected to each other.
- 46:42And I'm going to take them both seriously.
- 46:44Before that, next time we go back 122 years before Galton
- 46:50to a French aristocrat who proved mathematically
- 46:54that the majority vote converges on the truth,
- 46:56and who built that proof on an assumption so demanding
- 47:00that almost nothing in public life has ever satisfied it.
- 47:05He died in a prison cell during the revolution he helped design.
- 47:10Thanks for listening.
- 47:11The paper is Galton Vox Populi in Nature, March 1907,
- 47:14and it is two pages,
- 47:16and you can read the whole thing in 10 minutes,
- 47:18which I recommend.
- 47:19Ken Wallis's archive work is in Statistical Science in 2014.
- 47:24Aubrey Clayton's book is Bernoulli's Fallacy.
- 47:26The mammogram study is Kurvers and colleagues in PNAS in 2016.
- 47:30All of them are in the notes.