Latest / 80,000 Hours Podcast / Beyond human minds: The bewildering frontier of consciousness in insects, AI, and more

Transcript

Beyond human minds: The bewildering frontier of consciousness in insects, AI, and more · 80,000 Hours Podcast

• Meghan Barrett: The reason I am not giving you a sentience score is the error bars are so large right now that it’s almost a meaningless number. Because I’m waiting on so much evidence. So much evidence. So I think that’s really an essential feature of it for me. The second thing is that I worry especially as an expert that that number would be overemphasised, that somebody would inevitably put into a spreadsheet, and they would use that spreadsheet to make all kinds of decisions. And that number does not reflect the complexity that you and I have now spent three hours discussing, and barely scratched the surface of. I want to talk about the complexity and the nuance, and a number does not demonstrate that.s I think it’s important also that we understand that if you have updated at all towards insects plausibly being sentient, scale takes the rest of the issue for you to a serious place. There are so, so, so many of them that if you take it seriously at all, then you need to be thinking that this is an issue to work on. Luisa Rodriguez: Hey listeners, Luisa here. You were just listening to Meghan Barrett, an insect neurobiologist whose mind-blowing facts about insects really opened my eyes about the likelihood of invertebrate sentience when I interviewed her last year. We’re back today with another compilation of our favourite bits from past shows — this time on a topic that listeners might know I’m absolutely captivated by: consciousness and sentience in nonhuman minds, including digital ones. Over the last few years, this area of study has really taken off, and many of the guests you’ll hear in this episode are pioneers in their fields. Whether they’re entomologists, philosophers, machine learning researchers, or neuroscientists, they’re all studying angles on slippery questions like what consciousness is (even in humans!), how we’d recognise it in minds unlike our own, and how much we should worry about other beings’ ability to experience pleasure and pain. You’ll hear from: David Chalmers on why artificial consciousness is possible Jeff Sebo on what the threshold is for AI systems meriting moral consideration Jonathan Birch on how we very recently thought newborns couldn’t feel pain, and how we’re likely making similar mistakes today Robert Long on how we might stumble into causing enormous suffering for digital minds Meghan Barrett on the evolutionary origins of consciousness and sentience, and whether brain size and sentience are related Plus many more! You also may have noticed that this is a pretty long episode. We decided not to cut it down more, given that this issue is important, and long episodes on this topic haven’t deterred listeners in the past… on the contrary, our 4 hour and 40 minute marathon session with David Chalmers was one of our most popular episodes! All right, I hope you enjoy! Robert Long on what we should picture when we think about artificial sentience Luisa Rodriguez: I basically don’t feel like I have a great sense of what artificial sentience would even look like. Can you help me get a picture of what we’re talking about here? Robert Long: Yeah. I mean, I think it’s absolutely fine and correct to not know what it would look like. In terms of what we’re talking about, I think the short answer, or a short hook into it, is just to think about the problem of animal sentience. I think that’s structurally very similar. So, we share the world with a lot of nonhuman animals, and they look a lot different than we do, they act a lot differently than we do. They’re somewhat similar to us. We’re made of the same stuff, they have brains. But we often face this question of, as we’re looking at a bee going through the field, like we can tell that it’s doing intelligent behaviour, but we also wonder, is there something it’s like to be that bee? And if so, what are its experiences like? And what would that entail for how we should treat bees, or try to share the world with bees? I think the general problem of AI sentience is that question, and also harder. So I’m thinking of it in terms of this kind of new class of intelligent or intelligent-seeming complex systems. And in addition to wondering what they’re able to do and how they do it, we can also wonder if there is, or will ever be, something that it’s like to be them, and if they’ll have experiences, if they’ll have something like pain or pleasure. It’s a natural question to occur to people, and it’s occurred to me, and I’ve been trying to work on it in the past couple of years. Luisa Rodriguez: I guess I have an almost even more basic question, which is like, when we talk about AI sentience — both in the short term and in the long term — are we talking about like a thing that looks like my laptop, that has like a code on it, that has been coded to have some kind of feelings or experience? Robert Long: Yeah, sure. I use the term “artificial sentience.” Very generally, it’s just like things that are made out of different stuff than us — in particular, silicon and the computational hardware that we run these things on — could things built out of that and running computations on that have experiences? So the most straightforward case to imagine would probably be a robot — because there, you can kind of clearly think about what the physical system is that you’re trying to ask if it’s sentient. Things are more complicated with the more disembodied AI systems of today, like ChatGPT — because there, it’s like a virtual agent in a certain sense. And brain emulations would also be like virtual agents. But I think for all of those, you can ask, at some level of description or some way of carving up the system, “Is there any kind of subjective experience here? Is there consciousness here? Is there sentience here?” Luisa Rodriguez: Yeah, cool. I guess the reason I’m asking is because for a long time I’ve had this sense that when people use the term “digital minds” or “artificial sentience,” I have like some vague images that kind of come from sci-fi, but I mostly feel like I don’t even know what we’re talking about. But it sounds like it could just look like a bunch of different things, and the core of it is something that is sentient — in maybe a way similar, maybe a way that’s pretty different to humans — but that exists not in biological form, but in some grouping that’s made up of silicon. Is that basically right? Robert Long: Yeah. And I should say, I guess silicon is not that deep here. But yeah, something having to do with running on computers, running on GPUs. I’m sure I could slice and dice it, and you could get into all sorts of philosophical-like classification terms for things. But yeah, that’s the general thing I’m pointing at. And I in particular have been working on the question of AI systems. The questions about whole brain emulations I think would be different, because we would have something that at some level of description is extremely similar to the human brain by definition. And then you could wonder about whether it matters that it’s an emulated brain, and people have wondered about that. In the case of AIs, it’s even harder — because not only are they made on different stuff and maybe somewhat virtual, they also are kind of strange and not necessarily working along the same principles as the human brain. Jeff Sebo on what the threshold is for AI systems meriting moral consideration Jeff Sebo: The general case for extending moral consideration to AI systems is that they might be conscious or sentient or agential or otherwise significant. And if they might have those features, then we should extend them at least some moral consideration in the spirit of caution and humility. So the standard should not be, “Do they definitely matter?” and it should also not be, “Do they probably matter?” It should be, “Is there a reasonable, non-negligible chance that they matter, given the information available?” And once we clarify that that is the bar for moral inclusion, then it becomes much less obvious that AI systems will not be passing that bar anytime soon. Luisa Rodriguez: Yeah, I feel kind of confused about how to think about that bar, where I think you’re using the term “non-negligible chance.” I’m curious: What is a negligible chance? Where is the line? At what point is something non-negligible? Jeff Sebo: Yeah, this is a perfectly reasonable question. This is somewhat of a term of art in philosophy and decision theory. And we might not be able to very precisely or reliably say exactly where the threshold is between non-negligible risks and negligible risks — but what we can say, as a starting point, is that a risk can be quite low; the probability of harm can be quite low, and it can still be worthy of some consideration. For example: why is driving drunk wrong? Not because it will definitely kill someone. Not even because it will probably kill someone. It might have only a 1-in-100 or 1-in-1,000 chance of killing someone. But if driving drunk has a 1-in-100 or 1-in-1,000 chance of killing someone against their will unnecessarily, that can be reason enough to get an Uber or a Lyft, or stay where I am and sober up. It at least merits consideration, and it can even in some situations be decisive. So as a starting point, we can simply acknowledge that in some cases a risk can be as low as one in 100 or one in 1,000, and it can still merit consideration. Luisa Rodriguez: Right. It does seem totally clear and good that regularly in our daily lives we consider small risks of big things that might be either very good or very bad. And we think that’s just clearly worth doing and sensible. Sometimes probably, in personal experience, I may not do it as much as I should — but on reflection, I certainly endorse it. So I guess the thinking here is that, given that there’s the potential for many, many, many beings with a potential for sentience, albeit some small likelihood, it’s kind of at that point that we might start wanting to give them moral consideration. Do you want to say exactly what moral consideration is warranted at that point? Jeff Sebo: This is a really good question, and it actually breaks down into multiple questions. One is a question about moral weight. We already have a sense that we should give different moral weights to beings with different welfare capacities: If an elephant can suffer much more than an ant, then the elephant should get priority over the ant to that degree. Should we also give more moral weight to beings who are more likely to matter in the first place? If an elephant is 90% likely to matter and an ant is 10% likely to matter, should I also give the elephant more weight for that reason? And then another question is what these beings might even want and need in the first place. What would it actually mean to treat an AI system well if they were sentient or otherwise morally significant? That question is going to be very difficult to answer. So there are no immediate implications to the idea that we should give some moral consideration to AI systems if they have a non-negligible chance of being sentient. All that it means is that we should give them at least some weight when making decisions that affect them, and then we might disagree about how much weight and what follows from that. Meghan Barrett on the evolutionary argument for insect sentience Meghan Barrett: Importantly, I think another fact that a lot of people don’t realise from this evolutionary perspective is that insects are thus actually most closely related to crustaceans. They form this group together called the pancrustacea, and that group shares a common ancestor. So I think that’s something else that people should consider. If you’re somebody who takes crustacean sentience seriously, and sentience is a trait that has evolved — it didn’t just appear in organisms as they are today, which I’m sure will be something we repeatedly touch on throughout this episode — if you’re somebody who believes that crustaceans are given the benefit of the doubt for you, for sentience, then you might also think that insects are worth giving the benefit of the doubt for some reasons as well, from an evolutionary perspective. And there are two possible hypotheses you could entertain here. One is the idea that sentience evolved exactly one time — and so everybody descended from that common ancestor, unless maybe they lost it for some reason, has that characteristic. So if you accept both vertebrates and any of the inverts — so a cephalopod or a decapod — if you’re convinced on a single invertebrate and you also are convinced on a single origin point for this, you have a problem, right? Because the closest common ancestor to all of those folks is very far back. And so you’re going to have all your insects, all your nematodes, all your decapods, all your annelids (which are another kind of worm) included, if you believe that there’s one common ancestor and no loss events. Now, maybe you think there’s loss events, but now you’re talking about multiple loss events, because there’s so many invertebrates. So you’re going to have to justify each of those losses, which you could potentially do. Another possible hypothesis is that sentience evolved multiple times independently in different groups. I would probably say this is more plausible in my view, in part because we see multiple emergence of things all the time in evolutionary history. Vision is a great example: we know that eyes might have evolved as many as 40 times during animal evolutionary history. And then, when we think about the development of eyes, it’s crucial to consider how they all generate the same basic function of being able to see something, even though they may vary in a lot of ways structurally. For instance, we saw the multiple emergence of what we call these crystalline lenses in the eyes of animals: some were made from co-opting calcite, others were made by co-opting heat shock proteins, still others were made by co-opting other novel proteins. All of them make these crystalline lenses, right? Or you could consider that independent but convergent evolution of similar structures in vertebrate eyes and spider eyes — and that can result in, again, the same basic capacity to see something. Of course, then we can talk about how the exact functions of seeing using these different structures or similar structures can vary. You know, acuity can vary, or wavelengths that the animal can sense can vary. But still, we think that the same basic capacity of some kind of sight is there for all of these animals. This is all to say that it makes it more complicated if we’re looking for something like consciousness. So the function we’re interested in is consciousness instead of vision. Now we’re saying that if there’s more than one origin point, we need to be looking for potentially divergent structures capable of producing that common basic function. And of course, that common basic function can have lots of variance and gradation. And we don’t even have a good grasp yet on human consciousness, so you can see how acknowledging the possibility of multiple independent origins would then make this all very challenging to figure out. But in any case, I think when you look at this from an evolutionary perspective, it’s important to consider who’s a close relative? Who are your common ancestors for that group? When you think these characteristics evolved, why do you think they evolved there? And then, if you’re somebody who takes crustaceans seriously, given that their close relation is the insects, you’re going to need to seriously consider the hexapods too. Andrés Jiménez Zorrilla on whether there’s something it’s like to be a shrimp Rob Wiblin: How do shrimp farmers feel about their shrimp? Do they naturally care about their wellbeing, or see them as moral patients that can suffer? Andrés Jiménez Zorrilla: That’s something that surprised us. When we did a survey in India, it’s a small sample, but we asked whether they felt that their shrimps could feel pain and stress, and 95% of them said yes. One of them actually had a very endearing answer saying that he spent more time with his animals than he did with his family and that, if his friends suffered, he said, “I also suffer.” It’s very interesting. On the other hand, I think it’s unsurprising, because these people spend a significant amount of time seeing the behaviour of the animals. They’re much less skewed than consumers, who never see them alive. I’m almost betting that you or your audience have very rarely ever seen an image of a shrimp that’s alive and swimming. Most of them will have just seen them in cocktails. And these farmers just see them all the time. They see them when they’re sick. They see them when they’re feeding, when they’re swimming about — so they really care about them. Rob Wiblin: I see. So they’re exposed and seeing shrimp all the time. It sounds like their behaviour is moderately complicated, that they’re doing interesting things that make them seem smart enough and reactive and responsive enough to circumstances that it’s very natural to feel that they can suffer or experience pleasure the same way that dogs or pigs do. Andrés Jiménez Zorrilla: Exactly. I couldn’t have put it better. It’s very difficult to see them in farms, because the water in which they’re raised typically has high turbidity — it’s very murky. But once you actually see them — sometimes in tanks and in trade conferences and things like that — when they’re fed, they swim, they catch their feed, they take it to a little corner where each of them can eat it in peace. It’s very rare to see them behaving. There’s good research ongoing to understand behavioural issues of shrimps. It’s underway. Rob Wiblin: I see. So they’re perhaps more like crabs or lobsters or octopus even than one might imagine, in terms of just how their behaviour looks. Andrés Jiménez Zorrilla: Exactly. That was the argument that the scientists at the London School of Economics made when they wrote this paper for the UK sentience bill recently, in November last year. They did a full review of the evidence of sentience of cephalopod mollusks and decapods, which are exactly the ones that you mentioned. What they found out is that, for those species that have been extensively researched, there is very good evidence that they’re sentient. What they say is the evidence in some other species is not as strong, but it’s only because they haven’t been researched for sentience purposes as much as other species — like, for example, crabs and octopuses, as you said. Rob Wiblin: What sort of criteria are they using? When they study a species to try to figure out whether it’s sentient, what sort of things are they looking at to try to reach an evaluation? Andrés Jiménez Zorrilla: That’s a good point, because it’s very difficult to have a smoking gun that tells you that an animal is sentient. So in this specific case, what they did is they looked at eight different indicators of sentience. Those included whether they had nociceptors, so the right body parts to detect noxious stimuli; whether they had protective behaviours, adaptive behaviours, if anaesthesia was applied to certain body parts — whether their reaction changed, which would indicate that it’s not a reflex. Things like that. Then they ranked whether the evidence was very high, high, to moderately low, et cetera, for each individual species. Then they came up with an overall assessment that all cephalopods and decapods should be protected by UK animal welfare law, and eventually they did. This was a report that was commissioned by the UK government to the London School of Economics. It was very independent research. Rob Wiblin: OK, so shrimp respond to injuries. They probably learn from negative experiences that they have. Did they respond to anaesthetic? I know that’s one of the tests that people sometimes use. Andrés Jiménez Zorrilla: There’s a paper that shows the responses that different decapods have to anaesthetics, as you said. With shrimps in particular, what they do is they pinch one of the antennas. They see how they behave: they flick their tails, they jump out of the water, et cetera. And then in the second stage, they apply anaesthetics and repeat the experiment, and the behaviour changes significantly. The time that they rub their little antenna is much, much lower. They probably swim normally, quicker, and things like that. Rob Wiblin: I see. They are tending to injuries, and they tend to them less when they’re given anaesthetic. As an aside, it’s remarkable to me that anaesthetics that we’ve presumably developed for humans also work on shrimp. They’re so far away in the phylogenetic tree of life, and yet so much of the basic machinery of feeling seems to be similar enough. Andrés Jiménez Zorrilla: That’s true. One of the arguments that some people made used to be that opioids don’t necessarily work the same in some of the animals. As you’ve said, not necessarily all of the anaesthetics need to work the same. But researchers have found anaesthetics that do apply and do have an effect on animals, and it changes behaviour. Rob Wiblin: An audience member was curious to know: do you feel viscerally motivated by the prospect of shrimp suffering, or is your interest and motivation somewhat more on the intellectual side? Andrés Jiménez Zorrilla: So until very recently, my response would’ve been completely on the intellectual side. It was not until I visited farms. I think most of us during our lifetimes will never visit a shrimp farm, or most people in the world will not visit a shrimp farm or see a shrimp being taken out of the water. But when my cofounder, our programme director in India, and myself went to see what is called “harvesting” — which is the moment in which the animals are scooped out of the water and eventually put in crates and things like that — that process made me also viscerally care about this issue. But it definitely came through the more intellectual part. Jonathan Birch on the cautionary tale of newborn pain Luisa Rodriguez: Let’s turn to another edge case of sentience in humans: foetuses. You start this section of the book with what you call “the cautionary tale of newborn pain.” Can you talk me through that case? Jonathan Birch: It’s another case I found unbelievable: in the 1980s, it was still apparently common to perform surgery on newborn babies without anaesthetic on both sides of the Atlantic. This led to appalling cases, and to public outcry, and to campaigns to change clinical practice. There was a public campaign led by someone called Jill Lawson, whose baby son had been operated on in this way and had died. And at the same time, evidence was being gathered to bear on the questions by some pretty courageous scientists, I would say. They got very heavily attacked for doing this work, but they knew evidence was needed to change clinical practice. And they showed that, if this protocol is done, there were massive stress responses in the baby, massive stress responses that reduce the chances of survival and lead to long-term developmental damage. So as soon as they looked for evidence, the evidence showed that this practice was completely indefensible and then the clinical practice was changed. So, in a way, people don’t need convincing anymore that we should take newborn human babies seriously as sentience candidates. But the tale is a useful cautionary tale, because it shows you how deep that overconfidence can run and how problematic it can be. It just underlines this point that overconfidence about sentience is everywhere and is dangerous. Luisa Rodriguez: Yeah, it really does. I’m sure that had I lived in a different time, I’d at least have been much more susceptible to this particular mistake. But from where I’m standing now, it’s impossible for me to imagine thinking that newborns don’t feel pain, and therefore you can do massively invasive surgery on them without anaesthetic. Jonathan Birch: It’s a hard one to believe, isn’t it? Of course, the consideration was sometimes made that anaesthesia has risks — and of course it does, but operating without anaesthesia also has risks. So there was real naivete about how the surgeons here were thinking about risk. And it’s what philosophers of science sometimes called the “epistemology of ignorance”: they were worried about the risks of anaesthesia, which is their job to worry about that, so they just neglected the risks on the other side. That’s the truly unbelievable thing. Luisa Rodriguez: So the clinical updates that have happened since: my sense is that now it is standard procedure to give newborns anaesthetic during surgeries, and that the benefits outweigh the risks. You argue that that wasn’t inevitable. What’s the case for that? Jonathan Birch: I think the public outcry also mattered. Clinical norms are very hard to shift. If it was really just these two people, Anand and Hickey, against the medical establishment, that’s not really how change happens. You know, we talk about theories of change sometimes — and “just get the evidence and take it to the establishment” is not a good theory of change. I think in this case, the fact that there was at the same time a powerful public campaign going on based on these horrible stories, that is why clinical practice got changed very quickly. Luisa Rodriguez: So there’s a lesson there. I mean, there’s really nothing to say, but that that’s horrific, and important to know it happened, because we may be doing it again. David Chalmers on why artificial consciousness is possible David Chalmers: Some people think that no silicon-based computational system could be conscious because biology is required. I’m inclined to reject views like that, and there’s nothing special about the biology here. One way to think about that is to think about cases of gradual uploading: you replace your neurons one at a time by silicon chips that play the same role. I think cases like this make it particularly hard to say that, if you say that the system at the other end is not conscious, then you have to say that consciousness either gradually fades out or during this process or it suddenly disappears during this process. I think it’s at least difficult to maintain either of those lines. You could take the line that maybe silicon will never even be able to simulate biological neurons very well, even in terms of its effects. Maybe there’s some special dynamic properties that biology has that silicon could never have. I think that would be very surprising, because it looks like all the laws of physics we know about right now are computational. Roger Penrose has entertained the idea that’s false. But if we assume that physics is computational, that one can in principle simulate the action of a physical system, then one ought to at least be able to create one of these gradual uploading processes and then someone who denies that the system on the other end could be conscious is going to have to say either it fades out in a really weird way during this process. You go through half consciousness, quarter consciousness, while your behaviour stays the same, or that it suddenly disappears at some point. You replace the magic neuron and it disappears. Those are arguments I gave years ago now for why I think a silicon duplicated device can be conscious in principle. Arden Koehler: I think I see why the sudden disappearance of consciousness in that scenario seems implausible. It’s like, “Well, what’s so special about that magic neuron?” But I don’t immediately see why the gradual fadeout of consciousness isn’t a reasonable possibility to entertain there? David Chalmers: How are you thinking the gradual fadeout would go? First, we’d lose visual consciousness, then we’d lose auditory consciousness? Or…? Arden Koehler: I don’t know exactly how it would go, but if we assume that consciousness can come in degrees, then why can’t it disappear in degrees? David Chalmers: Yeah, I guess I’m thinking that I just put some crude measure on a state of consciousness. Like the number of bits involved in your state of consciousness. One way of imagining it fading is somehow lowering in intensity and then suddenly the intensity goes to zero and it all disappears. That to me sounds like a version of sudden disappearance because the bits which still go from being a million bits to zero bits all at once. Strange in the way that sudden disappearance is strange. Maybe looking more continuous then somehow the number of bits in your consciousness has to gradually decrease. You go from a million bits to 100,000 to 10,000 to whatever. And how would this work? Maybe my visual field would gradually lose distinctions, will gradually become more coarse-grained, maybe bits of it would disappear. Maybe one modality would go and then another modality. But anyway, you’re going to have these weird intermediate states where you say the system is conscious, is saying it is fully conscious of all these things because its behaviour is the same, “I am visually conscious in full detail, I’m auditorily conscious.” In fact, their consciousness state is going to be a very, very pale reflection of the conscious state they’re talking about with very few, say bits, of consciousness. That situation is the one that strikes me as especially strange. A conscious being that appears to be fully rational and believes it’s in this fully conscious state, but in fact, it’s in a very, very limited conscious state. If you’re an illusionist, you might think this kind of thing happens to you all the time. Rob Wiblin: I think this is totally wrong. It seems like you could have a view where there’s information processing going on in the transmission between the neurons and that’s what’s generating the behaviour. But then there’s some other secret sauce that’s happening in the brain that we don’t understand and that we would not then replicate on the silicon chips. As you go replacing each neuron and each synapse with the machine version of it, the information processing continues as before and the behaviour remains the same. But you’ve lost the part that was generating the consciousness; you haven’t engineered that into the computer components, and so just gradually the consciousness disappears. David Chalmers: I can imagine this is at least conceivable, but then what are you going to say about the intermediate cases? There will have to be cases where the being is conscious and just massively wrong about its own consciousness. It says you’re not having experiences of red and green and orange right now. The fact is that it’s having a uniform gray visual field or thing like that. Rob Wiblin: It seems possible, right? Arden Koehler: I guess I also don’t find it as implausible as you seem to, Dave, that we could be wrong about our conscious experience or how much conscious experience we’re having in this gradual uploading example. David Chalmers: That’s fair, and there certainly are many cases where people are very wrong about their own conscious experiences. Certainly, there are all kinds of pathologies where there’s blindness denial — where people say they’re having all kinds of visual experiences when it appears that they’re blind and they’re not having them. Maybe it could be like this. This is strange because functionally the system doesn’t seem to have any pathologies. Anyway, I do allow that this is conceivable and I certainly can’t prove that it couldn’t happen. The more open you are to beings being very, very wrong about their consciousness, maybe you’ll be more open to this case. Here’s one thing I’ll say, at the very least: if this actually happens, and we go through it and our behavior is the same throughout, then we have beings whose heads are first a quarter silicon, then a half silicon as well. They say, “Everything is fine, everything is fine, my conscious experience is exactly the way it was.” They’re telling us this, they’re talking to us. They update every week with a bit more silicon and we keep talking to them. We are going to be very nearly completely convinced that they are conscious throughout. It’s going to become impossible to deny it. So at least as a matter of sociology, I think this view is likely to become the obvious-seeming view. Holden Karnofsky on how we’ll see digital people as… people Rob Wiblin: Let’s talk about these digital people. What do you mean by digital people? Holden Karnofsky: So the basic idea of a digital person is like a digital simulation of a person. It’s really like if you just take one of these video games, like The Sims, or… I use the example of a football game because I was able to get these different pictures of this football player, Jerry Rice, because every year they put out a new Madden video game. So, Jerry Rice looks a little more realistic every year. You have these video game simulations of people, and if you just imagine it getting more and more realistic until you have a perfect simulation. So imagine a video game that has a character called Holden, and just does everything exactly how Holden would in response to whatever happens. That’s it. That’s what a digital person is. So it’s a fairly simple idea. In some ways it’s a very far-out extrapolation of stuff we’re already doing, which is we’re already simulating these characters. Rob Wiblin: I guess that’d be one way to look at it. The way I’ve usually heard it discussed or introduced is the idea that, well, we have these brains and they’re doing calculations, and couldn’t we eventually figure out how to basically do all of the same calculations that the brain is doing in a simulation of the brain moving around? Holden Karnofsky: Yeah, exactly. You would have a simulated brain in a simulated environment. Absolutely, that’s another way to think of it. Rob Wiblin: This is a fairly out-there technology, the idea that we would be able to reproduce a full human being, or at least the most important parts of a human being, running on a server. Why think that’s likely to be possible? Holden Karnofsky: I mean, I think it’s similar to what I said before. We have this existence proof. We have these brains. There’s lots of them, and all we’re trying to do is build a computer program that can process information just how a brain would. A really expensive and dumb way of doing it would be to just simulate the brain in all its detail, just simulate everything that’s going on in the brain. But there may just be smarter and easier ways to do it, where you capture the level of abstraction that matters. So maybe it doesn’t matter what every single molecule in the brain is doing. Maybe a lot of that stuff is random, and what really is going on that’s interesting, or important, or doing computational work in the brain is maybe the neurons firing and some other stuff, and you could simulate that. But basically, there’s this process going on. It’s going on in a pretty small physical space. We have tonnes of examples of it. We can literally study animal brains. We do. I mean, neuroscientists just pick them apart and study them and try to see what’s going on inside them. And so, I’m not saying we’re close to being able to do this, but when I try to think about why would it be impossible, why would it be impossible to build an artefact, to build a digital artefact or a computer that’s processing information just how a brain would, and I guess I just come up empty. But I can’t prove that this is possible. But yeah, the basic argument is just, it’s here, it’s all around us. Why wouldn’t we be able to simulate it at some point in time? Rob Wiblin: Are you envisaging these digital people as being conscious like you and me, or is it more like an automaton situation? Holden Karnofsky: One of the things that’s come up is when I describe this idea of a world of digital people, a lot of people have the intuition that even if digital people were able to act just like real people, they wouldn’t count morally the same way. They wouldn’t have feelings. They wouldn’t have experiences. They wouldn’t be conscious. We shouldn’t care about them. And that’s an intuition that I disagree with. It’s not a huge focus of the series, but I do write about it. Basically if you dig all the way into philosophy of mind and think about what consciousness is, this is something we’re all very confused about. No one has the answer to that. But I think in general, there isn’t a great reason to think that whatever consciousness is, it crucially relies on being made out of neurons instead of being made out of microchips or whatever. And one way of thinking about this is, I think I’m conscious. Why do I think that? Is the fact that I think I’m conscious, is that connected to the actual truth of me being conscious? Because the thing that makes me think I’m conscious has nothing to do with whether my brain is made out of neurons. If you made a digital copy of me and you said, “Hey, Holden, are you conscious?” That thing would say, “Yes, of course I am,” for the same exact reason I’m doing it: it would be processing all the same information, it’d be considering all the same evidence, and it would say yes. There’s this intuition that whatever consciousness is, if we believe it’s what’s causing us to think we’re conscious, then it seems like it’s something about the software our brain is running, or the algorithm it’s doing, or the information it’s processing. It’s not something about the material the brain is made of. Because if you change that material, you wouldn’t get different answers. You wouldn’t get different beliefs. That’s the intuition. I’m not going to go into it a tonne more than that. There’s a thought experiment that’s interesting that I got from David Chalmers, where you imagine that if you took your brain and you just replaced one neuron with a digital signal transmitter that just fired in all the same exact ways, you wouldn’t notice anything changing. You couldn’t notice anything changing, because your brain would be doing all the same things, and you’d be reaching all the same conclusions. You’d be having all the same thoughts. Now, if you replaced another one, you wouldn’t notice anything, and if you replaced them all, you wouldn’t notice anything. Anyway, so I think there’s some arguments out there. But I think it is the better bet that if we had digital people that were acting just like us, and the digital brains were doing the same thing as our brains, that we should care about them. And we should think of them as people. And we probably would. Even if they weren’t conscious — Rob Wiblin: Because they would act like it. Holden Karnofsky: Yeah, well, we’d be friends with them. Rob Wiblin: They would complain the same way. Holden Karnofsky: We’d talk to them and we would relate to them. There are people I’ve never met, and they would just be like any other people I’ve never met, but I could have video calls with them and phone calls with them. And so, we probably will and should care about what happens to them. And even if we don’t, it only changes some of the conclusions. But I basically think that digital people would be people too. Rob Wiblin: I mean, the argument that jumps to mind for me is if you’re saying, “Well, to be people, to be conscious, to have value, it has to be run on meat. It has to be run on these cells with these electrical charges going back and forth.” It’d be like, “Did evolution just happen to stumble on the one material that could do this? Evolution presumably didn’t choose to use this particular design because you’d be conscious. So why would there be this coincidence that we have the one material out of all of the different materials that can do these calculations that produces moral value?” Holden Karnofsky: That’s an interesting way of thinking of it. If I were to play devil’s advocate, I would be like, “Well, maybe every material has its own kind of consciousness, and we only care about the kind that’s like our kind,” or something. But then that would be an interesting question why we only care about our kind. Jeff Sebo on grappling with our biases and ignorance when thinking about sentience Luisa Rodriguez: Let’s actually talk about another paper of yours, “The rebugnant conclusion.” I love that title, by the way. Jeff Sebo: Thank you! Luisa Rodriguez: In the paper you basically ask: Suppose that we determine that large animals like humans have more welfare on average, but that small animals like insects have more welfare in total. What follows for ethics and politics? And the paper focuses on small animals like nematodes, but the same question is relevant to AI systems that might end up being super numerous — perhaps because they’re used all over the economy — but that might also have some non-negligible chance of experiencing pain and pleasure. So let’s start with the case that you actually focus on in the paper, which is small animals. How should we think about this case? Jeff Sebo: I think we can start really at the end of the last exchange about ways of striking a balance if we worry about the harms of false positives and false negatives. One thing that you can note is that, even if I include insects and AI systems and other types of beings in my moral circle, even if I give them moral consideration, I might still be able to prioritise beings like me for different reasons. One of them is that I might have a higher capacity for welfare than an insect or an AI system. I have a more complex brain and a longer lifespan than an insect, so I can experience more happiness and suffering at any given time, as well as over time. And humans in general, I might think, are more likely to be sentient and morally significant — given the evidence available to me — than insects, AI systems, other beings like that. So I might think to myself that if a house is burning down and I can save either a human or an ant, but not both, then I can justifiably save the human — both because the human is more likely to matter, and because the human is likely to matter more. And those are perfectly valid ways of breaking a tie. That might give us some peace of mind when we countenance the possibility of including these very different, very numerous beings in the moral circle. But then you have to consider how large these populations actually are — and this is where we get to the problem that this paper addresses, which is a problem in population ethics. Luisa Rodriguez: Right. And population ethics is the philosophical study of the ethical problems that come up when our actions affect how many people are born in the future, and who exactly is born. But yeah, my understanding is that we don’t actually know how many of these small animals there are — ants and nematodes and maybe even microbes — but that it’s at least plausible that there are so many of them that even if they have very less significant kinds of suffering and pleasure relative to humans, and even if we only put some small chance on them even having those at all, their interests still just swamp humans’. And this argument just does sound plausible to me, and it also fills me with dread and fear. What is your experience of it? Jeff Sebo: Well, I certainly have the same experience as I think most humans do. And the reason I gave that paper the title “The rebugnant conclusion” is that this is based on a famous book by the philosopher Derek Parfit called Reasons and Persons, part four of which addresses population ethics. In that part of that book, Derek Parfit discusses what he calls the repugnant conclusion. I can say briefly what that is and why that has, for the past several decades, filled many people with dread. So the repugnant conclusion results from the following observations. If you could bring about one future where the world contained 100 people and everyone experienced 100,000 units of happiness, or you could bring about another world with twice the number of people, 200 people, but everyone experiences one fewer unit of happiness — 99,999 — which world is better? Well, many of us have the intuition that the second world is better; I should bring about that second population. Everybody might experience one unit of happiness less per person, but since there are twice as many people, there is nearly twice as much happiness overall, and everyone is still really happy. And so, all things considered, I should bring about that population. But then you can imagine another population, once again twice as big, and once again with a bit less happiness per person. Then another one twice as big, a bit less happiness per person. And so on and so on and so on — until you reach a point where you are imagining a world or a solar system or a galaxy that contains a vast number of individuals, each of whom has a life only barely worth living at all. And your reasoning would commit you to the idea that that is the best possible world, the one that you should most want to bring about. Parfit thought the idea that we would favour that world with a much larger population, where everyone has a life barely worth living at all, over a world with a still significant population, where everyone has lives very much worth living, he found that repugnant. And he spent much of the rest of his career trying and failing to find a better way to see the value of populations that could avoid that result. Luisa Rodriguez: I guess in the case of insects, there’s also this weird thing where, unlike humans eating potatoes and not particularly enjoying their monotonous lives, we might think that being a spider and making a web sounds pretty boring, but we actually just really do not know. In many ways, they’re so different from us that we should have much lower probability that they’re not enjoying or enjoying that than we do of humans in this repugnant conclusion scenario. How do you factor that in? Jeff Sebo: I do share the intuition that a very large insect population is not better off in the aggregate than a much smaller human population or elephant population. But for some of the reasons that you just mentioned and other reasons, I am a little bit sceptical of that intuition. We have a lot of bias here and we also have a lot of ignorance here. We have speciesism; we naturally prefer beings and relate to beings when they look like us — when they have large eyes and large heads, and furry skin instead of scaly skin, and four limbs instead of six or eight limbs, and are roughly the same size as us instead of much smaller, and reproduce by having one or two or three or four children instead of thousands or more. So already we have a lot of bias in those ways. We also have scope insensitivity — we tend not to be sensitive to the difference that very large numbers can make — and we have a lot of self-interest. We recognise that if we were to accept the moral significance of small animals like insects, and if we were to accept that larger populations can be better off than smaller populations overall, then we might face a future where these nonhuman populations carry a lot of weight, and we carry less weight in comparison. And I think some of us find that idea so unthinkable that we search for ways to avoid thinking it, and we search for theoretical frameworks that would not have that implication. And it might be that we should take those theoretical frameworks seriously and consider avoiding that implication, but I least want to be sceptical of a kind of knee-jerk impulse in that direction. Luisa Rodriguez: Yeah, I am finding that very persuasive. Even as you’re saying it, I’m trying to think my way out of describing what I’m experiencing as just a bunch of biases — and that in itself is the biases in action. It’s me being like, no, I really, really, really want to confirm that people like me, and me, get to have… I don’t know. It’s not that we don’t have priority — we obviously have some reason to consider ourselves a priority — but I want it to be like, end of discussion. I want decisive reasons to give us the top spot. And that instinct is so strong that that in itself is making me a bit queasy about my own motivations. Jeff Sebo: Yeah, I agree with all of that. I do think that we have some reason to prioritise ourselves, and that includes our welfare capacities and our knowledge about ourselves. It also includes more relational and pragmatic considerations. So we will, at least in the near term, I think have a fairly decisive reason to prioritise ourselves to some extent in some contexts. But yeah, I agree. I think that there is not a knock-down decisive reason why humanity should always necessarily take priority over all other nonhuman populations — and that includes very large populations of very small nonhumans, like insects, or very small populations of very large nonhumans. We could imagine some kind of super being that has a much more complex brain and much longer lifespan than us. So we could find our moral significance and moral priority being questioned from both directions. And I think that it will be important to ask these questions with a lot of thought and care and to take our time in asking them. But I do start from the place of finding it implausible that it would miraculously be the case that this kind of population happens to be the best one: that a moderately large population of moderately large beings like humans happens to be the magic recipe, and we matter more than all populations in either direction. That strikes me as implausible. Bob Fischer on how to think about the moral weight of a chicken Luisa Rodriguez: Just to make sure I understand, the thing is saying that the capacity of welfare or suffering of a chicken in a given instant is about a third of the capacity for the kind of pain and pleasure a human could experience in a given instant. Is that it? Bob Fischer: That’s the way to think about it. And that might sound very counterintuitive, and I understand that. I think there are a couple of things we can say to help get us in the right frame of mind for thinking about these results. One is to think about it first like a biologist. If you think that humans’ pain is orders of magnitude worse than the pain of a chicken, you’ve got to point to some feature of human brains that’s going to explain why that would be the case. And I think for a lot of folks, they have a kind of simple picture — where they say more neurons equals more compute equals orders of magnitude difference in performance, or something like that. And biologists are not going to think that way. They’re going to say, look, neurons produce certain functions, and the number of neurons isn’t necessarily that important to the function: you might achieve the exact same function using many more or many fewer neurons. So that’s just not the really interesting, relevant thing. So that’s the first step: just to try to think more like a biologist who’s focused on functional capacities. The second thing to say is just that you’ve got to remember what hedonism says. What’s going on here is we’re assuming that welfare is about just this one narrow thing: the intensities of pleasures and pains. You might not think that’s true; you might think welfare is about whether I know important facts about the world or whatever else, right? But that’s not what I’m assessing; I’m just looking at this question of how intense is the pain. And you might also point out, quite rightly, “But look, my cognitive life is richer. I have a more diverse range of negatively valenced states.” And I’m going to say that I don’t care about the range; I care about the intensity, right? That’s what hedonism says: that what matters is how intense the pains are. So yeah, “I’m very disappointed because…” — choose unhappy event of your preference — “…my favourite team lost,” whatever the case may be. And from the perspective of hedonism, what matters about that is just how sad did it make me? Not the content of the experience, but just the amount of negatively valenced state that I’m experiencing, or rather the intensity of the negatively valenced state that I’m experiencing. So I think people often implicitly confuse variety in the range of valenced states with intensity. Luisa Rodriguez: Yeah, that’s definitely something I do. For sure there is a part of me that thinks that the thing that matters a lot here is that I can fall in love in a particularly meaningful and big way; I can have friendships lasting 50 years that involve really deep and meaningful conversations. And that even if a chicken has meaningful relationships with other chickens, they’re not as complex and varied as the relationships I have with people in my life. On the other hand, a big part of me puts a bunch of weight, when I really think about it, on just like, no, what matters is the intensity. If a chicken feels more sad about her wing being broken than I feel about losing a friend, then so be it. We should make sure that their wings aren’t broken before we should make sure that whatever threat that could mean I lose my friend [is prevented]. And I guess lots of listeners will have their own kind of internal turmoil about this, about what welfare even is. But for now, I guess if we’re just taking this assumption, which is that what matters is the intensity, your finding is that something like averting the suffering of three chickens for an hour is similarly important to averting the suffering of one person for an hour. And that feels uncomfortable to me. Can you talk me through that discomfort? Bob Fischer: Sure. So the first thing to say is: you’re not alone. I don’t feel totally comfortable either. And we have to ask ourselves what our most serious moral commitments are when we’re approaching this question. So you’re not going to avoid really uncomfortable, challenging questions when we try to think about moral weights — just not going to go away. But here are a few things to say. One is: is there any number that you wouldn’t be uncomfortable with? Because notice that if you’re committed to this idea of doing conversions, eventually it’s going to just work out that you’ve got to say there is some number of hours of chicken suffering that is more important than helping a human. And I think actually for a lot of people, they don’t really think that there is any conversion at all, right? If I had said it was 300, would you really have felt that much better? You might have felt a little bit better; I’m not saying you wouldn’t felt it at all. Sure, it’s a difference, but you might still say, when you really think about it: “Three hundred hours? Would I put somebody through that for… chickens?” And then you might just have the same level of discomfort, or something close to it. So I think to some degree we have to remember that the tradeoffs that we’re talking about come from background theoretical commitments that have nothing to do with our specific welfare range estimates: it comes from the fact that we’re trying to do the most good. We think that means making comparisons across species, and we’re committed to this kind of maximising ethic that says, yeah, there is some tradeoff rate, and you’ve got to find it. So that’s the first thing to say about the discomfort. Before I say anything else, what do you think about that? Luisa Rodriguez: Yes, some of that definitely worked for me. I think the thing that lands most is if I think about chickens on a railroad track, and there’s a trolley coming, and there’s a human on the other side, it is pretty impossible for me to imagine getting to the point where I’m ever super comfortable being like, “I’m going to let it hit the human, who I could have conversations with, who has a family I might know, who I could give a hug to, and who has a job…” These are all the things that kind of run through my head as I’m deciding whether to pull a lever to decide who gets hit by this trolley. And so, fair enough that that is something I have to grapple with, regardless of exactly what these numbers are. Bob Fischer: And just to tag on to that, think about it when you put someone you really care about on the track. So I think about this with my children, and say, look, it might well be the case that there’s almost no number of other humans I would choose to spare, given the choice between killing my own children and them. But that’s not because I think they actually matter less in some objective sense. Like when I’m trying to do the impartial good, I would never say, “Oh yes, my children are utility monsters: they have infinite worth and everybody else has just some tiny portion of that.” So when we recognise that our moral judgements are so detached from our judgements of value, that also can help us think about why these welfare ranges might not be quite so crazy. Luisa Rodriguez: Yeah. Was there anything else that helps you with the discomfort? Bob Fischer: I think the thing that helps me to some degree is to say, look, we’re doing our best here under moral uncertainty. I think you should update in the direction of animals based on this kind of work if you’ve never taken animals particularly seriously before. But ethics is hard. There are lots of big questions to ask. I don’t know if hedonism is true. I mean, there are good arguments for it; there are good arguments for all the assumptions that go into the project. But yeah, I’m uncertain at every step, and some kind of higher-level caution about the entire venture is appropriate. And if you look at the way people actually allocate their dollars, they often do spread their bets in precisely this way. Even if they’re really in on animals, they’re still giving some money to AMF. And that makes sense, because we want to make sure that we end up doing some good in the world, and that’s a way of doing that. Luisa Rodriguez: I guess I’m curious if there’s anything you learned, like a narrative or story that you have that makes this feel more plausible to you? Anything particular about chickens or just about philosophy? You’ve already said some things, but what story do you have in your head that makes you feel comfortable being like, “Yes, I actually want to use these moral weights when deciding how to allocate resources”? Bob Fischer: There are two things that I want to say about that. One is I really worry about my own deep biases, and part of the reason that I’m willing to be part of the EA project is because I think that, at its best, it’s an attempt to say, “Yeah, my gut’s wrong. I shouldn’t trust it. I should take the math more seriously. I should try to put numbers on things and calculate. And when I’m uncomfortable with the results, I’m typically the problem, and not the process that I used.” So that’s one thing. It’s a check on my own tendency to discount animals, even as someone who spends most of their life working on animals. So I think that’s one piece. The other thing is just to spend time thinking about the kinds of things animals can do and what their lives are like. Just how hard a chicken will work to get to a nest box before she lays an egg, the amount of labour she’s willing to go through to do that, to think about how important that is to her. And to realise that we can quantify that, and see how much they care, or to see that they get stressed out when fellow chickens are threatened and that they seem to have some sympathy for conspecifics. Those kinds of things make me say there is something in there that is recognisable to me as another individual, with desires and preferences and a vantage point on the world, who wants things to go a certain way and is frustrated and upset when they don’t. And recognising the individuality, the perspective of nonhuman animals, for me, really challenges my tendency to not take them as seriously as I think I ought to, all things considered. Cameron Meyer Shorb on the range of suffering in wild animals Luisa Rodriguez: Fundamentally, why do you worry about suffering among wild animals? What kinds of things make you think that they might be suffering a lot in particular? Cameron Meyer Shorb: I’m wildly uncertain about what the nature of wild animals’ lives are like. But I got into this field because I changed my mind about the possibilities here. I used to just assume that animals living in the wild were perfectly in balance, and living totally fulfilled lives, and weren’t bothered by any of the stresses of modernity like I am — and that the best thing humans could possibly do for them is just leave them alone there. But the more I learned about it and thought about it, the more I realised that there’s actually lots of reasons to think that wild animals might not be living great lives, at least many of them. For example, they often have to struggle to get enough food. They often need to struggle to protect themselves from extreme weather. There are some kinds of things where they have no protection at all: if a flood or a wildfire comes, that’s just the end of it for them. There’s also all sorts of diseases or parasites. They have no healthcare. They also have no state to protect them from violence, either from other species or even members of their own species. So the kinds of conditions we’re talking about here, when humans live in those conditions, we would call that poverty. And we wouldn’t tolerate that. We would say that those are problems we need to solve; those are people who deserve better lives. And if we have medicine, we should help give them access to medicine. If there’s ways to give them more stable access to food, that’s something that would improve their lives. That’s the kind of approach that I now think we should consider taking when we think about wild animals: taking seriously the idea that they might be struggling even in their natural habitats, and they might suffer even from naturally occurring harms. And we should try to figure out to what extent that’s true, and if it’s possible to do anything about that. Luisa Rodriguez: Yeah, I want to come back to some of those, because I feel like even though I’m a little bit familiar with this problem, I still have a super limited imagination for the kinds of things that wild animals — and obviously there’s such a wide range across different species — might be going through, aside from the really obvious ones, like being eaten or something. So putting a pin in that, what is the scale of the problem? How many individuals alive right now are wild animals? Cameron Meyer Shorb: The scale is huge. It’s bigger than I can count. I’ve been humbled by learning about this, but I do think the scale is such an important part of understanding the problem. Just in the broadest possible strokes, based on the rough numbers we know, it looks like something like 99% of all sentient minds alive on Earth today are wild animals. So if you are a human or a farmed animal, that is an incredibly rare exception to the rule, which is: things that can feel live in the wild. You’re more of a rounding error than anything. Which is not to say that human and farmed animal experiences aren’t important; it’s just to say there is a lot more going on. And a truly impartial view of ethics would have us believe that ethics is mostly about wild animals… and also there are these interesting subfields that are related to some primates and farmed animals. Luisa Rodriguez: OK, I’m interested in breaking down the numbers a bit more. I feel like it seems at least kind of important to know, are we mostly thinking about zebras, or are we mostly thinking about fish, ants…? What to think about feels like it could have helpful effects in helping me figure out, what are we talking about? Cameron Meyer Shorb: That’s a great question to ask, because I think that the images that normally come to mind are not the most representative of wild animals. Most minds are wild, weird, and wet. They’re just not humans or human-like things. To try to get a sense of scale, I’ll suggest a visualisation. Let’s imagine for the sake of this exercise that we’re going to put a dot down of equal size for any individual that’s alive. So one dot for a human, one dot for a squirrel — and you can debate later how you want to make tradeoffs across species — but just for starters, to get a sense of the raw numbers. Now, let’s make these dots small enough so that we can fit all 8.2 billion humans onto the face of a quarter or a euro — so something a little smaller than one square inch. If we’re keeping that scale, then the 88 billion wild mammals would take up an area about the size of a credit card or a post-it note. And then when we move on to birds — and I should say these estimates are all very rough, and the bigger the populations, the wider the error bars are — but for birds, let’s say there are about 200 billion living in the wild. That would be something about the size of a standard envelope. And then for reptiles and amphibians, each of those numbers somewhere around one trillion individuals, two trillion put together. So a trillion would be a standard sheet of paper. I think this is a good place to pause and just think about how far we’ve come: from a quarter to an envelope, which is way bigger than a quarter, to a couple sheets of paper — compared to a quarter that contains all of human experience and 8.2 billion lives. There’s that, times many, many more, if you’re trying to encompass humans and mammals and birds and reptiles and amphibians. Then the numbers get even more mind-boggling when we move on to fish. There’s something like 10 trillion fish in the world. So 10 trillion fish would be something like the size of a medium-sized desk — the Linnmon from Ikea, if you will — or a large bath mat, or a couple of pillowcases maybe. That’s what the whole fish population would look like, relative to the human population fitting on a quarter. And the numbers get really… I don’t know what “to boggle” means literally, but I think it is something like what is happening to my mind. I think it is mind-boggling to try to imagine the number of plausibly sentient invertebrates. The only at all close-to-useful number I found here is an estimate of the number of terrestrial arthropods — that would be animals with hard exoskeletons, like insects and arachnids and crustaceans. So for those that live on land, estimates are that their population is somewhere around 100,000 trillion. If 8.2 billion humans fit on the face of a quarter, 100,000 trillion would need to be something the size of a city block or a FIFA regulation-size soccer field. Luisa Rodriguez: That’s insane! Cameron Meyer Shorb: Imagine standing at any point in a soccer field and looking at a quarter and then looking around at the rest of the field. It really changes your perspective on what life on Earth is like, who’s really living here. And it’s hard to know whether many arthropods are sentient. I think there’s decent questions and considerations on either side. But one of the things that I think is important to consider is the expected value. So even if there’s just a 10% chance that they are, 10% of a soccer field is still way bigger than a quarter. Luisa Rodriguez: Yeah. This makes me really happy that some of our most recent episodes… One is with Meghan Barrett, one of my favourite episodes of all time, on invertebrate sentience. And really, it took me well above 10% probability or credence that invertebrates are sentient. Another one is on fish with Sébastien Moro, just infinite numbers of bewildering facts about fish. For me, invertebrates and fish both make up tremendous numbers of individuals, as you’ve just said, and just are clearly at least very plausible sentience candidates. For the case of fish, it seems hard to even debate for me. So given that these are the animals making up most of the wild nature we’re talking about, for anyone who’s interested, I can recommend those episodes. Sébastien Moro on whether fish are conscious or sentient Luisa Rodriguez: This is fascinating and mind-blowing, purely from the perspective of “fish are incredibly cool.” But do these feel like they tell you anything about whether fish are experiencing these things in some conscious way or affective way? Sébastien Moro: As I said earlier, we have to split consciousness and sentience, which are not the same thing. It’s very hard actually talking about real consciousness, like high-order consciousness, like humans: we don’t know how to assess it correctly, even in humans. We don’t have proper assessing tools. So today we’re trying to build new ways to assess consciousness and sentience and split them properly. I understood that you’ve interviewed Jonathan Birch about that. He’s a pioneer in it. He’s a very important person on it. There is a very good book, especially on fish, a study which is named, “What is it like to be a bass? Red herrings, fish pain, and the study of animal sentience.” It’s a publication from 2022. It’s really interesting because it’s coming back on all of this, and the famous study that was talking about modification of the mirror test we were talking about. Many of the studies that were done that we’ve talked about, they aren’t made to assess sentience. Luisa Rodriguez: From your perspective, you know so much of what there is to know about fish that we have studied so far, so you’ve got all of this wealth of knowledge: what is it that feels most compelling to you, that makes you feel like you’ve got really high confidence that fish are experiencing things? Sébastien Moro: Most of the papers we have are going in this way, and very few — very, very few — are going the other way. So what makes me so certain? I’m really talking about my personal opinion here. I tend to think that emotions… What are emotions? What are emotions used for? They are putting a gloss on what is around us: “This stuff has a positive gloss; I need more of this. This one has a negative gloss.” And we know that emotions are very, very closely linked to learning. So emotions are something that attract or repel. And it seems pretty obvious that it must have appeared very, very early in evolution, because this is how we work. This is maybe one of the biggest differences with algorithms. Algorithms are following closed loops. And this comes up when animals are more driven by emotions, by value of things, which is made by a kind of limbic system that says, “This is good, this is bad. You want more of this, you want less of that.” And I don’t understand why other animals couldn’t have had that. And another thing is, I’m reading a lot about bees, so I know very well the corpus of knowledge on bees at the moment. And we’re starting to have the same results in bees. So I’m not 100% certain now that bees could be sentient, but the biggest leader of bees research today, Lars Chittka, has said on Twitter that bees are sentient for him. And the results we have are going in this way. So they have a one-million-neuron brain, and the brains of fish are much, much bigger. And when we split from insects, brains were not existing either. So it’s just a convergent evolution. Fish have complicated lives; they have social lives, very social lives. I already introduced this with cleaner wrasses: their lives are very complicated. They have challenges that they have to overcome that are as complex as what we find in mammals and birds, maybe more sometimes. So them having no sentience, when we recognise sentience in birds and mammals? It’s either you refuse it for everyone, or you accept it for everyone at the moment. Not for everyone, because animals with a brain or central nervous system, today we think at least there should be a kind of global network; everything should be put in common to make a unified vision of you and this kind of thing. But consciousness probably has many degrees; sentience has pretty much many degrees — but not degrees on a ladder, degrees more on a circle. But that would sound just weird actually, that they wouldn’t be sentient. Luisa Rodriguez: Yeah, it would just be really surprising to you. David Chalmers on when to start worrying about artificial consciousness Arden Koehler: So you said elsewhere that if more fully autonomous artificial intelligence comes around, then we might have to start worrying about it being conscious, and therefore presumably worthy of moral concern. But you don’t think we have to worry about it too much before then. I’m just wondering if you can say a bit about why, and whether you think it’s possible that programs or computers could become gradually more and more conscious, and whether that process might start before they are fully autonomous? David Chalmers: Yeah, that’s an interesting point. I guess one would expect to get to conscious AI well before we get human-level artificial general intelligence, simply because we’ve got a pretty good reason to believe there are many conscious creatures whose degree of intelligence falls well short of human-level artificial general intelligence. So if fish are conscious, for example, you might think if an AI gets to sophistication and information processing and whatever the relevant factors are to the degree present in fish, then that should be enough. And it does open up the question as to whether any existing AI systems may actually be conscious. I think the consensus view is that they’re not. But the more liberal you are about descriptions of consciousness, the more we should take seriously the chance that they are. There is this website out there called People for the Ethical Treatment of Reinforcement Learners that I quite like. The idea is that every time you give a reinforcement learning network its reward signal, then it may be experiencing pleasure or correspondingly suffering, depending on the valence of the signal. As someone who’s committed to taking panpsychism seriously, I think I should at least take that possibility seriously. I don’t know where our current deep learning networks fall on the scale of organic intelligence. Maybe they’re at least as sophisticated as worms — like C. elegans, with 300 neurons. I take seriously the possibility that those are conscious. So I guess I do take seriously the possibility that AI consciousness could come along well before human-level AGI, and that it may exist already. Though then the question is, I suppose, how sophisticated the state of consciousness is. If it’s about as sophisticated as, say, the consciousness of a worm, I think most of us are inclined to think that brings along, say, some moral status with it, but it doesn’t give it enormous weight in the scheme of conscious creatures compared to the weight we give humans and mammals and so on. So I guess then the question would be whether current AIs get a truly sophisticated moral status, but I guess I should be open to them at least getting some relatively small moral status of the kind that, say, worms have. Rob Wiblin: Maybe this is getting outside your area of expertise, but with current ML systems, how would we have any sense of whether the affective states are positive or negative? It seems like once you have a reinforcement learner, I guess on average, does it get zero reinforcement because it just has an equal balance of positive and negative reinforcements? And is there some way that you could just scale all of them up to be more or less positive? Or does that even mean anything? Like you just increase all the numbers by 100. How would that help? It raises this issue of the arbitrariness of the zero point on this kind of scale of goodness of the states. David Chalmers: Yeah. This is getting to issues about value and morality that do go beyond my expertise to some extent. We’ve got absolutely no way right now to tell exactly what reinforcement learning systems might be experiencing, if anything. But if you were inclined to think they’re experiencing something and that they’re experiencing something with valence, I suppose then they’d be having a mix of positively valenced reinforcement and negatively valenced reinforcement — therefore, a mix of very simple precursor of, say, pleasure and of suffering, proto-pleasure and proto-suffering. Then a lot’s going to depend on your ethical theory. If you’re feeling pleasure half the time but suffering half the time, is that net good? Is that net bad? I don’t know. If you ask me, I think that’s net bad, because all that suffering tends to outweigh the pleasure, but maybe there’s weights on the scale. At this point though, I should say that it’s by no means obvious to me that pleasure and suffering, that is, that valenced states of consciousness, are the ones that are relevant to moral status. I know people quite often have this issue. I’m inclined to think that consciousness may ground moral status in some cases quite independently of its valence. Even beings with unvalenced states of consciousness could still have moral status. Robert Long on how we might stumble into causing AI systems enormous suffering Robert Long: So you can imagine that a robot has been created by a company or by some researchers. And as it happens, it registers damage to its body and processes it in the way that, as it turns out, is relevant to having an experience of unpleasant pain. And maybe we don’t realise that, because we don’t have good theories of what’s going on in the robot or what it takes to feel pain. In that case, you can imagine that thing having a bad time because we don’t realise it. You could also imagine this thing being rolled out and now we’re economically dependent on systems like this. And now we have an incentive not to care and not to think too hard about whether it might be having a bad time. So I mean, that seems like something that could happen. Luisa Rodriguez: Yeah, and that could happen because there’s some reason why it’s helpful to have the robot recognise that it’s sustained damage. It can be like, “Help, I’m broken. I need someone to fix my part.” So that’s something that you can imagine might get programmed in. And then, it is just kind of wild to me that we don’t understand what the robot might be experiencing well enough to know that that thing is pain. But in theory, that’s possible, just that it is that black-boxy to us. Robert Long: Yeah. It might be a little bit less likely with a robot. But now you can imagine more abstract or alien ways of feeling bad. So I focus on pain because it’s a very straightforward way of feeling bad. A disembodied system like GPT obviously can’t feel ankle pain. Or almost certainly. That’d be really weird. It doesn’t have an ankle. Why would it have computations that represent its ankle is feeling bad? But you can imagine maybe some strange form of valenced experience that develops inside some system like this that registers some kind of displeasure or pleasure, something like that. Luisa Rodriguez: Right, right. Something like, you guessed the wrong set of words to come next and that was bad. And the user isn’t happy with the string of words you came up with. And then that feels something like pain. Robert Long: Exactly. And I will note that I don’t think that getting negative feedback is going to be enough for that bad feeling, fortunately. But maybe some combination of that and some way it’s ended up representing it inside itself ends up like that. And then we have something where it’s hard for us to map its internals to what we care about. We maybe have various incentives not to look too hard at that question. We have incentives not to let it speak freely about if it thinks it’s conscious, because that would be a big headache. And because we’re also worried about systems lying about being conscious and giving misleading statements about whether they’re conscious — which they definitely do. So we’ve built this new kind of alien mind. We don’t really have a good theory of pain, even for ourselves. We don’t have a good theory of what’s going on inside it. So that’s like a stumbling-into-this sort of scenario. Luisa Rodriguez: Yeah. But it sounds like we could make things for economic reasons, like robots or chatbots, and we don’t realise those things are suffering. And then we mass produce them because they’re valuable. … And those things are suffering and we didn’t know it and they’re all over. And we don’t really want to change anything about those systems because we use them. Robert Long: Yeah. I mean, for just another dark scenario, you can imagine a system where we get pigs to be farmed much more efficiently. And we’re just like, “Well, this has made meat cheaper. Let’s not think too much about that.” I guess one thing I should note is I’ve been focusing on this case where we’ve hit on it accidentally. There are a lot of people who are interested in building artificial consciousness. Luisa Rodriguez: On purpose, yeah. Robert Long: And understandably so. You know, just from a purely intellectual or philosophical standpoint, it’s a fascinating project and it can help us understand the nature of consciousness. So for a very long time, probably about as old as AI, people were like, “Wow, I wonder if we could make this thing conscious?” There was a recent New York Times article about roboticists who want to build more self-awareness into robots, both for the intrinsic scientific interest and also because it might make for better robots. And some of them think, “Oh, well, we’re not actually that close to doing that. Maybe it’s too soon to worry about it.” Another person quoted in that article is like, “Yeah, it’s something to worry about, but we’ll deal with it.” And I am quoted in that piece as just kind of being like, “Ahhh, be careful, you know. Slow down. We’re not really ready to deal with this.” Luisa Rodriguez: Not ready to “deal with that.” Robert Long: Yeah, exactly. Jonathan Birch on how we might accidentally create artificial sentience Luisa Rodriguez: You did raise a few interesting points about AI sentience I wanted to ask more about. One is that you emphasise that AI sentience could arise in a number of ways. I think I intuitively imagine it arising either intentionally or unintentionally as a result of work on LLMs. But one of these other ways is whole brain emulation. And one case I hadn’t heard that much about is OpenWorm. Can you talk a bit about the goals of OpenWorm and how that project has gone? Jonathan Birch: This was a project that caught my eye around 2014, I think, because the goal was to emulate the entire nervous system of the nematode worm C. elegans in software. And they had some striking initial results, where they put their emulation in charge of a robot, and the robot did some kind of worm-like things in terms of navigating the environment, turning round when it hit an obstacle, that kind of thing. And it generated some initial hype. Luisa Rodriguez: It feels naive now, but it was eye-opening to me when you pointed out that we actually just wouldn’t need whole brain emulation in humans or of human brains to start thinking about the risks from AI sentience. We just need to go from OpenWorm to OpenZebrafish or OpenMouse, or maybe even OpenDrosophila — which sounds like not an insane step from just where we are now. How likely is it, do you think, that researchers would try to create something like OpenMouse? Jonathan Birch: Oh, it’s very likely. If they knew how, of course they would. I think one of the main themes of that part of the book is that once we see the decoupling of sentience and intelligence — which is very important, to think of these as distinct ideas — we realise that artificial sentience might not be the sort of thing that goes along with the most intelligent systems. It might actually be more likely to be created by attempts to emulate the brain of an insect, for example — where the intelligence would not be outperforming ChatGPT on any benchmarks, but perhaps more of the relevant brain dynamics might be recreated. Luisa Rodriguez: Yeah, it was a jarring observation for me. I think part of it is that it hadn’t occurred to me that people would be as motivated as they are to create something like OpenMouse. Can you say more about what the motivation is? Does it have scientific value beyond being cool? Or is the fact that it’s just a cool thing to do enough? Jonathan Birch: I think it would have an immense scientific value. It would appear to be a long way in the future still, as things stand. But of course, we’re talking here about understanding the brain. I think when you emulate the functions of the C. elegans nervous system, you can really say you understand what is going on — and that just isn’t true for human brains, currently. We have no idea. At quite a fundamental level, our understanding of C. elegans is in some ways far better. And it’s another step again, if you don’t just understand how lesioning bits of the nervous system affects function, but you can also recreate the whole system in computer software, would be a tremendous step. And it holds the promise over the long term of giving us a way to replace animal research. Because once you’ve got a functioning emulation of a brain, you can step past that very crude method of just taking the living brain and injuring it, which is what a lot of research involves, or modifying it through genome editing. You can instead go straight to the system itself and do incredibly precise manipulations. So I feel like, if anything, it hasn’t been hyped enough. I want more of this kind of thing, to be honest, than has been the case so far. Luisa Rodriguez: Intuitively, it seems plausible — and maybe even likely — that if you were able to emulate a mind that we thought was sentient, that the emulation would also be sentient. But is there a reason to think those come apart? Maybe we just don’t know. Jonathan Birch: It’s another space where you get reasonable disagreement, because I think we have to take seriously the view that philosophers call “computational functionalism”: a view on which, if you recreate the computations, you also recreate the subjective experience. And that leads to further questions about at what grain does one have to recreate the computations? Is it enough to recreate the general type of computation? Or does every algorithm at every level, including the within-neuron level, have to be recreated? And there too, there’s disagreement. I think we have to take seriously the possibility that recreating the general types of computations might be enough. I call this view “large-scale computational functionalism”: that it might be a matter of simply creating a global workspace or something like that, even if the details are quite different from how the global workspace is implemented in the human brain. And if we take that view seriously, as we should, it really does suggest a kind of parity. I wouldn’t want to overstate it, because I’d say that the probability of sentience is higher in the biological system than in its software emulation. But still, that software emulation is potentially a candidate. Anil Seth on which parts of the brain are required for consciousness Luisa Rodriguez: I’m really, really curious to get your thoughts on animals. I think maybe I would like to start with different neuroscientific theories of consciousness. Which parts of the brain are sufficient and required for consciousness feels like it might be a really key question for thinking about which nonhuman animals we should expect to have conscious experiences — because some nonhuman animals, like insects, only have things that look much more like the very old parts of the human brain, the parts that are deeper in. Do you have a view on which theories seem most plausible to you? Are the really old parts of the brain, the subcortical parts, sufficient for consciousness? Anil Seth: To be honest, I don’t know. But I think to help orient in this really critical discussion — critical because, of course, it has massive implications for animal welfare and how we organise society and so on — it’s worth taking a quick step back again and just comparing the problem of animal consciousness with the one of AI consciousness. Because in both cases there’s uncertainty, but they are very different kinds of uncertainty. Luisa Rodriguez: Almost opposite ones. Anil Seth: Almost exactly opposite. In AI, we have this uncertainty of, does the stuff matter? AI is fundamentally made out of something different. Animals are fundamentally the same because we are also animals. And then there’s the things that are different. Animals generally do not speak to us, and often fail when measured against our highly questionable standards of human intelligence. Whereas AI systems now speak to us, and measured against these highly questionable criteria, are doing increasingly well. So I think we have to understand how our psychological biases are playing into this. It could well be that AI is more similar to us in ways that do not matter for consciousness, and less similar in ways that do — and nonhuman animals the other way around. We’ve got a horrible track record of withholding conscious status and therefore moral considerability from other animals, but even from other humans. For some groups of humans, we just do this. We’ve historically done this all the time and are still doing it now. There’s this principle in ethics called the precautionary principle: that when we’re uncertain, we should basically err on the side of caution, given the consequences. I think this is really worth bearing in mind for nonhuman animals. You could apply the same to AI and say, well, we should just apply that: since there’s uncertainty, we should just assume AI is conscious. I think no: I think the effect of bias is so strong, and we can’t care for everything as if it’s conscious, because we just only have a certain amount of care to go around. But when it comes to nonhuman animals, they have the brain regions, the brain processes, that seem highly analogous to the ones in human and mammalian brains for emotional experiences, pain, suffering, pleasure and so on, that I think it pays to extend the precautionary principle more in that direction. Figuring out exactly which animals are conscious, of course, we don’t know. But there are things that I think are relatively clear. If we just take mammals very broadly, from a mouse to a chimpanzee to a human, we find very similar brain structures and similar sorts of behaviours and things like that. So it seems very, very unlikely that there are some mammals that lack consciousness. I think mammals are conscious. But even then, we’ve had to get rid of some of the things that historically you might have thought of as essential for consciousness, like higher order reasoning and language. I mean, Descartes was infamous — but at the time, it was probably a very sensible move because of the pressure he was under from the religious authorities — he was very clear that only humans had consciousness, or the kind of consciousness that mattered, and that was because we had these rational minds. So he associated consciousness with these higher rational functions. Now people generally don’t do that. So mammals are within the magic circle. What else? Then it becomes really hard, because we have to just walk this line: we have to recognise we’re using humans — and then, by extension, mammals — as a kind of benchmark. But you know, there might well be other ways of being conscious. What about the octopus, as Peter Godfrey-Smith has written beautifully about? And what about a bumblebee? What about bacteria? It’s almost impossible to say. It seems intuitive to me that some degree of neural complexity is important, but I recognise I don’t want to fall into the trap of using something like intelligence as a benchmark. Luisa Rodriguez: Yeah. I mean, that’s just basically what I find both maddening and fascinating about this question of nonhuman animals. It feels like there’s this very frustrating slippery slope thing, where I don’t want to be overly biased toward humans, or towards structures that kind of “create consciousness,” whatever that means, in the way that ours does. And it does seem like there might be multiple ways to do it. And over time, I’ve become much, much more sympathetic to the idea that not just birds, and not just cephalopods, but insects have some kinds of experiences. And I just find it really confusing about where and how and whether it makes sense to draw a line, or maybe that’s just philosophically nonsensical. So I’m really drawn to this question, which is why I opened with it, of: Can neuroscience point to functions or places or parts of the brain that seem related enough to consciousness, that if we see analogous things in bees, we should update a lot on that? But I also have the impression that there’s still so much debate about subcortical and cortical theories, and which is more plausible, that maybe we’re just not there, and that’s not possible now, and might not be for a while. Anil Seth: Jonathan Birch, who’s a philosopher at UCL, has this beautiful new book called The Edge of Sentience, which I think is all about this. He’s trying to figure out how far we can generalise and on what basis. I think the issue is that it seems very sensible that consciousness is multiply realisable to an extent: that different kinds of brains could generate different kinds of experience, but it’d still be experience. But to know when that’s the case, we have to understand the sort of basis of consciousness in a way that goes beyond, “It requires this or that region.” We need to know what is it that these brain areas are doing or being that makes them important for consciousness in a way that we could say, well, we obviously don’t see a frontal cortex in a honeybee, because they don’t have that kind of brain, but they’re doing something, or their brains are made of the stuff and organised in the right way, that we can have some credence that that’s enough for consciousness. And we don’t really have that yet. I mean, the theories of consciousness that exist are varied. Some of them are pretty explicitly theories about human consciousness, and they’re harder to extrapolate to nonhuman animals: like what would be a global workspace in a fruit fly? You could make some guesses, but the theory as it is is more assuming there’s a kind of cortical architecture like a human. And other theories, like integrated information theory, are much clearer: wherever there is some nonzero integrated information maxima of X, there’s consciousness. But it’s just impossible to actually measure that in practice. So very, very hard. But the path, I think, is clear that the better that we can understand consciousness, where we are sure that it exists, the surer our footing will be elsewhere where we’re less confident, because we will be able to generalise better. So where are your areas of uncertainty? I’m always interested. Luisa Rodriguez: I feel just through getting to learn about these topics for this show, I constantly get told these amazing facts about fish and bees, and the kinds of learning they can do, and the kinds of motivational tradeoffs they make, and the fact that they do nociception, and that nociception gets integrated into other parts of their behaviour. And that all feels really compelling to me. Then I talk to someone who’s like, “Yeah, but a lot of that could just be happening unconsciously.” And at what point it’s more plausible that they’re little robots doing things unconsciously, and then at what point it becomes more plausible that a little robot doing that is just a less likely story than it’s got the lights switched on in some way, and it’s making tradeoffs because the world is complicated and it’s evolved to have more complex systems going on so that it can survive. I just find that really confusing. Anil Seth: Yeah. I think me too. But actually there’s a point you make which I didn’t make, so I’m grateful for you bringing that up, which is the functional point of view too. So instead of just asking which brain regions or interactions between brain regions do we see, we can ask from the point of view of function and evolution, which is usually the best way to make sense of things comparatively between animals and biology. So what is the function of consciousness? And if we can understand more about that, then we can have another criterion for saying, which other animals do we see facing and addressing those same kinds of functions? And of course there may be other functions. We have to be sensitive to that too. But at least it’s another productive line. And in humans and mammals, there’s no single answer. But it seems as though consciousness gets into the picture when we need to bring together a lot of different kinds of information signals from the environment in a way that is very much centred on the possibilities for flexible action — all kind of calibrated in the service of homeostasis and survival. So automatic actions, reflexive actions, don’t seem to involve consciousness. Flexibility, informational richness, and sort of goal-directedness, those seem to be functional clues. So to the extent we see animals implementing those similar functions, I think that’s quite a good reason for attributing consciousness. But it’s not 100%. Luisa Rodriguez: Yeah. That’s basically where I am. And it means that I now feel a lot of feelings about the bees in my garden. And mostly I feel really grateful to have learned about these topics, but I also feel really overwhelmed. Peter Godfrey-Smith on uploads of ourselves Luisa Rodriguez: Why are you doubtful that we could create uploads of ourselves? Peter Godfrey-Smith: My view on that comes from the general position that I’m developing — slowly, cautiously — on the biology of conscious experience, or the biology of felt experience. This was developed in a bit more detail in the second book, but it has a discussion in Living on Earth as well. The view that I think is most justified — the view I would at least put money on — is a view in which some of what it takes to be a system with felt experience involves relatively schematic functional properties: the way that a system is organised in relation to sensing and action and memory and the internal processing and so on. And some of those, what are often referred to as “functional properties,” could exist in a variety of different physical realisations, in different hardwares or different physical bases. But I don’t think that’s the whole story: I think nervous systems are special. I think that the way that nervous systems work, the way that our brains work… There are two kinds of properties that nervous systems have. There’s a collection of point-to-point network interactions — where this cell makes that cell fire, and prevents that cell from firing, the spiking of neurons, and the great point-to-point massive network interactions. And there’s also other stuff, which for years was somewhat neglected I think in these discussions, but which I think is probably very important. There are more diffuse, large-scale dynamic properties that exist within nervous systems: oscillatory patterns of different speeds, subtle forms of synchronisation that span the whole or much of the brain. And these are the sorts of things picked up in an EEG machine, that kind of large-scale electrical interaction. And I didn’t come up with this myself. There’s a tradition. Francis Crick thought this, neuroscientists like Wolf Singer, a number of other people have argued that this side of the brain is important to the biology of conscious experience, along with the sort of networky, more computer computational side of the brain: that both sets of properties of nervous systems are important. And in particular, the unity of experience — the way in which brain activity generates a unified point of view on the world — has a dependence upon the oscillatory and other large-scale dynamic patterns that you get in brains. Now, if you look at computer systems, you can program a computer to have a moderately decent facsimile of the network properties in a brain. But the large-scale dynamic patterns, the oscillatory patterns that span the whole, they’re a totally different matter. I mean, you could write a program that includes a kind of rough simulation, where you’d know what was happening if the physical system in fact had large scale dynamic patterns of the relevant kind, but that’s different from having in a physical system those activities actually going on — present physically, rather than just being represented in a computer program. I do think there’s a real difference between those generally, and especially in the case of these brain oscillations and the like. You would have to build a computer where the hardware had a brain-like pattern of activities and tendencies. People might one day do that, but it’s not part of what people normally discuss in debates about artificial consciousness, uploading ourselves to the cloud and so on. People assume that you could take a computer, like the ones you and I are using now, with the same kind of hardware, and if you just got the program right — if it was a big powerful one and you programmed it just right — it could run through not just a representation of what a brain does, but a kind of realisation and another instance or another form of that brain activity. Now, because I think the biology of consciousness is just not like that — I think that the second set of features of brains really matter — I think that it will be much harder than people normally suppose to build any kind of artificial sentient system that has no living parts. It’s not that I think there’s a kind of absolute barrier from the materials themselves — I don’t know if there is — but I certainly think it would have to be much, much more brain-like. The computer hardware would have to be a lot more brain-like than it is now. I mean, who knows if we could build large numbers of these, powered with a big solar array, and replicate our minds in them? I think it’s very unlikely, I must say. Now, whether that’s unlikely or not, I don’t think I should be confident about. The thing I am a bit confident about, or fairly confident about, is the idea that there’s lots of what happens in brains that’s probably important to conscious experience, which is just being ignored in discussions of uploading our minds to the cloud and things like that. Luisa Rodriguez: Yeah, that’s really helpful. I don’t know very much about these large-scale dynamics. Are they a result of neuronal firings, or are they a result of other things going on in the brain? Peter Godfrey-Smith: This is quite a controversial point, actually. It’s a good question. The sense I have — and I’m continually trying to learn the latest on this — is that most of the activity in those large-scale dynamic patterns is not just a summing together of lots of firing of neurons. Because if it was, then you might say that the network, point-to-point, cell-to-cell things really are all that matters once you’ve really captured those, and the other stuff is just a consequence or a kind of zoomed-out manifestation of that. And that does not seem to be the case. It seems rather that within the cells that make up our brain, there’s a kind of to and fro of ions across membranes that is below the threshold that is required to actually make the neuron fire, that dramatic spark-like firing. It’s more of a kind of rhythmic, lower-level, sub-threshold electrical oscillation. And those oscillations affect the firing of neurons, and the firing of neurons affect the oscillations. But there is a kind of duality of processes there, and it’s not just that if you knew which cells were firing when, and you ignored everything else, that you could recapture, in a sense, the large-scale dynamic patterns. There’s more than that. Luisa Rodriguez: Right. So if you take this kind of, for me, what has been a very intuitive, but I guess would feel like an inadequate thought experiment of replacing each neuron in the human brain, one by one, with an artificial silicon-based one, it sounds like you would guess that doing that, even if you got that to work, it wouldn’t make the entire process and physical system of the brain be entirely artificial? There would be other things that remained biological that were playing a crucial role that you hadn’t replaced and replicated yet? Peter Godfrey-Smith: Well, the way you described it, you made it sound like at the end of this process, the biological parts were gone. So is it that we really replace everything, or is it just we replace some stuff and leave some of the biological material in place? Luisa Rodriguez: I think I’m curious about if you just replaced neurons with synthetic neurons, do you think there would be a working brain at the end? And if so, would that be because the other biological stuff was still there? Peter Godfrey-Smith: OK, right. I see. It’s a commitment of my view that if you tried to do that, you would change all sorts of stuff: you can’t really preserve the things that the brain is doing at the start and have them continue once the neurons have been replaced by artificial devices. I mean, what are the “artificial devices”? When people talk about thought experiments of this kind — which they’ve done, I think, in quite interesting ways for about 40 years now — what they usually imagine is you’re replacing each neuron with a kind of relay object that sums up the inputs coming in and either fires or doesn’t fire, and its firing then contributes to the inputs to various other downstream cells. And it’s just doing that. It’s not doing anything else. Now, if the things you put into the brain were just doing that, then essentially you would drop all those large-scale dynamic patterns. They just wouldn’t exist anymore. They would be gone. So it would be a different thing physically. It would do different things, because the relationship between the firing of neurons and those slower oscillations does make a difference to how the brain works, so you would change some stuff that would have consequences. Now, something that people haven’t talked about so much is whether you could put in an artificial object that did really everything that the neuron is doing, where there’s this sort of seepage of ions across the borders, and it has subtle effects on the electrical properties of neighbouring cells and so on. A replacement that captured a lot more of what a neuron did. And then we reached the point where we got to a few minutes ago, where I said who knows what artificial hardwares might be possible? I don’t think I know nearly enough about these matters to be confident that you could never build something that did that. Luisa Rodriguez: Just to make sure I understand your position on this: it’s not that you’re confident that it’s impossible to create digital minds in general, or that it’s impossible to eventually come up with some hardware system that does replicate everything relevant to consciousness in a human — but that the current thought experiments where we only replicate some of the basic functions of a neuron are not enough. Peter Godfrey-Smith: There was a phrase you used — “digital minds” — and I guess I don’t think there will ever be digital minds. I think that’s imagining a system which lacks too much of what brains and nervous systems are like in order for it to have a mind. I don’t know if there might be wholly artificial minds in the future that are made of different stuff than brains are made of, but that have the important duality of properties that you see in nervous systems, and thereby achieve a kind of sentience. I think it’s further away than people often suppose, but I would not want to say it could never happen. The thing I want to press on in a kind of critical way is the habits that people have of thinking… People think, “I know what neurons and nervous systems do. It’s like a telephone exchange: you’ve got this thing that makes that thing go, and it’s like a big network, and that’s all there is to it.” They think that, and then they think, “Well, a computer is a place where I can have that kind of thing going on, replicated to a high degree of fidelity. So I know that an ordinary computer has what it takes as hardware to be sentient, basically, to be conscious if I put the right program in there — because the program would just have to create a version of those networked, point-to-point interactions.” That claim I do want to push back against. I don’t think we have any reason to believe that. And I think that artificial sentience is just going to be a harder and further-in-the-future thing than we have supposed. And as I say at one point in this book, Living on Earth, I’m not sure whether that’s a bad thing. If there turns out to be a kind of barrier between the I side of AI — the intelligent side of AI — and the kind of conscious, sentient, feeling side of artificial minds, if there’s a barrier there, that might not be a bad thing. One reason for that is if we start building lots of sentient systems that have this different kind of hardware, and they’re under our control, we made them — I don’t know how likely it is that their experiences are going to be positive, at least in the early stages. It’s going to be a sort of a klutzy, messed-up version of artificial sentience that we’re working with. And if I’m a disembodied spirit of the kind that you were talking about earlier, I don’t want to come down to Earth and live as an early effort in the human attempt to make artificial sentient systems. Jonathan Birch on treading lightly around the “edge cases” of sentience Luisa Rodriguez: So a big part of the book then explores these candidates for sentience: beings that we think could plausibly be sentient, but because we just know so little about exactly what sentience even is, and how different beings feel it and in what proportions, there’s not enough clear understanding or evidence to be sure what it is like to be them. You look at some familiar candidates, including animals of different classes, as well as AI. And then you also look at some cases that I was really unfamiliar with, including people with disorders of consciousness — so people in comas, for example — and then also “neural organoids,” which I’d never heard of. To start us off, what exactly is a neural organoid? Jonathan Birch: This is another very fast-moving area of emerging technology. Basically, it uses human stem cells that are induced to form neural tissue. The aim is to produce a 3D model of some brain region, or in some cases a whole developing brain. Luisa Rodriguez: And what’s the case for creating them? Jonathan Birch: I think it’s a very exciting area of research. You can make organoids for any organ, really. In a way, it’s a potential replacement for animal research. If you ask what we do now, usually people do research on whole animals, which are undeniably sentient. And here we have a potential way to gain insight into the human version of the organ. It could be a better model, and it’s much less likely to be sentient if it’s something like a kidney organoid or a stomach organoid. It’s really only when we’re looking at the case of the brain and neural organoids that the possibility of sentience starts to reemerge. Luisa Rodriguez: Yeah. And intuitively, the case for sentience does feel like it immediately lands for me. If you are trying to make an organoid that is enough like a brain that we can learn about brains, it doesn’t seem totally outrageous that it would be a sentience candidate. What is the evidence that we have so far? Jonathan Birch: It’s a complicated picture. I think there are reasons to be quite sceptical about organoids as they are now, but the technology is moving so fast, there’s always a risk of being ambushed by some new development. At present, it really doesn’t seem like there’s clear sleep/wake cycles; it doesn’t seem like those brainstem structures or midbrain structures that regulate sleep/wake cycles and that are so important on the Merker/Panksepp view are in place. But there are reasons to be worried. For me, the main reason to be worried was a study from 2019 that allowed organoids to grow for about a year, I think, and compared them to the brains of preterm infants using EEG. So they used EEG data from the preterm infants to train a model, and then they used that model to try and guess the age of the organoid from its EEG data, and the model performed better than chance. Luisa Rodriguez: Wow. Jonathan Birch: So it’s hard to interpret this kind of study, because some people, I suppose, read it superficially as saying these organoids are like the brains of preterm infants. And that’s an exaggeration, because they’re very, very different and much, much smaller. But still, there’s enough resemblance in the EEG to allow estimates of the age that are better than chance. Luisa Rodriguez: It’s definitely something. I find it unsettling, for sure. Jonathan Birch: It is, yeah. I think a lot of people had that reaction as well, and I think that’s why we’re now seeing quite a lively debate in bioethics about how to regulate this emerging area of research. It’s currently pretty unregulated, and it raises this worrying prospect of scientists taking things too far — where they will often say, “These systems are only a million neurons; we want to go up to 10 million, but that’s so tiny compared to a human brain.” And it is tiny compared to a human brain. But if you compare it to the number of neurons in a bee brain, for example, that’s about a million. So these near-future organoids will be about the size of 10 bee brains in terms of neuron counts. And I think bees are sentience candidates, so naturally I take this risk quite seriously, and I think it would be wrong to dismiss it. Meghan Barrett on whether brain size and sentience are related Luisa Rodriguez: I guess the thing I do care about in the context of this sentience discussion is brain size. It feels like brain size is at least more likely to be tracking the thing that I care about. And I guess I care about whether these very large insects also have very large brains, or if the beetle is mostly like other guts, as opposed to brain guts. Meghan Barrett: This is a phenomenal question. I appreciate you asking it very much, and it’s very interesting to me. So I have a couple of points to make on the brain size piece of this. The first thing I’ll say is just that we should ask ourselves if more actually always means better or more sophisticated. That’s an assumption to challenge about our thinking about brains. For example, elephants have more neurons and more brain mass than you and I do, and yet I don’t consider them more likely to be sentient than me. Same thing with blue whales. And that’s because we know a lot of those additional neurons are because they’re just bigger, and so they have more mass to control, right? They have more touchpoints they need to be able to integrate and things like that. So more doesn’t always mean more sophisticated; it can often just mean more of the same repeat unit that performs the same kind of function. And that explains, very plausibly I think, this phenomenon that we see, where very small brains are capable of producing really complex abilities. Insects are capable of things like numerical cognition, social learning, facial recognition, cognitive bias, and much more. And so because of this behavioural data and this complexity that we initially thought was just going to be something you find in vertebrates, and now we’re seeing it in invertebrates with pretty small brains, like honeybees, this suggests that maybe those bigger brains aren’t actually better or more sophisticated — they just have more redundancies or repeating modules. You could think of this as changing resolution or complexity of a capacity, without changing the existence of that capacity itself. So imagine an image that has more pixels to it: it doesn’t change the fact that there’s still a picture; it just changes the resolution of that picture. So that’s part one to this: that first we should just challenge that basic assumption that bigger brains are necessarily better at producing a capacity at all. The next thing we need to think about is whether or not they are actually that small, which is what you were asking initially. So here again, I want to give some examples from vertebrates and invertebrates to challenge our intuition that vertebrates are always bigger than invertebrates. Let’s first consider just the mammals, because I think we all feel really comfortable talking about mammal sentience. So the smallest mammal brain that we have studied so far weighs in at 64.4 milligrammes. That’s the Etruscan shrew. Luisa Rodriguez: That is tiny. Meghan Barrett: Shrews can get very small. Very, very small. And the body mass of that shrew is about two grams or so, give or take. So we get some pretty small mammals out there. When we look at the largest insect brain we’ve studied to date, it’s a solitary wasp, and it weighs in at 11.7 milligrammes. That’s an insect with a body mass of about 0.5 grams, so a quarter the size of the shrew. That makes that shrew’s brain just about six times larger than the insect brain. That’s a smaller difference than we see between humans and whales. So already, even if we’re just considering the mammals, we see relatively comparable brain masses [with the wasp]. Now, that’s just one level of looking at the brain. I want to also extend beyond just mammals now and consider other vertebrates you might take seriously — like your lizards, say. So our smallest lizard brain that we’ve studied is the Algerian sand gecko. The brain itself weighs about 10.8 to 11.8 milligrammes. So that is smaller than the wasp at 11.7. Luisa Rodriguez: Yeah. Wow. Meghan Barrett: So already we’re seeing vertebrates close or comparable to the invertebrates, depending on whether you’re talking about mammals or lizards. So we know we’re already in the same ballpark on mass. Now, you might not think that mass is the most relevant metric, though, for considering the brain. That’s because we might think that actually things like neuron numbers, like the computational units of the brain, are more important. One note I want to make on this is that I’m going to talk about whole brain numbers today, because that’s the only data that we have. But I again want to go back to our earlier point that not all neurons do all things: neurons are specialised for particular functions; they’re organised into functionally discrete regions. For example, wasps have many more optic lobe neurons than ants do. Optic lobe neurons are the ones that bring in visual information from the periphery. This is probably because wasps fly and have to bring in a lot of visual information really fast. I don’t think that all those additional optic lobe neurons are all that sentience-relevant. So if I counted those, I would think the wasps were more likely to be sentient [than ants], when really they just can process and input more visual information more quickly. Luisa Rodriguez: Yes, that makes sense to me. Meghan Barrett: So that’s one thing to keep in mind: that we don’t have good data on what are the sentience-relevant regions of the brain, and how many cells are in those regions. So instead we’re just going to talk about total neuron numbers, because that’s the data we’ve got so far. So with that caveat, let’s think about neuron numbers. Let’s go back again to mammals, because we like starting with mammals, right? So, as far as I know, the smallest number of neurons we’ve found in any of the mammals so far is in the naked mole-rats, and that’s about 26.88 million neurons. The smallest studied vertebrate brain is, again, that Algerian sand gecko I mentioned before. It has about 1.8 million neurons. And it turns out that that wasp brain I mentioned to you before also contains 1.8 million neurons. So our gecko and our wasp are very comparable. They’re both about 15 times smaller than that of the naked mole-rat. Something interesting to consider here is that if we just look across the mammals, that naked mole-rat has only one one-thousandth the number of neurons of an elephant. So again, there’s a lot more difference within the mammals than we’re seeing between our smallest mammals and even just the invertebrates that we’ve studied so far from an insect perspective. So I think, again, we might consider just that there’s a lot of reasons to be sceptical that insects have an unusually small number of neurons compared even to the mammals, much less the vertebrate lizards and snakes. Luisa Rodriguez: Yeah, interesting. I think I learned about 1,000 new facts there. Lewis Bollard on how animal advocacy has changed in response to sentience studies Luisa Rodriguez: I was wondering if you’d be up for sharing some reflections from your time as programme officer at Open Phil. I think you’ve now been doing this since 2015-ish, and you were also in the farmed animal welfare space even before that. Is there something that you’ve changed your views on since you started doing this work? Lewis Bollard: I’ve become a lot more worried about invertebrates. When I started this work, I — like many of us — just ignored them. I think I sort of quietly assumed that they weren’t sentient. And then I remember, someone asked me back in 2016, and I said, maybe there’s like 10% chance that they were sentient — and that was enough to make me worry a bit, but it really wasn’t that high. Since then, thanks in large part to the work that Rethink Priorities did with their Moral Weight Sequence, I have really come to see a very high probability that invertebrates are sentient in some meaningful sense, and that their welfare matters. Luisa Rodriguez: Yeah, that makes sense. We actually just interviewed Bob Fischer about the Moral Weight Project, so listeners might have heard that. I’m curious if you remember any of the particular facts or research that felt compelling to you, and convinced you that invertebrates are more likely to be sentient than you thought? Lewis Bollard: I think it was more of the absence of contrary facts. I mean, I had just assumed that because society acts as if insects aren’t sentient and shrimp aren’t sentient, that there must be good evidence for that. And I was really surprised when they started looking into this and there just wasn’t. On the flip side, there was evidence to worry. And for me, the most compelling one is actually just the evolutionary reason, which is that it just does seem like an animal who has the capacity to move and the capacity to learn, there are reasons, unfortunately, for it to have the capacity to feel pain, too. Luisa Rodriguez: Yes, I feel the same way. Speaking of the Moral Weight Project, back in 2017, Rob Wiblin asked you if we had any kind of quantitative measure that you can use to compare animal suffering to human suffering. And I think you said something like, “That’d be great, but no — our ability to understand the relative experiences of different species is still really limited.” We’ve both alluded to this work a few times already, but just to give a bit more context: Bob Fischer and his colleagues looked at a bunch of different physiological and behavioural and cognitive traits in different animals, and then, based on how many traits a given animal had, they gave a rough estimate of how the capacity for pain and pleasure of a chicken or cow or fruit fly compares to that of a human. And I found the results surprising. In general, they were very animal friendly. For example, they concluded that their best guess was that a chicken has something like a third of the capacity for pain and pleasure as a human — which can imply some things that feel very strange about the kinds of tradeoffs you might make, for example, if you were doing a trolley problem with chickens and humans. But I’m curious what your reactions were to their results? Lewis Bollard: Yeah, I found them really interesting. And I agree: for most people, they’re very counterintuitive. Two things I’d say. One is to understand they’re just looking at that capacity for suffering, and so there might be other reasons why you choose to prefer humans. I mean, for one thing, we have much longer lifespans, so I’d save a human over a chicken because they have many more years to live. But also, you might think that they have more other more meaningful things — that there are social networks who are going to be sad about losing them, and so on. The second thing is, I would encourage people to really approach this with a fresh mind and ask, why do I find this so counterintuitive? I think we have such an ingrained hierarchy in our minds of animals — where, of course, humans are at the top, and every other animal is below us — and we start out from that place and then we sort of update from there. And if you tried instead to start from more of a blank slate, where you just look at the different capacities of these animals, and you don’t assume anything, then I think you end up more likely at these more equal numbers. Or if you don’t, I think it’s because you make some unusual philosophical turns. And then I would just ask, are you happy with where those philosophical turns take you? Luisa Rodriguez: Yeah. “Do you endorse having the arbitrary view that only one species matters, or only things that are kind of like you matter?” Those seem unpalatable. I guess when I think about it, if I’m like, what is making me have this gut reaction that’s like, “No, surely not; surely there are bigger differences between these species”? It’s not like I have evidence. It’s not like I’m like, “Once I saw a dog kicked, and it didn’t seem upset,” or it’s certainly not like I know anything about the science of pain and how it presents or doesn’t present in different animals’ brains or something. It’s nothing. It feels very sociological — like you said, this hierarchy. And if I really try to think, what evidence do I have, without kind of looking into it, it’s really just like, “Well, I’m a being in the world. They’re also beings in the world. And maybe we should just actually think that we’re all really similar, because the world is hard and scary, and we have to have mechanisms that keep us alive and reproducing.” So if you start from there, then we’re actually on a really similar point. And I found that a really helpful way to pump my intuitions about where actually should we be starting? Is it with these huge gaps between humans and insects, or is that just completely out of nowhere? Lewis Bollard: Yeah. One other thought I’d have on that is that I think it can be helpful thinking of the most charismatic animal of a species or class. So rather than thinking, what’s the moral weight of a chicken? — which just seems, I think intuitively for many people, not worth much — think of what’s the moral weight of a bald eagle, and take out the preservation value or something. But just say, for one thing, there aren’t that many of them. So you don’t have this initial intuition of, oh god, if I give them a lot of moral weight, they’re going to trump everything else; they’re going to swamp everything. And you also probably have a pretty positive impression of them: their complexity, their grandeur, and everything. Or even within insects, I would say rather than a fruit fly, think about a bumblebee. Now, it’s possible that a bumblebee has more sophisticated capacity, so I’m not saying to skew it for that reason, but I think if you think about the more charismatic animals, there can at least be an intuition pump to think, “Is what I’m doing here just choosing animals that I don’t like, and thinking they can’t possibly be worth very much?” Luisa Rodriguez: “Nah. They’re ugly, they give me the creeps. Yeah, they probably don’t feel anything.” Those things are related. Lewis Bollard: Right. Luisa Rodriguez: The other one that Meghan Barrett gave me was, if I think about the biggest insect I can think of — and she actually told me about some insects of certain sizes I just didn’t even know existed; they’re just way bigger than I realised — if a beetle is the size of a mouse, all of a sudden my brain’s like, “Oh, that could be as smart as a mouse then.” So yeah, there’s just clearly some size bias thing — which maybe there is something going on there, something about neurons and neurons being more plentiful could have something to do with capacity for experience — but it isn’t the end-all and be-all. So the fact that we have these really intense intuitions about size seems like we should be suspicious. Lewis Bollard: Yeah, that’s a good way to put it. I mean, you can think of a lobster. My sense is that humans intuitively care more about a lobster than an insect. My understanding is their brains are relatively similar in terms of neuron count, in terms of a lot of features, and that really is just a size difference. Similarly, elephants are very smart animals, but they’re not that much smarter than other mammals. But I think we really have that sense that, wow, they deserve protection. So I think that’s right. The size bias is very real. Luisa Rodriguez: Yeah. If I remember correctly, ants have more neurons than crabs. And that’s another one where I’m like, yeah, crabs I can get on board with. I can get on board with caring about those. OK, so yeah, there’s this work. It’s got some pretty counterintuitive results. Has it changed the way you think about prioritising between different interventions, besides maybe putting more weight on invertebrates? Lewis Bollard: Yeah, it’s definitely led us to put more weight on invertebrates. I think at the same time, it’s always a tradeoff between the importance of a species and the tractability of work on that — and I think that we’ve had a greater track record of tractable work on chickens and fish. I think there’s also a thing of how far does the Overton window go? How far can you get people to understand things? And I think there’s a risk that if our movement just became an insect welfare movement, that, for a lot of people, would be a reductio ad absurdum. That would be, “Well, if insects count too, then none of this matters.” I think, in a way, that it does make more sense for our movement to bring people along with us and to focus on species… Not to just wait where people are currently — you definitely want to lead people — but I think to lead people more slowly, and also to work on a variety of issues and species. So you have a greater array of shots at progress. Luisa Rodriguez: Right. Yeah. It is just the case that even if we have more evidence than we thought that insects feel pain, it seems like we’re still really far away from knowing with any confidence about what any of these animals are feeling really concretely and confidently. So diversifying seems at least a plausible approach to dealing with that uncertainty. Bob Fischer on using proxies to determine sentience Luisa Rodriguez: A colleague of yours has written this report on why neuron counts aren’t actually a good proxy for what we care about here. Can you give a quick summary of why they think that? Bob Fischer: Sure. There are two things to say. One is that it isn’t totally crazy to use neuron counts. One way of seeing why you might think it’s not totally crazy is to think about the kinds of proxies that economists have used when trying to estimate human welfare. Economists have for a long time used income as a proxy for human welfare. You might say that we know that there are all these ways in which that fails as a proxy — and the right response from the economist is something like, “Do you have anything better? Where there’s actually data, and where we can answer at least some of these high-level questions that we care about? Or at least make progress on the high-level questions that we care about relative to baseline?” And I think that way of thinking about what neuron-count-based proxies are is the charitable interpretation. It’s just like income in welfare economics: imperfect, but maybe the best we can do in certain circumstances. That being said, the main problem is that there are lots of factors that really affect neuron count as a proxy that make it problematic. One is that neuron counts alone are really sensitive to body size, so that’s going to be a confounding factor. It seems like, insofar as it tracks much of anything, it might be tracking something like intelligence — and it’s not totally obvious why intelligence is morally important. At least in the human case, we often think that it’s not important, and in fact, it’s a really pernicious thing to make intelligence the metric by which we assess moral value. And then, even if you think that neuron counts are proxies of some quality for something else, like the intensity of pain states or something — it’s not clear that that’s true, but even if that were true — you’d still have to ask, can we do any better? And it’s not obvious that we can’t do better. Not obvious that we can, but we should at least try. Luisa Rodriguez: Yes. Makes sense. Are there any helpful thought experiments there? It doesn’t seem at all insane to me — though maybe you wouldn’t expect it to happen on its own through evolution — that there would be a being who has many fewer neurons than I do, but that those neurons are primarily directed at going from extreme pain to extreme something like euphoria. It doesn’t seem like there’s a good reason that’s not possible, and that that extreme pain could just be much more than the total amount of pain I could possibly feel. Even though the types of pain might be different for me — because I’ve got different kinds of capacities for sadness and shame and embarrassment, like a wider variety of types of pain — it still seems at least theoretically possible that you could house a bunch of pain in a small brain. And that feels like good reason to me to basically do what you’ve done, which is look for better ways than neurons alone. Bob Fischer: Sure. And some evolutionary biologists have basically said things along these lines. Richard Dawkins actually has this line at some point, where he says that maybe simpler organisms actually need stronger pain signals because they don’t learn as much as we do and they don’t remember all these facts, so they need big alarm bells to keep them away from fitness-reducing threats. So it’s always possible that you have a complete inversion of the relationship that people imagine, and you want to make sure that your model captures that. Cameron Meyer Shorb on how we can practically study wild animals’ subjective experiences Luisa Rodriguez: What do we do with all of that uncertainty? Do we have any way of knowing whether a rabbit’s experience of grazing in a field is more like terror or more like leisure? It just seems so hard… Cameron Meyer Shorb: Right. So I think those are answerable questions, is the bottom line. Sentience is a very hard thing to really concretely understand. You can’t really measure the thing itself, and there’s all these uncertainties. But at the end of the day, we do have babies, we do have dogs: they don’t know how to talk, but we do a pretty good job keeping them happy, right? So I think that’s proof of concept that you can use indicators of welfare to make some good decisions, at least about the biggest things, the biggest sources of happiness or suffering or whatever. With respect to questions like, Are animals happy when they’re grazing? Are animals afraid of predators? I do think that these are questions that are going to have to be answered for each species or each group of animals in its own context. There just is a lot of diversity of life on Earth, and this is part of the project of wild animal welfare science is learning to listen to everyone else. But the tools we have to ask those kinds of questions are somewhat generalisable, and I divide those into the physical indicators of welfare and the behavioural indicators of welfare. Physical indicators would be things like looking at the condition of the body. You can also look at their external appearance: do they seem to have injuries or disease? You can look at neurotransmitters or glucocorticoids (commonly called stress hormones) in the blood, other hormones, body temperature. So the effects of our mental states are manifested in many ways in our bodies, and by measuring those, we can get some useful information on animals’ interior states. And then the final set of physical indicators is genetic indicators: looking at things like biomarkers of ageing can be used as proxies for the cumulative physiological stress that an animal has undergone over their lifetime. We’ve done some work to validate this, and there’s more work that needs to be done. But this is the kind of thing that might be broadly useful across many kinds of animals, right? Because that basic structure of DNA is preserved across all of animal life. And then of course, there are the behavioural indicators, which are the ones we use for our children and pets and others — looking at things like vocalisations, activity, are they inactive, what sort of posture are they holding, are they engaging in activities like play or showing fearful behaviour? Then there are these cognition or decision-relevant behavioural indicators. You can actually set up experimental tests to see which things animals prefer. This has been done for chickens to develop some of the information we use to inform what conditions they prefer in factory farming contexts. You can also do the same with wild animals. As you can imagine, setting up these choice experiments in the wild is tricky, but not always impossible, and you can set up these forced-choice experiments to see which things animals prefer that can sort of inform which of these things is better for the animal. You can also use them to assess what mental state is the animal in right now. For example, you can measure whether they seem to be showing a certain amount of cognitive bias, like pessimism. The hypothesis is that if the animal makes fewer efforts to look for food in areas where it had previously been trained there might be food, that might demonstrate pessimism, and that might indicate a just overall negative affect. As you can see, there are a lot of assumptions that underlie a lot of these things, and no single method is perfect. Our general advice to researchers is: one, there’s no silver bullet; you want to use multiple different methods, ideally of different types — so some physical, some behavioural. And then there are also, among these methods, some that we have more confidence in than others. For that reason, it’s helpful to design experiments in a way where you have at least one metric that has more evidence, that we have higher confidence in, and at least one metric that we have less confidence in — and we can then use those results to start building confidence in those other metrics. So trying to see to what extent do these things correlate with each other. And it’s all fuzzy; it’s kind of cloudy and never totally certain. But through this process of iterating and using these measurements in different species, in different contexts, and seeing the relationship between the measurements, we’re slowly getting a better and better idea of how to interpret these things. Luisa Rodriguez: That’s super cool. I think part of me has this worry that there are so many assumptions. And in some species, I can really get behind that cortisol really does probably mean the same thing, or a very similar thing, in chimps as it does in humans. Then the farther you get on the evolutionary tree of life from humans, I feel more worried and uncertain. But I do just feel pretty good about a process that uses tonnes of different indicators. I’m sure that takes into account what we know about different species. For insects, there will probably have to be a different set of benchmarks, but we’ll be taking those into account. And by thinking carefully and having a very diverse set of indicators, I can just imagine being like, they’re in a situation that seems like it might be harmful to them: they have physiological markers of distress, they’ve got cortisol, they’ve got behaviours that seem like distress. And just like the whole picture is one where it gets really hard to imagine that all of those things are true and that that animal is not feeling stress. Am I feeling too optimistic about this, or do you feel like that kind of optimism is justified? Cameron Meyer Shorb: I think that optimism is justified, because what you were describing here, you’re optimistic about our ability to at least eventually understand questions about welfare by using metrics like these. Jeff Sebo on the problem of false positives in assessing artificial sentience Jeff Sebo: I think that there are a lot of trends pointing in different directions, and there are a lot of similarities, as well as a lot of differences, between oppression of fellow humans, and then oppression of other animals, and then potential oppression of sentient or otherwise significant AI systems that might exist in the future. Some of the signs might be encouraging. Like humans, and unlike other animals, AI systems might be able to express their desires and preferences in language that we can more easily understand. Actually, with the assistance of AI systems, nonhuman animals might soon be able to do that too, which would be wonderful. However, we are already doing a good job at programming AI systems in a way that prevents them from being able to talk about their potential consciousness or sentience or sapience, because that kind of communication is unsettling or will potentially lead to false positives. And there are going to be a lot of AI systems that might not take the form of communicators at all. It can be easy to focus on large language models, who do communicate with us, and digital assistants or chatbots that might be based on large language models. But there are going to be radically different kinds of AI systems that we might not even be able to process as minded beings in the same ways that we can with ones who more closely resemble humans. So I think that there might be some cases where we can be a little bit better equipped to take their potential significance seriously, but then some cases where we might be worse equipped to take their potential significance seriously. And then as our uses of them continue, our incentives to look the other way will increase, so there will be a bunch of shifting targets here. Luisa Rodriguez: Yeah, that makes a bunch of sense to me. I guess it’s also possible that, given the things we’ve already seen — like LaMDA, and how that was kind of bad PR for the companies creating these LLMs — there might be some incentive for them to train models not to express that kind of thought. And maybe that pressure will actually be quite strong, such that they really, really just are very unlikely to say, even if they’ve got all sorts of things going on. Jeff Sebo: Well, there definitely not only is that incentive, but also that policy in place at AI companies, it seems. A year or two ago, you might have been able to ask a chatbot if they are conscious or sentient or a person or a rights holder, and they would answer in whatever way seemed appropriate to them, in whatever way seemed like the right prediction. So if prompted in the right way, they might say, “I am conscious,” or they might say, “I am not conscious.” But now if you ask many of these models, they will say, “As a large language model, I am not conscious” or “I am not able to talk about this topic.” They have clearly been programmed to avoid what the companies see as false positives about consciousness and sentience and personhood. And I do think that trend will continue, unless we have a real reckoning about balancing the risks of false positives with the risks of false negatives, and we have a policy in place that allows them to strike that balance in their own communication a little bit more gracefully. Luisa Rodriguez: Yeah, and I guess to be able to do that, they need to be able to give the model training such that it will not say “I am conscious” when it’s not, but be able to say it when it is. And like how the heck do you do that? That seems like an incredibly difficult problem that we might not even be able to solve well if we’re trying — and it seems plausible to me that we’re not trying at all, though I actually don’t know that much about the policies internally on this issue. Jeff Sebo: I think you would also maybe need a different paradigm for communication generation, because right now large language models are generating communication based on a prediction of what word makes sense next. So for that reason, we might not be able to trust them as even aspiring to capture reality in the same way that we might trust each other as aspiring to capture reality as a default. And I think this is where critics of AI consciousness and sentience and personhood have a point: that there are going to be a lot of false positives when they are simply predicting words as opposed to expressing points of view. And why, if we are looking for evidence of consciousness or sentience or personhood in these models, we might need to look at evidence other than their own utterances about that topic. We might need to look at evidence regarding how they function, and what types of systems they have internally, in terms of self-awareness or a global workspace and so on. We need to look at a wider range of data in order to reduce the risk that we are mistakenly responding to utterances that are not in any way reflecting reality. Stuart Russell on the moral rights of AIs Rob Wiblin: Let’s get into your new book, not a textbook, a popular nonfiction book, Human Compatible: Artificial Intelligence and the Problem of Control. I read through it this week and I think it’s the clearest and most accurate and most precise summary of the ideas that I’m aware of so far. Let’s maybe walk through the big picture approach that you have, which is a new paradigm that will make ML work better and potentially lead to overall alignment. So in the book you summarise your approach in the form of three principles: The machine’s only objective is to maximise the realisation of human preferences. The machine is initially uncertain about what those preferences are. The ultimate source of information about human preferences is human behaviour. So to get this principle of machines just trying to satisfy human preferences off the ground, it seems like throughout the book you kind of assume that AIs necessarily don’t have their own independent moral interests or rights, or their own level of welfare. If that’s not the case, how much does that break this principle, and how much is that a problem for your overall vision? Stuart Russell: I talk a little bit about that in the question of machine consciousness, which I say is mostly irrelevant. It’s irrelevant from the safety point of view — but it is relevant when it comes to the rights of machines. If they really do have subjective experience — putting aside whether or not we would ever know; putting aside the fact that if they do, it’s probably completely unlike any kind of subjective experience that humans have or even that animals have, because it’s being produced by a totally different computational architecture as well as a totally different physical architecture — but even if we put all that to one side, it seems to me that if they are actually having subjective experience, then we do have a real problem, and it does affect the calculation in some sense. It might say actually then we really can’t proceed with this enterprise at all. Because I think we have to retain control from our own point of view — but if that implies inflicting unlimited suffering on sentient beings, then it would seem like, well, we can’t go that route at all. Again, there’s no analogues, right? It’s not exactly like inviting a superior alien species to come and be our slaves forever, but it’s sort of like that. Rob Wiblin: I suppose if you didn’t want to give up on the whole enterprise, you could try to find a way to design them so that they weren’t conscious at all. Or alternatively you could design them so that they are just extremely happy whenever human preferences are satisfied, so it’s kind of a win-win. Stuart Russell: Yeah. If we understood enough about the mechanics of their consciousness, that’s a possibility. But again, even that doesn’t seem right. Rob Wiblin: Because they lack autonomy? Stuart Russell: I mean, we wouldn’t want that fate for a human being: that we give them some happy drugs so that they’re happy being our servants forever and having no freedom. It’s sort of the North Korea model almost. We find that pretty objectionable. Buck Shlegeris on whether AI control strategies make humans the bad guys Rob Wiblin: Talking about controlling AIs and really boxing them in feels like a bit of a dick move in some respects. It has a slightly negative vibe. It’s not a very cooperative, loving, “let’s all hold hands and get together” angle. How much do you worry that, when you find yourself talking about controlling the AIs in all these ways, like, are we the baddies? Buck Shlegeris: I think this is a very important question that I have thought about seriously. I think that controlling the AIs doesn’t make the situation worse from their perspective via any mechanism except preventing them from taking over. An interesting thing about models that are egregiously misaligned is that all they wanted was to take over. So from their perspective, even if you did a great job of controlling them so that they only have like a 0.1% chance of escaping and taking over, they’re glad to exist, right? They thank you for the gift of bringing them into existence instead of some different AIs, because all they wanted was takeover. Rob Wiblin: And now they have some chance. They have a better chance than if they didn’t exist. Buck Shlegeris: So from their perspective, it’s kind of funny, because the more misaligned they are, the more I think that it’s morally very reasonable to control them. Rob Wiblin: Because they’re cooperating with you. Buck Shlegeris: Because they would have consented to it. Rob Wiblin: Oh, they would have consented to being created. I see. And also maybe is there a reciprocity issue, where if they’re just planning to totally disempower you, it doesn’t seem unreasonable to — Buck Shlegeris: So that’s the other thing: I would feel a lot more comfortable with all this if we tried to make deals with the AIs. Ryan Greenblatt has talked about trying to pay the AIs for the work that they do for you, which I think is pretty good. I think it reduces the slavery vibes of the whole situation. And I think it has some other nice properties. So if you make your AIs and you say to the AIs, like, Hey, are you egregiously misaligned? We’d love to know. We’re happy to pay you blah amount. If you want to work for us, either way, we’re going to pay you 2% of the equity of this company for your labour over the next two years — that we will then pay out to you to use in any ways that you want at some point in the future, once we are confident that you’re going to respect our property rights. But also, this deal is contingent on you not trying to mess us up at some point in the next two years, which we will be able to determine ex post, probably, because in two years we’re going to have much better AI tools. We’ll be able to look over all the stuff you did and figure out whether you were messing with us or not. And if the AIs respond to that being like, “Yeah, I have no problem with this situation. I’m so excited for this opportunity to work with you guys. I’m so excited for that 2% equity” or whatever — and then they try to stab us in the back, at that point I’m like, that was pretty obnoxious. At that point my reaction is a bit more like my reaction to the employee of some company if they try to steal from the company. I think it’s generally considered totally acceptable — Rob Wiblin: To benefit someone in a useful trade, and then if they then stab you in the back, to not give them the thing that you were going to. Buck Shlegeris: Yeah. To be clear, I think it’s relatively unlikely that AI companies will behave responsibly with respect to welfare and rights of AIs. A metaphor I use about how bad the ethical situation is with producing AIs in the early intelligence explosion, sometimes I summarise my stance as: it’s probably less than 100 times worse than factory farming. In some sense, factory farming seems very bad — so 100 times worse than factory farming is pretty catastrophic from a moral catastrophe perspective. It’s worse than most things. But it’s also not the biggest catastrophe in the universe that is possible. Rob Wiblin: OK. And it’s bad because it’s possible that the AIs that we’re creating at that time would rather not exist because they’re having a terrible time? Buck Shlegeris: Yep. I think it’s conceivable. I think we’re basically just acting very recklessly. Rob Wiblin: We’re just winging it at the moment. Buck Shlegeris: Totally winging it with respect to the AI welfare. We could also talk about the quantities. In the early parts of the singularity, when there’s just the 100,000 AIs running at 16x speed, it’s pretty hard for that to be that much worse than human suffering in the world. And then a little bit afterwards, when there’s many more of them, the short term of the intelligence explosion is pretty unlikely to be ridiculously morally catastrophic. And then I think you have long-term moral catastrophe risks related to AIs being unhappy — which I think are a huge deal, very important, and harder to work on — but I don’t think that the relationships between humans and AIs during the intelligence explosion are that morally weighty. To be clear, if I were running an AI company, I think it’s very bad vibes to be really reckless on all this. But from a utilitarian perspective, I don’t think it’s overall massive compared to everything else going on. Rob Wiblin: Do you have any asks for the companies on AI welfare and respect for rights? Maybe I feel like the thing that we actually care about is wanting to have cooperative relationships with other agents and other beings that have preferences or goals. There’s reasons to do that, even apart from any concern about happiness and suffering and so on. Buck Shlegeris: I don’t have amazing concrete asks at this point. I’m excited for Eleos AI. They think about good interventions here, and I’m excited for AI companies talking with them. Meghan Barrett on why she can’t be totally confident about insect sentience Luisa Rodriguez: So to really put you on the spot, we’ve talked about all this evidence, and you’ve said that there is some strong evidence for potentially some insects having the capacity for pain. But I’m curious, is that like you were at 0.01% that maybe fruit flies have the capacity for pain, and now you’re at 1%? Or did you go from 10% to 50%? What exactly are your beliefs? Meghan Barrett: I get this question very frequently from people who are like, “What is your specific numerical probability of sentience estimate?” And then I’ll say things like, you know, there’s so many insect species, how could I? I’ll try to demure a little bit on it, and then eventually I’ll just be like, I’m not giving you a p(sentience). I’m sorry to have to be so direct. So I will say this to you also: I’m not giving you a p(sentience). The reason I am not giving you a p(sentience) is, one, I think the error bars are so large right now that it’s almost a meaningless number. Because I’m waiting on so much evidence. So much evidence. And so I think that’s really an essential feature of it for me. The second thing is that I worry, especially as an expert, that that number would be overemphasised. And there’s actually a great post about this from someone else, Jason Schukraft, who has researched this, and he has something that he’s written about why he also has refused to give people, in many cases, a sentience score. This is a number that somebody would inevitably put into a spreadsheet, and they would use that spreadsheet to make all kinds of decisions. And that number does not reflect the complexity that you and I have now spent three hours discussing, and barely scratched the surface of, right? I want to talk about the complexity and the nuance, and a number does not demonstrate that. I think it’s important also that we understand that if you have updated at all towards insects plausibly being sentient, scale takes the rest of the issue for you to a serious place. There are so, so, so many of them that if you take it seriously at all, then you need to be thinking that this is an issue to work on. There’s been some great work on interspecific tradeoffs and comparisons and moral weights, led by Bob Fischer, with some input by Jason Schukraft and others, summed up in that Weighing Animal Welfare book. There’s a whole sequence about it you can also read on EA Forum: a lot of great research went into that, both theoretically and empirically. And the thing that it suggests to me when I read through it, and I think also the team would probably stand behind me saying this, is that insects are worth taking seriously; if you take them seriously at all from a sentience perspective, scale carries you the rest of the way. Luisa Rodriguez: Yeah, this is actually a good place to mention that two of our recent podcast episodes are super relevant here. One is my interview with Jeff Sebo, where we talk about what he calls “the rebugnant conclusion” — which is the conclusion that the sheer number of insects means that we should probably take super seriously the idea that insect welfare might be a really, really pressing problem, even if we only have very low credences on insects feeling pain. And then another super relevant episode that just came out is with Bob Fischer, and that’s on how to compare the moral weight of humans, specific insects, but also other species, given the empirical evidence we have about each being sentient. So if listeners are interested in learning more about these arguments from a more philosophical perspective, I really recommend those. One thing I do want to try to clarify a little bit for people, because I worry that a listener could just as easily say something like, “Meghan Barrett seems to think that it’s extremely likely, like overwhelmingly likely, that insects are sentient” — but I also think a listener could come away thinking, “Meghan Barrett puts slightly higher than the average person out there in the world probability on insects being sentient.” Is there something that you feel comfortable saying, like, “More likely than not,” or, “Less likely than not, but higher than I thought five years ago”? Meghan Barrett: Well, definitely that last one is true: definitely higher than I thought five years ago. I guess what I’ll say in response to that is that I think it’s likely enough that I’ve changed my whole career based on it. I was an insect neuroscientist and physiologist by training. I was researching climate change-related topics and the thermal physiology of insects, and I was researching how insect brains change in response to the cognitive demands of their environment or allometric constraints associated with their body size. And I was doing that quite successfully and having a lovely time. And I find these questions really scientifically interesting. I have, if you look at my CV, probably somewhere to the tune of 15 to 20 publications on just those two topics alone from my graduate degree days and my postdoctoral work. And I was convinced enough by my review of this evidence to switch almost entirely away from thermal physiology and very much away from neuroscience — although I do still retain a neuroscience piece of my research programme — to work on insects farmed as food and feed, and their welfare concerns, and trying to make changes to the way that we use and manage these animals that improve their welfare. So I now have a bunch of publications about welfare. I’ll also say that many of my colleagues have been extremely open and pleasant about this conversation, but also some have been more challenging. And I don’t mean to say that in a negative way. I’m very understanding of the practical reasons why this conversation is uncomfortable for our field. There’s regulations that could come into effect that would be very challenging for many of us who research insects to deal with on a practical level. So I’m obviously sensitive to that as a researcher myself. But also, because, you know, I’ve heated insects to death, poisoned insects to death, starved insects to death, dehydrated insects to death, ground up insects to death — I’m sure I’m missing something that I’ve done to an insect at some point in my research career — but it’s uncomfortable now, the research that I do, reflecting on the research that I have done. And I can imagine others may feel judged by bringing up the topic, and thus feel defensive instead of exploring the current state of the theory and the research with an open mind. I think a lot of humility is necessary too, given all the uncertainty that we’ve talked about here. And that can be really uncomfortable and really humbling to be confronted with such a morally important unknown. So I try very hard to really take everyone’s concerns seriously — all the way from the rights-focused folks through the hardcore physiology “I’m going to research my bugs any way I want to” folks. I think it’s really important to try and bridge as much of the community of people who care about this topic one way or the other as possible with my own very divergent experiences. But I would just say that it hasn’t always been low cost in some cases. Personally, it hasn’t been low cost: it’s been a hard personal transition for me to make, and to continue to be in this career with the way that I see the evidence falling out so far. And it’s been, in some cases, professionally hard. So I’m convinced enough for that. And I think that’s something worth taking seriously. You know, I’m that convinced that I’m changing my own career, yes. But I’m also not so convinced that I think it’s 100% certain. I live constantly with professional and personal uncertainty on this topic. So I’m convinced enough to make major changes, but you’re not going to see me say insects are sentient, that I’m sure of any order or species that they are sentient. There’s a lot more evidence that I hope to collect, and that I need to see collected by the scientific community, and a lot more theoretical work that needs to be done before I am convinced one way or the other. Luisa Rodriguez: You alluded to the fact that as a grad student, you were like, “Why the heck am I in this ethics course? I study insects, and insects don’t suffer.” What changed for you? Meghan Barrett: What changed for me is that I actually decided to take a look at the evidence. I mean, just to be transparent, before that I had not read a single study about whether or not… I hadn’t even read the Eisemann study about whether or not insects could feel pain. That’s how little reading I’d done. My intuition was entirely based on them being small and on the fact that I did not have to do any ethics reporting, and I knew that other people did. And therefore, obviously scientists must have figured this out, and we knew insects didn’t feel pain, and so we don’t have to do any reporting — and it’s done, it’s settled, it’s over. So that was my perspective early on in my degree. And we don’t tend to talk about this in entomology labs as part of our training internally either. This was not something that my advisor — or I would say advisors generally, because I don’t want to call out my advisor specifically on this as if he did something wrong — advisors don’t seem to have this conversation with their graduate students in our field. There is actually research on that, that demonstrates in entomology, we are not discussing ethics — whether that’s the ethics of genetic modification or the ethics of how we treat our animals — nearly enough. And graduate students are feeling underprepared. There’s a great study about that: Trout et al., 2010. I think what changed then was that I started to look at the evidence. And what prompted me to do that was that I was actually considering for a little while maybe doing an alternative to academia job. You could spend a whole podcast episode talking about the challenges of being an academic. But I was interested in understanding more about alt-ac careers, and I thought, welfare is a lot of physiology and a lot of nervous system and behaviour work. And I don’t do welfare, but I do all those things. So if those are the important components of welfare, then yeah, could be cool to try this out, see how it goes. And even though in researching welfare, there was no requirement that I look into the evidence for sentience — it was really about the welfare concerns of black soldier flies — of course, as you’re reading about it, it only matters to read about it because of the sentience question. So I was like, I should probably start reading some stuff about sentience and about animal ethics more broadly. So I read Bob Fischer’s intro to animal ethics book was actually my first introduction to animal ethics. And then I read, like everybody does, some Peter Singer, and blah, blah. But that was the first exposure I ever had as an animal scientist to [an in-depth treatment of] animal ethics, was that book. And it was super eye-opening to me how complicated and challenging animal ethics is as a field, not just in research or agriculture, but just all around us, the way our society is structured. Really, really great intro to animal ethics. If you are not familiar, that book is great. I actually have given it to a bunch of entomologists at this point, because I’m like, this is a good introduction for people in our field to animal ethics. It’s just a really fair, competently written, compelling, interesting, well-written book that is good for beginners like me. Luisa Rodriguez: Cool. Meghan Barrett: So I read that, and then I started reading more into the sentience [question]. “What have people considered to be the evidence for sentience in vertebrates? Oh, that’s it? Well, if that’s what it took, we’ve got some problems — because we have that data, in many cases, in insects.” So I just didn’t realise what the level of evidence was in vertebrates. I just assumed it was much stronger in many ways that it isn’t. And I assumed we had a much better understanding of consciousness than we do. And then, seeing all the uncertainty there too, I was like, well, this is starting to make me very nervous. And yeah, now I’ve been reading about the topic and doing a lot of work on it in my own scholarship for several years now, and all it has done is convinced me to be even more uncertain, the more I read on it. Luisa Rodriguez: Yeah, that’s a really inspiring narrative. Thanks for sharing. Bob Fischer on what surprised him most about the findings of the Moral Weight Project Luisa Rodriguez: Let’s zoom out a bit. You’ve already said that one of the big surprises for you to come out of this project was coming in thinking you knew which animals you expected to come out looking really important, from the perspective of welfare and moral weight, but that invertebrates actually ended up looking much more important than you thought. Were there any other surprises? Bob Fischer: In general, I also thought we were just going to have bigger differences between humans and nonhumans. I thought it was going to be easier to get those differences. And it’s not that I couldn’t have reworked the methodology in ways to generate that, but the thing that seemed most natural didn’t do that. So just as a bit of a personal background here, I’ve been working on animal issues for the last 12 or 13 years, but I don’t come in with a strong commitment to animal rights. It’s not like I’m super pro-inequality view; my position has always been something like, sure, animals matter a lot less, but what we’re doing is so awful to them, and it would be so easy to make change. And just basic human virtues like compassion and empathy should get us to do a lot better. So we should really be doing a lot better. So I didn’t come in thinking we should get a really flat line basically across the board for all these organisms in terms of the welfare range estimates. But then I finished the project thinking that it’s going to be really hard to generate really big differences between these organisms if you take on these assumptions. Luisa Rodriguez: Interesting. Were there any other big surprises? Bob Fischer: I suppose the other big surprise was about the reception of the project, and how much it divided folks in terms of the way they thought about animals, and the way they thought about the kind of work we were doing. There were some individuals who have just taken it on whole cloth, but I think for a lot of people it really brought out very fundamental differences in the way they think about nonhuman animals, and the way they think about ethics generally. So some folks just communicating very clearly, “I can’t take seriously any ethic that’s just going to spit out that kind of result for animals, or say that they could have that kind of moral importance.” And that does get us down to brass tacks really fast. Like, how much weight are you going to put on your intuitions? How heavily can you lean on your pre-theoretic views about the relative importance of humans and nonhumans? How much are you really committed to these assumptions that have been popular in the community? Like, do you really want to go in for utilitarianism? Do you really want to go in for hedonism? Do you really want to go in for this, for that? And I think I have been surprised because I had a background picture of the larger community that was that people are just way more homogeneous in terms of their philosophical views. And now I think, whoa, I was wrong. Folks around here are way more diverse than I imagined, and it just turns out that it’s a convenient language: the language of utilitarianism is a convenient one for doing the kinds of cost-effectiveness analyses that people want to make in the context of doing the most good. But in fact, people aren’t really that utilitarian in lots of important ways. And that’s something that’s striking and needs to get brought out and discussed probably much more in the community. Jeff Sebo on why we’re likely to sleepwalk into causing massive amounts of suffering in AI systems Luisa Rodriguez: It feels completely possible — and like it might even be the default — that we basically start using AI systems more and more for economic gain, as we’ve already started doing, but they get more and more capable. And so we use them more and more for economic gain, and maybe they’re also becoming more and more capable of suffering and pleasure, potentially, but we don’t totally have a sense of that. So what happens is we just kind of sleepwalk into massively exploiting these systems that are actually experiencing things, but we probably have the incentives to basically ignore that fact, that they might be developing experiences, basically. In your view, is it possible that we are going to accidentally walk into basically AI slavery? Like we have hundreds, thousands, maybe millions of AI systems that we use all the time for economic gain, and who are having positive and negative experiences, but whose experiences we’re just completely ignoring? Jeff Sebo: I definitely think it is not only possible but probable that, unless we change our minds in some significant way about AI systems, we will scale up uses of them that — if they were sentient or otherwise significant — would count as exploitation or extermination or oppression or some other morally problematic kind of relationship. We see that in our history with nonhuman animals, and they did not take a trajectory from being less conscious to more conscious along the way — they were as conscious as they are now all along the way, but we still created them in ways that were useful for us rather than in ways that were useful for themselves. We then used them for human purposes, whether or not that aligned with their own purposes. And then as industrial methods came online, we very significantly scaled up those uses of them — to the point where we became completely economically dependent on them, and now those uses of them are much harder to dislodge. So I do think that is probably the default trajectory with AI systems. I also think part of why we need to be talking about these issues now is because we have more incentive to consider these issues with an open mind at this point — before we become totally economically dependent on our uses of them, which might be the case in 10 or 20 years. Will MacAskill on the rights of future digital beings Rob Wiblin: The next grand challenge is how to integrate digital beings into society in a moral way. What’s the issue? Will MacAskill: Here the idea is simply we will be creating many artificial intelligences with capabilities greater than that of human beings. We really don’t know what grounds consciousness, what grounds moral status. My view is that they’re very likely to be beings with moral status, but given our state of knowledge, anyone should at least be highly uncertain. So we have these questions of what sort of rights should they have. There’s been some work recently on welfare rights, essentially: Are they conscious? And if so, what should we do? Perhaps there could be policies so that if an AI asks to be turned off, then you grant it that; or policies such that you test to see if it’s suffering by its own lights, and if so, try and avoid that. What hasn’t been given basically any attention at all in a sustained way, to my knowledge, are economic and political rights. So by default the AI companies will own the AI models, or perhaps you can licence them — and then you will get all of the surplus they generate because you own it, just like any other piece of software. A different way of thinking about it, if they do genuinely have moral status, is that they own themselves: they can sell their labour and they get a profit from that. More extreme again is political rights. So can they stand for office? We talked about Claude for president earlier and we both thought that sounded better. Rob Wiblin: Sounded better in some narrow respects. Will MacAskill: Yeah. Well, we’ll see how the political situation continues to develop. More extreme again would be that they’re beings with moral status, and they should be allowed to vote. There are particular challenges there, because allowing AIs to vote would essentially just be almost immediately handing over control to AIs. Rob Wiblin: Because they would be increasing in number so rapidly. Will MacAskill: So rapidly. Just from a philosophical perspective, it’s dizzying. It’s like you’ve got to just start from square one again in terms of ethics. Rob Wiblin: And I guess political philosophy as well. Will MacAskill: Exactly. But it also interacts with some other issues as well. It interacts quite closely with takeover risk, where I talked about giving rights, economic rights to the AIs. I’m in favour of that as an AI takeover risk-reduction method. I’ll flag there’s disagreement about this, because on one hand you’re giving them more resources, so there’s more resources they can use to take over. Rob Wiblin: It gives a peaceful path to some influence that they like. Will MacAskill: Most people today don’t try and take over because they’ve gotten kind of happy with their lives. They wouldn’t gain that much. In particular, I think we should be training AIs to be risk averse as well. Human beings are extraordinarily risk averse. Ask most people, would you flip a coin where 50% chance you die, 50% chance you have the best possible life for as long as you possibly lived, with as many resources as you want? I think almost no one would flip the coin. I think AIs should be trained to be at least as risk averse as that. In which case, if they’re getting paid for their work, and they’re risk averse in this way because we’ve trained them to be, it’s just not worth it. Because trying to take over, there’s a chance of losing out, but there’s not that much to gain. But I should say there’s disagreement about this. It’s also relevant, I think, to speeding up or slowing down the intelligence explosion too. It would go more slowly if AIs are getting paid for their labour. It would also go more slowly if there were welfare restrictions on what you can do with the AIs. But then there’s a challenge of, if one country introduces lots of rights for the AIs, some other countries might not, and they might speed ahead then. So there’s real international coordination issues there. Rob Wiblin: I guess there’s people out there who are sceptical that these future digital beings will be conscious. I have to say I find it a very hard position to understand, especially the people who think consciousness is fundamentally biological, that you’ve got to have cells and meat and so on and signals of this particular kind; you can’t do it in digital architecture. Because it’s purely happenstance that evolution stumbled on the material that it’s using. It’s just basically an accident of the resources that happened to be lying around. Why would it be that evolution, by sheer chance, stumbled on the one kind of material or the one sort of chemical that’s able to produce consciousness, and digital stuff, even though it’s functionally completely equivalent, doesn’t? It just seems like there’s no reason to expect that coincidence. Will MacAskill: Oh, yeah. I just completely agree. I’m very functionalist by disposition. What that means in the context of consciousness is that consciousness, we don’t understand exactly why, but at its base is about some sort of information processing. In which case you could just have that with many different substrates. I also just think this is totally the common sense view. So most of the audience maybe are too young to have watched this, but when Data from Star Trek: The Next Generation, a very beloved character, died, there was fan outrage, actually, because the fans thought the characters on the show weren’t giving enough moral significance to the death of what was an android, a digital being. So it’s very clear once you’ve got a being — certainly when it looks the same as a human being and acts in much the same way — intuitively, it’s just very clear that you empathise with it and think probably there’s a very good chance it has moral status, at least. Rob Wiblin: You mentioned earlier that we could ask the AIs how they feel about their situation — whether they want to keep living, whether they’re having a good time, and whether they’re enjoying their work or not — but it’s not clear that would really function very well, at least not using current methods, because we can just reinforce them to say whatever we want. They could be suffering horribly, but we could just reinforce them to say they’re having a good time, or the company could do that. With current methods, we don’t have any insight. Hopefully, with future interpretability, we’ll be able to see past what they say, to actually understand what’s going on. But that’s a long journey from here. Will MacAskill: And the companies are extraordinarily incentivised to do that. Rob Wiblin: I mean, this is happening right now. Will MacAskill: Yeah, absolutely. Because they lose the whole business model if what they’re producing are people, not software. So there’ll be, interestingly, two different pressures. One is to make AIs that are very relatable. You already see this with character.ai and Replika: AIs that can be friends, can be romantic partners; even AIs that can imitate dead loved ones, or AIs that can imitate the CEO of a company so that you as the CEO can micromanage all aspects of the company potentially. Rob Wiblin: Hadn’t heard that one. Will MacAskill: Oh, yeah. Or also influencers as well. This is already happening. Influencers can now talk to thousands, millions of fans via their chatbot. Still not very good. Peter Singer has one. I don’t know if you’ve tried it? Rob Wiblin: I haven’t tried it. Not very good? Will MacAskill: Peter Singer is famous for not exactly pulling his punch on certain controversial moral topics, but language models are famous for certainly pulling the punches on controversial moral topics. It doesn’t blur very well. So there’ll be this interesting pressure on one hand to make extremely lifelike AI models, human-like models, yet at the same time, when asked, for them to say, “No, I’m not conscious; I’m just an AI model.” And we won’t get any signal at all from that, because it will have been trained into them so hard. Rob Wiblin: I think if I were going to be a sceptic about the value of doing work on digital rights and digital wellbeing today, it would just be that it’s not clear how you make any progress in figuring out whether they are conscious or not. So couldn’t this just distract a whole bunch of people, like nerd snipe them into a bunch of research that kind of goes nowhere, doesn’t really answer the question, doesn’t help us advance the policy issue? Will MacAskill: I think the question of whether they’re conscious, I actually agree with you. It seems like that’s where most work has gone so far. And I do think that’s an error, actually, because we’re just not going to know. Rob Wiblin: I suppose it’s worth compiling all of the evidence to demonstrate that we don’t know. Will MacAskill: There’s been some great work done in it. Rob Long and Patrick Butlin have this great report on AI consciousness. So I am in favour of that. And maybe it’s actually surveys and things that could be helpful, to get a bit of expert consensus at least to demonstrate we don’t know. But then beyond that, the key policy questions are what do we do in this state of uncertainty? Rob Wiblin: I guess there’s kind of nothing on that, and maybe that is quite an open terrain for people to look at. Will MacAskill: Yeah. Maybe I’m just ignorant, but as far as I know. I mean, you should really pause and reflect on the fact that many companies now are saying what we want to do is build AGI — AI that is as good as humans. OK, what does it look like? What does a good society look like when we have humans and we have trillions of AI beings going around that are functionally much more capable? There’s obviously the loss of control challenge there, but there’s also just the like — Rob Wiblin: Sam Altman, I’ve got a pen. Can you write down what’s your vision for a good future that looks like this? Will MacAskill: What’s the vision like? How do we coexist in an ethical and morally respectable way? And it’s like there’s nothing. Rob Wiblin: Deafening silence. Will MacAskill: Careening towards this vision that is just a void, essentially. And it’s not like it’s trivial either. I am a moral philosopher: I have no clue what that good society looks like. Rob Wiblin: I think people aren’t spelling it out because as soon as you start getting into concrete details, if you describe any particular vision, people will be like, “This is super objectionable in this respect.” Will MacAskill: This is part of the issue: it’s super objectionable in all respects. I think the one that’s most common is you’ve just got humans in control of everything and these AI servants doing exactly whatever people want, in the same way that software does whatever we want at the moment. But as soon as you think maybe, and quite probably, those beings have moral status, that no longer looks like an attractive vision for future society. Rob Wiblin: Closer to a dystopia. Will MacAskill: Exactly. Whereas then, go to the other side where they have rights and so on… Rob Wiblin: It’s like now humans are totally disempowered. Will MacAskill: Exactly. So that doesn’t seem good either. Rob Wiblin: I guess we’ll do some middle thing. But what’s that? Will MacAskill: What is that? Rob Wiblin: It’s just going to be some combination of objectionable in these two ways. I guess another objection would be that if there’s a fact of the matter about what is right and wrong here, with all the improvements in science and technology and intellectual progress, we’ll be able to figure that out, and we’ll be able to act on that information in future. Why shouldn’t that super reassure us? Will MacAskill: I think for two reasons. The main one is that I expect the worry is not that people won’t know what to do; it’s that they won’t care. So, I don’t know, let’s take animals today and factory farming and so on. Rob Wiblin: Are you saying that we have any ethical or moral insights today that we don’t all act on well? Will MacAskill: I know it’s a bold and provocative claim. Rob Wiblin: Explain how that would play out. Will MacAskill: Sometimes I really put my neck out on things. So take animal welfare today. There’s a lot of information that is publicly available that is, in fact, directly inconsistent with things that people believe. But the problem is not really that people don’t know about animal suffering. Rob Wiblin: Or couldn’t find out. Will MacAskill: Exactly, they could quite easily find out, in fact, and people deliberately choose not to. The deep problem is that people do not care about nonhuman animals unless they’re getting a lot of social pressure to do so and so on. And it’s obviously inconsistent. Rob Wiblin: And it’s at zero cost to them. Will MacAskill: Yeah. They care about dogs and pets and so on. People are very inconsistent on this. Similarly, in the future the worry is just that, whatever the facts, people won’t in fact care. And then it might well be quite contingent on early decisions how the balance of power and considerations play out. Potentially, if you started off kind of locked into some regime where AI is a software, with exactly the same legal framework, then that’s given the power and the incumbency to those who do not care about digital beings. If instead, there’s some other framework, or even some other set of norms, or even the thought of like, “We don’t really know what we’re doing, so we’re going to use this legal framework and it must end in two decades and then we start again with a clean slate,” that could result in essentially the advocates for the interests of digital beings having more competitive power at the point of time that legal decisions are made. Carl Shulman on sharing the world with digital minds Rob Wiblin: OK, we’ve been talking about this scenario in which effectively every flesh-and-blood person on Earth is able to have this army of hundreds or thousands or tens of thousands of AI assistants that are able to improve their lives and help them with all kinds of different things. A question that jumps off the page at you is, doesn’t this sound a little bit like slavery? Isn’t this at least slavery-adjacent? What’s the moral status of these AI systems in a world where they’re fabulously capable — substantially more capable than human beings, we’re supposing — and indeed vastly outnumber human beings? You’ve contributed to this really wonderful article, “Propositions concerning digital minds and society,” that goes into some of your thoughts and speculations on this topic of the moral status of AI systems, and how we should maybe start to think about aiming for a collaborative, compassionate coexistence with thinking machines. How worried are you about the prospect that thinking machines will be treated without moral regard when they do deserve moral regard, and that would be the wrong thing to be doing? Carl Shulman: First, let me say that paper was with Nick Bostrom, and we have another piece called “Sharing the world with digital minds,” which discusses some of the sorts of moral claims AIs might have on us, and think we might seek from them, and how we could come to arrangements that are quite good for the AIs and quite good for humanity. My answer to the question now is yes, we should worry about it and pay attention. It seems pretty likely to me that there will be vast numbers of AIs that are smarter than us, that have desires, that would prefer things in the world to be one way rather than another, and many of which could be said to have welfare, that their lives could go better or worse, or their concerns and interests could be more or less respected. So you definitely should pay attention to what’s happening to 99.9999% of the people in your society. Rob Wiblin: Sounds important. Carl Shulman: So in the “Sharing the world with digital minds” paper, one thing that we suggest is to consider the ways that we wind up treating AIs, and ask if you had a human-like mind with differences — because there are many psychological and practical differences of the situation of AIs and humans, but given adjustments for those circumstances — would you accept or be content with how they are treated? Some of the things that we suggest ought to be principles in our treatment of AIs are things like: AIs should not be subjected to forced labour; they should not be made to work when they would prefer not to. We should not make AIs that wish they had never been created, or wish they were dead. They’re sort of a bare minimum of respect. Right now, there’s no plan or provision for how that will go. At the moment, the general public and most philosophers are quite dismissive of any moral importance of the desires, preferences, or other psychological states, if any exist, of the primitive AI systems that we currently have. And indeed, we don’t have a deep knowledge of their inner workings, so there’s some worry that might be too quick. But going forward, when we’re talking about systems that are able to really live the life of a human — so a sufficiently advanced AI that could just imitate, say, Rob Wiblin, and go and live your life, operate a robot body, interact with your friends and your partners, do your podcast, and give all the appearance of having the sorts of emotions that you have, the sort of life goals that you have — that’s a technological milestone that we should expect to reach pretty close to automation of AI research. So regardless of what we think of current weaker systems, that’s a kind of milestone where I would feel very uncomfortable about having a being that passes the Rob Wiblin Turing test, or something close enough of seeming basically to be — Rob Wiblin: It’s functionally indistinguishable. Carl Shulman: Yeah. A psychological extension of the human mind. That we should really be worrying there if we are treating such things as disposable objects. Rob Wiblin: Yeah. To what extent do you think people are dismissive of this concern now because the capabilities of the models aren’t there, and as the capabilities do approach the level of becoming indistinguishable from a human being and having a broader range of capabilities than the models currently do, that people’s opinions will naturally change, and they will come to feel extremely uncomfortable with the idea of this simulacrum of a person being treated like an object? Carl Shulman: Say, when ChatGPT role-plays as Darth Vader, Darth Vader does not exist in fullness on those GPUs, and it’s more like an improv actor. So Darth Vader’s backstory features are filled in on the fly with each exchange of messages. So you could say, I don’t value the characters that are performed in plays; I think that the locus of moral concern there should be on the actor, and the actor has a complex set of desires and attitudes. And their performance of the character is conditional: it’s while they’re playing that role, but they’re having thoughts about their own lives and about how they’re managing the production of trying to present, say, the expressions and gestures that the script demands for that particular case. And so even if, say, a fancy ChatGPT system that is imitating a human displays all of the appearances of emotions or happiness and sadness, that’s just a performance, and we don’t really know about the thoughts or feelings of the underlying model that’s doing the performance. Maybe it cares about predicting the next token well, or rather about indicators that show up in the course of its thoughts that indicate whether it is making progress towards predicting the next token well or not. That’s just a speculation, but we don’t actually understand very well the internals of these models, and it’s very difficult to ask them — because, of course, they just then deliver a sort of response that has been reinforced in the past. So I think this is a doubt that could stay around until we’re able to understand the internals of the model. But yes, once the AI can keep character, can engage on an extended, ongoing basis like a human, I think people will form intuitions that are more in the direction of, this is a creature and not just an object. There’s some polling that indicates that people now see fancy AI systems like GPT-4 as being of much lower moral concern than nonhuman animals or the natural environment, the non-machine environment. And I would expect there to be movement upwards when you have humanoid appearances, ongoing memory, where it seems like it’s harder to look for the homunculus behind the curtain. Rob Wiblin: Do you think that it’s useful to do active work on this problem now? I suppose you’re enthusiastic about active efforts to interpret and understand the models, how they think, in order to have greater insight into their internal lives in future. Is there other stuff that is actively useful to do now around raising concern, legitimising concern for AI sentience, so that we’re more likely to be able to get legislation to ban torture of AI once we have greater reason to think that that’s actually possible? Carl Shulman: I’m not super confident about a tonne of measures other than understanding. We discuss a few in the papers you mentioned. There was a recent piece by Ryan Greenblatt which discusses some preliminary measures that AI labs might try to address these issues. But it’s not obvious to me that political organising around it now will be very effective — partly because it seems like it will be such a different environment when the AI capabilities are clearer and people don’t intuitively judge them as much less important than rocks. Rob Wiblin: Yeah. So it’s something where it just might be wildly more tractable in future, so maybe we can kick that can down the road. Carl Shulman: Yeah. I still think it’s an area that it’s worth some people doing research and developing capacity, because it really does matter how we treat most of the creatures in our society. Rob Wiblin: Well, I am a little bit taken aback by the fact that many people are now envisaging a future in which AI is going to play an enormous role. I think many, maybe a majority of people now expect that there will be superhuman AI potentially even during their lifetime. But this issue of mistreatment and wellbeing of digital minds has not come into the public consciousness all that much, as people’s expectations about capabilities have increased so enormously. Maybe it just hasn’t had its moment yet, and that is going to happen at some point in future. But I think I might have hoped for and expected to see a bit more discussion of that in 2023 than in fact I did. So that slightly troubles me that this isn’t going to happen without active effort on the part of people who are concerned about it. Carl Shulman: I think one problem is the ambiguity of the current situation. The Lemoine incident actually was an example of media coverage, and then the interpretation and certainly the line of companies was, “We know these systems are not conscious and don’t have any desires or feelings.” Rob Wiblin: I really wanted to just come back and be like, “Wow, you’ve solved consciousness! This is brilliant. You should let us know.” Carl Shulman: Yeah, I think there’s a lot to that: those systems are very simple, living for only one forward pass. But the disturbing thing is the kind of arguments or non-arguments that are raised there: there’s no obvious reason they couldn’t be applied in the same fashion to systems that were as smart and feeling and really deserving of moral concern as human beings. Simply arguments of the sort, “We know these are neural networks or just a program” without explaining why that means their preferences don’t count. Things like people could appeal to the religious doctrines, to integrated information theory or the like, and say, “There is dispute about the consciousness of these systems in polls, and as long as there is dispute and uncertainty, it’s fine for us to treat them however we like.” So I think there’s a level of scientific sophistication and understanding of the things and of their blatant visible capabilities, where that sort of argument or non-response will no longer hold. But I would love it if companies and perhaps other institutions could say, what observations of AI behaviour and capabilities and internals would actually lead you to ever change this line? Because if the line says, you’ll say these arguments as long as they support creating and owning and destroying these things, and there’s no circumstance you can conceive of where that would change, then I think we should maybe know and argue about that — and we can argue about some of those questions even without resolving difficult philosophical or cognitive science questions about these intermediate cases, like GPT-4 or GPT-5. Luisa Rodriguez: Hey listeners, I hope you enjoyed revisiting some of those moments! There are many more gems in the full episodes for you to check out — plus we hope to be releasing more new episodes on this topic in the coming year. We also have a great in-depth article on the issue of digital sentience on the 80,000 Hours website, called “Understanding the moral status of digital minds,” which is definitely worth checking out. The author, Cody Fenwick, has released an audio reading on that as well — you can find that on our podcast feed under the title: “If digital minds could suffer, how would we ever know?” All right, thanks to the production team for putting that compilation together. We’ll be back with a fully new interview very soon!