Latest / The Information Bottleneck / Surya Ganguli: The Physics of Intelligence

Transcript

Surya Ganguli: The Physics of Intelligence · The Information Bottleneck

• Ravid Shwartz-Ziv: Hi everyone and welcome back to the Information Bottleneck podcast. And today we have Surya Ganguli. He's a professor at Stanford and VP at the General Catalyst. And he's doing a lot of work in many topics, both in deep learning, neuroscience, statistical mechanics, and all the connections between them. We'll try to cover as many as topics as we can. Hi Syria, thank you for joining us. Surya Ganguli: Hey. Hey, good great to see you, Raveed, and great to meet you, Alan. Allen Roush: Yeah, great to meet you as well. Ravid Shwartz-Ziv: So I think we can start with maybe like the basic or like how you see deep learning or what is the the point of view that you actually think we should view deep learning and how it's related to the brain. Surya Ganguli: Yeah. Yeah. So I you know, my background was I I I actually trained in string theory. Like like the way I ended up there is I I I wanted to do AI as a freshman at MIT. and I I took my first AI course. This was the end of the nineties, so it was a very different time. And I asked my profess professor, shouldn't we try to reverse engineer the brain? And he said, No, no, no, just ignore the brain. we just need to figure out the software program the brain is running. And even as a freshman, that didn't seem right to me. So then I was kind of in the wilderness. I I kind of wasn't sure I wanted to do AI the way it was practiced then. I ended up sort of accidentally triple majoring in math, physics, and and electrical engineering computer science. And then I didn't know what I wanted to do, so I went to I decided on a whim to do my PhD in physics, and so then I I I kind of studied string theory because obviously that was the most fundamental thing to study. I didn't know what I was doing back then really. I did string theory, I don't regret it, I do it over again, it was super fun. But then I kind of took my first theoretical neuroscience course at the end of my PhD, and I decided that's what I wanted to do. And then when I took on my professorship position, this was in around 2012, AI was taking off. And I realized all of the emergent properties that I studied in complex physical systems, like in condensed matter physics and string theory, that allowed me to analyze biological neural networks, also allowed me to analyze artificial neural networks and understand them and improve them. So kind of this Thinking of emergent properties in complex systems is what unifies my study across physics, neuroscience, and AI. And I I find them all extremely interesting. Ravid Shwartz-Ziv: So w what what do you think? Like what is the like the connection between them or like what are the basic fundamental building blocks that we actually Surya Ganguli: Yeah. Ravid Shwartz-Ziv: should look on when we try to analyze the AI or like neuroscientists? Surya Ganguli: Yeah, exactly. So so so there's deep analogies between statistical physics and and AI and just generally neuroscience and optimization. So if you think about what physical systems are trying to do, if you cool them down, they're trying to minimize their energy over a high-dimensional landscape of, say, atomic positions or spins or something like that. So energy minimization is a fundamental driver of the dynamics of physical systems. When you're trying to train a neural network, you're trying to minimize the loss over a high-dimensional space of neural network parameters. So, in this analogy, energy in physics is like loss in in AI. And moving atomic positions is like changing atomic parameters, or or spins are like parameters as well. So so basically that's the kind of analogy, and then Physics has developed a lot of tools to analyze energy minimization in complex systems. When does it get stuck? When does it not? What is the geometry of energy landscapes that can translate directly into the geometry of lost landscapes? Sort of an early paper I had in 2014 showed that in in generic high-dimensional landscapes, local, minimum and high error just don't exist. It was always very confusing to people in AI why we don't get stuck in local minimum and high error. The basic answer is they generically don't exist because in a high-dimensional landscape, let's say you're optimizing for a million variables. If the gradient vanishes, what are the chances that the loss goes up in all one million directions? The answer is exponentially rare and exponentially small in the dimensionality, unless you're already near the bottom. So basically local minima all exist at only at low errors, right? Close to that of global minimum. So kind of deep connections like that are are sort of allow you to sort of transport techniques in physics to analyzing neural networks. And actually we have a whole grant, we want a kind of assignments collaboration on the physics of learning and neural computation. If you go to physics, physicsoflearning.org, you can see sort of a a whole bunch of work that me and several other PIs have done in this center. Allen Roush: do you have any opinion on how important gradients are in general? Like gradient-free machine learning techniques, like for example, neuroevolution, forward pass exclusive stuff. they seem to have been out of vogue for a while, but I claim that the forward pass in models is easily parallelizable. So I I'm fascinated by the lack of this work. Do you do you agree with that? Surya Ganguli: Yeah, yeah. It's I think it's slower, right? Gradient free methods are just slower than gradient methods. The comparison to the mapping to sort of the biological world, you can think of evolution as gradient free, right? And it took five hundred million years to go from like all of vertebrae brain evolution took place over five hundred million years. That's a long time, right? whereas human learning is actually even faster than what we can currently achieve with gradient-based learning. So gradients are like a very weak supervision that give you a direction to go. but I think when we train humans, like we tell them the algorithm, right? We give them more than just gradients. Like when I teach my son math, I tell him the algorithm for doing math, and he immediately length generalizes. So I think there's a kind of a hierarchy of information that we can use to train. systems. gradient free is like evolution. It's slow. Gradients and machine learning are much faster, but still like they take a long time to learn. and then there's whatever we do with humans, which is I think even more sophisticated than gradients. Allen Roush: And and and d do do you think there's a a di like a creativity or diversity trade off here? Because I I it seems that di gradient free methods, if you th if you would c you know, argue evolution is gradient free, the diversity is extraordinarily high, at least by our, Surya Ganguli: Yeah. Allen Roush: you know, subjective measure of it. So I guess like it it like from the perspective of if you want to make zero assumptions about the global solution space, Surya Ganguli: Yeah. Allen Roush: D do you think it has any value there? Surya Ganguli: Yeah, I I think what's happening with evolution and why you get such diversity is that you get these arms races where you're from the any from the perspective of any species, its fitness landscape is never static because it's co-evolving with other species that are competing with it. So as one species gets better, it affects the evolution of another species. And then you have like l a large number of species all co-evolving. And this competitive arms race between many, many different species. with different objectives and overlapping niches, creates an entire ecosystem which has a huge amount of diversity, right? It's not like there's one loss function that the entire ecosystem is optimizing. So you get this kind of growth of ecosystem diversity and complexity, which are actually legitimate open problems of study in the field of ecology. And actually there's a lot of physicists working in that field looking at how you can get open-ended diversity out of systems and open ended growth of complexity. So I think that evolutionary dynamics arms races are something that are not currently in AI systems at the moment. Ravid Shwartz-Ziv: So why is we think like we we see like so many like examples of like superhuman models, right? That like agents are better in math, for example. Like do you think this Surya Ganguli: Yeah, yeah. Ravid Shwartz-Ziv: is something that like we actually like make something better than the evolution? Like I don't know, billions of years of evolution? Surya Ganguli: Yeah, I I think that is sort of scaling up search. The reason AI has done really well in math and coding is because they're working in verifiable domains where you can do reinforced alerting against a correct reward signal. And then you can also do test time scaling with verification in the loop. So for example, in theorem proving, you know, you you could have LMs guess the next step and approve. The verifier checks the next step. If it's correct, the LLM keeps going. If it's incorrect, it can backtrack. So this loop between the LLM and a verifier is like a search process down a search tree. And as soon as you can turn a problem into search, you can achieve superhuman capabilities because LLMs can search much faster, right? The magic is that like brute force search on a proof tree would never work. Because the branching factor and depth of the proof tree, the set sp space of possible incorrect incorrect proofs, is so astronomically large. I think one of the magic questions of our age is why did LLMs get good enough to be able to guess the next step of a proof, such that LLM guided search with verification in the loop gets us much better solutions than humans can find at the moment, especially, for example, in counterexamples to AirDosh problems and things like that. So This combination of learning and search is incredibly powerful. And it will be powerful in other domains too, like material science discovery, where there the verification is a little bit harder because you have to do experiments in the loop. But there's companies that are working on that now. it'll be yeah. Ravid Shwartz-Ziv: Yeah, okay. We we talked like a few weeks ago with Max, Max Welling that this is exactly what what Surya Ganguli: yeah, yeah, yeah, I know Matt as well, yeah. Ravid Shwartz-Ziv: they're doing. Yeah. but like okay, so like we also talked with Julia Camp that like e she is like studying and and and trying to understand like math in in this context and she basically s said that like look math is a solved problem like at least like the math that we that we all know that like you try you suggest theorem and try to to prove it. from one hand and from the other hand, like we talked with with Daphne, Daphne Coller, also from Stanford, that you're they're doing drug discovery. And she said, look, like the the the main button like here is is not like like we can accelerate like real time the the the actual experiments and to actually give the drugs to to to humans. so we have the like these like these two two sides. So What do you think? Do you think like like most of our problems are closer to math or like to drug discovery? Most of the interesting problems. Surya Ganguli: Yeah. I actually know the numbers in drug discovery because I used to sort of look into the space a little bit. If you think about small molecule drug discovery, there's roughly about 10 to the 80 possible small molecules. Of those, 10 to the 40 of them are sort of biologically like. you know, and of those, there's still a small fraction that are useful drugs, right? That are also safe and non-toxic and so forth. So if you think about drug discovery, it's a needle in a haystack problem, right? The common and so imagine searching in a space for maybe, I don't know, 10 to the tenth useful drugs out of 10 to the 80 small molecules, right? We would have to have really good LLM guessers to search in that space. but you can sort of do, you know, computational methods for eradication might work, but ultimately the arbiter, the verifier is a clinical trial, right? and then there's earlier steps before that, sort of preclinical work and animals and things like that. and then in vitro work is even earlier than that. So as soon as you can connect AI to labs in the loop, then you can start to speed up that cycle of guessing and checking in biology. that's why math is the the first space where you've you've gotten like big advances. Because the verification can be algorithmically done within machines very, very fast. I think You know, if you go to other domains like I don't know, other problems in society, you know, legal work I think is somewhere in between. the laws are sometimes a bit ambiguous, which is why we need courts to weigh in so often. but yeah, I I think what you have with AI is you point it as a tool to those situations where verification is easy, right? And that's where it'll work. Ravid Shwartz-Ziv: So what like okay let's try to okay let's try to improve and and what do you think about how we actually can improve it and like like you know pushing it to more and more tasks. So we have we you have a recent work about scaling loss and like you argue that you can beat power law scaling and get closer to exponential. can you give us like the the intuition? Surya Ganguli: Yeah, so so then Ravid Shwartz-Ziv: At least for why pruning can change the exponent. Surya Ganguli: Yeah, so so there's we we actually have some later work where we explain why you get power laws. And then we have some earlier work where we show you can how you can beat them. So maybe let's start with the later work, we explain how we got power laws. So these especially like data power laws, right? So something that fascinates me is how much data do you need to learn, right? And one of the early results from OpenAI, you know, about six years ago by Jared Kaplan and colleagues, was these scaling laws for For data, for example. So if compute and trading time are not bottlenecks and model size are not bottlenecks, you can look look at loss as a function of training data. So right, so if this is the amount of training data, this is a loss, you get a power law with some exponent. The funny thing about this exponent is it's quite small. So it's a very shallow fall-off, right? and remarkably, like despite the fact that these scaling laws have captured the imagination of industry motivated significant investments in. in a compute and data collection and so forth, we've never had a theory for why those exponents take the values that they do on any language model on any realistic data set, right? And you might ask, what did the exponents depend on, right? Do they depend on sort of architecture or whatnot? So we recently showed that these scaling laws only depend on the statistical structure of natural language itself, right? And the basic idea is if you're trying to compute loss as a function of the amount of data, what is limiting you in getting a lower loss at a given amount of data? The basic idea is correlations between tokens separated by a certain amount of time decay with the time, right? So, so you know, like what I said 10 sentences ago will be less correlated with what I said one sentence ago, right? Now, to detect weaker and weaker correlations, you need more and more training data. So that implicitly means that at a given amount of training data, if this is sort of the next token and this is time into the past, as you get more and more training data, you get a certain time into the past that you can look back at usefully to predict the next token. But if you get more training data, your time horizon increases. So basically, there's a data-dependent time horizon over which an LLM can look back into the past to usefully predict the next token. We were able to Compute the data-dependent time horizon by looking at how token correlations decay as a function of time. And that gives you an exponent that you can measure in language. The second thing is, let's say you could back look back a certain time horizon in the past. You could ask what's the best you could possibly do to predict the future? You could the best you could do depends on how well you use the tokens in your time horizon. Let's say you use them optimally. Then you can show that the irreducible loss or cross-entropy loss in predicting the next token given the time horizon is the conditional entropy of the next token given a conditioning window given by the time horizon. So then it's just the conditional entropy conditioned on a number of tokens back in the past given by the time horizon. That's our theory for the scaling law. You can measure these conditional entropies in data as well using a very powerful language model. And then we get a we we get a theoretical exponent for the scaling law, and we compare it to the actual scaling law, and there's a perfect match on GPT two stu scale models, you know, the types of stuff we can train in academia. So so that that's actually the ver six years after they were discovered, that's the very first theory of why the scaling exponents take the values that they do. I don't know if that made sense or not. Like if Ravid Shwartz-Ziv: And yeah, and but like what are like how why like okay, or like how it depends on the data, like what are the properties of the data that actually can like make it like change these exponents, right? Surya Ganguli: Yeah, exactly. So I'll tell you exactly. So I'll tell you the two key properties. Right. So if you look at like the the next token and a token end time steps in the past, let's say your vocabulary size is say fifty thousand, like in bytepair encoding, you can measure the token token correlation matrix. It's a fifty thousand by fifty thousand matrix, and it has some maximal singular value. So that maximal singular value decays as a function of time into the past. And that decay occurs with a as a power law with some exponent. Okay? So that's the correlation decay exponent, is just a property of your data. The second exponent is you can look at the conditional entropy of the next token conditioned on n previous tokens. And as n gets bigger, I get to condition more. So because every time you condition on more, the entropy decays. H sub n, the entropy of the next token given n previous tokens in the past, decays as a power law with n. So that's a second exponent. That's the entropy decay exponent. It's another property of language itself. If you give me the correlation decay exponent and the entropy decay exponent, I can compute for you the data scaling law exponent, which is the loss of a language model as a function of the amount of training data. And that's it. It's a very simple. Ravid Shwartz-Ziv: But but this is but it's not related to the model itself. Surya Ganguli: No, it's not related to the model itself. So we're so remember, the key assumption to get our theory is that the only thing that limits model performance is a data-dependent time horizon where the time horizon grows further into the past as the data increases, right? And then we assume the model acts optimally using the tokens within that time horizon. So this is suggesting that model capacity is not a bottleneck and training time is not a bottleneck. So, in a situation where model capacity and training time are not a bottleneck, it's just the amount of data that's a bottleneck, then the model doesn't matter as long as it's good enough, right? So that's why we get a scaling exponent that doesn't depend on the architecture, it doesn't depend on the amount of training time, because none of those are bottlenecks. All that matters is the structure of the data. To get chinchilla-like scaling laws, where you need to scale up both model size and data set size. That's a much harder problem, and that remains open. We don't have a theory of chinchilla type scaling laws. Ravid Shwartz-Ziv: Okay, so in optimal, like in the optimal case when everything is infinity and you just need to to look on the data, you have these laws, right? now what will happen like now when we want to actually like make like data pruning and like actually like to choose. Now Surya Ganguli: Yeah. Ravid Shwartz-Ziv: I I I want actually like to choose like the best type of data. Like you have any special suggestions? Surya Ganguli: Yeah. Yeah. So so we had yeah, we had an earlier paper on data pruning where we said do we have to be content with power laws? And in particular, we're wondering why are the power laws so shallow, right? Like and the the basic idea is it's because the data sets are so redundant, right? So if you have let's say you train in a billion data points and they're random data points, right? This could be for machine vision, it could be for LLMs or whatnot. And then you just add one more data point. What are the chances that the next data point will tell you anything that's new compared to the billion you already have? Well, if they're random, it's not going to tell you much that's new. Basically, the new data point is redundant with respect to the billion, two billion or billion previous data points. Right? So the key idea is then coming up with non-redundant data sets where you try to let's say you could construct a non-redundant data set where each data point Gave you orthogonal or new information compared to all previous data points. If you could do that, then the addition of each data point should cut your error in half, or some multiplicatively drop your error. So if each data point multiplicatively drops your error, you'll get an exponential scaling law. So if this is data and this is loss, a power law would be straight and an exponential would curve down much faster. So we developed methods to construct such non-redundant data. Interestingly, we we're coming back to stat match. We first discovered this possibility theoretically in a very simple model, just the perceptron learning problem. And there we found that, like so in perceptron learning, if this is your input space, you have a true perceptron weight vector and you have a decision hyperplane here. And you know, if you just pick random points, they'll be kind of concentrated near the hyperplane. But but but let's say you could find just the hardest points that are closest to the hyperplane. Then if you restricted the hardest set of points, we could show analytically using techniques from cisclomechanics, like the cavity method and the replica method, that you could get an exponential drop-off in the scaling laws. Right? So we we saw that mathematically first. Then we asked how could you generalize this to say machine vision systems? so the key idea was to choose idiosyncratic problems, idiosyncratic inputs. So what we did is we we tried all sorts of methods. The one that worked best was you take a bunch of images, let's say you're training a classifier on, you feed them into the latent space of some foundation model, you cluster them so you find the clusters. And then in each cluster, you take the data points that are the most outlier elements within that cluster. So you get data diversity by picking different clusters, and you get data non-redundancy by picking the outliers within each cluster. And then if you look at them, Like, you know, say for ImageNet, say the the class of swans, you get a whole bunch of swans that all look identical to each other and they're near the cluster centroids, but the outliers are kind of idiosyncratic pictures of swans, right? And you know, you don't want to overtrain you don't want to waste your resources on like all the swans that look identical. You want to kind of spread yourself out across the idiosyncratic versions. And you get that across all the the the clusters. So that's the key is data diversity is is the key rather than data amount. Ravid Shwartz-Ziv: I I have a question. So like you mentioned that you had like some like this like theoretical analysis and then you actually like test it. what is your take? Because like it looked at like in recent years the gap between theory and empirical experiments are getting wider, you know, like increase in in AI specifically in with NLMs, right? but in in the other hand like statistical mechanics, for example This is like it should be like if if our models are getting bigger, if our data is getting larger, it means that like basically statistical mechanics should like now the constants are like don't matter anymore, right? So statistical mechanics should have much better predictions, right? what is your take on it? Like do you feel that like the theory that like the analyze that you can actually do today is less important or like like describe worse the the actual models or or the other way weren't Surya Ganguli: Yeah, I'm I'm optimistic about I'm optimistic about the application of theory to all of these systems because partially because I'd be very dissatisfied if it didn't work, right? Like how can it possibly be that we don't have a good theoretical understanding of the most consequential technology of our age, right? That's completely unsatisfactory if we don't. And so I would say that it's kind of a faith of mine that we will get there. And we've had enough like success points along the way that I see no reason to stop. I'll give you like one example of well I can tell you my philosophy of how how to do theory in this face space first. You know, th the key is once you see a phenomenon that seems puzzling and you can't think of any explanation for it, you try to find the simplest situation where you can model it and try to understand it there. That that's one thing. But you don't stop there. You try to organize your theory such that it only deals with measurable phenomena that you can measure in more realistic systems, but that you can model in simpler systems. And then you try to extract testable predictions. from the theory in the simple system that you can then measure in the more complex system and verify your predictions at least qualitatively if not quantitatively. So I'll give you one example, like diffusion models, for example. So so I, you know, the the first diffusion model was invented in my lab by a very talented, you know, former postdoc of mine, Joshua Soldicstein. We were actually thinking about non-equilibrium cyclical mechanics, right? And trying to violate the second law of thermodynamics, right? So, how do we violate the second law of thermodynamics? Well, we were inspired by things like the Drzezinski equality, where you look at the probability of forward trajectories and over reverse trajectories, the log probability ratio of forward versus reverse trajectories. And in our in a in an irreversible system, that log probability ratio is very high. The forward trajectory is very high probability, the reverse trajectory is very low probability, right? Think about a glass falling and shattering, right? That has high probability. But all the pieces of the glass coming together and rising up and falling on the on the table unshattered, that's very low probability. It's not impossible. If you arrange all the momenta of the shattered pieces correctly, you can have it happen. But it's just very low probability. So we thought why not why not replace the reverse process with the neural network and train it to reverse the forward process? And then you can get create structure out of noise and violate this the second law. Okay, fantastic. So then these diffusion models kind of eventually got taken seriously by industry. They they did have algorithmic improvements, scaled it up, and now you have all these images like Dali and all this kind of stuff. And then and and what? And text diffusion as well. Ravid Shwartz-Ziv: And tax diffusion. Tax diffusion, yeah. We we just like we talked with with with someone at Google that's like leading the like their like tax diffusion team that working on it, yeah. Surya Ganguli: Yeah. Yeah, exactly. So then you know a decade later, right, b after that first paper came out, I thought, okay, now we should really under this is a consequential technology. We should understand how it works, right? And this is now going back to your question. Can you actually have theory that explains how systems actually work, right? So the the thing that we asked was, okay, how do they creatively generate new images that are not in the training set, right? So it it turns out that this reverse process, so if you have a finite training set, like let's say you have kind of three points, right? Three training set images, in the forward process, you'll get Gaussians growing out of those three training set images. So that's a time-dependent family of distributions. It turns out you can exactly reverse this, right? So you can get a process that goes from a single Gaussian and goes back to the three training set points. Now, all these diffusion models are trained to reverse the forward process. So if there exists an exact reversal, Then if they can achieve that exact reversal, they'll never be creative because they go from noise back to one of the training set images. So, how can it go to a new image that's not in the training set? It must fail to achieve the very thing that it's trained to, which is to exactly reverse the diffusion, forward diffusion. So how can it fail in a sensible way? Well, we looked at just very simple inductive biases of, say, convolution-only diffusion models. And it turns out they're local, like when you compute the score, which is the direction you should flow to reverse the process. The score computation of a certain pixel only depends on a certain neighborhood. That's locality. And so then what we were able to do, we were able to write down analytically the exact reversal of the process subject to locality constraints and also equivariance constraints, but let's skip the details of that. And then once we did that, we got an analytic ODE that could go from noise to images. And then we could compare on a case-by-case basis what our analytic theory predicted for the image. With what actually a trained convolution-only diffusion model spit out. And they agreed on a case-by-case basis to with a correlation coefficient of point nine. Right. So this is an example where analytic theory can predict the outputs of a generative AI system on a case-by-case basis for every noise realization with a correlation coefficient of 0.9. Right. So that's an example of like a very successful theory experiment comparison. Now, just to just be clear, it doesn't completely solve the problem because these are convolution-only diffusion models that have a strong locality bias. That's also, by the way, in the statistics of the data. more sophisticated diffusion models have non-local transformer architectures that are in there that can break the locality. And we don't have a good theory of that, though we did show that our analytic model also explains what tr diffusion transformers do at least early in their training at 30 to 108 box. So we also get a correlation coefficient of 0.9 between theory and experiment there. Right. So I think yeah there's lots more to do there, but I'm I'm still optimistic about the theory of th theory in in deep learning. Like you know that famous phrase by I think by Wittgenstein, wir veden mussen wir wissen, we we we must know we will know, right? I I feel that way about theory here. Ravid Shwartz-Ziv: And and what do you think about like the are the I I don't know, the the the the fundamental components of of deep learning. Like you know, like in the past, like don't know, 10, 15 years ago it was like, yeah, the convolutional layers are giving us like the the biases that we actually need. And then there were a lot of talks about the SGD that like kind of like inject noise to the system or like the activation function and of course like your work about like linear networks there we can we can talk about them. So w what do you think are now in your update view Surya Ganguli: Yeah. Ravid Shwartz-Ziv: are the these maybe there are aren't like any fundamental components. Surya Ganguli: Yeah. I don't think I don't think deep learning will be like particle physics where we have a standard model of what's going on, right? like a unified standard model that explains all the phenomena, right? I think there's many interesting phenomena and there are theories of those phenomena and the theories act like a patchwork quilt that when you put them together you start to get a sense of what's going on. And again, the ultimate test of theory will will be can it guide design choices and building deep learning, right? That's the ultimate goal, I think. And I I don't think we're quite there yet, though it does give us intuitions and things like that. You know, the analogy, there's an analogy between steam engines and thermodynamics. Like steam engines came long before we understood the laws of thermodynamics. I think we're like that right here, right? Like Graduate student descent, so to speak, has created all these incredible graduate student descent, also powered by billions and billions of dollars of investment in industry and then industry research and descent has created these amazing artifacts, right? And theory's still catching up. we're still there. But like things that I think will stand the test of time are the geometry of lost landscapes in high dimensions, the I I think some of the work on stochastic gradient descent and its beneficial effects and regularization, implicit regularization, things like that will stand the test of time, but there's lots more to do. Allen Roush: So w when when you talk about theory as as being, you know, having having, you know, some questionable dynamics today with with the study of of these models, where do you think we we go in the future? do you think in particular that there's new paradigms outside of the current I've heard it called computationalist or connectionalist paradigm we're in now? Surya Ganguli: Yeah, potentially. I mean, the the one thing that sort of bugs me is is that I don't think we're teaching our AI systems in the best possible way we could, right? So for example, we have a standard pipeline now, right? We have pre-training, we have post-training, which could consist of several elements, supervised fine-tuning, RL. Then we have test time scaling where we decide how to use our inference compute. That often involves search and verification loops and things like that. And it works pretty well. Like, like, like, you know, that in and of itself has enormous economic value and will have enormous implications for society. But like, imagine, for example, that we trained our kids the same way we pre-train our LLMs, right? So I would give my son or my daughter like a natural a chunk of the internet. My son's, my daughter doesn't know how to read yet, so let's stick with my son. Like, I'd give him a chunk of the Actually, no, it's better to stick with my daughter because she doesn't know how to read yet. So how would I teach her to read? I'd give her a chunk of the internet and say, by the way, this is the next word. Another chunk of the internet and say, by the way, this is the next word. Right. And then maybe it's as as she gets older, I'll give her some RL. So I might give her like some math problems and say, by the way, this is the right answer for this math problem. This is the right answer for this math problem. That's it. That's all I did, right? Like it's hard to imagine that she would learn anything this way, right? But but That's what we're doing with AI, right? So what do we do with with our children? We we when we're teaching them math, we tell them the algorithm, right? We just tell them the algorithm, and then they immediately length generalize to larger problems, right, after some practice. So their sample complexity is way less, right? Just to throw some numbers out there. you know, we tr we pre-train our language models of about 10 trillion tokens. Humans get about 100 million words. Right. So we get many orders of magnitude words, fewer words. So our sample complexity is way better. Now we do get vision and and and all of that stuff, but I would argue that just looking at more images of of trees and rivers and forests is not necessarily gonna help you like figure out the standard model of particle physics or figure out group theory or something like that. Right. So so at least in terms of our language experience alone, where orders of magnitude have less language experience. And I think it's because we communicate how to think to our kids, our students, our grad students, and so forth. And so we give them a very rich vector reward, not even reward signal, a vector instruction signal. That just gradients alone don't have that, you know. So I think this whole paradigm of learning is something we're gonna have to change radically. You know, in-context learning is a rudimentary version of that. And that's why I think everybody got so excited about in-context learning. And it is kind of interesting and powerful that you can present rules in context to an LLM. It'll learn the rule on the fly without changing its weights. And then it'll give give the correct answer to new examples of that rule. Right? So so I think that was interesting, but we we should be able to do better than that. Yeah, we we actually got interested in in just talk about theory experiment. Like we we got interested in another qu like when in context learning came out or a couple of years after it came out, we just asked a very fundamental question. can an LLM learn something fundamentally new in context that wasn't in the pre-training data, right? Now that's very hard to answer about a general LLM because a general LLM, like you don't know what's in its pre-training data, it's the entire internet. So, in what sense is an in-context example different from the pre-training set when the pre-training set is the entire internet? Like it's hard to answer. So we just decided to look at again a toy model, right? A linear regression in context. We asked, like, let's say you give some number of linear regression problems in the pre-training set. So you train from scratch, and you ask how many linear regression problems do I need to see before I can solve any linear regression problem whatsoever in context, right? So you give it kind of inputs, inputs, and outputs in context, and you ask, give it a new input, and you ask for the new output, where the inputs and outputs come from one regression problem. Okay, so we found a phase transition, right? if the number of different regression problems you've seen in pre-trading is below a certain critical threshold, you cannot learn new regression problems in context. You can only solve the ones that were in your pre-trading set. But as soon as the number of different problems gets above this threshold, You suddenly have a phase transition where you can solve any new regression problem in context that was not in your training set, and you can solve it as well as the optimal problem, ridge regression, right? So so there's this phase transition. The next critical question is: how many examples do you need as a function of the dimension, the input dimension of the problem? And the answer is a linear in the input dimension. So you get a remarkable phase transition with not that amount of Not that much amount of diversity in your pre-training set, right? So yeah, so this suggests that in-context learning of fundamentally new examples is possible, actually. Which which was kind of hard to answer that question until you controlled the the pre-training scenario, the architecture, and all that kind of stuff. So I think insightful experiments like this is also by the way, I mean, it's not just about deep learning theory, it's about deep learning experiment as well, right? I I think again, looking at historical analogies, if you think about the steam engine to the laws of thermodynamics, we're kind of closer to the steam engine than we are to the laws of thermodynamics. Or to complete the analogy, we're closer to the experimental realization of intelligence than discovering the fundamental laws and principles of intelligence. Right? So in when you're in such a pre-theory setting, oftentimes you don't even know what the right questions to ask are. You don't even know what the right experiments to do are. You don't even know what to plot on the x-axis and the y-axis to construct a theory of, right? So when people were working out statmec in the early days, they would compute they would measure a bunch of things and they'd they'd compute nonlinear functions of those things on the x-axis and nonlinear functions of those things on the y-axis until they got a straight line. Then they said, okay, the straight line is worth explaining, right? And and we Ravid Shwartz-Ziv: Ha ha Surya Ganguli: saw that, for example, in scaling laws. We got a straight line on a log log plot. Okay, why do we get a straight line? Let's try to explain that, right? So part of the job of a theorist is also to suggest experiments and and actually in deep learning as opposed to neuroscience, the other field I work in, we can do the experiments ourselves, right? So part of our job is to do, is to discover insightful experiments that we can do. One of those was the phase transition in in context learning for regression, for example. Allen Roush: So you Ravid Shwartz-Ziv: But it Allen Roush: you you you t I I I do wanna change the topic a little because you told us ahead of time that you wanted to talk about some really interesting, you know, potential even results you had on consciousness and and thoughts. Love Surya Ganguli: yeah. Yeah. Allen Roush: to hear your thoughts on what it means for a machine to be conscious if that's even possible. Surya Ganguli: Yeah, that's an interesting question. Ravid Shwartz-Ziv: And and we will not say anything about the recent paper of of entropic about consciousness yeah yeah Allen Roush: Yeah. Surya Ganguli: JSpace paper, yeah. Yeah, so Jack Lindsay was a former student of mine. He's an absolutely brilliant guy. I I really like him a lot. JSpace work, I think, is very interesting explainability work. the connection to consciousness, I think, is a little bit more tenuous, but let's just leave it at that. on Ravid Shwartz-Ziv: Yeah. Surya Ganguli: the on the consciousness side, like you know, so okay, so how do you how do you think about it, right? So the I I think something like the sense of self is a little bit easier to think about, right? So we all have this notion of we are a being in the world, and we can we can affect the world and the world can affect us, and that's continuously with us. It's actually hard to turn it off, right? so where does that sense of self come? Or or to state it another way, every single word that we say, References a pre-existing pre-verbal object, right? For example, chair refers to the thing I'm sitting on, you know, laptop refers to the thing that I'm looking at right now. What does the word I refer to, right? Is there some pre-verbal notion of I that the word I references, right? Like who am I, right? So my thinking in this is heavily influenced by Antonio Damasio. He wrote a beautiful book called The Feeling of What Happens. And I'm gonna tran he's a he's a neuropsychiatrist and he has all sorts of interesting he he looks at all sorts of interesting disorders of consciousness, right? I highly recommend that book. It's it's very cool. But I'm gonna translate what he says into the language of control theory and causality, like mathematical language. So the basic idea is why does an organism even need a sense of self? Well, imagine an organism with a very complex array of sensory inputs. And a very complex of external sensors, so eyes, ears, taste, all so forth, hearing. and then it also has a very complex array of internal sensors, like what are my muscles doing, what is my gut doing, what is my hunger level, what are my O2 levels, what is my what are my thirst levels, right? So you have these two huge sensory arrays, right? We're not as u used to thinking about our internal sensory arrays, but we have a very complicated internal sensory array. Now you have to connect the external sensory to the internal sensory array somehow, right? In order to become an organism. What is the most important thing you have to take into account when you connect them? Well basically the external world can affect your internal state, right? And you'd like to causally model the impact of the outside world in your n internal state in a predictive way, right? So you're gonna have predictive causal modeling loops that predict how the outside world affects you. And you want it to be predictive because you want to predict stuff ahead of time so you can act on it, right? So for example, if you see a lion in the bushes, you can predict that it might eat you. So you wanna you wanna act accordingly, right? Or if you see water, you can predict that if you drink it, you'll feel better, right? So these causal predictive loops that connect the external world to the internal world, internal array, are kind of where your sense of self should come from, right? So I'll give you kind of one example, a kind of striking example where if you disconnect these loops, you can dramatically alter your sense of self. So, for example, there was a patient who had trigeminal nerve damage. So the trigeminal nerve goes from your face area to your thalamus to your cingular cortex. And so what that meant is that if there was a tiny gust of wind, the patient would keel over in in incredible pain because there's this Pain circuit that gets activated. So then what they did is they disconnected her, they they lesioned her cingulate cortex, right? So the pathway from the periphery to the thalamus was still fine, right? So then what happened, Ravid Shwartz-Ziv: Mm-hmm. Surya Ganguli: like right after she woke up from the surgery, they asked her, Do you f do you feel the pain anymore? She's like, Yeah, I feel the pain, but it doesn't bother me, right? So notice she used the word me. She used the word sense of self-sense of self, me. So her sense of self didn't feel like it she had the pain but not the suffering to the sense of self that happened, right? Okay, so so that's I think remarkable, right? Like you've dissociated the it's like the pain isn't happening to you anymore and so you don't suffer, right? And there's just a lesion in the singular cortex, which is actually involved in causal modeling of the impact of inputs on on yourself as well, actually. All right, so now how do we actually study this? In a concrete way that leads to scientific results. So the way that we've been going about it is we've been giving drugs to animals and then looking at what happens to their brain. So for example, one aspect of your sense of self is where you are in the 3D world, right? It's actually quite remarkable that and this connects to embodied intelligence and AI and things like that. It's actually remarkable that we have a sense of where we are, right? Like For example, if I were to close my eyes and you asked me where is the door, I I could immediately point behind me, right? If you just think about the computation that happened there, it's remarkable because when I entered my office, what were my sensory inputs? I just had the pitter pattern of photons on my eyes, and I had the the the pitter pattern of muscle contractions as I was walking, right? And through this stream of low level sensory inputs, I can compute a 3D model of the room, situate myself in it, and compute motor commands to point to Directions in the 3D model, right? It's absolutely remarkable, right? This is a world model, right? I mean, everybody in AI is falling all over themselves to build world models at the moment. The brain has them, right? Now, there's a part of your brain called the entrance cortex that has 3D grid cells. sorry, it has grid cells in it. These are cells that fire, for example, if and only if you're at the ver the vertices of a hexagonal grid in some some room. So that that grid cell area. computes where you are, right? It it it in a room, right? So now what happens if you give ketamine to a mouse or a human? If you give ketamine to a human, they report out-of-body experiences. They dissociate their positional sense of self from their physical sense of self. So we could ask what happens to a mouse, right? So what happens is you get pathological oscillations somewhere along the circuitry from the retina to the enterinal cortex where the grid cells are. Then you look at the grid cells and they start drifting actually, right? Drifting from where they should be. Or they can even dissolve in higher doses. So what's happening is the part of your brain that computes where you are is either drifting or dissolving. And that could be a neural correlate of your out-of-body experiences, right? So we we've also been we've also been giving LSD to mice and decoding the hallucinations that they have. We've been giving psilocybin By we, I mean this is a collaboration with sort of Carl Dyseroth and Lisa Giacomo, neuroscientists at Stanford. we've been giving Silo Sibin and trying to understand that. We've been stimulating the brain to try to induce schizophrenic hallucinations. and we have a way to do that as well. So we're we're kind of trying to understand all these kind of hallucinations slash alteration of the sense of self in animals and humans. And like for LLMs, I don't think LLMs are conscious in the way that Damasio talks about because you need embodiment for that. But I actually don't think there's anything stopping robots from being conscious in this way, or at least aware of themselves, right? it just involves complex sensory internal and external sensory arrays and causal modeling loops between them. And so, you know, in the same sense that we've developed intelligent machines and now we're no longer kind of weirded out by them, we will probably develop like robots that have a sense of self and then experim and then and then we won't be too weirded out by them, I don't think. Ravid Shwartz-Ziv: But do you think like it will make them better in some sense? Like both consciousness and also like internal world models? But because like I don't know, Jan Surya Ganguli: I think so. I think Ravid Shwartz-Ziv: for example, like this is his bet, right? That like better Surya Ganguli: Yeah. Ravid Shwartz-Ziv: world models will actually make better robots, better like models and that will solve like Surya Ganguli: In in some sense it yeah, in some sense it has to, right? Like I mean, one of the key control theoretic dividends of a internal world model is is the ability to do imagination-based planning, right? You could simulate the future, contition on contingent on different actions, and then do all sorts of experiments before you take the next action. And to the extent that your ability to simulate the future contingent on actions is accurate, then you can Do all sorts of imagination-based planning, pick the best action in the simulation and then implement it in the real world and then replan. Like that's always been the dream of world models. It's just been very hard to develop world models that can predict the future condition actions over a long enough time horizon to be useful. But it's sort of clear that we do that, right? We're always thinking about, you know, if I did this, what would happen? If I did that, what would happen? Like and and and then we make our best guess and then then we act. Right. So we have so many internal cognition loops between each action, right? it seems like the right way to go. Allen Roush: So I I I'm curious then, would you agree then that people who reject any possibility of consciousness, sentience, or qualia, would you agree that they're carbon chauvinists in this sense by rejecting even the idea? Surya Ganguli: Right miss w would I agree that there was? Allen Roush: Carbon chauvinists, like yeah. Surya Ganguli: yeah, totally. Yeah. I I there's nothing special about carbon. I think I think once we understand consciousness it it won't be that mystical. I think but there's lots more work to do there, right? Like that's why I really love what the neurosychica like you understand a phenomenon when you can break it, right? That's what happened with genetics, right? We understood To the extent that we understand genotype phenotype maps, we understood it by knocking out genes, kind of one by one, and then sometimes combinatorily figure out the change in the phenotype. And then we we had a bunch of puzzles and we tried to put them together, right? We're gonna need to do that in consciousness studies as well, right? Look at patients who have disordered disorders of consciousness, look at animals under different drugs that alter consciousness and and so forth. We're going the drug route at the moment because it's it's a very rapid experimental loop compared to waiting for humans to show up that have disorders of consciousness because of lesions or things like that. Yeah, there's also like Ravid Shwartz-Ziv: So Surya Ganguli: yeah, sorry, go ahead. Ravid Shwartz-Ziv: like like like I think like genes is is a good like like compare genes to to neuroscience or like understanding the brain. Like it's it's a good like example. W why you think like we actually basically almost like solve understanding genes and now are like in the phase of like just making like better like treatments for for different disease versus the brain that like we are so far from understanding it and like we probably like know or like understand very small components of of it. What is the difference? Surya Ganguli: Yeah. I mean I think I I think part of the reason is the brain is it's the most complex structure known in in the known universe, right? In terms of, you know, we have roughly a hundred billion neurons, we have a thousand times as many synapses, right? Like it's it's incredibly complicated. And it's also complicated across scales, right? So you you you go from molecules to cells to local circuits to global circuits. to whole brain, to behavior. And so, you know, a lot of the degenerative diseases start as molecular deficits, and you have to understand how the molecular deficits propag propagate across scales to behavior, right? Through through all the intermediate levels. So it's an incredibly complicated problem. But that's where physics comes in, right? In physics, we've been used to thinking about thinking across multiple scales and things like that, putting the puzzle pieces together to come up with current models. And and we also haven't had access to the brain the last decade has actually revolutionized our ability to access the brain. You know, like just like maybe two decades ago, we were recording from one neuron at a time. Now we can record from 70,000 neurons at a time. We can even record from multiple brain regions all at once, right? And so you know, we have papers we've been able to trace. information flow from sensation to action across the entire brain. Right. And and so I'm incredibly excited about advances in neurotechnology. And also we don't have to be we don't have to kind of be happy with just recording for the brain. We can even stimulate through the brain through the magic of holographic optogenetics. So for example, you can shine patterns of light into the brain and genetically modified brains that have light gated ion channels that have been stolen from bacteria. So when you shine light on a neuron, the light-gated ion channel opens and then the neuron fires. So you can start to play the piano of the brain. You can, you can at millisecond resolution, you can stimulate any arbitrary pattern you want. So now you can imagine a research program that's a neuroai effort where you have the brain, you construct a digital twin of the brain, which is an accurate model of both neural activity and behavior for a certain brain region and a certain behavior. Then you can use explainable AI to understand it. Then you can use control theory to control the digital twin to get it to behave in a certain way. Then you can take the control signals, which are injected patterns of neural activity, you can transform them to the actual brain to get the brain to behave in a certain way. Right? So so now you have these closed loop experiments where you have a brain and an AI system talking to each other. And imagine the things that you can do. You could do active learning of yeah, you can do active learning of the model itself. You can Try to get the brain to be in a clinically desired state. so we've done various things. We had a a science paper several years ago again with Carl Dyseroth, where again, I don't do any of the experiments. I rely on extremely talented experimental fibers to do the experiments. I do the theory and experimental design and so forth and the modeling and data analysis. But we were able to stimulate just 20 neurons in the mouse's brain and get it to hallucinate a particular percept, right? and and there's a reason we had a theory for why it's 20 and not. e not lower. basically the idea is your brain has spontaneous activity. So there's a kind of a a pitter pattern of neural activity in the brain. And 20 neurons is about four standard deviations above that activity. So if it were any lower, then your spontaneous activity would generate false positive percepts. And so what this is telling you is that the brain is as sensitive as can be possible to external stimulation without having false perse false positive percepts driven by its own activity, right? Then what we did is we in a separate paper, I mean this connects to schizophrenia, we started stimulating more neurons. And what we found is as soon as we stimulate above 50 neurons, there's a shutoff valve that goes off in the brain. There's a set of inhibitory cells that shut off the stimulation. Okay. Now what's interesting is that these particular cell types, you know, for the for the nerds, it's somatostatin cells, but But for this particular cell type, those are exactly the cells that get disrupted in schizophrenia. Okay? So now imagine what happens when your brain's shutoff valve gets disrupted. What this shutoff valve does, say in visual cortex, in lower level visual cortex, you can have higher brain areas come in. And these inhibitory neurons will prevent activity from the higher brain areas to propagate into visual cortex, right? Now what happens when you take away the inhibitory neurons? Now activity that wouldn't have gone into the visual cortex gets into visual cortex. So your top-down brain regions like prefrontal cortex, whatever, are now sending signals into visual cortex that shouldn't be there, right? But that's exactly what could cause schizophrenic hallucinations, right? You start seeing visions that aren't there in the world because your internal brain is generating them. Right. So now we're starting to get a mechanistic understanding of how high level psychological effects like Schizophrenic hallucinations, LSD hallucinations, ketamine-induced out-of-body experiences are happening in the brain because we have the ability to do these closed loop experiments with neuroAI. So I think it's incredibly exciting, right? I used to work in neuroscience because that was the only intelligent system in town as of like two decades ago. But now we have Ravid Shwartz-Ziv: Mm-hmm. Surya Ganguli: AI systems that are also quite intelligent from a computational perspective. So actually, I do a lot of work in deep learning now because I want to understand the principle of intelligence. On the neuroscience side, I work on those things that you cannot work on in AI. And that's like consciousness, drugs, things like that. Allen Roush: the so I'm I'm always fascinated by like cyberpunk media depictions of what the future might look like, and I'm especially fascinated by Ghost in the Shell for a number of reasons. There's the new Netflix or whatever anime about to premiere. One of the things they predict is that the soul or consciousness or whatever you want to call it, they call it the ghost. It is scientifically understood and removable from the human body and placeable into machines or computers. And then it becomes interesting when the opposite happens of a computer creating a scientifically understood ghost from scratch, and that's kind Surya Ganguli: huh. Allen Roush: of ensuing events. Do you think things like this are possible given what you've just said before, or is this like all s fantasy? Surya Ganguli: That's an interesting I so I haven't I'm not aware of this. I don't know if you guys seen Gossip Goblin. Do you know that one? It's a bunch of Facebook reels that are quite they're dystopian futures, actually. I was kind of quite intrigued by them. So I haven't seen what you're talking about, but in terms of sort of transferring consciousness from a machine to a human, I'm not sure how that would work because I take a very embodied approach. in my thinking about consciousness. Like consciousness and sense of self are ultimately limited to an organism. Like we have and it's very important for an organism to understand what it can directly control with its brain versus what it cannot control with its brain. Right. And that's the boundary between self and non-self. And you can imagine it's incredibly adaptive for an organism to understand the difference between the two. Right. Like I can control my arm, but I cannot control the lion that's about to eat me. And nothing I do in terms of just thinking alone will stop the lion from eating me. Right. So so now it it's hard to imagine what it means to transfer the boundary between yourself and your non-self to a robot. Right. I I'm not quite sure how to think about that. Ravid Shwartz-Ziv: How far are we you think from understanding the brain? And what are the main bottlenecks for that? Surya Ganguli: Yeah. I think we're actually further f much further from understanding the brain than understanding AI. the main bottleneck is experiments. it's harder to do experiments in the brain than it is to do experiments on AI systems because you have everything you need in the AI system. You have access to the entire connectivity, you have access to all the neural activity patterns, you know even the training data. Well, for open models, for some open models, you know the training data. You can look at checkpoints of training, like you have everything, right? You have But but for the brain, we don't have a lot of that stuff. So we have to do guesswork. Now, the the flip side of that argument is that it could be that AI trained systems are just r non robust Rube Goldberg devices that have a whole bunch of heuristics built to them to them through gradient descent and large training sets, and that they don't think as systematically as brains do. So they don't they currently don't have strong robust OD generalization. We don't actually know what OD generalization means when the training set is the entire internet, also. So that's an interesting question there. But brains are much more robust and systematic in their thinking and their OD generalization capabilities. And robust systems are much easier to understand than Rube Goldberg devices, right? So it's this competition between not having enough experiments in the brain versus having a potentially more robust system, right? The former makes it very Ravid Shwartz-Ziv: Wha why you think the brain is more robust? Like what is the fun like what is like fundamental like more robust in in the brain? Surya Ganguli: We can learn rules with much less data and then generalize systematically than AI systems can do. Right? So th that's what I mean by more robust. Like robust generalization with very little sample complexity. Right? And this may have to do with how we train humans. Like again, we can give them instructions and humans will follow them. Ravid Shwartz-Ziv: So h how much like you think is from like evolution perspective versus like current let's say like connections and like do you think like we should like at the end like AI machines like should have kind of like evolution component or this is something that it's like unnecessary? Surya Ganguli: I've so sorry, you you y I I missed the last part of that question. Ravid Shwartz-Ziv: So do you think like so do you think like we actually like for for the brain, like the most of the the the sample complexity efficiency is is from evolution process, from the evolution process? And if so, do you think we actually like should apply to AI models? Surya Ganguli: Yeah. I think it's a combination of nature and nurture, evolution and within lifetime learning. for arguing for the evolution side, if you look at the vertebrate brain plan, it's conserved across 500 million years of evolution. So for example, our last common ancestor of say humans and and and lampreys was like a fish like organism. And it it had and now if you look at even today, the the lamprey and the human, right, they're so far apart. The lamprey is little worm-like organism. We both have a cortex, a basal ganglia, a thalamus, and a little bit later, the descendants of lampreys have a cerebellum. Each of these brain regions have their own learning rules. They control motor actions on different timescales and a hierarchy. So this modular hierarchical architecture that's specialized with different learning rules and different communication protocols between more abstract regions and more finer scale regions. I think that has a lot to do with the sample complexity and robustness of learning motor control, for example. And ultimately motor control and navigation co-op or got co-opted by evolution to do higher level logical reasoning. Right. Like logical reasoning is navigation in concept space, and then navigation is the navigation in physical space. Right. So I think right now we're we're just taking these big transformers and training them end-to-end. They don't have any modularity, right? Other than the intentional MLP, intentional MLP. Like I think that modularity is is key to creating an efficient learning machine. In fact, there was a grant call for neuroscience inspired robotics, where we got together like a bunch of really good roboticists. I mean, if I name them, you know them, they're very famous. And and we got a a very good experimental neuroscientist and we got very good computational neuroscientists. And we got the three of us together to like go after this question of what are the principles of modular robotic control? So we're gonna record from multiple brain regions in animals and look at communication patterns between these brain regions and how they relate to behavior and how they get learned over time. And then the roboticists are gonna build modular control modular robotic systems and we're gonna try to put them all together and put it together. I I think we're probably gonna get funded, we'll see. about that. So so I think that's on the on the nature side. On the nurture side, it's like I said before, we just have better ways of teaching humans than we have of teaching AI systems. Ravid Shwartz-Ziv: Mm. Allen Roush: How you go? Ravid Shwartz-Ziv: And like more like more general question, d do you think we actually need to try to mimic the brain when we develop AI systems? Do you think like this is the only path forward or like there are like other ways to do it? Surya Ganguli: No, I I don't think so. I I I think when you're trying to build an AI system, you're trying to get it to work. Like if you're working in industry, you know, you have short time horizons, you gotta get something to work quickly, you're competing against other companies. I I think you you you you just go for it. And and and it's it's worked quite successfully, right? But it it still misses the boat on sample complexity and also energy efficiency, right? So I think on a longer time horizon, it can be interesting to think about brains, but from an engineering perspective, right? So I mean, I I talk to like experimental neurobiologists and AI practitioners, right? And they're very different people, right? Like oftentimes experimental neuroscientists get so caught up in the experimental details of a biological system that they sometimes forget to think about what the engineering purpose of that system is for. And the engineering people get so caught up in in Hacking and random exploration, that they sometimes forget to think about the theoretical principles guiding their hacking. But they've both been incredibly successful, right? I would like to bring all of that together and think about a principled way to understand both biological and artificial computation and improve both together, right? So bringing the engineering and the biology together, I think it's fun, it's exciting, but I don't think if you're an industry and you have these short time horizons, you necessarily want to do that. Allen Roush: So, w there was this really cool research, it even predates a current generative AI era around doing image reconstruction using human brain activity. And I'm sure there's been a lot of other research based on human brain activity sensors as control for all kinds of things. What Surya Ganguli: Yeah. Allen Roush: what do you think of this kind of research? Are you aware of it? Surya Ganguli: Doing image reconstruction using human brain. Allen Roush: Yeah, yeah, so like so like basically somebody's looking at an object and then using only their brain activity reconstruct using like neural nets the image they're looking at. So mind reading. Surya Ganguli: Yeah, yeah. Yeah, we've gotten a lot better at that collectively as a as a community, right? we even have some work where we can decode what a mouse is looking at from about 5,000 neurons from the mouse's brain. yeah, that's that's getting better, better and better. We had a work that went one step further, which was it's called like letting the neural code speak. So if you think about like trying to understand what a visual system of say a monkey can do, right? You have neurons deep in the monkey's brain that that have very complex selectivity, like like and we don't understand it, right? Early in the brain, we kind of know what neurons do, roughly to first-order approximation. They're edge detectors. They have gabor filters. But if you go deeper into the brain, it's much more complicated, right? And so we don't know how to mathematically characterize, for example, the set of images that make a monkey neuron deep in the monkey's visual system fire, right? For example, High level in the human brain, like an infertemporal cortex, we have this famous Jennifer Aniston neurons. It will fire if and only if the image is an image of Jennifer Aniston or the text Jennifer Aniston written there and stuff like that. So so in a monkey neuron, you don't have that, but you have other stuff. And so how do you understand this? Well, we figured out a way to get a monkey neuron to tell us in human language what it likes to fire to. And it's actually just putting together a bunch of like AI systems. So first We create a digital twin of the monkey brain. So this is an accurate model that goes from image to firing effect of monkey neurons. Right? So now we have a now we have an insilical model of the monkey's brain. Now we can do new experiments we couldn't do before. We can pick a neuron in the model and show a million images to it and find the 20 images that make it fire the most. So then we get a collection of images. They're very complicated but structured images. We don't know how to mathematically model them. Other than the digital twin itself. So then we feed those to a language model and we have the language model describe it. And it comes up with a pretty good description of those images. Now that's not science because we're not verifying the language description. How do we verify it? We can feed the language description into a language conditioned image generation model, like a diffusion model, and then generate many, many images. And then we can feed those images back to the digital twin and just check, does it make the digital twin neuron fire a lot more? And it does. And then previous work has shown that if you make a digital twin neuron fire a lot more, you can make the monkey neuron fire a lot more. So now, for the very first time, we have an algorithmic method to sample from images that will make a monkey a neuron deep inside the monkey's brain fire a lot, right? This would have been impossible to do without the AI because we'd have the cursive dimensionality had you search in this huge image space. Now I find that the deep irony of this procedure, the very first procedure to do so, factors through human language. That's kind of crazy, right? So this is a collaboration with Andreas Toliash led by Vidang Lad, a very talented student who's now gonna be joining my lab. he did this like before his PhD. Ravid Shwartz-Ziv: But but this is like this is based on on the fact that like you actually can make a d digital twin of the of the monarchy, right? And like Surya Ganguli: Yeah. Yeah, yeah, exactly. Yeah. Ravid Shwartz-Ziv: so do you think in general there there are like a lot of works that try to look on like the similarity between like Brain and AI models, right? And like they try to map specific layers to the brain and things like that. Surya Ganguli: Yeah. Ravid Shwartz-Ziv: What is your take on it? Like do you do you agree that there are some systems and I don't know, vision AI models or like language AI models that looks and you can actually map them to specific areas in the brain? Surya Ganguli: Yeah, people have done that. So my colleague Dan Yamans, who's kind of two doors down from me, he had the mo he had s sort of seminal work on that where he he tr he trained a model in ImageNet. So it's a multi-layer network. And then what he did is he did regression from the model to an actual monkey's brain in different layers, and he found the the best layer in Image Net that matches to a particular brain region in the monkey. And he found that early layers in ImageNet mapped to visual cortex, later layers mapped to V4, a deeper region. And even later layers mapped to infertile cortex in an even deeper region. So that was kind of a nice correspondence. And that actually set up, kind of started this whole line of work on training deep neural networks on a task and comparing these task trained networks to the brain, right? And you see like a decent amount of fraction of variance explained using that method. I think there's a lot more to do there. We're following a different approach. So Andreas Tolesh's lab is gathering some of the largest data sets of, you know, showing. You know, a thousand hours of movies to a monkey brain and recording from many, many neurons in the monkey's brain. And then we're just fitting neural network models to go from recorded images to recorded neurons. So that's not a task train network. That's literally mimicking the monkey's brain. And in those models, I don't trust the intermediate layers of the model at all. All we care about is getting an accurate functional simulation of the monkey's brain. And then this opens really interesting questions about Explainability, like how do you understand the geometry of neural representations on the monkey in a way that you can summarize it? So so this opens up a whole bunch of explainability studies. And I actually completely changed my theoretical neuroscience course that I teach at Stanford to become an explainable AI course because I believe there's a huge confluence between explainable AI and neuroscience. And I think one of the critical questions of our time is just explaining how. complex nonlinear distributed circuits compute, whether they're biological or artificial. And some of the old theoretical neuroscience that you know people used to teach before, like Hodgkin-Hogsley models, Hopfield networks, they're still important, but knowing them doesn't allow you to dive into research immediately. So I'm teaching the type of class that I think will help students dive into this modern world of neuroAI immediately. I actually put up the explainable AI kind of background reading list of papers last year when I taught it for the first time and it got a lot of attention and excitement. Ravid Shwartz-Ziv: So do you think like when we will understand how to explain like AI models we'll also like success to explain the brain? Surya Ganguli: I think so. There'll be common principles. The the analogy that I like to talk about is is people often use the following analogy. Birds are not like planes, and therefore brains are not like machines. But that's too facile of an analogy. It's not going deep enough. So if you think about the design space of aero of birds and planes, like there's two problems you have to solve in a flying object. You have to solve lift and you have to solve thrust. Right? So birds and airplanes solve the problem of lift in exactly the same way. They have an aerodynamic wing shape that creates high pressure below and low pressure above, and that keeps you up in the air. But thrust is solved completely differently, right? You have jet engines for the airplane and flapping of the wings for the bird, right? But so so there are differences between the birds and planes in that way, but there's also commonalities. But also the design space is very different. A jet engine is is Is supposed to fly very, very fast and carry a very heavy load. Whereas a bird has to land on a tree branch, right? Or even and so it's a very light system that can stop on a dime, right? So it's very different. But but then there's these common laws of aerodynamics, and woe unto any flying object that violates the laws of aerodynamics, it'll fall out of the sky. Similarly, I think there's principles of intelligence that govern how intelligence emerges from nonlinear circuits. And AI and humans a AI and brains will have some common principles and some different principles. And what I'd like to understand is the com is the principles, right? Ravid Shwartz-Ziv: So but like w why you think like okay, so first like can you give us some examples of like these principles and and then like when we actually like okay, let's assume that like we see some phenomena, how we know if like this is like a common principle for both like these types of systems? Surya Ganguli: If if you see sort of convergent evolution between brains and machines. So for example, I mean Gabor filters, it's very early in the visual system, but there's deep principles to r believe why they're there, right? There's many ways to get them. The simplest one is projection pursuit. You want to find linear combinations of pixels such that projection of images onto that linear combination is highly non-Gaussian. Right? So and it turns out that projections of images onto Gabor filters have a very highly kertoic distribution. So it's a very interesting direction in that sense. and then there's many methods that can that can discover that. So for example, if you do Hebian learning on a neuron with a nonlinear output, right, then you'll find non-curtotic direction. So that that's a kind of a very simple one. there's many possible principles, right? Disentangling things so that complex nonlinear manifolds in input space translate into simple manifolds in output space. So that can do linear discrimination in the output space. I'll give you one. For example, few shot learning, right? We can learn concepts using one or two examples, right? So, for example, if I give you a couple of images of a new species called a numbat, right? You may have never seen a numbat before. the example we used in the paper was numbat and kawadi, right? So you've probably never heard of a numbat or a kawadi. It turns out they're ImageNet classes. I give you a couple of images of a numbat, a couple of images of a kawadi. Then I give you a new test image and you say, is it a numbat or a koate, right? We can do it. Okay. What is a theory that governs that? The basic idea is if we can do it with a couple of images, there's probably no need for plasticity. So the idea would be: a very simple theory for that would be the two numbats come in and you get two points in some latent internal representation space. You get the two koadis and you get two points. And you just form a maximum margin hyperplane between them, right? Then the test example comes in and you just check which which side of the hyperplate it's on. Very simple theory. It actually works really, really well. You can do few shot learning based on image net representations. You can even ask how well you would do if you had monkey neural representations. So for example, somebody showed these numbats and Kwatis to a monkey, right? And we recorded like 150 neurons, they record 150 neurons, and we put it into our theory and computed analytically the the few shot classification error. And it was it was very high, similar to ImageNet. So the basic idea is those ko no no how would this have to work? It it basically means that all the numbats would have to come into some delimited region of of latent space. And all the Kuwaris would have to come in some limited region of decision space. And if that's true, the max margin classifier I learned in only two examples would work well in any test example. So we came up with an analytic theory for few shot classification error in this limited setting. And it depended on the geometry of concepts in latent space, their their the separation between their means and their covariance ellipsoid. And we found that concepts that humans could learn in Fuchshot were also well separated in the in the visual representation space of monkeys, right? So so basically, like I think that that's what it boils down to is that neural representations. learn to put things together for which we'll need similar behavioral outputs and separates things for which we'll need different behavioral outputs, right? and that's what a lot of self-supervised Ravid Shwartz-Ziv: With Surya Ganguli: learning methods do, by the way, right? Like simClear and so forth. Ravid Shwartz-Ziv: But like I will try to like to push back a bit because like here you you basically say like okay what is like the simplest solution that actually works, right? The simplest like hypothesis that actually walk, and you say, okay, well like let's try some linear class classification and let's develop like theory and and check if this is actually works. But what if this is not the case? Like what if like It's not only the simplest solution, right? Like this is like one solution, I agree. But like what if like maybe in our brain this is something like we actually like use some more sophisticated decoder, for example? And Surya Ganguli: Yeah, yeah. Ravid Shwartz-Ziv: and this is actually and and and it may be that like linear decoder will work, but also that this like complicated decoder will also work, and this is what we actually do. So how you how you try to separate this hypothesis? And actually Surya Ganguli: Yeah, it's a good question. Ravid Shwartz-Ziv: like we are not using like AI is using something and we are using something totally different. Surya Ganguli: Yeah, exactly. So we also tried quadratic decoders, which which are then sensitive to the second order statistics of the class conditional distributions in latent space. and they didn't work that much better than linear decoders in the settings that we looked at. the key Allen Roush: That that Surya Ganguli: thing is to design experiments. So one fantasy experiment I have is to say, okay, we know that few shot classification accuracy depends on the structure of covariance ellipsoids in the In the latent space of some representation, could we design stimuli that elicit different different covariance ellipsoids and then do the psychophysics in humans to test their accuracy and few shot classification as we make the task harder or easier? Bring the ellipsoids together and change their shape to make it harder, or skim bring them apart and change their shape to make it easier. So we do psychophysics in humans, and then we do look at neural representations of monkeys and compute the theoretical performance there and look at many different two-way classification tasks. And if our theory in monkeys and our measurements in humans match using the simple linear readout, then that's good evidence for the simple linear readout. Plus also it's hard to come up with if you only have two examples of each category, there's not much you can do. Right? That's the other argument. Like what is a more sophisticated readout that does better than linear readout if you only have say two exam? Or if you have just one example of each category, what are you gonna do? Right? Allen Roush: Isn't isn't this a pretty common result? I think the diffusion and text diffusion, image diffusion folks have also mentioned the whole quadratic samplers not being or decoders being better than linear ones, which is really surprising. Surya Ganguli: Yeah, it's it's not so surprising if your task is easy, right? If your class conditional distributions are well separated, then a linear decoder will do well and a quadratic decoder won't do that much better. The the the hard example where a quadrat quadratic decoder necessarily does better than a linear decoder is let's say your the means of your two distributions are identical, right? But your covariance ellipsoids are different. So maybe one has a covariance ellipsoid this way and one has a covariance ellipsoid this way. Then the quadratic decoder will do much better than the linear decoder, right? Because there's no information in the linear decoder, because the means are the same. Right. So that but but if you look at like pairs of classification tasks for which few shot learning is easy, you don't have a hard situation like that. You have a kind of a well-separated situation like that. Ravid Shwartz-Ziv: What do you think about agents? You know, like some I heard like some opinions that basically you have like agents are like the brain is like is like it needs some like tool usage and like it kind of like Surya Ganguli: Yeah, yeah. Ravid Shwartz-Ziv: agent in the like in the real world. Like what do you think about it? Surya Ganguli: Yeah. Yeah. lots of stuff to say there. I I did take Marvin Minsky's class, Society of Mind, when I was an undergraduate MIT. he he was kind of ahead of his time in that respect. you know, all of the like feedback loops that agent use, like refinement, self-deliberation, self-creating self-cr critiquing LM as judge. We do that all in our brain, right? Like as well just through introspection. It's not like we have experiments that prove that, but but it's through introspection, I feel like we do that. And it can be very powerful, right? it's it's not like we zero shot the solution to complex reasoning tasks. We make guesses, we check it using a calculator, we check it using mathematica, we verify it or we prove ourselves wrong, we do an experiment, right? Like and in some sense, like the entire Effort of science and engineering is a bunch of human agents working together with specialized capabilities and things like that. And we have interdisciplinary teams. So I feel like it's the right way to go. Early on, my hot take on agents, like was that we're stuck with agents because LLMs are so crappy. And so we need to create loops of interacting LLMs in order to correct each other's errors. But maybe that's inevitable because we're not gonna get a single LLM with a single context. window to auto regressively generate the answer to a very complicated problem, right? That just seems very hard to imagine how it would work. Especially if at every step of the auto regressive generation the LM has a probability of making an error. So Ravid Shwartz-Ziv: But this is like the limitation of of like the current paradigm, right? Surya Ganguli: That's the limitation of the current paradigm. The question is, will we have like a more powerful paradigm where we just have one shot auto regression? You just unroll in the future and you solve a complicated problem. I I feel like that's very too much to hope for, you know? Humans don't Ravid Shwartz-Ziv: Mm-hmm. Surya Ganguli: even do that, right? We I mean, if you think about the history of physics, there are many false starts and there were many experiments that killed off theories, right? And and We actually don't have a precedent for one shot generation of solutions to complicated problems anyway. Ravid Shwartz-Ziv: So it means that we start with agents. Surya Ganguli: We might be stuck with agents, but it might not be a bad idea, you know. I think we will Allen Roush: What do you Surya Ganguli: always have verification loops in verifiable scenarios. Like I don't think we can get around that. I mean, in some sense, science has had verification loops, rather it's the scientific hyp the scientific method is a verification loop, right? Hypothesis test, hypothesis test, update, right? Ravid Shwartz-Ziv: But do humans have this verification loop all the time? Surya Ganguli: Yeah, peer review, right? Have you never experienced three or two? As a joke. But but yeah, human Ravid Shwartz-Ziv: Yeah. Allen Roush: Ha ha ha Surya Ganguli: I mean you know, I can metacognition is something that humans have. We have a sense of when we don't know the answer. I'm not sure that LMs have as sophisticated metacognition as humans do. I think that's a big gap. but we have a sense of our own confidence in whether we're correct or not. and then we we recalibrate. In the face of new data, we update our beliefs, like we collect data to test our beliefs, right? Like we're always doing that. in AI research, I might have an idea, I might do an experiment. The experiment might kill my idea, then I modify my idea. Like it happens all the time, right? That's tool use by humans. Ravid Shwartz-Ziv: Yeah. Allen Roush: H how important do you think incremental slash online learning is for like really advancing AI further? Surya Ganguli: Yeah, so there is no train test split in human experience, right? Like this whole paradigm of you train an AL LM for a long period of time, then you're done. You deploy it, right? And you deploy it out there in the world. And and like I understand why industry has to do that because of, you know, the the mechanics of deploying a system, having it be stable over time and so forth. But ideally we would want our AI systems to be learning all the time. And I think the ultimate future is one AI per person, right? Like we wanna that's that's not the business model of current Frontier AI labs, but that seems like the AI that would be most useful to me, right? Something that observes everything I see. It adjusts its weight somehow as it sees what I see and as it talks to me, and and it it becomes personalized to me and it knows what I want and it solves what I want. Right. I I think eventually we will get there. But it might be a completely different type of company that has to Ravid Shwartz-Ziv: Okay, I I think we are out of time. do you have anything else that you want to add? Surya Ganguli: No, I think we covered a bunch of stuff. Yeah, I think I'm good. Ravid Shwartz-Ziv: It was really a pleasure to have you. Thank you so much. Surya Ganguli: Yeah, thanks for taking the time. It was fun, fun to chat. Allen Roush: Yeah, it was a pleasure. Ravid Shwartz-Ziv: Thank you so much.