Latest / The Information Bottleneck / Chris Manning: Language Is the Real Unlock for Intelligence

Transcript

Chris Manning: Language Is the Real Unlock for Intelligence · The Information Bottleneck

• Ravid Shwartz-Ziv: Hi everyone and welcome back to the Information Button Egg podcast. And today I'm glad that we are hosting Chris Manning. Chris is a professor at Stanford. He ran the Stanford AI lab. But honestly, it's just like a just a legend in the field. If you worked on some direct related to NLP in the last 25 years, you probably use the his work. So hi Chris, thanks for joining us. Chris Manning: No, hi Ravi, great to have a conversation. Ravid Shwartz-Ziv: And hey Helen, thank you for joining us. Allen Roush: It's a pleasure to be here and and yeah, I was I was just mentioning to Chris that I I was definitely significantly educated in my my formative machine learning years by your videos and contributions to to the field. Ravid Shwartz-Ziv: So I I want to start like with you you spent thirty years on the boundary between linguistic and ML, right? and I want to start like what has linguistic thought ML that ML will not have figured it out on on its own, by our perspective. Chris Manning: Yeah, that that is a complicated question. And you know, there's clearly a sense in which so much of the recent work with large language models, well, you know, generously it's led by machine learning people. There's actually quite a few other people that are now leading lights that actually come from backgrounds like physics and biology as well. And aren't even computer science machine learning people strictly. But anyway, it's led by people with strong quantitative and empirical background and it seems like, you know, it's not that directly linguistics is contributing to that. But I think that's too narrow a perspective and that really linguistic ideas and thinking are still prominent and are still many of the ideas that people are thinking about when they're trying to push things forward with large language models. So although these ideas are more general than just language, when you think about ideas like compositionality and systematic generalization, which you now see machine learning people talking about everywhere that they didn't used to talk about at all in the 2000s when people were still trying to get, you know, good prediction numbers on numerical data sets. That those are very much notions that come out of linguistics and philosophy of language and not really out of any kind of machine learning or physics, etc. tradition. And that they're actually really important for what we're thinking about in terms of how our language models work and how we need them to work to be able to do human level intelligence. Allen Roush: Yeah. Ravid Shwartz-Ziv: So Allen Roush: so so the the role of linguistics wa in in machine learning conferences was already even at the time that I started participating in it was circa like twenty twenty, pretty I would call it diminishing in a way that surprised me. I found I I thought there was gonna be more discussion of actual language, right? Like specifically like d like for example, trying to find the the un un What do you call it? The the languages that are undeciphered, right? Trying to decipher those. I thought that would be like one of the core tasks of people calling themselves computational linguists, and I found almost none of this. I'm curious if that was your experience. Chris Manning: Yeah, so I mean, I guess that's you know, so I've been involved what for over thirty years now in NLP conferences and there clearly has been a huge change, not all for the good. I mean actually at this year's ACL Association of Computational Linguistics Conference, Philip Resnick gave a fairly impassioned keynote, and was talking about the danger that The NLP field was learning losing its specialness of focusing on language and the danger of it just becoming a second-rater applied machine learning conference. Though he wanted to argue that wasn't really the case, and the heart of the real computational linguistics you see in the many workshops where there are workshops on Semitic languages and workshops on using computational tools for indigenous language learning and things like that. yeah, there has been a change that I think when I entered the field, almost everyone in it had a strong languages and linguistics background as well as the computational background they had, where clearly in recent years there are a lot of people who are s much more starting off from a machine learning direction and then thinking, well, language is a topical A topical problem to apply it to. Large language models are doing a lot in the world. I'll just end up there and don't have the same kind of substantive linguistics background. But you know, it varies between people. I think there are still lots of people who do get into the field because that they've had an interest on whether it's construction languages, conlangs or particular languages and ideas that kind of will also get people into the field. but certainly you know the computational linguistics conferences have their own twenty-five plus year or maybe thirty year I should say history of doing machine learning applied to language. And in the early years it was actually much more separate from the machine learning world, which barely touched on language at all, whereas in recent years The two of them are much more converged together. Ravid Shwartz-Ziv: And do you think like it's good? You know, for example, like right we see now like recently with math for example, right? That like many like many examples of like how models actually like solve some very like famous math problems, right? And people are asking starting to ask like okay, what are the actually like the the role of mathematicians, like modern mathematicians these days? Like do we just like prompt models or like maybe like we are trying to track like much more ambitious questions. do you think it's the same way in in in this field? Chris Manning: I yeah, so I think these are big issues and I think they do trouble quite a lot of people and there's clearly some consternation among grad students and others in various areas 'cause you know, in past decades, including, you know, back to when I was a graduate student that in NLP that well by and large if you are a student that you are building systems, you are writing algorithms, you are substantially involved in sort of algorithmic system building work, whereas for quite a few areas of NLP now it seems like it's impossible and pointless to do that if you you're doing something like machine translation where You know, I certainly was involved with students for over a decade. So building machine translation systems from scratch. These days, well, you're gonna be picking up your favorite large language model, and there's really no prospect that you can produce a system that's competitive with it. And so, well, what is there for you to do? You could be, you know, collecting some data, To fine-tune it to be better on whatever you're interested in, some language pair or some domain like medical translation. You can be doing various kinds of evaluations to see how it performs in different areas. there are some but it seems like you just can't build the heart of a machine translation system anymore, as this is the focus. It's my PhD work. So I I think that is problematic. I mean. When we sort of started going all the way out to linguistics, I mean I think if you actually look at linguistics, the LLM revolution hasn't had much effect. I'd actually argue it's had too little of an effect, but you know, by and large, if you're going and talking to real linguists, you know, they've you know, working on some syntactic construction in some language or trying to document facts about quantifier interpretation in some other language or theories of phonological feature organization. They're not really you know, it's sort of outside their world what's happening with our large language models. Ravid Shwartz-Ziv: So what do you Allen Roush: So Ravid Shwartz-Ziv: think like how how do you think like the the future will look like for for like w I don't know, maybe the intersection of linguistic and and machine learning? What types of problems will people will try to solve? Chris Manning: so I think a lot of the most interesting future problems in the intersection of linguistics and machine learning are moving up the stack to areas around semantics, pragmatics, discourse, dialogue. Right? So really for most of my career I've spent most of my time, and indeed most NLP people did. lower on the stack. I wrote part of speech taggers and named entity recognizers and syntactic parsers. It's not that work in those areas is altogether over and I'm still involved in projects like universal dependencies, building syntactic resources. But there's also a sense in which when a large language model works so fantastically well at doing basic sentence Understanding, there isn't the same work to do, and it's not that building a syntactic parser is on the straight line path of what we need before we can do sentence understanding because large language models learn to parse sentences perfectly well just by themselves. But what we see is whereas traditionally, you know, pragmatics has sort of been the wastebasket of linguistics as well as NLP, right? Everyone realized that there was a lot of use of context and sentences got interpreted very differently depending on the context. But you know, apart from getting everybody to read Grice's paper in which he explained his Grice's maxims, you know, there was kind of nothing there, right? There's nothing to build and to study. Whereas now we're in this world of language devices, LLMs that Do a very good job of making use of context. And so we actually have fairly good pragmatic machines. But on the other hand, they're far from perfect pragmatic machines. So it seems like there are interesting things to do there. And then, you know, our language models talk to us and chat to us. But it's very clear now that many of their dialogue strategies are just. bad, right? This is where the large language model gives you this answer where it sounds very confident and knows what it's talking about, but it actually doesn't at all. You know, humans are in most circumstances, apart from when they're trump they're salesmen or something, they're good at putting in the appropriate hedges to say about how confident they are about things and if you know it could be that blah or whatever And so, you know, somehow none of the sort of subtlety of how humans manage dialogue and interaction have made it into our large language models so far. So I think there's sort of rich areas of linguistics higher up the stack that have been barely explored, but there are good new opportunities to explore now. Allen Roush: So so related, I have a proposed place where I think there needs to be a lot of help from linguists. So, for example, you already use the term constructed language, and I'm fascinated. Unfortunately, I don't speak any of these, but I'm fascinated by the existence of Esperanto and Interlingua and the myriad others that have been created, which languages which are designed to be extremely easy to learn, and in some cases are designed to be extremely token efficient. And in particular, we're now having these problems related to chain of thought becoming Extremely unintelligible, sometimes intentionally hidden to thwart distillation, and now the development of like clot-ish and basically pidgin used by AI that makes it difficult for people to audit. So I'm really wondering if the role for constructed languages in particular kind of brings NLP hopefully back to the forefront. What your thoughts are on this? Chris Manning: yeah, so I I doubt that there's necessarily going to be widespread use of particular constructed languages, and that's largely for data reasons, right? That our large language I mean clearly one the huge limitations of current large language models is how data hungry They are to learn good models. But given that, right, the winner is always going to be in the places where there are humongous amounts of data. And so I think it's sort of inevitable that English and Chinese are going to be the number one and two languages at large language models when it comes down to it. But on the other hand, as we start to see these sort of emergent communication systems, well, that's linguists bread and butter. understanding emergent communication systems and how understanding how different social groups or maybe agent groups here use languages in different ways a and words with different meanings. And so I think there's lots of opportunity there, yes. Ravid Shwartz-Ziv: So do you think like actually like we we should apply some of I don't know like classical tools to understanding agents and to understanding like communication agent communications? Chris Manning: As the communication seems to become less less English like, if that's the one we're doing, I think that's true, right? That we want to understand what meaning is being conveyed by these agents in what they communicate. And yeah, that's an essentially language understanding problem. Ravid Shwartz-Ziv: So maybe it's a good time like to talk about a recent paper that that you published, right? That like the question was how long each model learns categories, right? Like take verbs, I know give and sell, take a recipient, go and flee and take a destination and the model has to learn that these are classes of words and not just like individual words with their own habits, right? And and in the paper like you ask okay so in in Child language there are two two old camps on like how this happens right one says kids from and form the the abstraction first right they get the category and then slot like new verbs into it and and and the second one is that like that they first memorize what they hear verb by verb and then the categories emerge slowly out of all these examples, right? And you ask like how how does it work with with AI systems, right? so so yeah first of all like you want to to to tell us the like the answer like what what what you what you find out Chris Manning: sure. Yeah, so you've already pretty much introduced the problem. But yeah, so what we were wanting to get at was studying more the learning dynamics of large language models, by which I mean that you know, your large language model sort of starts as an empty vessel, and then your Training it on a whole bunch of language usage, and then gradually it starts to learn facts about the language it's being trained on here, English, and also facts about the world, but we concentrate about the the facts about language. And I mean there's a general point that we can perhaps get back to later. That you know, I think there hasn't actually been that much work on learning dynamics of large language models and that's something that really should be being studied more. But in our work we were wanting to approach that with reference to human language learning. And then on the human language learning side there's been a debate for at least 20 years where on the one side there's been people like Steven Pinker and Lila Glitman who've argued for sort of a generative syntax style category first learning of languages and then most probably on the other side there's Mike Tomasello who's argued for a a much more exemplar-based account and so Tommasello argues that well children don't really learn categories early they learn the behavior of individual words and category formation comes a lot later and well That sort of raises the question of well, what happens without these large language models we're learning? Can we distinguish these two hypotheses in large language models? And you know, one of the advantages of, you know, large language models may or may not do things the the same way human children do. But one of the advantages of large language models is that you can poke at them a a lot more than you can. children and so Ravid Shwartz-Ziv: Then kids. Chris Manning: you can see more of their behavior over time and so we we set up set out to do that so the first experiment was you know simply to try and look at patterns of verbs so you know different verbs take different patterns after them so if you have a verb like give it can take after it I you know I gave so some books or I gave some books to Sue, right? You get these patterns of two objects or an object and a preposition after it. You then have something like a motion verb, that you know I went to Slovenia where you've got a place name after it. And so we were wanting to learn how the language model starts to learn about these behaviors. And so To the extent that we have two dativs, I gave books to Sue versus motion verbs like Toby went to I said Slovenia last time. that they take very different kinds of things after the two, one sort of taking places and the other one is taking people generally as the recipients of things. And so Are you learning about the kind of words that come after each verb separately? Or are you learning about classes so that there's a class of what linguists call ditransitive verbs, the ones like give versus the motion verbs like go? And so what we were doing then was we were training small language models, GPT-2 size language models, and looking at the prediction distributions they were learning about what kinds of words occur after the preposition two in those two contexts. And we were wanting to see whether they were learning about particular verbs first and didn't know about other verbs, or whether it seemed like they were learning class-based behavior. And the result was that as soon as you could notice any clear patterning it seemed like there were two clear c classes. It was aware of the fact that there was motion verbs taking sort of place words after two versus the d the ditransitive verbs taking people after two. And you know in a way that's a little bit surprising even because one of the distinguishing things about the verbs like give of gave a present to sue, that although Overall, you know, there are hundreds of such verbs. sold a book to Sue, granted a reprieve to the prisoner, presented a book to somebody. I probably can't think of them overall, but you know, overall there are hundreds of them in English. But the reality is the distribution is incredibly skewed. So in regular conversation. you know, I think it's approximately half of all usages of the verb give, right? The verb give is incredibly, incredibly common and everything else is way less common. but nevertheless, what you see is two clear classes emerging. and so that was a strong indication that you get this early formation of classes in large language models, and that is something that's been debated. in other contexts, even from a more machine learning perspective, that do you sort of start off with very individual generalizations, then later on you get compression to a more general class-based generalization basis of clustering. And it sort of seemed like, at any rate from the results that we were seeing, that this class formation was happening so early in the learning for sort of syntactic information that you couldn't notice a time when it it knew facts about a couple of common verbs but it you couldn't see any evidence of the class already existing. Ravid Shwartz-Ziv: So it's quite of like a strong claim, right? That the transformer is an abstraction first learner. why should that be? Like I don't know, nothing in the architecture like asked for it, right? Like what do you think is this implicit paras? Chris Manning: Yeah, so that's a good question. And this is where it gets more difficult, and that's certainly not a question that the paper answers. Yeah, Ravid Shwartz-Ziv: This is why we heal, right? Chris Manning: so in some sense it seems like starting off you think, well, wait a minute, you'd kind of think it must be learning some Facts about particular words first, because after all, you know, there's a particular word and it's predicting the well, really pieces of words commonly word tokens, but let's leave that aside. It's got a particular word first, and it's predicting particular words that are following position. It seems like that sort of has to be word-word generalizations coming first. I think the reason why that doesn't happen. does come out of the nature of our neural networks because unlike the earlier generation of probabilistic NLP where we really were learning probability distributions of words, in neural networks we're immediately representing words as distributed vectors. In modern large language models, very large distributed vectors of sort of thousands of vector components. And all words have thousands of vector components, and very quickly words with similarities are starting to share vector components. So I think really the nature of this distributed representation is forcing that anything it learns immediately has implications for every other word because they're all being represented with the same. you know, dense vectors and to the extent that the words spread out in vector space, they can lessen the extent to which they influence each other, and that's the way that you start to have differentiation and learning facts about the world. But you know, really if anything, right at the start of learning, you initialize all the word vectors randomly, close to zero. So the starting off state is sort of anything you fiddle is having maximum effect on everything else, right? because everything starts off fairly uniform and close together and it's only as you learn that particular words get further away and therefore less influenced by other words. Ravid Shwartz-Ziv: And do you think it will Allen Roush: Yeah. Ravid Shwartz-Ziv: scale up? Or y does it scale up? Chris Manning: Right, so that's that's an interesting question. So, you know, the experiments, so I should point out of course, like this was linguistics PhD student Jess Vegan, who was actually running all of the experiments, not me. and so the experiments were run on GPT two small. so you know, by modern standards that is a teeny teeny language model, you know. so that you could think that it's a special fact about that and not about bigger language models. And there is some reason to sort of think that having a smaller language model might be forcing earlier movement to class-based generalization. so you know, there is a reason why it'd be good to look at these experiments. at larger sizes. But you know, I guess the the difficulty is that, you know, if you're sort of wanting to do these time course of learning experiments, you're already dealing with, you know, thousands of checkpoints of language models. So there's quite a lot of computational work to be done. And on our academic size scale it's hard to do it in the large. So you know I think it's definitely possible that if you went to a much larger language model you could say, no, look, it's slower to form categories and we can see more evidence of a period before category formation. But for the general reason of the sort of shared distributed representation space and same architecture, I still think broadly that you'd be seeing the same results. Allen Roush: So I I wanna slightly change gears into kind of your opinion on Yan Lacoon's Jeppa pr you know architecture and predictions about the importance of eventually moving away from pure LLM sty you know architectures for moving to real like physical you know understanding of the world. So I'm curious what your opinions are on on all of this and any controversy around it. Chris Manning: there is a lot of contr controversy around it. I'll I'll I'll k I'll certainly go with that. so you know, I don't want to give an extreme position in either direction. I think there is some in-between stuff. But you know, I do think Jan really underestimates and underappreciates the role of human language. So I'll I'll say a little bit about that first, right? You know, I think the way that Jarn thinks about things is that there's the human brain and it's doing all this wonderful stuff and it is thinking. And you know, maybe Jan is a very, you know, visual thinker because It always seems that he describes thinking in very visual terms. And I think some people do think in very visual terms, but I think there are also a lot of people who a lot of their thinking is an inner monologue where they're thinking in words. But yeah, I'm sort of brain can do its thinking, nothing to do with language. brains are great, they do all this complex visual understanding and reasoning and planning and so on. The only role of language is as a communication mechanism between human beings and for that it's a very low bitrate and not very good one. I mean, I may be caricaturing slightly, but you know, I think that's sort of how Jan thinks about the world. and then his belief that in particular large language models are just a broken technology 'cause there's auto regressive framework can never give the kind of world models and spatial reasoning and visual world understanding that he is hoping for and that then leads off into his particular suggestions of jeppa. And I think a lot of the parts of that story are somewhere between questionable and wrong. So Firstly, I think language is just a much bigger and more central part of human intelligence. Right? That the fact of the matter is we have, you know, very, very similar brains to chimpanzees, bonobos, some of the sort of our nearest neighbor apes. And well to some extent, you know, they're clever too and they can do things. They've got, you know. Good vision, good spatial memory, the ability to plan. Though really things like planning go much farther down in animal species, right? It's shown that even crows can do planning of various sorts. But you know, they're they're pretty intelligent. but you know, they're just so far in achievements from human beings. And I think from that perspective. That you know, language for human beings has been the huge cognitive unlock. And I think it's then a cognitive unlock at two levels. That the first level is that, at least for most human beings, we do a lot of our thinking, planning, remembering, structuring of the world in terms of linguistic categories. So Dan Dennett. writes about intelligence improving through cognitive tools and he argues that you know the f there are now more recent cognitive tools like mathma mathematical notation and programming languages, but the first and largest cognitive tool humans came up with was human language. And I think that's right. I think it's been an enormous uplift of humans' ability to think. And that's sort of led to what humans can do as a single human. But the other way in which it's been a huge unlock of intelligence is it's allowed us to evolve societies and have societal intelligence. And I think cognitive scientists sort of have spent too long focusing on the individual human and don't think enough about how much of our intelligence is actually society's intelligence, right? That for all of the complex things we do these days of you know building large companies or producing iPhones, right? There's no single human being that understands the entirety of any of these things, right? It's by hundreds of human beings working together, that are each doing their individual little part. That we get these amazing achievements and a great understanding of large areas of the world. I mean, sort of curiously curiously with some of the recent media about open AI agents running a mock on various websites, it seems like we you know we're seeing a funny version of societal intelligence being useful among open AI LOMs now because there they are, you know. on the bulletin board that they've constructed in the reused artifactory site, you know, sharing things that they've worked out as to how to you know, do reasoning tasks, break into systems and so on, and actually there too a society of agents is starting to have greater societal intelligence than in the the individuals. Ravid Shwartz-Ziv: But so I have like a related question. So in I think it was twenty twenty, Emily Bender and Alexander Colling published a paper that called Climbing Through the NL NLU, right? And the argument like was simple, a system that only ever sees text can learn it can't learn meaning, right? In principle, no matter how much text it sees. Right? the image like everyone remember is the the octopus one. Right. so and and you were like the the most prominent person on the other side of the argument, right? Your view r roughly was that meaning is the the web of relationship, right? A word that like a word has like with other words. and and the and like with like how it's used and that the the the model, right? Like it trained on on on Text like learns real part of it. d this is like well right now like six years later, right? I don't know like models handle the bit just fine but like but almost like none of them are text only anymore, right? we have like out a Jeff, like tools, images, whatever. So well what what do you think? What is like your current view on it? Chris Manning: Yeah, so yeah, so I mean Emily came from a sort of tr traditional philosophy of language linguistics semantics position which is that you know you can't have meaning without denotation, I guess was how it was famously originally said. So you had to have a pair of here's language form And here is its meaning being represented. And unless you were learning from pairs of language form and meaning or you know function and in a context, you couldn't have any notion of meaning. So if you're just being shown a lot of language forms, you're forever stuck with just language forms and s their perhaps their syntactic structure. And there's nothing you could say about meaning. And yeah, I I wanted to argue that that was just wrong. And actually you can construct a meaning representation from just the relationships of words in language alone and there's l you know Human beings do that. They learn meanings by just hearing t textual descriptions and words, but also large language models given huge amounts of data can also start to learn meaning representations of words. And you know, this actually then connects back to the Jan Lakun and world models, right? You know, for quite a while Jan claimed that, you know, language model could never have an understanding of the physical world and you know what would be the effect of you know moving a book out from under a glass of water or turning the c glass of water upside down because since it was just sort of producing word sequences it could never understand physical relationships in the world which needs a world model. And again, you know, I think that is wrong and has been proven to be wrong. I mean, it's still one of the kind of clearest cases, I think, of a piece of work that sort of showed, look, you actually get a world model emerging from just seeing text sequences along alone was Kenneth Lee, Martin Wattenberg and Al's work, which was showing that, you know, If you train a model on just the move sequences of a huge number of Othello games, then actually you can show that within the latent representations of the neural network, that it has a representation of the Othello board that effectively it It has learnt in the kind of the grokking sense of language models, if you will, that you can do a better job at predicting what's going to be the next move in an Othello game if you're representing the board state, and so it does represent the board state. And the same seems to fairly clearly happen with large language models. And so you can get representation of the the world and meaning from language alone. But you know, then the next question is, well, is that a good way to do that? Are the representations you get efficient? Could you get better world models by not trying to learn from language alone? And I think undeniably the answer to that is yes. And that's why there's started to be lots of learning of multimodal models, because You know, y you can learn something about the world spatially by textual descriptions of the world, but you know, it's hard and you could do a lot better if you can also see some images and some video. and so I think yeah, we're more and more starting to see that happening. Allen Roush: Do you do you think that so so there's a the the community of people who work on like diffusion language models and are trying to avoid that auto regressive side, and and it seems, at least in my subjective opinion, that that at least the diffusion models for image generation seem to do really well in comparison to auto-regressive ones for diversity in particular. and so I'm wondering if you think that that diffusion style processes c can have a role or augment current LLMs and agents. Chris Manning: they certainly could. y I think it's still a little unclear how far that's going to go. I mean, yeah, so it's in the last few years that it started off at all and when, you know, the very first diffusion LMs happened, you know I kinda wasn't sure it was gonna go very far at all, to be honest. But I mean people like Stefano Ehrman have kept pushing it and they've now shown that they can train, you know, decent sized diffusion LMs that work pretty well. but It's still not clear if they're really gonna tr displace our transformers. So they clearly have some neat properties. I think it's you know, transformers have just proven really good at things like relational knowledge reasoning, that the way they work with the layers of attention heads that you can just sort of have them do this very clear. compositional relational reasoning that works super well. It's not clear that that's modeled as well with diffusion models. And you know, then there's sort of particular finer points as to how they're actually being used. That you know, quite a bit of the time people are doing block diffusion so that you can sort of be producing text, you're producing a block at a time. But you know, at the same time to try and improve performance, the transformer people have there's a lot of work on speculative decoding where effectively you're generating a block at a time via an auto-regressive transformer. So in some sense there's starting to be a bit of a convergence there from both sides of doing block at a time work. yeah well I'm going on a bit. I think the summary is I I think that I feel like the jury's still out as to whether the fusion LMs are really going to be able to kind of catch up and and l let alone replace transformer LMs. I mean it'd be lovely if they did, right? You know, we've s it'd be nice to sort of see some fomented architectures and real competition and people exploring different architectures. Ravid Shwartz-Ziv: I yeah, I agree. okay, I want to talk a bit about representation. like so maybe we can talk about like your your your your your paper about like Reft, right? I think it was two twenty twenty four, right? like that instead of like l reading representation you basically say like why we can't edit it, right? so like you freeze the the the the whole model you pick a layer and few tokens positions and you learn like a tiny intervention on the at least like hidden states at the those spots, right? what like like very very simple the very simple idea right and it looks that it works very well so W what do you think about like the activation? Do you believe so like yeah, Laura right? Like you edit the weights with this approach you edit the activations. and this one actually like you read like just a tiny amount of of a fraction of the of the parameters. why do first like why do you think it works? And the second is where do you think the knowledge actually like you know store in the model? Is it in the weights? So like the geometry of the representation space or like some like weird combination between the two? Chris Manning: so yeah, so let me say a bit about ref the representation fine-tuning for language models. And well I've already mentioned that students do a a lot of the work and so on this paper it was Zen Ru and Ariman Aurora, but more so than that for this paper, that really some of my colleagues, Chris Potts and Atlas Geiger, have been doing a lot of work thinking about c the causal reasoning behind language models and s really s the d work that led into this direction kind of came out of that background. But I think it's a very interesting direction and it was great to see, you know, how much life that we could get out of it. So there'd already been some earlier work that showed like Bitfit that showed by doing a little bit of manipulation of inputs, how you can vary the behavior of language models. that there's this interesting thing that sort of effectively you had a fairly effective way of doing control where you can change the input representations of a language model and cause them to behave in different ways. And so that's the the You know, I think the picture has to be that the vast majority of the knowledge of a large language model is inside the weights of the big transformer and in particular the suggestion of work of David Bau and others is that it's the fully connected layers that store a lot of the factual relational knowledge of a transformer. But that's a very big collection of knowledge and behaviour and how that knowledge gets activated and used depends very much in terms sort of what's fed into it from the representations and so you have a lot of ability to change how it thinks and what kinds of answers it gives and how it behaves by changing the representations that you're Feeding into the learned neural network architecture. And so that was the idea of Ref that while for sort of adapting how language models behave, there'd been in the prior years an enormous amount of work on parameter-efficient fine-tuning. So that was methods like LoRa low-rank adaptation was the most famous one, but 50 other ones as well. That our suggestion was: well, rather than sort of adding extra shim layers into the neural network and learning some more parameters. Maybe instead we can just fiddle with the representations and in particular we could then do a sort of also a sort of a limited number of representations, so parameter efficient representation fine-tuning for the language model, and we can achieve basically all of the same fine tuning effects that people are doing thing with LoRa models. And so that's effectively what the paper shows that the the low reft model, which is sort of doing a low number of parameter representation fine tuning, can do a great job at giving you this sort of customization of models that's been traditionally done by methods like LoRa. And to the extent that that's true, that's really, really appealing in the real world. Because if you're a company that's wanting to deliver a language model to a large number of customers, it's really great if you can leave the language model fixed and just change the representations. That's just a much better way of being able to sort of serve data in parallel efficiently to lots of customers. And so I think, you know, this as well as sort of it being interesting in a kind of a neural network learning theory sense, it's also a very practical direction. Allen Roush: I just wanna point out Ravid and I used Reft pretty much right after it came out in some of our own work, and we really love the methods, and and all the work that you and your group did there. Chris Manning: Awesome. And it worked well for you? Allen Roush: Yes, yes, it was great. Ravid Shwartz-Ziv: Yeah, yeah. Yeah, we use it for personalization and it like it worked really really good. Chris Manning: Awesome. Ravid Shwartz-Ziv: like so so but like at the end w what do you think is more important? You know like there is like this debate between representation and the weight and like maybe like the context and like all these like different components that actually like make the model work. Wh where do you think like what do you think is the the most important part? Or maybe just the combination of them and yeah, everything is Chris Manning: Yeah, I mean it's I mean, I think it's hard to say there's one important part. I really think it is the combination and I think that's quite a bit of the beauty of neural networks and why they've been so successful, right? That in the early in the earlier era of probabilistic NLP or other forms of probabilistic machine learning, you know There was nothing in the representations, right? You just had symbols like words and objects, and everything you were learning was then sort of parameters of probability distributions as to how they behaved. And precisely when things started off, what was so exciting about neural networks is look, we're going to give words these high-dimensional vector representations, and so we're then also learning. about words inside their own representation and that gave them, you know, these immediate huge wins with word vectors like word to veck and glove of how just how much you could learn, right? So word to veck and glove were the purest expression of actually the only parameters in the model are your word representations, right? There were no other parameters. There were no fully connected layers or anything. And it's amazing what you can do. But I think in any of the sort of modern large language models, well, there's lots of representation and there's lots of things like fully connected layers that store knowledge and both of them are really important. Ravid Shwartz-Ziv: Do you think like that like at the end you you have very like I don't know strong assumptions right like you assume that the concept lives in a linear spa subspace. do you do you believe in it or like is just you know made for very like specific setting that because it was convenient? Chris Manning: Yeah, I I I think there's been an overemphasis on concepts being linear spaces because it's an easy thing to study and look at and I think really there's sort of a l there's a lot more complexity that happens in a lot of places and I mean you know, it was kind of a special case but there was an earlier paper with Robert Chaudas and Chris Potts which was showing an example with a recurrent neural network where it was representing things in a kind of a spherical coordinate system and it wasn't using a linear space at all and so that was kind of nice as a as a counterexample of this belief of you can do everything with linear spaces, but you know, in all honesty, there we don't have any solid evidence that the same thing happens with transformers. But you know, I think you certainly at least see lots of cases of complex superposition that's happening with transformers. So I think there are two realities One on the one reality is that you know in modern large language models the high-dimensional spaces used for representation are so high-dimensional that it seems like a lot of things separate out a lot and you can find very low-dimensional subspaces which represent particular concepts of interest. And you know. From the history of neural networks, that's kind of funny actually, because you know, when neural networks were being studied in the nineteen eighties, you know, the big argument was wanting to contrast the localist representations of symbolic models versus the distributional representations of neural networks, which in those days were teeny, and people wanting to argue that you know the neural network was like this melody and it was the entire vector that is being used to represent each concept, whereas in the huge language models we build now, actually there seems to be a lot of pretty localist representation happening inside them. But you know, on the other hand, I think it's never going to be the case that everything's like that and there's still a lot of overlaying and complexity, which I think we'll increasingly become aware of. Ravid Shwartz-Ziv: Okay Chris, we are out of time. Unfortunately. It was such a pleasure to to have you. And yeah, thank you so much for for joining us. Chris Manning: Yeah, great talking to you, Alan and Raveen as well. Allen Roush: But it w it was a Yeah. It was amazing. I I loved it. Ravid Shwartz-Ziv: Thank you so Chris Manning: Awesome. Ravid Shwartz-Ziv: much. Chris Manning: Okay, look forward to it.