Latest / The Information Bottleneck / Pierre-Carl Langlais on Building Models from Data You Can Account For

Transcript

Pierre-Carl Langlais on Building Models from Data You Can Account For · The Information Bottleneck

• Ravid Shwartz-Ziv: Hi everyone and welcome back to the information bottleneck. And today we are hosting Pierre Carl Langlais. He is the co-founder and C CEO of PlaySt. This is a French German AI lab that focused on data research. So hi Pierre. Thanks for joining us. Pierre-Carl langlais: Hi, nice to meet you all. Ravid Shwartz-Ziv: And today we we're going to talk about many topics but I think like we we can start with synthetic data or like data generation and why it's so important these days and what you think are the maybe the major bottlenecks in this area. Pierre-Carl langlais: Totally. So I think right now I think it's important for two different aspects we are kind of focused on on different sides. The first one is about learnability and efficiency to learn something within the model itself and especially memorization. So that's the focus we had when we release this thing called synth, which is a synthetic pre-training data set, which is kind of unusual because most synthetic data is more done for post-training usually. But we did this idea of okay, what is the minimal amount of information a model should memorize, basically. And since I was in Wikipedia for a long time, I was a Wikipedia admin, I knew there was this thing called Wikipedia Vital Articles, which is basically like a curation from the community of everything that you should know about, which is kind of a good shortcut of okay, what is actually everything a model should know? Because if we are realistic, actually everything we read for our life is kind of tiny. We were like maybe a few hundred books, this type of thing, but we still know a lot and we can still converse. And the idea was whether okay, how can model memorize all these things? So there was this first idea of how we can use synthetic data for what is called static rephrasing, but more like a way to amplify information, to create variation, to avoid collapse you'll get if you just repeat the same information inside of the models. And Now we are exploring a variety of other techniques to enhance in liability, and I can talk about that later. the other topic which is really usually important because we have a French German lab, as you underscore. So we are in Europe and we are mostly working with regulated sectors, and it means working mostly with sensitive data. So just last week we released a model with the subway of Paris. We work mostly also closely with the telecom sector, with banks, and all these kinds of things. And there are two issues here. The first one is that Their data is not really in pre-training data right now. It's not really in the big crawls. So the models are not that great actually for many tasks they want to do, even the frontier ones. And the other issue is that they cannot really use their data for training because it's sensitive. And even internally it's complicated for them to share, actually. Even from one service to another is complicated. So here I we can talk about that more in detail. So I think it's really about how we can simulate data we don't have. basically and simulate it in all kinds of way in terms of formatting, in terms of content, in terms of process, term of infrastructure, all these kinds of things, which is kind of very connected to what frontier labs are doing right now with environment and all these things. But with this more specific target that we have right now as a as a company actually, not just a labs, but it which is really about how we can basically create good models for customers which are not having them right now. Ravid Shwartz-Ziv: But but the idea is like to create a model that like can we can pre train on this data and that this model will be like comparable to to models that are trained on like large amount of data. and and and w what are like so why to do it? Like one is like I say like we have like we want like I know. smaller like amount of data. And the second one is that because like now we know exactly the the license and like the thing that like actually like the model, like the data, like the properties of the data, right? Pierre-Carl langlais: Totally. I think that's one of there's a fair a maybe there's three reasons and in the end, but I think the first one is the difficulty that you want to train a models that are going to be much smaller because it can run on premise. That's what we did actually with the French subway. We trained a six hundred million parameters model for the monitoring systems. They used to use a very large model for this, but then they had to send the data through API. And it was kind of sensitive because it's a lot of complaints, for instance, by users, it's a lot of Tweets, for instance, sent by travelers when there is someone sick on the subway, this type of thing. And it's not a trivial task, either, because there is a lot of under understanding, interpretability you have to put to really understand, okay, it's not a hoax, it's really something happening, this type of thing. So you have to take all these context clues. But yeah, we try we create a simulation of the kind of data that we're receiving at scale. And we publish this paper in a CL, it's been accepted by CL Fending, and yeah. We manage basically this way to kind of reduce the size of the model. Second one is really about auditability, which is again really important for this sector. Actually, it's kind of mandated also by the I Act in Europe, but even interprofessional agreements, I think even in the US you can have this kind of requirement that you want to audit the system, you want to know exactly why it does behave in such a way without having a black box. So aircetic data is really nice in a way because I think and We can talk later about Ciophetical data because we worked a lot on this before. But it's kind of the way to make sure that we can get data that can be releasable and still very competitive because you can use vertically all the latest advancement and say it's reasonable because where by default I would argue that all static data is public domain because there are no authors, there are no copyright. Send the cell is the approach we taken by Frontier Labs. Right now we see a lot of talks about model IPs and control over static outputs. But Still, it definitely allows much more flexibility into sharing the data, also because you can remove anything that can be sensitive, because you can simulate it and so on. And I think the third one is really about even unlocking new capabilities inside of the model itself. Because once you work with environments that can be very different from the ones models are being trained. And that's a kind of thing working especially in telecom right now. We have a partnership with GSMA, which is one of the conglomerates for telecom industry. we're going to release the copies together actually this week. And it's really about yeah, we want to get good models for dealing with log systems, in telecom systems, this type of thing. And right now it's really the kind of thing we try and aware the frontier because not really the kind of thing when even good best models are not really being trained on that right now. Allen Roush: so so so so what what do you w what do you consider to be like ethical in the context of data sets? Because I think that's something that you've worked in for a while, is trying to create data sets that are not, you know, mired in claims of copyright infringement and and myriad other notions of problems. So so how how do you define that? Pierre-Carl langlais: Yeah. I think it's a very good question because when we released the paper on common corpus we did not say just the largest like okay freely reasonable copyright free whatever you want to data data set but we put intently as the largest ethical data set for training which was kind of intentional on our part. So I think ethical usually the most basic concept is more about data that you can release, that we can audit, that you know there has been some kind of consent. for shareability. And so usually but there are a variety of approaches and several actors try to do that and we try to do it with common purpose. What we've made concretely is first and foremost navigation of everything while you have a guarantee that you can reproduce it, including for commercial use. So basically it means everything under free license, everything under public domain because copyright have expired, everything that is that another Have any kind of rights because for instance, it's public domain of information, everything that is open source, for instance, for everything that is under free software. And importantly, there's something you can have attributions, which is another criteria on top of it. But especially important for anything that is free licensed, you want particularly to maintain and to quote some common corpus. We have the mention of the authors, everyone that we can know it, but equally so that we have this kind of attribution being maintained. Obviously. Not once the model is trained, which is also a part of contention in that space, but still we wanted to maintain it at the scale of the data set. We want everything that is transported to maintain attribution. But then there is the third one, actually we are kind of very conscious about that, which is more about other things like for instance multilinguality, the question bias, structural bias, which is something actually we worked much more on right now on common corpus. And we're also kind of Really expecting also synthetic data to unlock that because originally Common Couples were really, really Western biased. It was mostly like Europe and US, mostly because that was kind of easy to get the source of information. But in the latest release we've made, February, we added a lot of content. We have a big Chinese data set. We added a lot of content also from India, from Japan, from other countries. And with JSMA, actually, we have now a collaboration where we train the very small language for language detections, which is kind of the kind of core bricks you need to be able to identify even low resource languages because there's kind of negative feedback loops where frequently we don't even have the tools to identify it. So I would say ethical is the so this part of things which is really about how actually data is kind of representing the world you put inside of the model. So it's not just about okay, just the information in a vaccine, but it's really much even like worldview and everything you have to get accessible. And kind of the vision that we have right now is also about yeah, correcting for this type of bias, creating also the tooling that is going to help, for instance, to digitize content in many African countries. We have deployment projects in Senegal right now, and we see definities interest to have also models that are being much more culturally attuned. So I think that's also the dimension by the KI that we want to push for and also the reasons why we also push for this concept of ethics in a kind of broad sense. But I I think it's also a bit kind of loaded because ethics is not really not the same as anthropic will claim to have a concept of ethics, which is definitely not ours, which is much more anchored in openness and transparency. But we still want it to push towards this wider conversation. Ravid Shwartz-Ziv: So how can actually like you you build it in practice? Like okay, you we have the like the idea, we want to build like a data set for pre training, we understand why it's good. What you actually do it in order to build it. Pierre-Carl langlais: Yeah. So I think the night the funny part is that actually I started building it way longer before I got inside of LLMs actually, because I was in all these open movements. So I was in Wikipedia. I was in open science. I was really activist at the time, a bit like the time where Aaron Schwarz had all these issues and I even leaked a severe contracts and stuff like this. So And yeah, I was kind of close. I I was doing my test in syntax mining on public domain corpus, so I know so that it was this large iron structural heritage that I said they were kind of lying around. And so I would say it really started the idea in twenty twenty, so very early when yeah, we started to scale basically and Before that, actually it was kind of more commonplace to use like Wikipedia data set I could release, but suddenly when the scaling began with GPT-3, you had this big crawl data set, and effectively GPT-3 paper was almost the last one where it to really detain data sets they have been using. So and suddenly there was this kind of discourse, but yeah, it's completely impossible to train on data that we can release because it's the right size and all these things. But size not necessarily. Or at least definitely much more than you believed, actually. And you can definitely scale to a large amount enough to basically create almost a quick training ecosystem. So it really started just next there, and actually it was the original journey of players. So now we have kind of expanded a bit in the other direction, especially aesthetic data, but that was really the original impetus. So Common Corpus was our first project. We started it around public domain originally, so everything kind of digitized cultural heritage because it was really like really big sources actually which were kind of not being collected or curated. And gradually we expanded it. So it was more like a continuous project, which is a bit different from other project that was also undertaken in around the same period like Common Pyre from Ellioter, which was more like okay, there is a one shot and a lot of work. Here we worked very gradually aggregating other things, every one passing by so there have been multiple updates. And yeah, so I think it The aggregate aggregation was like two years and it involves a lot of work in terms of provenance, I would say, which is really the kind of different outlook than when you just take a crawl. Because when you take a crawl, okay, you have this kind of agnostic crawler going through the web, and then maybe you try to filter afterwards. Yeah, it's kind of the reverse because you start from a situation when you will know already, you have to know already what you're going to find, and then you go in depth on these sources. Because What we discovered is that right now there is less than two percent of common corpus which is inside crawl dataset. A very tiny amount which is being in both sides. Because if you just do featuring of crawls and you take only the path which is the free license, you have very, very little amount. And the reason for this is that crawl as an infrastructure is kind of faulty. Because it does not have PDFs, it does miss a lot of content because everything that is kind of not cross-linked, because the Well, the crawler by definition has to crawl. He has to move to one part to another. Anything that is not being indexed, it's a large amount. Especially for anything that is large data set, like large digital libraries, type of things that are not being indexed right now. And so actually in a way, and especially for the global soft content we get it, we have now almost a role of preservation, because a lot of the data we've been aggregating actually have been either at high risk of disappearing or already disappeared. like for instance the big chain is that I've mentioned it had been collected thanks to two CEDERS that kept it on torrent because it was all kind of open data, Chinese, KSL and stuff like this, which the government stopped opening basically adjusted during the pandemics, even though it's open data. But there was no institution, no place to keep it. So you really have to do this kind of investigation to actually create open data and maintain it, create it and put it back. into a place where you can you you can get so yeah there is a collection part and there is a processing part and one of the biggest challenges is that most of it is pdf which is a big difference with web sources which are HTML which comes also with their own challenges but with PDF it means okay you can have also our errors you can have this type of thing you might want to correct it at posterior you might want to digitize now we digitize a lot now especially scientific data We have own pipelines with visual language model for this. But yeah, it's a different it's a different approach. The nice part is more on the side of, for instance, context length. Like you get much longer documents through this, which is kind of interesting. Plus, not only longer, but better contextualized. Because we get a lot of metadata about the documents. We know the authors, we know where they've been published, in which date, in which context, this type of thing, which is not what you have when you have a web crawl. Usually you just have the URL. So the moment it was collected, that's the end of the story basically. So my hope right now, but I'm not sure it's going to happen, but it's that's going to be more interesting open data for this purpose, especially in the context of agentics system, where you are more interested in content which you can trace, which you can know about, so that you can train model to be good at search, to be good at all of these things. so it gets more value basically. Allen Roush: How important is the harness, in your opinion, with with running agents and going forward for synthetic data creation purposes where there's a huge demand for agentic data that that helps models perform better in common agentic harnesses, Claude Code, Cursor, open code, codecs, etc. how how have you and your company been responding to the the kind of latest development and demands for synthetic data? Pierre-Carl langlais: So I think it's kind of a new development in a way for synthetic data. The approach we had used originally was more about fine-tuning models, which was kind of unusual because usually when you do synthetic data, people will just take an existing generalist model prompted at the end of the story. Whereas we really wanted to have a high level of control, for instance, even on the style of reasoning, how it's being structured. So there was very specific steganographic style we put inside of synth. Which is something which is very, very complicated to get if you just get prompting because it's too much. You ask for questions, then you ask for styles, then you want for other things and so on. And I think there are still now a need we have been very interested, especially in a recursive language models approach that being pioneered recently, to basically yeah, create agentic traces in a kind of recursive way and deeper sequences of things. And especially using adjunct also and that kind of look back to the previous questions for data curation and data processing, because a huge amount actually of work you need to do to get good static data is actually seedings. You want to have good sources of inspiration, you want to use you you want to ground things basically because it's always based. So the quantity can be kind of elaborate because you're not bounded to just use the text. You can definitely use structured data. We use a lot of Wikidata for this, for instance. Or you can use any kind of information you want to access. But more and more we are in a situation right now when we want to get and I think that's really the future where we're directing ourselves, not really our pipeline as it is right now, but more like more and more this kind of process of creation, this kind of process of deciding even what is the interesting task we could create, what is the interesting static environment we could create. But this process will have to be created by models themselves at some point. Because that's the only way you can scale. Because if you start to Minutiusly constructed okay, this environment and this and that. You can do it in your large labs and you can ask for millions of people doing that. It's not something which is possible right now. So we need to find a kind of scaling perspective. And yeah, yes, the RS can be definitely useful. I think the other part is also about I think it's really a concept from Eliaso Schedular Direct, which is called training on deployment. because once you have a good I don't think I already Azure Text systems are what is something you want to train on, but again you get the model better at it, better and understanding okay, what is infrastructure I'm currently built on ill. So a lot of the thing we are working right now with our partners is how we can adapt well, it can be the MCP but even straight the API system in such a way we can simulate it and create a kind of good harness for it first and afterwards use it for simulation and then it loops back to we can use it. But but yeah, no, I'm not seeing really I'm definitely seeing the NS as something that is going to be more important in the months to come, even for static pipelines. Ravid Shwartz-Ziv: And how you like think about evaluation, you know, like how you evaluate this data set? Because like it there are so many works about like the connection, right? Or like how you actually want to evaluate models like during pre-training, right? so how you evaluate the quality of the data? Like you have like one like two problems, like two like different axes here. Like one is the what actually like the the metric Pierre-Carl langlais: Yeah. Okay. Yeah. Ravid Shwartz-Ziv: for the model, right? Like when you pre-train, like do you measure only on the post-training like metrics? This is like one question. And the other is like at the end you you tie to a model, right? Because like you have the data and the only way to evaluate, or like maybe not, but like a reasonable way to evaluate it is like to train a model and to say okay this model like is good or bad with this data. But it's only like one specific model. So What's your thought about it? Like do you need like several models, several metrics? Pierre-Carl langlais: So the approach we have been using for static pre-training, which is interesting, is that we can train from the start with reasoning traces. Which is kind of nice when you want to basically not just use the kind of very soft metrics that are being used in pre-training, like LSWAC, this type of thing, which just okay, the model is kind of having a broad common sense or linguistic capacity, but more you want to measure reasoning capabilities from the start. And here our big inspiration was especially the work of physics of language models and the idea, okay, how we can basically assess capabilities from the start and even start thinking in terms of selecting, for instance, doing architecture choices for targeting specific capabilities from the start. So the approach we have in SIT in synth and in the paper we have been preparing for Synth, which is not outs out yet but soon. is really about okay we kind of have we have a correction right now of about forty five different evaluations in different domains, especially specialized domain because we want the model to be kind of broad. well as we say more booba than kiki basically so having not just MMLU but really all kinds of capabilities and we see basically it was really interesting for us synth because it was already a scaling bet that we take all the vital articles of Wikipedia, but it's kind of a good conversation of most of human knowledge, even kind of specialized. So it's not just okay it's good at MMLU and then you can say okay it's circular because MMLU obviously is measuring high school capacities. But what we have been seeing instead actually is that the model is good also even for more specialized telecoms, domains type of thing. And the nicest part really we show in the paper is the fact that we have reasoning traces to assess these things, even for small models, basically. And what we saw especially, it was really interesting, was we saw basically an emergence of the model capacity to correctly assess in in certain uncertainty. Like for instance, stating, okay, I'm not really sure about this information. And because we have this in the reasoning traces where the model can memorize things and say, okay, I'm kind of circling on this specific domain on specific topic. And then what we saw is that the smallest model, basically when they were trying to edge or whatever, it's not really making sense. But the larger model when they were trying to edge, definitely it was making sense. It was kind of connected actually with what you knew and don't know. So this is the kind of thing basically that does really allow to have a closer, I would say, outlook of how it can I think also an advantage also working on SLMs is that a complete pre-training is kind of relatively fast, even more so with synth. so we can definitely make decisions at this scale, but we can definitely assess this. the other side of evaluation which is really important actually is with our clients because the big issue that we have obviously is the fact that what is relevant as evaluation is necessarily going to be available. So here there are two different situations. Like for instance, it can be that for instance some industry decide to create an entire evaluation suite, which was the case with GSMA, so they have this kind of big telecom benchmark. They have been releasing with six different sys evaluation systems. So it can be knowledge, it can be analysis of logs, this type of thing which we have been using. And what was interesting is that they showed by the caliph there was very little improvement even for frontier models on their evaluations because it's not the kind of thing that is being targeted. But the other interesting collaboration we had was especially that for instance, for the subway models, it was by Cali's Our client that designs the evaluation itself internally with travelers and everything like that, trying to assess already what is working or not. So yeah, I think it's evaluation is really like a key skill, not just using existing evaluation, but building some actually are really needed because in the end it's really the target you're going to have. And that's what the frustration I got right now with a lot of models we call okay, they are kind of benchmarks or whatever. it's not just that they are bad, but they are more they're targeted towards a specific use. that it's not necessarily going to meet the variety of even the intent actually because a model can be kind of limited range actually is fine. especially if you make a small model designed for a specific industry, for a specific use case. But within that it should work relatively okay. so yeah, there's a lot of intuition that have to come into this to be able to design properly. Ravid Shwartz-Ziv: But do you think like the a at the end from a pre training data like we want something more like general, like it like will be good for different domains, or like we want something even for pre training we want something very specific? Or like some domain together. Pierre-Carl langlais: I think yeah. Yeah. I think that's really the key thing. We wanted general. So definitely for instance MLU and this type of thing, okay, that was kind of the primary objective we had. What was really interesting is that we got good results on this type of specialized or semi-specialized because sometimes it's not necessarily advanced knowledge, but all kinds of evaluation we have been collecting through the existing literature in all kinds of domains. even some where we don't necessarily actively worked on. But for instance yeah it turned out the model which we're trading were really good at things like petroleum, for instance, this type of thing. So we we have no idea of what is really inside, but it turns out yeah, just drawing from the knowledge we have put inside of the model turns out to get the model better at it. So I think yeah, I think we need both because especially I think it's really interesting to use specialized evaluations you in a domains you're not necessarily familiar with. Because it's a good warning in terms of okay, your model is definitely getting better as something else you're not necessarily anticipated. And I think it's a larger issue because right now I think the system models are so complex, especially if you want to get toward agentic models. There are so many things you can do, so many environments you have designed. So you have to think at some point about something like a generative principle. You want to generate the universe of possible environments, which you cannot control by definition. So that's why it's super important to have this scope of evaluations which is not limited. Allen Roush: And in general, I think you mentioned as part of this, the term benchmark, you know, like maxing, right? Like people kind of over Yeah, yeah, yeah. how do you fight that? And in in particular, I guess the second question is if you've had any experience with GLM five point two, do you think that they're also benchmark maxed because they like everybody else seem you know, people online are like, it's an Opus class open source model and I I just have never seen that with any of these even big Chinese models. So I I wanna hear your opinion. Pierre-Carl langlais: Next max, yeah. Yeah, I think it's no I think it's a very good question because I think it's right at the heart of whether or not there is a big lag between open and frontier frontier close frontier models. And I think the biggest issue for the lag is the fact obviously frontier labs they have their internal evaluations that are not open, that are not public, that's not all these kinds of things. And I think what's also going the direction of the lag is the fact that we have almost a market mechanism right now. People are definitely voting for closed source model. We can say right now there have been a big catch-up of this kind of open model. A bit now tips for pricing reasons, but not necessarily for capabilities. So definitely I think it goes direction of the lag. And I think the reason for it is also what we have been having also this cycle with reinforcement learning and with syntic data that didn't make it really hard actually to incline basically an evaluation. So definitely, yeah, you can benchmarks in specific. But then whether or not it's going to give you a kind of broad ability and whether or not you're going to cover all the range of use cases that people want to do with it and even if you're not going to arm it in the end because it's so the issue, sometimes best model are better for a right range of use cases, but you're going to break it during post training because suddenly you make a that's the thing actually we saw with some Even synthetic datasets, well I'm going to name some but are on a game phase where okay model strain audits are kind of better on the benchmark, but they're completely broken if you are using it for anything other than English, for instance. So i I think it's really about this kind of equilibrium and also the part where it's kind of interesting always to think of model for deployment. So one of the things I do all the time when I'm printing a model is just reading the output. Like I I have my set of questions I'm going to ask. I want to get the feel of the model, I want to get an idea of how it's going to behave, even in a kind of more adversarial way. And yeah, it's obviously it's very non-scientific, I must say, but I think it's not very that dissimilar to from what I hear Frontier Labs are also doing when they are training a model they want to get this vibe, you know, of okay. how the model is kind of behaving, even in a way which is completely un orthogonal, unrelated to any kind of benchmark optimization, but you want to get by the case this behavior right. Ravid Shwartz-Ziv: And so and d do you think like there is like I don't know some specific data types that you that will be more like I don't know robust to benchmark sync and versus data types that are like are more like specific and like you can actually like climb up on on some specific domains? Pierre-Carl langlais: I think the case, but I realize I haven't really answered you about GLM five point two because I'm right now trying to deploy it for Citic Data and because I've been seeing all the interest for it and well I don't have the answer yet because I haven't tested it enough. But I think it can be pretty interesting because from what they described in their paper, but again to be seen whether they did try to make basically this idea of generate C Citic environment, which is also something which is really close to my mind. Like okay, you're not really just going to encraft all of these things because also for there's a cost issue but still but I think it's also the issue of how you can cover a wide range of things. But you're going to try to generate the task, generate the range in a kind of expensive ways, in such a way that you're going to cover much more information. So and but yeah I think right now it's a bit difficult right now to say it's just about one piece of data that are going to allow to you to overcome this because anything can be reward, anything can be any climb. So it's more about a good practice, I must say, of ensuring diversity of even the targets you're going to have that is going to help you out to to overcome this benchmark thing I would say. But I I think it's also about what's the models you want medically. Because for some organization definitely they're just fine with okay, they want to get good benchmarks at the end of the story. So they're not necessarily going to be interested in actual deployments, which is not really our case. but yeah. U it's more about the practice of things. Allen Roush: and and from you know, when when you're talking about like diversity with synthetic data generation, what are your favorite ways to improve that? And what do you think works best for improving creativity diversity? Pierre-Carl langlais: So it's a very very good question because it's a lot on my mind. so I think first obviously it's going to be seedings. you don't just want to prompt a model because that's how you're going to get diversity collapse. So you want to get some alignment with other things. But then on top of it, you might want to control for the model behavior. Something that was actually pretty well for synth is that we had this kind of controlled features that we used to basically ensure that we get a range of combinations over For instance, the fact you want to ensure that the model has at least 20% of questions that are going to be negative, like going to elicit a refusal, going to be absurd because it helps with getting the right world models. You want to get also, for instance, all kinds of input that could contain, for instance, typos, all kinds of diversity to basically increase the surface area of what the model is going to meet. So I think that's one of the first things. But then what is really interesting for me right now is also how you can create even something more diverse. than the kind of uncleanic sources you can access to, which is something which is very interesting for me. And really at the source of the collaboration was with Nvidia, which is about creating static personas. So the core idea is how you can we made that for France who have been their partners there, started sooner with the US. And the core idea was really about how we can model the demography so that you can get realistic persons which are kind of the right combination. So we use a probability graphic model for this, but the case that okay You start from a region, then obviously in this region sometimes there's a of old people, for instance, and then since they are old, they are not going to have a large educational background, for instance, are more like less likely to have a to be immigrants, all these kinds of things. And what's interesting once you got that is that we can get particularly also seedings for a lot of people that are not going to necessarily, for instance, write a lot of text. Like it's not going to be that presented in clear in the common in the cross. It's been More important because this is the kind of approach we also exported even for our projects in Africa. We have met some personas in Senegal, this type of thing, and so again of people that are never being represented in crawl data sets right now. So I think it's kind of interesting this kind of approach because it goes a bit against the grain where okay, stick data is always thought, you know, like power representation of the thing you're going to get into organic data. But My belief right now is that if you get your pipeline right, actually you can even correct some of the bias, you can even enforce some presentation and say, okay, we want it for instance this environment to be from the perspective of someone, let's say in Senegal that are going to have health issues and so on, and then obviously there is no hospital open whatsoever in the region. How do they manage and how do you get access for instance? But so there's this kinds of things right now which are not being really simulated. And Obviously it's creative, so it comes with risk because obviously you're sim you're doing simulations, but in a way which I find very, very interesting. Allen Roush: Interesting. and do you do you think that model distillation attacks and similar kind of problems are like actually reduced or you know by reducing like thinking trace visibility as we've seen from like Claude Code and Codex and the closed source model providers, you know, they're trying to hide the thinking traces. So I think you alluded to this earlier as well. What's your opinion on that? Pierre-Carl langlais: Yeah. Well, I think I have a bit of a I would say controversial idea around that because I definitely see as a way right now for large model providers to put some kind of intellectual property on the model output right now. So it's something which is complicated on the perspective because obviously they train on all of the web They don't really ask for permission, they push really hard for fair use, and then suddenly we start to see this discussion even about model IPs, which is now being mentioned. Open I recruited someone to deal with our model IPs. And this idea of distillation is kind of a way to have a claim of intellectual property and static outputs in some ways and to prevent people from using that. And so I think obviously there are a lot of academic interest around that, and I seize upon because right now We have been seeing this maturation over CFI, so now there is this need of okay, they want to protect the output. And I think it's something we're going to see even much push once the model will get clever enough to create actually very powerful actually output, which is not really the case yet, but what we have been seeing with okay, models for instance that can solve complex mathematic conjectures, this type of thing, we can get into very fast into a situation where even model could make scientific discovery and then wound the patterns, this type of thing. So We're seeing kind of this push right now. And I think that's why it's really important, but it's not done, for the open model community to kind of step up its skills in terms of synetic data generation and curation. Because just pure distillation in any case is kind of putting the ceiling in any case, because okay, you're not you're not aiming to go further than the model you're targeting for. which is already well, it can be pretty powerfully if you have a very small model, but still it's going to put you a ceiling. whereas if you have this kind of good process, you kind of train models for this and you put a lot more information that were available for instance the product at this type of thing, you can definitely go beyond the existing capabilities. And that's how basically you can create this kind of open ecosystem I would like to see at this point. So yeah, I see it more like a com an emerging conflict I'm seeing right now between okay there is This large industry which has its own interest right now basically to close things down, which makes sense. But on the other hand, this kind of newcomers, definitely in China, but also hopefully maybe in Europe, which are going to be more interested into the side of okay, how we can pull even our efforts together to create good data practices that are going to allow us to get competitive, not necessarily at the frontier level, but at least on some industrial application of I Ravid Shwartz-Ziv: And wha what do you think about pre-training versus post-training data? Do you think like we need specific data? it can be I don't know synthetic or like not synthetic or like what people are doing with their environments, our environments these days? Like so do you think like do we need some specific synthetic data there? And like if so, like what are the the paths to to a good Data. Pierre-Carl langlais: So I think it's it's no, I think it's a very good question because I've wrote one of the first I think it was almost the first blog post on mid-training two years, one year and a half ago. And at the time already I had this suspicion, okay, mid-training is basically about scaling, post-training. So medically at somebody want to create much more environment, much more data, much more memorization, stuff like this. And my suspicion already at the time, which is actually the thing we've made with synth, but which is particularly static pre-training that okay. At some point if you continue to grow these things, is it not actually what you want to train on? And I think right now we have this kind of divide. It's kind of unknown whether or not we're going to go this road, but definitely we push forward in this direction where what are we actually modeling for actually? Is it just language or is it more like infrastructure, knowledge, particularly any kind of complex systems you want to model? And I think adjuncts are kind of moving in this direction. And should we really wait? To have a complete pre-train before we start doing the thing which is actually interesting for us during mid-training, during RL, doing all of these kinds of things. So the idea we pushed actually with Synth was really about okay, we want to train directly on reasoning traces. Zifra did the same thing in their latest model too. A few others have been doing that, and I'm kind of suspecting even Shiny's labs. So from what we can gather from Moonshot, I think they also did something a bit similar in terms of putting a ton of synthetic data, likely with stick reasoning. very early in training and then you can get basically. I think it comes with the direction again of scaling because once you get all these environments you're going to create, to create all these kind of agentic scenarios and stuff like this. And maybe suddenly the main entry point with the model is not going to be a chatbot, but really the kind of thing you have a clot code. Then obviously if you're interesting with clot code Your versioning is going to be generated by definition because it's not something which exists in the real life. Intelli versioning never never exists in the real life. So you need to generate it at some point, or you need to create it to create the condition for it. So my expectation right now is that yeah, we're probably going to have this kind of a break where what used to be pre-training is going to collapse into something else. So actually that's the title of the paper we have so we have submitted long ago for synth, which is called It's All Training. Which is really about this collapsing of okay, maybe we don't need to separate right now between pre-training, mid-training, post-training and okay, we can collapse it all. I think it makes at least a lot of sense at least for small models. to be seen whether or not there are still some benefit to start with a large training of unstructured data first, again, maybe for diversity purposes for large model, but at least for small models where You really want to target as much as possible the efficiencies, their use for applications and it's okay basically if they're not going to be having large breath, it makes much more sense. Ravid Shwartz-Ziv: And d do you think like these small models like can be competitive to larger models? At least in like narrow domain. Pierre-Carl langlais: So I think I depend for its few space. Yeah, yeah. Yeah, exactly. That's basically what we did for the subway of Paris. We is it's a model in production right now, so it's not a poke, and it's being used actively for the monitoring system, so it's very tiny, 600 million parameters, it does its task. And I think we're going to see a lot more of this because I think actually for some of this kind of task, small models are good almost by design because the nice part is that it can be very bonded. You want them not to go all over where like the issue actually we had seen with other applications, for instance in the early days of ChatGPT where like for instance a lot of people were creating commercial chatbot, but then you can ask all kind questions, it will go off rails. yeah, you can get much more constraints, you can deploy it even offline on safe infrastructure, and it can be really models for infrastructure. So I think that can be the first push. The second one, which is a bit more speculative, but I see a lot more interest for that, is whether or not for instance we can get a market for subad jones, for instance. Because we see kind of the emergence of this needs, I have it very frequently that okay, interacting with clouds, I'm interacting with opules, but then suddenly I need to annotate for instance a large corpus. So it can use IQ for that, but it's really a quiet, it's not really being well done right now. And For instance, I want to authorize something. I want to let's say interact with the documents, I want to maybe access Wikidata, for instance, to do some search. All these kinds of things, they kind of call for actually having this kind of specialized model and large model could be the ideal orchestrator for this. Because I think the reason actually good specialized model, because they do exist, never really took off, is really also the mental cost. You have to know they exist, you have to use them, you have to deploy them. But if we are in a situation where it's kind of smooth, because okay, there is this kind of large model doing this orchestration already, and maybe I think the issue is that large labor necessarily interested in this approach right now. But I think it's definitely a kind of commercial model that could work out, where you could have all kinds of specialized providers, especially if we are in a future, for instance, where our training recipes are much more distributed across a lot of many industries. Yeah, they could make available all kind of sub agents and you could have shared retributions or whatever you can imagine for influence cost, and you can have particularly this kind of healthy ecosystem. so that's kind of the two directions. You can have particularly very large organizations for their internet purposes, they need to have their own systems. And also for another reason actually where I had not really anticipated, but I see a lot right now is really companies getting concerned about their IPs. About their own capabilities, about what actually they have to defend in the long run and the C basically I has a general transform general transformative technology basically at this point. And okay, if you get everything automated even from an external system, actually it's not that great at some point. You want some defensibility. And also you want to ensure also you in some cases, especially in regular sector, maybe you want you won't want an external actors to access what you have. So you want to preserve it. But even more deeply it's really about models as yeah something which is part of your technological assets. Allen Roush: And are you seeing a big movement and desire for more stuff around cybersecurity? Nah with with all the hoopla around fable and mythos and related Pierre-Carl langlais: Totally. I think it was been the actual wake-up calls because we had many in Europe at this point. Because suddenly they realize okay, there is this kind of big strategic mess. Like suddenly it's not just about not accessing the latest chatbot, but basically we know that our infrastructure, banks or whatever, well, they are not going to get protected. For any kind of assets are going to get a I think it's so yeah, definitely there's a real concern over this right now. I would say it's a real unknown, but I don't have any kind of solution right now, and there's a lot of speculation around that whether it's something that comes for pure scaling capabilities from generalist model in this case, okay, it's going to be pretty hard to compete with frontier labs. Or it's something which is more the direction of specialization. I think the signals from Frontier Labs are kind of mixed in a way. Because I think they are getting pretty good at things they are knowing in the end. Like again, counterexample is telecoms. la evaluation has not been progressing for two years because that's really something they are interested in. Whereas okay, math, cybersecurity, computing, okay, they have internet capabilities, they know about these things, they know it pretty well. So they can push the model pretty far in this direction. So I would say it's kind of a real unknown, but I think it's also the general issue also of information asymmetry between what we know labs are capable of doing whereas what they actually know. yeah, so I think it's kind of an unknown at this point, but definitely we're going to see this push. the case at this for Europe is really about the fact we don't have a large amount of labs that are even have the capabilities even to train not just frontier model but even like capable state of the close to state of the art model is really something we're kind of lacking. So definitely it creates a lot of imbalance. Ravid Shwartz-Ziv: And I have another question. So like we we talked about like right like orchestration and like small models that can like I don't know make some like joint decision. But do do you think we actually like need the bigger model, like a general bigger model to orchestrate between them? Because like It looked at like a lot of people are trying to do a lot a lot of companies are trying to do this orchestration, right? There is like for example today or like yesterday I saw a Sakana AI, right? It's like making like some yeah, make a orchestration level that you like you call different models and like and the idea is that you try basically to predict like what is the the best model to do this task or like to do the next step. But Pierre-Carl langlais: Taking that, yeah. Yeah. Hmm. Ravid Shwartz-Ziv: Do you think we can come up like with like specific types of data that actually like make it easier, like more more like or like even better than general models that like just train on all the data or like the general proposed models? Pierre-Carl langlais: yeah, I think it's complicated because the key issue is really about bitter lesson in the end. And the key the advantage of large model that you don't have to have a classifier or router or whatever because the model is just knowing already what what you want to do and that's actually the reason because we see the click at would say around one year ago because before that Opus was not the most used model at all. people were using Sonet and Opus was seeing okay, it's kind of clunky, it's too big, it's not being deployable enough. And then suddenly there was I think both the fact okay, Opus Inference got better, but on top of it, there was certainly this kind of need for more complex use cases that where you want to get an understanding of things. So I think right now it's a bit different. The kind of vision that I have, it's more about okay, is there models that can delegate some things at some point, but I think maybe we need to still have this kind of model orchestrator at some point, which is going to say, okay, this is going to be the intermediary I'm going to be using. And then On the contrary, because I think secana approach is more like okay, they use a very small classifier which is going to route you to potentially powerful model, which I'm not sure is the right approach. I think it's better to start with very large model, which has the good understanding of what you want to do. And then it can push for especially if you want, for instance, to analyze a large amount of corpus, large amount of text, today models are very reluctant to do that actually. They're really reluctant to look at the data, which is my main complaint with them right now. but yeah we can get even better capabilities in the end through this kind of focus rated process. So I think it's I think it's really the selling point at some points, not just to get something well there is the issue of cost obviously, but still if you can get systems that are better than the one that you have right now, I think it's kind of good. there's a question which is actually more interesting and more frontiery in a way, which is how good can you get general adjunct medically, general cap general capability model. like e even like this concept of well capacity this concept of cognitive core, which is one of the inspiration actually we got for when we created synth, even though we revised it a bit because the core of the concept of cognitive core was really a model with almost no memory, but just the ability to know basically where to find things. And what we discovered actually through our approach CTP training is yeah, the model needs to have some memories. But generally intuition that we have right now is that again, the information that we have about the person isn't that big again. what we have seen through all our life is not that big. and quote the set are kind of very lossy. So I think there still could be this general bets, especially if we get maybe also a better knowledge infrastructure around it. So maybe much better API, much better way and smoother way to access information. Maybe we could get actually a good enough like actually general intelligence. Because I think sometimes we kind of forget also what was general intelligence necessarily going to be about someone knowing all about everything, but having kind of the cleverness and ability to find information when needed. Maybe this kind of general diligence doesn't need to be that that big. I think I see need to have some size, definitely. but definitely I think we could get and I think we see this doubts. I think it could be something emerging like we have seen like this kind of 30 B model getting pretty good right now for many tasks. And I think even more with specialization we could make very good pair almost like personal agents, you know, like okay, the kind of because for this definitely it's a use cases where you don't necessarily want to send all information to a bit frontier model you and you want actually the model to be pretty good at accessing for instance all the information you have on computer, all the information about yourself, your life, whatever in a kind of very smooth way. I think that's kind of the use cases where not necessarily big but generalist model could be pretty useful and definitely the kind of thing we don't see yet, but that's kind of shapes of model to come. Ravid Shwartz-Ziv: Thank you. Allen Roush: And w what what are what are your opinions on the the politics of models right now in terms of like the the countries kind of leading the pack on AI development? So for example, France has at one point had Mistrol, which was, you know, held in really high regard. but there has been some, you know, people claiming Mistral has kind of fallen off and and and now there's this push around sovereign AI across basically every country that's not the United States. what what do you think about that that like po the the geopolitics and how it intersects with these models and the development of them? Pierre-Carl langlais: Yeah, so I must say I'm very, very ambiguous right now with the notion of sovereign sovereign eye. Because the key issue is that okay, first it creates fragmentations and then it's not necessarily very conducive actually to even emancipation. Because my own vision right now of I think even for Mr. it has been kind of an issue at some point because I've been kind of pushed direction well, okay, they are the national champion, but then they have to be super focused in becoming something a bit like actually a big French consulting firm to me to mean extent. That is has spent much less time right now reading frontier model reports and much more time okay going through all the big French administration and kind of optimizing their workflows. which is not that conducive to a kind of research spirit, you know, like you're not getting and I think as part of the discourse around Mesra, I Metra is optimizing for what he wants to look for, which is definitely going to get good at corporate business in France, but they're no longer optimizing for being your frontier labs right now at all. And they're kind of open about that. They're not even hiding it. So I think that's kind of the issue. But I would say right now my vision more and more is not even about French eye, not even about European eye, because I'm kind of very skeptical in the direction right now taken by even the European Commission in terms of the expertise at the Avengernality is very poor and they're not making good policy discussion. I think what we need is more like almost like international coal coordinations because we see these concerns right now, between okay there is US blocks, there is Chinese blocks which seems kind of more open, but we can't know for sure, and there have been kind of mixed signals even from big Chinese labs. They start, for instance, to remove free license for the more powerful models in some cases. So whether we can rely on that is not that obvious. And I think it's a shared concern not just with Europe. we see it also Korea as a pretty good program right now for developing their own training capacities. But Japan has Sakana, we have mentioned this before. There is India even with Sarvam, which definitely has been training models which are not that bad. Also on this kind of basically innovation, I think it's really call for having this kind of international community where we share knowledge about AI and it share in a way where we can practice I, which I think is right now the most lacking thing in Europe really. Because We we we have actually this kind there's always the idea that in Europe we have the talents, but I think it's no longer true at this point. Because all the talents that we have were kind of trained, you know, in the kind of pre LLMs AI, so they're pretty good at it. But what we have right now is a current practice of LLMs, like even how you do for instance adjunctive valves, how you can do deploy very large picture of experts, how you can train that, doing L all these kinds of things. There are very few people in Europe doing that right now. And I think it's kind of becoming a concern that we are kind of lacking the practice. What has been working the best, I would say, right now in Europe, is really the concept of public clusters, which is actually even working better than the US, because I see a lot of people in the US in universities are kind of frustrated because they have very little access to compute and the local clusters are not great. Yeah, at least in Europe, what worked great because it was kind of an investment being done almost 10 years ago, not even for LLMs, but really for i in all kinds of shapes. even Frontier project, we do have this kind of much smaller obviously than frontier level cluster, but still sizable clusters in France, in many countries for these kind of Euro specific things, which is right now almost the only place where we do practice, where we do the distributed computer multi-date scale, this kind of thing. But that's why it's complicated. But I think really it's kind of the most pressing concern right now. So the reason actually we joined this project from ILILAN's CoTapestry, which was launched by Jan Lecker not long ago. And we are one of the stakeholders in this, which is basically a place also to get this kind of exchange of information, experience and expertise across different countries, in our Ravid Shwartz-Ziv: I won we we don't have a lot of time, but I just like have like more questions about scanning loss and like what you think. So like till now, right, like people have built like scanning loss and like how the model performance with with like some time like some amount of data and like when you scale the compute and the interaction between them. and but but do you think when we actually when we control the data? Pierre-Carl langlais: Yeah. Yeah. Yeah. Ravid Shwartz-Ziv: we we can have more like I don't know smarter ways to to build these scanning laws for example you can ha you can imagine that like you need different types of data or like data properties when you when you have small amount of data versus when you have like large amount of data, right? So like it the kind of like the the sampling is not like ID if you want when you when you scale it. Do you agree or like what's your thoughts about it? Pierre-Carl langlais: Yeah. No, no, I completely agree. I think one of the most interesting things we have been discovering is always the scaling laws are completely true, obviously. You want to scale the models and scaling across different dimensions, obviously, it's not just about number of parameters, but it's about diversity. It's about the amount of knowledge you want to embed. We you want basically these kind of scalings, but you want to do it intelligently. I think the most important revision is that the lot of scaling laws were kind of conceived for one specific data source. And which is Vaticali Web Crawls originally, which is much more specific than a lot of people believed at the time, because it's collected in a certain way, certain text, which are mostly very shorts, because again for everything was kind of dependent on common crawl and their infrastructure issue at some point. They were not they were not having money to collect PDFs, so obviously we don't have PDFs. so even in time of coverage where there was very little multilingual sources for various reasons, so Everything was kind of made from this perspective. And what we're seeing right now when we can train model and static data at scale is that scaling laws are definitely there, but they are behaving differently. We have not this issue of stability. A lot of batch issue and spikes and stuff were mostly due to correlation issue in terms of how you were going to get the data. And we can we can very very closely inspect that. And yeah. What's interesting is basically I think how we can even compile, you know, for reasoning at some point. Like how we can get the models to get certain capabilities to unlock some capacities to solve math problems or whatever. You can observe that in a very minute way. And here the inspiration is really been physics of language model with the idea of controlled environments. Like okay, you know exactly the data shape you want to put inside of the models, you want to see basically how it's going to adapt to it, what kind of thing it's going to memorize, what kind of exercise, like for instance, multi-up reasoning, this type of thing. You can really abstract it and you can see how the model can gain that under different conditions. And I think it's very powerful as a kind of approach. I like it a lot because I find it's much more scientific than what we did before, which was like treating data like an opaque box and then okay. And now yeah, I think we Really at the start right now, at least in the open. I think probably this work was done in the closed labs already, maybe it's a possibility. But at least in the open, I think we're really at the moment where we start to be doing proper scientific work on this getting glows intently plural. And there are many things we still don't know in terms of for instance what is the right amount of repetitions you need to have for rephrasing, for instance, something we don't know yet. what is how much time for instance you need to repeat a mat exercise until the models knows it at a given parameters, budget, we don't know yet. This type of thing right now and I think it's really exciting to be part of this because there's much more experiments we can do now in a very controlled way, very scientific, very reproducible way. also since we can open the data and with the data, we don't have the issue of okay, it's kind of a quad, we know where it's coming from, maybe it's a liabilities. so I think it's kind of exciting. Ravid Shwartz-Ziv: Yeah, I I agree. okay, we're almost out of time. Like maybe let's talk a bit about the newtron personas datasets and what do you want to maybe to explain a bit about it and like what are the the what is the goal and what your role in this process? Pierre-Carl langlais: So it's a project which started by NVIDIA, but even before that by Gretel, which is a aesthetic data company which was bought by Nvidia about I think one or two years ago. And the core idea was really similar to us. So we're working a lot of people, for instance, in insurance processing in banking or whatever. And the key issue is that again if you want to train models, you want you want to even want to exchange data, even for doing any kind of thing you can imagine. you have the issue that okay you're going to train on forms, for instance, shared by a person, but then they put a lot of information about themselves, you cannot really share. And it's it's a general concern in the US, so it's not just about GDPR or whatever, and it's a general concern about this domain. So the current concept of personas was originally born from this kind of concern, but I think it grew beyond that because it's used a lot right now for static pipe plans in general. because as I mentioned before, it's a way also to create more diversity, to create also even situation, especially in agentix systems, because you want a user, but you want to flesh out the user. It's not just a random person, but maybe it's someone in a specific place, specific case, with specific ideas and so on. So the personnel originalists are really maybe the idea of okay, we can simulate demographics, and demographics of the US is very different from the demographics of France. And one of the key parts, for instance, is even It's kind of information you can access. So we're using open data usually and at least in France we are kind of available here to get a lot of information. But even then it's tricky because for instance, at least in France, we don't have any kind of good recording data for people who were becoming French, basically, like immigrants but first generation immigrants because they're not being born there. So they're not being trusted at all. So we had to correct for this to be able to kind of reflect better the population. We use statistics and we corrections, this type of thing to be able to correct for this. And so Nvidia this program started in the US. then they expanded it. I think first in Brazil, in Saint-Camp, I think we the fourth country in France. then it continued, and now we are the tenth country for Belgium, which was an interesting use case because it's a multilingual country. So we had to generate data in French, Dutch, and German. And we we had to ground all of these things. I think really the common thread is this thing, I think it's really about how it kind of connects what I was saying before about international collaboration, which is how we can create these shareable assets for data pipelines in eye, because at some point if you get good personal data is a kind of general purpose asset in a way. Because it's the kind of thing you can use again for alignment, you can use it again for just fleshing out any kind of static data. So you kind of need that. So so yeah, that's I think I'm also interesting that's also the kind of thing we don't necessarily going to require actually large scale compute because we did that actually on Janzé on on national on national sovereign compute because also part about simulation of metadata is done just with the PGM, so it's a very very light model I can write on my computer to do millions of rows in a in a in a blink. And then yeah, we kind of create this kind of fictive bib biographies people can use or cannot use medically about okay the person where it is, what it's becoming from and so on. there's a lot of interesting research to be done still because one of the issues we find that even with the control of okay, the person was coming from model it can still collapse a bit in terms of for instance the diversity of tastes. Like okay. at least in France, like okay, all people are reading the same books. All people are kind of having the same taste in kitchen and so on. So at some point you have to kind of extend probably even the range of features you want to put inside. But yeah, I think it's like what I like is really about data as a kind of never ending project, basically at this point. You you can still make it better. Ravid Shwartz-Ziv: But the the idea is like to what? Like now given this data set, I can w what are the like the the the goals, like what I can do with that? I can like sample persons and then I don't do some like survey about something or as like instead of like asking real people like to to learn some demographic kind of shifts or like what what do you actually do with these types of data? Pierre-Carl langlais: I think there are mostly two use cases. So the first one is really about again simulations of sensitive data. Because it's a ton of personal data. And you want it to be realistic. Because you want, for instance, if you're going to train, we have a project right now, insurance filings in France. So it's a lot of people be writing complaints about okay, okay, for instance there was an accident with their cars, or whatever. They put a lot of information, they put their address, they put their names, put all these things. Now if you want to train a model on this and you want it to be able to afterwards being able to process real information, you have to ensure there is this kind of alignment where the name of the person, its age, situations, choices they could have made because they all impact afterwards, even the case what is going for instance to be the monetary situation, all these kinds of things is going to be impacted. You want it to be in a kind of it's fictive once again, it's not real person, but they have to be realistic. They have to be kind of grounded into this specific region, for instance, where for instance and a person right riding a car for instance is not going to be one of the outsole, for instance. You want to avoid this kind of torture situation. So you want to control for the distribution. I think that's the first aspect or using it a lot because the original motivation. And the second one is really general purpose pipelines. Because once again, if you're going to create environments beyond just coding, but like say environments like for instance situation for healthcare assistant we are doing right now for instance. And basically you want to get people which have had an issue with their own health, for instance, the back pains or whatever, they want to ask questions. But again, the ability of having this kind of symptoms is conditioned on demographics. It's condition of okay, what is going to be your head? And then after all the application is going to ask for information, stuff like this. So you want to ground all of these things. So I think it's really about this kind of jump where We're not just going to train on text, but on the preconditions that made text happen in some ways. which I think is really really interesting. Like almost the recipe of how a text came to be. We are almost to unlearn it. Like especially if you're dealing mostly with forms, it's relatively easy to see. But I think even for a laboratory text actually which talk about like say a Keslow for instance. Okay, some person arguing another person, but again they are part of society, they are part of a specific region and so on. So you can ground all of this and then you can create endless variations around. That. But I think personas are not the ultimate thing because what you want really is to ground that with all kind information which can be again simulated or non-simulated, you can have all kinds of things about the places, the kind of thing like that. So I think it's the reason actually that's been taking up so much recently because it I think it's really the part where we're going to move on from the syntic pipelines which are really closely related to text, like what they stick rephrasing. Okay, we're just thinking existing text to Word. This kind of more creative approach. We are going to create data not out of thin air, but out of realistic combination of reality. Ravid Shwartz-Ziv: Okay, I think we are out of time. do you have anything else that you want to add? Pierre-Carl langlais: No, I think it was it was a great conversation and I think what I really like is that we can have it right now because I see that working on data as we church is not obvious until very recently. It's not something you could do openly, but I still think right now between all the advancements we have in open data, we have in static data now it's something we can definitely discuss and we see how much it can impact the capabilities of the models. Ravid Shwartz-Ziv: I agree. Pierre Carl, thank you so much for joining us. it was a b pleasure to have you. Pierre-Carl langlais: Excellent. Allen Roush: Yeah. Yeah, thank you. It was really a pleasure to to see you again. Pierre-Carl langlais: Thanks for that. Ravid Shwartz-Ziv: Okay, thank you everyone.