Latest / Agentic Conversations (formally mlops.community) / Distilling 200+ Hours of NeurIPS: What’s Next for AI // Nikolaos Vasiloglou // #336

Transcript

Distilling 200+ Hours of NeurIPS: What’s Next for AI // Nikolaos Vasiloglou // #336 · Agentic Conversations (formally mlops.community)

• 0:00
  There's been only one area that for many years people are trying
• 0:03
  to use deep learning to beat traditional like solo models and
• 0:07
  they haven't managed to do that. And that's your typical XT boost
• 0:11
  type of models like tabular data, you know, predicting on on
• 0:16
  on data frames. My name is Nicolas Vasilavlue.
• 0:26
  People usually call me Nick the Greek.
• 0:28
  I'm the VP of research ML, but relationally I and I got an
• 0:34
  award for finishing my PSD without drinking a single cup of
• 0:37
  coffee. But as I get older, I need to
• 0:41
  get some caffeine. I'm also a bath and runner, but
• 0:45
  I'm allergic to to to coffee, you know, so this is how I take
• 0:50
  my coffee. I take coffee pills I've they
• 0:54
  make my day. I wake up in the morning, I open
• 0:56
  the bottle, I take a pill, drink some water and then everything
• 1:00
  looks great. Take them in the morning, not at
• 1:02
  night to wake me up. You went to Nurups.
• 1:07
  You distilled the plethora of information in talks and papers
• 1:12
  that came out at Nurups into almost like a CHEAT SHEET.
• 1:17
  Can you explain what you did and what made you?
• 1:21
  What drives you to do such a thing?
• 1:23
  Yeah, I don't think it's some exaggeration to say that I tried
• 1:27
  to become a human language model, which is a little bit of,
• 1:30
  you know, strange because language models came were
• 1:34
  invented by humans. This is how we speak.
• 1:37
  And I think now we're trying to imitate sense.
• 1:40
  So I I do have a passion for research, although I work for
• 1:44
  the for the industry. Since I finished my PhD, I've
• 1:48
  been very close to research. I tried to publish.
• 1:50
  I have interns, but I think I remember what somebody told me
• 1:54
  many years ago. He said it was a researcher in,
• 1:58
  in biology. And he said, well, we should
• 2:00
  stop doing research and just read our papers over the last 10
• 2:03
  years to see what we've done. And there's this particular
• 2:09
  thing that, you know, people always want to publish, they
• 2:11
  always want to talk. But, you know, we produce a lot
• 2:16
  of material, especially in research these days.
• 2:18
  It's it's, you know, amazing, but I don't think we ever take
• 2:22
  the time to read what we produced.
• 2:25
  And I've noticed through the years that there's always papers
• 2:30
  that they just don't get on the spotlight for some reason,
• 2:32
  whatever reason. And then it takes us 10/15/20
• 2:36
  years. Oh yes, actually somebody has,
• 2:38
  you know, mentioned that, but we never paid attention to it.
• 2:43
  So I think there's a gap in the scientific world, in the
• 2:47
  industry world, which is, OK, we say what we say, we'll produce
• 2:52
  all this research. Can we somehow try to invest
• 2:59
  that material and communicate it to a, you know, to the leaders,
• 3:04
  to the decision makers? These decision makers can be,
• 3:07
  you know, research managers, they can be CE OS, they can be
• 3:12
  VCs so that we actually know what we're doing.
• 3:17
  So yeah, that's a pasta that I've been doing for for many
• 3:19
  years. This was the most difficult year
• 3:21
  because, you know, we're talking about a conference with close to
• 3:26
  8 1/2 thousand publications, hundreds of hours of video, just
• 3:32
  like reading, going through, you know, skimming through the
• 3:35
  material. This is like, you know, 10
• 3:37
  hours, just like going In Sync everything one by one.
• 3:42
  I think probably the last year that I'm I'm I'm able to do that
• 3:45
  for new Rich. I think next year I will have to
• 3:47
  use the help of language models. Maybe we'll have to train the
• 3:49
  language model with with a. So you didn't use the help of AI
• 3:53
  this year? No, no, I didn't.
• 3:57
  Actually. I did in the end, let me tell
• 3:59
  you this, which was a little bit, you know, so they say,
• 4:05
  disheartening. So after I finished my
• 4:07
  presentation, I created, you know, 600 slides, which I broke
• 4:12
  it into 12 different lectures. If you want, I recorded the
• 4:16
  videos. I had this idea people are using
• 4:18
  Notebook LM to, to generate podcasts.
• 4:22
  And to my surprise, you know, notebook LM did such a great job
• 4:28
  explaining the material. I thought, I thought I did a
• 4:33
  good job creating the presentation, but just taking
• 4:35
  this the transcripts and and giving them to do to, to
• 4:38
  notebook LM was experimenting with something else called PDF 2
• 4:42
  audio. We can talk about that if, if
• 4:43
  people are interested in that. I said, wow, you know, you can
• 4:48
  communicate ideas better than than me.
• 4:51
  Now I just want to put in asterisk because AI is running
• 4:54
  so fast and I had the, the the honor to meet the, the product
• 4:59
  manager of of notebook LM as Google.
• 5:02
  So when we started notebook LM was a little bit hard to tame.
• 5:07
  It was sometimes was going off road, it was oversimplifying, so
• 5:11
  we had to kind of like learn how to tame it.
• 5:14
  But in the past six months it has improved a lot and I don't
• 5:17
  really think it needs that taming anymore.
• 5:21
  Fascinating. So what you did, if I understand
• 5:24
  this correctly, is you took all of the recordings and all of the
• 5:28
  papers that were submitted to NURUPS this year, this last
• 5:32
  year, right? Sort of I didn't go through
• 5:34
  here, going through 8 1/2 thousand papers would require
• 5:37
  four years of work for me. I just went through the
• 5:40
  presentations, the worksops selectively, some papers and
• 5:45
  I'll explain why I did that. So it's kind of interesting
• 5:48
  knowledge. It's already self organized
• 5:51
  through the so we can talk about that hard works.
• 5:56
  But essentially I try to like, if you see my process, it really
• 6:00
  looks like the process of doing Zen AI this day.
• 6:05
  So I think there was another benefit for my professional job
• 6:09
  about how, how do you basically use Zen AI in the enterprise?
• 6:14
  Like if you think that I got, you know, this huge material and
• 6:19
  I tried to identify places where this is well summarized or you
• 6:24
  know, well ranked if you want and working alternatively by
• 6:29
  myself and getting to this 600 slides.
• 6:34
  I think this is something that now you can start doing with
• 6:38
  language model where you have to prepare material.
• 6:41
  Yeah, why did you choose certain papers over others?
• 6:50
  So here's the thing, let me explain you how it works.
• 6:52
  So there's three major conferences every year in AI and
• 6:56
  they happen with this order. And this is the order of
• 6:59
  importance ICLR, which actually happened a few weeks ago in in
• 7:03
  May. Then in the summer, we have ICML
• 7:06
  which is bigger and then we have new ribs in December, which is
• 7:09
  much bigger. And I think it's the most
• 7:10
  complete and probably the oldest.
• 7:13
  What happens is when you go to new ribs, there's a lot of
• 7:17
  keynotes. You have the orals, the oral
• 7:20
  presentation. No, not all these 8 1/2 thousand
• 7:23
  papers. It's about 4 1/2.
• 7:25
  I don't remember the numbers I have in presentation that you
• 7:28
  have them, the main track and another 4000 in, in workshops.
• 7:32
  Not all of them make it to oral presentations.
• 7:36
  OK, so you first, you know, go through the keynotes, you go
• 7:39
  through the orals and then you go through the workshops.
• 7:43
  You know, the keynotes and the workshops and, and presentations
• 7:46
  that they look interesting. I think that's part of me.
• 7:49
  You know, I've been many years in the area.
• 7:50
  So I think I have some, I'm a good ranker, you know, say, so
• 7:55
  I, I, I basically use what the new rips reviewers have done by
• 7:59
  elevating a paper to an oral, to a keynote, the works of
• 8:02
  organizers, by selecting which one's going to be, you know,
• 8:05
  it's more important. Now, all these presentations,
• 8:08
  all these oral presentation, they usually cite and reference
• 8:13
  papers that they've already published, let's say within the
• 8:16
  last or two years. And sometimes when they go to
• 8:19
  the conference, because things are running so fast, they're
• 8:21
  also presenting about things that they have submitted for
• 8:24
  publication in the next year. So by attending one conference,
• 8:28
  you kind of like get an idea about what is happening in the
• 8:32
  past six months and what's going to happen in the next 6 months,
• 8:36
  at least in, in as a publication.
• 8:40
  And now then I start, you know, as you watch the presentation,
• 8:42
  you start digging through the references and then you find
• 8:44
  out, you know, this is a reference that maybe wasn't very
• 8:47
  well received or, you know, or you got published like a year
• 8:52
  ago. And we needed a year for people
• 8:54
  to kind of like start talking about it.
• 8:57
  And that's how it goes. You take snapshots, you take, I
• 9:00
  took up about 2000 screenshots, you know, from papers here and
• 9:03
  there. It's, it's, I have a way of
• 9:04
  organizing the material anyway. And then you, you keep refining
• 9:08
  and then you know, you keep cutting, you know, you keep
• 9:10
  cutting, consolidating, cutting until you bring it.
• 9:13
  This year, typically I, I can make it fitted in an hour and a
• 9:17
  half. The previous years, this year I
• 9:19
  had to to basically have 12 presentation close to I think 6
• 9:25
  hours, 6 1/2 hours of of of me presenting.
• 9:30
  Well, it's 200 slides, you said. No, no, it's 650.
• 9:35
  They're public. I have, I actually have
• 9:38
  everything available. The initial screenshots was I
• 9:42
  think it's close to 2000 and and then after they still got to I
• 9:46
  think 600 slides. And then you had the six hour
• 9:52
  presentation that you yourself are talking about your
• 9:55
  understanding of all of this and why it's important.
• 9:58
  And then after that, the podcast, which makes it even,
• 10:01
  you know, even smaller. The Notebook LLM broadcast.
• 10:06
  What are some things that stood out to you?
• 10:08
  Yes, you know, surprisingly I don't know, but the, the, I
• 10:12
  managed to break it into 12 topics.
• 10:14
  That's why I named the, the series like the 12 days of new
• 10:17
  ribs. I, it kind of coincided with
• 10:19
  Christmas, but I didn't do it on purpose.
• 10:20
  It just happened to be 12 things but and the presentation was
• 10:24
  broken. You know like there's like 2
• 10:27
  slides that there's a nutshell. Then I have a slide with, you
• 10:30
  know, 12 topics and then there is, you know, 3 slides for every
• 10:34
  topic and then there's like a whole section for every topic.
• 10:37
  So you can go hierarchically as deep as you as you want.
• 10:42
  I think the, the very first thing that kind of like stood
• 10:45
  out is that, you know, we see that with every technology.
• 10:49
  It, it pops, there's buzz. It's kind of like the hype cycle
• 10:53
  either makes it or or or doesn't make it.
• 10:55
  So now it's the year that we can say that language models are are
• 11:00
  everywhere, but not for everyone yet.
• 11:03
  OK. And it seems to me that I don't
• 11:08
  remember Europe's skiing around the deep sick kind of boom or,
• 11:16
  you know, buzz. But it is true that the, the,
• 11:21
  the open source language models are, are winning.
• 11:25
  There were a lot of, you know, there were a lot of how can I
• 11:33
  say this papers about language models that are very, you know,
• 11:46
  they are very open. They give you everything.
• 11:48
  They give you all the the checkpoints you can reproduce
• 11:54
  them, you can pre train them from those checkpoints.
• 11:59
  You can, if you want, you can take their data sets.
• 12:05
  So they open sourced a lot of data sets that you can you can
• 12:11
  modify if you want to, you can include them in in your own in
• 12:16
  your own deployment. OK, so and there there was a
• 12:21
  presentation, I think if there was 1 presentation to towards
• 12:24
  was from the inventor of the LSDMS. So for the for the
• 12:33
  audience that doesn't know what an LSDM is, it was the precursor
• 12:36
  of the transformer that made the the language model what it is.
• 12:40
  And he basically had a slide where it says, the way that I
• 12:45
  translate it is everybody loves to hate language models, OK.
• 12:49
  And, you know, at least like a bunch of complaints from people,
• 12:54
  you know, it's not going to give us AGI.
• 12:56
  He can't do this. He can't do that.
• 12:58
  And when you hear people complaining about something, it
• 13:00
  means that they're using it a lot.
• 13:02
  And these complaints are actually constructed because
• 13:04
  these are the complaints that they're going to lead us to the
• 13:07
  next to the next, you know, big leap.
• 13:14
  I think, you know, it's kind of fascinating that this was the
• 13:21
  10th year anniversary from the first sequence to sequence
• 13:25
  model. So the sequence to sequence
• 13:27
  model is a basically what a language model is.
• 13:30
  And, you know, you give it a sequence and it generates
• 13:33
  another sequence. You give it a prompt, it
• 13:35
  generates. And that won the Test of Time
• 13:38
  award. OK, so but the author was Ilya
• 13:42
  Suskiverse, the famous you know, open the eye guy with the whole
• 13:45
  drama. He's left and so he was there
• 13:47
  and he gave a very nice presentation about that.
• 13:51
  It was also the 10 year anniversary of another
• 13:55
  monumental paper, instrumental paper.
• 13:57
  It was the gas. It was the first time we
• 13:59
  synthetically generated images and it's been, you know, it's
• 14:03
  been 10 years. You know, it's those years.
• 14:06
  I remember when these papers came out, everybody was very
• 14:08
  critical because they said, oh, you know, your references were
• 14:12
  only one year old. And you know, that's not a nice
• 14:15
  thing. When we write a paper, you want
• 14:17
  to always refer papers that they go back in time.
• 14:20
  And but that was the beginning of deep learning.
• 14:23
  There was nothing before that. Now we now we can go 10 years
• 14:28
  back and, and reflect and talking a little bit about
• 14:33
  DeepSeek. You know, I just wanted to, to
• 14:36
  mention that this is kind of like this is the second time
• 14:38
  that it's happening. You know, I remember, I think it
• 14:42
  was 2023, he was in the middle of Europe's where open AI
• 14:45
  announced SAT GPT and all of a sudden everybody was talking
• 14:50
  about that. You know, I think they do it by
• 14:53
  choice. If you ask my opinion, like
• 14:55
  these announcements are coming around the conference to kind of
• 14:58
  like leverage the congregates. We're talking about 16,000
• 15:02
  people. OK, so this is the place where
• 15:04
  you want to go and throw the the bomb.
• 15:08
  And obviously there was a lot of discussion and, and something
• 15:10
  that I noticed is that both years like back in 2023 and now
• 15:14
  in 20 actually, no, it was 2022. Yeah, 2022.
• 15:20
  And this was news 2024. I was two years ago, the there
• 15:26
  was no one from open AI or very few, you know, people from open
• 15:30
  AI then and now there wasn't anyone from deep sick.
• 15:34
  I think I found only one paper that a small paper that was
• 15:37
  someone from deep sick. And that's a sign in my opinion
• 15:41
  that these people are waking, you know, day and night.
• 15:44
  That's why you don't see them there, I think.
• 15:46
  Or maybe they don't want to be there because they there's so
• 15:49
  much that they want to withhold they they don't want to be.
• 15:52
  They'll get trampled, yes. But you know, this year there
• 15:58
  were a lot of great talks by Open AI.
• 16:00
  It was my my favorite 1. So a specific one about how
• 16:05
  trying to improve ourselves in gaming basically brought us to
• 16:09
  the reasoning models and, you know, very, very nice
• 16:15
  presentation. So which means that now Open AI
• 16:17
  starts opening more and more and, and giving out to the word
• 16:24
  and explaining what they're doing and how they're doing.
• 16:30
  So, you know, that was obviously there was a lot of stuff around
• 16:36
  agents, you know, if you ask this, they basically dominated
• 16:43
  the conference. And, you know, I could go
• 16:46
  through the whole 12 of those, but I'll just, you know, you let
• 16:49
  me know. Maybe our on the agents thread,
• 16:52
  was there anything that jumped out at you with the agents that
• 16:57
  still to this day? Because it's been 6 months since
• 17:01
  Nareps and it feels like you were saying that things move
• 17:06
  very fast and sometimes it's a flash in the pan, sometimes it
• 17:08
  is a signpost that needs further research.
• 17:14
  Did you see anything there with respect to agents that now
• 17:19
  you're seeing more and more a common pattern or it is
• 17:23
  something that is more established?
• 17:27
  There were two things. First of all, there was a lot of
• 17:30
  buzz about, you know, the smaller models and if you
• 17:33
  remember like NVIDIA stock and open AI, there's not a stock.
• 17:37
  But anyway, the kind of like the Dipton, there was this thing are
• 17:40
  the smaller ones challenging them really.
• 17:43
  And I remember there was a slide about there's a lot of
• 17:48
  evaluation frameworks about, you know, it was a benchmark on
• 17:52
  agents between different language models.
• 17:55
  OK. And that benchmark had the
• 17:57
  Frontier model as we call it, like clogs, open AI, go here
• 18:00
  Gemini. These are the frontier models as
• 18:03
  being better than others. And of course, they hadn't
• 18:07
  included in their benchmarks the newer models, you know, the Quan
• 18:12
  and DeepSeek. And there was this impression
• 18:16
  that, you know, are these outdated or what's going on?
• 18:21
  I think six months after new ribs and all these bars of, of
• 18:25
  smaller models, I think they're useful, but the bars they make
• 18:30
  they made was a little bit bloated.
• 18:32
  I think the frontier models are still ahead of the game and that
• 18:37
  they're worth their their money. Of course, you know, people are
• 18:43
  now experimenting with these reinforcement learning, you
• 18:47
  know, technique for fine tuning and, and we're learning a lot.
• 18:53
  So that track kept going on and I think it's producing results.
• 18:59
  We personally use them in the, in the, in the project at
• 19:03
  traditionally I it has to do with, I guess we can talk about
• 19:08
  that since we're going to talk about it in a week at the
• 19:11
  Snowflake Summit about text to sequel competitions and, and now
• 19:14
  we've modified and use the, I want to say simpler but more
• 19:18
  effective method of using that. So I can see that from a
• 19:20
  personal experience, but I follow, you know, the what the
• 19:23
  others are doing. So this idea of reinforcement
• 19:27
  learning chain of thought is, is producing results.
• 19:32
  The other thing is something else, which I personally find it
• 19:39
  fascinating. There were papers that they were
• 19:41
  talking about building a, a, a, a lab from agents, you know, a
• 19:46
  biology lab or a materials lab or an antibiotics lab where
• 19:52
  there's agents trying to do research and discover a new, you
• 19:55
  know, antibody, a new, a new material and.
• 20:01
  That was, you know, really impressive.
• 20:03
  These are the cases, you know, where the hallucination, the
• 20:06
  imagination of language model is a feature.
• 20:08
  It's not a pack because you want them to to think out-of-the-box
• 20:11
  and you don't care if if not out of 10 are wrong or they're
• 20:15
  hallucinations. You know, 1 of this
• 20:16
  hallucination could be useful. And of course there was a small
• 20:21
  set back because there was a paper from MIT that supposedly
• 20:25
  discovered that came a bit later, a new material out of
• 20:30
  the, you know, out of nothing. And it created even bigger hopes
• 20:36
  and it got retracted. I, I read recently that it was
• 20:39
  also cited by Nobelist in economics.
• 20:44
  Now they took it back. But if we see ICLR that
• 20:48
  happened, you know, 5-6 months after in Europe and we see the
• 20:52
  workshops, we see a lot of activity around that, like
• 20:55
  building an army of agents that they're actually going to
• 21:00
  produce new, new stuff, invent things.
• 21:05
  I think there's definitely a lot of interest by by companies, but
• 21:12
  there's also a lot of pitfalls. We can talk about that if you
• 21:14
  want more, yeah. Let's go down that path.
• 21:18
  Yes. So I think while it is useful
• 21:21
  when you're working on a research lab or where you're
• 21:23
  trying to, you know, it doesn't have to be a research library.
• 21:27
  Say you are the finance department and you want to run
• 21:29
  like a simulation. You can ask what if we do this
• 21:31
  or that, what's going to happen? And you can have, you know,
• 21:34
  agents talking and creating scenarios and ideas that you you
• 21:37
  never thought about it. This is great.
• 21:40
  But remember this is what I called agents as decision
• 21:44
  support. OK, no decision making and this
• 21:49
  is your execution, OK, So and I think that's a great path and I
• 21:54
  think that's the way that you should be using agents.
• 21:58
  But I think slowly companies are trying to build authentic models
• 22:03
  that they're doing decision making, decision execution
• 22:07
  actually for you. And you might have seen like
• 22:09
  some companies that they're trying to organise your
• 22:11
  calendar. There's a nice research preview
• 22:16
  from from Opening eye again, it's called the operator where
• 22:20
  you know, say book a restaurant for two people in a nice
• 22:23
  restaurant with booking meal for yes with seafood wherever you
• 22:29
  want and it starts opening a browser and then comes back to
• 22:31
  the reservation for you, which is probably fine.
• 22:35
  If it messes up, it's not a big deal, but I don't know if you're
• 22:40
  going to say go buy me a car or, you know, I'm giving you the
• 22:42
  bank account making a decision for me.
• 22:44
  I don't think we can trust them. And I think in many cases,
• 22:49
  remember, these are kind of also probabilistic systems.
• 22:55
  People are trying to trust deterministic tasks to
• 22:59
  probabilistic systems that they're extremely difficult to
• 23:03
  control and extremely difficult to debug.
• 23:07
  So I feel like we're reliving the time where multi threading
• 23:10
  came out and people jump to adopt that.
• 23:14
  And then they very quickly realise that, you know, you can
• 23:17
  put yourself in a very difficult situation where the system is
• 23:20
  failing and you don't know why and you can't fix it.
• 23:23
  So I think there's a bit of a, you know, a rush over there.
• 23:29
  You know, we're asking things. There's some threads we need to
• 23:32
  pull on and we need to get things that are a bit more
• 23:36
  reliable before we can really make the significant advances
• 23:40
  we're looking for. There's also a few other things.
• 23:44
  I think the things that are more important in, in my opinion, if
• 23:47
  you want like ages, you know, obviously people were talking
• 23:53
  about them. We see them now.
• 23:55
  What, what I think it's interesting from this
• 23:57
  presentation is research that we haven't really seen, you know,
• 24:03
  in production yet, but it's coming very aggressively, like
• 24:06
  it's being baked, you know, like you can see a lot of momentum,
• 24:10
  but you probably don't see, you know, like, or the audience, you
• 24:14
  know, you don't see it in your everyday life because there's no
• 24:16
  companies producing the products with them.
• 24:20
  So, you know, there is, for example, then AI or deep
• 24:26
  learning has conquered every traditional area of machine
• 24:29
  learning. You know, they did it with
• 24:31
  images, they did with text, you know, anything you give them,
• 24:37
  they can, you know, there's this recipe, just give us lots of
• 24:40
  data to pre train and try to predict the next token, the next
• 24:44
  thing you know, the next something unsupervised in any
• 24:47
  way. And and we can, we can finish
• 24:49
  that. There's been only one area that
• 24:51
  for many years people are trying to use deep learning to beat
• 24:54
  traditional like solo models and they haven't managed to do that.
• 24:58
  And that's your typical XT boost type of models, like tabular
• 25:02
  data, you know, predicting on, on, on data frames and tables.
• 25:07
  And I think this is the first year that we have the series of
• 25:10
  models that they managed to beat the, you know, the undisputable
• 25:16
  queen of models XT boost that everybody's using for predictive
• 25:21
  analytics, of course at a higher cost.
• 25:23
  So you know, there's a Worksop that I've been attending for
• 25:28
  three years and it shows signs right now that I think next year
• 25:33
  we're going to see language models for tables for database.
• 25:37
  They're going to take database sets of tables as input, get pre
• 25:43
  trained and you will be able to do, you know, predictive models
• 25:47
  and ask questions directly on those data.
• 25:51
  So this is one of the things. On that point, I know that in
• 25:57
  the community, a lot of folks have asked time and time again
• 26:00
  about time series and forecasting models, and you have
• 26:04
  these forecasting foundational models, but the jury's still out
• 26:10
  on how valuable they are. And it feels like that's kind of
• 26:14
  in that same vein of, yeah, maybe the research is really
• 26:21
  pushing in that direction, but is it has it overtaken what the
• 26:26
  common industry uses are? Not yet, but it's coming fast.
• 26:32
  Yeah, it's coming fast, definitely for the first time we
• 26:34
  saw them. No, there's there's this
• 26:36
  argument which is like even if you are marginally or a little
• 26:39
  bit better, is it worth spending all these GPU's and these
• 26:42
  resources? But I think people are waking
• 26:45
  like them, they're making lighter models.
• 26:48
  And you know, there's another aspect that you have to see,
• 26:52
  which is in the enterprise world, people that think that
• 26:55
  you hate is having 20 different platforms, you know, one for
• 26:59
  your predictive one, your prescriptive, one for your, you
• 27:03
  know, graphs, one for your whatever BI people want one
• 27:07
  thing, one platform. This is, by the way, why the,
• 27:11
  the data cloud AI data cloud companies are marching right now
• 27:14
  like Snowflake, you know, data breaks big query.
• 27:18
  That's because they're kind of like consolidating this, you
• 27:21
  know, this set of tasks in one place.
• 27:26
  OK, so you know, so if you have like 1, you know, right now,
• 27:32
  let's say foundational model that can do everything for you.
• 27:34
  It can do your sentiment analysis, it can do your
• 27:37
  knowledge extraction. It can do the query answering,
• 27:39
  it can do the predictive analytics, it can do the
• 27:41
  optimization for you, you know, prescriptive analytics.
• 27:44
  This is what people call the generalized rag, something I'm
• 27:48
  working on. But you know, Amazon also
• 27:50
  presented the auto blue and this is part of their vision as well.
• 27:53
  So you'd find more in my presentation where they're
• 27:56
  trying to have one box where you ask a question and if that
• 28:00
  question means a predictive model, you know, cause
• 28:03
  predictive model. If it needs a, you know, typical
• 28:09
  rug retrieval, it uses that. So if it means to solve an
• 28:13
  optimization problem, it's also an optimization problem.
• 28:15
  I guess that's a little different than this model that's
• 28:18
  trained and it's like a foundational model for tabular
• 28:21
  data, right? But the idea that you're saying
• 28:23
  is that will hopefully that type of a model will be able to
• 28:30
  generalize across many different of these questions.
• 28:34
  So you don't need to think call a RAG call a a prescriptive
• 28:39
  model called XYZ. You're just going to call that
• 28:41
  model. So, so here's the thing, let me
• 28:44
  make a small digression. There isn't why we don't have
• 28:47
  language models that can deal well with, you know, with
• 28:51
  tabular data is because we don't have a lot of public tabular
• 28:56
  data. You know, enterprises don't open
• 28:58
  source their databases, OK, we don't have an Internet of now
• 29:03
  there was there's been a lot of effort to gather that the
• 29:06
  equivalent of AC4. So C4 is the big data set that
• 29:10
  people crawl the web and train all the the language models.
• 29:13
  I think there's been an effort and that's part of the the
• 29:17
  workshop over there that described all these efforts to
• 29:19
  create that. In some cases, you know, the
• 29:23
  language model can natively do that task.
• 29:26
  OK, that can natively if if you pre training with databases,
• 29:30
  there's language models where you say, OK, here are some
• 29:32
  examples of training data. Here's my test data based on
• 29:36
  that predict, you know how much I'm going to sell tomorrow and
• 29:40
  that's great. There are some other tasks like
• 29:42
  for example, if you want to do integer optimization, we don't
• 29:47
  have language models that they know how to do that.
• 29:49
  And right now we have some pretty good solvers that you
• 29:51
  know, Simplex, Kurabi and others that people have worked for for
• 29:55
  many years and they are quite accurate.
• 29:58
  So in some cases, you know, you want the the optimization to be,
• 30:03
  you have to be careful because if it's a predictive model, A
• 30:05
  language model, making a mistake is not a problem because any
• 30:09
  other solver you know will do the same.
• 30:11
  But you know, when you're solving optimization problem, in
• 30:15
  some cases, it could be you can have like soft rules like or
• 30:19
  soft constraints where it doesn't really matter if you're
• 30:21
  violating them. But in some others, you might
• 30:25
  have hard rules, you know, like you're trying to do your supply
• 30:28
  chain and or you're trying to schedule your your airplane
• 30:33
  fleet and you want to make sure that you're not going to have
• 30:36
  more than 100 people. Like it gives you 150 that
• 30:39
  aeroplane does not fit 150. You got other problems.
• 30:42
  And you have some pissed off customers.
• 30:44
  Yes, which of course takes us to another thing that was very
• 30:50
  popular during the conference, which is verification systems.
• 30:54
  OK, So this is becoming a central theme.
• 30:57
  You know, how are we going to make AI models more reliable?
• 31:07
  And Long story short, if the task you're trying to solve, you
• 31:13
  can attach a verifier, that's the perfect competition, that's
• 31:18
  the perfect combination. For example, you're trying to
• 31:21
  prove theorems. You got a language model, which
• 31:24
  is very creative, also a huge part of the conference like math
• 31:28
  for AI or AI for math, you know, you being creative to prove a
• 31:34
  theorem. But in the end you can check if
• 31:36
  you're right or if you're wrong. OK, but there's other tasks
• 31:40
  where you don't have a way to verify them.
• 31:43
  OK, so that's as, you know, what's the capital of France.
• 31:47
  You you don't have to, you know, if it does you wrong, there's no
• 31:50
  way you can you can verify that. So you're relying, you know,
• 31:55
  100% on the the language model. So the verifiers are going
• 31:59
  beyond like the guardrails and output checks type thing.
• 32:05
  Yeah. So the verifiers could be, you
• 32:08
  know, verifying, for example, a mathematical theorem.
• 32:11
  It's not, you know, there's a system called Lean Four that
• 32:14
  does it for you. It's it's a complicated system.
• 32:17
  We can do it. But in some of the cases.
• 32:21
  So here's the interesting thing. This, this is something we all
• 32:23
  have to get used to thinking One Direction and immediately try to
• 32:27
  think the other direction, OK, So whenever you have the
• 32:30
  language model doing something for something, then you have to
• 32:35
  reverse like what can this something do for the language
• 32:37
  model? OK, So for example, when we
• 32:43
  generate software, OK, let's go to the biggest elephant in the
• 32:46
  room. Everybody's working on using
• 32:47
  language models to generate software.
• 32:50
  OK, You need to verify, OK, you got your compiler that it
• 32:55
  compiles. That's like, OK, that's easy,
• 32:59
  but how do you know that the software does what you want to
• 33:05
  do? OK, that's like a so there are
• 33:09
  AI systems that they're trying to take your task.
• 33:12
  Like I'm saying, I want the software to do this reverse a
• 33:15
  string, OK? And they're trying to create
• 33:18
  their verification tasks. The tests, if you want the
• 33:21
  verification tests that their produce code has to satisfy,
• 33:28
  this is, you know, this is one of the most difficult tasks
• 33:32
  actually the reason why verification has been there.
• 33:36
  And this eventually awards in model psychic and all that stuff
• 33:39
  for many years. The problem is, you know,
• 33:43
  writing the tests like or writing the verification, you
• 33:48
  know, processes that you need to pass through.
• 33:53
  And if you pass through, you say, well, that it should be
• 33:55
  correct, OK. Yeah, it's.
• 33:58
  It's basically unit tests for the output.
• 34:01
  Yeah, it's a little bit more than a unit test because it
• 34:04
  could be abstract. You know, you have to, you know,
• 34:07
  it's like show me that these variable never, you know, prove
• 34:11
  that this variable never becomes negative in the code.
• 34:15
  You're saying that I'm having some variables and and you have
• 34:19
  to prove that they will never exceed those bounds.
• 34:22
  Like which boxes would be all the all that stuff.
• 34:25
  There was a workshop over there and and lots of people talking
• 34:29
  about that, you know, also a great topic.
• 34:35
  But the verifiers themselves are not generated by foundational
• 34:40
  models, they're generated by humans.
• 34:43
  Yeah, the verifiers are symbolic systems.
• 34:46
  You know, this is how we like calling this symbolic system.
• 34:48
  Like if I tell you, for example, if let me give you an example, a
• 34:52
  verifier that everybody was going to understand.
• 34:55
  Let's say that the I'm asking a language model to generate a
• 35:01
  path from Atlanta to to San Diego and it gives me a path.
• 35:06
  Now I can go to Google Maps that they have a graph and I can say,
• 35:10
  OK, you told me to go from this road to this road and take this
• 35:15
  intersection. Well, does this intersection
• 35:16
  exist? I have to go from this note to
• 35:19
  the other note. Where does this exist?
• 35:21
  And then you told me that the whole thing is going to be, I
• 35:24
  don't know, 3000 miles. Like it could go and measure.
• 35:27
  Well, you actually it's not, it's 2000 miles or it's 3 1/2.
• 35:31
  So that's what I'm saying that you can think about.
• 35:34
  Most of the verifiers that we would use in in enterprise
• 35:38
  applications will involve some type of a graph or some types of
• 35:42
  rules. We like calling them the whole
• 35:44
  packets. That's knowledge graphs in in
• 35:47
  the enterprise. This is, you know, again,
• 35:49
  relational AI where I work. So the knowledge graph and
• 35:52
  that's why they're becoming very popular with 10 AI is, you know,
• 35:57
  they have a very strong interactions with language
• 35:59
  models. You produce something and I'm
• 36:01
  using the knowledge graph to to verify that this is working.
• 36:05
  Or sometimes, you know, maybe I'm creating a, a, a symbolic
• 36:10
  path that I go and I execute it on the knowledge graph.
• 36:14
  That's lots of things. And sometimes even people use
• 36:16
  the knowledge graph to train the the language models and we we
• 36:19
  divert a little bit, although there was some discussion about
• 36:22
  that on yours. Wow.
• 36:26
  Well, so there I'm trying to think what we were talking about
• 36:29
  before we were talking about the verifiers.
• 36:31
  And we went on that little tangent because there there was
• 36:34
  also some good stuff. They're but man, these verifiers
• 36:39
  are wild in trying to figure out the actual verification of them
• 36:46
  is quite difficult. And so all the talk and all the
• 36:50
  fun stuff around that I'm I can only imagine is going to be
• 36:56
  getting more and more complex as time goes on.
• 37:00
  No, no, you're absolutely right. And just keep in mind that our
• 37:04
  you remember when I talked before and I said gaming, trying
• 37:07
  to play poker, trying to, you know, play chess and all these
• 37:11
  games, They basically evolved and brought, you know, the the
• 37:16
  kind of like chain of thought and test time inference, like
• 37:20
  what we see right now, the old class of models or the deep sea
• 37:23
  class. I think people use the old class
• 37:25
  because open AI was the first one O one or three O 4.
• 37:29
  You know, that was by working with games.
• 37:31
  That's a beautiful talk. It's historical.
• 37:33
  It's nice. It has a lot could be a movie,
• 37:35
  you know, because there's students beating, you know,
• 37:38
  playing poker, winning. We can we can make it Hollywood,
• 37:42
  You know, kind of like the same thing that's happened with the
• 37:44
  verifiers and math. I mean there is there's two
• 37:46
  reasons why the community is so focused on math.
• 37:51
  The first one is because this is kind of human nature.
• 37:53
  You know, we want to prove the Riemann hypothesis.
• 37:55
  You know, we want to go to the moon.
• 37:57
  Like people have this aspiration that, you know, this is, you
• 38:02
  know, we, we have the money, we have the, the brain power.
• 38:06
  Let's go and solve a difficult problem.
• 38:08
  OK. The second thing though, is
• 38:11
  it's, it's hilarious. You know, you know how you want
• 38:14
  to study history at Yale and people tell you, well, you know,
• 38:18
  in order to increase your times to get at Yale, you should take,
• 38:22
  you know, AP calculus and AP algebra, like advanced algebra.
• 38:26
  And you say, well, I want to study history, say, well, yes.
• 38:29
  But when you're going to submit your application, they want to
• 38:33
  make sure that you are smart and you're, you're hard working.
• 38:36
  And you know, if you are good at math, then it means that you're
• 38:40
  smart and hard working. Of course, this is a little bit
• 38:43
  unfair because there's other ways of testing people if they
• 38:46
  are smart or, or hard working, but this is what people know
• 38:51
  over the years. You're good at math, you're
• 38:53
  smart. OK, it's it's a little bit, I
• 38:55
  don't like that, but that's how the now we do the same thing
• 38:59
  with language models. It's actually been proven that
• 39:01
  if you teach a language model to do math, it becomes a better
• 39:05
  agent, it becomes a better thinker.
• 39:07
  So you would say that people are pushing math theorems and code
• 39:11
  into the training data, not because they want the model to
• 39:14
  do math, but it wants to be the better reason.
• 39:17
  It's like Wall Street, you know, they're saying, oh, this guy's a
• 39:20
  quantum physicist, you know, the studies abstract math, let's
• 39:24
  hire them because, you know, they're going to be able to
• 39:26
  figure out the stocks. So, so these are the two reasons
• 39:30
  why and and that's where the whole discussion about the
• 39:32
  verifiers of course, is coming up.
• 39:38
  No, you've been working with graphs a lot.
• 39:42
  Talk to me about what you're working on.
• 39:44
  Let me first give you my definition.
• 39:47
  It's not actually mine. It it it came from someone who
• 39:51
  worked for us in sales. What is a knowledge graph?
• 39:53
  OK, Knowledge graph is a language that both humans and
• 39:59
  machines understand. OK, if I write code, code is
• 40:04
  something a machine understands really well.
• 40:06
  I don't think the average human or anybody understands.
• 40:08
  It's very difficult even for a software engineer to to read
• 40:11
  code. And knowledge graph is kind of
• 40:14
  like an intermediate representation, OK, It has like
• 40:18
  facts, it has qualifiers, relations, entities, rules.
• 40:23
  OK, so this is, this is, you know, the, the, the truth for
• 40:33
  every company, for every organization.
• 40:36
  Like if even if you know, my favorite example of knowledge
• 40:39
  graphs is, you know, Euclid, Euclid basically created the
• 40:45
  first knowledge graph of mathematics, you know, with his
• 40:48
  theorems, his axioms and not axioms.
• 40:50
  He had, you know, he built theorems and, and that wasn't
• 40:54
  imperfect. There's actually a wrong
• 40:55
  knowledge graph. Like it took about 2000 years, 2
• 40:57
  1/2 thousand years until David Hilbert completed and put all
• 41:01
  the pieces together. And because of that knowledge
• 41:03
  graph he built, it helped him pose the famous 20 or 10, I
• 41:08
  think. I don't remember questions about
• 41:12
  that. I have to be answered.
• 41:14
  So how beautiful this is. You know, the knowledge graph
• 41:17
  helped us systematize everything in the back in between, even
• 41:20
  before Hildweh it, it let us go to different geometries like,
• 41:23
  you know, non Euclidean geometries like hyperbolic,
• 41:27
  spherical and all of them. And then Hilbert said, OK, now
• 41:30
  that we know what we have, these are the 20.
• 41:32
  I think they're 20 and it's either 20 or 10.
• 41:35
  I think they're 20. There was that he said in the
• 41:37
  famous conference in 1900s and that, you know, because he said
• 41:41
  those questions then you know, Geddle came and the others and
• 41:44
  there's still some of them unproven.
• 41:47
  The same thing is happening on a on a corporation, an enterprise.
• 41:50
  You kind of like have to create this ultimate source of truth,
• 41:54
  which is your knowledge graph. OK.
• 41:57
  And once you have that, it helps you build apps on top of that,
• 42:01
  answer questions, do your research as we described before,
• 42:04
  in the future, like, show me what product I need to to
• 42:08
  develop. OK.
• 42:11
  The problem is, as I said, you know, building the perfect
• 42:14
  knowledge graph, it's not a task, it's a journey.
• 42:16
  And it can take time. And usually what happens is that
• 42:19
  the information is spread all over, you know, around here, you
• 42:22
  have them in databases, you have them in documents, you have them
• 42:25
  in code. You know, on Slack, you have
• 42:27
  them in Yeah, you have them in Slack.
• 42:29
  You have them in Zoom, transcripts, emails and emails
• 42:33
  everywhere. And somehow this needs to be,
• 42:36
  you know, you need to build like this skeletal if you want like
• 42:38
  this, this ultimate source of truth.
• 42:44
  Now, you know, I remember when I started my, my job at
• 42:49
  Relationally high back in 2018, I would go to customers and say,
• 42:54
  hey, you know, you know, if you have a knowledge graph like we
• 42:56
  have the system and we can help you do XYZ, this is great.
• 42:59
  But I don't have a knowledge graph.
• 43:01
  What would it cost me? And you know, you put down the
• 43:03
  cost it would cost you, I don't know, $1,000,000 or something
• 43:05
  like that because it was too custom.
• 43:07
  You know, you, you'd really needed a lot of work annotators.
• 43:10
  And there's example you can go, if you go to my Europe's
• 43:13
  presentations a few years back, not that that long, you can
• 43:16
  find, you know, the famous talk by Luna Dong from Amazon where
• 43:19
  she was saying that the product graph they built was 100 million
• 43:23
  projects, you know, basically more than 100 million.
• 43:27
  I think it was 100 year project. There was several millions of
• 43:29
  dollars to build up. That's for the products of
• 43:31
  Amazon. LLMS didn't exist.
• 43:33
  An army of data scientists, annotators, blah, blah, blah.
• 43:36
  Another language model comes and all of a sudden that process
• 43:40
  the, the, the bar is, you know, the cost is, is going, is going
• 43:44
  down. Like everybody can do that.
• 43:48
  I, I wouldn't say it's trivial. You just don't get things.
• 43:50
  You put them on the language model and it happened on it's
• 43:54
  own, but you need to build tools.
• 43:55
  So part of my day-to-day job is building tools natively in
• 44:00
  Snowflake because there's another aspect that we haven't
• 44:04
  talked about, what's called security.
• 44:06
  OK, so moving things around makes them insecure.
• 44:10
  So you basically want to do everything.
• 44:12
  I talked before about people wanting to do things in the same
• 44:15
  stack, in the same, you know, cloud provider, you know, in the
• 44:21
  same space, in the same language if possible and not and not move
• 44:26
  around. So my job is how I can take all
• 44:31
  the sources of data and create a notice graph in a, in a, in an
• 44:37
  environment that is very friendly to databases because
• 44:40
  because the majority of the data is stored there, it stays.
• 44:44
  And that has, you know, of course, it's silences because
• 44:47
  databases, they are optimized for a particular, you know, task
• 44:51
  which you have to respect and you're trying to do something
• 44:56
  else. So I'm I'm there bridging that
• 44:59
  that gap. Well, there's a lot of questions
• 45:01
  that come up instantly because I can imagine from the data
• 45:08
  engineering perspective, you have hot data and cold data.
• 45:12
  And so how does that play with the knowledge graph?
• 45:13
  You also have data retention policies, so how does that play
• 45:19
  with the knowledge graph? All of these things seem to make
• 45:22
  it a very sticky situation. It does.
• 45:25
  And you know, we tend to conflate to different and I
• 45:30
  didn't make it clear because then we're confused, but maybe
• 45:34
  it's the time to talk about that.
• 45:35
  There's something called the ontology and there's something
• 45:38
  called the knowledge graph. The ontology is like the type
• 45:40
  system, you know, you got a location where you got this and
• 45:43
  that generally no types that these are kind of like invariant
• 45:47
  over time. And then on that you attach like
• 45:50
  a fact the facts, the value types, as we call them these
• 45:55
  which change and they have an expression there.
• 45:57
  Now I think here's a clever thing that we're doing.
• 46:01
  We belong in the category of the relational knowledge graphs,
• 46:05
  which means that our view of the world is very, very closely tied
• 46:13
  to the database. So you can think about a
• 46:17
  knowledge graph as a as a view, as a layer, semantic layer that
• 46:23
  writes and reads everything from a relational database.
• 46:27
  OK, so relational databases have kind of like solved this
• 46:30
  problem, he says. Retention, access roles, you
• 46:35
  know, scaling all these things. So of course you know, there are
• 46:43
  still some talents because you know, when you when you fine
• 46:45
  tune a language model, you, you know, just doing that, like
• 46:49
  incrementally adding facts you can do, removing them is not
• 46:52
  that easy. So you got to be careful what
• 46:55
  you fine tune and what you do dynamically.
• 46:58
  So it is a scientific, but it's mainly a, an engineering
• 47:04
  project, you know about how you let's say it's an engineering
• 47:06
  problem where you need to be smart about AI and how you use
• 47:10
  it. So it's, it's R&D, you know, if
• 47:13
  you feel. But how are you making sure that
• 47:16
  your worldview is not limited to only that database?
• 47:20
  Or is that the way that you like to do it where you say 1
• 47:23
  database, 1 worldview, 1 type of knowledge graph?
• 47:27
  It could be multiple databases, but when we bring, for example,
• 47:29
  text data, you know, we, we ingest them and they see it as,
• 47:33
  you know, let's say triples. NVIDIA is located in Mountain
• 47:39
  View or Mendo Park, wherever they are, you know, that's like
• 47:44
  a fact. It has a time stamp it's, but it
• 47:46
  leaves. So the document is there.
• 47:48
  You ingest it like after you adjust everything, it sits on
• 47:51
  the knowledge graph and the knowledge graph sits on the
• 47:53
  database. Some of the data they already
• 47:55
  sit on databases. But I think what people don't
• 47:58
  realize that when they build databases, they build them to
• 48:02
  serve an application and they put more emphasis on how it's
• 48:05
  going to be fast, how it's going to serve the application, but
• 48:09
  not how it reflects to the concepts that you as a business
• 48:13
  owner or stakeholder think about them.
• 48:16
  So that's why you see some database that they're super wide
• 48:19
  and you look at all this column. So you don't know what's going
• 48:21
  on with this, you know, cryptic columns.
• 48:24
  And some of this is, yeah, if you actually need the customer,
• 48:27
  you have to say select this when this and this like it's like AI
• 48:30
  don't know a 10 pages query that basically defines what a
• 48:34
  customer is something. Like that?
• 48:35
  Your handshake, yes. So that's because that database
• 48:40
  was built to serve, you know, production, you know, to be
• 48:43
  optimal, to minimize whatever you want, like whatever
• 48:46
  constraints you had. But that's not the way that you
• 48:51
  think about the data, the word your company, you know, like
• 48:55
  when you're thinking about changing a gear in your car, you
• 48:58
  know, it's a sequence of things happening.
• 49:00
  You don't care about all these things you're saying.
• 49:01
  I'm saying it's a gear. It's it's a wheel.
• 49:04
  You know, you don't, you don't need to know the material that
• 49:06
  the wheel is like, you have to abstract that, OK, when you're
• 49:09
  writing things now, of course, there's a lot of collaterals.
• 49:12
  You know, 1 is, you know, answer your questions that the other
• 49:15
  thing is it's not very well known to the masses is that if
• 49:21
  you had designed your database according to how you, you, you
• 49:25
  think of your business and that representation could also be
• 49:29
  efficient. OK, Then it means that you can
• 49:32
  build your apps much faster because every time you know,
• 49:35
  like somebody wrote an app and defined the, the active customer
• 49:38
  somewhere with a big SQL query and it's like in a Java file.
• 49:42
  And then somebody else wants that and doesn't know how to
• 49:44
  find it and goes and writes that query.
• 49:46
  And that query is a little bit different from the other and
• 49:48
  others like inconsistencies. And so if you have things in the
• 49:54
  knowledge graph and you're trying to build an app, you can
• 49:57
  do it much faster, much cleaner, it's more maintainable.
• 50:00
  And we do have, you know, like we have customers like, you
• 50:03
  know, where we, we took, you know, hundreds of thousands of
• 50:05
  lines of code, like UI, where we could be able to compress them
• 50:09
  with something smaller because we had a better representation
• 50:13
  of data. So that's something that's
• 50:15
  probably not very popular or familiar to the AI audience, but
• 50:20
  when it comes to application development and enterprise
• 50:22
  application development, this is another huge use of of no, this
• 50:26
  glass, you know, the whole enterprise is running on
• 50:28
  applications. There's a lot of meat on them
• 50:32
  bones. That is 100% things that I've
• 50:37
  heard people talk about, especially when it comes to
• 50:40
  self-serve data and analytics. And each person is, like you
• 50:45
  said, creating their own data pipelines in their own way, or
• 50:49
  they have their own dashboard and then they leave the company
• 50:52
  and the data engineer doesn't know, do I need to keep
• 50:57
  babysitting this dashboard? Is anybody looking at it?
• 51:00
  Can I just start turning things off?
• 51:02
  I know I talked to a guy who said, yeah, we had this big push
• 51:06
  about five years ago to go to self-serve analytics and it was
• 51:09
  great. And now we have over 10,000 data
• 51:14
  pipelines that the data team has to support.
• 51:19
  And so they're literally just chaos engineering and turning
• 51:23
  things off and then waiting for somebody to come and say, hey,
• 51:26
  why is this not working? And again, I'm just going to go
• 51:30
  back to the mathematicians who are very disciplined.
• 51:33
  You know, every time you prove a theorem, you don't say and
• 51:35
  according to that you can say according to that theorem, it
• 51:38
  refers to specific. So there is a very well known,
• 51:41
  you know, knowledge graph of theorems where you go, it was,
• 51:45
  you know, Newton's dot or, you know, like you go to the
• 51:48
  specific person and it has a name.
• 51:51
  Sometimes they have a name, but sometimes we refer to them as
• 51:53
  publication. You know Theorem X.
• 51:56
  So there's a reference system. So it's very rare that the same
• 52:00
  theorem exists proven twice in two different places where you
• 52:04
  don't know. About it in two different kind
• 52:08
  of unique ways. That's the other thing here
• 52:12
  because you don't have to be limited to just doing it one
• 52:16
  way. Maybe you do it one way and
• 52:18
  then. So think about the, you know,
• 52:21
  you brought a very nice example here, the Pythagorean theorem.
• 52:24
  You know, there's about, I don't know, 50 a hundred ways to prove
• 52:26
  it. But when you're proving
• 52:29
  something that says according to the Pythagorean theorem and you
• 52:32
  don't care if it's like, you know, 50 different ways to prove
• 52:34
  it. Imagine now if we didn't have
• 52:36
  this, you know, knowledge graph. You would reference, you know,
• 52:41
  the Knicks proof and then somebody else would have the
• 52:44
  Jones proof that they refer to the same thing, but they don't
• 52:47
  have the same name. And I will say, what are they
• 52:50
  doing the same thing? Is it the same?
• 52:53
  Yeah, you have to be a expert on all 50.
• 52:56
  Different ways exactly. Or you at least have to go in
• 52:59
  reference and do a little bit of Sherlock Holmes and figure out
• 53:03
  is there actually that way to prove it?
• 53:06
  Yeah, that's typically the person you can fire from the
• 53:08
  company who lives last, like the person who's been there for 20
• 53:10
  years. And remember that in 2021 there
• 53:13
  was an Outers and you know, we weren't getting data was it was
• 53:17
  a bug. So whenever you're writing a
• 53:20
  query, you have to exclude those dates.
• 53:22
  Like you see this query and it says, but not in these dates,
• 53:24
  you know, to get the average or something because that app was
• 53:28
  polluting the data. So yeah, we have lots of stories
• 53:31
  like that. Man this is great.
• 53:34
  Anything you want to end with? Well, you know, there's 12
• 53:37
  topics. I think I'll close with one,
• 53:41
  Maybe I close with two because I have two more favours where I
• 53:43
  didn't cover the first one is something, and I'll close with
• 53:48
  that because I think I have a little bit of a contribution
• 53:51
  over there. Something has to do with data
• 53:53
  monetization, OK? There wasn't any works or any
• 53:58
  paper about data monetization, but there was something about
• 54:00
  called data attribution, OK? We all know that, you know,
• 54:05
  there's a ton of copyright lawsuits, all sorts of problems.
• 54:10
  You know what's going to happen, You know, the famous Facebook,
• 54:13
  Facebook case where meta case where they downloaded illegally,
• 54:17
  you know, pirate books to train the model.
• 54:19
  What are we doing with people? You know how you know, flank
• 54:21
  with if Jenny I is doing all that stuff, how are we going to
• 54:25
  leave? You know, how are we?
• 54:27
  Is it fair? So it was the first time that I
• 54:31
  saw papers that they can say I'm generating this token.
• 54:34
  You know, I'm generating this sentence and I did that because
• 54:39
  of that page in my training data, OK.
• 54:43
  And if that page, you know, training page was created by
• 54:46
  Demetrius, then we'll say let's go and give one penny to
• 54:49
  Demetrius for that or whatever like 10% of what we're making
• 54:52
  from that. So, so that is huge for me
• 54:55
  because if we continue in that direction, we should be able to
• 55:00
  basically because right now you're going to say, well, you
• 55:02
  know, I do that, but I, there's no way I can pay, you know, I
• 55:04
  found it on the web or something like that.
• 55:06
  So if we can have an attribution model, especially which is like
• 55:11
  per token, like per per sentence, let's say per word
• 55:14
  that I generate, then we can start paying off people
• 55:18
  dividends. And these models are smart.
• 55:21
  It's not that they find the same thing, you know, like if you go
• 55:25
  in my presentation, they they're just saying, you know, to the
• 55:27
  language model, I need to shut it down.
• 55:29
  And the language model complaints says no, I still want
• 55:31
  to be alive. And and they traced it back to a
• 55:34
  novel, a sci-fi novel that was about a robot that was supposed
• 55:37
  to be sat down. OK, that was actually written in
• 55:41
  a different language. So but the language model was
• 55:43
  able to do that. So that's one thing and the
• 55:46
  other thing that again, it's one of these things that we haven't
• 55:49
  seen that taking off, but I think it's going to take off.
• 55:52
  It's basically building smaller language model and putting them
• 55:56
  like Legos, composing them to build a bigger one.
• 55:59
  Because you know, right now one of the problems that language
• 56:01
  models are very monolithic, you know, like you, you train them
• 56:05
  in, you know, put everything, all this effort is millions of
• 56:08
  dollars to build 1. You know, it's like as if you
• 56:10
  were writing code, you know, the whole Windows system in one
• 56:13
  file. So, but that's not, you know,
• 56:16
  like this composability, there's a competition really great and
• 56:21
  I'll leave you with that. Yeah, this isn't in your eyes
• 56:26
  the directions that Laura's are trying to take.
• 56:29
  Well, Laura's are, you know, they're still, you know, you
• 56:32
  still have the llama, which is, you know, 100 millions of
• 56:35
  dollars to train and you're modifying it.
• 56:38
  That's one way. But I think what people are
• 56:41
  trying to do is it's if it's like you can take, let me give
• 56:45
  you an example. You take a language model that
• 56:47
  is specialized, very good at translating from, I don't know,
• 56:49
  Greek to English and another one from English to Italian.
• 56:52
  And you can literally like add them And now you can translate
• 56:54
  from Greek to Italian if you can compose the tasks.
• 57:00
  So it's much faster than going and training at Laura, you know,
• 57:03
  so there are some pros and cons, but it's, it's, it's an
• 57:07
  interesting direction. I think it can also help when
• 57:10
  you're building the language models like we could say you
• 57:13
  train on this, I train on those, let's put them together and and
• 57:17
  build, you know, it's more about, you know, all of us
• 57:20
  working in a collaborative way.