Latest / Agentic Conversations (formally mlops.community) / Becoming an AI Evangelist // Alex Volkov // #215

Transcript

Becoming an AI Evangelist // Alex Volkov // #215 · Agentic Conversations (formally mlops.community)

• 0:00
  Everyone. My name is Alex Volkov.
• 0:02
  I'm in the evangelist with words and biases.
• 0:04
  I'm also the host of the Thursday Eye weekly newsletter
• 0:07
  and podcast. And I my I don't drink coffee
• 0:10
  anymore for the past six months. But I do have caffeine.
• 0:13
  So this is my coffee basically either yerba matte or just black
• 0:16
  tea. It's another ML OPS community
• 0:21
  podcast. I am your host, Dimitri Olsen.
• 0:24
  What a show we've got today with Alex.
• 0:26
  He is absolutely kicking ass on Twitter.
• 0:29
  I love all the stuff he posts. Not going to lie, I don't spend
• 0:33
  a lot of time on Twitter, but when I do, I see his posts and I
• 0:36
  smile. He's also got a weekly show
• 0:39
  that's happening on the Twitter spaces.
• 0:42
  I guess that's what the cool kids are doing these days.
• 0:43
  They've got their Twitter spaces and he is one of those cool
• 0:47
  kids. Bring it on, incredible experts
• 0:49
  and talking about all of the newest updates that are
• 0:52
  happening in the AI space. And today we got into so much in
• 0:58
  our conversation. Everything from podcast, nursery
• 1:00
  on our dear and what microphones we're using and how we travel
• 1:03
  with it and what not. All the way to evaluation,
• 1:06
  multimodal, and even a bit of agent talk.
• 1:10
  Autonomous agents are, as you may know from listening to other
• 1:16
  podcasts with me, something that I think are complete bullshit.
• 1:19
  But he was not giving me the runaround.
• 1:24
  He came with his feet on the ground and I appreciate the
• 1:27
  answers and I appreciate his viewpoint on agents, which TLDR
• 1:31
  is just they're still not there yet, but they're getting better.
• 1:35
  So don't expect like they're not going to be or they're going to
• 1:39
  stay where they're at forever. The other thing on this I want
• 1:44
  to mention, we got deep down and dirty when it comes to
• 1:47
  evaluating systems, evaluating your LMS, evaluating everything.
• 1:51
  You need to be thinking about a little bit of a hack week that
• 1:54
  he did and how they are evaluating these different
• 1:58
  products that they did at their hack week And it makes me want
• 2:04
  to let you know that we are currently running a little bit
• 2:10
  of and valuations survey that is right.
• 2:16
  I broke down the past evaluation survey.
• 2:21
  I looked at it all and I wrote a blog post on it.
• 2:25
  But now time has passed and it is high time for us to do
• 2:33
  another survey, figure out what are people doing in this space,
• 2:36
  how are they going through it. And we're not just doing it on
• 2:40
  our own this time. I found an incredible partner
• 2:43
  Bendy eye to do this with. So and it's not a by any means a
• 2:48
  vendor or anything. It is someone that actually
• 2:51
  knows how to do surveys. And so that helps a lot because
• 2:57
  if you know me, I've been in the dark when it comes to how to
• 3:05
  formulate the surveys, how to extract the data.
• 3:07
  I've been leaning on a lot of the community to help me extract
• 3:11
  all the insights and now I've got some trusted friends at
• 3:15
  Bandai to help me with that and it's much needed.
• 3:19
  So I encourage you if you are doing anything with LLM's or
• 3:23
  not. Even I would love if you would
• 3:27
  take the 5 minutes, fill out this evaluation survey that
• 3:30
  we're running and we will give you access to the final product.
• 3:37
  We're going to make an insane report that is going to probably
• 3:42
  cost me a few grey hairs. That's all I'm leaving.
• 3:46
  All right, let's get into this conversation with Alex.
• 3:48
  Here So dude, tell me about what you've been up to for the last,
• 3:59
  like, what, 10 years? 12 years?
• 4:01
  15 years? You've been doing this for a
• 4:03
  while. And.
• 4:05
  Yeah. Give me the breakdown real fast,
• 4:09
  because I think you you came on my radar relatively recently and
• 4:13
  it was through your antics on Twitter, bitching about Apple
• 4:17
  pros or Apple Vision pros and stuff like that.
• 4:21
  I'm a Twitter native. I'm an ex native.
• 4:23
  I think that's that's a big part of how it came to be kind of in
• 4:29
  this profession, but generally in everything that I did before
• 4:33
  and maybe we'll talk about this later, but it's a big part of
• 4:36
  early adoption as well of technology.
• 4:38
  I I saw Twitter and once I I kind of got to the professional
• 4:42
  thing of it, I was living in another place.
• 4:45
  And I quickly realised that I can get the upper hand on
• 4:49
  everybody else around me professionally just by knowing
• 4:52
  what they don't know yet, but like the new stuff.
• 4:55
  And that's how I grew up, just professionally.
• 4:58
  I'm a Ukrainian in origin. I lived in Israel for most of my
• 5:01
  life, most of my adult life. And so I you know grew up as a
• 5:04
  as a software engineer in in Israel and there so most of my
• 5:10
  career has been in software development mostly on the front
• 5:14
  and side and then it kind of became like a full stack
• 5:17
  everybody tries to do everything.
• 5:18
  So full stack I used to call myself the fullest of full stack
• 5:22
  which was foolish but I used to do like product and design as
• 5:25
  well including like front end and back end.
• 5:27
  So very generalist approach nothing at all related like
• 5:31
  this. This whole world was like so far
• 5:33
  from me. I'm self other deduct and really
• 5:36
  bad at math. Like I'm really like I think I'm
• 5:38
  number deficient or something so math is not.
• 5:40
  Search for Mill. Yeah, and so the visual stuff
• 5:43
  pulled me more than than everything else, but I love just
• 5:45
  to create in code. And the stuff that you probably
• 5:49
  saw me through is I worked for a decade at one specific startup.
• 5:56
  I started there very early on. So I did pretty much everything
• 5:58
  I did from front end, then full stack now let teams that became
• 6:01
  like director of product, director of this director
• 6:05
  marketing. And then I basically got tired
• 6:07
  of it and I moved here. I live in Denver, so I moved
• 6:11
  into like this office worker mode just before COVID.
• 6:14
  So that like worked out. And then after maybe eight or
• 6:18
  nine years, I get so bored. I literally like hit all the
• 6:21
  possible ceilings and did all the lateral moves that are
• 6:24
  possible. And at some point, it was like,
• 6:26
  OK, I need to, I need to do some stuff.
• 6:30
  So I looked around about like, OK, what's interesting, what's
• 6:33
  new, What's cool. Grifter was my my thing for a
• 6:36
  while. For about 3 months.
• 6:38
  I went and see. I went to see if that was a
• 6:40
  whole thing while web three was exploding.
• 6:42
  I'm here in Denver ethos here. And then I kind of stumbled onto
• 6:46
  Dali from U in the Eye and back then I already kind of used
• 6:50
  Copilot and Daley just just blew me the hell away because I
• 6:56
  looked at this and I was like, I can't draw and I wish I would be
• 6:59
  able to draw and and now it seems ridiculous to talk about
• 7:03
  this in 2024. Oh you know Daley won like what?
• 7:06
  Or Dali 2. I think it was like the image
• 7:08
  quality. The facility was not that great
• 7:11
  but to me this this show that like you know a I is not only
• 7:14
  this autocomplete. Back then we didn't have a lens
• 7:16
  which is little auto complete in in GT, but it's also something
• 7:20
  that can add to me a power that I don't have.
• 7:26
  Yeah. And Dolly started my deep dive
• 7:28
  into the eye which you can talk about separately from I I was
• 7:32
  working with Dolly and then stable Diffusion was starting to
• 7:35
  come up in in the Discord, in the beta and as they release
• 7:40
  stable diffusion release daily exactly like that, they started
• 7:43
  charging money. So you used to get from like, I
• 7:46
  don't know, an impossibly free tier in Delhi that you can do
• 7:49
  everything and you have to iterate through this fund to
• 7:52
  like, hey, I'm going to start charging you and then also stay
• 7:54
  with you. Like, hey, here's the free thing
• 7:56
  and that's the thing was a a model awaits file and that's it.
• 8:01
  And the ability of some Python code.
• 8:03
  And so that's that's where my kind of AI programme started.
• 8:05
  And then we can talk about how I got into AI evangelism from
• 8:09
  there. Incredible.
• 8:10
  And before we get there, what do you feel like burnt you out you
• 8:15
  mentioned and what were you looking for like doing all these
• 8:17
  lateral moves? What was it that you were
• 8:21
  looking for? I think now in retrospect, so
• 8:28
  since then we can probably chat about this.
• 8:29
  Like about a year after I discovered the I also switched
• 8:32
  careers. I don't code anymore unless I
• 8:35
  really have to, like I do some demos here and there I code for
• 8:37
  fun. But I decided personally that
• 8:39
  the the coding days for Alex is somebody who like, commits code
• 8:43
  and and you know maintains code and production are done.
• 8:46
  And I noticed this after a while doing my own startup,
• 8:49
  specifically in that company it was about.
• 8:53
  It was a fintech company and we did like a very cool product,
• 8:56
  but nobody that I knew in the worlds that I'm in on X or
• 9:02
  whatever, nobody cared what we did.
• 9:05
  And you could talk about like the cool stuff that we build,
• 9:07
  but it wasn't really like very interesting and I just loved
• 9:09
  talking to people. I love the the community
• 9:11
  building aspect of this and there was no community to build
• 9:14
  around that thing specifically. And I think that I tried
• 9:18
  multiple things within the org. So I did a bunch of, you know,
• 9:21
  hackathons internally. I did like a bunch of stuff and
• 9:25
  I think when I move the remote, so a whole community piece of a
• 9:31
  workplace that exists in the workplace, I think it was
• 9:33
  missing for me. And so when I worked remote, I
• 9:37
  stayed there for a while still to like keep it going and I have
• 9:40
  a family. But after a while I was like,
• 9:42
  no, I need, I need more. I need something that I enjoy in
• 9:46
  my life and and it just didn't have a lot of enjoyment.
• 9:49
  And then also they did a layoff. So you know the longest
• 9:53
  employees that were the the 1st to get hit super quick, but
• 9:56
  actually it was a blessing in disguise.
• 9:57
  So you went through the layoff you were trying to do.
• 10:00
  So you were trying to do community stuff at a company
• 10:03
  that was basically had no need for community.
• 10:06
  You also were trying to find your footing and recognise what
• 10:09
  it was that you were really passionate about.
• 10:11
  It seems like you you knew what you were passionate about, but
• 10:15
  at the place that you were at, it wasn't able to service those
• 10:19
  passions. No, no, there's no developer
• 10:23
  experience to be done because we didn't offer an API and me
• 10:27
  coming up on ex. Most of the folks that I
• 10:31
  followed are were at least back then folks who are developer
• 10:34
  experience with Google on the Chrome team.
• 10:36
  I remember like all of the names Jack Archibald and like you know
• 10:39
  all of those folks and to me those were like the the coolest
• 10:43
  positions possible. Those are the the guys who go to
• 10:45
  give talks. And I used to when I lived back
• 10:47
  in Israel, I went, I loved giving talks on stage.
• 10:50
  I loved just like presenting and and teaching people what I
• 10:52
  learn. And oftentimes this is my
• 10:54
  process of learning myself. And even back then I just commit
• 10:58
  to a talk about the topic that I had no idea about.
• 11:01
  And I promised myself that by the time that this talk is
• 11:03
  ready, like I'm going to be the person, the expert on stage.
• 11:06
  And a lot of this came from literally from just like being
• 11:09
  able to be on Twitter, learn super quick about very new niche
• 11:13
  concept. I I just didn't and find a way
• 11:16
  to turn this into my actual career.
• 11:18
  And so you know I don't love managing and I got stuck into
• 11:21
  this like management route where I had to like make sure people
• 11:24
  executing et cetera that that was my wasn't my vibe.
• 11:27
  So I scaled back a little bit to individual contributor and still
• 11:31
  it wasn't like what I actually wanted to do.
• 11:34
  And after they laid me off or I guess after the laid off
• 11:37
  happened, it wasn't like a specific personal thing for me.
• 11:40
  It was a blessing in disguise. I was already like super tapped
• 11:43
  out nine plus years. You know all the upsides for
• 11:46
  stock, equity, whatever. That way, in the past I decided
• 11:51
  to embark on the solo journey of my own to to open the startup.
• 11:55
  And this was around the time where AI was happening very,
• 12:00
  very strongly. Open the Eye released Whisper
• 12:03
  and yeah this was a few months of testable diffusion release.
• 12:06
  So stability fusion gave me the the kind of the understanding.
• 12:08
  OK, here's the weight file. Here's how you going to rod this
• 12:12
  brain to do what you want. And then Whisper came out.
• 12:15
  I was like, oh this is so cool. Let me let me try Whisper real
• 12:17
  quick. And so I I tried Whisper real
• 12:19
  quick as I was building already a startup around AI art and
• 12:23
  stable diffusion. I tried Whisper and the first
• 12:25
  thing that came to mind was, hey, I can use this for
• 12:28
  translation. It's not only ASMR, it's not
• 12:30
  only automatic speech recognition.
• 12:32
  Whisper also does decent translation.
• 12:34
  If you just say, hey, that person is actually talking, you
• 12:37
  know, English and not Ukrainian whatever, like there's a
• 12:40
  translate like. And so I used Whisper to put
• 12:42
  subtitles on the video from Ukraine and Ukrainian origin,
• 12:47
  right. So.
• 12:47
  So everything's happening in Ukraine at the time I'm
• 12:50
  ingraining in origin. Something happened there that's
• 12:52
  very interesting. And I don't speak Ukrainian.
• 12:55
  I'm from the like Russian speaking area of Ukraine.
• 12:58
  So I don't like very natively speak.
• 12:59
  Again, I really was interested like Ohh I can do this with
• 13:02
  Whisper and so I built like a super quick think that
• 13:05
  translates videos and put subtitles and put it on X and
• 13:09
  that blew up, that blew up and I was like ohh OK.
• 13:14
  This took me like 5 hours to set up to to put the you know,
• 13:17
  Python notebook to like do FFM pack to the stitch stop there or
• 13:20
  whatever. But like this can happen
• 13:24
  automatically and so I went into a cave even though I already had
• 13:27
  like a starting brewing and I for a weekend spent time in that
• 13:32
  game and came out of it with a bot on X.
• 13:35
  The auto translates videos O actually like you know, a lot of
• 13:40
  text but also started to to show a lot of videos from different
• 13:42
  worlds, especially around the war in Ukraine.
• 13:45
  I built a bot that just replies with the translation and I think
• 13:50
  I was the first one to do this automatically and that became my
• 13:53
  startup. For the next I want to say seven
• 13:55
  months. Putting that aside those super
• 13:58
  ohh so we we can talk about the envelopes of this.
• 13:59
  I have to like scale this. I had to put the models in
• 14:01
  production had to monitor them. I had to do, I had to learn like
• 14:05
  another full stack on top of my already full stack and that was
• 14:08
  that was fun. But I think it's behind me we
• 14:11
  can talk about. It but is that when you went to
• 14:13
  the local meet up in Denver? Are they going there a little
• 14:17
  bit after? So in Denver specifically I met
• 14:19
  with the envelopes, folks in Denver shout out and then Bianca
• 14:23
  and yeah, Phil, I wanna say Yep. And yeah, I forgot the names.
• 14:30
  I'm not really good at names on the spot.
• 14:33
  I'll I'll try to remember them. And I I went there because we
• 14:38
  also have a local meet up here in Denver because one of the
• 14:40
  biggest parts of being a remote worker is I missed a community
• 14:45
  in real life, in physical life and I should say and we started
• 14:49
  our own like I meet up and some of them came and then a a glare
• 14:53
  as well. I think Claire showed up an
• 14:54
  armed up and said hey tomorrow we're doing our own thing.
• 14:57
  MLP you should come. And so I came like a day after
• 15:00
  day, so the meet up so super fun.
• 15:03
  Yeah, probably Pete too, I imagine.
• 15:05
  Yes, yeah, yes, I feel it. But definitely beat he he he
• 15:09
  comes to ours as well. If you combine forces now, we
• 15:12
  should. Exactly.
• 15:14
  The whole team there. What is super cool for me to
• 15:18
  watch the Denver community grow is that the first meetups I
• 15:23
  think had like 5 people, and then there were seven people and
• 15:28
  then there was like 10 people, and then there was like 50
• 15:31
  people and boom, there's 100 people, so.
• 15:34
  And now they're sold out. Yeah, and now they're school
• 15:36
  events. It's one of those things where
• 15:38
  you just you get to see that it takes on a mind of its own and
• 15:42
  the dedication from the local organisers to keep going with it
• 15:45
  and keep figuring out ways to make it better.
• 15:48
  We've been having these like local organiser meetings where
• 15:53
  we share ohh in Berlin. They did this like we have.
• 15:57
  I'm part of the Berlin chapter, right?
• 15:59
  So, Well, kind of. I guess it's a bit of a stretch
• 16:03
  these days to say that I'm part of it.
• 16:05
  I used to help organise it more, but now I just drop in to the
• 16:09
  events occasionally and what they do, which I think is super
• 16:14
  fun, is that we will get together and at the beginning of
• 16:17
  the events to encourage people to break out of their shell and
• 16:22
  meet other people. We'll have like trivia games.
• 16:26
  And so we say, alright everybody, you get put into a
• 16:31
  team depending on when your birth date month is.
• 16:34
  So if you're born in May, you're gonna be with everybody else
• 16:36
  that's born in May and that's your team.
• 16:38
  If you're born in June, doesn't matter the year, it's just the
• 16:40
  month, right? And then you get together and
• 16:44
  there's a trivia quiz and the person who or the team that wins
• 16:48
  the trivia quiz gets like Bose speakers or whatnot.
• 16:51
  And so we make sure to have some cool prizes.
• 16:55
  But at the end of the day, it's so cool to see the people
• 17:00
  interacting on a totally, you know, I can go there with no
• 17:04
  friends and come out with a group of comrades because you
• 17:08
  could say. And I think I, I, I this piece
• 17:13
  in my kind of position at that startup, Defender startup was
• 17:16
  sorely missing from me. I was missing that exactly.
• 17:19
  I love building communities. I did this throughout everything
• 17:22
  including within the startup. As I moved countries, as I as I
• 17:26
  kind of moved cultures essentially I still kept doing
• 17:29
  this and I think that's a big piece of what you guys doing
• 17:32
  with them. All apps like everywhere, local
• 17:34
  meet ups and everything. That's what we do.
• 17:36
  That's how we kind of we we we met with the envelopes community
• 17:39
  because they came to ours and then into there and you know a
• 17:42
  lot of the people are pretty much given the same talks
• 17:45
  talking about the same things Claire talked about you know
• 17:48
  arise for example and that's something that is in milliamps
• 17:52
  but also fits the AI meet up that the tinkers do.
• 17:55
  And I think one of the coolest things that I've noticed in in
• 17:58
  our meet ups is we had a month after month conversation where a
• 18:03
  month after somebody came up and started talking about their
• 18:06
  process and then something that a light bulb moment that he had
• 18:10
  during the previous meet up that somebody else talked about the
• 18:13
  approach to AI tinkering. And that other person Matt was
• 18:17
  in the room and the guy was on stage with Sam and he's like ohh
• 18:21
  yeah that's the guy you know we're like yes this is exactly
• 18:24
  what we're here for. This is exactly like these like
• 18:27
  osmosis learning thing doesn't happen necessarily online it
• 18:31
  doesn't act. But you know not everybody's as
• 18:33
  native to access as as I am. But definitely that's that's the
• 18:37
  community part that we want. All super cool And then we had
• 18:40
  beer almost who. Drink.
• 18:41
  Yeah, you make it a good time. You make it.
• 18:44
  It's like guaranteeing that you're going to go there, you're
• 18:47
  going to enjoy yourself and then learn something to and honestly,
• 18:52
  like after whatever two years of being stuck inside, everybody
• 18:55
  just wants to go and be around people again, right?
• 18:58
  The tops in my eyes are an excuse to just hang out with
• 19:01
  people and have. Us through something.
• 19:03
  Sometimes there's too much talking and not not not enough
• 19:07
  person to person talking the. Other person?
• 19:09
  Yeah, 100%. And you see that too with all of
• 19:13
  the different in person events like conferences that are
• 19:17
  popping up again. And so I I understand that is
• 19:23
  there's a value in that connection.
• 19:25
  I also am huge, obviously. I'm huge on virtual stuff.
• 19:29
  We're doing all these virtual conferences and I think that has
• 19:33
  its value because you can get people that are in different
• 19:38
  parts of the world and they can jump in.
• 19:41
  You don't have to be in Denver to get the knowledge from the
• 19:44
  experts in Denver, right. It's just kind of like the
• 19:47
  pressures on me, the organiser, to make sure to go out there and
• 19:51
  get everything sorted. So these experts want to come
• 19:55
  and talk and they want to share the value and then we make sure
• 19:59
  that the talks are good. But.
• 20:02
  Dude, so talk to me about the transition that you went
• 20:05
  through. You you mentioned how, yeah, you
• 20:08
  stopped in COVID and then you had your own stuff.
• 20:11
  You started working on AI, you had your own for seven months.
• 20:15
  You were doing all kinds of Milo stuff.
• 20:17
  I imagine you learned quite a bit in a short amount of time.
• 20:22
  So I'm sitting in Denver. Ohh no friends were into AI.
• 20:28
  One friend who's it's more of a dialogue between me and him and
• 20:34
  not a conversation where I talk about the stuff that he's very
• 20:37
  interested, but it's not like we're brainstorming, right.
• 20:40
  And a very good friend of mine. And like I'm very happy for the
• 20:43
  opportunity to be able to be vocal as well and not just look
• 20:46
  in front of the screen. Because sometimes you just need
• 20:48
  to bounce ideas off of folks, but not as your reputation.
• 20:52
  Yeah, Robert Ducky, exactly. But not for debugging
• 20:54
  specifically. It was just for like, hey, this
• 20:56
  cool thing happened. That cool thing happened.
• 20:57
  Can connect the whisper model, I can connect with translation
• 21:00
  model, and I can put up a pipeline.
• 21:02
  I can do evaluations on production.
• 21:03
  I can do, like, all these cool things.
• 21:05
  And you're like, yeah, but he's like not a technical person at
• 21:08
  all. And that was very helpful.
• 21:10
  And I just turned to my community online.
• 21:13
  And so there's a million additional thing where I got
• 21:16
  lost on the X for a while during 2016 to 2020 in the politics era
• 21:21
  and my whole feed the game just look awful, just awful and not
• 21:25
  fun. And it kept scrolling and like
• 21:26
  engagement farming like got to me and everything.
• 21:28
  And as I was starting like the startup, I decided hey I'm gonna
• 21:33
  like clean this out. I want only AI stuff.
• 21:37
  And as I'm learning I think you you touched the very interesting
• 21:40
  thing as I was learning. My process of learning is is
• 21:43
  speaking about the stuff that I learned, that I I learned the
• 21:45
  best publicly. When when I use X as my kind of
• 21:49
  note taking nap or brain dumping up.
• 21:51
  I noticed that like the content that performs the best is the
• 21:55
  content I just put out for myself to be able to find it
• 21:57
  later. And I used to do it with a blog
• 22:00
  a long time ago. And that's the content that
• 22:02
  resonates with people. Not the performative staff, not
• 22:04
  the like, Hey, here's 7 tricks and tips and whatever, just the
• 22:07
  counters. Like, hey, I just found out this
• 22:09
  cool thing. I just put that on production
• 22:11
  and this worked and this didn't work.
• 22:12
  Like that content started showing up and so I just noticed
• 22:15
  that I'm building this community on X and slowly getting followed
• 22:19
  by folks. We're also discovering this as I
• 22:22
  was building this Chaji came out like as I literally started, I
• 22:25
  want to say October with with whisper stuff and those
• 22:29
  building, building, building. And then he came out.
• 22:31
  I could not ignore what's going on.
• 22:33
  I could not. I used GPT 3 before the auto
• 22:36
  completion stuff. It was night and day.
• 22:38
  Something about the memory retention in the in the product.
• 22:41
  It was night and day and to me it wasn't the night and day in
• 22:44
  the UI have obviously everybody's now on the, you know
• 22:47
  the chat bot is incredible. You talk to it.
• 22:49
  It gives you so much. To me the night and day was
• 22:52
  because I used these tools before to do like a bunch of
• 22:54
  stuff, but now I wanted that as an API.
• 22:58
  I wanted there like memory management in context.
• 23:01
  I wanted like the the ability to handle the conversation, the
• 23:04
  ability to role play it, like all these things.
• 23:06
  So actually came out with like a bunch of side projects and I got
• 23:10
  sidetracked when we started just for a little because I had to.
• 23:12
  I had to make sure that like I know what's going on with this
• 23:15
  new technology. So I did a bunch of stuff like
• 23:18
  that went kind of viral. Learned a bunch, learned about
• 23:21
  prompting about sandwich prompting.
• 23:22
  I came up with like a bunch of techniques and I love the new
• 23:25
  stuff like we we just talked about the Vision Pro that you
• 23:28
  know I just bought 1 release. I I my best existence as as as a
• 23:34
  person is on that frontier that like some we all got something
• 23:38
  new as humanity and I get to discover this first and talk
• 23:41
  about this publicly and when this whole thing which actually
• 23:45
  came out I started noticing that slowly started noticing that
• 23:48
  while I'm building my startup it's an AI startup.
• 23:50
  It's very exciting. I'm kind of way more excited to
• 23:54
  share than to build to share one of the building to build and and
• 23:59
  I think that slowly rotated like after a couple of months I went
• 24:01
  down to the hole again. I built the whole UI and the
• 24:04
  payment system and everything. I released my product to the
• 24:07
  world. It's called Targum video.
• 24:08
  It's still live. Every like people still use this
• 24:11
  and I just noticed that as I released it I was like OK how do
• 24:14
  I market this? OK I released it.
• 24:16
  I'm I'm a solo entrepreneur I don't have employees I didn't
• 24:18
  get fundraised. I how do I market this Ohh you
• 24:22
  know keep doing what I'm doing publicly I keep talking about
• 24:24
  the stuff that I'm doing. Yeah, I slowly started noticing
• 24:28
  the talking about AI and innovation and cool stuff is
• 24:32
  taking over the the the excitement about the product
• 24:36
  itself, the excitement about building more features, the
• 24:38
  excitement about delivering value to customers, all these
• 24:41
  things. And that's where I started the
• 24:44
  audio stuff. So X or Twitter, formerly known
• 24:49
  as Twitter, some of us still refer to this Twitter.
• 24:51
  There's a Ex Spaces thing, which is kind of like Clubhouse for
• 24:54
  folks who who are not Twitter natives.
• 24:57
  Basically a radio show with an audience.
• 24:59
  Live presented the radio show with an audience and I hopped on
• 25:03
  one with my friend Travis, also a thinker in AI when GT4 came
• 25:09
  out, so I remember that date. March 14th, 2022, 2023, Ohh
• 25:15
  2022, March 14th, 2023 is a GT4 comes out and we're all been
• 25:22
  waiting and it's incredible. And we hopped on the Twitter
• 25:24
  space of somebody elses that ended.
• 25:27
  And we were like, we were so excited about talking about
• 25:28
  this. Like, hey dude, let's just open
• 25:30
  a space of our own, keep the conversation going.
• 25:32
  And we had like a nice vibe. And so we started with this and
• 25:35
  that happened on a Thursday and next Thursday we we came on
• 25:39
  again. And next Thursday we came on
• 25:41
  again. And next Thursday we came on
• 25:42
  again. And then at some point Travis
• 25:44
  started his own startup. So he's like, hey Alex, I don't
• 25:46
  have time for this, you go. And so I kept going and I have
• 25:50
  been doing this every Thursday since then, I think more than 70
• 25:54
  times. And around the summer I noticed
• 25:57
  that OK, the startup is no longer of an interest to me.
• 26:00
  Like, honestly, this was a one person thing.
• 26:03
  It still runs, it works. I haven't added any features for
• 26:05
  the past six months. I apologise to the customers out
• 26:08
  there. I just noticed that preparing
• 26:11
  that weekly conversation, learning throughout the whole
• 26:14
  week about the new stuff, talking about this evening, it
• 26:17
  became a show. It became a show with segments,
• 26:19
  with, with repeating customers, with guests, with, with
• 26:22
  entertainers is way more fun to me than everything else.
• 26:26
  And that's around the summer. People told me, like, hey,
• 26:29
  you're doing a live thing. Why don't you just, you know,
• 26:31
  put it on the podcast? Because I don't want to be live
• 26:33
  on Twitter when I listen to you. I want to drive to pick up my
• 26:36
  kids and, you know, on a Sunday listen.
• 26:38
  And I was like, I don't want to pass on the podcast.
• 26:40
  I don't like voice unlike this, unlike that.
• 26:42
  I was like, OK, at some point I just did it.
• 26:45
  And now now here we are. Basically there's a let me just
• 26:50
  like a say where we are exactly. There's a weekly show called
• 26:54
  Thursday I because we talk about AI every Thursday that I host
• 26:58
  with a bunch of friends of mine who are experts in different
• 27:01
  fields. We'll recover everything that's
• 27:03
  interesting in the world of a I for the past week mostly focused
• 27:07
  on open source and LM's. But also we talk about vision.
• 27:10
  We talk about new papers that are exciting, they'll come out
• 27:13
  and it's been, it's been a while to build like a wild time to
• 27:16
  build that community as well, because that's now a huge
• 27:19
  community that I did not expect to happen.
• 27:23
  Yeah. And I think that is how I
• 27:26
  stumbled across what you were doing.
• 27:27
  And it's incredible to see the dedication that you've had.
• 27:32
  I know we talked about it before like I hit record, we talked
• 27:35
  about the travel gear and how you've you almost are like, oh,
• 27:38
  I don't want to travel right now because I got to do this
• 27:40
  Thursday, every Thursday. And so breakdown like over the
• 27:44
  whatever the past, let's say month or two, what are some huge
• 27:48
  things, maybe papers that have stuck out to you or big news
• 27:53
  that has come across and you're like well this could lead to if
• 27:58
  we extrapolate this out a little bit, where can we lead to with
• 28:01
  this? Definitely.
• 28:03
  What what happens is, I think you've touched on a very
• 28:05
  interesting point is because I get to do this every week and
• 28:09
  because I only focus on what's new that week, I noticing that
• 28:14
  we talk about the new stuff, not necessarily only papers.
• 28:17
  I do try to be very practical about OK, which open source
• 28:20
  model just released that like which you know performance
• 28:24
  optimising thing happens that that makes this whole thing go
• 28:28
  better. And because I have this record,
• 28:30
  a weekly record, I'm able to look back during the summer and
• 28:33
  say, OK, we thought that this is gonna happen.
• 28:35
  And now we're here during winter of 2024, beginning of 2024,
• 28:40
  which of the things that that we kind of suggested might happen
• 28:43
  happens and several trends are showing up and it's very
• 28:47
  interesting to be able to like see which ones were predicted or
• 28:50
  not. And multimodality is to me one
• 28:54
  of the biggest ones. Agentry is another big one.
• 28:57
  And I think multimodality in 2024 is going to be remembered.
• 29:00
  The year of multimodality. Well, I mean by multimodality is
• 29:06
  LMS as a concept then naturally just spitting out kind of next
• 29:11
  token predictions that's how they're built.
• 29:13
  But the architecture behind them, Transformers architecture
• 29:16
  can be applied to multiple other things like envision like in the
• 29:20
  whisper for example ASR field, so NOP and more and more folks
• 29:27
  realise. And I think GD four started
• 29:30
  actually being the first announced multimodal model or a
• 29:34
  VM visual language model where they combine language and and
• 29:41
  I'll be understanding with visual understanding as well.
• 29:44
  And I've been on this almost every week since GD Four came
• 29:48
  out was like, hey, we need this also on the open source because
• 29:52
  that's how are we humans talk even on the thing example, I
• 29:57
  take a screenshot most I know that the most engaging tweets,
• 30:01
  whatever, they're like the visual ones with video, with the
• 30:03
  screenshot, not only text, but also because these models have a
• 30:07
  better world understanding, like what happens in our environment
• 30:10
  when they have like proverbial eyes, let's say.
• 30:13
  And it's harder to train those models, it's harder to evaluate
• 30:17
  those models. The evaluation sets for kind of
• 30:20
  the combined things are not necessarily exist.
• 30:22
  But there's a very interesting the Iliad Solskjaer who was a
• 30:26
  chief scientist and Open the eye actually don't know his status
• 30:29
  there right now. But he was like one of the Co
• 30:31
  founders He talked to with Jensen from NVIDIA about how if
• 30:37
  you build this model with the concept of eyes basically with
• 30:41
  visual understanding, you get to the same concepts that large
• 30:45
  language models get to like what the concept will read but
• 30:48
  quicker because the model can see and and and I think since
• 30:53
  that point I was very very interested in how visual
• 30:58
  multimodal models perform and multimodality.
• 31:01
  I I'll just like say this one thing and then we can talk about
• 31:04
  this multimodality often gets referred to as OK, there's only
• 31:09
  two modalities, there's text and there's video or text and images
• 31:13
  for example. Video is a whole different
• 31:15
  beast. We can talk, but it's so much
• 31:18
  more than this. Meta has released the paper and
• 31:21
  I think it's not a full model, but like a an example model that
• 31:26
  has six modalities in it. Next images or video.
• 31:31
  They also have Imus which is motion units, so they can
• 31:34
  understand from video motion. So if it trains comes in the
• 31:37
  model natively understands that you know distance and and motion
• 31:40
  units etcetera. And if you think about this, you
• 31:42
  want to embody AI at some point in in, in robots, they walk
• 31:46
  around, they have to understand motion as well.
• 31:47
  It's not like they're gonna take a picture every second.
• 31:50
  OK? Between this picture and that,
• 31:51
  picture will change. And there's example of
• 31:54
  multimodality to go even crazier.
• 31:56
  Alright, let's be quick. Break from the episode and talk
• 31:59
  to you for a second about our sponsors of this show, One DB,
• 32:04
  and what they've done with their newest LLM course.
• 32:07
  You can elevate your machine learning skills with weights and
• 32:10
  biases. Free course on training and fine
• 32:12
  tuning large language models looms as they're called out in
• 32:16
  the wild. You can learn from industry
• 32:19
  experts like Jonathan Frankel from good old Mosaic, ML friend
• 32:23
  of the Pod, We were Yang and Mark Serafin, and while you're
• 32:29
  doing it, learning from them, you can deep dive into the LM
• 32:34
  fundamentals, advanced training strategies like Laura and prompt
• 32:41
  tuning, all while getting your hands dirty and having some
• 32:44
  hands on applications. This engaging course offers 37
• 32:49
  lessons and over 4 hours of videos and a certificate upon
• 32:52
  completion. Enrolled now to master well EMS
• 32:56
  and advanced techniques. Check out the link in the
• 32:59
  description to start your journey and let's get back into
• 33:02
  the pot I had. The folks from a prophetic
• 33:05
  prophetic I think I believe they called they have a multimodal
• 33:09
  transformer architecture that the modalities are not text or
• 33:12
  vision at all. The modalities are inputs from
• 33:15
  an EEG signal and F MRI signals right?
• 33:18
  So functional magnetic resonance brain brain scan signals
• 33:20
  basically and EEG Electro something and they use this to
• 33:26
  turn the output of their transformer architecture into
• 33:30
  locations in your brain where super highly precise ultrasound
• 33:35
  can hit to invoke a specific state in your brain for lucid
• 33:38
  dreaming. O that's a dream about you
• 33:39
  completely. That's on a I system that takes
• 33:42
  in brain waves and outputs directions, right.
• 33:45
  And that that that type of multimodality is also like very
• 33:47
  interesting to me. So definitely that's an area of
• 33:50
  extreme excitement in 2024, so the.
• 33:53
  Fascinating thing about multimodality, besides claims of
• 33:59
  invoked lucid dreaming. Because who doesn't want the
• 34:02
  easy way of lucid dreaming? I don't know if you've tried.
• 34:05
  Have you ever tried lucid dreaming?
• 34:07
  Yes, I've I've been. Pursuing lucid dreaming most of
• 34:09
  my life. I still do 12345 and try to poke
• 34:12
  a finger through my hand to see if I'm in the reality or not.
• 34:14
  I do reality cheques every day. Yes.
• 34:16
  Is that the one so? I I tried to turn off and on
• 34:20
  lights because that's that's another one that you could do.
• 34:23
  So I'm guessing then if you do that, you've been through a
• 34:27
  lucid dream. Just I think I got like a once
• 34:31
  or twice. Even though I've been pursuing,
• 34:33
  I'm not really a great lucid dreamer.
• 34:37
  Well, there you go. Prophetic is perfect for you,
• 34:39
  right? That is.
• 34:40
  Yeah, that's why that's. Why I trust technology to get me
• 34:43
  there eventually. Yeah.
• 34:46
  And but on the other hand, when it comes to multi modality, one
• 34:49
  thing that I think about is multimodal rags and the
• 34:54
  difficulties there and all the challenges that need to be like
• 34:58
  there's so many hurdles. I feel like when it comes to how
• 35:01
  we do those, I'm by no means an expert in the field at all when
• 35:06
  it comes to multimodal rags. But I have heard people talk
• 35:09
  about why what we have the current tooling and what is out
• 35:15
  there right now is not U to standards.
• 35:17
  Especially if you're getting into video or if you're getting
• 35:20
  into audio and you can think of I guess 11 way that you can
• 35:25
  think about it is yeah, looking through long movies and being
• 35:29
  able to grab context not from just the transcript but also
• 35:34
  what is in the scene in the movie, right.
• 35:36
  But that's really hard if you are taking a screenshot every
• 35:42
  frame and then you're actually using that for your rag.
• 35:47
  And so there are those pieces. I do wonder like how that will
• 35:53
  change over, as you're saying, hey, multimodal is 2024 that
• 35:57
  it's, it's the year for multimodal and every aspect, not
• 36:01
  just lucid dreaming, but in multimodal rags too.
• 36:04
  That was definitely an. Extreme example of showcasing
• 36:07
  how modalities are not only image and text as as most of the
• 36:12
  people will get to think about multi modalities F MRI inputs as
• 36:16
  a modality in addition to text for example or facial structure
• 36:19
  inputs as in addition to you know webcam feed in addition to
• 36:23
  what you say and intonation for example also could be
• 36:26
  modalities. I just want to highlight that
• 36:29
  multimodality as the most practical one is probably the
• 36:32
  imagery in text, but what GBT has for us right you could
• 36:36
  consider audio and the modality as well but honestly until they
• 36:39
  detect intonations in my voice until that point happens and it
• 36:42
  doesn't happen, I haven't seen that happen yet.
• 36:45
  Audio modality is just text modality because of ASR and LP
• 36:49
  just translates everything I say into text.
• 36:51
  So it's just text and and and and imagery, not even video for
• 36:54
  most of it. I think the next jump like you
• 36:56
  said is OK how do you understand from frame to frame because we
• 37:00
  humans, we don't experience you know 24 frames a second.
• 37:02
  We experience. I think our refresh rate is
• 37:04
  higher than that and we actually.
• 37:07
  So there's there's two things here in terms of difficulties
• 37:10
  right. One of them is the actual scene
• 37:12
  to scene understanding and scene boundaries and cuts and like all
• 37:15
  these different things and streaming audio enough hardware
• 37:18
  to be able to actually get those frames.
• 37:21
  So everybody knows that OK Transformers have this problem
• 37:25
  called quadratic attention problem like they're incredible
• 37:28
  because of their attention. Attention is only need is the
• 37:30
  paper that came out with all this.
• 37:32
  However the quadratic scaling with this attention, the more
• 37:34
  tokens you give the harder it is to to to run inference the the
• 37:39
  more imagery you add the the Super like way quicker that the
• 37:43
  tension grows right. And if you consider that every
• 37:47
  image from a video is also like you know 24 frames a second, you
• 37:51
  start getting into serious let's say MLA's problem just just
• 37:56
  before they even transforms problem of how do you even store
• 37:59
  all these things. There's a bunch of stuff that
• 38:01
  like video included. So shout out to folks at 12 Labs
• 38:04
  that came to one of our AI thinkers.
• 38:06
  Meet up in Denver. One of them is in Denver and
• 38:09
  showed us kind of the future they're working on video
• 38:11
  embedding and the video foundational model that
• 38:15
  understands video specifically. So like they have a video
• 38:18
  modality built into their model and video embeddings built in.
• 38:21
  So hopefully folks check this out.
• 38:23
  Definitely very interesting. And the other problem that
• 38:26
  you've touched on, very, very interesting problem even in text
• 38:30
  right now is evaluations. How do we know that the model
• 38:33
  that was trained is actually better than the model trained
• 38:35
  yesterday? It's literally a competition now
• 38:38
  like a competitor's game of releasing a better model and
• 38:40
  it's better by like 1.7% on the MU patient stack and here's like
• 38:45
  where weights and biases comes in.
• 38:47
  The company that eventually hired me because of the podcast
• 38:49
  so now Thursday is part of its summarises evaluations as part
• 38:53
  of the building of the of the model is very important.
• 38:57
  So there's like the open evaluation stack on hugging face
• 39:01
  and folks follow that. There's like a very specific,
• 39:03
  you know, MU and GSM, 4K and AK and math and all these things.
• 39:07
  But also the the problem you touch on is this gets maybe an
• 39:13
  order of magnitude more complex into multimodality because how
• 39:17
  do you then evaluate, how do you evaluate text and imagery, How
• 39:20
  do you evaluate in if we get the video, how do you, how do you
• 39:23
  run the valuations, what the person saw and then said and if
• 39:27
  there's any donation In addition to this, if the model reacts
• 39:30
  differently based on if I'm angry or sad, like how do you
• 39:32
  evaluate all this? And I think that it's fine for a
• 39:36
  technology to change, evolve and then also the supporting tools
• 39:39
  like the MLR tools, the tracking tools like businesses, they have
• 39:43
  to catch up to technology because it moves, it doesn't
• 39:45
  wait for anyone. But definitely it's gonna be
• 39:47
  challenging the multimodal evaluation sets, multimodal
• 39:50
  understanding of OK, how do we actually know what's good or
• 39:54
  not. It's funny you mentioned.
• 39:55
  That because we did a survey on evaluation last year, at the end
• 40:01
  of last year in the community and one thing that is.
• 40:05
  Absolutely clear from that survey is when it comes to
• 40:08
  evaluation, you kind of have ways that you evaluate depending
• 40:15
  on your use case. So if you're doing translation,
• 40:17
  you use one evaluation set. If you're doing summarization,
• 40:20
  you use another evaluation set. And that's basically the
• 40:23
  standard right now, which doesn't feel like it's the most
• 40:28
  robust at all. It's like this is a little bit
• 40:30
  janky. It's it's very interesting.
• 40:33
  Because that keeps happening also in the open source LM world
• 40:38
  the right track very closely week to week so, so I'm very
• 40:41
  fortunate to host from week to week folks who build or fine
• 40:45
  tune the open source all models and that's kind of the the, the
• 40:48
  bread and butter of Thursday night.
• 40:50
  We talk about kind of the progresses in LMS and a bunch of
• 40:54
  those evaluations like you said are task specific or at least
• 40:57
  like you know the instructions specific.
• 40:59
  And what happens is we are building generally generative
• 41:03
  models like the the you know most of the people's here is a I
• 41:07
  artificial general intelligence whatever that means.
• 41:09
  Your definitions aside the whole the the, the main concept there
• 41:13
  is generality the the ability of this model to do this thing and
• 41:16
  that thing and that thing. So often we see in the open
• 41:19
  source world like this model beach GT, GT4GT3 on this
• 41:24
  specific task that model be GT4 and this specific task and
• 41:27
  that's incredible. But GT4 is like this general
• 41:30
  model that can do all of these tasks and still perform like at
• 41:33
  the top but from the evaluation side all these creations are
• 41:37
  also on the specific Gen like a specific task evaluation said
• 41:41
  word error rate you know Yeah captioning for video and imagery
• 41:46
  etcetera. And the problem is we also need
• 41:49
  general evaluations. We need, we need we need
• 41:52
  evaluations that tell us like how overall the models are
• 41:55
  doing. I think a lot of those are now
• 41:57
  for example, average of all the scores, average of all the
• 42:00
  specific task and valuations. And one shout out I think is to
• 42:03
  LMS. If folks don't are not familiar with LMC's, they're the
• 42:07
  folks who tuned fine-tuned Vicuna.
• 42:11
  I either Vicuna and Alpaca always forget which one but they
• 42:14
  also since then raced like a bunch of very interesting
• 42:16
  things. One of them is their
• 42:19
  leaderboard, the LMC's leaderboard where they actually
• 42:23
  measure an ELO score or rank based on conversations that
• 42:26
  people have with these models blindly with like 2 models face
• 42:30
  to face. People ask the prompt and I
• 42:32
  suggest everybody here goes and tries this out and contributes
• 42:34
  to this data set and they just humans just prefer OK, this
• 42:37
  answer is better. That answer is better.
• 42:39
  And I think one of the coolest evaluations that came out of
• 42:43
  this world of the digital crazy world that we live in is the
• 42:46
  vibe check, like the Internet vibe check.
• 42:48
  So you go to LMS and there's like the human vibe check of
• 42:50
  what what they thought was more helpful.
• 42:52
  There's not there's not a number there of they just put the
• 42:56
  number there. But also if you go to Reddit,
• 42:58
  there's like a whole subreddit of people who like try things
• 43:01
  for specific purposes. Some of them are building
• 43:04
  rifles, but they just like, say, OK, this model performs better
• 43:07
  in role play than this model. And so we get this like very
• 43:10
  weird, like life checky, understanding that the Mistral,
• 43:13
  for example, is performing better than ye from from 0,
• 43:19
  whatever they're called. And so we get this, like, life
• 43:21
  trick, but nobody knows really why.
• 43:23
  And that's kind of incredible to me as well.
• 43:26
  Yeah. And the the other.
• 43:27
  Piece that I think is interesting to note when it
• 43:29
  comes to evaluation, besides the vibes and how that is now a
• 43:34
  thing is that you have to look at evaluating in so many
• 43:39
  different ways, right? You're are you evaluating the
• 43:42
  training of the model, are you evaluating the system or are you
• 43:47
  evaluating the output, right. And then if you're using Rags,
• 43:51
  are you evaluating the retriever and how you're evaluating all
• 43:56
  these things? And each piece of the system or
• 43:59
  the greater system all plays into it.
• 44:02
  So it's not just like, Ohh, let's look at the model for this
• 44:09
  whatever. I'm asking it to write me a poem
• 44:12
  in the style of Bob Dylan. It's like, what is the use case?
• 44:17
  What kind of model? The size of it, the speed of it,
• 44:21
  the accuracy, All of that can come in, like how many tokens it
• 44:25
  spits out, How well does it listen to you?
• 44:28
  What's the toxicity? All these different pieces need
• 44:31
  to play a part, especially if you're trying to use something
• 44:34
  in production, right? It's not just like, oh, let's go
• 44:38
  and see what this model can do. And that was another piece that
• 44:43
  jumped out at me because funny enough, like timing is perfect
• 44:46
  on this because I'm writing the takeaways from the survey.
• 44:49
  Even though we did the survey three months ago, now four
• 44:52
  months ago, I'm writing the Takeaway blog and justice,
• 44:56
  looking through all these answers and there I think like
• 44:58
  over 120, thirty people filled it out.
• 45:02
  And so you get to see what people are saying and how
• 45:05
  they're evaluating and what is important to them.
• 45:09
  Basically, everybody's like truthfulness, hallucinations.
• 45:13
  I want to make sure to mitigate that as much as possible.
• 45:17
  So accuracy, hallucinations are big ones.
• 45:20
  And then cost is another huge one, like evaluating the models
• 45:25
  based on cost. And that's where the open source
• 45:27
  stuff comes in. And it's huge.
• 45:30
  And so looking at all that and the reason I forgot to mention,
• 45:34
  the reason I'm trying to do this is because of the fact that
• 45:37
  we're going to run this survey again and see like what's
• 45:40
  changed. It's only been three or four
• 45:41
  months, but a whole lot has changed in the evaluation work,
• 45:45
  in the evaluation space, right. And now we know much more.
• 45:49
  And if people are doing multimodal rags, how are they
• 45:52
  evaluating it? Like you said, there's
• 45:54
  incredible details that you need to be thinking about.
• 45:57
  And how do you know that The model says OK, or the output
• 46:03
  says, yeah, there was a red dress in this scene of this
• 46:06
  movie, and it extrapolates out from that.
• 46:09
  But like, how can you double check that?
• 46:11
  That's really hard. And so I know that there's a lot
• 46:14
  of things happening as far as, and there's a great paper, I
• 46:17
  don't know if you've read this one, on mitigating
• 46:19
  hallucinations. I'll throw it in the show notes
• 46:22
  and send it over to you, too. But it just talks about all the
• 46:25
  ways up until now that people have tried to make sure that the
• 46:30
  models don't hallucinate. And so it's not exactly
• 46:33
  evaluation, but it's like, hey, what can we do so that we can be
• 46:37
  a little bit more confident things are working and then how
• 46:41
  do you? Actually evaluate the that you
• 46:43
  were able to mitigate 100 nations.
• 46:45
  Yeah only had some fresh learnings from this from
• 46:48
  actually micro and western versus the growth and melting.
• 46:51
  We I I literally went live yesterday with part of the team
• 46:55
  and then the second part is is the next Monday.
• 46:57
  I can send you the link to have to show that as well where our
• 47:00
  team was tasked to build because you know we build the tools but
• 47:03
  we also need to actually experience the rag kind of see,
• 47:08
  see what new, what change, like how do people evaluate.
• 47:10
  Like we also need to listen and build and build intuition around
• 47:15
  these tools ourselves just to know what is the best product to
• 47:19
  deliver to customers. And our team back in December
• 47:22
  got a week off different other tasks to just work on build week
• 47:27
  projects or hack week projects that we do.
• 47:28
  We said all of them around LM use cases, all of them around
• 47:31
  like hey, let's build an actual thing for the company and just
• 47:35
  to see what are the pain points, what are the pain points to
• 47:38
  solve. And so here is kind of the
• 47:40
  outline. I'm going to give, like some
• 47:41
  alpha to your listeners. What happened in those
• 47:43
  conversations? So we had one person who built a
• 47:49
  a rag bought basically a retrieval called want bot where
• 47:52
  if folks go to a Discord or Slack and I think now it's part
• 47:55
  of the GBD store as well. They can just talk do a weights
• 48:00
  and biases document documentation and all the health
• 48:03
  and the code and everything and this is about will give them
• 48:05
  answers that that's been up on production for a while and
• 48:08
  that's a way more mature than the the the weak build but that
• 48:12
  we were able to do right. And so he he joined this build
• 48:16
  we can he kept enhancing the performance everybody else
• 48:19
  started pretty much from scratch.
• 48:20
  Here's the learnings. It's fairly easy to build
• 48:24
  something that looks like ohh yeah download all my data from
• 48:27
  documentation. I build like a retrieval button
• 48:28
  on top of this I did chunking with long chain or or lava index
• 48:33
  and like you know embedded this with this, embedding that,
• 48:35
  embedding whatever and see you ask it the thing and it works.
• 48:39
  It's fairly easy to get to that point.
• 48:42
  The the here's a few very interestingly problematic
• 48:45
  points. First of all you have to figure
• 48:47
  out what use case you're solving for actually, whether or not
• 48:50
  you'll have users that will send you evaluations and say hey this
• 48:54
  works and doesn't work. Second of all, most people don't
• 48:57
  build in the the even these things.
• 48:59
  Not even to mention like storing everything to be able to run
• 49:02
  this. Again, folks should have
• 49:04
  considered like build this and if it's a slack bot, build the
• 49:07
  buttons, whatever store that information.
• 49:09
  It's very very crucial for you to ask your users whether
• 49:13
  something worked or not. Third is educate your users into
• 49:17
  actually using those buttons into actually saying what you
• 49:20
  need for which use case and whether or not it worked because
• 49:24
  that would be your data set going forward and then once you
• 49:27
  have this bar in production. So I I just literally I'm
• 49:30
  comparing the the mature bot that people actually work and we
• 49:32
  have users for and kind of the building projects that were cool
• 49:35
  but here's kind of their next steps, figuring out what happens
• 49:40
  with the data set when it changes.
• 49:42
  So everybody talks about RAG, but not a lot of people talk
• 49:45
  about ingestion pipeline, what happens when your documentation
• 49:47
  changes. It changes often like SDK
• 49:49
  change. We all know the problem with
• 49:51
  LM's where there's a cut off line or knowledge and then
• 49:54
  doesn't know everything and above it.
• 49:56
  So you have to augment this with RAG.
• 49:57
  So the mature bot, the one bot has and ingestion pipeline that
• 50:02
  runs automatically every week and now we're getting into MLA,
• 50:05
  right? Like you have to set this out.
• 50:06
  The scrolls involved, there's a bunch of stuff and he like the
• 50:10
  the person barat incredible. He gets a report of how much
• 50:15
  change which ingestion because we we have the documentation, we
• 50:18
  have translations, we have GitHub code, we have a bunch of
• 50:21
  stuff that like this both learns and the more the more it learns
• 50:24
  the the kind of the more important it is to make sure
• 50:27
  that you keep up. Because it's not only about
• 50:29
  those nations anymore. At some point it's also about
• 50:31
  not giving the right answer just because you didn't update your
• 50:35
  data set against you know what's actually running.
• 50:38
  Especially when there's like multiple people in the
• 50:39
  organisation. So ingestion pipeline is very
• 50:41
  interesting because nobody builds an ingestion pipeline
• 50:43
  when they just like hack on Lamma index.
• 50:46
  And the other side of this is automatic evaluations.
• 50:48
  Both this thing and so very interesting is that suddenly we
• 50:53
  use LM to evaluate their lens. So we after data set has been
• 50:57
  built, after you get your users to do thumbs up, thumbs down.
• 51:00
  After getting enough of examples of what your users would want,
• 51:04
  you now have enough to actually start building an evaluation
• 51:08
  pipeline that's automatic, that keeps running.
• 51:11
  And that's incredible because that unlocks this next thing.
• 51:13
  We're like iterations are suddenly possible.
• 51:16
  Water iterations, for example. You can switch from an open GT4
• 51:19
  model, for example, to a local Mistral model and then see your
• 51:23
  evaluations are running. Is it better?
• 51:25
  Is it worse? Is it significantly better?
• 51:27
  Significantly worse? One evaluation has to be about
• 51:31
  speed, inference speed. Because the more prompts you
• 51:35
  get, the more raging there you get.
• 51:36
  At some point you're user is just sitting and waiting and and
• 51:40
  it's really hard to measure that.
• 51:41
  But for many users that's a pain point where they would stop
• 51:43
  using this completely, they would just go and look it up
• 51:46
  right. Most accommodation places have
• 51:48
  surged so like at some point is diminishing returns.
• 51:51
  So also evaluate your speed to to to user inference.
• 51:54
  And I think another thing that is possible to evaluate
• 51:58
  automatically is embeddings. Not many people consider the
• 52:01
  embeddings settled themselves. They're not fine tuning their
• 52:04
  beddings model, for example, which is possible.
• 52:06
  They're not doing ranking, for example, which is also possible
• 52:09
  in significantly better at at some cases.
• 52:11
  And so now once you have automatic evaluations running,
• 52:15
  you're we are able to now switch embedding systems.
• 52:18
  So open the eye. A week ago, two weeks ago came
• 52:20
  up with a completely revamped embeddings.
• 52:23
  They moved from other 002 to text embedding light or small.
• 52:27
  And text embedding large and large is like the significantly
• 52:30
  performance. Small is very, very performant.
• 52:32
  Both of them are cheaper. I love how everything gets
• 52:34
  cheaper but but then how do you know, do you just switch and
• 52:37
  hold the open? The I gave you like the best
• 52:39
  tool. So we now have the automatic
• 52:41
  evaluations on the more mature bot that now we can actually
• 52:44
  replace run for a week and say, OK, yeah, that's actually kind
• 52:47
  of, you know, faster and better, less hostile nations.
• 52:51
  So cool. And it's there's a great point
• 52:54
  that you just kind of skipped over.
• 52:55
  But if you're not automatically ingesting, then it could be the
• 53:00
  right answer that you're getting for the documentation.
• 53:04
  But it could be the right answer from 2 weeks ago and since
• 53:08
  things have been updated and now it's not the right answer, but
• 53:11
  it was, but now it's not. So evaluating that.
• 53:15
  It's also huge that you're talking about these embedding
• 53:18
  models too. And those are there's a lot of
• 53:20
  cool stuff happening out there in the embedding model space.
• 53:23
  There's a lot of open source embedding models that you can
• 53:26
  use. There's a lot of justice
• 53:28
  different. I want to.
• 53:29
  I want to shout. Out a few, if you don't mind,
• 53:31
  because just recently we go This is what I do, right?
• 53:34
  So now we're moving into the Thursday I podcast because I
• 53:36
  specifically have a thing about embeddings.
• 53:38
  Our friends from Normal AI, the folks behind Atlas, if you're
• 53:41
  familiar with Atlas, is a way to visualise embedding you're
• 53:44
  talking at. The next conference?
• 53:46
  Oh yeah, awesome. So so.
• 53:47
  Nomic folks released a fully open source Anomic embedding end
• 53:51
  to end, including the training set and everything, and that's
• 53:55
  incredibly awesome. The more open source things are,
• 53:57
  the more you're able to take on the training and the fine tuning
• 54:02
  of embedding model, something that people don't often
• 54:04
  consider. People consider fine tuning the
• 54:06
  GP, for example LM, but you can fine tune embeddings on specific
• 54:10
  tasks. For example.
• 54:11
  Another shout out is Gina Gina. I released the Gina Embeddings
• 54:15
  V2 and recently they released a a fine tune on code for example.
• 54:18
  And when you fine tune and beddings on code you get better
• 54:21
  code examples rather than text examples.
• 54:24
  Because if you consider what embeddings are, usually they
• 54:26
  came from the world of just text embedding and understanding, not
• 54:30
  necessarily code code it's a different thing.
• 54:32
  But now we're getting into the world, we're like Microsoft
• 54:34
  trained M5 which is a is embedding model on top of
• 54:38
  Mistral which makes no logical sense besides the the
• 54:42
  theoretical fun for academics or academic sounds.
• 54:46
  But like it's super slow to run. You have to run like in 14
• 54:50
  gigabyte vram whatever this mistrial model.
• 54:52
  But if you consider what embeddings are, they have a they
• 54:54
  have a brain in there the the understands the concept of what
• 54:58
  you're trying to embed and Mr is like as this very open source
• 55:03
  very nice brain. So they used that to train and
• 55:05
  that obviously outperformed everything else but again on the
• 55:09
  problem of evaluations outperforms everything else.
• 55:12
  On benchmarks we have to consider how much like hardware
• 55:14
  and RAM and and speed to embed. You have to evaluate those
• 55:19
  things again. So embedding this like
• 55:21
  incredibly vast area that people are now starting to get to
• 55:27
  towards, hey, we can fine tune for this specific task, we can
• 55:30
  find them for Q&A like literally you can fine tune embedding
• 55:33
  model for specific UN retrieval tasks, you can find them for
• 55:36
  code tasks. So definitely people sleeping on
• 55:38
  this, but it's coming like this area is now like opening up.
• 55:42
  If we're talking about trends for 2024, thinking better is
• 55:45
  another interesting trend. Yeah, and it's.
• 55:48
  Just with including just one. Sorry, sorry, just one less like
• 55:52
  sentence including multimodal embeddings as well, because
• 55:55
  embeddings now are not no longer about just text.
• 55:57
  You can embed the images, you can embed videos, you can embed
• 56:00
  concepts, you can embed users preferences, you can embed like
• 56:03
  a bunch of stuff. And all of these you have to
• 56:06
  embed in order to retrieve from, right?
• 56:08
  So like, yeah, the multimodal embeddings are also up and
• 56:10
  coming and a very big deal as well.
• 56:13
  Yeah, I was. With the bird models out there,
• 56:16
  you can basically fine tune and embedding model for like 200 or
• 56:20
  500 bucks and if you know what you're doing, that's really
• 56:25
  cheap and that gets you off to a really good start, right?
• 56:29
  And hopefully if you're doing that, then later on down the
• 56:34
  line, like you don't have to go and fine tune other models that
• 56:39
  much. Maybe some off the shelf model
• 56:41
  will work good enough for you in what you're trying to do and
• 56:45
  what you're talking about here. Everything that you're saying
• 56:47
  just reminds me of how much of A trade off each piece of this is,
• 56:52
  right? Like if you're evaluating each
• 56:54
  one of these and you're trying to figure out what you want to
• 56:58
  do, you're you're basically looking at like, OK, what are
• 57:02
  the pillars of what I'm trying to do?
• 57:04
  Do I want the most accurate thing?
• 57:06
  Do I want the fastest thing? Can they be accurate and fast
• 57:10
  together? Like, how much longer is it
• 57:12
  going to take me to build that? All of those different design
• 57:16
  decisions come into play and I'm sure you think about that quite
• 57:19
  a bit, just with the product background that you have and how
• 57:23
  that all will look at the system.
• 57:26
  Again, it goes back to that looking at it as a full system,
• 57:30
  should we add the thumbs up, Thumbs down?
• 57:32
  Should we ask users to actually give us robust feedback?
• 57:36
  And how do we make sure that they give us robust feedback?
• 57:39
  What does that even look like? And then what do we do with that
• 57:41
  robust feedback once we have it? And so all of these different
• 57:45
  pieces get you gotta like, kick him around in your head before
• 57:50
  you actually have something that you can confidently be like,
• 57:53
  this is about as good as it can be right now.
• 57:57
  Right? 1.
• 57:58
  Consideration to take into account is how it looks for the
• 58:03
  user and who is the user, right? So obviously looms came in and
• 58:07
  chat bots are like the number one thing.
• 58:09
  Like chat bots, chat bots by almost necessity mean real time
• 58:15
  interaction with the user who waits.
• 58:17
  And because everybody get used to the best chat bots around GP4
• 58:20
  and now Gemini Ultra is going to release very very soon.
• 58:24
  Those are huge companies they run wasn't looking extreme
• 58:27
  extreme scale compute where you get instance like a response
• 58:31
  like super super quick. So your users may get used to
• 58:34
  something like this and here you are trying to run your own like
• 58:37
  model open source model on production let's say and so you
• 58:41
  have to take into consideration the use case and I'll give you
• 58:44
  another example from from the build week that we did and by
• 58:46
  the way folks they're listening to work at companies.
• 58:49
  This is incredibly important thing to do for your own
• 58:52
  developers as well. Just give them like a week to
• 58:54
  play and try out things. The learning it will get from
• 58:58
  not building two production necessarily, but just like going
• 59:01
  to hackathon or playing around, the learnings that you'll get
• 59:03
  will carry you through actual production deployments
• 59:06
  significantly. So here's another example.
• 59:08
  Thomas built a translation but, and that's on GitHub, so it it
• 59:12
  sits there and get high actions, waits for a new pull request to
• 59:19
  come in with a new documentation update or a change, etcetera,
• 59:21
  and just like uses LM to translate that.
• 59:24
  So if you think about OK, one example is a chatbot that rides
• 59:28
  a copy for ads. Somebody in marketing needs that
• 59:31
  answer right now. Otherwise they're going to GP4
• 59:33
  and just shove everything they have in the context and just go
• 59:36
  there And basically the diminishing returns will will
• 59:39
  play in GTA4's favour here. Documentation essentially our
• 59:45
  thing that if you don't do automatically user will have to
• 59:48
  go and copy. And remember like automation
• 59:50
  here is a benefit but documentation doesn't have to be
• 59:53
  real time. The user does not sit in front
• 59:56
  of the answer and waits. And So what this allows you to
• 59:59
  do is now use the trade off calculation that you just
• 1:00:02
  mentioned in favour of a higher quality.
• 1:00:05
  Maybe you run the translation once and you run this again.
• 1:00:09
  Maybe you do two or three problems.
• 1:00:10
  Maybe you do different like thing because the user doesn't
• 1:00:13
  necessarily wait. Then suddenly your condition
• 1:00:16
  changes. Your cost calculation could
• 1:00:17
  change because of that. Maybe you go for a cheaper model
• 1:00:20
  that's lower your your time to inference.
• 1:00:23
  Maybe response time is not the only thing.
• 1:00:26
  Maybe also evaluations are different in that case as well
• 1:00:28
  because like it's different. So I definitely encourage folks
• 1:00:31
  to consider the user and how this use case is getting
• 1:00:36
  deployed at the end to definitely rollback and say OK,
• 1:00:40
  what are we optimising for? We're optimising for the
• 1:00:41
  inference speed. We're optimising for alternation
• 1:00:44
  removal because those like you said there's a bunch of papers
• 1:00:47
  retrieval or rerank your or all these like tools three of
• 1:00:52
  thought, like all these things that take three of the chain of
• 1:00:54
  thought, they take more you know money in in form of more tokens
• 1:00:58
  and more prompts that you run, but they can lower significantly
• 1:01:03
  hallucinations as well. So yeah, that's one
• 1:01:05
  consideration also to take into account in in the back and forth
• 1:01:10
  between features. Yeah, you mentioned the.
• 1:01:12
  LLM as a judge type thing and that is something that I feel
• 1:01:16
  like is super hot these days, but it is is another trade off
• 1:01:20
  because now you're making another call, you're making
• 1:01:23
  another API call and you got to pay for that, right?
• 1:01:27
  But the the other thing that I was going to say about all of
• 1:01:31
  these different tradeoffs that you're looking at and what the
• 1:01:34
  end user experience is, I like to talk about something that
• 1:01:42
  Sahil. Jane, I think.
• 1:01:45
  Spoke about from he was a he's a product engineer at u.com and he
• 1:01:51
  said that for them it's important to always be as fast
• 1:01:54
  as possible, right? But you can't always be fast
• 1:01:58
  because these models just aren't always fast.
• 1:02:01
  And So what they've been trying to do when it comes to
• 1:02:04
  optimising for speed is he talked about the two metrics
• 1:02:09
  that they would look at. One is the actual time it takes
• 1:02:12
  to get the answer, and then the other is the perceived time that
• 1:02:16
  it takes O. What they're trying to do is
• 1:02:18
  build in these little tricks while you're waiting to make the
• 1:02:23
  perceived time go down. And so maybe that's you take a
• 1:02:28
  quiz while you're waiting or you get some kind of an example of
• 1:02:31
  this. That I built, if you want to
• 1:02:32
  hear what you do. Yeah, yeah, of course I I built.
• 1:02:36
  A build OK, I created the GPT custom GT on the GPD store and
• 1:02:41
  it went kind of viral as well. It's called visual weather GPT.
• 1:02:44
  So basically you provide like Denver or whatever Seattle and
• 1:02:48
  it will generate A dowdy image. It will go to Bing, browse the
• 1:02:53
  weather for that area and then generate the daily image using a
• 1:02:57
  bunch of like prompting and start prompting for that weather
• 1:03:00
  that day. So that's pretty cool, right?
• 1:03:01
  You get like a super custom art for your thing and that went
• 1:03:04
  viral and that was singing there and I was like, yo, this is
• 1:03:08
  slow. Dolly is kind of slow.
• 1:03:09
  Browsing is kind of slow. Look the user is waiting.
• 1:03:12
  What what's fast? Speeding tokens look super fast.
• 1:03:14
  So I came up with a thing where like it spits out the bone about
• 1:03:17
  the weather and like it spits out like 4 paragraphs of the
• 1:03:20
  poem and then it starts generating the image.
• 1:03:22
  So by the time the image starts everything, the user sits there
• 1:03:24
  and reads the poem. By the time to read the poem,
• 1:03:26
  boom, the image generated. Nice.
• 1:03:28
  Nothing. Changing before if anything I
• 1:03:30
  made it slower, but essentially for the user like the experience
• 1:03:32
  is better. So definitely agree.
• 1:03:34
  Like perceived. It's like that airport story.
• 1:03:38
  I don't know if you know that like people complained to the
• 1:03:41
  way the long for luggage and so the airport routed the people
• 1:03:44
  around like a long way to walk for 10 minutes and you know they
• 1:03:48
  they walk basically encircled 10 minutes.
• 1:03:50
  By the time we get to the luggage thing, luggage was
• 1:03:51
  there, nobody complained anymore.
• 1:03:53
  So it's kind of like but these are.
• 1:03:56
  New challenges, right? The I think about one of the
• 1:04:00
  stories I heard with Instagram when it was first coming U and
• 1:04:04
  how they would make the uploading time of your photos be
• 1:04:07
  so fast and what they would do is you.
• 1:04:12
  Take your photo. Put it on the app, upload it,
• 1:04:15
  but you wouldn't actually host it yet and you would have to
• 1:04:19
  write the caption. And so while you were writing
• 1:04:22
  the caption, they were already uploading the photo, which made
• 1:04:25
  it seem like the everything was so much faster because then when
• 1:04:29
  you actually did post it, it was either already posting in
• 1:04:36
  milliseconds or it was like a second that you had to wait.
• 1:04:40
  And so it just makes me think about how we can look to the
• 1:04:45
  past for inspiration on this. It's not like it's necessarily a
• 1:04:49
  the first time that we've encountered challenges where
• 1:04:52
  like it's very laggy and it takes a while to do something
• 1:04:55
  because the tech isn't quite there yet.
• 1:04:58
  So I would say if. Anything this is maybe a sign of
• 1:05:01
  maturity of this field that we kind of all stumbled on.
• 1:05:04
  Many people did ML for a while and then charged with came out a
• 1:05:07
  year and two months ago and like completely upended everything.
• 1:05:10
  So this is the maturity of building these tools in the
• 1:05:13
  production that we're getting to the same user centric kind of
• 1:05:17
  issues, UX issues that have been there for a while even in these
• 1:05:20
  tools. And I think that's a good thing
• 1:05:22
  that's like a generally like a step forward in the direction.
• 1:05:24
  We're like, OK, there are people who have been building these
• 1:05:27
  type of solutions for these type of problems.
• 1:05:29
  It doesn't matter whether the backbones Allam or the backbone
• 1:05:32
  is something else. I think we're getting towards
• 1:05:34
  like, OK, now we're no longer in the play phase.
• 1:05:37
  We're now in the hey, let's go to production.
• 1:05:39
  In production there are users and that phase is is very
• 1:05:42
  interesting to me as well. The Chief dude.
• 1:05:44
  So talk to me. About another what I would
• 1:05:48
  consider topics that tends to blow U especially on Twitter
• 1:05:53
  which is agents and how much experience do you have with
• 1:05:57
  agents? What are your feelings about
• 1:05:59
  them and just give me the overall vibe check that you have
• 1:06:04
  on them so. A few things here from personal
• 1:06:08
  learnings, but also from talking with people who built a bunch of
• 1:06:11
  very cool cool things. O maybe people remember JT
• 1:06:15
  blowing U sorry excuse me. Maybe people remember Auto GT
• 1:06:20
  blowing up on GitHub, becoming the fastest growing GitHub
• 1:06:23
  project in stars, etcetera. I met with the with the Auto GPT
• 1:06:27
  team and during one of the conferences there.
• 1:06:30
  There are very cool, cool folks as well, and many people get
• 1:06:33
  excited about the idea. And then fairly quickly learned
• 1:06:35
  that, OK, the grass is not that much greener in the ancient
• 1:06:39
  world because of hallucination. Because of inability to execute
• 1:06:41
  tasks, for example. And then I met with a bunch of
• 1:06:44
  folks that did agents. I remember distinctly this one.
• 1:06:49
  Very important milestone in the agents world where many folks
• 1:06:54
  went to hackathon about building agents and Andrew Karachi from
• 1:06:57
  open air was there and Andrew Chapati basically said this
• 1:07:00
  thing. He had a like a 7 minute talk
• 1:07:02
  and said hey if you go to open the eye and you suggest like
• 1:07:05
  another transformer architecture, you suggest like
• 1:07:07
  state space Transformers whatever you suggest the RWV
• 1:07:10
  people have tried this out. There's like PHD's in there,
• 1:07:13
  they wrote papers or why this work doesn't work, like they
• 1:07:16
  tried all the permutations with Transformers, mixture of mixture
• 1:07:18
  of experts, like all these things.
• 1:07:20
  If you talk to him about agents, at least that was the the case
• 1:07:23
  in the summer. They're fairly as new to this as
• 1:07:26
  you would He was talking about this in the afternoon, he said
• 1:07:30
  basically you guys are the at the frontier of what we can do
• 1:07:33
  with these models because, you know, the idea of running them
• 1:07:36
  in loop is also new to us. Like we we just came up with BBC
• 1:07:39
  Four before GB Four, it wasn't impossible.
• 1:07:41
  GB Three could not execute on tasks that it worked for itself.
• 1:07:44
  It just didn't have as much. Sorry for the blunt agency.
• 1:07:49
  And now what happens is there's a bunch of these agent
• 1:07:52
  frameworks and the latest that I I know about and I know people
• 1:07:56
  are getting sad about this crew AI.
• 1:07:58
  But this guy Juan Mora and I had the I had drawn on the podcast
• 1:08:02
  actually. So if you only like check this
• 1:08:03
  out I'd I'd be happy to like send you a link with the Deep
• 1:08:06
  Trouble and show them. Up and.
• 1:08:08
  And specifically why he chose to write his own framework where
• 1:08:11
  like 17 other frameworks exist already.
• 1:08:14
  And we also had a chat with Killian Lucas about Open
• 1:08:17
  Interpreter which is an agent on your Mac that actually executes
• 1:08:20
  code and runs it on your Mac. So kind of like the code
• 1:08:23
  interpreter but like an interactive one.
• 1:08:25
  And the interview killing is also great and what I notice is
• 1:08:30
  that we're getting better and by we I mean kind of elements are
• 1:08:35
  getting better in context generally, right.
• 1:08:38
  So like 1 trend of 2023 to me personally was context length
• 1:08:42
  are exploding. GD4 started with four and now
• 1:08:45
  with 128 and I had the folks from the rope scanning paper as
• 1:08:50
  well on. So like context is getting
• 1:08:52
  solved even though the quadratic convention still is a problem
• 1:08:55
  and context is very important for these agents because you can
• 1:08:59
  only do so much with rag. You can OK you can go and save
• 1:09:01
  the memories and the task and everything in like to database
• 1:09:04
  retrieve them at the time and say OK the user wanted this and
• 1:09:06
  that at some point you need context you need this this model
• 1:09:10
  that runs your agent to remember well you asked me a preference
• 1:09:14
  of yours to send your e-mail in your tone seven days ago and
• 1:09:18
  that's like on the top of the context if at all and so I think
• 1:09:21
  context grows and the more so auto GPT when it came out we
• 1:09:24
  only had like 4000 tokens in the context possible window.
• 1:09:28
  Now these tools the crew I they can they can send like books.
• 1:09:33
  So The Great Gatsby I think is around 70,000 tokens or so,
• 1:09:37
  right. So you know you know supports
• 1:09:39
  128 over the API. So you can literally send The
• 1:09:42
  Great Gatsby book like context and would basically understand
• 1:09:44
  all of it. I think that's a crucial piece
• 1:09:46
  for agents as well. Another very important piece is
• 1:09:49
  evaluations, and that's also horrible in agents.
• 1:09:52
  If we talk about evaluations for a basic thing for a rag, you
• 1:09:55
  know, a very basic like, you know, you go, you retrieve some
• 1:09:58
  stuff, you answer the user, maybe you do something else.
• 1:10:01
  Evaluations of whether or not the agent did what you needed to
• 1:10:05
  do, whether the task was completed successfully, that's
• 1:10:08
  significantly more difficult because you don't build them to
• 1:10:11
  be the specifically tasked focused.
• 1:10:14
  You build them to be generic, generic.
• 1:10:16
  And so there's a bunch of approaches to agent as well.
• 1:10:18
  The cool thing about Crew AI, the guy that I talked to is he
• 1:10:22
  actually kind of came up with a basically an assembly of agents
• 1:10:25
  each with a specific task and specific prompt.
• 1:10:28
  They're all they're going to assemble together like a team to
• 1:10:30
  do a thing and that's pretty cool.
• 1:10:32
  That's why he called the crew I because you know in crew there's
• 1:10:35
  one person shouting directions everybody else looked growing.
• 1:10:38
  So he he specifically has this approach where every agent is a
• 1:10:41
  specific task and kind of like said about the valuations, you
• 1:10:44
  can evaluate every agent on the specific task as well and then
• 1:10:47
  in accordance of all of them together they come up and do a
• 1:10:50
  thing. I also tried multi on, which is
• 1:10:53
  super cool. I've tried like a bunch of them.
• 1:10:56
  I'm still waiting. I'm still waiting because I
• 1:10:59
  haven't yet been able to replace a core part of what I do, which
• 1:11:04
  is, as I learned on X every week about the cool stuff in the eye,
• 1:11:08
  I just drop a note and say, hey, I'm going to talk about this in
• 1:11:10
  my podcast. I'm gonna cover this.
• 1:11:11
  I'm gonna cover this every Wednesday before Thursday.
• 1:11:14
  I need to go next, find all those replies, compile them into
• 1:11:18
  a thing to show notes that I have the show notes.
• 1:11:20
  I wasn't able to find or build an agent that does this for me
• 1:11:23
  easily. And until I do, I'm still
• 1:11:26
  waiting on agents as like this panacea or promise.
• 1:11:29
  In this AI world, you've got a clear use.
• 1:11:31
  Case though, that's awesome. I have one and it's.
• 1:11:33
  Like an evaluation? Basically, they can test Yeah.
• 1:11:37
  Have you messed? Around with DSP I so.
• 1:11:42
  I've seen this fight on my feet like everybody else.
• 1:11:45
  And this week we're gonna talk with with folks we actually
• 1:11:49
  like. I'm I'm planning to learn a
• 1:11:52
  little bit before the conversation.
• 1:11:53
  I'm going to talk with Benjamin, Clavie and Connor from the way
• 1:11:56
  we had podcast because they dove into the spy and had very
• 1:11:59
  interesting content about this. So actually if you ask me next
• 1:12:02
  week, I'm probably going to have more about the spike.
• 1:12:05
  But I know for a fact that that's very, very interesting
• 1:12:09
  and has been as people mature in these products, they understand
• 1:12:13
  the OK, maybe something like you would talk to a human.
• 1:12:15
  That's not the thing that we want.
• 1:12:17
  Maybe we want to actually like treat those machines as machines
• 1:12:19
  and to programme them a little bit.
• 1:12:21
  I think that's the main idea from this fight.
• 1:12:24
  Excellent. Yeah.
• 1:12:25
  When you do play around with it, let me know.
• 1:12:28
  We talked to Omar a while back and I'll drop this that in the
• 1:12:33
  show notes too, because I think you're going to like it a lot.
• 1:12:36
  And while you're doing a little bit of studying on it, you can
• 1:12:39
  listen to the episode that we did with them.
• 1:12:41
  And it is so exciting to see how much because it takes a little
• 1:12:47
  bit to understand what's going on, right.
• 1:12:49
  And I look at it as like you throw something at it and then
• 1:12:55
  it gets broken. U and there's like all these
• 1:12:58
  prompts that kind of race to see which one is the best.
• 1:13:01
  But you don't have to go and you don't have to create all these
• 1:13:04
  prompts. It just automatically does that.
• 1:13:06
  Well, man, I've taken up a tonne of your time.
• 1:13:08
  I appreciate you coming on here and chatting with me and and us
• 1:13:12
  geeking out on everything from podcast gear to evaluation
• 1:13:16
  metrics. It's been a blast and I love
• 1:13:19
  what you're doing with Thursday. I This is great dude.
• 1:13:24
  Hope to have you back. Thank you.
• 1:13:25
  Thank you so much. Ohh yeah.
• 1:13:26
  So folks are more than welcome to join our live streams.
• 1:13:29
  They're very interesting folks who build the stuff that we talk
• 1:13:32
  to often, come and talk about the stuff they built.
• 1:13:35
  And I find that like maybe one of the best parts of the
• 1:13:38
  community that I, you know, I report to something that was
• 1:13:41
  released this week. And oftentimes, at least in the
• 1:13:44
  smaller labs, you know, Jeff Dean from Google is not going to
• 1:13:47
  come probably. But we, we had folks some news.
• 1:13:50
  We have folks in the open source community.
• 1:13:51
  We have folks for Gina embedding.
• 1:13:53
  Yeah, we have like a bunch of very interesting people to talk
• 1:13:55
  about the actual things they built and I find that incredible
• 1:13:58
  in the Thursday community. So definitely invite folks to
• 1:14:01
  check that out and also shout out to where the masses the, you
• 1:14:04
  know, the the closest place to build your lens and track your
• 1:14:07
  LLM data. So definitely give us a follow.
• 1:14:11
  Well, this has been. An absolute blast.
• 1:14:14
  And we'll leave all the show notes for everything or we'll
• 1:14:18
  leave everything in the show notes.
• 1:14:20
  I should say thanks so much, Alex.
• 1:14:24
  Cheers. Bye, bye.