Latest / Agentic Conversations (formally mlops.community) / Ax a New Way to Build Complex Workflows with LLMs // Vikram Rangnekar // #259

Transcript

Ax a New Way to Build Complex Workflows with LLMs // Vikram Rangnekar // #259 · Agentic Conversations (formally mlops.community)

• 0:00
  My name is Vikram Ramneker. I'm not, you know, I'm not with
• 0:05
  any specific company right now. I'm just consulting, building my
• 0:09
  framework and, you know, like trying to figure out where this
• 0:13
  whole level on thing is going and you know, where I I fit in
• 0:18
  my coffee. I I just like any good like a
• 0:21
  light roast, you know, love all the stuff that came out of, you
• 0:25
  know, comes out of SF and like the Pacific Northwest and they
• 0:29
  got and if you ask me what's my favourite coffee, I'd say Caden
• 0:32
  Cloud in Santa Cruz, Verva Caden Cloud, both both those places
• 0:38
  are you cannot go wrong with them.
• 0:41
  What up MO OPS community, we are back for another podcast.
• 0:45
  I am your host Dimitrios. Today it's all about DS pie.
• 0:49
  Vikram broke down so many concepts that I thought I
• 0:53
  understood, but I didn't. I'm not going to lie, I didn't
• 0:58
  and you get to hear me working it out in real time.
• 1:00
  Hopefully it helps you understand.
• 1:02
  Let's get right into it. But before we do, I've got a
• 1:05
  little recommendation engine helper for your music listening.
• 1:11
  If you have not heard of these guys, they're called Black
• 1:15
  Coffee and they are so good. They are so good.
• 1:20
  I'm just absolutely in love with them right now.
• 1:23
  I can't get enough. Let's have a little listen to
• 1:28
  their top rated song called The Rapture.
• 1:31
  And I want you to picture the scene.
• 1:36
  It's 5:00 AM, the sun's coming up, you're looking out over the
• 1:43
  Mediterranean Sea. You've been dancing all night
• 1:49
  just on vibes. No help from any kind of
• 1:54
  substance Let's get into this conversation
• 3:17
  with Vikram all about DS Pie. As always, if you enjoy this
• 3:22
  episode, just tell one friend. All right, real quick, I want to
• 3:26
  tell you about our virtual conference that's coming up on
• 3:28
  September 12th. This time we are going against
• 3:31
  the grain and we are doing it all about data engineering for
• 3:35
  ML and AI. You're not going to hear rag
• 3:39
  talks, but you are going to hear very valuable talks.
• 3:42
  We've got some incredible guests and speakers lined up.
• 3:48
  You know how we do it for these virtual conferences.
• 3:50
  It's going to be a blast. Check it out right now you can
• 3:53
  go to home dot ML OPS dot community and register.
• 4:00
  Let's get back into the show dude.
• 4:14
  Well, let's start by telling me what you've been working on
• 4:19
  because I want to start there and then go into like the
• 4:21
  nuances of it and all the different challenges you've
• 4:24
  encountered. Well, I started like building
• 4:28
  with LMS a while back and you know, like a bunch of startup
• 4:31
  ideas, first one being this thing called 42 papers back
• 4:35
  before I for all the archive excitement, I I predicted sort
• 4:39
  of like the archives, kind of like take off and and and I
• 4:44
  wanted to build this like way to discover like what everyone's
• 4:48
  reading like right now on archive and stuff related to you
• 4:52
  like, you know, I want. To see the trending.
• 4:54
  Sort of exactly, but also like related trending.
• 4:58
  Yeah. And so I was like exploring ways
• 5:00
  and, and I didn't just want to throw the papers out there and
• 5:02
  want to like extract the, the, the key points and stuff like
• 5:05
  that and just show that like, you know, forget, forget the
• 5:08
  papers kind of thing, the whole thing.
• 5:09
  No one's reading the math anyway.
• 5:11
  So. So I was like, yeah, and we're
• 5:15
  trying to figure that out. And we had this whole idea
• 5:17
  around let's get like, let's get and this was early, like 20-17
• 5:22
  ish maybe, and and called it 42 papers, you know, named it after
• 5:26
  the Douglas Adams whole 42 things.
• 5:29
  Yeah, but and we were saying we had this idea where let's get
• 5:33
  all these people sort of like, you know, the Amazon service
• 5:36
  where they can like all these like PhDs or whatever can help
• 5:39
  us and do this and we give them like points or money or
• 5:42
  something karma and, and, and we had this whole thing and we're
• 5:46
  planned and, and I was like, you know what this is?
• 5:48
  I don't know how fast it's going to scale.
• 5:50
  It's going to be hard. And you know, like get people
• 5:52
  like wheat papers and extract value and stuff like that.
• 5:54
  And so when I stumbled onto like the early models coming out of
• 5:57
  Allen EI, these were like Burt models and stuff like that.
• 6:00
  And and they had trained them on archive maybe like small models
• 6:04
  like today's standard, like probably like I, I don't even
• 6:06
  remember probably like 500 million parameters.
• 6:09
  Yeah. Maybe.
• 6:10
  And the thing could summarise. Yeah.
• 6:11
  And this thing could like summarise the paper and.
• 6:13
  And the first time I actually saw it, I was like blown away,
• 6:16
  honestly. You know, till then, I was, I
• 6:17
  was, I was very familiar with the ML world, you know, from
• 6:21
  being at LinkedIn. They used ML everywhere.
• 6:24
  Yeah. And they're the best, but they
• 6:26
  definitely use it. Yeah, I mean they, they, they
• 6:29
  use it right. It looks like ad relevance, a
• 6:31
  fraud and all kinds of stuff right everywhere.
• 6:33
  Yeah, these are like not LM models, these are not GPU
• 6:36
  models. These are models that ran on
• 6:37
  Hadoop or whatever. These are most statistics
• 6:40
  leaning sort. Of the traditional ML world.
• 6:43
  Traditional ML and and I and I this thing was like magic
• 6:48
  because that I could kind of, you know, I could roughly wrap
• 6:51
  my head around what's going on, but this thing could summarise
• 6:54
  like a BLOB of text into like points.
• 6:56
  I was like, how does that what just happened?
• 6:59
  And that, that was like really mind blowing.
• 7:01
  And, and then kind of, I took, you know, went down that rabbit
• 7:04
  hole and I need, I needed sort of like a framework, I realised
• 7:10
  to like build with these things. And so I started like an early,
• 7:15
  like the early work of building a framework.
• 7:18
  And, and there was a lot of learnings around the way and,
• 7:21
  you know, and it like evolved along the way.
• 7:25
  And that's where like I am today.
• 7:26
  But in terms of EX. Incredible.
• 7:29
  So starting in with the fine-tuned verb model for 42
• 7:34
  papers and then going from Mechanical Turk I guess is what?
• 7:40
  You dropped that real quick when we realised the model could just
• 7:43
  do that right, I mean. Yeah, it's, it's incredible to
• 7:48
  see. And now it feels like you can
• 7:52
  still probably get what you're looking for with that fine-tuned
• 7:57
  bird model, or maybe like a fine-tuned small model.
• 8:02
  I think a fine-tuned small model, there were a lot of like
• 8:04
  issues with that bird model. There was like a lot of like
• 8:07
  literal heuristics like, oh, if you, if you, if your word is
• 8:11
  like broken here, then let me just like join it together.
• 8:14
  There was, you know, a lot of like and, and they were sort of,
• 8:19
  I think super like, obviously they could do it, but a small
• 8:21
  model, I think would not cost too much more or be slower.
• 8:25
  And, and I'm not even sure if you probably need a fine tune
• 8:29
  model at this point. I I don't know, I mean, some of
• 8:32
  these like really small models coming out today are, are like
• 8:36
  super powerful. You know, this Nisten guy, he
• 8:38
  had this like merged model he launched yesterday.
• 8:40
  It was like you did this ternary one bit quantizing on it.
• 8:45
  It came down to like one like 65 MB or something.
• 8:50
  And this this model was like, apparently like super fast and
• 8:54
  really good. And so you know.
• 8:57
  Yeah, exactly. You, you can get so much further
• 9:00
  with some of the stuff that's out there these days and that it
• 9:05
  is funny to think about how no, you out of necessity had to
• 9:12
  create something to help you along the way from that, Burt.
• 9:15
  But along the way, things just blew up with in the space.
• 9:21
  And so you were able to level up really quickly.
• 9:25
  And we mentioned too, before we hit record that you're very much
• 9:29
  focused on the JavaScript TypeScript world because it is
• 9:32
  so fast and you can iterate and you can do things really
• 9:36
  quickly. So talk to me about first, just
• 9:41
  like maybe give us a bit of background on what the actual
• 9:44
  tool is that you built. And then why this love for
• 9:49
  JavaScript, TypeScript? Sure.
• 9:52
  So the tool I built it's, it's basically a framework.
• 9:55
  It's called AX and it doesn't stand for anything really just
• 10:01
  cool sounding name. So for me, you call it agent
• 10:05
  framework or whatever. I mean, so the idea was like
• 10:08
  when I, I started, I needed initially a way to abstract out
• 10:13
  the LLMS. So that was my first thing.
• 10:15
  It was like, OK, all these like client libraries have their own
• 10:17
  set of bugs and all of this. And these are just like API
• 10:19
  calls. Let me abstract this out.
• 10:21
  And that was just a simple goal. And I think I achieved that
• 10:25
  with, you know, like this pretty quickly essentially.
• 10:29
  And then I realised on top of that that's like, you know, I
• 10:32
  need a way to improve prompting. And then I realised these blobs
• 10:37
  of text that I have to maintain like, you know, it's .1 do this
• 10:40
  .2 don't do this .3, you know, do that.
• 10:43
  And like I, I hated that honestly, that I did not, it
• 10:45
  caused me not to reach for an lol more often in my like code
• 10:50
  'cause there was like this whole like dance I had to do and then
• 10:53
  there was like loops around it to make sure it didn't break.
• 10:55
  And you know, there was a whole bunch of stuff and I.
• 10:58
  Find myself too, just on that fact really never being able to
• 11:03
  quit because I always think the next prompt is going to be
• 11:07
  better. And so it's like, yeah, that's
• 11:10
  there's, I can never be like, yeah, that's good enough.
• 11:12
  You know, it's 'cause prompting is so yeah, yeah.
• 11:16
  And. You don't know what word in
• 11:18
  there is messing things up and you know.
• 11:20
  Yeah. And and that just like just
• 11:22
  feels the key to me. And I, I didn't like it.
• 11:24
  I know instruction kind of following is obviously the way
• 11:27
  all of these things have been trained, you know, at the, you
• 11:29
  know, final stage of training. But I was like, try to look for
• 11:32
  a better way and that and I tried a bunch of stuff.
• 11:36
  I was like, there was this templating thing and blah, blah,
• 11:38
  blah. And it just didn't quite fit.
• 11:39
  And then I stumbled onto the DS by papers or, you know, the
• 11:43
  demonstrate search predict papers from Omar and Stanford.
• 11:46
  So the, the key, I know you've like covered this before, but
• 11:50
  essentially to me that they were like, so what's really
• 11:53
  interesting about that paper is there's this trend where people
• 11:56
  just take like an idea and then chop it up into like 10 papers,
• 11:58
  right? And then all these papers are
• 11:59
  really light because they're now they have 10 public published
• 12:02
  papers. But Omar like took all his best
• 12:05
  ideas and put it like into this one paper.
• 12:07
  And people don't notice that like people like, oh, OK, it's
• 12:09
  like these examples. You can tune these models, but
• 12:12
  there's a lot more to that. If you start the paper, the
• 12:14
  first part starts with signatures, right?
• 12:16
  Like that signatures concept. Really like to me it made sense
• 12:20
  because they're basically functions.
• 12:22
  What they're saying is they're a bunch of inputs, bunch of
• 12:24
  outputs and and then, you know, obviously with the X, we took
• 12:28
  that even further. We added the concept of types on
• 12:30
  those things. So essentially we say I have two
• 12:33
  string inputs and I need a boolean and, and like, and a
• 12:37
  number output it. Then we'll enforce that
• 12:41
  automatically. And that's basically what you
• 12:43
  need in your code. You want something with inputs
• 12:46
  and something with outputs. You don't want, you know, blobs
• 12:48
  of text that you're like trying to substring through, right?
• 12:52
  So that that was a real clear abstraction to me.
• 12:55
  And then I was like, you know, there's so much more that could
• 12:57
  be built on the abstraction because these inputs and
• 12:59
  outputs, what are the, you know, when put together, what is this,
• 13:02
  this an API, right? Once you have this kind of
• 13:05
  structure on a prompt, it's very easy to like say, oh, that this
• 13:08
  is not a prompt anymore. It's an API.
• 13:11
  And now if you, if you have and and also another realisation
• 13:15
  came on is you don't want to do too much in a prompt.
• 13:17
  There are, there are, you know, talks by open the eye guys and
• 13:20
  stuff. I've, I've, you know, so I don't
• 13:22
  know the exact, I can't tell you the exact like technical reasons
• 13:29
  behind this, but ascent at a high level from like a builder,
• 13:33
  you don't want a prompt to do too much.
• 13:36
  You want it because there are all these like tasks that these
• 13:39
  things are trained on. And when you essentially ask it
• 13:42
  to do something, the LLM is pulling on those tasks.
• 13:45
  And kind of like there is some sort of like a, like a
• 13:47
  limitation because of the size of the LLMS and all of the stuff
• 13:51
  you contact site, you know, honestly, I don't know where
• 13:53
  that limitation comes from, But if you say, oh, can you classify
• 13:56
  this? Can you summarise this?
• 13:57
  Can you also call these functions?
• 13:59
  Can you also do this? Like it's going to mess up more.
• 14:01
  So you want to like focus it down and say, OK, let's
• 14:04
  summarise here. Let's take that output.
• 14:06
  Let's put into another one that then does function calling based
• 14:09
  on it, augments it with more data.
• 14:11
  Then let's take that result, put it in a third prompt.
• 14:14
  Like if you have these like long contact, I mean, yes, we do have
• 14:17
  longer contexts now, but that does not mean you can have a
• 14:21
  longer multi step, blah, blah, blah.
• 14:23
  Because these things are not trained for that kind of a
• 14:27
  thing. Even if you notice, like Google
• 14:29
  talks about needle in the haystack like tests, which is
• 14:33
  fine, which is basically an extraction step and saying we
• 14:36
  put a lot of text and we can find the right step.
• 14:39
  They don't say you can also then do function calling and then do
• 14:42
  this and then do that, right? I mean, so there's this whole
• 14:45
  like multi step like task thing. They, I think they refer to it
• 14:49
  as long horizon tasks and and most of these models are not
• 14:53
  trained on that yet. So, you know, even though you
• 14:55
  have the long context, you still want to like get your work done
• 14:58
  as quickly as possible. Yeah, you can put a lot of text
• 15:01
  in there and then say, oh, you know what, find the things you
• 15:04
  need from here. But you know, that's you are
• 15:07
  limited to that. And then you want to take those
• 15:09
  texts and then do something else anyway.
• 15:11
  So the point is that it defined an API and that API made it
• 15:15
  really easy to then plug in all these like work flows together.
• 15:18
  So if I have like an extraction step, a classification, a
• 15:21
  function calling augmentation step.
• 15:23
  And so it's basically sort of like code.
• 15:27
  Yeah. And that that, that that
• 15:28
  signature piece was really, really like, honestly, that blew
• 15:31
  my mind first. I mean, it's not, it's not
• 15:35
  anything like out of the blue. It's functions, I mean, but like
• 15:38
  to apply that idea to like LMS was is, you know, a paper on to
• 15:43
  itself, sort of. Right.
• 15:45
  Yeah. Gives you so much more
• 15:46
  stability. Yes, exactly.
• 15:48
  So. And like with AX, we think that
• 15:51
  even further because now these functions we, we have, we, we
• 15:55
  focused on the concept of agents.
• 15:57
  These are basically these like, sort of like prompts, but they
• 16:00
  have the capability to do multi turn reasoning, function
• 16:04
  calling, I mean, reasoning, you know, as much as these things
• 16:07
  reason function calling and like try to get a task done.
• 16:11
  But when they call these functions, these functions
• 16:13
  aren't necessarily just functions like like other, like
• 16:18
  some remote API, but they can call other agents because there
• 16:21
  is this like input output structure.
• 16:23
  You can easily plug agents into agents as functions.
• 16:27
  You can have a tree of agents and you could have something
• 16:30
  like, hey, you know, this is like the one that you asked this
• 16:33
  agent about the stock, about a certain like, you know, stock.
• 16:38
  And then it has like a agent as it calls like an agent to do
• 16:41
  like a analysis on something and then something else and
• 16:44
  something else. And then these four agents like
• 16:46
  return their results. And then this main thing like
• 16:49
  decides, OK, here's your answer sort of thing, right?
• 16:52
  So it makes it really easy to plug agents together and stuff
• 16:55
  like that. The other thing with the paper
• 16:57
  that really blew me away was the concept of examples.
• 17:01
  So essentially a lot of times we like, you know, we just talked
• 17:04
  about it. You put all these like
• 17:06
  instructions in there and instructions are really great
• 17:10
  with large models, right? I mean, like you want to like
• 17:13
  the, the bigger the model. And the way I, I understand it
• 17:17
  from the paper is you want to use these big models to generate
• 17:19
  these examples from these instructions.
• 17:22
  And what are example examples are?
• 17:25
  If this is my input, then this should be mapped.
• 17:28
  If this is my input, these should be mapped.
• 17:30
  And these should be rich. Try to cover as much as
• 17:33
  possible. And then you plug these into
• 17:36
  your prompt. And now this has this allows the
• 17:40
  LMP in four patterns that you and I might not even see.
• 17:45
  So it's sort of like if I told you to write something sort of
• 17:49
  like Shakespeare, but not, I mean, how do I really
• 17:52
  communicate that I could use just his name, but.
• 17:55
  And that would infer patterns. But if I give it like an example
• 17:58
  of what I want, it's much more powerful, right.
• 18:00
  There are patterns in there that I couldn't even use, you know,
• 18:03
  English words to communicate to you.
• 18:05
  Yeah, right. So, so these patterns are what
• 18:08
  these LMS are able to like leverage, you know,
• 18:11
  distributions, whatever they want to call them.
• 18:12
  I I don't know the exact term. And, and so this is just like
• 18:17
  this took all my work flows like to the next level, like they
• 18:20
  were a lot more solid. And, and how many, like, you
• 18:24
  know, how many examples can you add in there?
• 18:26
  There's, there's a new Google Paper that said add 1000.
• 18:29
  Obviously I'm not going to do that.
• 18:30
  But like then input tokens are cheaper than output tokens.
• 18:34
  So use those input tokens to get more solid output tokens.
• 18:39
  You're not error correcting and all of that and wasting your
• 18:42
  output tokens, right? Yeah.
• 18:44
  That's such a good point. Yeah.
• 18:46
  So that that's a sort of like the second piece here and.
• 18:49
  In DS Pi it that's when the when they do the teacher student type
• 18:56
  of stuff. That that is also in there.
• 18:59
  That is actually like a third step.
• 19:01
  So once you set the examples and then you do the teacher student
• 19:03
  bootstrapping, that's when it uses these basic.
• 19:07
  So in DS by examples are very simplistic.
• 19:11
  You don't have too much information in them.
• 19:13
  You just have like the input and the quick output and and that's
• 19:17
  only of the whole tree. So it's sort of like if you have
• 19:19
  a tree of of tree of functions or prompts, whatever you want to
• 19:24
  call them DSPY programmes that does like a like a whole
• 19:26
  workflow. The examples, you just set it at
• 19:29
  the top, say this is the inputs and at the bottom these are the
• 19:32
  outputs I'm expecting. And then when the bootstrap
• 19:35
  iterates, it captures all the inputs and outputs of the whole
• 19:38
  tree. Do you get it?
• 19:40
  So like, so essentially you have this like database of all those
• 19:43
  traces whenever you have a good output.
• 19:46
  So if you're, you know, every time you get a good output, you
• 19:48
  say pause, save that. Yeah, pause, save that.
• 19:51
  So essentially now you have a database of all these like good
• 19:54
  out inputs, outputs that when you are putting your code into
• 19:57
  production, you apply them on those things.
• 20:00
  So this one gets it's set of 10 great examples.
• 20:03
  This gets it's set of 10 great examples, this one.
• 20:05
  And then the whole tree works so much better.
• 20:09
  Yeah. And that's funny because I had
• 20:12
  heard others talk about how and they weren't using DS Π but they
• 20:17
  talked about how important it was for them and their workflows
• 20:21
  to have the high quality examples.
• 20:23
  And they were saying, I spend so much time just going around the
• 20:28
  web and collecting examples that I weren't.
• 20:32
  And I have a database of examples.
• 20:35
  So, and this was for marketing use cases, right?
• 20:39
  So if it's an intro to a blog, they go around and anytime they
• 20:45
  see a great intro for a blog, they'll collect that and throw
• 20:48
  it in their database. And now with these different
• 20:51
  tones, and then they actually they try to label it so that
• 20:55
  they can say this is this type of tone and this is this type of
• 21:00
  formality. And then they can give it in
• 21:03
  those prompts. They can almost reference like
• 21:06
  this database and say, here's here's what you're doing.
• 21:09
  It sounds like with the way that DS Pi does it, you wouldn't even
• 21:16
  need that. Or would you still want it have
• 21:19
  something like that? Or are they 2 separate things?
• 21:25
  So those are, it really depends. You're right.
• 21:28
  DSPI is about having exact, exact examples for your inputs
• 21:32
  and your outputs. So if you're expecting like
• 21:34
  three things, inputs that you have a question, a context and
• 21:38
  like and another thing as inputted like, and then and then
• 21:43
  you want the outputs and like a classification or something,
• 21:45
  some three things. And then you want like a
• 21:48
  category and then you want an output that's like a summary and
• 21:51
  something else. And you want the exact inputs
• 21:56
  and outputs like the whole chain as an example, right?
• 21:59
  But if, if you're like, if you're getting it to do
• 22:03
  something and you want to make it better and you want to add
• 22:06
  these examples in there as one, as like just a thing to, you
• 22:10
  know, improve the prompt. That's fine, right, Because your
• 22:12
  inputs could just be write me a marketing, you know, your, your
• 22:17
  first input could just be essentially you could give it a
• 22:20
  marketing task. You could build like an agent to
• 22:22
  do marketing tweets or whatever. And then you could have the
• 22:25
  first input could be the task. The second input could be the
• 22:31
  the examples that you're like sort of inserting yourself like
• 22:35
  a context, you know, and and then it would just like the
• 22:39
  output would be the tweet, right?
• 22:42
  So, so essentially, yes, you could provide your own examples,
• 22:46
  but the examples that you know, at a high level, they're not
• 22:51
  different. They're basically putting
• 22:53
  something in there based on what you want, like output.
• 22:55
  So you could use those as example outputs and build your
• 22:58
  own like sort of tree. And they're, they're just high
• 23:03
  quality things that you have hand picked, you know, in the
• 23:08
  end. Because yes, even though the
• 23:10
  bootstrap thing is sort of like it captures all the traces in
• 23:13
  between, but the final input, the input and the final output
• 23:17
  is something you still pick, right?
• 23:18
  You say, OK, even though you're going through 10 steps, It's
• 23:21
  like when this is the input sort of, I mean, it's not exactly, I
• 23:24
  want this to be the output, right?
• 23:26
  Yeah, like sort of I like an output like this and then maybe
• 23:29
  15 steps in the middle that you don't have to go and set values
• 23:33
  on, right? The LLM is the the values are
• 23:36
  flowing. The LLM, the first LLM activates
• 23:38
  and that generates an output that goes into the second, then
• 23:41
  that activates and then goes into the third, right?
• 23:43
  I mean sort of like function calling the next function
• 23:45
  calling the next function. And now you don't have to sit
• 23:48
  and capture all the variables within all those like the state
• 23:52
  within the the the whole chain of functions, right?
• 23:56
  And. That's really the key to like
• 24:01
  DSP. It's basically saying that, you
• 24:03
  know, yes, sure, set high quality examples, found them
• 24:06
  from the web or whatever. That's cool.
• 24:07
  And then we'll help you capture all of the traces in between so
• 24:11
  that when you put this into production, like there are good
• 24:14
  examples set for everything. Because your your tweet example
• 24:18
  might be good for like the final output, but it might not be good
• 24:20
  for the one classification or extraction step you have in the
• 24:23
  middle, right? That might need it's own sort of
• 24:26
  example. So that that essentially was
• 24:29
  like sort of blew my mind because I was like, it's really
• 24:32
  simple. Again, none of this is like, you
• 24:33
  know, like a bit it, it, it really like solved problems that
• 24:38
  I was having and I, I could see others kind of struggling with
• 24:41
  that as well. Yeah, it also allowed me to like
• 24:44
  take my models to smaller, take my work to smaller models and
• 24:50
  still kind of have that same performance.
• 24:52
  And something I'm like exploring now is there is more and more
• 24:55
  tuning APIs that are popping up, right?
• 24:57
  Every service provider has like their inference API, but now
• 25:01
  they have a tuning API. In fact, Gemini is like sort of
• 25:04
  made it like almost free to run to small models.
• 25:07
  So if you take this information that DSP wise capture and
• 25:12
  essentially just push it to the tuning API and have like a tune
• 25:16
  model, right? So then you might not even have
• 25:18
  to set those on the. So, you know, there, I think
• 25:23
  there's a lot of value in in in, you know, taking this ahead.
• 25:27
  Fascinating. Yeah.
• 25:28
  And retracing our steps and going back to something that you
• 25:32
  said originally that IA 100% agree with is the idea of making
• 25:39
  sure that the tasks themselves are as small as possible.
• 25:44
  But then where you can really let yourself go is the context
• 25:48
  that you're giving it on the task.
• 25:50
  You're able to say, and that's why I think the context windows
• 25:54
  are great. Having larger context windows
• 25:56
  are really useful. But the task itself that you're
• 25:59
  trying to get done, you're not trying to get 3 or 4 tasks done
• 26:02
  at the same time in the same prompt.
• 26:05
  You're just trying to say, do this one thing, do it really
• 26:07
  well. Here is all the context you need
• 26:10
  on how to do it, and then it outputs that.
• 26:13
  And then we can go along to the next thing.
• 26:17
  And, you know, there's actually a really interesting thing
• 26:19
  you're even though we see that people often do not realise
• 26:24
  they're adding another task. That's another thing I've seen
• 26:26
  often. So they're they're basically
• 26:28
  it's like this, let's do this thing and then I want Jason
• 26:31
  output. Guess what?
• 26:33
  You've added another task. So you don't realise that it
• 26:36
  like, oh, you know what? I got Jason everywhere.
• 26:38
  Like, yeah, but you know, dude, this is like it's another task
• 26:42
  and it's quite a heavy task now, you know, So like that's
• 26:47
  another, that was another breakthrough that the paper had
• 26:49
  for me. Like I've struggled with outputs
• 26:53
  and I've tried to figure my way around and I and even early on I
• 26:56
  realised that Jason and stuff was just like a nice to have.
• 26:59
  I think people without like overthinking it.
• 27:01
  They were like, oh, I use Jason everywhere.
• 27:03
  So this should just pop out Jason.
• 27:04
  Oh, Jason doesn't work. Well, let me try Yamils.
• 27:07
  I was like, no, you know, what about key values?
• 27:09
  Just like, do you really need the nested structure out from
• 27:12
  this thing? Like mostly not so, so I the, so
• 27:17
  if you look at the paper, they use key values and that's really
• 27:21
  great because a lot of the models, when they're trained,
• 27:23
  they're also trained on the key value format.
• 27:26
  There's always like a title of what the The thing is.
• 27:29
  And then The thing is right. And then and there are a lot of
• 27:33
  advantages to key value. Firstly, yes, there are no extra
• 27:36
  tokens that this thing has to figure out.
• 27:38
  It's not, it's not really like the you're wasting tokens.
• 27:41
  Sure, that's maybe one thing, but it's, it's like it has to
• 27:45
  generate those tokens and then it has to, and it's got to do a
• 27:48
  worse job on something else because it's making room to do
• 27:52
  that, Jason, right. And then if you you're like,
• 27:55
  yeah, but you know, the, the so and so API does this really
• 27:58
  fine. This model provider, you know,
• 28:01
  if you notice often those things are slower because they're doing
• 28:04
  something else. It's sort of like having a
• 28:06
  second model or, or like trying to force it somehow and the
• 28:10
  error correcting in the background and, and they're like
• 28:12
  hiding that from you. So it's and then what about
• 28:15
  streaming? Often these things completely
• 28:17
  break for streaming, even like there are model providers, there
• 28:19
  are just like Jason streaming function calling is broken,
• 28:24
  right? And they just don't because no
• 28:25
  one's using that or something, you know, maybe people using
• 28:28
  that just like whatever. But so the key value, the
• 28:33
  advantage of going just key value for inputs and outputs or
• 28:36
  outputs specifically is you can now process things as they
• 28:40
  stream in. So once you get the key and then
• 28:42
  you start getting an output and you're expecting the output to
• 28:44
  be a set of points like say you know, one dot blah 2 blah, 3
• 28:49
  blah. You can now ensure you can like
• 28:52
  literally validate it as it flows in.
• 28:55
  And if it's not doing the one dot blah, 2 dot blah, whatever,
• 29:00
  then you can like early stop and say, you know, hold on, hold on.
• 29:03
  This is not what I want. I, I specifically want it to
• 29:07
  follow this format. That's the concept of assertions
• 29:10
  that, that the DSPY paper has. So and so in EX.
• 29:16
  We took that further and we, we built something called streaming
• 29:18
  assertions. So streaming assertions allowed
• 29:21
  you to basically do this like early stopping on like whether
• 29:26
  on key values, like OK, where, where there is like an output,
• 29:29
  which is like a summary and then the summary.
• 29:31
  And sure, I look at the whole summary and I can then decide,
• 29:34
  oh, this is wrong, blah, whatever reason.
• 29:36
  Or while the summary is streaming out, I can actually
• 29:39
  look at the thing and say, oh, you know what I wanted this is
• 29:41
  in points and this is not in points stop.
• 29:44
  Now this obviously is good because it's early stop.
• 29:48
  So you're going to, you know, your latency and all that is
• 29:52
  better and you're going to save tokens.
• 29:54
  So you know, this something is really that's possible, That's
• 29:57
  not possible if you go any other route, right?
• 30:00
  And it's built into the framework.
• 30:02
  So you don't have to like sit, enter, write this complex thing.
• 30:05
  And because the framework is vertically integrated like we
• 30:08
  there, there's no dependency. So Axe has like 0 dependencies
• 30:11
  in the packages. So it's, it's the API's, all the
• 30:14
  LLM AP is, are built from scratch.
• 30:17
  So the the beauty of that is we can actually like, you know,
• 30:21
  vertically build the most solid thing we want.
• 30:24
  We want streaming assertions or whatever, right?
• 30:26
  We can just build it now because we don't have to go past the
• 30:29
  boundary of someone else's like, you know.
• 30:32
  And I didn't quite understand the idea of these assertions and
• 30:40
  building out the API, basically being able to validate the type
• 30:45
  and how you said, all right, well, one abstraction is what
• 30:50
  they were doing with DS Π, but another abstraction is saying
• 30:53
  this can now be an API. And so when we give whatever
• 30:58
  output to someone, it comes in inherent with the idea of it's
• 31:03
  going to be in the type that you specify.
• 31:06
  Is that what I understand? Yes.
• 31:09
  So there are two types of assertions. 1 is.
• 31:11
  You put them, you assert a type. Essentially you say, you know,
• 31:15
  while you're creating the function signature, you just say
• 31:17
  this output type is string or string array or number or
• 31:21
  boolean or whatever, or you know, and.
• 31:23
  And the second type of assertion is like a custom function where
• 31:27
  you want to like, like actually do something with the output.
• 31:30
  You might just you might actually call another LLM if you
• 31:32
  really want, you'd be like, hey, is this, is this good enough?
• 31:36
  If not, throw a like a error, you know, essentially.
• 31:39
  And then the first LLM can correct itself, right?
• 31:41
  So it's, it's, it's a function called on this input output
• 31:45
  value. And, and this can be done either
• 31:48
  when the whole value is available, you can wait till the
• 31:51
  end of value. You might have like multiple key
• 31:52
  values that are outputted. So you know, in which case then
• 31:55
  you know, you might want to look at the whole value or you can do
• 31:58
  it as the value streaming out because it's not Jason.
• 32:01
  If you can somehow validate it, maybe you know, you're looking
• 32:04
  for some prefix or something before, so you can chop up every
• 32:08
  line that's coming, wait for a line to come in, then then kind
• 32:11
  of look at it with the regex and think, you know, sorry, you
• 32:13
  know, you're messing up sort of thing.
• 32:15
  And this early failure is is great because now, firstly, the
• 32:19
  assertions concept makes it easy to code because now you have all
• 32:22
  of this like captured within your DSPY programme.
• 32:27
  And when you initially put in the examples and you run the
• 32:29
  bootstrap, all of this gets engaged, right?
• 32:32
  So the error correction, all that.
• 32:33
  So your outputs, all the traces are high quality in between
• 32:36
  because the assertions have ensured.
• 32:38
  So in production, less of the stuff happens, right, because
• 32:43
  you have already fine-tuned it down to the correct what you
• 32:46
  need in terms of code. You're not saying please add a
• 32:48
  dash in front of you know, like you have an assertion.
• 32:51
  It's explain that so you got high quality examples, you just
• 32:53
  save those in there and the LLM goes, oh, I know what this guy
• 32:56
  needs. I see.
• 32:59
  OK. And one thing that you mentioned
• 33:01
  before we hopped on too, is that you feel like there's not a lot
• 33:08
  of people really thinking about building with LLMS or sometimes
• 33:12
  the way we go about building can be off.
• 33:15
  And it makes me think about some of the stuff that I, I tend to
• 33:21
  talk a lot of shit on how the things that we are building
• 33:25
  aren't really that valuable. And so I am looking around and
• 33:31
  saying like, this is the future we were promised.
• 33:34
  You know, this isn't what I was hoping for.
• 33:37
  So what? What's your take on that?
• 33:41
  For one caveat, I would say we are still early.
• 33:46
  So you know, we we shouldn't forget this game are like, you
• 33:50
  know, production ready, sort of like, you know, beginning of
• 33:52
  this year, mid last year, maybe maybe, you know, so we're like
• 33:56
  super early. I think people just haven't
• 33:58
  built their LLM muscle, right? They're they're just like a lot
• 34:02
  of people don't reach for an LLM naturally yet.
• 34:05
  I mean, there's so many like, you know, who was this is that
• 34:07
  joke about how like, you know, just Google it or whatever,
• 34:10
  right. And now just don't ask me sort
• 34:12
  of thing and like just LLM it. Then I think even in code,
• 34:15
  people don't like haven't wrap their head around what's
• 34:17
  possible. Like, you know, I've I've come
• 34:19
  across people who do not believe Melan can write stuff like the
• 34:23
  whole they're like, yeah, but it's not as good as law.
• 34:25
  You know, I was like, yeah, that's because you just like
• 34:26
  prompted it wrong. He's like, oh, I tried it.
• 34:28
  I was like, did you OK, hold on, Did you try that for chat?
• 34:31
  GPD 3.5 free? And, you know, it's like, yeah,
• 34:34
  I was like, dude, you're like, whatever engineer in this
• 34:37
  company pay the 20 bucks, you know, get get GPD 4 and then,
• 34:41
  you know, like prompted better give it some examples, right.
• 34:44
  So it's sort of like people like, I think haven't like
• 34:48
  internalised it. And yes, I, I don't think a lot
• 34:51
  like like I have experimental tooling start-ups.
• 34:54
  So I was, I was experimenting the observability sort of
• 34:58
  spacing, you know, maybe there's something I can build there that
• 35:01
  I tried, like I experimented with this, another tooling idea
• 35:05
  where we'll build like make it easy to build these LLM work
• 35:07
  flows and, and you know, everyone, it's so early.
• 35:11
  Then obviously these spaces are super crowded.
• 35:13
  So I, you know, I'm not like I, I'm kind of holding off to
• 35:16
  figure out what really is the thing here.
• 35:19
  But also customers, right? I mean, they're, they're most
• 35:23
  are like just not, they don't have the people to build this
• 35:26
  essential. I think that's really like the
• 35:28
  job. I feel like AX can help with
• 35:30
  this, make it really, really accessible, right?
• 35:33
  Like I, I think the model guys are doing a phenomenal job.
• 35:35
  They're accelerating much faster than anyone else.
• 35:38
  And a lot of the people building like on the framework end of
• 35:41
  things, I almost feel like they're not building with real
• 35:45
  customers because I, I see a lot of like over complication of
• 35:50
  stuff and, and like, you know, it's not production ready.
• 35:53
  There's like no, you know, there's no telemetry.
• 35:55
  Like like EX has open telemetry built in.
• 35:58
  So there's always tracing happening.
• 36:00
  So you plug in a tracer like from, you know, open telemetry
• 36:03
  is like an open standard, you can plug it.
• 36:06
  So a lot of these stream bus don't have that, for example,
• 36:08
  just built in. And you know, there's a lot of
• 36:10
  like production stuff that you need.
• 36:12
  It's you know, there is just lots of dependencies.
• 36:15
  And then you're, you know, like every little like NPM for this
• 36:18
  and that and whatever, you know, Python.
• 36:20
  Break all the time, yeah. And people, you know, obviously
• 36:22
  don't want to use stuff that as all that.
• 36:24
  And then they have the link that's created the whole world
• 36:26
  on top of their own, like terminologies and blah blah.
• 36:28
  Like that's why I like dspy because it's sort of like
• 36:31
  standardising some of that. And there's a lot more people
• 36:35
  saying, Oh, OK, you know, what if all of us can have the shared
• 36:37
  understanding, this open thing with like that's, you know, an
• 36:40
  open paper, it's going to help. And, and then the framework on
• 36:45
  top of that helps with all of this other stuff because it's
• 36:47
  actually implements it and takes it like, you know, steps ahead.
• 36:51
  But yes, I, I will tend to agree with you.
• 36:54
  People need to build more useful stuff.
• 36:57
  And you're thinking about it at not even at the level of what's
• 37:00
  the actual output there. You're just thinking about how
• 37:04
  difficult it is to get to that output because of how we've
• 37:10
  architected this ecosystem. The ecosystem that we live in is
• 37:14
  just convoluted. And it's like, I mean.
• 37:16
  You know, I got to hit it. Ellen's also raw, right?
• 37:19
  I mean, they're like super. They're like raw power.
• 37:22
  And you have to sort of like, you know, help someone like, you
• 37:26
  know, you just like cannot throw a guy into a Ferrari and be
• 37:28
  like, you know, just like you sort of have to give him some
• 37:31
  way to control it. And, and I feel like like 11
• 37:35
  good example is really like, you know, like stop, stop, stop.
• 37:41
  What do you call those things like word, like generation
• 37:44
  output limits, right? And often, sometimes you might
• 37:47
  have like a small output limit and you hit it and then a person
• 37:51
  doesn't really know why his workflow is not like working
• 37:53
  because like somehow the frameworks not exposing the fact
• 37:56
  that this has stopped for this reason or and, and I've seen
• 38:00
  people struggle with like things like that because a lot of these
• 38:03
  frameworks and stuff are sort of immature in that sense, right?
• 38:06
  They haven't, Yeah. Or for example, why not just
• 38:09
  continue, right? You know, why not have like some
• 38:11
  capability in there saying that if you do hit the, the output
• 38:15
  count of like 200 tokens, then just, you know, like then run it
• 38:19
  again and then again until you get your exact structure that
• 38:22
  you need out, you know, or this whole thing about like Jason
• 38:25
  everywhere. And you know, people struggle
• 38:26
  and cry about it not generating Jason.
• 38:30
  Like, you know, do you just have a key and a value?
• 38:32
  I have another key and a value, you know, I have key and a
• 38:34
  value. Do you really need this value to
• 38:35
  have another key and value? And then like, you know, this
• 38:38
  whole whatever you don't, I mean, or just leveraging
• 38:43
  multimodal. So the AX, like when you have
• 38:45
  the signature, these are inputs and outputs, right?
• 38:47
  So the inputs don't always have to be text.
• 38:51
  It could be because now these frameworks model support
• 38:54
  multimodal, they could be images.
• 38:56
  So EX has that built in and how do you make that work with DS by
• 39:00
  that's not in the paper or anything.
• 39:01
  So there's a lot of stuff that EX is like taking the paper and
• 39:05
  then build stuff on top. Like yes, those those things are
• 39:09
  still examples. So people forget those images
• 39:11
  are base 64 encoded or whatever. They are still part of the
• 39:14
  prompt. So they can still be captured as
• 39:16
  examples and there are still there are patterns in those
• 39:19
  images. So, you know, so you can have
• 39:23
  images as examples with text. So if you have like something
• 39:26
  that classifieds cat, you know, like, I don't know, cat faces,
• 39:30
  you know, or something like is this cat happy or sad or
• 39:33
  whatever. And then you want examples.
• 39:35
  And now if you run through this and you give it a few examples
• 39:37
  and get all the traces right now you have your, your examples
• 39:41
  contain like, you know, stuff about the cat, maybe some text
• 39:44
  BLOB and then like the image of the cat's face or something.
• 39:47
  So you so that like almost no frameworks doing stuff like
• 39:51
  that, right? I mean.
• 39:52
  Yeah, trying to just think about it and as how can we give it a
• 39:55
  richer prompt so that there is better understanding.
• 39:59
  And make that work with the whole DSPY examples, tuning and
• 40:03
  all of that, yeah. One thing that I know it took me
• 40:07
  a while to wrap my head around and now I can't even remember
• 40:11
  how it works, but I would love to know if you messed around
• 40:15
  with it. Was the idea of like compilers
• 40:19
  in DS Π or I don't think they're called compilers, they're called
• 40:23
  something else. But Omar was telling me to think
• 40:27
  of them as compilers. And that's where they, Omar and
• 40:32
  team have had to spend the majority of their work because
• 40:37
  it is something that most people just have a really hard time
• 40:41
  wrapping their head around. But it is very powerful.
• 40:44
  So I, I, I think what you're talking about is what they
• 40:48
  initially called teleprompters and then they realised people
• 40:52
  are getting confused and, and then they, I think they call it
• 40:55
  compilers now, I'm not 100% sure.
• 40:57
  I think that it's basically the whole bootstrapping stuff,
• 41:00
  basically a thing that takes the, the, the basic set of
• 41:03
  inputs and examples where you, so you have a model, you have a
• 41:06
  teacher model. Yeah.
• 41:07
  I mean, you don't even a teacher model is just basically a big
• 41:10
  model and you don't, you know, you're not really teaching a
• 41:13
  small model, you're just doing a context.
• 41:15
  You don't really need the second one, but sure.
• 41:17
  And, and then you set some examples on it in terms of like
• 41:21
  your top level input and your final output.
• 41:24
  So and then the compiler, basically it creates on it with
• 41:27
  some kind of strategy where it it runs that once and then maybe
• 41:31
  it like changes the temperature a little bit or it changes
• 41:34
  something else a bit and then runs it again with another sort
• 41:38
  of like adding some sort of some sort of entropy to the whole
• 41:42
  thing. And then like that keeps running
• 41:45
  it. And then you evaluate the
• 41:46
  output. You can like manually do it or
• 41:48
  you have some sort of function or another LLM and then you find
• 41:51
  like the the traces are in in the DSPY paper, they call it
• 41:55
  demos, demonstrations. That's that's basically the
• 41:58
  inputs and outputs of the whole chain.
• 42:01
  Yeah. That store that that is a
• 42:03
  compiler, but they have other compilers too.
• 42:05
  They have something called the pro so that I haven't they have
• 42:08
  a whole range of them and I haven't really dug in this this
• 42:11
  new research coming out about MIP Pro, which is actually kind
• 42:14
  of optimising the the So there is, I know I'm talked a lot
• 42:18
  about examples, but there are some level instructions.
• 42:22
  So there are like the prompt signature does get compiled into
• 42:25
  a prompt and there are some level of like text that the high
• 42:30
  level instructions that you do give it like, OK, here I need
• 42:33
  you to classify the faces of these cats or whatever.
• 42:36
  There is some instructions and the Mipro at least a cursory
• 42:41
  what I kind of glanced at the paper is a way of like doing
• 42:44
  this kind of creative trying to come up with better text that
• 42:48
  improves the output of the LLM with the like interesting
• 42:52
  classify the faces of these cats.
• 42:53
  It'll say, you know, whatever dwell on the features of this
• 42:56
  cat and come up with, you know, whether it's happy.
• 42:59
  I don't know whatever right. I mean and and just like
• 43:02
  changing the language there gives you better outputs.
• 43:06
  So there are there are there's research happening there solid
• 43:10
  stuff. I mean, I am and I definitely
• 43:12
  I'm going to incorporate all of that, but I've also like I've
• 43:17
  just focused on like, you know, people aren't even using this
• 43:19
  right now. You know, that's like, well,
• 43:21
  let's get people to use this and even just setting examples,
• 43:24
  you're already like, you know, 8080 X, like 100 X from where
• 43:29
  you were. Yeah, your capabilities go up so
• 43:33
  much. And I had a friend talk about
• 43:37
  all the different ways that you can think about using LLMS
• 43:42
  because a lot of times you you get stuck in your capability or
• 43:49
  what you think it should do and how it should do it.
• 43:53
  And he just has been compiling different personas that he tries
• 43:59
  to give the LMS like 1 is a critic, 1 is the, what was he
• 44:05
  calling it, the adapter. So if there and there's all
• 44:10
  these different personas and then when you look at that, you
• 44:12
  can get inspired and you can think, OK, well, maybe I could
• 44:17
  plug it into this little piece of my workflow or whatever.
• 44:21
  In the EX, those can be agents. Essentially.
• 44:23
  You could have these like personas, which are basically
• 44:26
  like agents. You can even give them some
• 44:27
  tools like you can say this has this persona and it can now it
• 44:31
  also is able to use these tools to do this thing.
• 44:34
  It's like, I don't know, the searcher or something in it and
• 44:37
  then the researcher and then you connected into like an archive
• 44:40
  API or something, you know, like, so it's, it's able to
• 44:42
  like, and then you improve that little guy over time, giving him
• 44:47
  a little more capabilities, improving him to, you know,
• 44:49
  like, like sort of like independently.
• 44:51
  And then you can plug him with the X into any workflow you
• 44:54
  want. So you like you're doing
• 44:55
  something with doing something with like stocks or like some
• 44:59
  papers research. And you can say, oh, here I have
• 45:02
  this like, like scientific researcher.
• 45:05
  Let me plug him in. And now suddenly everything
• 45:07
  improves because now it has that extra capability, right?
• 45:10
  That that little agent comes with his own context, his own
• 45:14
  examples, his own functions. He doesn't pollute the main
• 45:18
  flow, essentially. And when I'm building out that
• 45:21
  little agent I am, as you said, I'm tuning it so that I get the
• 45:26
  best performance for that specific small task that I'm
• 45:29
  going for, and then I can plug him in into the greater Lego
• 45:33
  Masterpiece. According to the DS by when he
• 45:36
  would get tuned into your workflow, so there would he
• 45:39
  would not come with his own example as a state.
• 45:41
  So essentially when you plug him into your workflow and then you
• 45:44
  run the DSPY compiler or the bootstrapper or whatever you
• 45:48
  want to call it on it, then he would get his like as the as the
• 45:53
  the the as the text flows from the top to the bottom.
• 45:58
  His examples would get preserved the ones that have, you know,
• 46:01
  result in good outputs for you in your workflow.
• 46:04
  Does. That make sense?
• 46:06
  Sort of like. Yeah.
• 46:07
  I, I'm just thinking about, So I guess maybe I'm not thinking
• 46:13
  about Asian examples per SE, but I'm thinking about one use case
• 46:18
  that we had on here like a year and a half ago when Phillip from
• 46:22
  Honeycomb came on and he said they have certain things in
• 46:27
  their product and the product, Honeycomb can be fairly
• 46:32
  difficult sometimes to figure out what, what and how to do
• 46:36
  what you need to do. And so for that they said, oh,
• 46:39
  we'll just prompt it, figure out and then ask what you're trying
• 46:44
  to do or tell us what you're trying to do.
• 46:45
  And then we will try and do it for you.
• 46:49
  And on some level, I can imagine that's a lot of agents going,
• 46:55
  especially if it's the doing it for you.
• 46:56
  If it's just, hey, let me pull up a document page that'll show
• 46:59
  you how to do this, that's one thing.
• 47:01
  But if it's let's go and now let's try and do what you're
• 47:04
  trying to do, it's a whole another thing.
• 47:07
  And so. In EX, there's some support for
• 47:10
  that. The way to do that again, you
• 47:12
  know, I'm coming back to the start where I just said, like,
• 47:15
  you got to chop the problem right.
• 47:16
  And now there are different ways to chop the problem up.
• 47:18
  So when initially you get a task that's asked of you and that's
• 47:21
  broad saying, Hey, I, I either I need you to do my taxes and then
• 47:25
  you come in, Hey, I need you to, you know, I don't know, classify
• 47:29
  my cat's face, right? They're two pretty different
• 47:31
  things. And now if you put a lot of
• 47:34
  agents, a lot, you get an LLM to make this decision, right?
• 47:37
  You can, I mean, that's one way to do it.
• 47:39
  You could have these, but there is something you can give like,
• 47:42
  you know, 20 function basic agents or functions, whatever
• 47:45
  you want to call them into the main thing and say, Oh, you're
• 47:47
  only your job is only to send the control to one of these
• 47:51
  guys, right, Basically hand off control flow.
• 47:53
  And then you know that that that agent sort of takes off from
• 47:56
  there or there's this concept called a Symantec router or
• 48:01
  routing. So that there is this whole
• 48:02
  world around routing LLMS. There are start-ups that are funded
• 48:05
  here. And AX comes with something
• 48:08
  called a Symantec router where basically it uses embeddings to
• 48:12
  to classify sort of like find the closest match to send your
• 48:18
  task down. So if you send it like a BLOB,
• 48:21
  you say, I want you to like, you know, classify cat faces for me,
• 48:24
  then the then the then there is like a router set up which says,
• 48:28
  oh, anything to do with cats, cats faces.
• 48:30
  I want you to call this function sort of or anything do with like
• 48:34
  taxes call this function, you know?
• 48:36
  But that assumes you've seen the question before, right?
• 48:41
  So no, when the question, I mean you have to you're working with
• 48:45
  a finite set, right? You have functions that I mean,
• 48:48
  there is only finite number of things you can do.
• 48:50
  You cannot like just you know, if you ask the honeycomb thing
• 48:53
  to go, you know, I don't know, find your grocery list or hotel
• 48:56
  bookings, it won't be able to do that, right?
• 48:58
  So there is going to be a finite list of things.
• 49:00
  So once the question comes in, you embed it and then you have
• 49:04
  this router which already has embeddings in it for each one of
• 49:07
  its routes you find, you do like a dot product, find the most
• 49:11
  closest similarity and you send it down that and say, OK, let's
• 49:14
  activate that pipeline. So the the advantage here is
• 49:17
  this dirt cheap, the embeddings that cost like you literally
• 49:21
  it's it's almost free and there's no overhead of an LM
• 49:25
  being engaged until it like hits that path, right?
• 49:28
  So now you can have four more number of things possible.
• 49:33
  You can have hundreds of things that could potentially or
• 49:37
  thousands that your your initial input couldn't go down.
• 49:44
  And all these workflows have been created and then tuned
• 49:50
  through this through these methods that we're talking about
• 49:53
  and. Exactly, Yeah.
• 49:55
  So. So it's really not like.
• 49:58
  The workflows are just like code.
• 49:59
  It's just code. It's like functions, right?
• 50:01
  It's like a function which has more functions in it.
• 50:03
  That's you know, that's the only way to think about.
• 50:05
  And because the signatures are so tiny, you want to call and
• 50:08
  use these. You just want to, it's almost
• 50:10
  one line to engage ADSPY programme in your code.
• 50:13
  And it looks like a function call.
• 50:15
  You have some inputs that you pass in.
• 50:17
  You call this forward function because like that's why DS π is
• 50:20
  in Pytorch and Pytorch had the forward function.
• 50:24
  I kept that. I'm still like conflicted.
• 50:26
  I think you should call it run or something.
• 50:27
  But you know, it's not like like none of the prompt people care
• 50:31
  about Pytorch, right? I mean, it's too low level a
• 50:34
  thing. But anyways, so it's just, it's
• 50:38
  so easy to invoke, so people want to use it.
• 50:40
  It's like a function call. You just get some inputs, you
• 50:42
  wait for the output, you know, inside it, they might do a whole
• 50:45
  bunch of stuff, you know, engage your examples, do multiple
• 50:48
  function calling, you know, do assertions, error correction,
• 50:52
  you know, whatever, whatever. And and boom.
• 50:55
  And finally you get your like three things that you want out
• 50:58
  and then you call the second function, you pass those as
• 51:00
  inputs to that get it out. You might even call 3 in
• 51:03
  parallel, like use some, you know, whatever async thing and
• 51:06
  say, oh OK, I'm going to send it down three more paths and then
• 51:08
  I'll wait for those to complete and then I'll get the answer and
• 51:11
  then it's essentially a graph or whatever.
• 51:13
  It's like a small programme, you know, but it's all LLM based.
• 51:18
  And all of these different workflows, you're shepherding
• 51:23
  them along or you're tuning them to make sure that they are
• 51:26
  giving you the best output. So it's not like these workflows
• 51:30
  are just created by the LLM and then you stand around and say
• 51:37
  like, is it going to give me the right thing?
• 51:39
  No, you're, you're setting examples at each one of these
• 51:42
  subtasks, the best examples you can have, and you keep improving
• 51:46
  them over time. And that makes each one of these
• 51:50
  subtasks solid because. The.
• 51:52
  Subtask is like it's its own clean context, right?
• 51:55
  Has to have its own set of examples.
• 51:56
  It cannot have the examples of the top level task of the OR the
• 52:00
  the task besides it or the task below it.
• 52:03
  It has its own unique inputs, its own unique outputs.
• 52:06
  It could be a subtask in your subtask somewhere else, but it
• 52:09
  sits independently. It says my job is to take this
• 52:12
  text that's coming in and classify it into these four
• 52:14
  things. Maybe or or my job is to extract
• 52:18
  out the the right, I don't know, tax sections based on this
• 52:23
  sustain. So now it needs it's right
• 52:26
  examples for. You know, so cool, man.
• 52:29
  Well, thank you a turn for doing this because you're opening up
• 52:32
  my eyes to ways that agents are being worked with that I
• 52:41
  obviously I've talked to Omar before, but I'm, I love seeing
• 52:44
  you take that inspiration and then build on top of it and say,
• 52:50
  here's as you mentioned, here's the open paper, here's what we
• 52:54
  can standardise against and now what can we build with it?
• 52:58
  Exactly. Yeah, that's you're right on.
• 53:00
  Let's let's like let's start building.
• 53:02
  That's basically it. You know, I'm their help.
• 53:04
  Let's let's build all these things.
• 53:06
  Let's fill into that valuation. Let's let's prove Goldman wrong.
• 53:11
  I. Love it.