Latest / Agentic Conversations (formally mlops.community) / FrugalGPT: Better Quality and Lower Cost for LLM Applications // Lingjiao Chen // #172

Transcript

FrugalGPT: Better Quality and Lower Cost for LLM Applications // Lingjiao Chen // #172 · Agentic Conversations (formally mlops.community)

• 0:00
  Hey everyone, I'm Lingya. I am a CS PhD student at
• 0:05
  Stanford University. I'm working with Matthew Zaharia
• 0:08
  in Jamestown and more recently I've been working on the machine
• 0:12
  learning as a service market and more specifically land after
• 0:16
  model as a service marketplace. I I drink coffee a little bit
• 0:22
  and typically just go to stop bars and buy a tala table.
• 0:27
  Yo-yo, yo, welcome back to another ML OPS community
• 0:31
  podcast. I'm your host, Dmitry iOS.
• 0:33
  And what a conversation today. Let's go over a few of the key
• 0:38
  points that we were talking about, because WOW in GAO is.
• 0:45
  Spot on. I've been a fan since I saw
• 0:48
  Frugal Mill, but then he came out with Frugal GPT and we go so
• 0:52
  far into that paper. He basically walks me through
• 0:57
  each part and I had to sit there and stop him a few times and
• 1:01
  say, wait, am I understanding this correctly?
• 1:04
  That there's this, there's that. All of the fun stuff and the
• 1:07
  technical implementations of Frugal GPT?
• 1:11
  He goes through and he does it with so much patience for me.
• 1:15
  Everyone who was struggling on a few of these concepts to really
• 1:20
  understand it. Hope you all enjoy it and for
• 1:23
  those who are wondering who Lingao is.
• 1:25
  He is working with Matt Tay. I first got tipped off to him
• 1:29
  because when Matt Tae, the founder of Databricks, for all
• 1:33
  those who do not know you like how it we're on first name basis
• 1:37
  now, Matt says Zaharia. He came on the podcast and after
• 1:41
  he came on our podcast I asked him.
• 1:44
  So who are some people that I should be interviewing besides
• 1:46
  you and everyone at Databricks? And he mentioned Frugal GPT.
• 1:52
  While he mentioned at that time it was only frugal meal, but
• 1:54
  about a month later he came out with frugal GPT.
• 1:58
  And since then, he has also come out with a paper that stirred U
• 2:05
  quite a bit of controversy talking about how.
• 2:08
  ChatGPT and GPT 4 have been getting worse overtime.
• 2:13
  It's something that we've all been feeling and we've all
• 2:16
  intuitively thought, but they put some numbers around it and
• 2:21
  they released the paper and it caused quite a bit of stir.
• 2:24
  So yes, this is that author. Let's dive into it with him, the
• 2:30
  Stamford pH. D candidate, and both.
• 2:35
  Before we do, here is a quick word from our sponsors, Quantum
• 2:40
  Black. Alright listeners, I have the
• 2:43
  pleasure of announcing right now with you that the EMOS community
• 2:47
  and Quantum Black, the AI branch of McKinsey and Company, have
• 2:51
  partnered up. And for this special partnership
• 2:54
  I brought off Nair Khan. Your official title is Partner,
• 2:58
  but your unofficial title is Professional Data Nerd Nyer,
• 3:03
  what exactly is Quantum? Black really, really great to to
• 3:06
  be here. Let me start briefly and say
• 3:08
  look our origins. We started off in Formula One,
• 3:11
  we realised. Beyond that, we could use data
• 3:14
  in very smart ways off the track to have big impact on track
• 3:17
  times. And as I know we took that
• 3:19
  mentality of using data in a in very smart ways help companies
• 3:24
  and industries on their toughest AI use cases really using AI to
• 3:27
  drive impact. And we've done this across a
• 3:29
  whole myriad of different industries.
• 3:31
  I've mentioned Formula One and sports, but there's also natural
• 3:33
  resources, health and life sciences, consumer and so on and
• 3:37
  so forth. We are about 2000 colleagues
• 3:40
  globally in 40 different countries and we have a whole
• 3:42
  host of different types of. Skills and technologies, data
• 3:45
  science, data engineering as you know, but also DevOps, software
• 3:48
  engineers, product design, SRE, cloud.
• 3:51
  So we bring and fuse all these different types of talent
• 3:54
  altogether. I had the pleasure of going and
• 3:56
  meeting you in person about a year and a half ago and I went
• 4:00
  to the Quantum Black office in London and I loved it there
• 4:04
  because you set up a nice little Lego welcoming Matt for me.
• 4:08
  That said, ML, OBS and we also had some whiteboarding sessions
• 4:11
  and you blew my mind with some of the projects that you are
• 4:14
  working on internally. Can you tell us a little bit
• 4:16
  about what makes Quantum Black special and that vibe that I
• 4:19
  felt when I was there? So I think you know we all
• 4:22
  thought of McKinsey and company, but one of the things that we've
• 4:24
  maintained as QB is trying to keep our culture and there's two
• 4:27
  different things. It's our always like learning
• 4:30
  type of mindset and being very curious about the problem, but
• 4:33
  also curious about the solution. There's a lot of great
• 4:36
  technology and solutions out there, but there's also a lot of
• 4:38
  white space that hasn't been sold and I think our, our, our
• 4:41
  teams really get down and really are happy to try to drive
• 4:44
  through solutions with that. The second is.
• 4:46
  These fun aspect you mentioned Lego and Lego is part of our DNA
• 4:49
  and really doing things in new and creative ways.
• 4:52
  I think this is really not only with our our the organisations
• 4:55
  we work with is very energising, but also our people, they
• 4:58
  really, really dig this way of working.
• 5:00
  Incredible. So if anyone wants to go deeper
• 5:03
  down the rabbit hole and learn more about what you all are
• 5:05
  doing at Quantum Black, we're going to leave a link in the
• 5:08
  description. Quantum Black is always on the
• 5:10
  lookout for incredible talent, so if you think you fit, the
• 5:13
  bill will also leave a link for you to contact them.
• 5:18
  Thank you, Nair for coming and joining us.
• 5:20
  Let's today I am super excited to talk to you man, because I've
• 5:33
  been reading not only frugal ML, but I read your frugal GPT paper
• 5:38
  that you put out recently. And I know we were organising
• 5:41
  this podcast when you were in the throes of putting that paper
• 5:46
  out. And then I saw that you put it
• 5:48
  out. You said, Oh yeah, I got
• 5:49
  something really happening in May.
• 5:50
  So maybe we could do this in June.
• 5:52
  And then in May, I think you put out the paper or April.
• 5:55
  Put it out. And I absolutely loved it.
• 5:57
  I know we're gonna get really deep into it today, and you're
• 6:02
  gonna explain to me all the pieces that I didn't quite
• 6:04
  understand because it's a pretty beefy academic paper and I just
• 6:10
  thought it was so useful considering so many people talk
• 6:14
  about cost when it comes to large language models and frugal
• 6:18
  LGBT is an awesome way to think about costs and think about how
• 6:22
  you can save money. But before we get into that.
• 6:26
  And I think we should just start off with saying that you're
• 6:29
  currently doing a lot of amazing work at Stanford.
• 6:33
  You're working with Matt Tay, who is a friend of the pod.
• 6:36
  He's come on here. And I want to hear your story.
• 6:39
  Like, how did you get to be doing all this research at
• 6:43
  Stanford? Thank you Dimitris.
• 6:46
  I'm super happy to be here talking with you.
• 6:50
  I'm also they're very glad that you like the productivity you
• 6:53
  paid for. So we should definitely go over
• 6:55
  the paper in the next few minutes.
• 7:00
  What is the first question? How did I come into Stanford?
• 7:06
  I I think I'll super lucky. I was super lucky.
• 7:10
  Like I didn't start as a CS major when I was an undergrad,
• 7:16
  but then I went to consent to do a master.
• 7:21
  Dear, I was focusing on databases and then apply for PhD
• 7:25
  programme. An matey was very kind to get me
• 7:32
  accepted. So yeah, it's it's just, I was
• 7:36
  super lucky. I get help from a lot of amazing
• 7:39
  people and their size. They're very smart, full of
• 7:45
  wisdom and just give me a lot of help.
• 7:47
  I was so grateful to all those people.
• 7:51
  Alright, so what's the day-to-day like in your research
• 7:54
  and what draws you towards? More industry specific topics,
• 8:00
  because I know there's a lot of cutting edge research out there
• 8:03
  that is really trying to optimise things that aren't
• 8:06
  necessarily useful when it comes to actually using these LLM's in
• 8:12
  the wild. But it feels like you have your
• 8:15
  eye on things that can be useful in the real world.
• 8:19
  And that's not a dig at other researchers.
• 8:21
  It's just more of like, I see what you're doing and I see it
• 8:25
  as something that tends to have a different.
• 8:28
  Attention. I think one nice part about
• 8:32
  being in a university or being at academia is that you have the
• 8:37
  freedom to pursue whatever you like.
• 8:39
  So I think people could have totally different opinions on
• 8:42
  what to do or what is a good direction to pursue.
• 8:48
  Personally, I do have the intention to work on things that
• 8:54
  could be practically useful. That's actually part of my
• 8:59
  research principle. If you ask me in the what do you
• 9:02
  want to work right in the next few years, right.
• 9:07
  I would say I want to work on things that are like
• 9:11
  mathematically profound but also practically useful so that other
• 9:18
  people would be able to benefit from this kind of research.
• 9:23
  It's just, you know, it's just a few super nice.
• 9:26
  When I say, oh, people could benefit from this work or they
• 9:30
  could, you know, get something from this, this paper.
• 9:34
  That's just my personal take. Excellent.
• 9:38
  Yeah. I'm not gonna lie, man, I think
• 9:40
  you are. You may be the first researcher
• 9:44
  that we actually have on here, like practising researcher.
• 9:47
  We've had a lot of ex researchers that have crossed
• 9:50
  over and gone into academia but or into the industry.
• 9:54
  But I think you're the first one that we've had that's writing
• 9:58
  papers and doing academia full time.
• 10:02
  And it is because of these things that you're talking
• 10:05
  about, like you're thinking about how to make an impact.
• 10:07
  On the the industry, not just the academic world.
• 10:11
  So let's jump into this idea of the research that you're doing
• 10:18
  and go a little bit deeper onto like one thing that you told me
• 10:21
  before we hit record, which is that you do research on
• 10:25
  inventing data management inspired abstractions.
• 10:29
  What does that even mean? It sounds very cool, but I'm not
• 10:33
  sure I totally understand it. Yeah, that's a that's a
• 10:36
  fantastic question. So I I'm going to start by
• 10:40
  giving a little bit of background right.
• 10:41
  Like I mentioned I actually started doing research when I
• 10:44
  was a master and there I actually focus on like you know
• 10:48
  database related stuff that's pretty much data management
• 10:52
  research. So what that basically means is
• 10:54
  that you want to process your data and user friendly and
• 10:58
  system friendly way and their typical fathers is like you know
• 11:02
  you're doing you know some some transaction processing or some
• 11:06
  additive processing there. So that's a great experience, I
• 11:11
  love that experience. But after doing that I realised
• 11:14
  that ohh you know, there is actually a lot of a connection
• 11:19
  between the data management community and the community.
• 11:25
  More specifically the very poor part of both like data
• 11:32
  management and machine learning is the data itself.
• 11:37
  If you think about this ML word, yeah we have like fancy Mr them.
• 11:41
  We have great hardware optimised for those ML workload, but
• 11:46
  without data we couldn't do anything, right?
• 11:49
  So the main philtre within data and data management is kind of
• 11:53
  the same thing, right? Like yeah, we have those fancy
• 11:55
  tools, but you know, the key is the data.
• 11:57
  We want to actually process this data and then by processing it's
• 12:01
  like we want to get some insights from this data, right.
• 12:05
  So I think that a big connection between data management and
• 12:08
  machine learning which is the data part, how do we deal with
• 12:11
  the data in an efficient and user friendly way.
• 12:15
  So I think that's kind of the key motivation like we want to
• 12:18
  see hey data management steer for a couple of years, it has
• 12:22
  the same focus as more people have.
• 12:25
  So now what's a good way, right, to bridge those two-part And why
• 12:32
  would I want to do like like what is a data management
• 12:35
  abstraction or systems or outright said even main and what
• 12:39
  it means is that and I'm a word and MO word, you don't have to
• 12:44
  always view your research or the whole community as hey we're
• 12:50
  inventing new algorithms to process data or we're inventing
• 12:54
  you know new hardware or systems to make it faster.
• 12:59
  Another way to view it right is to say, hey, we actually have
• 13:03
  tonnes of data and we want to guide inside out of it there and
• 13:08
  how do we do that in a nice way and that's data management,
• 13:12
  right. So I think data management is
• 13:15
  basically a aspect and angle to view this machine learning
• 13:21
  research, machine learning, engineering, machine learning
• 13:23
  and deployment. It's just like a different way,
• 13:26
  a different angle of data management and where we can
• 13:30
  actually draw a lot of insights from data management community,
• 13:33
  for instance, was a good way to manage your data, to organise
• 13:37
  your data so that it fit into the machine learning workload
• 13:41
  much better like hopefully right?
• 13:44
  Or if you view those ML workload as a data workload, what
• 13:50
  particular similarity or what particular connection could see
• 13:56
  from Dan management and use that to help you.
• 13:58
  If I'm understanding this correctly, you are agnostic on
• 14:02
  the type of data. It doesn't matter if it's.
• 14:06
  Structured, unstructured, whatever complex data is can be
• 14:12
  called. You're just saying what are
• 14:16
  better ways that we can manage any type of data and you're also
• 14:21
  a little bit agnostic on the approach to managing it too.
• 14:24
  Is that the what I'm hearing is like we don't necessarily need
• 14:29
  to say that just because it's been done like this in the past
• 14:33
  that's the best way that we do it.
• 14:36
  Yep. Yeah.
• 14:37
  Well I want to add us to a database or data management by
• 14:43
  definition is data type agnostic.
• 14:48
  Like it is definitely true that a lot of the, you know where no
• 14:52
  one database data management research is focused on
• 14:55
  relational data or like tabular data, right.
• 14:59
  And you're pretty much familiar with traditional SQL language to
• 15:04
  process those additional data set, that's absolutely right.
• 15:09
  But actually even at the very beginning the data management
• 15:12
  community view like they like build data as like any types of
• 15:19
  data like you could you could deal with say relational data,
• 15:24
  you could also deal with the graph data and they're like we
• 15:28
  actually have say image or video or audio like multi media data
• 15:37
  or they call it objective database or objective relational
• 15:41
  database, right. So what I want to say is that
• 15:44
  you like database was typically viewed as saying, oh, it's
• 15:47
  dealing with relational tabular data, but actually it has a much
• 15:51
  broader scope. 100% so then you were getting into Frugal GT and
• 15:59
  how this was a perfect example of of what you were talking
• 16:02
  about. I think there's a there's a very
• 16:05
  interesting connection between like frugal ML, frugal tribute
• 16:09
  kind of thing to the data management or database
• 16:11
  community. One of the end goal is all those
• 16:16
  fruity programmer work is focused on the cloud, our as a
• 16:24
  service marketplace. Rightly all those training
• 16:27
  companies like Open AI and one Microsoft, right, they provide
• 16:32
  us e-mail service as a cloud service right across the cloud.
• 16:36
  But the data management community already say like cloud
• 16:41
  computing cloud for big data like 20 years ago, right?
• 16:46
  Like while we're not, while we're not inventing, you know,
• 16:51
  the sky kind of stuff. There was actually a notion
• 16:56
  right after cloud computing that the sky computing, right.
• 17:00
  If you're saying, hey, you have pumped up clouds and now we
• 17:04
  could automate the sky on top of that, right?
• 17:08
  Let's try to abstract the whole cloud into a sky so that people
• 17:13
  don't need to worry about the internal details of all those
• 17:17
  cloud services. Cloud computing, we can only
• 17:20
  offer it on the top abstraction. So for programmer for GPT, it's
• 17:26
  very similar like in in spirit, right?
• 17:28
  You don't need to worry about the details path desk cloud
• 17:32
  allow providers services, you can just operate on top of that.
• 17:38
  So in that sense you see that's kind of the sky I'm not seeing.
• 17:43
  So basically it's like yeah we're in data management, we
• 17:46
  have cloud data, cloud computing.
• 17:49
  Now people say hey we can view sky as abstraction to handle
• 17:53
  that. And in programmer will say, hey,
• 17:55
  you have this cloud services, cloud ammo, now we can actually
• 17:58
  view another abstraction to operate on top of that.
• 18:03
  So that's kind of a very similar, right.
• 18:06
  Another aspect to look at is is like, you know, there's there's
• 18:09
  a big trend in the data community to to think about what
• 18:13
  you can do with the was a data marketplace, right?
• 18:18
  So there's a huge interest in buying and selling data.
• 18:22
  Yeah, right. And it's the same thing, right?
• 18:24
  Like what is data? Is data necessarily relational
• 18:26
  stuff, or could data also be the predictions produced by MO
• 18:33
  service? And we should also draw
• 18:36
  inspiration from that aspect. If they just doesn't have to be,
• 18:41
  you know, relational tabular data, if it could also be a, you
• 18:45
  know, predictions, then what you can do with that like data
• 18:50
  market. Yeah, right.
• 18:52
  And that sounds like mill market like cloud ML services, it's
• 18:56
  just another version of data market?
• 18:59
  Whoa, wait a minute. Let me see if I'm understanding
• 19:01
  this correctly. It's you're basically saying,
• 19:03
  hey, these APIs from Open AI or whatever anthropic you name it.
• 19:08
  They are just another data provider.
• 19:10
  In a way, it's like a data marketplace and you're using
• 19:14
  Open AI as one of the data marketplace providers.
• 19:19
  Yep, that's that's one way to to to think about it like this
• 19:23
  services. So instead of leaning heavily on
• 19:27
  OK, this is going to be the be all, end all and I'm going to
• 19:31
  just do everything through Open AI.
• 19:33
  It's like no Open AI is going to be one of my data streams and
• 19:38
  one of my providers, but then I also have 10 other ones that I'm
• 19:43
  using and I can then build some kind of abstraction on top of
• 19:48
  that. OK, now I'm starting to see
• 19:51
  more. OK, so we should probably break
• 19:53
  down what Frugal GPT is and also maybe breakdown what Frugal ML
• 19:57
  is because we may have put the cart before the horse.
• 20:00
  And so anybody who hasn't read Frugal LGBT paper, this is my
• 20:04
  understanding. And you you tell me if my
• 20:06
  understanding is correct and where I'm getting it a little
• 20:11
  bit lost. But what you're doing and I love
• 20:14
  the idea of having these three different circles in a Venn
• 20:18
  diagram. And so that made it very.
• 20:21
  Easy for me to understand. Basically if someone is making,
• 20:27
  let's say someone wants to interact with a model or a large
• 20:31
  language model, for that case you're saving that person money
• 20:36
  because you're doing 3 different things before and after and
• 20:40
  during the interaction with the large language models.
• 20:43
  And so the first thing is you're running some algorithm or
• 20:48
  something to make sure that the prompt is fully optimised so
• 20:51
  that the prompt it's not using unnecessary amounts of tokens
• 20:56
  and not spending unnecessary amounts of money.
• 20:59
  So that's the first thing. It's like, how can we fully
• 21:01
  optimise this prompt to make sure that it is running at the
• 21:06
  most in the most efficient way. So we'll put a pin in that
• 21:10
  because I want to ask you questions about how you do that
• 21:13
  exactly. But let me just explain the
• 21:14
  whole thing before we go back to that part.
• 21:17
  Then the second piece is all right, Now that we have this
• 21:19
  optimised from, we're going to see if we can fulfil the
• 21:25
  questions that are being asked through open source models
• 21:28
  first. Because those are cheaper and
• 21:30
  you don't have to pay per token and hit the API, and especially
• 21:34
  if you get to a certain scale, it may become unreasonable.
• 21:38
  And so you see how much of the question or the task can be
• 21:44
  serviced by these open source models.
• 21:48
  And then if there is a delta between what you need done and
• 21:54
  what the open source model can get to, then you say, alright,
• 21:58
  now we're going to go to these APIs and we're going to say out
• 22:02
  of all the APIs, let's figure out which one.
• 22:06
  And this is where I think, and correct me if I'm wrong, this is
• 22:09
  where the frugal ML comes in, where you say out of all the
• 22:12
  API, we're going to use machine learning and AI to go and figure
• 22:18
  out what the cheapest model API call is going to be here.
• 22:23
  And so we say, alright cool, this one, whatever, Open AI is
• 22:28
  going to be the cheapest for this query.
• 22:30
  We're going to use Open AI or actually Claude is going to be
• 22:33
  the cheapest. So let's go and use that.
• 22:35
  So that's my understanding of it and it's almost like like you
• 22:39
  said, it's it's an abstraction layer on top of all of that.
• 22:43
  So that if you are trying to interact or get something done,
• 22:48
  you can save money because it goes through all these Nets and
• 22:54
  it's able to optimise your query.
• 22:57
  And then it's able to say, alright, let's see how much we
• 23:01
  can do with open source. And then if you can't get it all
• 23:03
  done from open source or you can't get anything done from
• 23:05
  open source, it goes to an API. But it makes sure that the API
• 23:09
  that it calls is the cheapest and most reasonable price due to
• 23:14
  the prediction from the machine learning model.
• 23:18
  What else? What did I miss?
• 23:20
  I think the very high level understanding is super precise.
• 23:27
  Yes, the that's what I'm talking about.
• 23:30
  But tell me all right now that's high level, but now let's go low
• 23:33
  level. I think you probably missed the
• 23:35
  third part of the. As the fragility paper.
• 23:41
  So so as I mentioned, the first two like how to deal with the
• 23:45
  language models prompt, how to deal with the multiple service
• 23:50
  providers. There's another.
• 23:53
  Little hard about how to approximate those sort of
• 23:58
  models. Like like.
• 24:01
  One of the key things here is that there there could be a lot
• 24:06
  of redundancy among your queries.
• 24:11
  You could actually use some system techniques or ideas, say
• 24:15
  caching, to deal with that. Like like for instance, right?
• 24:20
  If you use moms for say a search engine right?
• 24:24
  Or to support your customer service.
• 24:30
  You know from the users perspective at your.
• 24:36
  Your process queries could be quite similar, or there could be
• 24:40
  a large scale in your query distribution at some point,
• 24:44
  right? Say you know, let's say it's a
• 24:46
  election of Seed, and there are a lot of queries about elections
• 24:51
  for automotives, and they're pretty much asking the same
• 24:54
  thing, right? And then you don't have to.
• 24:57
  You know, ask a lot of model multiple times the same
• 25:01
  question. You can actually ask them once
• 25:03
  or a few times. Or locally and whenever you need
• 25:07
  it, you can reuse it. So I think that's another.
• 25:13
  Missing part from from your understanding, but that's
• 25:15
  totally OK. Like I think this first two-part
• 25:17
  are are pretty close. Yeah, that's a huge cost saver.
• 25:23
  That is very true because at the end of the day if people are
• 25:26
  asking the same question over and over, you don't need to ask
• 25:29
  the LM over and over. So if you have something which
• 25:32
  is probably a very popular use case for LM, it's like that chat
• 25:37
  with your data type use case and so whatever your data is, you
• 25:42
  slap a vector store together with an LM and you have a chat
• 25:48
  with your data. It's very easy to get up and
• 25:49
  running. I think even that's like one of
• 25:51
  the most. It's like the hello world of
• 25:54
  working with MLM's in a way. So you're you're doing that and
• 26:01
  that is like the first piece, then I I understand.
• 26:06
  And so from there if you say, alright first we're gonna just
• 26:09
  check and see if we have the answer cached because then we
• 26:12
  don't even need to go through the whole mess of trying to
• 26:15
  figure out if this needs to go through an LM.
• 26:18
  And then if it's not cached, then it's like, OK, now we go
• 26:21
  down the rabbit hole and we go deeper, deeper down the rabbit
• 26:24
  hole of what I was explaining earlier.
• 26:27
  Yep, Yep, OK, cool, makes 100% sense.
• 26:32
  And now what I want to know is when, when we're talking about
• 26:36
  that first step of optimising the prompts, how do you do that?
• 26:41
  Yeah, that's a great point. So first of all I want to say
• 26:46
  like there always exists more than one solution to to to any,
• 26:50
  in my opinion to any research questions, right.
• 26:52
  So I I don't want to say hey, we we we exhausted all possible
• 26:57
  answers. No, I just want to give like.
• 26:59
  Examples so that we kind of see like what are some reasonable
• 27:04
  and feasible ways, right? OK, so going back to your point,
• 27:08
  I think a lot of ways to optimise the prompt one.
• 27:14
  One interesting idea is to do certain query concatenation to
• 27:22
  to. To minimise or to reduce size of
• 27:25
  front. So what that means?
• 27:29
  Is that you take two or a few queries.
• 27:35
  And compress them into a large one.
• 27:39
  And then send this large query to the.
• 27:44
  Large number model so that you can actually allow those two
• 27:50
  queries to share some similar part in the prompt and therefore
• 27:56
  reduce the. The size of it.
• 28:01
  OK, wait, I guess I'm gonna too complicated.
• 28:04
  No, no, no, I I follow you. So if I'm understanding it
• 28:07
  correctly, the the thing that I guess I'm not understanding is
• 28:11
  that, alright, so you're taking two different queries and you're
• 28:13
  saying there's some overlap here.
• 28:16
  It's not the same question. So it's not going to hit that
• 28:18
  vector store caching model, but there is some overlap here and
• 28:24
  we're expecting the output to be kind of similar or we're
• 28:32
  expecting the output to have overlap.
• 28:34
  So why don't we put these two prompts together and then send
• 28:38
  them as one prompt? That's correct, but I was
• 28:43
  actually thinking about something even simpler.
• 28:46
  Ohh. Alright, that's even better.
• 28:48
  I like that. I guess it's probably easier if
• 28:51
  I if I if I show you one page on the slide.
• 28:55
  So yeah, sure what that show us what you got and for those that
• 28:59
  are listening all make sure to describe what I'm seeing on this
• 29:03
  slide so that they they can understand.
• 29:07
  Yeah. Hey everyone.
• 29:08
  My name is Aparna, founder of Verizon.
• 29:10
  And the best way to stay up to date with ML OPS is by
• 29:13
  subscribing to this podcast. All right, sweet.
• 29:18
  So we got that sorted out. We are now looking at a slide
• 29:21
  and there's all kinds of crazy math or what looks like prompts,
• 29:26
  questions, answers what what are we seeing on here?
• 29:29
  Yeah, I I I would say it's not really a mass, but we do have
• 29:33
  some symbols there. OK so I want to use this slice
• 29:38
  to show how one of the particular.
• 29:44
  Around automation, the query concatenation words, so that's
• 29:49
  actually pretty straightforward. So let's say you have two
• 29:52
  queries, right? One query is what's the result
• 29:55
  of end to end or at high temperature?
• 29:58
  You also have another query what helps pretty high.
• 30:01
  And you want to add a few in contacts example to those
• 30:08
  queries. Right to to have to our arms
• 30:14
  think about what's the right answer.
• 30:16
  So let's say you have like 4IN context examples appended to
• 30:21
  each of those questions. And this is just so the
• 30:26
  listeners understand what we're looking at is we've got a layer
• 30:30
  that is like a prompt layer and it says like Q1 plus A1 comma,
• 30:36
  Q2 plus A2 comma, Q3 plus a three comma, Q4 plus A4 and
• 30:40
  those. Then right under that we've got
• 30:44
  a a layer that says a question, which is what you mentioned
• 30:47
  earlier, what is the result of N2 and O2 at high temperatures.
• 30:52
  N2 and O2 I'm guessing are like the good old periodic table
• 30:57
  we're talking. That's what we're referencing
• 30:59
  here like nitrous or not, I can't remember my chemistry
• 31:04
  class but into is one of those. And then O2 I guess is oxygen
• 31:09
  and yeah so and then right under that directly under that we've
• 31:13
  got the same kind of thing where there's a prompt one plus A1
• 31:17
  comma Q2 plus A2 etcetera, etcetera.
• 31:19
  And then directly under that is the question what helps prey
• 31:24
  hide and both of those your smashing into.
• 31:29
  So all four of those things, or just the prompt you're smashing
• 31:32
  into the query, concatenate or. Right, right.
• 31:37
  And the query concatenation is what you were talking about,
• 31:39
  where you take both of those and you put them together.
• 31:43
  Right, right. We we put us two questions
• 31:47
  together and we compress the the income tax.
• 31:54
  Example So um. The new prompt only contentious
• 32:00
  Q1 plus Q2A1 plus A2Q3 plus Q4A3 plus A4.
• 32:07
  Only once. Right, so now you can actually
• 32:11
  compare the the new query and the old query.
• 32:16
  Right and and old quarry to process those two questions.
• 32:21
  I need the alms to to process the in context examples twice.
• 32:27
  Right. Because you have two questions.
• 32:30
  OK. So so if I'm understanding this,
• 32:32
  basically what you're saying is the money saving comes in
• 32:36
  because the prompt, which is the Q1 plus A1Q2 plus A2, that kind
• 32:42
  of thing you're only doing once. And then you're throwing in
• 32:45
  these different questions with those prompts and you're seeing
• 32:51
  the output and the output is going to distinctly have like a
• 32:57
  clear line as to where the question one and the question
• 33:00
  two car are isolated. They don't.
• 33:04
  There's no blend in those two answers.
• 33:06
  I guess is my biggest question is like doesn't GPT And so for
• 33:11
  people listening to. So basically it goes in these
• 33:14
  these two things go into the query concatenated and you're
• 33:16
  left with a sandwich of well not quite a sandwich.
• 33:21
  So it goes into the concatenated and you're left with the prompt,
• 33:24
  just one layer of the prompt of the Q1 plus Q2.
• 33:27
  And then you have both of these questions together and then it
• 33:32
  goes into CPT four and then the output is 2 circles where you
• 33:37
  have Q1 answer and then you have Q2 answer.
• 33:41
  And so my biggest question is, doesn't Q1 answer in Q2 answer
• 33:47
  get blurred? No, it won't get blurred big
• 33:52
  because you could add. You you you could add certain
• 33:58
  identifier right? So one way to add it right?
• 34:01
  So you you could write but you could have format your prompt in
• 34:05
  a way such that that the output would be something like answer.
• 34:12
  And then Note 2 and then answer um come flat or like answer 102
• 34:19
  an answer 2 blah blah blah. So this answer one answer two
• 34:24
  would be the identifier that separate those two answers.
• 34:29
  So my biggest question is aren't the two answers going to blend?
• 34:35
  And you're saying no because you can do specific things to make
• 34:39
  sure that doesn't happen. Yeah, so so the basic idea.
• 34:45
  Or the basic approach right is to add some identifier to
• 34:50
  separate the two? Answers.
• 34:54
  Ohh of course you just ask GT to say tell us what the first
• 35:01
  answer is and tell us what the second answer is.
• 35:04
  Alright, cool, we're back a day later.
• 35:06
  I'm in different clothes. You're in different clothes
• 35:08
  because I had a little problem yesterday with my Internet, but
• 35:12
  not gonna let anything stop me from learning from you.
• 35:16
  It's not gonna happen, man. So back to this slide that
• 35:20
  you're showing on the screen. I had the day to think about
• 35:24
  what exactly it was that you were saying to me and I totally
• 35:28
  understand it now. So basically when you're
• 35:31
  economising the prompts, what you're doing is you're saying
• 35:36
  we've got these two questions, they're completely different
• 35:39
  questions, right? Like you can see on the slide
• 35:41
  here, you can see that there's a question about chemistry and
• 35:44
  then you also have a, you know, a question about nature and what
• 35:48
  helps prey hide, etcetera, etcetera.
• 35:51
  And what you're doing is before each one of these.
• 35:56
  If they were their own prompts, then you would give a question
• 36:01
  and answer kind of like few shot a description to the model
• 36:07
  saying hey, here's what I want you to do, here's the question
• 36:11
  one and answer it like this. And then question 2, answer 2,
• 36:15
  et cetera, et cetera. And then you give it question
• 36:19
  which is the chemistry question. But you say, look, you know
• 36:22
  what? We don't even need to do 2
• 36:26
  different rooms because we can just smash these two questions
• 36:32
  together and ask the large language model at the same time,
• 36:38
  alright and say well here's question one, here's question 2
• 36:42
  looks like here's a question. And actually when what I see in
• 36:47
  this slide is even better. And I didn't even recognise this
• 36:50
  yesterday, but you have question one and question two together
• 36:56
  and then you have answer one and answer 2.
• 36:58
  So you're pairing the questions so that the model isn't thrown
• 37:03
  off when you give it two questions and ask it for 2
• 37:06
  answers and it all makes sense. Now I totally understand it and
• 37:11
  hopefully the listeners do too. And if they don't, then I'm
• 37:14
  going to encourage them to go and look at the YouTube video
• 37:17
  because you can see the slide clearly and you can really get
• 37:21
  an idea of what's going on. Yeah, thank you so much for
• 37:25
  explaining the whole thing. I mean that's awesome because
• 37:29
  the big question that I had yesterday I remember was like
• 37:32
  how can you tell would like I couldn't see how the model would
• 37:39
  not blend the answers. But now I easily see it because
• 37:45
  the prompts that you're giving it are questions and answers and
• 37:48
  pairs. So then the prompt that you give
• 37:50
  it is or the the last question answer pair, you just say like
• 37:56
  here answer these last two questions and boom then you have
• 38:01
  it and so back to the $1,000,000 question which is how does this
• 38:06
  actually? Save money, right?
• 38:08
  That's a great point. So why would this money right,
• 38:12
  basically compare the original 2 queries or to questions versus
• 38:19
  the new question we have, right. So in the previous version we
• 38:23
  have those two queries. You need to process those two
• 38:26
  questions as well as the prompts wise.
• 38:31
  Right, but now in this like batch or this concatenated query
• 38:37
  you only need to process a prompt once her.
• 38:42
  Right, so therefore you actually avoid redundant from processing.
• 38:49
  And that saves you the money. And have you tried like how much
• 38:55
  can you push this? Can you do it with two
• 38:57
  questions? Can you do it with 10 question
• 38:59
  pairs? Like how far did you try and go?
• 39:04
  Right, so so far we try. To push this to up to four
• 39:11
  questions. So that roughly speaking saves
• 39:13
  you about 70% of the cost, which also does not hurt your
• 39:20
  accuracy. So that's what we try so far.
• 39:24
  And I think in general this could be pushed too much further
• 39:28
  depending on the depending on the the maximum prompt size.
• 39:36
  So you know, for instance, you know if you have the if you're
• 39:39
  using like standard security for was like 1000 token window size.
• 39:45
  And if you each of your queries like a few hundred they can push
• 39:49
  us to 2040 or even 80 examples. So that can save you a lot.
• 39:55
  Yeah. OK, it makes complete sense.
• 39:57
  Alright, so now this is awesome if I wanted to go about
• 40:01
  implementing this. How do you recommend it?
• 40:05
  Like what is a way that I can start doing this now with my
• 40:10
  prompts? Is it just looking at hey where
• 40:13
  do I have similar questions? Where do I have similar prompts
• 40:18
  that I can use? Or is there some kind of
• 40:22
  programmatic way to do that? Not yet, but we will release the
• 40:29
  code for frugal just between a few days so then people can go
• 40:33
  away. So by the time this is out, by
• 40:36
  the time this podcast is out, it will be out.
• 40:39
  So we'll leave a link in the description then and I'm.
• 40:44
  Sure. I hope so.
• 40:45
  Yeah. Let's go out.
• 40:47
  Yeah, exactly. And so then next is you're gonna
• 40:50
  go make the rounds and raise a couple $100 million and it's
• 40:55
  straight to the moon. Yeah, yeah, yeah.
• 40:58
  We, we, we should do that. Right, guys.
• 41:01
  So, OK, so interesting to hear then.
• 41:05
  Right now it's, it's kind of manual just looking at where can
• 41:09
  you crunch these prompts together and how can you be more
• 41:12
  economical with your prompts. But you do have something that
• 41:16
  you're going to release that will help people do that.
• 41:19
  And I feel like there was another way.
• 41:22
  So this was one way of economising the prompts.
• 41:27
  There was more ways though, right?
• 41:29
  Absolutely. Give me all of it and tell me
• 41:32
  everything. This one might be the most um.
• 41:37
  Interesting. And also very different from the
• 41:39
  well we just see. Hmm, right.
• 41:42
  So, so so far we have been saying OK, have one.
• 41:45
  Service or one loud model, right?
• 41:48
  You want, you want to play with the problems you want to
• 41:51
  concatenate. Different queries get better
• 41:53
  results. Now we can take a very different
• 41:55
  angle. Right.
• 41:57
  So let's say you do have that fixed.
• 42:00
  Query or or prompt, but you have the ability to leverage all
• 42:08
  available loams on the market. Right.
• 42:12
  So if that is the case, how could you actually possibly save
• 42:19
  money from from that marketplace O1 deep inside there?
• 42:24
  Maybe not deep, but one interesting insight said there's
• 42:27
  a large performance and cost difference among those models.
• 42:33
  So therefore you could actually build a large language model
• 42:37
  cascade. To adaptively select like which
• 42:42
  level model to use for your own application.
• 42:47
  So in this example, right, you can think of this cascade as a
• 42:51
  sequence of LLM APIs or service and we have a new query.
• 42:59
  You pass that query to the models.
• 43:03
  In this sequence sequentially. And whenever one model says you
• 43:11
  know this is the this is the answer, and you believe that's a
• 43:14
  high quality answer, you stop there.
• 43:18
  Return answer and you'll never need to query any remaining
• 43:23
  service. Otherwise you'll keep query the
• 43:26
  service in the sequence until they reached the last one and
• 43:31
  return the answer there. So and and that's example right,
• 43:35
  we have a cascade that consists of three service, GPD, JPD 3 and
• 43:42
  P4. And the cost of them also grows
• 43:46
  in in its sequence and you pass the first query to JBJ.
• 43:53
  Will answer, but you think the answer is probably not good
• 43:56
  enough, right? So there you pass the query to
• 44:01
  the second one. So each of these three entry
• 44:03
  three gives you another answer. And look at answer.
• 44:05
  You say, OK, this is probably the right answer, so I'll stop
• 44:10
  there and then return answer. So now there's no need to
• 44:14
  courageously for, so that avoids, you know, calling the
• 44:18
  expensive G4 model. So the the hard part or the
• 44:23
  interesting part here is how do you actually know whether an
• 44:28
  answer produced by Sanjay or other model is high quality or
• 44:33
  is good enough, Right you're. Right.
• 44:36
  That's where the machine learning or AI technique comes
• 44:39
  in. We're basically build a.
• 44:43
  A model. That takes the query.
• 44:47
  And the answer from say Gypsy J or J3AS input and generate a
• 44:54
  score. That measures the quality of the
• 44:59
  generation. And then you know you can tune
• 45:03
  A. Score threshold to determine
• 45:08
  hey, if you think this answer is high quality or low quality and
• 45:14
  you can do all of those things automatically.
• 45:17
  Without the, you know, like human manipulation or
• 45:22
  inspiration. So that's kind of the.
• 45:25
  May idea like ultimate the whole scoring and cascade and a
• 45:32
  cascade building process. Dude.
• 45:36
  This is one of the things that a lot of people were very excited
• 45:39
  about when I posted this paper in Slack originally was the idea
• 45:46
  of the cascade method and I didn't realise that you were
• 45:50
  using another model to score it. I just thought like I understood
• 45:55
  the cascading part where it's like hey first start with an
• 45:58
  open source model. If that doesn't work go to a
• 46:00
  cheaper model like the ChatGPT or GPT 3.
• 46:04
  Then if that doesn't work then go to the expensive one, because
• 46:07
  obviously not all queries are going to need GPT 4 level
• 46:13
  answers and that is very clear. And GBT 4 is the most expensive,
• 46:19
  so your price can really get out of hand if you're always using
• 46:23
  GP4. However, I stopped there at my
• 46:27
  understanding. I didn't realise that your but
• 46:30
  it makes complete sense because as you were explaining this I
• 46:33
  was thinking to myself, OK, So what I'm gonna have to like
• 46:37
  validate each output and then if I don't like it I send it back
• 46:42
  and or I click a button like say like no, give me a better one.
• 46:46
  But if you're using a model to score it and then if it gets
• 46:52
  above a certain score and it's acceptable, then you serve that
• 46:56
  up to the end user. That's awesome.
• 46:58
  That's perfect. Yeah, I I'm, I'm glad you like
• 47:01
  it. I'm your biggest fan, dude.
• 47:03
  I don't know if you realise that by now.
• 47:06
  I'm gonna be just coverage you're you're permit first fan.
• 47:09
  I I would really appreciate it if I become a star.
• 47:12
  You know? Exactly.
• 47:13
  I'm gonna get your autograph. Yeah.
• 47:17
  Before you become famous. Ohh.
• 47:19
  So I can give you one right now. There we go.
• 47:22
  Yeah. Ohh, that's classic.
• 47:23
  That is good. Yeah.
• 47:25
  But so one thing I did want to ask you about the cascade method
• 47:30
  is when you were messing around with this, did you also hang
• 47:34
  around to different API's? Like right here you're just
• 47:37
  talking about GST and basically the Open AI APIs.
• 47:41
  But were you also looking at OK, what how can we cost compare
• 47:45
  with anthropic or cohere or whatever else is out there?
• 47:49
  Ohh, you gotta slide for it. Alright, here we go.
• 47:51
  I know it's real. If you got a slide, here we go.
• 47:54
  Yeah, edits here. I guess that's probably not that
• 47:57
  friendly for a podcast because there's so many slides.
• 48:01
  But I guess I got your point and I think the answer is yes, we do
• 48:05
  look at a bunch of APIs from different providers.
• 48:09
  So those providers are, I mean like first, there are a lot of
• 48:14
  providers here. There's opening eye, there's a
• 48:16
  21 Co here. So we measure the apps from all
• 48:19
  those providers. For a few different datasets,
• 48:22
  this is just one example O. In the cascade you'll see three
• 48:26
  of them because. Because we want to see the fast
• 48:32
  for the users, right. So let's say there are like 20
• 48:35
  APIs or like 30 apps in the market and you include all of
• 48:40
  them in your cascade, right. Then that itself could be very
• 48:44
  expensive for a user, right. Like you matter for error query,
• 48:47
  you have to you know pass through you know like all the
• 48:50
  five or six cheap APIs that's that's that's not quite desired,
• 48:55
  right. So what do we actually did is
• 48:58
  that we're going to include. All those APIs in the market as
• 49:03
  handed APIs and build an optimization framework to select
• 49:09
  which APIs should be included in the cascade.
• 49:14
  For make different users application or data ohh wait so.
• 49:20
  Right. So that's why it's the story.
• 49:22
  So OK, if I'm understanding it correctly, then it's you're
• 49:28
  using different APIs based on the different inputs that you're
• 49:32
  giving, and you're only doing 33 is the Max, because otherwise it
• 49:36
  just doesn't make sense. You're not saving money.
• 49:39
  You're probably spending more money.
• 49:41
  Yeah, actually you could use even more, no.
• 49:45
  Yeah, like we we, we actually tried a few options.
• 49:48
  We tried two or three, tried 4. So generally speaking the more
• 49:53
  you have the better prediction quality or generation quality
• 49:59
  you have, but the the higher cost you need to pay.
• 50:03
  So yeah, like 3 is a is a kind of a sweet spot, but 4-5 would
• 50:07
  also work. So biggest thing that jumps out
• 50:11
  at me on this slide right here is that accuracy for the Dolly
• 50:17
  model is basically the lowest that it can get and Matte is
• 50:21
  your advisor, he let you publish this.
• 50:25
  Ohh yeah yeah. Well he used the data Brace
• 50:28
  survey system to server Dolly model here.
• 50:33
  So you can see yeah, like the algorithm is probably not as
• 50:36
  good as other Moto, but that's also released as part of Dolly's
• 50:41
  earlier paper or blog post. They say that, hey, Dolly was
• 50:45
  not developed to be the state-of-the-art model there,
• 50:50
  right. But you can see the cost is much
• 50:53
  smaller than before. Yeah, it's so low.
• 50:57
  That's so true. Ohh man.
• 50:59
  So it's interesting to me too that GPT Neo is so high and the
• 51:06
  accuracy and that's a model that's been out for a long time
• 51:13
  And it yeah, wow. I mean then you go and it's like
• 51:18
  GT Neo GT3J1I. I'm not sure what the J1 is.
• 51:25
  J1 wasn't Model 81. OK, cool.
• 51:30
  Yeah, publishers have expressed somewhere say ohh, there's more
• 51:33
  from this company that's at that company.
• 51:35
  Yeah, yeah, it makes sense. OK, so interesting.
• 51:38
  So basically it just bringing this back to the end user.
• 51:42
  And if I'm at my company and I want to try and do something
• 51:45
  like this, is there like How can I get started and try and
• 51:50
  implement this? Or is it even worth it unless
• 51:53
  I'm racking up millions in API calls?
• 51:56
  Yeah, we will also. Include this in the.
• 52:01
  Upcoming. Korea is investors are happening
• 52:06
  there, yeah. Yeah, that's a few work too.
• 52:11
  I mean, just a little bit more work to make the code easily
• 52:15
  accessible. That's why we spend so much time
• 52:18
  there. But yeah, I I promise the the
• 52:20
  the code will be released in the building is then people can
• 52:23
  directly use that to help with their own applications.
• 52:28
  Ohh, I love it. Yeah I love it.
• 52:30
  So alright then this is this is again on that cascading piece.
• 52:35
  There was another another piece that you had and maybe we can go
• 52:40
  over one more before we finish it off.
• 52:44
  There's another piece that's called lottery model
• 52:46
  approximation right like that one.
• 52:49
  Basically it's about how you could approximate performance of
• 52:54
  Florida model using much cheaper infrastructures or simpler
• 53:00
  technique. So I actually want to go over
• 53:02
  like one very simple but in my opinion useful technique.
• 53:10
  So that's as simple as storing and reusing.
• 53:15
  At our arms answer for similar queries.
• 53:18
  The key idea right is to add a hash layer.
• 53:23
  On top of the. Large language model and
• 53:29
  whenever you send a query to the launch model, you first send
• 53:33
  that to the cache and to see if there's any query you already
• 53:38
  processed before. That's very similar to this new
• 53:42
  query, and if so, you use that Sir from your cache.
• 53:46
  No need to query any router models.
• 53:49
  Otherwise you go to the other model, you get an answer and you
• 53:53
  send it back to your cache, right?
• 53:56
  So. It's it's just as simple as
• 53:58
  that. But I want to make two points
• 54:00
  here. First this.
• 54:02
  This is quite useful when you have.
• 54:06
  Like similar queries. For instance, if you have a
• 54:10
  search engine powered by Rams and users are asking similar
• 54:14
  questions at certain time period, you can avoid sending
• 54:19
  all user queries to a language model.
• 54:21
  That's kind of like quite a common error of application
• 54:25
  right now. Second point I'll make is that
• 54:28
  there's a lot of interesting, interesting research and also
• 54:34
  private questions to consider even for such a simple.
• 54:38
  Architectures, for instance was a good way to to design the
• 54:44
  cache policy. Right, like like your cache.
• 54:48
  Like if your cache is is is is huge or like has infinite
• 54:54
  memory, then there's no problem. But typically you have a fixed
• 54:58
  bandwidth with your cache. Maybe it's like you know just a
• 55:00
  few GB or something, right? And if you have a lot of
• 55:03
  queries, at some point you need to decide which query to achieve
• 55:09
  in a cache and which query to move out of the cache, right?
• 55:14
  So how to design good caching policy?
• 55:17
  That's a that's a one big problem.
• 55:19
  And second, you need to sort of have a way to determine, hey, if
• 55:23
  this query is similar to queries in my database or in a cache.
• 55:29
  But how similar is similar, right?
• 55:31
  How similar? Similar.
• 55:32
  I give one simple example, right?
• 55:33
  So you could have two questions that are like very similar, or
• 55:42
  like, like, like, like superficially, but the meaning
• 55:45
  is totally opposite. Right.
• 55:47
  Like like what if I ask like ohh what is outcome of into and O2
• 55:52
  at a high temperature? And could I could also ask what
• 55:55
  is not the outcome of O2 and and two at high temperature, right.
• 56:01
  So those two cores could look similar but they're semantic
• 56:05
  meanings are quite different. So how do you actually enjoy the
• 56:09
  similarity there in a systematic way, Right So.
• 56:16
  All I'm trying to say is that this looks simple, but there's a
• 56:20
  great opportunity to. Make him a real fast and and and
• 56:28
  and efficient and also there's a lot of application of this.
• 56:33
  Yeah, it's like there still is, because the idea of using a
• 56:37
  vector store is probably one of the most common design patterns
• 56:43
  these days when you're working with looms, I think.
• 56:46
  And so you cache the query in a vector store maybe, and then you
• 56:51
  hit the vector store before you go to the LLM.
• 56:53
  But there are all of these really tricky situations where
• 56:59
  you like you said, the query may look similar, but the answer is
• 57:06
  the exact opposite of what the other query would be.
• 57:09
  And so I know there's a lot of pain points or just big
• 57:14
  questions out there when working with vector stores how what's
• 57:17
  the best practise and how to do that how to go about it.
• 57:20
  O that's man, that's awesome. This is some good stuff.
• 57:26
  I mean have you seen best practises and are there some
• 57:31
  quick trip tips and tricks that you know or you came across
• 57:36
  while you were doing it? I.
• 57:39
  I think it actually application dependent.
• 57:43
  Ohh interesting. Yeah, yeah, so.
• 57:45
  Like. I don't think there's any like
• 57:49
  good and principled solution yet.
• 57:53
  That's not to my knowledge. I think there are like a few
• 57:57
  tricks that you could use for certain cases.
• 58:03
  For instance if like for for the negation example I talked about,
• 58:07
  or you can manually insert that as a rule right in our method B.
• 58:13
  Or, you know, just add another layer on top of that.
• 58:16
  But. I don't think so far there's
• 58:19
  any. Principled way?
• 58:22
  Yeah, that's kind of a good thing from a research
• 58:25
  perspective because of research. You lots of open area.
• 58:29
  Yeah, yeah, totally. Yeah, but it was set to paying
• 58:32
  for users. Yeah, exactly.
• 58:35
  But I mean, hopefully it won't be after a while.
• 58:39
  Yeah, one big question that I have is do you use any of these
• 58:45
  together? Like are you using the for
• 58:49
  example this caching layer with cascading and maybe another one?
• 58:56
  Or is it? If you use one, you kind of have
• 58:59
  to stick to that one, and then you don't want to veer too far
• 59:03
  out. Yeah, there are.
• 59:05
  There are actually a lot of composition I ideas or
• 59:11
  composition messages I can give, like a few of them.
• 59:16
  For instance, it's caching idea could be easily composed with
• 59:20
  the. What's the LCM cascade, right?
• 59:25
  So yeah, you could have a drawing cache that store the
• 59:30
  prediction or generation or other services in your cascade,
• 59:36
  right? And whenever you query the
• 59:39
  similar question, you look at the cache and say, oh, it's
• 59:43
  already processed by the previous model, or sorry,
• 59:47
  there's already processed by some other service before you
• 59:52
  can actually fetch it. The year and there's no need to
• 59:55
  query anything on the on the on the cascade anymore, so this
• 1:00:00
  could also be composed with the with the query concatenation
• 1:00:04
  thing, right? Because you can just catch
• 1:00:07
  everything from that batch query as well.
• 1:00:14
  You can also combine the UM CASCADE with the query
• 1:00:20
  concatenation, right? Because you can say OK, I now
• 1:00:24
  have all of my poor race as a batch query, right?
• 1:00:32
  Because you say you have 1000 queries, you first concatenate
• 1:00:36
  them into 500. And then you cascade there,
• 1:00:40
  right? Then you cascade it, right.
• 1:00:42
  So yeah, like, like, like there's a lot of ways to to to,
• 1:00:47
  to combine or to to compose all those things.
• 1:00:51
  Ohh, that is so cool man. Well dude, this has been
• 1:00:55
  awesome. I know.
• 1:00:56
  I think you have been feeling how excited I am for what you're
• 1:01:00
  doing and I think there's a lot of people out there that are
• 1:01:03
  using large language models that.
• 1:01:06
  Definitely feel the burn of. Using the large language models
• 1:01:12
  and it can get quite expensive, especially if you're at scale.
• 1:01:16
  And so these tips and tricks on how to bring that cost down is
• 1:01:21
  super helpful. And so I thank you for that and
• 1:01:26
  I am very excited for the tool that you come out with and we'll
• 1:01:29
  make sure to highlight it in the.
• 1:01:33
  Description and I'll probably try and throw it in the
• 1:01:35
  newsletter, so man, I'll push it back.
• 1:01:39
  Yeah, this conversation's been great.
• 1:01:41
  Yeah, it's, it's, it's, it's all my all my pleasure.
• 1:01:45
  Like like it's really nice to talk with you and help.
• 1:01:48
  And thank you so much for helping me explain a lot of
• 1:01:51
  things because it's way better than my own words.
• 1:01:56
  It's just to make sure that I understand it, you know, I wanna
• 1:02:00
  recuse since I have you here. It's not.
• 1:02:03
  Not every day that I get to actually talk to the author, and
• 1:02:07
  basically it's a talk to your talk to your PDF, but in real
• 1:02:12
  form. And you can explain it to me
• 1:02:14
  much better than any large language model can.
• 1:02:17
  So I appreciate that and I really, I love the work that
• 1:02:21
  you're doing. And I think we can we can end
• 1:02:24
  the pod here, man. Ohh.
• 1:02:26
  Yeah, sure, sure. Thank you so much.
• 1:02:32
  Hey, this is Mike Delbalso, Cofounder and CEO of Tecton.
• 1:02:36
  ML Office Community is the best way to stay in the loop on the
• 1:02:39
  latest ML OPS news and best practises.
• 1:02:42
  It's also a great way to connect with experts and get support
• 1:02:45
  from an amazingly helpful community.
• 1:02:47
  Subscribe and stay