Latest / Agentic Conversations (formally mlops.community) / From Arduinos to LLMs: Exploring the Spectrum of ML // Soham Chatterjee // #162

Transcript

From Arduinos to LLMs: Exploring the Spectrum of ML // Soham Chatterjee // #162 · Agentic Conversations (formally mlops.community)

• 0:00
  So my name is so awesome. I'm currently leading the
• 0:03
  machine learning team at when Tech startup in Singapore,
• 0:06
  counting the Sleek. I try to make coffee with like,
• 0:12
  you know, with high Precision grinder with a, with a
• 0:15
  temperature-controlled cattle and with, you know, with a very
• 0:19
  sensitive scale. I use distilled water.
• 0:23
  That has been specifically, you know where they've added some
• 0:26
  minerals in some specific formulas that In percentage.
• 0:30
  Is that supposed to make your coffee tastes better?
• 0:33
  I have I I don't buy beans that I like older than that.
• 0:38
  Haven't been roasted like more than a week ago.
• 0:41
  Yeah. I mean, I do all of these things
• 0:43
  and you know, my coffee. Still tastes the same actually
• 0:49
  it doesn't it taste good but but you know it doesn't taste like
• 0:53
  like very expected to it is based on the amount of effort I
• 0:57
  put in but you know the good thing.
• 0:59
  It smells. Lee great.
• 1:00
  When you grind the coffee, it's really, it's very calming like
• 1:03
  grinding the coffee, and just going through those motion of
• 1:06
  making it. And I think that's the best
• 1:08
  part. It doesn't matter how it tastes.
• 1:11
  It gives it gives you a sense of calm.
• 1:15
  What's up, everybody? This is Demetrios and you are
• 1:17
  listening to another edition of the MLA Ops Community podcast.
• 1:21
  Today, I am graced with the presence of a b.
• 1:24
  What is going on? Abby?
• 1:25
  How you doing another? Very fell down with fever.
• 1:29
  I just took Look meds. Oh, no.
• 1:32
  Oh, I didn't realize that, we just recorded the whole episode
• 1:35
  and you seemed absolutely all together.
• 1:40
  Everything was fine. So I didn't even think to ask if
• 1:43
  you weren't doing that well, but I guess you hold your composure,
• 1:46
  well, under stress and Under Pressure now, but it was an
• 1:50
  excellent conversation. I was actually excited for this
• 1:53
  conversation, maybe that's difference close as opposed to
• 1:57
  other ones where you're bored out of your mind.
• 1:59
  And You are in pain even though you're not sick.
• 2:04
  I ate don't think I've been bored out of my mind in any
• 2:07
  conversation but this is also like one of the areas that I'm
• 2:10
  working in right now. So there's a little bit more
• 2:14
  excitement. If something is a little bit
• 2:17
  more recent you know, there are there things that you can learn
• 2:20
  for the sake of learning and then there are things that
• 2:23
  you're like, I'm doing this right now.
• 2:25
  I'm looking for a new perspective on those things, and
• 2:29
  I think, He had a very interesting idea of, or at least
• 2:34
  I could see his thought process on a lot of things did.
• 2:39
  But line very well with what I've been studying and
• 2:42
  observing. Oh my God.
• 2:43
  So we should probably just mention soham.
• 2:45
  Came on here, he just talked to us about two main themes, one
• 2:49
  being tiny ML, and the other being actually using large
• 2:53
  language models in production. I loved it, because it was a
• 2:56
  refreshingly real take on the Pains of use Large language
• 3:00
  models right now in their current state.
• 3:03
  And he has been trying to use them.
• 3:05
  I mean, he started using them when there was GPT to that was
• 3:10
  out and he's continued to use them.
• 3:11
  And I mean, for me, the biggest highlight was how he talked
• 3:16
  about, you know, things, it's very easy to stand something up
• 3:20
  and get that toy demo. Working really well, but when
• 3:23
  you want to actually go to scale, that is where it gets
• 3:27
  really hard and you can't just Rely on open a eyes API because
• 3:32
  who knows what they're s. LA's are they haven't actually
• 3:34
  released any SLS. And you also start paying a lot
• 3:39
  of money like really fast. So as a side for those who do
• 3:44
  not know who soham is, he leads the room machine learning team
• 3:47
  at sleek and he builds tools for automated accounting and back
• 3:52
  office management. He started out as an electrical
• 3:56
  engineer which we get into and he has This passion for tiny ML,
• 4:02
  and he loves talking about it. So we started out with that, and
• 4:05
  we went down that road for a bit, and the end Reno's Ardrey
• 4:09
  knows, I think is whether called, and how he loves
• 4:12
  deploying machine learning on them.
• 4:15
  And then we went to the opposite side of the spectrum and we
• 4:17
  talked about the large language model.
• 4:18
  So, basically how small can you make the machine learning and
• 4:21
  how big can you make the machine learning?
• 4:23
  Was there any key takeaways that you had before we jump into the
• 4:25
  actual conversation? Episode 1, which is the kind of
• 4:31
  glass that he looked at coming from Tiny ml.
• 4:34
  He said, you know, the scale is quite similar, at least the
• 4:37
  skill difference ratio, which is earlier.
• 4:40
  We were trying to deploy these models from very tiny devices.
• 4:44
  And we had the same challenges. These challenges were, how much
• 4:47
  power is the consuming. How much latency does it have?
• 4:50
  How private they are sharing is happening?
• 4:53
  How much excess in control? Do you have over these models?
• 4:56
  How are you compressing your models?
• 4:58
  And most of these things. Now translate into the realm of
• 5:02
  elements as well. So I love that one.
• 5:05
  So good. So he's also got a sub stack.
• 5:08
  I think I found him through sub stack or she came into the
• 5:10
  community and he's one of those people that it was obvious when
• 5:14
  he came into the community that he knows what's up.
• 5:16
  He's been doing this for long enough and he's been thinking
• 5:19
  about it deeply and so, I encourage everyone.
• 5:21
  If you want more of so, hums work, go and read his sub stack
• 5:26
  will leave a link to that in the description for now.
• 5:28
  Let's jump into this conversation and oh wait, no
• 5:34
  before we go anywhere, do you? It would mean the world to us.
• 5:37
  If you could share this episode with one friend that would just
• 5:41
  be incredible if you know someone that is in this field
• 5:46
  and would appreciate this type of podcast, please share it
• 5:50
  because that's how we can grow and get out to more and more
• 5:54
  people and hopefully spread the good word and let's go.
• 5:58
  Let's do it. A b.
• 5:59
  Uh, you ready? Here we go, right?
• 6:08
  So him just have to make sure it's am I pronouncing it
• 6:11
  correctly. So is so.
• 6:13
  Um, so it's more hum like, like, you, like humming music?
• 6:19
  Yeah. In yoga classes.
• 6:22
  So hum is kind of a popular word that they use, huh, do you know
• 6:27
  that? Yeah, I've heard, yeah, I know
• 6:30
  that but like, none of the yoga classes?
• 6:32
  I've been to have used that word so Other people say that.
• 6:39
  Yeah, the name mean, by the way. Yeah, so from what I understand
• 6:44
  it means I am. Like what I am.
• 6:48
  So I guess that can be interpreted in a lot of ways.
• 6:51
  Yeah, I am. I am.
• 6:52
  What I am, you know, this accept it.
• 6:56
  Yeah. Enjoy it.
• 6:57
  That's all. Yeah, exactly.
• 7:00
  Exactly, dude. Awesome.
• 7:02
  Well, I'm super excited to talk to you because we are going to
• 7:06
  go down almost like like two paths.
• 7:08
  Were you do a lot of fun stuff with large language models?
• 7:11
  But you also have this Alter Ego that is doing stuff with tiny
• 7:15
  ML. And you mentioned to me before
• 7:18
  we hit record, that one of your passions is to look at how
• 7:21
  people are using and try and figure out yourself how to use
• 7:26
  machine learning at the edge. So let's get into all of that.
• 7:31
  But before we do, I think it's worth just having a bit of a
• 7:35
  brief explanation on How you came to find yourself where
• 7:38
  you're at right now? Yeah, sure.
• 7:41
  So, so, my background is not in computer science, like, at all,
• 7:46
  I have a bachelor's and a master's in electrical
• 7:50
  engineering, but I mean, I can do anything that is electrical
• 7:55
  related apart from, like change the occasional like, light bulb,
• 7:59
  right? So when I was in, when I was
• 8:01
  doing my undergrad, right eye, this was back in late, 2014 or
• 8:06
  15. And iot was, like, really
• 8:08
  popular at that time. And and what happens in in any
• 8:13
  iot devices, it generates a lot of data.
• 8:16
  And at that time, there weren't really a lot of people trying to
• 8:19
  use that data and do something smart with it, right?
• 8:23
  Or do something intelligent with it machine learning and deep
• 8:25
  learning. Was, it was this gaining
• 8:28
  popularity at that time? Lord of people didn't know how
• 8:30
  to deploy these modules really well?
• 8:33
  Or even like how you could use the data and build something
• 8:36
  smart. Third at that time really good.
• 8:38
  So so yeah, it was kind of at that time when I started
• 8:42
  exploring like, how we can take this, all the data that these
• 8:46
  iot devices are doing and build something smart with it.
• 8:49
  So that's and that's that path. Kind of led me down to studying
• 8:54
  machine learning and then deep learning.
• 8:56
  And, and then combining iot with deep learning cute at your
• 9:00
  Bachelor's in electrical engineering and then you will
• 9:04
  turn to work with a company called Summer.
• 9:07
  Knowledge is as deep learning researcher.
• 9:10
  Yeah, how did that happen? Yeah.
• 9:13
  So so spite of this student run a lab in our University.
• 9:18
  Where, you know we was just like learning stuff experimenting
• 9:21
  with all this stuff. You all the cool things that
• 9:24
  were happening at the time. I would T deep learning virtual
• 9:27
  reality stuff like that. And so the person who eventually
• 9:32
  went on to be the head of the deep learning Lab at some Give
• 9:37
  give a talk at that lab and that's how we got connected.
• 9:41
  And when I was graduating he one day he just randomly message me
• 9:47
  on messenger. At the time I was in the middle
• 9:49
  of class any message me and he was like hey can you come down
• 9:53
  to the to our office which is like a four hour plane ride?
• 9:57
  I think it was. Wow and you know just this this
• 10:01
  like see if the Vibes if you like The Vibes and if you want
• 10:05
  to join me, we do an internship or Thing.
• 10:07
  Yeah it was like a brook student at the time and he essentially
• 10:12
  line and you know like here is yeah exactly.
• 10:15
  So I'm really happy that I got got that opportunity.
• 10:19
  I think it's really hard to get to get a job in machine learning
• 10:22
  especially at that time if you didn't have like a PhD or like
• 10:25
  or like prior experience and I'm glad that his name is maile by
• 10:30
  the way he still at Summa Sean and yeah shout out to my life
• 10:35
  for giving me the opportunity. The interesting me.
• 10:39
  So you ended up with a background, which was between
• 10:43
  deep learning Edge, Computing, and Quantum.
• 10:46
  Computing. That's a very interesting
• 10:49
  combination. Yeah, I did that bill a bit in
• 10:52
  Quantum Computing. I won't say, I know like the a
• 10:56
  lot about it. So the research lab at summer
• 10:59
  was like we were allowed to experiment a lot and work on
• 11:03
  things that we thought would benefit the company.
• 11:07
  You at that time sasame is like a farmer IT company so we build
• 11:12
  tools and products for pharmaceutical companies.
• 11:15
  At the time, a lot of people were using Quantum Computing to
• 11:18
  do, I think protein synthesis and like, yeah.
• 11:21
  So so yeah, we just tried experimenting and seeing if
• 11:25
  something worked, that's how I dabbled a bit in Quantum
• 11:28
  Computing. But yeah, not, I wouldn't say
• 11:31
  I'm like, I'm qualified to talk much about it because you've
• 11:35
  worked in tiny. Like this, you've explored it.
• 11:39
  What are some of the challenges that you've seen?
• 11:41
  But deploying machine learning models on Tiny ml, or kind of
• 11:46
  inch devices as far as Tiny ml goes.
• 11:49
  What do you consider? Tiny ml?
• 11:51
  Yeah, good question. So for me, tiny animal.
• 11:55
  Like the really, really tiny animal.
• 11:57
  It's probably like devices, like an Arduino Nano, which has only
• 12:02
  a few kilobytes of ram. So, and that Ram has to hold,
• 12:08
  not only your model but also your code for running the
• 12:12
  executing, the model, reading data from sensors, you know,
• 12:16
  processing that data, all of that stuff that that's that's I
• 12:20
  think like the really really tiny stuff.
• 12:23
  And then what makes it more challenging is if you deploy
• 12:26
  that device to like a remote area, where maybe you don't have
• 12:30
  have access to utility power rate.
• 12:33
  So you have to run it off of a battery.
• 12:36
  So how do you Baker code, like really optimized so that the
• 12:40
  device can autonomously for, like, a year at least.
• 12:44
  Yeah. So so that would be like the
• 12:46
  really, really tiny like at the really Edge, right?
• 12:50
  But then, I think, you know, phones right now really
• 12:53
  powerful. But that's also I guess in some
• 12:56
  ways I wouldn't call it tiny ml but it's an embedded device, at
• 13:00
  least. And and then another popular
• 13:03
  embedded device would be a Raspberry Pi.
• 13:06
  Yeah. So, So I guess like there's this
• 13:09
  levels of tiny ml. You have like the really really
• 13:12
  constrained devices with no power no connectivity.
• 13:16
  So you can't even make like a query to a server somewhere and
• 13:20
  then, you know, those constraints slowly get less and
• 13:22
  less. As you move further up the chain
• 13:24
  to like a Raspberry Pi and then a mobile phone and and so on.
• 13:29
  Yep and you're playing around with what when you mess around
• 13:32
  with tiny ml. What is your favorite area to
• 13:35
  play in on that chain? Yeah, I think I so I really like
• 13:40
  working with Arduino 90's because that's that's where the
• 13:44
  that's where it's most challenging.
• 13:46
  Yeah. How do you how do you even see
• 13:49
  something work on a device like that?
• 13:51
  It's probably going to be like an LED that blinks right and and
• 13:55
  the joy that a blinking LED you can give you.
• 13:58
  It's the I can't describe the feeling a little.
• 14:03
  So what? Yeah, plication stock you were
• 14:05
  exploring on on these These devices.
• 14:08
  Yeah sure so. So one of the first applications
• 14:12
  that I worked on was that summer, we were working with the
• 14:15
  medicine manufacturer and Manufacturing medicines.
• 14:19
  It's it's highly confidential. You need to have a lot of safety
• 14:23
  and there's a lot of protocols around it.
• 14:25
  So this particular manufacturer, they wanted to make sure that
• 14:30
  any box of medicine that was going off of the factory.
• 14:34
  You know, production line had like not only Medicine.
• 14:37
  But also medicines usually have like some piece of paper with
• 14:41
  like some information and some warnings on it stuff like that.
• 14:44
  They wanted to make sure that all the boxes that they were
• 14:47
  shipping out, had all of those things.
• 14:49
  And and the reason we need a tiny ml was because because they
• 14:54
  couldn't make a query to a model out in some server somewhere
• 14:57
  because that would be sending like confidential information
• 15:00
  outside and then Factory floor. Space is also very expensive so
• 15:05
  they didn't want something. Really large rate.
• 15:08
  They wanted something small that they could fit maybe next to a
• 15:11
  camera somewhere. So, so that was one of the first
• 15:14
  applications that I worked on with tiny ml.
• 15:17
  What we ended up building for that was at the time, there was
• 15:20
  this device called a neutral compute stick and the neural
• 15:24
  compute stick. It's like a USB.
• 15:26
  It's like a thumb drive that you can stick on into a Raspberry Pi
• 15:31
  and and and and it's called an accelerator.
• 15:34
  So what it does is you can offload The difficult and
• 15:38
  compute intensive ml workload on that accelerator and that
• 15:42
  accelerator can only execute ml modules.
• 15:46
  It can do anything else and because it can only do that,
• 15:49
  it's really efficient at that so it can do it really quickly as
• 15:52
  well. So three are so that was one of
• 15:55
  the first applications more recently, I was working with
• 15:59
  neuromorphic Hardware. This is so when you have to
• 16:01
  deploy an application to a place where there is no utility power
• 16:06
  and you have to use Use a battery.
• 16:07
  You want to make sure that the devices you're using consume as
• 16:12
  less power as possible, right? And sensors are one of the most
• 16:16
  power hungry devices that you can have in a tiny immune
• 16:19
  system. So, so one of the solutions is
• 16:21
  to use new term of fake Hardware, which are highly
• 16:24
  efficient. So so yeah.
• 16:26
  So we built and, and another thing in this field is do more
• 16:30
  custom. You make your Hardware.
• 16:31
  The more you can save on energy. So what we did was we built a
• 16:36
  chip Good interface with this neuromorphic camera and and it
• 16:41
  would consume a few Milli watts of power and and it could
• 16:45
  monitor like for instance, you could deploy it in the jungle
• 16:49
  and you could monitor for movement of like of cars or you
• 16:53
  know, any illegal activity stuff like that.
• 16:55
  IL love the fact that you mentioned latency you mentioned
• 16:59
  size and you mentioned popper energy swap because there's
• 17:04
  something you said earlier when we started this conversation.
• 17:06
  That this skill difference is very similar to how many things
• 17:11
  are, how many operational challenges have you seen
• 17:14
  transform, or like, come from one domain to like this?
• 17:19
  New to me? Yeah, sure.
• 17:21
  So latency is the biggest challenge?
• 17:24
  I would say because you want your, you want your
• 17:28
  architecture? And you you want the results
• 17:31
  from your module to come really quickly, right?
• 17:34
  So, so you want to optimize for latency?
• 17:37
  That that's been a really big challenge.
• 17:40
  One Challenge died. Well, it's dead in tiny ml, but
• 17:42
  not a lot is the cost. So at work, when I deploy these
• 17:47
  large language models, we have to really think about whether it
• 17:51
  makes business sense to provision and instance, with
• 17:54
  like one or two gpus, just to deploy a model because a lot of
• 17:58
  times just the cost of that doesn't doesn't make business
• 18:02
  sense at the time. So to kind of solve these
• 18:05
  problems, what people are doing there.
• 18:06
  Trying to find ways to fine-tune the models and compress the
• 18:10
  models, right? And there's a lot of techniques
• 18:12
  to do that. For instance, you know, the
• 18:15
  what's a module called alpaca model which which was which was
• 18:19
  compressed down from like you're from a llama model similar to
• 18:23
  how in time in 10 mL, we do knowledge, distillation.
• 18:26
  I feel like they've. They've kind of.
• 18:28
  They've done it. Similarly and, and you have
• 18:30
  other projects that are doing quantization, what these
• 18:34
  compression algorithms do is they actually have Been shown to
• 18:37
  increase bias in your models which has been a huge engines in
• 18:41
  tiny ML. And I'm sure that will also come
• 18:44
  up right now in lme's as well. Fascinating dude.
• 18:48
  So I love the fact that you are talking about all these
• 18:53
  constraints and really looking at when you're using tiny ml.
• 18:57
  It's because you have certain constraints that you have to fit
• 19:00
  in like there's not enough space on the factory floor or we can't
• 19:05
  actually send the data anywhere. Because that would be illegal.
• 19:08
  And so it forces you to come up with certain ways of doing
• 19:12
  machine learning that you wouldn't necessarily think about
• 19:17
  when you're just out of computer and you're creating some kind of
• 19:20
  SAS software, right? Like I also know that you have a
• 19:26
  sub stack and that might be what got me hooked on what you've
• 19:29
  been doing? You talked about building a
• 19:32
  Chrome app on this sub stack, I think and the different phases.
• 19:37
  Of building with large language models at your job, you're using
• 19:42
  large language models, right? And you're always thinking
• 19:44
  about. And from what I understand from
• 19:46
  talking to you is that it still isn't really clear if you should
• 19:49
  be using them or not and so maybe we could go down that road
• 19:53
  a little bit. But I also want to hear about
• 19:55
  your your basically your journey.
• 19:58
  When you talk about in in your articles on sub stack building,
• 20:02
  with large language models, and how you think about those
• 20:05
  things. And so So I'll leave it very
• 20:08
  open for you to go down whatever path you want.
• 20:10
  And then I'll follow up with what whatever you miss that, I
• 20:13
  still am curious about Yeah sure.
• 20:18
  So the reason we started this Chrome extension.
• 20:22
  It's because well, me and archana who I'm building this
• 20:26
  with feet. Like like anyone else in the
• 20:28
  field. Got really interested with and
• 20:29
  the limbs and how they work. But you know we weren't seeing a
• 20:33
  lot of people who are just building apps and and Sally and
• 20:36
  you know like hooking it up to and just you know trying to
• 20:40
  build a startup around a trait and and in the process we we
• 20:44
  weren't liking how closed-source everything was becoming.
• 20:47
  I wanted to go back to like, a few, like, the times, like, we
• 20:52
  had like a few years ago where people who would build
• 20:55
  something, and they would share how they built it and so that
• 20:58
  others could learn from it. And, yeah, I felt like that
• 21:01
  wasn't happening, and I was in learning a lot about how these
• 21:05
  models about the challenges of deploying these modules.
• 21:08
  And, and what are the constraints, when, when you go,
• 21:11
  when you try to build the product around the.
• 21:14
  So that's why we start we thought.
• 21:15
  Okay, fine, you know? Let's just build like a toy
• 21:18
  example and and you know like see what kind of challenges we
• 21:22
  face and and you know try to solve them and tell people about
• 21:26
  it right. What we've learned is.
• 21:29
  So from so what your would you mention right?
• 21:31
  From this up stack and like these stages of building, like
• 21:34
  stages of building a product that uses mlms, we've learned
• 21:38
  that if you are like a start-up and you want to build an LP
• 21:43
  application, it's very easy right now to build that it's A
• 21:47
  cheap to just call Panera's API and and, you know, have a few
• 21:52
  prompts and, and just build a wraparound prompts.
• 21:55
  And I think that's the first level of products that we're
• 21:58
  seeing out there right now, which are.
• 22:00
  And this is a, this is an oversimplification, but it's
• 22:03
  basically a rapid over crumbs, right?
• 22:06
  And, and, and, and what we realized is building a product
• 22:10
  like that is, is not feasible because as your product gets
• 22:15
  more complex, And as you start adding more features and as you
• 22:19
  try to kind of Meet the output from your model moves from the
• 22:23
  EPA blakemore's reliable, what you end up doing, is you have to
• 22:26
  create these long prompts you have to do stuff like, like,
• 22:30
  just few short, prompting is, usually not enough.
• 22:33
  You have to do like Chain of Thought, prompting, and an even
• 22:36
  more complex from straight. And, and your outputs also
• 22:40
  become longer when you go to Chain of Thought, prompting and
• 22:43
  stuff like that. Yeah.
• 22:45
  And your courses starts to scale up.
• 22:48
  It's also not that reliable, especially if you try to build
• 22:51
  chains and agents and where, you know, the output of one LM EPA
• 22:56
  call feeds into another. The reliability becomes even
• 23:00
  worse. Yeah, with every whatever we
• 23:02
  think will happen, it goes down and down and down.
• 23:05
  Yeah, yeah yeah. And and and I've seen people to
• 23:11
  fix that what they have done is they have like Watcher models
• 23:14
  that watches the output checks for You know, checks to see if
• 23:18
  there's errors happening. There's and, you know, people
• 23:21
  have built these systems where they also have models that
• 23:25
  watches for prompt injection attacks and trust issues.
• 23:29
  And you know, watching for incorrect output stuff like
• 23:32
  that. It it doesn't make sense to me.
• 23:35
  It. I don't think that that's
• 23:36
  something that's scalable and and I don't think that that is
• 23:41
  something that makes a lot of sense from the cost perspective
• 23:44
  as well. So what I think will happen is
• 23:47
  People will have to start fine tuning, the models and start to
• 23:50
  own the modules, so that they have more control over it, so
• 23:53
  that they can deploy it on their own, their own servers so that
• 23:57
  should reduce costs, right? Because if right now what we
• 24:01
  pay, when we make an API, call this on the tokens, which
• 24:05
  doesn't really make that much sense.
• 24:07
  And and yeah and and eventually I think as things get more
• 24:11
  complex, you have to start building custom modules.
• 24:14
  So I've seen a lot of people like Become scared and I was one
• 24:18
  of them as well. And I was, I thought that.
• 24:21
  Okay, you know, these modules with all the emergent properties
• 24:25
  I going to make data scientists obsolete.
• 24:28
  Well, I don't think that's that's the case because while it
• 24:32
  make, while it makes it easy to build applications, it pretty
• 24:36
  soon. You'll have to move pretty soon.
• 24:37
  We'll have to build custom modules, but what I really like
• 24:41
  about the about the apis is how easy it is for Forest.
• 24:47
  Up to build a product and get it out there really quickly.
• 24:50
  And and when you do nowadays, you don't have to be a company
• 24:54
  with a lot of funding, or any incumbent with a lot of data to
• 24:58
  build these products. You can be.
• 25:00
  So, for instance, our Chrome extension, right?
• 25:01
  It does, you know, like some changes like the tone of the
• 25:05
  text makes the text more formal summarizes.
• 25:07
  It stuff like that building a product like that.
• 25:10
  With no data would take us months, right?
• 25:13
  I don't think, I don't think any company in the right mind would
• 25:17
  but try to even attempt to do that because this is not enough
• 25:20
  training data out there to do it.
• 25:33
  But now you can do it and you can create something very nice
• 25:37
  really quickly, get a lot of customers.
• 25:39
  And that's, I think the really powerful thing about LL arms
• 25:43
  especially like in machine learning there was that one
• 25:46
  phase where everybody was picking up the most optimal
• 25:50
  model, you know, like everybody was doing hyper parameter
• 25:53
  tuning, and then eventually people were like, we don't
• 25:56
  really need to hyper parameter to now much.
• 25:59
  We really need to build models that are very specific.
• 26:03
  Affective domain and is that something that you're seeing
• 26:07
  with this as well which is if we want to run these massive
• 26:11
  particles in to be able to make a business case for them and to
• 26:15
  be able to scale them, be need to have smaller domain specific
• 26:19
  models as compared to larger models because of the
• 26:23
  reliability. And you know the other issues
• 26:26
  which come at scaling Yeah exactly.
• 26:30
  Yeah. You just yeah that's what I've
• 26:32
  been trying to say. One thing that I definitely
• 26:34
  wanted to know but it's totally changing gears.
• 26:38
  So maybe we can keep going down this lme's path a little bit
• 26:44
  because I do like the idea and I think it's been, it's become
• 26:48
  very obvious that you can validate ideas very quickly with
• 26:53
  an opening. I API call and Once you start to
• 26:59
  hit scale, it very quickly becomes unreasonable.
• 27:04
  And so it's almost like your success with a app that you
• 27:11
  built on open. AI is also your downfall and
• 27:14
  especially I really liked what you said is when you want to
• 27:18
  start going and making a few calls or you're chaining things
• 27:22
  together or you're making it more complex, that's where Start
• 27:28
  to see really difficult problems coming in that.
• 27:31
  It doesn't feel like it's fully solved and so maybe you have
• 27:34
  Watcher models. And I've, I've heard some really
• 27:37
  cool stuff that people are working on to try and figure out
• 27:42
  how to do this, but it is very much like, I mean, I guess the
• 27:46
  big companies have been doing it for a while.
• 27:48
  Like, if you talk to people at Facebook, they're not just using
• 27:51
  one large language model, they're using a bunch of them
• 27:54
  together and each one has its specific tasks and it's doing
• 27:57
  Certain things. But then you have like so much
• 28:01
  complexity of trying to get all these models to work together
• 28:04
  and how are you calling them? Is it just like 50 API calls
• 28:08
  that doesn't really work because there's not like 50 different
• 28:11
  models that you can call out to. So it's a little bit of a
• 28:15
  conundrum right now where we sit maybe by the time this podcast
• 28:19
  comes out, it won't be as bad because there's some new open a,
• 28:22
  I update that changes everything, but I do think that
• 28:26
  you are hitting Something very, very important that gets swept
• 28:32
  under the rug a little bit in the conversation.
• 28:35
  What everyone loves to talk about is how much these large
• 28:40
  language models have, democratized the ability to use
• 28:43
  machine learning and how you can do so much with them.
• 28:46
  But then the actual production use cases are a whole different
• 28:51
  story because like you said, you still need to have these chops
• 28:55
  to actually bring the model in house and Then really see,
• 28:58
  actually what part of this gigantic model is what we need.
• 29:03
  And can we fine-tuned for that? And actually, can we prune it
• 29:05
  down? Can we see, can we get the same
• 29:08
  kind of benchmarks if we're using a much smaller model, we
• 29:11
  don't need to use this gigantic model so I love everything about
• 29:15
  what you're talking about it is ringing true on so many
• 29:18
  different levels and I love the part you mentioned on prompt
• 29:22
  engineering especially because this is another conversation
• 29:25
  I've been having but many people say Are open and saying, let's
• 29:29
  pumped into new and get as much Optimal Performance out cover
• 29:34
  models as possible in. There's another set of people
• 29:37
  who are saying, you know, there's human feedback which is
• 29:40
  going into these models which is updating and making these models
• 29:44
  more powerful as they come. And there's a very good chance
• 29:46
  that by the time. Gbt 5 comes out, we may not need
• 29:50
  to use these process much. So I wanted to get your opinion
• 29:55
  on prompt engineering like as a Like discipline as well.
• 29:58
  And how do you feel like how much prompt engineering but
• 30:02
  people do in terms of like just a research project or like a toy
• 30:06
  project voices, a production in the nation?
• 30:10
  Yeah, absolutely. And yeah, so it's really hard to
• 30:14
  talk about these things because like you said, you know, like G
• 30:18
  PT 5 could come out. Like, I don't know, in a few
• 30:20
  days and, and just make everything I said Obsolete and
• 30:25
  and I will be like the weird guy.
• 30:27
  Give it a podcast on YouTube. He's is you know said random car
• 30:31
  going to date it today is April 25th.
• 30:35
  So in case by some like Stroke Of God, G PT, 5 comes out
• 30:41
  tomorrow or between now. And when we release this,
• 30:45
  everyone knows that. Today, when we were talking
• 30:47
  about this, it is the world is like, we are saying it is if
• 30:51
  it's but continued. Sorry, I didn't mean to derail
• 30:53
  this conversation. Yeah.
• 30:56
  No, that's all right. So about prompt engineering,
• 30:58
  right? When we started building this
• 31:00
  extension GPT for hadn't come out yet, and the API that was
• 31:06
  available was the, the wind she 002 API and we, you know, we had
• 31:12
  built like a pretty good. We've done some prompt tuning,
• 31:16
  some prompt engineering and and had a set of prompt and they
• 31:19
  were working really well, you are getting really good results,
• 31:22
  but as soon as you pity for came out, a lot of those problems, It
• 31:25
  stopped working, or maybe they work, but they weren't as good.
• 31:29
  So so, you know, we had to so we had a choice at that time,
• 31:33
  either continue with the Vinci 002 or, you know, move to the
• 31:39
  the Vinci 003 and point. And, and we didn't really show a
• 31:42
  lot of people say that it's really easy to move but but in
• 31:46
  reality it's not a lot of the work you've put into prom is
• 31:49
  very easy to change the change where you're making the API call
• 31:52
  but but all the prompt engineering and stuff you've
• 31:54
  done May May be Obsolete and may not work.
• 31:58
  So we decided to, you know, stick with the 002 Epi.
• 32:02
  But I think open a.i., they decrease, the amount of load
• 32:06
  that the TT I could handle. And at times we it would take
• 32:10
  like a minute. Maybe even more to get a
• 32:12
  response especially like I think around February or something it
• 32:16
  makes it got really slow. So so you know we had to move to
• 32:20
  the 003 API. So so yeah so that that that's
• 32:24
  also a Another challenge with from Tim's in earring, because a
• 32:27
  new model could come out. Your old, crumbs could become
• 32:30
  obsolete. It will level the playing field
• 32:32
  for sure, but at the same time, it reduces the reliability in
• 32:36
  your product. It reduces, you know, if you
• 32:39
  have delays or kpis about how fast or you know, you want the
• 32:42
  results to be out there, how trustworthy ends and how good if
• 32:47
  you want to Benchmark of your results, you know, you may not
• 32:50
  be able to maintain that anymore especially because open a
• 32:53
  actually doesn't have any essays or kpis.
• 32:55
  For the endpoints. Yeah, they haven't come out and
• 32:59
  said anything official about that.
• 33:02
  So yeah. So there's that that's also
• 33:03
  another reason why I think it's crucial that that you move away
• 33:08
  from from apis and try to. I don't think it's worth
• 33:12
  training. These models would at least
• 33:14
  fine-tune these modules for your application.
• 33:17
  That's so funny. You say that there is something
• 33:19
  that you mention there, which I think is worth noting one is
• 33:25
  That there is no SLA that open AI has set out with and we
• 33:31
  probably all have at some point or another gone to chat gbt and
• 33:35
  it's like, we're experiencing too much of a load.
• 33:38
  So let me write you a poem about how we don't have anything
• 33:42
  working right now. So like, I don't want to fucking
• 33:44
  pull men. Yeah.
• 33:46
  How can I be able to get this? Whatever, my task is done.
• 33:50
  And so, if you're building a product that is fully reliant on
• 33:53
  that API, it's It's very scary in my eyes and I'm sure there's
• 33:59
  Sr. He's out there that are
• 34:00
  listening in there. Just like that's a no-go you
• 34:02
  cannot do that and that I think is another reason why you see a
• 34:05
  lot of it feels to me like there's a lot of great projects
• 34:09
  that are out there but they're toys projects and they're doing
• 34:12
  it there in that first phase of hey can we validate quickly
• 34:16
  with? Open AI but then we have to
• 34:19
  figure out something else because we don't know if it's
• 34:22
  going to be up or not. And then the other thing that I
• 34:24
  wanted to mention from what you said, Ted and I was just talking
• 34:26
  to somebody about this yesterday, is, if you have let's
• 34:29
  say and this may have been your scenario.
• 34:31
  I don't know. But you have 300 prompts that
• 34:35
  you were using with GPT to and now G, PT 3 comes out.
• 34:40
  How are you going to migrate everything?
• 34:42
  All the infrastructure you've created around those prompts and
• 34:44
  everything over to GPT 3. And then you realize like oh
• 34:48
  half of these prompts don't actually work like they did in
• 34:50
  gbt to, you know. So it's just an absolute
• 34:54
  headache and especially every You bring in a new model, it is
• 34:57
  a little bit weird and then so top that off like let's think
• 35:02
  about 00 if you're using GPT for for some some prompts but then
• 35:08
  you want to go to an open source model and it totally reacts
• 35:11
  completely different. It's like night and day.
• 35:14
  You can't even use the same prompts at all and so you have
• 35:18
  to restart from scratch again. And that's that's just a bit of
• 35:22
  a mess and I appreciate that you're pointing these things
• 35:25
  out. Because Cuz again, like a lot of
• 35:28
  this gets swept under the rug when an especially if you hear
• 35:31
  VC's talking about how this is the future and there's the llm
• 35:35
  Ops hype that I've hopped on the wave and I love it.
• 35:39
  But I also am very, very realistic about it and so, yeah.
• 35:45
  And so I, I didn't mean to interrupt you, but I'll be also
• 35:48
  mentioned, how these models are also getting better in some ways
• 35:52
  because they're incorporating feedback.
• 35:55
  So what you have to do is you have to maintain like, maintain
• 36:00
  these prompts, and you also have to maintain like some outputs
• 36:04
  and you have to constantly Benchmark.
• 36:06
  The the outputs that you are getting today with your ideal
• 36:10
  outputs and if you move to a new module, you have to do that
• 36:13
  benchmarking as well. And it's not easy to it's
• 36:18
  possible to do that, but but it's not that simple and
• 36:22
  especially if you have a lot of Trumps, right?
• 36:24
  And if your, if your In is really complex, so, so, yeah,
• 36:28
  definitely. I don't see a lot of people
• 36:29
  saying this, but right now, your crowns are your gold, right?
• 36:34
  That that's what differentiates you from other companies.
• 36:37
  So you should keep your prompts, you know, lockdown as if they
• 36:40
  were your API Keys. That's one thing you should do.
• 36:44
  And at the same time, you should hear because that's, you know,
• 36:47
  if you come up with a, a prompt right now is the equivalent of a
• 36:51
  custom Tree in deep learning Model A few years ago, right?
• 36:56
  Yeah, if you want very special song.
• 36:59
  Yeah. Yeah, exactly, exactly.
• 37:02
  You need to keep it locked down. You need to have, you need to
• 37:05
  test your outputs with your benchmark outputs.
• 37:08
  And yeah, you need to set up tests for that guy.
• 37:11
  See, a new shirt coming. Keep your promise to yourself
• 37:17
  that. That's a good one.
• 37:17
  Yeah. Yeah, dude, I want to know so I
• 37:22
  got I got one for you. You've been you've been like
• 37:28
  really sinking your teeth into all of this.
• 37:30
  Where do you think you have succeeded?
• 37:33
  Where others have failed when it comes to any of this?
• 37:36
  Whether it's tiny ml or it's working with large language
• 37:38
  models. That's it that's a good
• 37:40
  question. So with tiny a middle.
• 37:42
  I think something that I'm really proud of is the standards
• 37:48
  and blessed. Best practices that I've come to
• 37:52
  learn about tiny and middle. I've been going to put that out
• 37:55
  there and help other people in the field.
• 37:58
  So I'm really proud of that I guess and with a little m's I
• 38:01
  think I'm I'm just happy I'm not that caught up in the hype.
• 38:06
  A lot of people I like so caught up in the hype.
• 38:10
  They, they don't realize that these applications don't really
• 38:14
  work like they do work level. But then not, you know, it's not
• 38:18
  really great for production and yeah I think.
• 38:22
  Yeah. Okay that's where we'll be
• 38:24
  succeed. I think you definitely hit the
• 38:27
  hammer and the nail. I don't know what they he did
• 38:30
  Ms. But despite the fact that these models are openly
• 38:34
  accessible via API, did not open sourced and they are not as
• 38:39
  accessible as most people would like to believe that they are.
• 38:44
  Yeah, yeah. Absolutely.
• 38:48
  It says there's very few scenarios where we think it
• 38:52
  makes sense to declare an llm, well, heartache, and I might get
• 38:57
  hit messages after this whole podcasters, but 2008 rushing
• 39:02
  lie. Yeah, it's refreshingly real
• 39:05
  takes and I appreciate it because, man, there's just
• 39:10
  something that since like GPT chat gbt came out.
• 39:14
  And then if you scroll Twitter, At all, and your Twitter feed is
• 39:18
  probably just blowing up with this change in the world.
• 39:22
  Nobody knows how to use chat gbg.
• 39:24
  Like I do. Here's the best prompts or here
• 39:26
  is like how this is going to change life forever.
• 39:29
  And there's so much hype in it and it's nice to actually talk
• 39:34
  to somebody. Who's like, yeah, I've been
• 39:35
  doing this for a while and here's where there's some real
• 39:39
  Hang-Ups that if you are going to use chat, gbt you have to
• 39:45
  think about these things and think.
• 39:46
  Through these things. Yep.
• 39:48
  Yep. You think another thing though.
• 39:50
  Me a note. So no doing really well with
• 39:52
  this product. Is we taking it we're not trying
• 39:55
  to get caught up in the hype. You're trying to analyze things
• 39:58
  objectively and trying to trying to grow the product, how we
• 40:03
  think it would, or trying to build this application in grew
• 40:06
  at how we think any other application would would grow,
• 40:09
  you know, like add features slowly, verify them, make things
• 40:13
  more complex, slowly. So for instance, right now we
• 40:15
  haven't the Thing we want to do is actually find doing a model
• 40:19
  and maybe compress the module and then try that out and see if
• 40:23
  that helps we haven't even like tried on tried out Vector
• 40:26
  databases or any of the it it's great.
• 40:30
  We have those tools but we haven't really had the need for
• 40:33
  them. You know, we could integrate
• 40:34
  though. But but here doesn't make sense.
• 40:37
  Yeah, you don't see it yet. So with the compression here
• 40:40
  angle, I find it fascinating that you mentioned it adds to
• 40:44
  the bias and also I imagine you've been trying to figure out
• 40:48
  yeah. Like pruning down these models
• 40:50
  and also the thing that we didn't say yet that probably
• 40:56
  needs to be said when it comes to open source models, there's a
• 41:00
  lot that's left to be desired out there on the open source
• 41:03
  market and yeah there's been some releases or leaks, you
• 41:10
  know, like Mama you can't actually use llama, you can play
• 41:13
  around with it but if you want to create an app with it, you
• 41:16
  can aunt. And yeah.
• 41:18
  Like that's that's a whole nother topic for another day,
• 41:22
  but talk to me about this bias and how did you recognize that
• 41:25
  you? There is more biases when you
• 41:27
  try and compress it, sorry not make it more so maybe I'll take
• 41:31
  an analogy from tiny little. So one of the people who found
• 41:36
  out about this bias was this researcher called saddle hooker.
• 41:40
  She has a paper on it. I can't recall the name right
• 41:43
  now, but it's a really nice paper.
• 41:45
  I think it goes. The problem really quick.
• 41:47
  Really, he's like you know, in a way that's easy to understand.
• 41:51
  So so what happened? So how the bias comes out is,
• 41:55
  let's say you're trying to, let's say you're building a
• 41:57
  camera trap, right? So with that will, so it's like
• 42:01
  it's checking to see if some animal crosses and, you know,
• 42:05
  identify the species, right? You know, you could build a
• 42:10
  module that identifies like common animals.
• 42:13
  Like, I don't know, like elephants zebras.
• 42:16
  I don't like animals that are big.
• 42:18
  We have a lot of pictures of them.
• 42:20
  It's easy to and also integrate like other animals that are less
• 42:24
  common because that's what you want to capture it in a camera
• 42:27
  tripod like animals. Like I don't know, like a snow
• 42:30
  leopard, something like that. You won't find them in the same
• 42:34
  habitat but like, let's see. You have the same model for
• 42:36
  that. Now the snow leopard you'll it
• 42:39
  will be in your module, it will be a part of like the long tail
• 42:42
  at the end where you have few images and the model.
• 42:46
  Your large model might be able to recognize it really well.
• 42:49
  But what happens when you compress your model is your
• 42:52
  model when try to optimize for those examples in the data set
• 42:57
  that it seemed a lot and forget these at the cost of these
• 43:01
  examples that it hasn't seen a lot, right?
• 43:04
  So you'll have a bias towards things that occur more often
• 43:07
  when you when you compress down your modules and that's
• 43:10
  something I think we will see happen with with large language
• 43:13
  models as well. When we compress them, you might
• 43:16
  lose. Lose vocabulary.
• 43:18
  You might lose a lot of a lot of the richness I guess of the
• 43:22
  larger models, when you move, when you try to compress your
• 43:25
  models. So good man.
• 43:28
  Well this has been awesome and and I really appreciate you
• 43:31
  coming on here and given this real take I wonder if how things
• 43:36
  are going to play out and if it is going to be something that we
• 43:40
  see people going back to this in a few years and saying like, oh
• 43:46
  yeah, that Is a little bit of a bubble and that was a bit of a
• 43:49
  crazed. Like the crypto craze that
• 43:51
  happened, or if it's just going to continue advancing in all of
• 43:55
  this stuff that we're talking about, all these pains that were
• 43:57
  mentioning right now, we're going to overcome them in the
• 44:01
  next couple of years. And we're going to figure out
• 44:02
  ways to work past them. I'm actually absolutely
• 44:06
  fascinated by that. And yeah, I think that's it and
• 44:11
  we'll leave it there. Thank you for calling.
• 44:14
  Thank you so much for having me. Hey, I'm Vishnu.
• 44:20
  I'm a data scientist at first hand, and I definitely think
• 44:24
  that you should subscribe to the ml Ops coffee sessions podcast.
• 44:28
  It's the best podcast out there to stay on top of what ml Ops
• 44:31
  actually is to talk to the true thought leaders in the space.
• 44:34
  And oh, by the way, dimitrios is absolutely hilarious.
• 44:38
  What a weird guy, you should definitely subscribe to the
• 44:40
  podcast.