Latest / Agentic Conversations (formally mlops.community) / Retrieval Augmented Generation

Transcript

Retrieval Augmented Generation · Agentic Conversations (formally mlops.community)

• 0:00
  My name is Sayed Assad, you can call me Assad and I'm a senior
• 0:04
  or a lead AI engineer here heading the AI vertical in Kiwi
• 0:08
  Tech. And Kiwi Tech is actually a US
• 0:10
  based company headquartered in DC and having his offices in New
• 0:14
  York and one of his offices in this Delhi, New Delhi where I am
• 0:18
  right now. OK, so I'm not a very hard
• 0:21
  coffee drinker. I would say I would be loving a
• 0:24
  flat white coffee with some cream on it.
• 0:28
  That's all. Welcome back to another Envelope
• 0:32
  Community podcast. I'm your host Dimitrios and
• 0:34
  today talking with Assad, this was a breath of fresh air.
• 0:38
  He gave us all the tea, as the youngsters say, on what is
• 0:43
  working and what is not working when it comes to tools.
• 0:46
  I appreciate this conversation because he did not sugarcoat
• 0:50
  anything. He said what he has been testing
• 0:55
  and what has worked and what has not worked in production as far
• 0:58
  as open source projects as far as tools and we mainly played on
• 1:03
  the inference layer, but he was able to go a little bit in and
• 1:07
  out. We referenced a conversation
• 1:09
  that we already had which was a live conversation that will link
• 1:13
  in the description where he talked a lot about Rag and the
• 1:17
  problems and challenges he was having when building rags.
• 1:20
  For those who do not know, he is a consultant and he is, as he
• 1:24
  mentions in the conversation, time bound.
• 1:26
  He's basically put on the spot and needs to get something into
• 1:30
  production as fast as possible. I really like that because he
• 1:35
  has to go and figure out the best ways to do something.
• 1:39
  Right now it's not like he can explore and then maybe in a few
• 1:45
  weeks or a few days this could get into production with a lot
• 1:48
  of work. He has to see what's out there,
• 1:51
  what's working, how can it work for me.
• 1:53
  And then he tests it and he iterates and he talks about how
• 1:56
  it falls over or what's working and what's not working.
• 1:59
  We get right into it. He talks about some production
• 2:02
  level problems that he was having with one of the rags.
• 2:05
  That was the first thing that he says.
• 2:07
  And I also really like this that we you'll if you stayed at the
• 2:12
  end you'll hear it because it's almost like this bonus part that
• 2:14
  I got out of him after I asked him what his favourite type of
• 2:17
  coffee he was. He talked about how his company
• 2:20
  has a whole ML research team, and this is not ML research in
• 2:24
  the way that they're trying to find new architecture for
• 2:27
  models. It is mill research team that
• 2:30
  goes out there. They test and they pit two
• 2:32
  different frameworks or two different tools against each
• 2:34
  other. They figure out which frameworks
• 2:37
  and tools are better for which scenarios and then they push
• 2:40
  that all to a GitHub repo so that they have it in mind in
• 2:44
  case a client comes to them with certain requirements.
• 2:47
  They already know they've tested this tool.
• 2:49
  It works for that scenario and I think that is brilliant and then
• 2:54
  I appreciate him sharing that information with us right now.
• 2:57
  It's a really no holds barred conversation.
• 3:00
  We're not making any friends in this conversation, especially if
• 3:03
  you are a company that plays at the inference level.
• 3:08
  We may lose a few sponsors from this, but it's worth it because
• 3:11
  the truth needs to get out there and he is speaking his truth
• 3:14
  100%. Let's get into this
• 3:15
  conversation. As always, in case you like it,
• 3:19
  please share it with a friend, let others know and give us some
• 3:24
  stars or some likes or feedback, anything of the sort.
• 3:29
  We love to hear from you. Let's start with this, man.
• 3:38
  You've just mentioned you're having a production issue at
• 3:41
  this very moment, just before we started recording.
• 3:43
  What's going on? Yeah so so it it will seem to be
• 3:48
  a very easy issue. There is a huge chunk of CSV
• 3:52
  data which many people claim on the Internet or the LinkedIn
• 3:58
  that we deal with vast amount of data and that can be handled in
• 4:01
  just a blow. So here the problem goes.
• 4:04
  So we have to develop a sort of ragged on that CSV data.
• 4:10
  So the size of that, I'll be very precise with the terms.
• 4:15
  The size of that CSV file is 133 MB.
• 4:20
  That is a huge fight as far as end.
• 4:22
  The file is perfectly formatted, there is no data cleaning
• 4:25
  required, everything is fine. Basically there needs to be a
• 4:29
  chat bot which could answer that.
• 4:30
  So the first approach would be to go ahead and straight away
• 4:35
  develop vector embeddings, do some sort of push it into a
• 4:40
  vector database and do a retriever engine type of thing.
• 4:43
  So the primary challenge was that when you directly convert
• 4:46
  that CSP into a vector embedding, it will not convert.
• 4:49
  No, it will not convert. So, so there is.
• 4:53
  So there is a huge fuss. Going it is going.
• 4:55
  It doesn't matter what model you're using, it doesn't.
• 4:58
  You've tried. It is not making any effect.
• 5:00
  I'll name the models also mixed bread, I'll snowflake, Arctic,
• 5:04
  normal sentence Transformers, mini embedding, any dam model
• 5:08
  that is not making any difference, Any context size,
• 5:11
  any vector size. It is not even starting to
• 5:14
  generate the embeddings. I have tried best of the systems
• 5:18
  I've tried, best of the Max, tried desktop, it is not even
• 5:22
  started. So I was stuck at 0 point .0 I
• 5:27
  mean. Do you know why that is?
• 5:29
  Not sure why it is happening. I just tried to gather the logs,
• 5:33
  anything like that, but this was never happening.
• 5:35
  I mean it was having some sort of public data.
• 5:38
  I mean the data is public, it is not a private data.
• 5:40
  So I can just tell what was inside.
• 5:42
  It's some sort of agricultural produce data of the US
• 5:46
  government. So it is a huge file and
• 5:49
  somebody makes to make a query on that file, some sort of
• 5:52
  analysis. So the first approach which I
• 5:54
  applied without since the vector embedding was not working, I
• 5:57
  used pandas AI, pandas Ki to do the vector search.
• 6:03
  It is failing miserably. Two or three queries error error
• 6:07
  error no module found. There is an error no module
• 6:10
  found it is gone. And when I check that error on
• 6:12
  the Internet it is still open GitHub issue so that is of the
• 6:16
  list. So the second thing is that I
• 6:19
  needed to do some sort of ETL on that data, then transform that
• 6:25
  data into a vector embedding and then push it into a vector DB
• 6:29
  and then query from there. So now the problem was that
• 6:32
  problem is in creating vector embedding.
• 6:35
  So I I tried to use mix speak, the one which you are pointing
• 6:40
  out that day and that person who is the founder of mixed speak,
• 6:44
  his name is Ethan and then Ethan, he was a very helpful
• 6:48
  person. He went on a call, he was on a
• 6:50
  call with me also a few days back.
• 6:52
  He was doing back and forth. He created the entire pipeline.
• 6:55
  So he used the mixed breed model and that model and that pipeline
• 7:01
  was able to convert the vector embeddings to a mongo DB vector
• 7:06
  search. But the point is that that file
• 7:09
  was very small which I had given to him as A to play around.
• 7:14
  I mean, I was given around the 133 MB file and today is
• 7:20
  Tuesday. I need to go into production by
• 7:22
  Friday, so Friday is my deadline including the testing.
• 7:26
  So so this approach. Although I would not say that
• 7:29
  this approach is failing but this approach needs some time on
• 7:32
  iteration and testing. I need to be sure it is working.
• 7:36
  And so when you look at this again and if you encounter
• 7:41
  something like this, do you feel like the danger in this scenario
• 7:47
  is the size of the CSV file? Or where is there the danger and
• 7:53
  where is there something that you can now have a bit of a red
• 7:56
  flag off in your head if you see it again?
• 8:00
  The danger is actually in the size of the file also.
• 8:03
  I mean the amount of data also, I won't say it is a big, it
• 8:07
  comes under the category of big data analytics.
• 8:09
  I know there are tools Airflow can handle this very easily.
• 8:12
  But seeing the timeline of me going into production, I mean
• 8:17
  everything is not possible in that particular time span.
• 8:20
  So the size is one of the factors which is the bottleneck.
• 8:23
  The other factor which I would say is that the complexity of
• 8:26
  the data because there is similar data also in that file.
• 8:31
  What if what if somebody's querying a data regarding the
• 8:35
  production of sugar cane in Mexico?
• 8:38
  Yeah, so that one that what is the guarantee that it will
• 8:41
  result the correct result. So that is one of the things.
• 8:44
  So vector embedding, I figured out that vector embedding will
• 8:47
  not be able to handle this type of data because Mexico was
• 8:50
  coming many at many places, Sugar King was coming at many
• 8:53
  places. So it will go with the context
• 8:56
  and it might result in some sort of confusion inside the
• 8:59
  document. Ohh wow.
• 9:02
  OK. Yeah.
• 9:04
  So vector embedding will not work in this case even if I try
• 9:07
  hard with the ETL process. So at the end there is a
• 9:10
  solution for this which fortunately worked.
• 9:15
  I converted that entire CSV into parkit format.
• 9:19
  Parkit that parket format reduce the size from 1:33 to 9:00 MB
• 9:28
  without too much loss. Then I made a normal lamma index
• 9:32
  agent, it queried and it is giving just blazingly fast
• 9:37
  responses in one or two seconds maximum.
• 9:40
  Everything is correct. OK, so I really like you
• 9:44
  pointing out the fact that this doesn't.
• 9:46
  It doesn't work with normal embeddings because of the amount
• 9:50
  of times that the same topics come up in very different
• 9:54
  scenarios and so understanding your data in that regard really
• 9:59
  helps you recognise if a vector or sorry if an embedding model
• 10:04
  will be the right use case and so you didn't do any embedding
• 10:10
  model. Number.
• 10:11
  Yeah, you said get rid of that whole idea of trying.
• 10:15
  To make things. Because.
• 10:17
  Gone with the So I didn't do that because and to be very
• 10:21
  frank when you check today's AI is development in the field of
• 10:25
  AI agents came after rag after axo.
• 10:32
  They are actually replacing the complexities of the.
• 10:35
  I won't say that vector search is losing its significance.
• 10:37
  It is still relevant and it is still relevant for many of the
• 10:40
  things like PDF data. And I would rather say hybrid
• 10:44
  search is also something which is required.
• 10:48
  Sparse embeddings plus vector embeddings combine the results
• 10:50
  and all those things. Yeah SO and one of the tools
• 10:54
  which I personally favour is. I have been continuously using
• 10:59
  in production. Is fast.
• 11:00
  Embed fostered is very very fast.
• 11:03
  I mean very, very fast. Yeah, shout out to Niran, who
• 11:07
  created that. That is great and you're able to
• 11:10
  use it in production with no problems.
• 11:12
  Yes, it is being used in production with no problems and
• 11:14
  it is even reducing the infra size also of the docker
• 11:17
  containers also OK so that is also a good thing.
• 11:22
  So I wanted to play around a little bit when it comes to the
• 11:26
  topics of this conversation on the inference layer and really
• 11:30
  dive into what you've been doing there, what you've been
• 11:34
  discovering, what have you experimented with thus far when
• 11:38
  it comes to inference? And maybe just start with what
• 11:42
  are some of the use cases that you have and then go down the
• 11:48
  rabbit hole of what you've been trying to use for these this
• 11:51
  inference layer. OK, I'll go.
• 11:55
  I'll go with the one of the use case which which was two days
• 11:58
  back having some issues The you remember that I talked about the
• 12:04
  multimodal rag in which in which there was a memory in which you
• 12:09
  can pick it out by searching SO from the last talk.
• 12:13
  So that that thing was working perfectly fine.
• 12:15
  But suddenly what happened? The docker container on which it
• 12:19
  was running or the infrared it was running, it was consuming
• 12:22
  around 3:00 to 4GB of space because the data was limited not
• 12:26
  not very much. The embedding model was using
• 12:29
  three to 4GB of space on the AWS infra I'm talking about.
• 12:34
  Suddenly three days back, it dropped from three GB to 400 MB
• 12:39
  and that embedding model stopped using the GPU although GPU space
• 12:43
  was allocated to it So and this resulted in the queries becoming
• 12:50
  more stale, the retrieval times all automatically increased.
• 12:56
  There is no problem with the embedding model, everything is
• 12:58
  working fine. When you check that code on a
• 13:00
  different system, it works fine. Absolutely.
• 13:03
  So what happens? It is actually linked with the
• 13:08
  asynchronous processing of the data.
• 13:11
  How the back end is integrating into the into the final app.
• 13:16
  So one of the backlogs which I think is that the software
• 13:21
  people don't understand what is happening in the AI.
• 13:25
  They do it the software way and so that that particular thing
• 13:30
  there were multiple videos and those videos required sequential
• 13:34
  processing rather than parallel processing.
• 13:36
  They put all the videos to be searched parallely.
• 13:41
  So it was taking too much memory and it was not giving any
• 13:44
  results. So this is one of the thing and
• 13:47
  at this point of time I would say the auto scalability option
• 13:50
  of AWS which is a widely popular option was not also working and
• 13:55
  handling the issue. The.
• 13:57
  Which usually handles all the issue, so I don't think it's.
• 13:59
  Using which service from AWS? Because I know there are almost
• 14:05
  like infinite services here. Yeah, the AWS service which we
• 14:09
  were using is that the EC2 instance which we are using on
• 14:13
  that. And so I don't think so at this
• 14:17
  point of time. Such level of complexities can
• 14:20
  be handled by these startups. They might improvise in the
• 14:24
  future but the startups which I used because because because I
• 14:31
  personally go ahead with that Kubernetes layer and what is
• 14:38
  because monitoring of logs is a very important concern.
• 14:41
  How and why things are failing? We use salary containers also.
• 14:46
  So usually what happens whenever you deploy a model, the workers
• 14:50
  in salary often get overloaded. That is one of the bottlenecks.
• 14:55
  Celery is where you were getting problems because it couldn't
• 14:59
  handle the load. Celery.
• 15:01
  Celery was handling the load, but you never know when celery
• 15:05
  could pipeline bus. That pipeline keeps on bursting.
• 15:09
  The workers get overloaded. They bust.
• 15:13
  Signing. Off.
• 15:15
  I think personally Mill OPS is something which is a hugely
• 15:19
  hugely, say having scope, huge scope.
• 15:23
  Is there an envelopes at this point of time?
• 15:26
  Yeah, because just in this conversation, you're talking
• 15:29
  about going from the Kubernetes layer to the data engineering
• 15:34
  layer to the inference layer to the model layer.
• 15:39
  Yeah, there's a lot you need to know in each one of those
• 15:43
  subsets. And so your production was
• 15:47
  struggling because of the auto scaling on AWS because celery
• 15:52
  was just kind of flaking out every once in a while and you
• 15:55
  didn't have the correct notice time or you didn't have the
• 16:00
  confidence that it could stay working in production.
• 16:05
  What did you change? So what we did is that we tried
• 16:10
  to we tried to test the load on every worker.
• 16:15
  The first step we started. So there were few workers which
• 16:18
  were getting overloaded. So initially we tried to
• 16:22
  increase the GPU, GPU and the V CPU also.
• 16:26
  But to be very frank increasing the VCP when the GPU does not
• 16:31
  really make any difference. It will it make a difference It
• 16:35
  it made the production running those salary workers were not
• 16:38
  overloaded. But you need to have some sort
• 16:42
  of mechanism to test because the AWS logs are not that accurate,
• 16:46
  which can give you an idea from the standpoint of a machine
• 16:49
  learning engineer. Then.
• 16:52
  So that is the one of the bottlenecks.
• 16:53
  Also the AWS needs to upgrade. Yeah, so they're giving you logs
• 16:59
  that are helpful for the SRE or for the DevOps team, but it's
• 17:02
  not very helpful for the machine learning engineer.
• 17:06
  Yes. And what happens the team, those
• 17:08
  teams also they get a log. I mean they do we do.
• 17:13
  We deploy time loggers to check where exactly the time is going
• 17:16
  on. So there is one more issue, the
• 17:19
  embedding model keeps loading again and again.
• 17:22
  Although I have loaded that model globally once but it does
• 17:27
  not work. It keeps loading again and
• 17:29
  again. It consumes memory.
• 17:30
  It takes time. So that is one of the more most
• 17:32
  problem. I mean that is still not sorted
• 17:34
  out. O Let's dive into the inference,
• 17:37
  because I know you've been doing a lot of testing with different
• 17:40
  startups and AW itself. What have you found?
• 17:45
  The one thing I found is that you can start like Salad.
• 17:50
  Because I have explored Salad extensively, I've deployed my
• 17:53
  own POC also and I would rather say it was working fine, it was
• 17:58
  working. For the salad for those, because
• 18:00
  I think they have an interesting and unique value prop.
• 18:03
  Yet Salad is actually UH. You can say Salad is an infrared
• 18:08
  inference as a service platform and they they claim that they
• 18:13
  are the most affordable cloud for AIML inference data scale
• 18:18
  and they're starting price is 0.02 I think .02 dollars per
• 18:25
  hour for the lowest bandwidth. I won't say I mean it will, it
• 18:29
  will suit you. But what I did was that my bill
• 18:32
  was $58 for approximately 1 month for a small POC.
• 18:38
  Nice. OK.
• 18:39
  And the reason that they can do that, correct me if I'm wrong,
• 18:42
  is because they are trying to do distributed computing or
• 18:46
  distributed GPUs. Yes, even if you are having your
• 18:49
  own GPU, you can register and salad and they can use your GPU
• 18:53
  also provided you have an NVIDIA GPU.
• 18:56
  So our clients are in, most of the clients are in the US so
• 19:02
  they're having, they're having good parallel computing
• 19:06
  infrastructure in US. Yeah, and is it fast enough?
• 19:12
  It is not fast. OK, there's the So if you want
• 19:17
  fast, maybe it's not the best option, but it is cheap.
• 19:21
  It is cheap. And why did it fall over for
• 19:24
  you? Like why are you not using it?
• 19:26
  Because the primary problem is that I cannot monitor the logs,
• 19:29
  they they gave a Jupiter notebook type of terminal to
• 19:33
  monitor the logs and I need to train my DevOps team to how to
• 19:36
  observe that Jupiter notebook, so that is another task so I
• 19:39
  dropped that. Wait, so it almost the exact
• 19:43
  opposite problem that you were having with the WS1.
• 19:45
  You had the DevOps team getting DevOps type logs in.
• 19:49
  The MLB team couldn't understand it.
• 19:51
  Now you have Jupiter notebook type logs that the DevOps team
• 19:54
  can't understand. You understand?
• 19:56
  Yeah. That's where it comes to the
• 19:58
  idea of you talking about how this is still a very pointed
• 20:02
  solution, but there's a lot of scope and a lot of design
• 20:05
  decisions as a startup that you have to make to say here's where
• 20:10
  we're going to plant. Our flag in the sand, we believe
• 20:13
  that you should get logs, but you should get them in a Jupyter
• 20:16
  notebook format. And for the people that don't
• 20:18
  like that, well, right now at this very point in time, they're
• 20:22
  not going to be our customers. That's not our ICP or our ideal
• 20:25
  customer profile, right? So that is where I dropped with
• 20:28
  Salad, because I cannot handle the hassle of handling the logs
• 20:33
  myself and just checking the logs every time something is
• 20:36
  failing because because logs are at that stage when you plant the
• 20:41
  product into product production and and at that point of time if
• 20:46
  anything is bursting that is hampering the client experience.
• 20:49
  Also, because we have to, we have to give answers to the
• 20:52
  clients also keep them involved. Yeah, yeah, you're on the hook,
• 20:57
  right? You can't just let something be
• 21:01
  a problem for days. It has to be hours because you
• 21:05
  are basically, and correct me if I'm wrong, you're going into a
• 21:09
  client and you're saying we are going to help you with your
• 21:14
  production environment. We're going to make sure that it
• 21:16
  is bulletproof and we almost give our seal of approval and
• 21:20
  our guarantee that if something goes wrong we can handle it.
• 21:24
  Yes, absolutely, because that is where I'm standing.
• 21:26
  I'm heading the AI part here in my company and I'm in and out
• 21:29
  responsible for everything happening as far as the AI
• 21:32
  product development is concerned and as the deployment is
• 21:35
  concerned because I won't be blaming on a DevOps team that
• 21:38
  they are handling this. So they should be doing that
• 21:40
  because I am primarily responsible for everything like
• 21:43
  what type of model we are handling.
• 21:45
  So that is why I stopped using models from hugging phase.
• 21:48
  They are not suitable. I mean you can go with
• 21:50
  production. I cannot say that is not
• 21:52
  suitable production but they consume lot of space S not
• 21:56
  production friendly. I would say fast embed is a good
• 21:59
  option to go into production. As a lawyer you as usual works
• 22:04
  good, but at the end you need to have your own framework.
• 22:07
  No longer no lama index, no bullshit.
• 22:09
  So you need to have your own things.
• 22:11
  And so you've looked into quite a bit of services it feels like
• 22:16
  and you mentioned the idea of OK, you've got certain options,
• 22:23
  you mentioned the different flavours that are out there.
• 22:27
  So you can go with the Together AI and you can say I'm just
• 22:31
  going to grab a model off the shelf from Together AI.
• 22:35
  It's fast, it works and I'm confident that it is going to be
• 22:40
  working in production. Or you can go a little bit
• 22:43
  deeper down the rabbit hole and say I'm going to go with
• 22:45
  something like a VLM where I understand more, I'm going to
• 22:50
  bring bringing my models, I get more control with the models
• 22:54
  that I want to put into production and it obviously has
• 23:00
  a cost. There's an overhead cost to it.
• 23:03
  And then of course you can just use Open AI right off the shelf,
• 23:08
  right? And so how much of it do you
• 23:10
  think is over engineering when you can just be using Open AI?
• 23:16
  Open AI, there are two concerns. One is the obviously the cost.
• 23:20
  The other concern which is readily coming out is the data,
• 23:24
  data part. I mean nobody knows what they're
• 23:27
  doing with the data, so at the end the client wants something
• 23:31
  which is localised. But if you go to the Olama type
• 23:35
  of thing or lama dot CPP again Olama is difficult to deploy in
• 23:40
  production because when you deploy that Olama call any model
• 23:45
  with Olama and then you go with deploying that container on the
• 23:50
  AWS, it gives it throws out errors regarding remote
• 23:54
  connection issues. I don't know.
• 23:56
  I don't know what exactly happens but I have started
• 24:00
  testing it more rigorously. I won't be able to comment much
• 24:03
  on this because because that is one thing because error
• 24:06
  debugging is too much. So that is one of the pain
• 24:10
  points at the end. Open AI is Open AI works fine
• 24:14
  but I would recommend going with your going with the framework in
• 24:19
  which you have more control on your designs, only more control
• 24:23
  on your things, especially the data and the type of
• 24:26
  customization which you can do like for example the VLM.
• 24:30
  And so for those who don't understand the difference
• 24:33
  between Obama and BLM, can you break that down real fast?
• 24:38
  Yeah Ohh Lama is something which in which you can run the models
• 24:42
  locally by downloading them a day.
• 24:45
  It gives you some quantity in models and quantized fashion
• 24:48
  also and VLM is something which increases the inference speed.
• 24:54
  It does not localise this there. Although there are options for
• 24:57
  localization. It can offer some sort of
• 24:59
  localization, but it increases the inference speed.
• 25:04
  So Olama is more for running models locally.
• 25:09
  It's for figuring out if something works and it distils
• 25:14
  or it compresses the models by default.
• 25:17
  I think I've seen that as like one of the.
• 25:19
  Default Yeah like lama three 8 billion parameters is of 4.7 GB
• 25:25
  in Olama. Yeah, in reality, Which sport
• 25:28
  70? It can be good and it can be
• 25:32
  difficult because if I've seen some people who wanted to use
• 25:35
  like A1B or the, I think what is the Microsoft Phi, Phi or Phi?
• 25:41
  Phi came out how they pronounce it, but it's Phi said here.
• 25:45
  Yeah. And I think I saw some people
• 25:47
  talking about how it wasn't really working with Obama
• 25:50
  because of the compression that you get from Obama.
• 25:53
  And it's interesting to note that basically, if you're trying
• 25:56
  to go into production for now, at least stay away from Olama
• 26:00
  because of the debugging issues, and just trying to figure out
• 26:03
  the errors that it's throwing is not worth your.
• 26:06
  Time. I mean if if you have time you
• 26:08
  can debug the errors. I would say go with Olama but
• 26:12
  not in a time bound environment in which I am working Spring
• 26:16
  system Agile I I work in an Agile scrum based formats.
• 26:21
  Yeah, and it could be something that's good to prototype.
• 26:25
  And then once you see that there's some value, then you can
• 26:30
  go and figure out what the next steps are.
• 26:32
  Yes, absolutely. I mean, it requires some sort of
• 26:36
  research. Usually I'm not able to get that
• 26:39
  research done, so that's the reason why I tried to deploy it.
• 26:43
  But at the end I need to just quickly switch on to a different
• 26:47
  one. And so you're all about this
• 26:49
  flexibility, but also understanding the scope and the
• 26:53
  needs that you have. As someone who is trying to find
• 26:58
  a tool that's going to be doing the majority of the hard work
• 27:03
  for you, did you look into something like a model or a base
• 27:08
  10 when it comes to these services that are out there?
• 27:14
  Because these are the only other two, and maybe there's a
• 27:17
  beam.cloud I know is another one.
• 27:20
  I'm judging by the look on your face that you weren't going too
• 27:24
  deep along the lines of either of these.
• 27:27
  Last this base 10 I started studying out but it is still on
• 27:33
  my hit list. I have not gone into the I mean
• 27:36
  the bot, I mean the grassroots of base 10.
• 27:38
  It is also an inference service but I would really like to try
• 27:43
  it. The reason why I like to try it
• 27:45
  is that because I have got feedback from my peers in the UK
• 27:51
  that based in and Vulture about these two platforms.
• 27:56
  So I have not readily explored them to be very frank on that,
• 27:59
  because there are too many platforms to explore.
• 28:03
  And yeah, I need to at least get on a 15 minutes call with
• 28:06
  anybody on their platform to just to get a good idea of what
• 28:10
  they are trying to offer for my use case.
• 28:13
  Yeah, exactly. That's why I like talking to
• 28:15
  you, because it feels like you've gone out there.
• 28:17
  You've done the hard work of figuring out what is out there,
• 28:20
  what your requirements are, and you've put it into practise, and
• 28:25
  then you give us a little bit of that wisdom of how you've
• 28:29
  learned from it, where it falls down, where it holds U.
• 28:32
  All of that and it's very unfiltered, which I appreciate.
• 28:36
  One more thing I have found out a few days back is the fine
• 28:40
  tuning of small Elms using techniques using Orpo and a
• 28:45
  third party tool unslotted AI. They are working wonders and it
• 28:49
  they're very very easy to do that.
• 28:52
  So all poor technique from that odds ratio preference
• 28:56
  optimization which actually eliminates the two layers, this
• 29:02
  fine tuning and preferential treatment layer and it it
• 29:05
  combines them into one orbo layer.
• 29:07
  So instead of doing massive level of rag and just checking
• 29:14
  on vector embedding, we directly go to Unst Sloth.
• 29:18
  They have ready notebooks available.
• 29:21
  Click on those need notebooks. Select the model that models Rs
• 29:25
  scalably quantized in a four bit format.
• 29:28
  That means if you test that model, that model might be
• 29:32
  having some sort of lossy compression, but that is not
• 29:34
  relevant to you. You train data, train it on your
• 29:38
  data from having face or any of the JSON format and then develop
• 29:43
  a sort of a small rack. So it is a 2 billion or 1
• 29:46
  billion parameter model small model and it works fine.
• 29:50
  And so breakdown the system on one of your use cases that are
• 29:55
  you creating pipelines that are hitting this and then they are
• 29:59
  going out and they're using these inference engines and then
• 30:02
  you have some kind of evaluation framework on the back end, so
• 30:06
  you can retrain. How does that look?
• 30:10
  Yes, evaluation framework I have.
• 30:13
  I have. I have checked deep eval very
• 30:17
  rigorously but deep eval is having an issue.
• 30:21
  You have to give a context and that is the point.
• 30:25
  The question, actual output, expected output and the context.
• 30:29
  They would have three options. So you will be stuck in the
• 30:33
  context part. What you will do in the context
• 30:35
  part. They will give you a score, but
• 30:36
  I don't think so that score is relevant to your unless you give
• 30:41
  a context. So the context part, they have
• 30:43
  already explained in their docs that what you need to put in the
• 30:46
  context, but I don't think so that is relevant.
• 30:50
  The second thing is that I tried to explore two other tools auto
• 30:55
  ragas, the ragas rag evaluation that is working good.
• 31:01
  Did you notice A? Because I've heard people talk
• 31:03
  about ragas and it's a great tool, but some of the challenges
• 31:08
  involved in that are that it can bring your cost U.
• 31:11
  Yes, it brings the cost up. And if you actually do that
• 31:17
  because there is one more thing which can be done to automate
• 31:20
  the testing of large language models.
• 31:22
  From the GitHub it is the CIA creating a CICD pipeline and on
• 31:28
  which Andrew Ng created a course also from LLM eval course.
• 31:33
  I'm forget Circle CI is the name of the tool Circle CI it is a
• 31:38
  very good tool for evaluation. For really concerned for
• 31:41
  evaluation part, there is a full blown free tutorial available
• 31:45
  for on deep learning website for Circle CI but it comes with a
• 31:50
  cost. There API is costing you but it
• 31:52
  is a serverless instance and it will be less as compared to and
• 31:57
  it creates A CICD pipeline so the automation is done.
• 32:00
  It is automated, actually you don't need to interfere in that.
• 32:03
  Yeah, that seems like the dream and that is very interesting
• 32:07
  that it's coming from the CID level where the software
• 32:11
  engineers are used to playing and you can get that eval, you
• 32:15
  can create those pipelines. Yeah.
• 32:18
  And hopefully it's retraining. And I guess I see it in a way as
• 32:24
  like you're going to the sloth as part of your CI CD and you're
• 32:28
  just creating the new fine, fine-tuned model.
• 32:33
  Yeah, I mean, I have still not deployed that Circle CI.
• 32:37
  The reason why I have not deployed is because at the end I
• 32:40
  need to get approval from the clients.
• 32:42
  And at this point of time, I would also like to talk about a
• 32:45
  very important thing, the client or the OR the end user who is
• 32:50
  actually paying for the project. They do not understand the value
• 32:54
  of evaluation. They don't understand how
• 32:58
  valuable it is. They still they still think that
• 33:01
  manual software testing is the way forward and we need to do
• 33:04
  that only. So that is the thing.
• 33:06
  So this needs to be there is lack of awareness and I think it
• 33:11
  will pick up in the coming few months.
• 33:14
  But still did. People don't know what is the
• 33:16
  relevance of LLM evaluation. People are usually testing it
• 33:19
  the manual way now. Yeah, this is fascinating, man.
• 33:24
  So any other pain points that you've come across as you're
• 33:28
  trying to put these different models into production or as you
• 33:32
  have your different use cases that you want to just mention
• 33:35
  here? Because I feel like you've been
• 33:37
  very vocal at the different parts that have been painful in
• 33:43
  your journey and hopefully we can all recognise this and we
• 33:47
  don't have to. Doesn't fall over the same
• 33:50
  potholes that you fell on, and we can skip that part of the
• 33:54
  learning. If somebody is learning from
• 33:57
  this, I mean I would say that they would be having a a huge
• 34:01
  exposure on what is right and what is wrong.
• 34:04
  But at the end I would say if there are any pain points also
• 34:08
  then there are good points. Also you keep learning many new
• 34:10
  things and just checking on how things are working in the
• 34:14
  background. So you actually come with a set
• 34:17
  of tools which you need to go forward and add.
• 34:20
  Then make your own own ones. Like recently the canned paper
• 34:25
  came. It is challenging.
• 34:26
  You might have heard about the can.
• 34:28
  Kolmogorov, Arnol Networks. It is making waves.
• 34:31
  It is challenging the transformer architecture, so
• 34:34
  even I'm trying to, I'm trying to develop a framework for that
• 34:39
  so that it fits the GBD model. I mean, it is one of my personal
• 34:42
  projects, not a client project. So let's see how it how it goes.
• 34:46
  It is still our research paper. Yeah.
• 34:49
  And I do know that the last time that I talked to you mentioned
• 34:53
  how hard it is to ingest data also and the idea that you don't
• 34:58
  have the best tools out there for ingesting data like an air
• 35:03
  bite. I I imagine all the data
• 35:05
  engineering tools out there that are used to ingesting data, they
• 35:10
  have something for unstructured data now.
• 35:12
  But again, it goes to this scope and it feels like this is an
• 35:15
  underlying theme that you're talking about as we're trying to
• 35:19
  shoehorn in all of these traditional methods with the new
• 35:24
  AI methods. And you have to choose a persona
• 35:27
  that you're going to be creating a tool for.
• 35:30
  So if you are an air bite or a tool that has traditionally
• 35:33
  catered to data engineers on structured data, it's not like
• 35:37
  you can just be like, OK, now we also do unstructured data and it
• 35:40
  works just as well. So have you made any updates
• 35:45
  since the last time I talked to you?
• 35:47
  Yes, I have tried unstructured also for production and I think
• 35:52
  it is unstructured is the most advanced tool right now for data
• 35:57
  ingestion and because it partitions the data first, it
• 36:01
  goes with the partitioning of data.
• 36:03
  Also, coincidentally, yesterday I had a call with Byte Wax.
• 36:09
  Ohh yeah nice. The Bite Wax.
• 36:11
  Yesterday I had a call with Bite Wax because I need you to
• 36:14
  discuss with them a special connector of what they can
• 36:18
  offer. So they said that the connector
• 36:20
  falls under the category of premium connectors though
• 36:22
  costing $1000 per month. I said WHO connector is $1000
• 36:29
  so. And by wax, just for those who
• 36:32
  do not know, what do they do. Bite Wax is a streaming platform
• 36:37
  and they deal with live data streaming and they act as a
• 36:41
  connector for streaming data basically.
• 36:43
  So they have both open source and closed source format.
• 36:46
  But anything specialised you need will fall under the
• 36:51
  category of premium connectors that will cost you depending
• 36:54
  upon the use case. For me they give you $1000
• 36:57
  costing per month for a single connector for a database.
• 37:01
  Well, and so this was just connecting the streaming data.
• 37:04
  Just connecting. To a database.
• 37:06
  Yeah, through a database. Through a public database API.
• 37:09
  OK, interesting. Yeah.
• 37:12
  These, that's a whole another can of worms that we didn't even
• 37:15
  get into. The pricing and how you
• 37:18
  determine the pricing on all of these tools because a lot of
• 37:21
  them it is usage based. Which makes sense because you
• 37:26
  aren't going to have problems. Like if your scale is very
• 37:31
  small, you don't want to be paying the same as a gigantic
• 37:33
  enterprise that has lots of queries per second, but at the
• 37:38
  same time it can be prohibitively expensive and
• 37:44
  cumbersome for the developer. If at the end of the day you
• 37:48
  need to use four or five or six different tools in your workflow
• 37:54
  just to get that desired result, it feels very painful.
• 37:59
  And so I don't know if you've seen that in your own workflow
• 38:04
  that you just aren't open to bringing on another tool because
• 38:09
  you have so many tools in the toolkit.
• 38:12
  Yes, because I think at this point of time I'm open to
• 38:16
  exploring new tools, but it actually consumes my time and to
• 38:20
  go and do research in that tool and then test that tool for that
• 38:23
  production thing is something which is very tiring for me.
• 38:26
  So at this point of time what I have done or what we have done
• 38:31
  here in my company is Q deck which I'm working on.
• 38:35
  We have developed a machine learning research lab.
• 38:39
  This lab keeps on researching new tools and sidelining the
• 38:45
  ones which we which we don't have to use.
• 38:48
  I mean we don't we don't necessarily research on those
• 38:51
  tools which are are working with which we are doing the projects.
• 38:55
  Even there are like for example there is a resource who is
• 38:59
  sitting idle. So we do keep doing Pocs every
• 39:03
  day like like we pick LLM, evaluation framework, Prometheus
• 39:09
  and DBL. So just start comparing them and
• 39:12
  put it in a ranking method like what type of use case this will
• 39:17
  fit in, what type of this will fit in.
• 39:20
  We quickly create a code for that, push it onto the GitHub
• 39:23
  and keep it for our reference. So the next time we just need to
• 39:25
  do a copy and paste that is all. So we have started the method of
• 39:29
  machine learning research lab. So research to production is our
• 39:32
  goal. There's the.
• 39:35
  That's smart. That feels like a very good
• 39:38
  idea. It also feels like it could be a
• 39:40
  little bit dangerous if in six months from now one of these
• 39:45
  tools that you've are testing and you put into GitHub has
• 39:50
  completely pivoted and it has breaking changes all over the
• 39:53
  place, you have to be very careful with how you're
• 39:56
  implementing that. Yes, yes, that is that is one of
• 39:58
  the bottlenecks that I can say that is a risk, but we usually.
• 40:03
  Because this is something which Lang Chan usually does does
• 40:07
  every day. They keep changing the libraries
• 40:10
  and your pipeline is gone with the wind.
• 40:13
  That's again why it helps to have a little bit more control
• 40:17
  when you create those frameworks yourself.
• 40:19
  You know that at least you have the stability that it's.
• 40:23
  Yeah, it's not gonna change daily or weekly.
• 40:25
  Monthly So well, man say this is always a pleasure.
• 40:31
  I appreciate you coming on here and talking to me about this
• 40:33
  stuff. If anyone wants to relate or get
• 40:38
  in touch with you, I encourage them to either follow you on
• 40:42
  LinkedIn or connect with you on LinkedIn because I think you
• 40:44
  also are sharing brilliant stuff on there.
• 40:46
  I appreciate your transparency and you going out there and
• 40:50
  doing this research for us. I think it is very valuable for
• 40:54
  the community to understand what has been stress tested, what has
• 40:58
  been very useful for you, especially when it comes to this
• 41:03
  inference layer that you were talking about, how you tested
• 41:05
  these different pieces and it takes you so much time.
• 41:10
  That's why I really love that you come on here.
• 41:12
  You share what your findings are after.
• 41:15
  I imagine you've probably spent a combined 204060 hours
• 41:22
  researching the tools, trying to put them into practise, figuring
• 41:25
  out where they work, where they don't work best.
• 41:27
  All of that is very valuable information and so I thank you
• 41:32
  for that. Thank you.
• 41:33
  I mean it is always a pleasure to share these types of
• 41:36
  information from you also. So I mean same feeling from my
• 41:40
  end also. So I didn't realise.
• 41:47
  The other funny thing is that ML engineers are now all AI
• 41:52
  engineers. Yeah.
• 41:53
  Have you noticed that pivots? Everybody has an AI engineer.
• 41:57
  Exactly. Except again, going into that
• 42:00
  scope idea that you were talking about.
• 42:02
  It's so funny because an A I engineer like what does that
• 42:05
  mean? These days the scope for an AI
• 42:07
  engineer is so large and so many different people coming at it
• 42:11
  from so many different angles. You have an ML engineer who's
• 42:15
  just changed their name to now being a I engineer, you have
• 42:19
  maybe a full stack dev who now says OK I I know how to fine
• 42:22
  tune the model, now I'm an A I engineer.
• 42:25
  And so it's interesting to me that it's like, OK, there's AI
• 42:29
  engineers now that's the term of the, the month or the year I
• 42:34
  would say. And I would also say, I mean, I
• 42:37
  was mentioning previously also with you that there would be
• 42:41
  specialised SY engineers also in the future.
• 42:44
  I mean that is such a big thing. I'm still testing it in
• 42:47
  production. I have not because that is a
• 42:49
  very hard framework to understand and to then implement
• 42:52
  it. So that's the big thing.
• 42:53
  So I am into a POC stage right now.
• 42:55
  I've still not completed. I'm pretty confident next time
• 42:58
  when we speak, we'll be in a production stage.
• 43:01
  What's been so hard about it? To understand the modules,
• 43:05
  actually to understand the modules, to understand how they
• 43:09
  are aligning with your prompt strategies.
• 43:11
  So that is the hardest part of them even I've created a GPS on
• 43:17
  that the chart GBT the the offers the GPT version to help
• 43:23
  me understand those things. But I have had a good hold now
• 43:27
  with DSPI but py button. Right now I need to work on
• 43:31
  getting it aligned with my use cases first.
• 43:34
  That is something very hard, but it is very, very promising.
• 43:38
  I think not many people are having those skills.
• 43:42
  Yeah, I think a lot of people are very in love with the idea.
• 43:45
  But as you say, when you try and actually use it in production or
• 43:50
  you try and actually use it. Yeah.
• 43:53
  It's a it's like you're trying to take a Ferrari to the
• 43:58
  supermarket, yes? They say practise, that is the
• 44:01
  thing, yeah. Yeah, just fascinating, man.