Latest / Agentic Conversations (formally mlops.community) / GPU Considerations, Labeling Privacy, Rapid Fine Tuning, and the Role of Private Eval Pipelines to Benchmark New Models

Transcript

GPU Considerations, Labeling Privacy, Rapid Fine Tuning, and the Role of Private Eval Pipelines to Benchmark New Models · Agentic Conversations (formally mlops.community)

• 0:00
  I mean, these models are the quickest depreciating assets,
• 0:03
  right? They've got a half life and.
• 0:05
  The most expensive? Depreciating.
• 0:07
  Assets, they're hundreds of millions of dollars that, you
• 0:09
  know, after weeks or months have lost their value because a
• 0:13
  better model came out. We talked with Paul and Zulkov
• 0:21
  today about training models for agentic use and evals for
• 0:26
  agents. Paul is the VP of AI at Process
• 0:30
  and Zhukov is the Director of AI.
• 0:33
  Got 2 hard hitters and myself, the founder of the ML OPS
• 0:37
  community. My name is Dimitrios.
• 0:39
  Let's get into this conversation.
• 0:45
  Paul set the scene. What are you training?
• 0:47
  How you doing it? Right.
• 0:48
  So we're talking about what we do at Process in terms of
• 0:53
  bringing AI, in particular agents into production in that
• 0:57
  in the scope of our work, we, we care about making sure that the
• 1:02
  systems work for our use cases, that they are scalable and that
• 1:07
  they're reliable, right? And so to do that, we have to do
• 1:12
  a lot of different things, right?
• 1:14
  And I'm going to focus on the AI part mostly because of course
• 1:17
  it's the product side and so on and the user needs that we
• 1:19
  obviously also spend designing new experiences with AI, but I'm
• 1:22
  going to focus on the AI piece. Well, we need to make sure we
• 1:25
  understand how well models perform on our use cases.
• 1:29
  And to nobody's surprise, the benchmarks that we see on this
• 1:32
  space are not good, you know, indicators for that.
• 1:36
  We saw recently even that the yeah meta is basically
• 1:40
  optimizing for the benchmarks. That's the oldest ML problem,
• 1:44
  right? Like over fitting to the test
• 1:45
  set. So I think that is something
• 1:48
  under evaluating these models and they come new models come
• 1:52
  out every week now very quickly for whether they can be used for
• 1:55
  Polish car dealers call summaries or for predicting
• 1:59
  ingredients in a recipe in in a Brazilian food dish.
• 2:03
  Those are all the use cases that we care about how they perform.
• 2:05
  So evaluations is 1, 2 is how much, how well do they fit to
• 2:11
  our use case. So it's domain understanding,
• 2:14
  right? But also language capabilities.
• 2:16
  We benchmark voice models and, and, and in our eval sets, which
• 2:21
  we'll talk about in a second in, you know, Afrikaans, Polish,
• 2:24
  Brazilian, Hindi and so on to see how well do they perform for
• 2:28
  our voice calls because most of our traffic is not English.
• 2:33
  Then we also care about cost. So once we believe that we found
• 2:38
  a good fit, we need to make sure that these models are actually
• 2:42
  affordable to scale to hundreds of millions of calls because we
• 2:45
  have, you know, that many users interacting with their products
• 2:48
  on a daily basis. So how do you optimize for that
• 2:52
  cost? So we need to find either need
• 2:54
  to fine tune models smaller, distill them or find open source
• 2:58
  models that you maybe want to host somewhere and control the
• 3:00
  full, let's say inference stack on.
• 3:03
  So then you get into the world of how do you actually fine
• 3:06
  tune? How do you distill them?
• 3:07
  How do you pick the GPU's? How do you optimize for
• 3:09
  utilization? So we'll talk a little bit about
• 3:11
  that. And then the last piece, once
• 3:15
  you've actually gone through this sort of ability to take the
• 3:19
  performance of a top level Ferrari level model and getting
• 3:23
  that performance out of a fine-tuned smaller model.
• 3:25
  How do you set up the entire inference stack to work for you
• 3:30
  with the same SL as that you can get from an anthropic and open
• 3:33
  AI or any of the other commercial model providers?
• 3:35
  And that brings in a whole new set of challenges.
• 3:39
  So, and we've had to solve things across this entire
• 3:43
  spectrum of problems because we care about bringing Gen.
• 3:48
  AI or a genetic systems into production at scale.
• 3:52
  You run the gambit basically from soup to nuts, you got to
• 3:55
  figure out from top to bottom. That's it, Yeah.
• 3:59
  So I don't know where you want to start, but maybe we can talk
• 4:02
  a little bit about the evaluation piece, because we
• 4:05
  have Zukov here, right? And we've went since the
• 4:07
  beginning of, of, of, of working together.
• 4:09
  It was very clear that a lot of things change all the time.
• 4:13
  Models get better. They can now code, they can now
• 4:16
  translate, they can now do it, generate images.
• 4:18
  But one thing that will not change is our need to be able to
• 4:22
  evaluate how they perform on the tasks we care about.
• 4:26
  And so Zukuf has spent a lot of time on developing the in house
• 4:30
  capability for us to do that. Breakdown how you're evaluating
• 4:34
  right now. That AI is the short answer.
• 4:39
  Which is what it's probably good to give a little context on
• 4:42
  what. Problem is actually it's about
• 4:44
  how we share our evolution like we have many different evolved
• 4:47
  sets touching different tasks from our group companies, from
• 4:50
  our own AI team and then we build pipelines, evolution
• 4:54
  pipelines. Every time there is a new model
• 4:56
  out there, we are, if I mean it's set from credible
• 4:59
  resources, then we are putting that model in our pipeline and
• 5:03
  in I think matter of hours if not days, we are getting the
• 5:06
  results from that model on different tasks and we are
• 5:09
  sharing it with outside. And the evaluation sets are not
• 5:17
  public. They are not.
• 5:19
  That's the critical point. Actually.
• 5:20
  Maybe I can give a bit more background to the how we started
• 5:23
  like it's like I think two years ago we you know, process on
• 5:27
  Stack Overflow and at that time we really wanted to understand
• 5:31
  the value of Stack Overflow data, how it's helping because
• 5:34
  everybody was training on their data.
• 5:36
  So we wanted to see, OK, how much we are really helping these
• 5:39
  models and it turned out a lot, but that's a different topic.
• 5:43
  But what we were doing that we are training the models and then
• 5:45
  we need to see OK, how these models perform after training on
• 5:50
  stack data and without stack data.
• 5:52
  And we needed to really like a thorough evolve sets to see.
• 5:57
  And also, you know, in stacked data you can really see
• 6:00
  different programming languages, different type of questions,
• 6:03
  debugging, non debugging like explaining, conceptually guiding
• 6:08
  all different types of this. And we wanted to go deeper in
• 6:11
  the evaluations. We just didn't want to see you
• 6:15
  put some data on the model and how good it's just writing
• 6:17
  problem, how good it's answering Python debugging question
• 6:21
  compared to JavaScript writing. So then we built all this evolve
• 6:26
  set with different attributes. And we were using this basically
• 6:30
  to follow attract the models we train.
• 6:33
  But it turned out then occasionally you get some claims
• 6:37
  say we built this great model on very small, you know, it's just
• 6:41
  three billion parameters, but it does better than GPT for all.
• 6:45
  And hey, then we are curious, does it really do it?
• 6:49
  Then we get our evolve set. We benchmark not at all.
• 6:54
  Not even. Close.
• 6:54
  It's just marketing. Yeah.
• 6:56
  And now we say, hey, there's a lot of hype there and it came a
• 6:59
  bit kind of, you know, more often.
• 7:03
  So we say, let's then take these benchmarks since we already have
• 7:06
  them, let's put them somewhere so we can also follow up to all
• 7:09
  models because we will have more of those.
• 7:11
  And then we have this. Then we said, hey, everybody's
• 7:14
  asking us then in our group companies which model should I
• 7:17
  use for this use case? And we said, hey, why don't we
• 7:19
  make it, share it with the outside?
• 7:21
  And then that's how it started, actually.
• 7:23
  Yeah, and you have a few different types of benchmarks.
• 7:27
  There's the stack data set and benchmarks, but there's tool
• 7:31
  calling benchmarks, right? What other ones do you have?
• 7:34
  I mean this stack benchmarks, we have two benchmarks there.
• 7:37
  One is typical Stack Overflow. Stack Overflow questions,
• 7:40
  historical ones like 1000 questions, we got there.
• 7:43
  And that one, it kind of saturated.
• 7:47
  Actually. I can give you a good example.
• 7:48
  Like we were benchmarking the models on this historical data
• 7:52
  and a Stack Overflow had some API agreement with Google.
• 7:56
  And then after that agreement, we saw that Google models
• 7:59
  suddenly picked on that data set V6.
• 8:02
  They start training on it. So he said, hey, I mean, this
• 8:05
  benchmark is good, but we need something more fresh, something
• 8:09
  really like some data from the models hasn't seen, haven't seen
• 8:14
  in their training data. Then we created Stack on Scene
• 8:17
  and that's we are getting from Stack Overflow from the recent
• 8:20
  data, we hope models haven't seen those ones.
• 8:24
  And so this is like a bit more, you know, new that they're
• 8:28
  asking about the new libraries or new ways of solving the
• 8:30
  problems. So that became Stack on Scene,
• 8:32
  but both of them are Q&A and then we have tool calling for
• 8:37
  our agent and use cases, how good models are picking up the
• 8:41
  right tool and using it. We have summarization and just
• 8:46
  to see how good it's following the instruction.
• 8:49
  I think that's we should also talk about because not every
• 8:52
  summarization is same and we have open book K and A where you
• 8:56
  give like for example, that's very relevant for our use case.
• 8:59
  We have, for example, finance colleagues bringing large data
• 9:03
  sets, putting there and asking some questions.
• 9:06
  So we want to see if you give such a big context length, how
• 9:10
  each model performs in answering those questions.
• 9:13
  And I think we have a couple of more like entity extraction and
• 9:18
  what else. We have Q&A from token, actually
• 9:20
  from token question and answers, but that's also very New South
• 9:24
  models haven't seen that data. Yeah, you need to keep the
• 9:28
  freshness of these evals up to date.
• 9:32
  How often are you refreshing the data?
• 9:35
  Yeah, that's the problem actually.
• 9:37
  Like they are getting outdated. They're getting automated
• 9:40
  quickly. So the the E assets, I think
• 9:43
  after six months or something they all start, you get the
• 9:46
  smell. So hey, you need to update it.
• 9:49
  And so like I'm seeing we are trying to do it every three
• 9:52
  months for other data sets, we are a bit slower, but we are
• 9:56
  adding more like a kind of and getting out some stuff, but that
• 10:00
  requires also resources. So I cannot say that we are
• 10:03
  doing it every month, but I think every six months or
• 10:07
  something you should show some attention to this.
• 10:09
  It was just check what's going on there.
• 10:12
  I mean models are getting way better, but you also see some
• 10:15
  parts they are not that still good.
• 10:17
  Maybe you put more emphasize on that, but also depending on our
• 10:21
  team needs like, hey, we need this type of like for example in
• 10:24
  Tocom, we start using Go. So it's important for us which
• 10:29
  models are doing good in Go, right?
• 10:30
  So suddenly it's a requirement for us.
• 10:33
  Then we say, hey, let's make sure we have some enough
• 10:35
  coverage for Go and we see which models are doing good there.
• 10:39
  But that's also based on our needs.
• 10:40
  We are changing the scope of the eval sets too.
• 10:44
  And you can see what's really nice if you go to pro LLMS or
• 10:46
  pros S LLM dot AI, you can see that there's basically this
• 10:51
  leader boards on all these tasks that, you know, we've scored
• 10:55
  hundreds of models on because some something will happen,
• 10:57
  right? So DeepSeek comes out and of
• 10:59
  course, well, we're very cute. Like we see, Oh, deep seeks out,
• 11:03
  like how well does it perform? And then, like Zuke was saying,
• 11:05
  somebody will claim XYZ performance.
• 11:08
  We run it through our, our basically evaluation, the
• 11:11
  problem evaluation pipeline and within an hour or a couple
• 11:15
  hours, we know exactly where it shows up in the leaderboard and
• 11:18
  we know what it costs and we know whether it does better
• 11:21
  function calling or or summarization, whatever.
• 11:23
  And then it's like, hey, great. So now we should consider this
• 11:26
  model and move to let's say solving the inference piece
• 11:30
  which we'll talk about the GPU's and how whatever for this model
• 11:33
  because actually we've got 3 or 4 use cases which you know which
• 11:36
  with 5 billion tokens each, right.
• 11:38
  So then we but we have then confidence because of the quick
• 11:41
  evaluation we're able to do. It's worth investing more time
• 11:44
  and bringing this model into an actual Legentic or AI product
• 11:47
  that we have. Live, it's great signal amongst
• 11:50
  all the noise and all of the hype on social media that you
• 11:54
  see, oh, this model got released, It's beating all these
• 11:56
  benchmarks you get to see for yourself.
• 11:59
  And I can imagine you did the same thing with the long before
• 12:01
  model, and you probably were a little bit unimpressed.
• 12:06
  We can say that there, but also good point there.
• 12:10
  And now we have agentic systems, right?
• 12:12
  Like there are many tools, like summarization can be a tool
• 12:15
  there. I mean, you can, for example, if
• 12:16
  you look at latest Google framework, you'll see agents can
• 12:21
  use agents as a tool. So if you know that's for
• 12:24
  summarization, which consumes a lot of tokens, like it's very
• 12:27
  costly. You put a huge text in it and
• 12:30
  then you ask questions. If you know that hey, there is
• 12:34
  an open source model that can almost deliver the kind of same
• 12:38
  performance, It's a very expensive flagship models, then
• 12:43
  you can just replace that model there.
• 12:44
  You don't have to use the same model for all the tasks.
• 12:47
  Then that Pro LM gives us this kind of freedom to choose.
• 12:50
  You look around for that type of task.
• 12:53
  Here we can use this model. What is the best alternative?
• 12:55
  Or we can cut cost here. We can get better performance
• 12:58
  here than this kind of. We can also do kind of routing
• 13:01
  around 2 models. One point to add about Pro LM,
• 13:06
  as you may ask, how do we get these eval sets created right
• 13:11
  because we know that need is there like we said, you all the
• 13:16
  time new models comes out, you want to know is it worth
• 13:18
  investigating further considering it for moving into
• 13:20
  production and so you need these eval sets.
• 13:25
  But this is 1 area where, you know, we found that there's a
• 13:28
  lot of solutions on the market, tools that that, that, you know,
• 13:31
  claim to help. But actually they're not the
• 13:34
  biggest problem that we have for eval sets or for evaluations.
• 13:37
  It's the eval sets themselves. And so how do we get those?
• 13:40
  Well, this is we have a couple of advantages that we have.
• 13:44
  Like I said, we're a big group, lots of companies with real
• 13:46
  world data. So what we typically do is we'll
• 13:49
  gather, you know, a whole group of people that are hungry and we
• 13:54
  organize a labelling party. We provide them some pizza and
• 13:58
  some other snacks and we put them in a room and explain, hey,
• 14:02
  we're going to go through, you know, a bunch of summaries or
• 14:04
  we're going to go through a bunch of, you know, I don't
• 14:07
  know, listings on OLX and figure out what are the entities that
• 14:11
  we would want to recognize. And we sit down and we manually
• 14:16
  with, you know, fuelled by pizza curate these eval sets, which we
• 14:22
  use. And that's the hard part because
• 14:25
  that's laborious. You need people to kind of come
• 14:28
  in and do that. And it's, by the way, it's been
• 14:30
  a lot of fun. And I think we've discovered
• 14:31
  that this is a great way also to give people who are not so
• 14:34
  familiar with the work that's happening inside an AI team with
• 14:36
  a little bit of exposure of how, you know, we, we work.
• 14:40
  And so we start to organize these labelling parties, design
• 14:44
  labeling, and then you have eval sets that we now have ready to
• 14:50
  be evaluated against anytime a new model comes in.
• 14:53
  And maybe it's worth also talking a little bit about how
• 14:56
  we use LM as a judge to very quickly be able to evaluate a
• 15:00
  new model against that eval set. Yeah, tell me about that.
• 15:05
  Yeah, I mean we have been using a little message judge for a
• 15:09
  long time. I think the the first thing we
• 15:11
  discovered that I mean for example for Stack Overflow, even
• 15:14
  like Stack unseen Stack create data sets like benchmarks, there
• 15:20
  is a technical answer. So there are many ways of giving
• 15:23
  this answer. Even if you go Stack Overflow
• 15:25
  CD, you will see different scores, you know for different
• 15:28
  answers. So there is not one ground truth
• 15:32
  but so how are we going to emulate that?
• 15:34
  We cannot just compare it. So what helps is using whatever
• 15:38
  the ground truth is there like people you in stick overflow of
• 15:41
  case we always have a true answer and we also have a couple
• 15:45
  of recent answers that get lots of good scores.
• 15:48
  So if you bring those answers to the model and then you bring the
• 15:51
  question and then you bring the answer from other models and ask
• 15:56
  the model basically OK, is this answer correct or not?
• 15:58
  And here is the clue kind of hint how ground truth looks
• 16:03
  like. And then using that the model
• 16:05
  then kind of I think we got like 8 to 5 now around 90% accuracy
• 16:10
  in evolating if that answer model generates the answer is
• 16:13
  correct or not. So as long as you have a kind of
• 16:16
  credible ground truth there, I think models are pretty good in
• 16:21
  judging the accuracy of the their own generation or
• 16:24
  generation of adult models. The part there we judge, but you
• 16:29
  can also, you know, then you need another.
• 16:31
  It was it like you need to judge the judges find out which judge
• 16:35
  you should use as a kind of judge because every model
• 16:38
  performs differently in that. And then to the poll point, then
• 16:42
  we organize a labeling party. We generated many labels from
• 16:47
  the models by putting each of them as a judge position and ask
• 16:50
  it ask each model to emulate and then humans emulated their
• 16:55
  emulations and from there we had a OK labeled data set.
• 16:59
  And then we use it. We have it in pro LLM as LLM as
• 17:02
  a judge benchmark and that benchmark shows how LLMS are
• 17:07
  performing as a judge. And from there we pick up the
• 17:10
  our judge and then use it. I think now at the moment it's
• 17:14
  GPT 4.1 is our current judge but if something else becomes better
• 17:20
  we will just replace our judge. And how are you setting up the
• 17:23
  judging system? Because there's a million and
• 17:26
  one different ways to have an LLM as a judge.
• 17:31
  Yeah, what we do, I mean you always need to explain the task
• 17:35
  for I think that's no brainer. Like a what this task about is a
• 17:39
  technical training. So you need a technical answer.
• 17:43
  And there there are also many ways, like say, OK, you some
• 17:49
  people use from zero to 10 scoring, you know, OK, give us a
• 17:53
  score. A model gives 7 another model
• 17:56
  says 8. What are what are we going to do
• 17:59
  with that? Right, like so in then the point
• 18:01
  we came, what matters for the user, like SAU technical
• 18:06
  question answering person. What's important for me is I get
• 18:10
  an answer and I stop searching. So that's good enough.
• 18:14
  I just go and then continue my work.
• 18:16
  Hey, reset. OK.
• 18:18
  But second also has something similar like accept acceptance,
• 18:21
  right? It's acceptable or not.
• 18:23
  So in that we wanted to bring it a kind of binary that solves my
• 18:27
  problem or not for technical question.
• 18:29
  But we also realize, hey, models are easy to flip around.
• 18:34
  So we better give a bit room for them to make mistakes.
• 18:37
  So rather than 0 and one, what we did to 01 and 2/3.
• 18:43
  So if 0 means really like completely off, 1 is a bit more
• 18:48
  OK, it's got some clue, but this is still uses for me.
• 18:52
  Please, I can leave it as answer.
• 18:54
  I don't need to. And three is like that's
• 18:56
  perfect. So actually we put two and one
• 18:58
  there. It's a bit room to if it flips,
• 19:00
  it still flips into the same area.
• 19:02
  It doesn't go the other side and makes it completely wrong.
• 19:05
  And with that, I think we could get pretty good results.
• 19:10
  So it's indent what is important for you for each task.
• 19:14
  If summarization for example, then that definition changes
• 19:19
  what's important there some. In summarization, it hits the
• 19:23
  follower instruction. If you ask for T key two key
• 19:27
  takeaways, it should give you 2, not 3 and things like that.
• 19:31
  So for each task you have to go a bit deeper, see what matters
• 19:35
  and then reflect it to your rubric basically.
• 19:39
  One thing I think about, because I am a huge user of Gemini, and
• 19:45
  if you would have asked me six months ago what model I was
• 19:49
  using, it would not have been Gemini.
• 19:52
  Have you been surprised by certain models gaining traction
• 19:57
  or losing traction over time because you've been tracking
• 20:00
  this for so long? I mean, in the beginning, open
• 20:05
  AI was always there on top. So in every benchmark like from
• 20:10
  summarization to technical question answering, it was
• 20:13
  always open AI models on top and that was a very big margin.
• 20:17
  The second best model if it is not from open AI.
• 20:20
  But over time what we see now that margin gets narrower and
• 20:24
  narrower. And now open AI is also not
• 20:25
  always on the top. We see other models getting very
• 20:29
  close or getting ahead of open AI and sometimes you just see OK
• 20:34
  for different task also like the models are changing now it's a
• 20:37
  we got lots of more variety now with Google models, with
• 20:41
  ontropic models. And sometimes I can even Mistra
• 20:43
  like as a small model was really doing great job on the
• 20:46
  summarization, better than the open IF flagship model.
• 20:49
  And now we also have the China dimension.
• 20:52
  Maybe Paul, you can also share your talks about Chinese model,
• 20:56
  what we see them getting way better and better over the time.
• 20:59
  And that's also helping us as a open source community.
• 21:02
  Like we are training a lot of models and now with those
• 21:06
  benchmarks, we are just using them to see which model can be
• 21:10
  our baseline for different type of tasks by looking at the
• 21:13
  parameters, OK, that model 32B that can be our really like a
• 21:17
  great reasoning model. And then in the from the
• 21:20
  benchmark, we can decide on that one or you look at the small
• 21:23
  models in the benchmark, what is the smallest best model?
• 21:25
  Yeah. If you go back to this angle of
• 21:28
  building agents in production, let's take token which we talked
• 21:31
  about our internal sort of platform for productivity and
• 21:34
  others. Over the last two years of of
• 21:38
  building Token, we've had continuous challenger champion
• 21:44
  models to kind of see which ones are better at certain tasks that
• 21:47
  the agent, you know, whether it's summarization or image
• 21:50
  generation. And we've gone, you know, over
• 21:53
  100 models that we've put into production.
• 21:56
  And I can tell you with confidence that the models we
• 22:00
  have today in Tokan, right, that are answering the questions in
• 22:04
  six months all those models will be different.
• 22:08
  And so think about that as a product builder, right?
• 22:10
  That means that we need to understand which, you know, in
• 22:13
  six months, which models are going to be the ones that we
• 22:16
  need to be replacing with, right?
• 22:20
  Because there's going to be tons.
• 22:21
  And a Zuku mentioned, you know, initially was actually easy,
• 22:25
  like it was just open AI. That was the best one, right?
• 22:27
  Unless there was some cost consideration and so on.
• 22:30
  It was easy to know which one was best.
• 22:32
  Today, what's been very interesting is to see the rise
• 22:36
  of, you know, new model providers, in particular open
• 22:40
  source ones coming from China and other parts of the world
• 22:43
  that have basically become very, very good alternatives for many
• 22:48
  use cases. And, and we're starting to use
• 22:52
  those in production. Which again then brings us to
• 22:54
  the point which we'll talk about is how do you organize inference
• 22:57
  for models that are open source at that size that are available?
• 23:02
  You know, and the first open source options that were really
• 23:06
  the only competitive ones were essentially Llama and Mistral's
• 23:09
  models. Now all of a sudden you have,
• 23:12
  you know, Gwen models coming out, of course, you've got the
• 23:15
  DeepSeek ones. And as we switch to more
• 23:19
  reasoning workflows, right, in particular for the agents, it's
• 23:23
  really cool to be able to get much more visibility on how
• 23:27
  these models were trained, what they work well on.
• 23:30
  And we see that, you know, because we've got two years of
• 23:32
  history of pro LM, these leaderboards, you can actually
• 23:36
  play the, the, the, the video to see, OK, how many, you know,
• 23:40
  what does this leaderboard look like over time?
• 23:42
  In the beginning was very concentrated, just three or four
• 23:44
  players. And now you start to see that,
• 23:46
  you know, there's a dozen players that are in the top ten
• 23:50
  that are continuously, you know, catching up to each other.
• 23:53
  I mean, these models are the quickest depreciating assets,
• 23:57
  right? They've got a half life and.
• 23:58
  The most expensive? Depreciating assets, they're
• 24:00
  hundreds of millions of dollars that, you know after weeks or
• 24:03
  months have lost their value because a better model came out.
• 24:08
  And so again, coming out to this product builder perspective,
• 24:12
  we're agnostic to where the model comes from, you know,
• 24:15
  because we just want the best model, cheap, fast, whatever,
• 24:19
  whatever criteria used for best. But then once we know which
• 24:23
  one's best, we still have to solve.
• 24:25
  How do we actually host this, right?
• 24:28
  How do we do? Is it a commercial API as a
• 24:30
  service, or do we actually need to solve for figuring out where
• 24:34
  this model is going to run on the cloud, on our own GPUs and
• 24:38
  so on? Yeah, it's it's probably worth
• 24:39
  talking a bit about the infrastructure side of things
• 24:42
  and going out there and figuring out the GPUs.
• 24:46
  And I am fascinated by the GPU market right now because if you
• 24:51
  do decide to go the open source route, you need GPUs.
• 24:55
  There's no questions. All other tools are kind of
• 24:57
  optional. We talked with Bruce about what
• 25:00
  you chose to buy and what you chose to build yourself.
• 25:04
  There's no way you're building your own GPUs now.
• 25:07
  You might go and you might decide to buy GPUs, but that is
• 25:15
  a very hard way of solving this problem.
• 25:19
  So can you talk about this journey of getting out there and
• 25:23
  looking at GPU providers? There's so many right now, so
• 25:27
  maybe you can break it down on what key considerations you had
• 25:31
  to look for when you were going out there.
• 25:34
  Yeah, actually as you say, there are ways so many players there
• 25:38
  and so I think we kind of talked to almost every leading player
• 25:44
  out there from a media to core wave together AI tech mosaic
• 25:48
  about became data breaks later. So we had lots of exploratory
• 25:52
  face and if you talk to GPU providers, I mean, initially
• 25:57
  everybody is very open and then it's all, everybody is the
• 26:00
  cheapest, everybody is the fastest.
• 26:02
  About once you start talking like it's shifting, like I think
• 26:08
  most of them are kind of asking, you know, some dedication like
• 26:12
  commitment. So you need to get reserve
• 26:14
  capacity for certain number of GPUs for certain number of
• 26:18
  months. And then if you do the math,
• 26:21
  hey, that's a lot of money. And if you don't, if you're not
• 26:24
  going to use this GPS immediately, you are just
• 26:26
  wasting like burning your money. And then you're they're just
• 26:29
  idle there and they're waiting. So that's kind of dilemma we had
• 26:33
  initially. Hey, we know that we are going
• 26:36
  to explore a lot and there will be lots of times we are not
• 26:39
  training, we are just manipulating data, preparing for
• 26:42
  another experiment. Then we will have a, you know,
• 26:45
  extensive week of trainings, so many parallel trainings.
• 26:49
  So we need many GPS those weeks like a peak times, but then we
• 26:54
  again get the experiment results, analyse, learn and
• 26:58
  prepare for the next set of experiments until we are sure
• 27:01
  that OK, now we can do a big training because now we know
• 27:04
  which data to use and how everything is settled.
• 27:08
  But that part we need on demand. So when we start talking is on
• 27:13
  demand was not something every GP provider is very enthusiastic
• 27:17
  about. So that was a tricky point.
• 27:20
  We couldn't I think get it from, yeah, almost anybody like hey,
• 27:25
  if you want to do on demand and suddenly either price goes super
• 27:29
  high and it becomes wow, almost you pay as much as a reserve
• 27:35
  capacity or you get very small, you know, number of GPUs
• 27:40
  assigned to you. So you cannot really run any
• 27:43
  meaningful experiment. So it was a kind of tricky
• 27:46
  point. So with our current GPU
• 27:48
  provider, so that was the differentiator for us.
• 27:51
  We have a kind of freedom we can use on demand GPUs and scale
• 27:57
  quickly and then we need a very bigger capacity.
• 28:00
  We can reserve a week ahead and then get more GPUs for bigger
• 28:04
  trainings. Maybe to add to that, because I
• 28:07
  think you've described a very specific type of need that we
• 28:10
  have for GPUs, which is when we train models for which for a
• 28:13
  period we need, you know, a few dozen, a few 100, maybe a few
• 28:16
  thousand GPU's to run a training run.
• 28:19
  It's very spiky, spiky. And this is sort of the on
• 28:22
  demand piece, right? Which is like then a commitment
• 28:25
  of one year or whatever longer is, is really hard to make
• 28:29
  because we just need it for that batch, that training run, right?
• 28:32
  And then, you know, we look at the results and we'll run it
• 28:34
  again. Then there's other users
• 28:37
  inference related, but we also need GPUs, right?
• 28:40
  So we looked at camp, do we need to, let's say again, commit
• 28:45
  dedicated GPUs or do we do on, you know, on demand spot
• 28:49
  instances or whatever? And on the cloud providers is an
• 28:52
  option, but they typically are, are they have these quotas,
• 28:56
  they're very slow to respond. They're not great for the, for
• 28:59
  the set up because they tell you, oh, you can have, you know,
• 29:01
  8, 881 hundreds or whatever, right?
• 29:04
  And but now it's sometimes we need 12 or sometimes we need
• 29:07
  whatever. So then for those infant
• 29:11
  workloads, we had the issue that if we had committed, let's say a
• 29:16
  certain number of GPU's resources to this, this, this
• 29:20
  service, right? Which was doing classification
• 29:22
  of whatever or translation, whatever.
• 29:24
  Like in real products, you also have a lot of cycles, right?
• 29:27
  Like daily or even weekly where people come in and they post
• 29:31
  pictures or they order food. And, and so you need to kind of
• 29:35
  adjust because if you don't adjust your underlying GPU's for
• 29:38
  inference, you have a really low utilization and then you cast
• 29:42
  out the roof again. And that's we're talking to
• 29:44
  providers like together AI has been interesting because that's
• 29:47
  more inference related. Like for the training, we don't
• 29:50
  really, we haven't worked with them yet, but they've been great
• 29:53
  because they basically offer, you know, on demand or token as
• 29:58
  a service for open source models with all the privacy guarantees
• 30:01
  that we needed. And so depending on the
• 30:03
  workloads, you actually need to solve for very different, you
• 30:07
  know, the number of resources and and so on.
• 30:09
  The elasticity is key for you if if I'm hearing that correctly.
• 30:13
  But also you bring up a great point that potentially if you
• 30:17
  can, you're going to try and just go with tokens versus I'm
• 30:24
  going to figure out how to put the model onto this
• 30:28
  infrastructure and then figure out everything around that to
• 30:31
  make that inference very fast or or whatever.
• 30:33
  So how do you break down the, oh, we'll just hit the open
• 30:37
  source provider in a token API way versus we're going to go
• 30:42
  into the GP US and do it ourselves?
• 30:44
  But we have our own models. So I think sometimes it's not
• 30:47
  like if you, let's say, let's take a DeepSeek example, right?
• 30:50
  We want to, you know, use DeepSeek for a workload.
• 30:54
  Then we could do a few things right.
• 30:57
  So we either go to DeepSeek themselves, which typically for
• 31:01
  privacy reasons doesn't work. So then we can say what we
• 31:06
  download the models from Hugging Face and put them on some bare
• 31:10
  metal right in the cloud or other.
• 31:13
  Then you get into the issue. Well, actually, we still have
• 31:16
  the sort of day-to-day cyclical usage patterns.
• 31:19
  So we probably can't get the economics to be very, very
• 31:23
  favorable because utilization fluctuates too much or is too
• 31:27
  low. If you have guaranteed bare
• 31:28
  metal on your own. So then you know, then let's say
• 31:31
  a service where it's token as a service for DeepSeek, which is
• 31:34
  just the vanilla model, right? The, the, the, the model
• 31:37
  everybody has access to. And Together AI is great because
• 31:41
  then you just send the request and you get it back and they
• 31:43
  organize the workloads. But we're also training our own
• 31:46
  models. And so that's where Mizukov
• 31:49
  trained a model. Like how do you expose that for
• 31:52
  people to use? Then you can't use it Together
• 31:55
  AI or a commercial provider. Obviously we don't have that
• 31:57
  model. So then you still need to go
• 31:59
  back and say OK, how do I provision make this work?
• 32:02
  Use for our own trained model. And the other piece is on the,
• 32:08
  when you're out there and you're looking for hardware, I'm sure
• 32:12
  there's a lot of other considerations that you've
• 32:14
  thought about. Maybe in this specific training
• 32:17
  instance, there's it comes with its own set of considerations.
• 32:21
  Yeah, in the inference instance, it comes with a whole different
• 32:24
  set of considerations. How do you look at what you
• 32:29
  need, what boxes you need to check as you're getting for the
• 32:32
  training? I mean, initially we actually
• 32:34
  didn't know. We learned it by experience.
• 32:36
  So when we start training the models for Stack Overflow and at
• 32:39
  that time Mosaic I think was the only provider helping there.
• 32:43
  And then we start using their platform and they were great
• 32:46
  guys helping with the support, but we learned that we cannot
• 32:50
  use every tool or every model out there.
• 32:52
  So we have to just go with whatever they support and that
• 32:55
  kind of start being a limiting factor for us in later.
• 32:59
  So that's OK, important checkpoint for us.
• 33:02
  Can we use the frameworks and the model every model out there
• 33:07
  immediately with this GPUs or do we need to wait a kind of
• 33:10
  support to come to the platform? I guess to make it real like so,
• 33:13
  So what you're, what we're facing is, you know, Llama 2
• 33:17
  came out and we wanted to train that model on the Mosaic
• 33:21
  servers, right? And that wasn't possible because
• 33:24
  their, their, their, their middleware, essentially their,
• 33:27
  their, their framework, I think Converge it was called, it was
• 33:31
  not compatible with that model yet or that model wasn't
• 33:34
  compatible with it. So we had to wait weeks to get
• 33:37
  that model available. So then you would want then we
• 33:40
  would want to have access to the bear underlying GPU's and run
• 33:43
  our own everything on there to be able to use their services.
• 33:48
  Well, it feels like it's almost the, the bigger question is when
• 33:52
  you're looking at the GPU provider, do they what kind of
• 33:56
  support do they have? What kind of software comes with
• 33:59
  the GPU's? Exactly.
• 34:00
  And most of them are nowadays supporting slur, for example.
• 34:03
  That's a kind of default out of box.
• 34:05
  You get, you always need Hugging Face kind of integration.
• 34:08
  We can bring every model from Hugging Face to the GPU's.
• 34:12
  And then you also see later, hey, how are we going to bring
• 34:15
  the data here? So initially you think, hey,
• 34:18
  that's a no brainer. It should be Dash, there should
• 34:21
  be many ways. But you see, hey, not everybody
• 34:24
  is really supporting every way. And most of our data is in S3.
• 34:28
  How we are going to do the secure connection to bring that
• 34:31
  this data? And once this data is here, how
• 34:34
  are we going to handle that data?
• 34:36
  Like what kind of privacy guarantees we get?
• 34:39
  Can we remove the data anytime we want?
• 34:41
  Do we keep the replica? Like those kind of simple
• 34:44
  questions don't have the same answers for every provider.
• 34:47
  So then they all turn to kind of check boxes for us after start
• 34:51
  talking to the GPU providers. And then there are natural
• 34:54
  things like OK network speed and like how is it scaling OK if
• 35:00
  there is an auto scaling or not. So they're also kind of
• 35:03
  technical check boxes. We also ask the GPU providers.
• 35:08
  One thing that was interesting is that availability was also
• 35:13
  hard, right? I mean this shortage we felt
• 35:16
  that sort of very real, right, Like it was actually hard to get
• 35:20
  access and even some of the providers said, well, we only do
• 35:24
  you know millions of dollar dedicated capacity of course.
• 35:28
  And so and then of course, the quotas and the cloud providers
• 35:31
  and so that that shortage was real.
• 35:37
  Yeah, especially if you have certain type of GPU in mind,
• 35:40
  like, hey, we give you a a hundreds as much as you want,
• 35:45
  but for age a hundreds we have to wait.
• 35:47
  And for age 2 hundreds, well, we need commitment, things like
• 35:52
  that. And then we, OK, we always want
• 35:55
  the best, fastest GPU. But then you sometimes there's a
• 35:59
  big cost difference, like, you know, with Nebulous, for
• 36:03
  example. Now we got the same price for
• 36:06
  age 2 hundreds, so we migrated there.
• 36:08
  But that's also changing from GPU provider to GPU provider.
• 36:13
  And I can imagine it's not as easy to swap out the GPU
• 36:17
  provider as it is to just swap out the API if it's tokens.
• 36:21
  So going back to the point of if we can, let's think about just
• 36:27
  using the API. If we need to go further because
• 36:31
  of some certain criteria, then let's figure out what we have to
• 36:35
  do to make that the most efficient as possible.
• 36:41
  Like swapping out GPU providers actually is easier than you
• 36:44
  think, other than maybe the data and so on.
• 36:46
  But like at the end of the day, we've got pipelines of code that
• 36:50
  we run, especially if it's not tied to their frameworks.
• 36:53
  It's just, you know, getting the environment set up and, and, and
• 36:56
  so on. It's, it's more if you switch
• 36:58
  from an H200 to A-100, obviously there's that you need to change
• 37:02
  a few things, but it's not super sticky, right?
• 37:05
  It's it's pretty commoditized. No.
• 37:06
  I mean you need to change your data routes like where data is
• 37:10
  coming and also like on paper everybody's supporting every
• 37:15
  framework, but in practice it's never the case.
• 37:19
  So we start using a certain frame for framework for training
• 37:22
  our models and then the RGP providers needed to do some
• 37:27
  adjustments. If you go to another one, I'm
• 37:29
  not sure if out of box support will be enough for us to
• 37:33
  continue as this. Probably they will also need to
• 37:35
  do some adjustments and but yeah, it's not a big deal.
• 37:40
  I know you also mentioned one big thing for you is the support
• 37:45
  that the providers give you. And I can imagine that if the
• 37:50
  scenario is you're inundated with a lot of requests and you
• 37:53
  have to prioritize which ones you want to go with.
• 37:58
  You're not going to be offering support to the person who's
• 38:00
  saying, hey, can we get like spot instances?
• 38:03
  And we'll see. Maybe we use a little money this
• 38:05
  month, maybe not as much next month versus someone who's going
• 38:08
  to pony up and pay the actual big dollars.
• 38:12
  So did you find providers that would offer you support despite
• 38:18
  not doing these large commitments from the get go?
• 38:21
  Yeah, we are lucky. And then we see the importance
• 38:24
  of support later. You know, initially you think,
• 38:27
  OK, it's just a bare metal. We take it, We put our
• 38:30
  frameworks and libraries and we do the training about the many
• 38:35
  things can go wrong. Like it's a huge network of the
• 38:37
  GPU's clusters, connected notes. Like there are so many reasons
• 38:42
  that things can go wrong and then you always need to support.
• 38:45
  And we were lucky with the neighbors on that front.
• 38:47
  We had the select channel and the guys were always available.
• 38:51
  Anytime you ask a question, you got an answer back.
• 38:54
  That's how we could move fast. Otherwise, I cannot really
• 38:57
  imagine it. After seeing all this in
• 38:59
  initially, I didn't think really we needed that much support.
• 39:01
  But after seeing on practice how much we needed support, I think
• 39:06
  it's a big prior. I mean, I will definitely ask,
• 39:09
  this is one of the first questions.
• 39:10
  That moves up on the hierarchy. Yeah, definitely.
• 39:13
  I think it, it, it, it's easy to forget how hard it is to run
• 39:17
  clusters of thousands of GP us and not all companies are
• 39:21
  equally experienced in that, right?
• 39:23
  And I think just having the GP us doesn't mean that we can run
• 39:28
  our training runs uninterrupted, right?
• 39:30
  That was AI mean sometimes we'd wake up like the next day and
• 39:33
  the runs were failed, but you got any alert and so then you
• 39:36
  know you're you're losing time, right.
• 39:38
  So that is definitely something that that's important to keep in
• 39:41
  mind. Yeah, it also helps that they're
• 39:44
  right down the street from you, so you can go and pop into their
• 39:46
  office, right? Exactly, it's just the next door
• 39:49
  so we can go knock their door that's helping.
• 39:54
  I I wanted to get into the labeling that you have on the.
• 40:03
  What is it exactly that you're doing?
• 40:05
  It's you. You've evolved from your
• 40:07
  labeling with Tokan answers. This is changing gears.
• 40:12
  All right, so let's change. Let's change gears for a moment
• 40:15
  because I want to talk a little bit about the output and the
• 40:20
  labeling that you've done for Tokan and the answers and how
• 40:26
  you are able to it. It's not evals you mentioned,
• 40:32
  it's what is it? It's so it's tagging and it's
• 40:36
  different than the parties, like the labeling parties that you
• 40:39
  were talking about too. So let's change gears for a
• 40:42
  moment, because I think you've done some innovative stuff
• 40:45
  around figuring out how people are using token without needing
• 40:51
  to read tons of messages or a lot of output from Tokan, right?
• 40:56
  And also that helps because then you have a bit of privacy.
• 41:00
  Yeah, I mean, it's more than a big privacy.
• 41:02
  It's really a lot of privacy there.
• 41:04
  So we cannot read any message and not even in our team without
• 41:08
  the consent of the person. So Tokan messages are super
• 41:12
  private. Nobody can touch and read them,
• 41:14
  even the database admins. And that's when you have that
• 41:17
  requirement. And also we want to understand
• 41:20
  how people are using token, right?
• 41:21
  Where token is helping them. What?
• 41:23
  And also our companies, they come to us and they say, I want
• 41:27
  to know how my employees are using it.
• 41:29
  They are really getting a kind of value from it.
• 41:32
  Or can you tell me any, give me any insights on how we can
• 41:36
  support them, make it more useful and things like that?
• 41:39
  That's the first point, like understanding how people use it,
• 41:42
  but also it's the builders of Tokan.
• 41:45
  It's important for us to see if Tokan is fulfilling the, you
• 41:49
  know, the expectations, if it is delivering and where it's great.
• 41:54
  I mean, where it's failing, where it's performing great.
• 41:57
  So understanding Tokan performance.
• 42:01
  The third thing is also like understanding the impact where
• 42:04
  it's providing the most value and where we can kind of improve
• 42:08
  it and those kind of dimensions we want to focus when we get to
• 42:12
  tagging in place. So what we do there, we build a
• 42:16
  system that takes a token conversations, not just a
• 42:20
  question, the entire conversation with the model,
• 42:22
  like a user question, model answer and follow up questions
• 42:26
  from the user and follow up answers from the model.
• 42:30
  And then we give this conversation to our LLM and we
• 42:33
  ask like I think 25 tags, for example, what is the domain of
• 42:37
  this conversation? Finance, for example, or HR or
• 42:43
  data science IT? And what is the task type?
• 42:47
  It can be like coding, debugging and what is the kind of use
• 42:52
  scale because some but I mean, it can be coding, but it can be
• 42:55
  somebody from finance writing, you know, some analysis work for
• 43:01
  regulating A competitor or for startup like and OK, this is the
• 43:05
  use case. This is a task type, this is
• 43:08
  this in this domain. And then we kind of make it very
• 43:11
  granular to see how exactly token is being used.
• 43:16
  And then we also ask the model, OK, how much time people are
• 43:19
  gaining with this task? And then in model is also like,
• 43:22
  OK, the first thing is how people would do this manually
• 43:26
  without token says OK manually he should do this, write this
• 43:30
  code, go check Google. Probably he would spend 60
• 43:34
  minutes with token. Now with conversation announcer
• 43:37
  it took 10 minutes. So he saved 50 minutes.
• 43:41
  That means OK, 50 minutes saving.
• 43:43
  I think in average each each token session, you say like a 17
• 43:48
  minutes or something and then in a day, if you use it three
• 43:51
  times, you save one hour by using the token.
• 43:54
  But that's also kind of, you know, the metric we surface and
• 43:58
  then see OK, how much we gain by using token.
• 44:01
  That's one of the tags. But I don't know if you want to
• 44:04
  deep dive into technical. Yeah, let's talk about the tags,
• 44:07
  because I know the tags were a bit of a headache since it is so
• 44:10
  open-ended. Yeah, that's the point.
• 44:12
  So you have two choices there. You either kind of restrict the
• 44:17
  models to certain number of texts and then you say OK, you
• 44:20
  have to choose one of those texts.
• 44:22
  But token is being used by different companies, different
• 44:26
  language for different type of things.
• 44:28
  So there is no way to capture everything front end.
• 44:33
  You have to like it will always try and go beyond those texts.
• 44:37
  And then basically what you do, you just miss cluster your text,
• 44:41
  then you introduce, you know inaccuracy into your system if
• 44:45
  you do that. All of a sudden you have 10,000
• 44:48
  tags. Yeah.
• 44:48
  I mean, we didn't want to go that way.
• 44:50
  We say, OK, we are going to let it keep it free and after, but
• 44:54
  what we see then and there's no end to end, it's going all wild.
• 44:58
  And the models also, you know, it's not always consistent even
• 45:02
  if you keep the temperature at 0.
• 45:04
  Sometimes it calls it Python. I mean it's coding, sometimes
• 45:08
  script writing, sometimes programming.
• 45:11
  Like you can get many different texts or the same task actually.
• 45:15
  So. We needed to do lots of things
• 45:18
  there to kind of tame the model in that we didn't touch the
• 45:23
  model, we kept it free, but we did lots of post processing and
• 45:27
  use the model again in this post processing.
• 45:29
  So kind of we built new way of LLM powered clustering here.
• 45:35
  If you go, I mean use direct embeddings.
• 45:37
  Embeddings also are not perfect. You will get also inaccurate
• 45:40
  clusters. Hey, then the idea we have such
• 45:44
  powerful models, why don't you use them for clustering?
• 45:47
  Technically you think I put all the text, pull all the text into
• 45:51
  model prompt and it will give me clusters out of box nicely
• 45:55
  organized text and I can use it. It turned out after 100, text
• 45:59
  models are failing, so some of them are not coming out at all,
• 46:02
  like they're forgetting the text and the clusters are like in the
• 46:06
  beginning it creates a cluster, but indent, it creates very
• 46:09
  similar cluster and then suddenly it's not consistent at
• 46:13
  all. Again, hey, we cannot use that.
• 46:15
  It turned out we cannot really put more than 100.
• 46:17
  Maybe new models had changed, but like a year ago it was like
• 46:22
  around 100. He said, OK, we use that model,
• 46:26
  but we do the clustering ourselves.
• 46:28
  We create the hierarchy of text. We kind of created good enough
• 46:33
  clusters, then use the model to refine those clusters and that
• 46:38
  worked the best. So you get a kind of good enough
• 46:40
  cluster. You ask the model if it is good
• 46:42
  or not. The model can say OK, this stack
• 46:44
  is good, this kind of not that should be the name for this
• 46:48
  cluster. So we go over all the clusters
• 46:51
  and after that we did another clustering because sometimes you
• 46:54
  just need 5 clusters to communicate the, you know,
• 46:57
  audience who needs to know very high level.
• 47:00
  But we also wanted to go low level and see what people are
• 47:03
  doing. So we need the kind of three
• 47:04
  level of clusters there, deep dive into what people are doing.
• 47:09
  And it turned out OK. I know the cluster like you are
• 47:12
  writing, you know, Python And you are using, I mean for
• 47:17
  finance like somehow Python, but what exactly are you?
• 47:20
  I want to know a bit more, but it's also private.
• 47:23
  I cannot look into it. So we also kind of created
• 47:27
  another tech kind of giving you a bit more detail, like the goal
• 47:30
  of without anything private, anything sensitive, but just a
• 47:34
  few words more. So if you want to read get a bit
• 47:37
  more idea, you can go read those texts in a kind of nice UI.
• 47:41
  So in that we came up with a solution, you know, from all the
• 47:44
  way down, you can go all the way high in different clusters,
• 47:47
  different on your needs. So we can serve it to our users
• 47:50
  that can tell you how people are using token, how Tocan is
• 47:54
  performing and what is the impact.
• 47:56
  One of the reasons that we wanted to do this is because,
• 47:59
  you know, we have large volumes of data and we need to get some
• 48:02
  understanding in a way that scales and in Tocan's case is
• 48:06
  compliant with the privacy requirements of the stock to
• 48:09
  certification and so on. So we built Flow which is our
• 48:12
  data analyst agent. And of course, this is, it's an
• 48:15
  interesting case because Tooken handles millions of questions
• 48:19
  and the interest we had was, you know, what types of topics, what
• 48:22
  kinds of, you know, how helpful is it?
• 48:25
  How much time is it saving? And we, so we built the flow
• 48:27
  data analyst. It took a lot of time, right?
• 48:29
  But now flow analysis these millions of questions
• 48:32
  continuously and provides this data.
• 48:34
  So it's very useful. But it turns out this pattern of
• 48:37
  understanding large volumes of data is useful for a lot of
• 48:42
  different, you know, settings in, in the group, right, People
• 48:45
  that are dealing with customer interactions or support tickets
• 48:49
  or, you know, other kinds of, you know, like sales calls.
• 48:54
  So, you know, we are now moving more heavily into this AI
• 48:58
  workforce as, as a, as a topic and theme and creating, you
• 49:03
  know, AI agents that become parts of the team to, to help
• 49:07
  things make easier. We now have lots of data
• 49:09
  analysts connected to all sorts of internal data sets that allow
• 49:14
  people to ask this data analysts questions.
• 49:17
  So we've got flow or data analyst, but there are many
• 49:20
  token data analysts out there in the group in ifood and OLX and
• 49:23
  so on that are doing very similar things, taking raw
• 49:28
  customer data, you know, adding a layer of intelligence,
• 49:32
  clustering, tagging and so on that then can be used by the
• 49:37
  teams to make informed decisions about what to do.
• 49:41
  So you did mention before that models are the fastest
• 49:44
  depreciating asset. And at the same time, I think I
• 49:49
  heard you just say that you're training models and you have to
• 49:53
  go out there and procure GPU's. How do you weigh those two
• 49:58
  seemingly opposite ideas in your head?
• 50:02
  Yeah, I mean, I think you must train the models.
• 50:04
  If you are in that game and you are serious about it and then
• 50:07
  you have a really good domain data, you have to leverage that.
• 50:11
  And in our case, we have really companies having lots of
• 50:14
  valuable data. So we wanted to create
• 50:17
  specialized domain models for these companies, their own
• 50:20
  business in their geographies and language.
• 50:23
  Like for example, we are creating a large commerce model
• 50:25
  now that can help the customers of these companies way better
• 50:29
  than the generic models that models those models will know
• 50:33
  their domain better and they can later build on top of those
• 50:37
  models. Their agents, conversational
• 50:39
  assistants even use it for their existing AI systems.
• 50:43
  And I think we should be mindful about the depreciation part.
• 50:47
  There is like what depreciate is the base model.
• 50:49
  So we are not interested in creating a language model from
• 50:53
  the scratch. And we are just we have our
• 50:56
  pipelines, we have our data and if the base model depreciates
• 51:01
  for example, it was Quan 2.5 a few months ago, now last week we
• 51:06
  got Quan 3 QVQ. So we can just replace the base
• 51:10
  model And now we continue and just enjoy the performance gains
• 51:16
  coming from the base model. On top of on top of that we
• 51:19
  continue fine tuning and even further pre training alignment.
• 51:24
  But that part doesn't really add that much on the cost I think.
• 51:29
  If if you think about effort spending on training, it is
• 51:33
  about fine tuning and and pre training for our specific tasks
• 51:38
  on top of an existing base model like Zukov just said that we
• 51:43
  will see and expect to see many more of our proprietary models
• 51:47
  going into production today. That is still a, you know, small
• 51:51
  percentage of the use cases we we try and test open source
• 51:57
  models on that we've fine-tuned because we want to make sure we
• 52:00
  know how to do that. I think in the next 12 to 18
• 52:04
  months, it's very likely that we will displace a lot of the
• 52:08
  commercial model traffic to our own models for a very specific
• 52:11
  tasks because it's higher performance and much more
• 52:15
  scalable in terms of costs. Yeah, the cost is a big one
• 52:19
  there. It's especially once you get it
• 52:22
  out to that many users and you're starting to see, well,
• 52:27
  it's not only our internal company that is using it or our
• 52:30
  portfolio companies that is using it.
• 52:32
  But now when we start to push this out to their users, then
• 52:37
  that cost is going to go through the roof.
• 52:40
  If you're using just off the shelf APII can imagine.
• 52:44
  And also the customization factor, like you have lots of
• 52:47
  flexibility. Then you have a model you can
• 52:49
  train on multiple tasks there and you can use it not just for
• 52:53
  one thing. Now you can also scale it to
• 52:54
  different tasks. Like why don't we change?
• 52:57
  I mean, we have lots of statistical models, right?
• 52:59
  In every company AI infra host, I don't know how many models and
• 53:05
  other models, then hey, we have a better intelligence here.
• 53:09
  Maybe we can also replace some of those models with that one.
• 53:12
  Now you start thinking, you know, in that way of how to
• 53:14
  leverage that resource that I have, I trained.
• 53:16
  Can we also introduce this task, maybe change the data structure
• 53:21
  differently, but also start utilizing a better intelligence,
• 53:23
  get a bit more performance gain from it.
• 53:26
  And now once you have your model and the training second, start
• 53:29
  also thinking around that. And then you also suddenly find
• 53:32
  yourself coming up with some innovative ideas like, hey, I
• 53:35
  can also use it here. I can also let me try it there.
• 53:37
  Let's do this, maybe then put this agent on this model rather
• 53:41
  than the open AI models, and suddenly you have kind of more.
• 53:44
  Well and interesting, it comes right back around to Pro LLM and
• 53:48
  you're not going to know which ones are performing better in
• 53:51
  what ways unless you have something like Pro LLM.
• 53:54
  Exactly. Actually we also published that
• 53:57
  like a kind of eval driven development.
• 53:59
  We maybe AI use case, we always try to start from the end.
• 54:03
  Like we create eval set for pro LLM, we put it there.
• 54:07
  What is our baseline, OK, what we expect from the model.
• 54:10
  We also do initial labeling for some evolve set if you don't
• 54:13
  have any labels. And then after that we start
• 54:16
  developing training and then experimenting.
• 54:19
  But we always have the benchmark there in program.
• 54:21
  We recently created private spaces.
• 54:23
  So for that that those are not public.
• 54:26
  It's only for our team or processed companies.
• 54:29
  So when you start the development of a project, every
• 54:33
  time you do benchmark on your private space, you can see which
• 54:36
  model is doing how, which technique work best, which
• 54:39
  prompt work best. You can do the kind of follow up
• 54:41
  on that space and then we iterate over through the
• 54:44
  benchmark. Are you allowing folks from the
• 54:47
  portfolio companies to add their own?
• 54:49
  Eval. That's the idea.
• 54:50
  Yeah, yeah, that's cool. Everybody can use it.
• 54:53
  They can create their iPhone space, they can create their own
• 54:56
  X space and put the we give the pipe.
• 54:58
  So the pipelines, the only thing they need to do come up with the
• 55:01
  eval set and then they put the wall set and everything else is
• 55:04
  automatically ran through all the models they want to test.
• 55:10
  Even we can also test fine-tuned models so that they will be able
• 55:14
  to see it. That's all we've got for today,
• 55:17
  but the good news is there are 10 other episodes in this series
• 55:21
  that I'm doing with process deep diving into how they are
• 55:26
  approaching building AI products.
• 55:30
  You can check it out in the show notes.
• 55:32
  I leave a link and as always see you on the next one.