Latest / Agentic Conversations (formally mlops.community) / Agents of Innovation: AI-Powered Product Ideation with Synthetic Consumer Testing // Luca Fiaschi // #306

Transcript

Agents of Innovation: AI-Powered Product Ideation with Synthetic Consumer Testing // Luca Fiaschi // #306 · Agentic Conversations (formally mlops.community)

• 0:00
  Luca Fiasco. I'm a partner at by MC Labs.
• 0:04
  I take my coffee. I just drink espresso.
• 0:07
  I'm an Italian, hardcore espresso is my way to go.
• 0:12
  Letters make words, words make sentences, and sentences make
• 0:16
  paragraphs. Welcome back to another Enelops
• 0:19
  community podcast. We're getting into it today
• 0:21
  talking about how Gen. AI can help the world of
• 0:26
  traditional ML with Luca. We also go deep onto leading
• 0:31
  data teams at the end. Hope you enjoy and a huge shout
• 0:35
  out to folks that have been joining the ML OPS community
• 0:38
  because I've got the best music recommendations now from you all
• 0:45
  since I set up one of the triggers to ask for music
• 0:50
  recommendations. It is so cool to see some of
• 0:54
  these suggestions that I'm getting back and I'm going to
• 0:58
  play one for you today. This recommendation that I'm
• 1:01
  going to play for you today is by the Band Boards of Canada.
• 2:20
  So this is like phone a friend basically.
• 2:23
  I get to phone a friend and then you've been doing a lot of cool
• 2:28
  stuff in your career. We should probably just go over
• 2:32
  a bit of your journey at HelloFresh, HelloFresh,
• 2:37
  HelloFresh and that stick Stitch Fix.
• 2:41
  Both of those are not hard words to say, unless you are me today.
• 2:45
  And so which one came first? HelloFresh came first and even
• 2:53
  before that I used to work with this large VC in Europe called
• 2:59
  Rocket the Internet. It really moved me out of my
• 3:02
  academic track into the start up world and Rocket was a lot of
• 3:08
  fun. I built some of the largest
• 3:10
  e-commerce in the world outside of the US and literally was
• 3:16
  empowering young people like me to solve big, big problem,
• 3:21
  putting a lot of money behind them and behind bold ideas.
• 3:25
  And. And there I got to work with
• 3:28
  companies like Zalando, Delivery Hero, a lot of these, you know,
• 3:32
  like pretty. Yeah.
• 3:36
  And I got to work with them when they were relatively small, like
• 3:40
  20 people in the room and also when they became like these huge
• 3:43
  giants that they are today. So HelloFresh was one of the
• 3:46
  company of the Rocket Internet Group.
• 3:48
  And at that time, the founder asked me to move to the US and
• 3:53
  build the data team for the US component of the business.
• 3:56
  It was one of the few rocket companies operating in the US as
• 3:59
  well. And when I arrived in the US, it
• 4:03
  was like 4 people working in data.
• 4:07
  And when I left 35 S within four years, then you're there was a
• 4:12
  tremendous ride. It was also the years of the
• 4:14
  pandemic. So the business was booming and
• 4:17
  we did a lot of crazy and interesting stuff by
• 4:20
  implementing, created a platform, great analytics
• 4:23
  processes, recommendation engines, forecasting models and
• 4:27
  so on and so forth. Yeah.
• 4:30
  The the interesting thing about this is that the ML aspect of
• 4:36
  the business is integral to whether or not that business
• 4:39
  succeeds or fails because it is such a hard thing to do when you
• 4:44
  have fresh produce. And you need to know how much of
• 4:47
  this am I going to need tomorrow?
• 4:48
  How much am I going to need next week?
• 4:50
  And if you get that wrong, that can really damage the business.
• 4:54
  And if you consistently get it wrong, you don't have a
• 4:57
  business. Yeah, I can tell you stories of
• 5:00
  missed forecast and people having to run Whole Foods, Whole
• 5:04
  Full Foods around the area with the company credit card.
• 5:09
  Wow. High as much lime as possible to
• 5:12
  fulfil our clients demand it. It is it is really like
• 5:16
  interesting where like small mistakes in your prediction can
• 5:20
  have too big impact on the company bottom line.
• 5:23
  And definitely the forecasting aspects of other fresh was
• 5:27
  extremely interesting and complex to actually solve.
• 5:30
  Because the product has so many in the box, you ship so many
• 5:33
  ingredients. So think about Amazon as a
• 5:35
  complex business models, but you ship mostly items that are not
• 5:40
  perishable and in quantities that are kind of like pretty
• 5:45
  defined in like HelloFresh. Every every box I remember every
• 5:48
  recipe contains an average, every box contain three recipes
• 5:53
  and every recipe has like 7-8 perishable items that you need
• 5:57
  to package accurately and shipped to the to the users.
• 6:02
  And then you add the recommender systems in with that.
• 6:04
  And so I imagine the recommender system chops that you had from
• 6:09
  there translated nicely to Stitch Fix, because that's
• 6:12
  another similar thing. Now you don't have to worry
• 6:15
  about perishables as much, I guess because clothes only go
• 6:21
  out of style, they don't go bad. And some clothes never go out of
• 6:26
  style. So if you're lucky, or if you're
• 6:29
  just oblivious to fashion like I am, then you're good.
• 6:33
  That's right. I think Stitch Fix is an
• 6:36
  interesting different set of problems.
• 6:39
  Actually, Stitch Fix are the very interesting business model
• 6:42
  where when you ask where is your inventory, 50% of your inventory
• 6:48
  is most of the time at FedEx. It's always in transit between
• 6:53
  the user or the final customers and the fulfilment centres.
• 6:59
  And you want to keep it like that because you want to be as
• 7:05
  efficient as possible with the fulfilment space.
• 7:09
  And so Stitch Fix at this very interesting business model where
• 7:13
  you need to be very accurate also on your forecast to always
• 7:17
  have available inventory and relevant inventory to present to
• 7:21
  your customers. Otherwise you can make the best
• 7:25
  recommendation, but if the inventory is not there, you can
• 7:28
  fulfil it. And so there is this very tight
• 7:31
  and nice interrelationship between the two.
• 7:34
  Sometimes the forecast or the recommendation engine can be
• 7:37
  super accurate, but the problem is that the items you retrieve
• 7:41
  are out of stock. So.
• 7:44
  The interplay between the two is extremely interesting and
• 7:48
  complex and that this is problem.
• 7:50
  You have gone down the rabbit hole and I know that you've
• 7:53
  talked a lot about Bayesian theory and working with Bayesian
• 7:58
  algorithms. Can you tell me a bit more about
• 8:00
  that? Yeah.
• 8:02
  So Bayesian algorithms is something that I came to from
• 8:07
  the sideways, meaning I have a background in ML engineering and
• 8:14
  I have a background in things like, you know, traditional ML
• 8:17
  and deep learning and so on. However, a lot of problems that
• 8:22
  I tried to solve is actually to convince stakeholders of the
• 8:27
  reliability of the forecasts we can putting up for them.
• 8:32
  And it's super complicated because the, the stakeholders
• 8:36
  ask you how, what's the logic on the model?
• 8:39
  What is the prediction interval for the model?
• 8:45
  What features are you based this on?
• 8:47
  And you can do that with a traditional ML and there are
• 8:50
  techniques for doing these explanatory variables and some,
• 8:53
  but they're intrinsically hard to to explain.
• 8:56
  And the confidence interval you get out, sometimes they're not
• 9:00
  well calibrated. And so Bayesian models is a
• 9:03
  solution for that because it gives you two things
• 9:06
  out-of-the-box, which is interpretability and confidence
• 9:11
  intervals in which is the ability to add easily
• 9:14
  constraints to your forecast. For example, if you if you know
• 9:17
  that the the output needs to be positive, you can constrain that
• 9:20
  very easily in a Bayesian models.
• 9:22
  So for things like high stakes scenario where you need to do
• 9:27
  like if Invest Ella Fresh had spent $800 million in marketing
• 9:35
  budgets across 4050 media channels every year or at least
• 9:40
  when I was there. So you that's a high stakes
• 9:44
  scenario. So you really want to understand
• 9:47
  how the model works, what are the causal relationship between
• 9:51
  the variables, not just statistical relationships
• 9:54
  between the variable and the Bayesian model is perfect for
• 9:57
  doing something like that. When you have talk to the final
• 9:59
  stakeholders and you see it, CMO and CFO, you can really motivate
• 10:04
  why you actually put up a certain forecast.
• 10:08
  And yeah, and, and therefore I we saw those with those tools
• 10:13
  because they were the right tools to solve the specific
• 10:16
  problem. And you've kind of taken that
• 10:19
  and run with it because now you're still doing stuff with
• 10:22
  it, right? Right.
• 10:24
  So I that's actually solved the the like what I want to solve
• 10:29
  right now. So we with the models, we build
• 10:31
  very sophisticated models for marketing allocations that
• 10:34
  through the years you know that have been published.
• 10:36
  Then companies uses Pine C Labs as companies that has been
• 10:44
  developing the open source library of supporting the
• 10:48
  person's library by MC, which is a statistical libraries that's
• 10:51
  very much used in the industry and is perfect for building
• 10:56
  these type of models. Now the key idea there is that
• 11:02
  in my career always at this problem that you're over short
• 11:06
  of very, very small people to hire.
• 11:10
  And the other problem is, especially when you're running
• 11:14
  analytics teams, what really kills you is the delivery of the
• 11:17
  insights, but it's all the follow up that comes from
• 11:20
  stakeholders. So how do you solve that
• 11:23
  specific problem augmenting your data analytics workflow and data
• 11:28
  scientist workflow with AI? So the idea is that you can take
• 11:32
  these models and talk to them, chat to them using LLMS and
• 11:38
  basically you can do two things. 1 is to simplify the process of
• 11:44
  building these models by having as warm of agents.
• 11:48
  Some are modellers that actually put together the relationship
• 11:53
  between the variable that you need them right the code that
• 11:56
  you need. Others are quality control
• 11:59
  agents that control the quality of your incoming data and these
• 12:03
  accelerate the production of these models and coming out with
• 12:08
  them. And the other one is once the
• 12:11
  model has been built well, you can ask questions of make a what
• 12:14
  if scenario analysis. So if you make the specific
• 12:18
  forecast, what could be a certain scenario that you
• 12:23
  haven't thought about? Please right now answer to the
• 12:26
  questions a would happen if I drop my marketing budget $100
• 12:30
  million in Google next day. The CMO and the in the usual
• 12:36
  process scope, it wasn't really there.
• 12:38
  This question, you thought about this afterwards, No problem.
• 12:42
  I can give you the answer right away without having to involve
• 12:44
  my analyst, because I know how to talk to the model and I don't
• 12:48
  need specialised expertise to rerun that code again.
• 12:52
  So I think that to me, it's a way of solving this very hard
• 12:59
  problem I always had in my careers that it's so complicated
• 13:02
  to scale up data teams effectively and and really
• 13:06
  augment the workflow of your data scientists and analysts
• 13:11
  using these identical technology.
• 13:15
  Yeah, I love this because you are combining the new world Gen.
• 13:19
  AI capabilities with the traditional ML capabilities and
• 13:24
  each one has their purpose and has their value.
• 13:26
  And so there's going to be times where you need to use
• 13:29
  traditional ML and so being able to augment the capabilities and
• 13:35
  augment your understanding of how this works and your benefits
• 13:43
  from them are, are such an incredible superpower to have.
• 13:47
  So if I'm understanding you correctly, you're saying
• 13:52
  something like on the front end when you're building the model,
• 13:56
  you're getting help from LLMS And are you also getting
• 14:03
  features that are being suggested to use you mentioned
• 14:07
  like making sure that there is clean data or the data that
• 14:11
  comes in is is good data. How are you doing that?
• 14:14
  Because that seems like a bit of an impossible chop.
• 14:18
  I think these are very hard jobs, but these are The lamps
• 14:22
  are surprisingly good. Maybe I should also prefix this
• 14:26
  somewhere by Asian models are particularly tailored towards
• 14:31
  this problem because the first things that you would do with
• 14:33
  the Bayesian models, you don't work in a scenario of like very
• 14:36
  large data sets with millions of features.
• 14:41
  So Bayesian models are really, really good where you have
• 14:43
  relatively small data sets, you have several features, but maybe
• 14:47
  in the area of 3040 features themselves.
• 14:51
  It's a small data application, very high tailored problem in a
• 14:56
  very high stake scenario, but relying on relatively small data
• 15:00
  sets. And the reason is why when you
• 15:02
  have a small data sets, the prior of this Bayesian models
• 15:05
  help you with completing the missing gaps in the fact that
• 15:10
  you don't have enough data to draw full inference.
• 15:13
  So they're pretty particularly tailored to that.
• 15:16
  So that allows you to give these data in the context of the
• 15:22
  leather lamps of. And so in the leather lamps by
• 15:25
  having access to this data in the context of the lamps, we
• 15:28
  realise for example, things like hey, there are missing values
• 15:33
  here. Hey, there is a trend in the
• 15:35
  data that's a little bit strange.
• 15:36
  And then when you actually prompt the LLM appropriately by
• 15:41
  telling the LLMS, hey, probably these are type of analysis that
• 15:46
  you need to actually do for quality control and so on and so
• 15:51
  forth. It often comes out with some
• 15:54
  interesting insights about the data that allows you to then
• 15:59
  catch early specific problems. It's not perfect, certainly
• 16:02
  still developing the technology, but there is a series of quality
• 16:06
  control checks that you can try and tell them to and they're
• 16:11
  going to come up even with follow-ups and further checks
• 16:14
  and they can execute that quickly, quickly on this type of
• 16:17
  data set and the next LLM to the next agent for the next steps of
• 16:23
  the analysis. Yeah.
• 16:25
  And how are you doing, How are you operationalizing this?
• 16:27
  Is it some DAG that you have the LLM cleaning the data as one of
• 16:34
  the steps of the DAG? Right.
• 16:36
  So right now we are setting this up as a long graph application
• 16:40
  and there is there is a full graph, there is behind these and
• 16:45
  one step is a quality control agents, another step it's an
• 16:49
  insight agents that taking the data can actually draw some
• 16:53
  interesting plots of it just to explain what is in the data in
• 16:56
  the main trends and the main insights, the relationship
• 16:59
  between the variables. Then there is a modelling agent
• 17:02
  downstream that actually builds the model on top of it.
• 17:06
  And there is forecasting agents allows you to make inference
• 17:09
  with the model about the future and the scenario planning agents
• 17:13
  that allows you to create scenario plans, optimised
• 17:17
  configurations for your variables, for your forecasts
• 17:20
  and so on and so forth. And even at some, which is a big
• 17:25
  problems for analytics teams and time consuming things which is
• 17:28
  creating PDF and X out of the insights of the models of your
• 17:33
  analysis. There is even they are also
• 17:36
  building right now at PowerPoint agents that actually creates a
• 17:41
  deck with the final recommendations for the
• 17:44
  stakeholders so that they can have it at hand and reading it.
• 17:49
  And how are you confident that the data you're getting after
• 17:54
  it's gone through all of these different steps is high quality
• 17:59
  data? So at this stage, of course
• 18:01
  there is there's some parts of automated checks you can make.
• 18:06
  For example, you can check for new virus, you can check for
• 18:11
  things that are out of scale. For example, if you see that
• 18:15
  you're spending in a marketing channels in the MMMMM example,
• 18:19
  if you're spending a marketing channel hundred $200 million in
• 18:24
  a week, that's probably a little bit too much.
• 18:27
  So there is something that's certainly out of scale and the
• 18:29
  LLM will not will note that down.
• 18:33
  Otherwise you still rely on workflow where the human is in
• 18:37
  the loop. It doesn't need to be a human
• 18:41
  which has specialized technical expertise in terms of coding
• 18:45
  because the LLM, that's the coding for you.
• 18:49
  But it needs to be a human that has business context good enough
• 18:53
  so that can understand whether the data is the right data and
• 18:57
  the output of the entire workflow is something that's
• 19:00
  sensible and you can make a decision on reliably.
• 19:05
  That's fascinating to think about.
• 19:08
  The most important human to be in this loop is less of a data
• 19:13
  scientist, and it's more of the one who understands the business
• 19:17
  context so they can flag something when it looks a little
• 19:21
  off. Right.
• 19:22
  So in fact, who was the target customer of an application like
• 19:26
  this is like busy analytics and data science tips that want to
• 19:31
  augment the workflow of their analysts and data scientists
• 19:35
  without having them to write the code necessarily from scratch
• 19:38
  for some of his models or come up the semi automated analysis.
• 19:43
  And the other one is really tech pics in the stakeholders that
• 19:47
  are technical enough to understand what's right and
• 19:50
  what's wrong, but they have also like a deep business context
• 19:53
  that they can help them guide the the analysis and guide the
• 19:59
  results of the models. Yeah, It reminds me of the
• 20:06
  sequel Analyst Agent that process put into production
• 20:10
  recently and they were talking about how they have this bell
• 20:14
  curve in a way where you have the very advanced sequel queries
• 20:19
  that you need to ask and you need to spend a lot of time
• 20:22
  digging into the data. And then on the other side of
• 20:25
  that, you have the do it yourself and everything is going
• 20:30
  to be written by an LLM. So there's like a spectrum of
• 20:35
  how much LLM you're using and on one side it is only LLM
• 20:41
  self-serve and on the other side it is no LLM because it's far
• 20:45
  too complex. And what they were mentioning
• 20:49
  was the majority of the lift is in the middle where it is
• 20:53
  something like you said, it's augmenting those who already
• 20:55
  understand or have enough experience to be able to get the
• 21:00
  value from it. But they aren't on either side
• 21:04
  of this spectrum where it's highly complex and you can't use
• 21:08
  an LLM, or you don't know anything about it and you're
• 21:11
  relying 100% on an LLM. That's exactly right.
• 21:16
  And it's a great example that you make because the idea comes
• 21:20
  a little bit from that and it's a development of the idea of SQL
• 21:26
  Agents. Because when I look at that
• 21:28
  idea, I thought, oh wow, that's great.
• 21:31
  But it stops at descriptive statistics, which is the
• 21:35
  one-on-one of analytics. What if instead of just doing
• 21:39
  descriptive statistics and get sequel out and nice plots, you
• 21:43
  can actually do predictive analytics and mixed forecasts
• 21:47
  and close the gaps between those advanced statistical models and
• 21:52
  the the the delivery that you need to give to the business in
• 21:55
  a very fast way. So it was inspirational the work
• 22:00
  of what companies like, you know, data bricks are doing, for
• 22:03
  example, like agents like Genie, the Dust X to sequel completely
• 22:09
  integrated in, you know, the the data bricks ecosystem and data
• 22:13
  lake. However, our idea was like, you
• 22:17
  shouldn't stop there. You should actually go the last
• 22:19
  mile now. Help automating or like
• 22:22
  supporting and augmenting the entire analytics and data
• 22:25
  science classes. And so now there's a lot of
• 22:29
  agents that you have working and you also have the human in the
• 22:33
  loop and being able to oversee the agents.
• 22:38
  How are you doing evals, if at all?
• 22:42
  And that feels like it could get very sticky very quickly or it
• 22:48
  could just become a ton of work if you have all of these agents
• 22:52
  and you're trying to make sure that they are producing high
• 22:55
  quality data. Yeah.
• 22:56
  But they're still learning about what's the appropriate way of
• 23:01
  doing evils there. We have some reference
• 23:05
  workflows. So we have some reference data
• 23:08
  and workflows and we do know that the agents applications
• 23:15
  needs to come to certain conclusions.
• 23:16
  And we have way of verifying that the agent applications you
• 23:20
  know, doesn't get stuck or gets results and parameters for the
• 23:25
  models that are aligned with reference parameters for the
• 23:29
  models because you can verify that even generating synthetic
• 23:32
  data, for example. And so we have a set of those
• 23:36
  that are in place. We have Lang Smith and other
• 23:41
  telemetry that allows us to check what the queries of the
• 23:44
  users are and see whether the user gets stuck.
• 23:47
  But finding the precise way and the more systematic way of
• 23:52
  evaluating this type of applications is something that I
• 23:54
  don't think we have solved yet. And I don't know if the industry
• 23:58
  has entirely solved yet. There's still kind of an open an
• 24:01
  open debate. Yeah, about 100%.
• 24:05
  And you mentioned to me before too that you're leveraging this
• 24:09
  for product development and almost getting like user
• 24:13
  research done. Can you talk to me about that
• 24:16
  use case? Yeah, that's, that's a, it's a
• 24:18
  little different use case of what I thought thought about,
• 24:22
  but it's kind of the same idea. And in, in, in a bit of a, in a
• 24:28
  bit of a different way. So let's the the idea here is
• 24:31
  that you have an entity like a Bayesian model, a machine
• 24:34
  learning model. You want to took to it like it
• 24:37
  was your friend and ask how you've been built, took me a
• 24:40
  forecast and so on and so forth. Now there's another problem
• 24:45
  these often is company have is that they don't know that much
• 24:49
  their customs and product managers, business stakeholders,
• 24:52
  marketing people, designers. They wish they could talk to
• 24:55
  their customers and user every day to learn and get, get more
• 25:00
  insights about what they are doing out there using the
• 25:03
  products of why they're using the product.
• 25:06
  And so another applications we are building is some virtual
• 25:11
  representation of your consumer that we call synthetic
• 25:14
  consumers, which you can ask any type of question too.
• 25:18
  And the typical applications would be, hey, you are, I don't
• 25:23
  know, it's CPG company for example.
• 25:25
  And you are developing a new type of, I don't know, like
• 25:31
  let's say a new type of toothpaste.
• 25:34
  And you want to know whether this resonates with the user
• 25:37
  base and the consumer base. Well, the only way you'd have to
• 25:41
  do it right now, you, you need to do very expensive panel
• 25:45
  interviews with tons of consumer research.
• 25:49
  And it's very times consuming and expensive while you could
• 25:52
  have a swarm of other lands that are primed and prompted to
• 25:57
  behave like real people. And you can just ask to show
• 26:01
  them pictures and ask them questions of what you buy these
• 26:04
  products for a certain price points.
• 26:05
  What are the features of this product that you like?
• 26:08
  And there is research that have shown that within certain
• 26:12
  constraints, this is can be actually representative of the
• 26:16
  real population. So especially if you build these
• 26:20
  synthetic consumer in the right way, using the right type of
• 26:23
  data and so on and so forth. And we are still in the early
• 26:27
  stages of building this application for a Fortune 500
• 26:30
  client, but that's the the next type of things.
• 26:33
  And I think in the longer term this could becomes really an
• 26:37
  interesting technology. So imagine that you're a product
• 26:40
  manager and you're building a new website.
• 26:43
  Well, you probably want an operator or synthetic consumer
• 26:47
  that browse through your website, try to do specific
• 26:51
  actions, and then you want to ask questions.
• 26:54
  Oh, how did you like the bottom up there?
• 26:56
  Or did you find this confusing? There's a kind of like user
• 27:00
  workflow and why? And I think this could be
• 27:03
  extremely powerful to really close the loop between and,
• 27:07
  design better UX and close the loop between, you know, product
• 27:10
  development and real users. Yeah, you can take that a step
• 27:17
  further to and anyone that's developing software tools or
• 27:23
  infrastructure tools, you can get synthetic data or just have
• 27:29
  an LLM use your software tool and tell you what they think
• 27:35
  about the API and how your specs where it's confusing.
• 27:40
  And so that is a a huge promise. I wonder how in line it is with
• 27:48
  real humans and how many insights.
• 27:50
  I guess at the end of the day, even if it's not totally in line
• 27:54
  with humans, like what real humans would be doing, if you
• 27:59
  can gather insights, that's what you need and you can at least
• 28:03
  get a few iterations before you bring something to market or
• 28:07
  before you come out of stealth or whatever.
• 28:10
  And so this is a fascinating way of doing it.
• 28:16
  Now the the main question that I have is how are you making sure
• 28:20
  that the LLMS are properly set up to be these consumer
• 28:26
  profiles? Are you prompting them?
• 28:28
  Are they fine-tuned to a certain person?
• 28:33
  What does that look like? And also what kind of models are
• 28:36
  you using? Yeah, that's, that's a very good
• 28:39
  question. Like I, I, I want to answer
• 28:40
  first your your first insult. This can be fascinating.
• 28:45
  So the, the idea comes from a very personal actually
• 28:48
  experience. I'm, I'm a little bit of a timid
• 28:50
  person. I kind of like, I wish I could
• 28:53
  speak freely with and get connections with a lot of
• 28:56
  different peoples. And sometimes the applications I
• 28:59
  have and the discussions I have with LLMS, it's extremely deep.
• 29:03
  So that's why I wish I could talk to anything.
• 29:06
  I could talk to, you know, a scientist so I could talk to a
• 29:12
  famous scientist like Richard Feynman, or I could talk to a
• 29:16
  user of my final applications that have no problem asking the
• 29:19
  presence of them. That's why I'm, I'm so excited
• 29:22
  about building stuff like this. Now the second part of your
• 29:26
  question is, are you actually building it and making sure that
• 29:29
  they are aligned with the entity you want to represent, meaning
• 29:35
  the user that are the user of this application?
• 29:38
  So at the moment, we're exploring variations of
• 29:42
  prompting and that brings you or like a, let's say, 7060% there.
• 29:51
  And the typical way you would do this is by really prompting the
• 29:58
  Lambor with things like, hey, you are a black woman who's
• 30:03
  living in Brooklyn, you have a tech job, your confidence about
• 30:09
  this. You bought items of the specific
• 30:11
  product category recently and that brings you there up to a
• 30:17
  certain level. What you can also do, and we are
• 30:22
  experimenting with that is that you can take this company's past
• 30:28
  consumers research and surveys that where as worthy in the past
• 30:35
  do supervised fine tuning. And basically they had already
• 30:39
  like massive data sets in the past of product they've shown to
• 30:42
  consumers research users, but they collected demographics of
• 30:46
  those users and specific behavioral and you can use those
• 30:50
  to do supervise fine tuning and then you know, like create
• 30:55
  another lens that are specifically for that.
• 30:57
  Of course you base these on open source models to do supervised
• 31:03
  fine tuning like the Llama models for example.
• 31:06
  Now it's really complex and there are there's a lot of
• 31:11
  nuances of it. For example, you know that and
• 31:18
  most of these LLMS, they have specific political biases.
• 31:21
  For example, has been proven that American lamps, they are a
• 31:25
  little bit like left-leaning in general.
• 31:28
  And what you can do, and we are also exploring is techniques to
• 31:35
  remove these spices to make sure that the base of a lamp you
• 31:38
  start with is actually impartial.
• 31:43
  And you can do that with also technique like ablation for
• 31:46
  example, which is a technique to remove specific biases from LLMS
• 31:52
  by removing and killing specific neuron within the network.
• 31:56
  So we are still at the beginning of it.
• 31:59
  The basic applications bring you 6070% there, especially when you
• 32:03
  go in fine grained sub segments of users takes the IT gets the
• 32:07
  average and the bulk of the American population's right.
• 32:10
  But when you look and slice your data in sub segment of the users
• 32:14
  can be very often. So we're into these techniques
• 32:18
  like supervised fine Q and encabulation to refine the model
• 32:23
  outputs, especially on sub segments.
• 32:26
  Well, the idea of having someone go through your website too and
• 32:31
  looking at heat maps and all of that.
• 32:33
  One of the first things that I did when I was at a startup that
• 32:38
  kind of gave me a bit more freedom to play around with the
• 32:42
  product was we had a tool installed on our product called
• 32:48
  Full Story, and that would record everyone's sessions in
• 32:52
  the product and I would just sit there.
• 32:54
  I was mesmerized by how people were using the product because
• 32:57
  you can really see where folks get caught up and where they're
• 33:02
  snags. And Full Story I remember had
• 33:05
  this feature, I don't know if it was a feature, it had an alert
• 33:10
  when someone would do what they called rage clicks.
• 33:14
  And that was where if it wouldn't get, if someone
• 33:16
  wouldn't get what they wanted on the first click and then they
• 33:19
  would click like four or five more times because they wanted
• 33:22
  that. But for some reason the button
• 33:25
  wasn't working or the page wasn't loading, whatever it may
• 33:28
  be. And so it's, it's incredible to
• 33:35
  have that almost before you do anything or before you actually
• 33:41
  have real users and you don't have to have real users go
• 33:44
  through the pain and that rage and you gather the insights and
• 33:49
  then you can talk with the LLMS to gather those insights and
• 33:54
  say, what are some of the key things that I'm missing?
• 33:56
  Here's my write up maybe, or here's some things that I'm
• 33:59
  seeing. What am I missing?
• 34:01
  That's that's right. You need to do it.
• 34:03
  If you gather that just from data, you do it need to do a lot
• 34:06
  of guess work. And that's the reason why UX
• 34:10
  research exists and qualitative surveys exist because
• 34:15
  quantitative data bring you, brings you like probably 80%
• 34:18
  there, but you still want to understand human behaviour also
• 34:23
  from a qualitative background. And these techniques I hope in
• 34:29
  the long term will be a way of like bringing UX research to the
• 34:35
  next level. Yeah.
• 34:38
  And so are you trying to operationalize this or is it
• 34:42
  something that is much more handcrafted for each use case
• 34:47
  that you encounter with the companies?
• 34:48
  Yeah. So at the moment we are
• 34:52
  basically consulting companies. So we have clients interested in
• 34:55
  building specific applications which is for product innovation
• 35:01
  and in the CPCPG space. So we are developing this with
• 35:06
  that angle in mind. I do think that unlike other
• 35:11
  consulting companies, we are a little bit different at IMC Labs
• 35:16
  because we have a strong belief in innovation and open source.
• 35:20
  And that's the reason why we called Labs, because we are
• 35:23
  recognised internally as a group of researchers that it's wants
• 35:28
  to solve interesting problems and everybody has fantastic
• 35:32
  backgrounds both in academia and in the industry.
• 35:36
  And so we think that some of these applications we are
• 35:40
  developing right now is using a concrete problems that comes
• 35:45
  from our clients in the industry in the longer term can be
• 35:47
  strategic for us to develop and they're going to be released as
• 35:51
  open source where we can of course.
• 35:54
  And, you know, they're going to consider the body of work that
• 35:59
  that's going to remain for everyone.
• 36:02
  Now, when you look at other use cases and ways that you can
• 36:07
  merge the Bayesian world with the LLM or just language model
• 36:11
  world, have you seen other stuff that you want to attack?
• 36:15
  Maybe you have put time into trying to make it work.
• 36:19
  Maybe it's an idea that's floating around in your head.
• 36:23
  Yeah, I think this area of probabilistic deep learning and
• 36:29
  probabilistic neural network is really interesting and it's
• 36:34
  another year you want to talk. So we talked about the idea of
• 36:38
  augmenting the Bayesian works below the lamps.
• 36:41
  We talk about the idea of running even simulations of like
• 36:46
  agents that behave like users. And then what you can do, and I
• 36:50
  mentioned later on, is to add to the simulation some Bayesian
• 36:54
  prior later on. But intrinsically there is also
• 36:58
  another angle, which is your fundamental deep learning model
• 37:02
  can become a probabilistic models if you add probability
• 37:06
  distributions on the weights, for example, and you sample from
• 37:09
  the neural networks. That's the reason why you know,
• 37:14
  like the technique you use in Bayesian modelling like building
• 37:17
  computational graph sampling from GPUs.
• 37:20
  They're actually the same technique, computational
• 37:23
  technique fundamentally that you use in deep learning.
• 37:26
  The result, even libraries like tensor probability, which is an
• 37:30
  adaptation of tensor flow to the problem of building Bayesian
• 37:33
  models. Now a deep probabilistic deep
• 37:37
  learning, it's extremely interesting.
• 37:39
  It's very hard to solve computationally and there is
• 37:42
  still a lot of research to be done on whether the probability
• 37:46
  distribution to get from this deep learning models, it's
• 37:50
  really calibrated and the correct probability
• 37:53
  distributions you can rely on. So lots of research to be done.
• 37:59
  There is a body of research that comes from Ath Zurich that came
• 38:05
  a couple of weeks ago out in a very nice summary paper.
• 38:13
  So I think that's where I want to go there next.
• 38:18
  It's because it actually addresses very key problems,
• 38:21
  business problem, especially in the area of, you know, high
• 38:26
  stakes predictions like for self driving cars, for example, where
• 38:30
  you need to know the confidence of your like predictions for
• 38:36
  instead of pedestrian there. Or things like financial
• 38:40
  forecasts or for I stick scenario like for example, hedge
• 38:43
  trading and hedge fund trading or like things like medical
• 38:49
  predictions for like knowing whether a person has a cancer,
• 38:54
  for example. So these are extremely
• 38:57
  interesting topics that we probably are going to get also
• 39:01
  like you know, some work from our side in future.
• 39:07
  And I missed how exactly that corresponds.
• 39:11
  Or did you say that that's just somewhere where you want to
• 39:16
  start focusing your attention? Yeah.
• 39:18
  So I think it's the bleeding edge of research today.
• 39:23
  It's still I, I would always divide, let's say, the world
• 39:29
  into categories, the bleeding edge of research.
• 39:31
  It's not yet ready for industrial applications, then
• 39:34
  things that are ripe for building applications on top.
• 39:37
  And then there is the state-of-the-art that everybody
• 39:40
  uses and keeps building up. I classify, and this is of
• 39:44
  course just my opinion, but I would classify probabilistic
• 39:47
  deep learning as the bleeding edge of research, almost at the
• 39:51
  brink of becoming something that you can implement in industrial
• 39:55
  applications. And of course, you know, people
• 39:58
  was going to shoot me and they're going to say, oh, there
• 40:00
  is already some applications built on top of probabilistic
• 40:03
  deep learning. I'm a deep expert in that topic,
• 40:07
  but that's kind of my my high level assessment of the field at
• 40:10
  the moment. It's so cool, man.
• 40:13
  That's it's really nice that you're so deep on this stuff and
• 40:19
  you're thinking about it and then thinking back to how it can
• 40:23
  be tied into the companies that you're working with and how can
• 40:27
  we bring business value with this, right?
• 40:30
  And so one thing that I Speaking of asking users for feedback,
• 40:35
  it's so funny that you mention this because I literally this
• 40:38
  morning, before we hopped on this podcast, I spent probably
• 40:42
  an hour and a half emailing folks that sign up for the ML
• 40:46
  OPS community and that have written me because one thing
• 40:48
  that I asked for just to find out if they're human or not is
• 40:52
  what's your favorite song? And so on that first e-mail when
• 40:56
  you join the ML OPS community and says, Hey, this is the ML
• 40:58
  OPS community. I'm demetrious here just so that
• 41:01
  I know what's your favorite song?
• 41:03
  And so people write me back with like favorite artists and all
• 41:05
  that. And by the way, it is an
• 41:07
  incredible way of finding new music.
• 41:09
  I have such cool playlist now that I've never had heard of,
• 41:14
  but what I was doing for an hour and a half today was writing
• 41:18
  everyone back saying great suggestions, finding new music,
• 41:25
  this is awesome. And also asking for what's going
• 41:28
  to be the most valuable thing that this community can offer
• 41:32
  you. Because I always want to know
• 41:36
  what else can we be doing in the community to make it more
• 41:39
  valuable to people. And so that type of stuff I
• 41:44
  need, I need the LLM, I want to talk to the LLM also.
• 41:49
  And then also just the reason I was, I was bringing this up is
• 41:53
  at the end of last year, I sent out a bunch of emails and
• 41:58
  messages to close friends asking them what's something that we
• 42:02
  can do more of in the community in 2025 that will be valuable
• 42:09
  for you, right? And one that has stuck with me,
• 42:12
  I can't remember who told me this, but somebody said, I
• 42:15
  really appreciate it when you talk about bringing the ML and
• 42:21
  AI field to the business side and tying it to business
• 42:28
  metrics. I think I'm paraphrasing
• 42:31
  obviously, but tying it to business metrics.
• 42:33
  And so I feel like you're someone who sits in a unique
• 42:36
  position because of the area that you're in right now, being
• 42:42
  able to work with a lot of these Fortune 500 companies, but also
• 42:45
  all the stuff you've done in the past with these hyper scalers.
• 42:49
  Do you have ways that you can sniff out high value use cases
• 42:57
  or high value or or basically bridge that gap between the
• 43:02
  machine learning and AI side of the house and the business
• 43:05
  metrics and tying it back to it, that isn't just an LLM creating
• 43:09
  a PowerPoint for you? That's a very, very, very good
• 43:12
  question. I think that I I don't have the
• 43:18
  magic answer to that, but there are two principle 2 principles
• 43:23
  that are used that are really guided me and my career.
• 43:26
  So the first principle is try to understand very deeply the
• 43:32
  business model and what moves the needle percentage wise.
• 43:38
  Meaning is in this specific business, business models you
• 43:43
  can always model as a graph. And then if you change X percent
• 43:47
  in this variable, let's say marketing spend, how much the
• 43:51
  revenue profitability of the company changes, that's
• 43:56
  percentage wise. If you find those variables of
• 44:00
  very, very high elasticity, meaning a small change here
• 44:04
  changes kind of a lot your top line and bottom line, then you
• 44:10
  found an area where you want to dig deeper.
• 44:12
  And often what I do when I join a new company, I do an analysis
• 44:17
  of the business model. I try to find those areas of
• 44:20
  high elasticity looking really at the composing, the business
• 44:23
  modeling, kind of like a graph of variants.
• 44:28
  The second one is that, and this is like more like a leadership
• 44:32
  principle. You don't need to come up as a
• 44:37
  leader with all the answers all the time.
• 44:40
  You can rely on your team. And often if you think about
• 44:44
  your portfolio of teams as a head of data, you have the data
• 44:48
  platform teams, your analytics teams, you have a machine
• 44:51
  learning teams and so on and so forth.
• 44:53
  Your analytics teams, it's the gold nugget Finder team.
• 44:59
  So they work closely with stakeholders, they often uncover
• 45:05
  needs insights, opportunities that you did a platform teams
• 45:09
  and you did assign these, the machine learning engineers
• 45:11
  wouldn't see it otherwise. And you can use them to really
• 45:15
  to evangelize the approach that your team has and to uncover new
• 45:21
  interesting problems. And so often, or actually a
• 45:25
  trick that I learned that I used in all my teams is that the the
• 45:28
  teams should have business calls.
• 45:30
  And a business call that I often give and KPI loves to give to to
• 45:34
  my analytics teams is by the end of the quarter, you need to find
• 45:38
  ten $20 million of new opportunities.
• 45:43
  That's why I keep the problem back to them.
• 45:45
  And I asked them to solve the problem in this way.
• 45:48
  And it's, it's a core KPI that they have, they got it.
• 45:53
  And it really doesn't matter whether it's 10,000,000 or
• 45:55
  20,000,000. And if they achieve, you know, X
• 45:58
  percent of it only. But it puts them on the
• 46:00
  offensive to going to their stakeholders and instead of like
• 46:05
  just getting the problem from them, helping them think through
• 46:09
  their problems and finding new opportunities.
• 46:13
  So this makes me think almost in in a different way that I
• 46:19
  normally will look at it. But if you are looking almost at
• 46:24
  a pipeline of use cases or potential use cases, you have
• 46:30
  the analysts that are out there like Sherlock Holmes trying to
• 46:33
  uncover new use cases and then figure out how much that is
• 46:38
  going to make for the business or save the business.
• 46:43
  And then that is properly scoped and there's an idea there.
• 46:49
  Then they can go and hand it off to the respective teams or they
• 46:52
  can go and champion for it. And then it's up to the leader
• 46:55
  to make the decision if it's worth pursuing or not.
• 46:58
  And then you go and you say, OK, well, this is actually going to
• 47:02
  be implemented by the machine learning team or it's going to
• 47:05
  be something that the data platform or machine learning
• 47:09
  platform team has to go and implement.
• 47:12
  Because we see that if we can increase the velocity of getting
• 47:18
  a machine learning model to production by 2% or 10%, then
• 47:22
  that's going to save us or make us X amount of money.
• 47:24
  Or if we can bring down the fraud by .3%, then that's going
• 47:29
  to save us X amount of money, whatever it may be.
• 47:32
  That is a really cool way of thinking about it as like the
• 47:38
  analysts are out there searching through the business just like
• 47:43
  lifting up rugs and trying to find dirt.
• 47:47
  Yeah, or or they get it from the stakeholders, for example.
• 47:49
  It's completely new problems we haven't thought about.
• 47:52
  For example, I mean, in hindsight now at Ello Fresh
• 47:56
  always had this problem of getting amazing pictures of food
• 48:00
  and creatives of food. And you know, there is entire
• 48:03
  photo op team that's dedicated to that.
• 48:05
  Well, I mean, like the analysts could have noticed that specific
• 48:09
  angles specific like type of pictures got tons more
• 48:12
  engagement and down go figure with your MML team are the
• 48:17
  actually scale that process up with generality that I so here
• 48:21
  it's a new business problems that was uncovered by an
• 48:23
  analyst. A good observation it's not the
• 48:27
  team as a head of data that I work with that often the the
• 48:32
  team of like photo op for like food.
• 48:36
  So that insight is brought to me back to from my analyst.
• 48:42
  I say wait a minute, there is something to dig further here.
• 48:46
  Let me go make the connections with my data science team.
• 48:49
  So let's try to automate that process and get value out of it.
• 48:52
  So that's the IT actually works. The value of the analytics teams
• 48:57
  is to be decentralized in your data portfolio and to really get
• 49:02
  this inside golden act that that comes across the business from
• 49:07
  everyone. Yeah.
• 49:11
  And then figuring out, I imagine there's got to be some tough
• 49:15
  calls that are made that potentially, like you were
• 49:18
  saying, the technology isn't there yet.
• 49:21
  I know that there was someone who I was talking to a few weeks
• 49:24
  ago that is embedded in the finance team.
• 49:27
  And there's over 42 people on this finance team.
• 49:30
  And the majority of the stuff that takes this finance team
• 49:34
  ages to do and what they are constantly being flooded with
• 49:39
  are PDFs from banks. And they've tried so hard to
• 49:45
  figure out how to ingest those PDFs so that LLMS can do the
• 49:50
  bulk of the work, but it has been a tedious process and they
• 49:55
  haven't been able to get there with technology to ingest the
• 49:57
  PDFs and allow the LLM to fill in the PDFs for them.
• 50:04
  And so there it's almost like the, is it worth it to spend X
• 50:11
  amount more time? I, I guess if it's a gigantic
• 50:14
  business problem like you were saying back to the first part of
• 50:18
  your answer on how important is this?
• 50:21
  If we spend a year on it and we get 1% savings, if it's
• 50:27
  $1,000,000 company, that's probably not good.
• 50:30
  But if it's a billion dollar company, that's going to be well
• 50:34
  worth it. That's exactly right.
• 50:37
  There is also like or there. There are principles that you
• 50:40
  can also apply here depending on the phases your company is in.
• 50:45
  So for example, if you are in a growth stage company, it's
• 50:51
  rarely the right thing to focus your team on cost saving
• 50:57
  opportunities and automation opportunities.
• 51:00
  Why? Because your business is growing
• 51:03
  really, really fast. And typically when it grows
• 51:06
  really, really fast, the opportunity cost of solving the
• 51:10
  growth problem like how do I get to the next $10 million of
• 51:13
  revenue versus solving a a relatively small scale
• 51:18
  automation problem? How do I get four people to
• 51:22
  spend 10 hours less on this specific problem?
• 51:26
  It's fairly worth the. However, if you are already a
• 51:30
  business at scale where growth is really hard, then actually
• 51:35
  focusing on cost saving opportunities it's important.
• 51:38
  For example, if you are in Amex and you have thousands and
• 51:41
  thousands of agents doing phone calls and that's big chunk of
• 51:46
  your like costs, then of course working on automations of like
• 51:52
  client success processes is extremely important.
• 51:55
  So you need to think where your company is at, where the
• 51:59
  strategy of the company is AT, and of course this gold Nuggets
• 52:02
  that your team identify needs to be contextualized within the
• 52:06
  broader fiction. Now you mentioned at the first
• 52:11
  part of that answer a while back how you will get very intimate
• 52:17
  with the business model. What are some things that you do
• 52:21
  there? Is it just you're reading their
• 52:24
  S ones or 10 KS if it's a public company?
• 52:26
  And if not, are you going to the CFO and saying, hey, how does
• 52:31
  how does this thing work? Or do you have other strategies
• 52:35
  of making sure that, again, you're able to uncover rocks
• 52:38
  that seem like pebbles, but really they're boulders?
• 52:42
  Well, that's such AI. Don't know exactly if I have a
• 52:46
  good recipe for doing this. I definitely talking to people
• 52:52
  is extremely important. I mean, if you're just sitting
• 52:55
  in the company as a head of data, the VP or chief executive
• 53:00
  level, you of course need to work very closely with your
• 53:05
  peers to understand the business model or the CEO to understand
• 53:08
  the business model and how the company works.
• 53:10
  In a way, as a head of data, you also have the superpower of
• 53:15
  being able to look at the data very closely.
• 53:18
  And what I typically do, I look at concept like LTV, which is,
• 53:25
  you know, a very historically, it's a very concept that's being
• 53:29
  built and started throughout. But understanding where the
• 53:33
  value is created in your business model and how these
• 53:37
  accrues to revenue, it's the first step.
• 53:40
  If you really understand that, that you can model it even with
• 53:43
  a Bayesian model, for example, then you have understood your,
• 53:48
  your business model. And then I look at consumer
• 53:50
  research, like I look, for example, I've NPS surveys that
• 53:56
  we did to our users and what problems they surface within our
• 54:00
  product to identify other areas of opportunities.
• 54:04
  Often talking to your CPO is extremely insightful.
• 54:09
  CP OS are obsessed with using the products themselves and they
• 54:13
  offer uncover like very interesting insights and
• 54:15
  problems. So this is the tricks that I use
• 54:20
  to really get familiar and acquainted with the business
• 54:22
  model. Most lately, I also use deep
• 54:27
  research from open high a lot power user.
• 54:30
  I think of that often I have just the prompt with I, if I'm
• 54:34
  working with a company I never thought or never worked with
• 54:36
  before, I asked deep research to give me like an overview of the
• 54:41
  industry, the the trends and why this company differentiate from
• 54:45
  the competitions is what there's, it's their strategy.
• 54:47
  And that's actually is a good way from an external point of
• 54:52
  view to familiarise with the new business model and with the new
• 54:56
  companies I took even to BCS that are using deep research to
• 55:00
  think about the investment here and opportunities.
• 55:03
  So. Actually, now that you say that,
• 55:07
  it is so cool to think about how you can use deep research just
• 55:11
  to get an understanding of what the competition is doing also
• 55:16
  and what their most valuable products are or services that
• 55:21
  they may be selling. And then it can inspire you in
• 55:26
  different ways potentially, or it can show you where, oh, maybe
• 55:30
  there's a product line or an offering that we can incorporate
• 55:34
  into our company also. So I've just been using Deep
• 55:40
  research and I use the Gemini version, but I've just been
• 55:43
  using it to like research stuff that I want to buy.
• 55:46
  I didn't think about researching that kind of thing.
• 55:49
  You are much further ahead of me and I appreciate that little
• 55:54
  tip. So now whenever I talk to a
• 55:56
  company, I'm going to use deep Research and get a full report
• 56:00
  before I talk to them. That's right.
• 56:02
  Very, very, very important. And it's like, yeah, if you're
• 56:05
  ever comparing what shoes you want to buy or what, you know,
• 56:10
  like I went from watching a ton of YouTube videos to now asking
• 56:14
  it, hey, I want to buy a new car.
• 56:16
  What's the best hybrid or what has the highest score?
• 56:20
  What cars do people like the most?
• 56:22
  You know, that kind of thing on those bigger ones, just so I can
• 56:26
  know, like, where's the signal? Where's the noise?
• 56:28
  And it does a pretty good job of it.
• 56:31
  Or yeah, shoe I, I think I've done it for shoes, cars.
• 56:36
  And my buddy told me about it. He told me he was doing it for
• 56:40
  his watches for like exercise, you know, like garments.
• 56:44
  Because there's a anything where it's like there's a million
• 56:47
  different models and each one has its own little things.
• 56:51
  However, I have tried it for GPU's and it did not work.
• 56:56
  I was like, I want to know all yeah, like GPU reserve.
• 57:00
  Like basically if I want to pay for some GPU's, I want to know
• 57:05
  how much, what's the pricing models?
• 57:07
  What are the different value props that each GPU provider
• 57:10
  has? And there's like manage GPU
• 57:14
  services and then there's non managed.
• 57:17
  And So what are the ones that do this or that can't really get a
• 57:24
  good handle on it? And the whole reason I wanted to
• 57:27
  do that was because we're creating this like GPU buyer's
• 57:30
  guide. And so we're trying to put
• 57:35
  everything that folks would want to know when they are trying to
• 57:39
  buy GPUs or just like rent GPUs even what do you want to know?
• 57:44
  You're on the market for some GPUs.
• 57:45
  What do you want to know? And so we're we're putting that
• 57:48
  all together. I tried to do it with Gemini and
• 57:51
  I couldn't get a good. Because I what is the reason do
• 57:56
  you think and they didn't get to the right answer?
• 57:59
  I think it's super confusing and so I don't know, maybe the maybe
• 58:04
  the LLM, I shouldn't underestimate the LLM like that.
• 58:08
  But one thing is to be able to find.
• 58:10
  First of all, where I think deep research fell down was that it
• 58:16
  wasn't able to find all of the different providers.
• 58:20
  But you would think like with Google, it would be able to.
• 58:23
  But I don't know if there's a million GPU providers out there.
• 58:27
  And I know of a lot of them. And I told her even more now
• 58:30
  after doing this research and the report that it put together
• 58:35
  did not have half of them. It also had a bunch that weren't
• 58:39
  really what I was looking for. So it was, I don't want to say
• 58:44
  it was like scam pages, but it wasn't high quality pages.
• 58:48
  And then the value props and the pricing they had no like they
• 58:52
  really didn't get well. And so I don't know if it was
• 58:56
  hallucinated or if it was just especially on the pricings, you
• 59:01
  know, and, and I want to go as deep as when somebody's on the
• 59:05
  market for a GPU, they're probably on the market or they
• 59:09
  could be convinced to use ATPU or Amazon Awss like Inferencia
• 59:16
  or Tranium. And so I also want to
• 59:20
  incorporate those in. But as when you say GPU for deep
• 59:24
  research, it's only looking at GPU.
• 59:26
  It's not looking at now like OTP use and Inferencia and all of
• 59:30
  that. I, I, of course, I could prompt
• 59:31
  it better by just saying now also look at Inferencia, but it
• 59:37
  might be my prompts. It might just be that, you know,
• 59:40
  like, but we're really, I can invite you to the notion space
• 59:45
  where we're trying to do all of it.
• 59:46
  In case you have any feedback on, have you ever been on the
• 59:49
  market for GPUs? I feel like you have.
• 59:51
  No, I haven't. I haven't.
• 59:53
  No. All right then, maybe not the
• 59:55
  best. Yeah, Cloud, cloud resources off
• 59:57
  the shelf as well. We haven't shopped ourself or
• 1:00:00
  GPUs, but it's so fascinating. I I I actually never thought
• 1:00:05
  about using it for shopping, which is an obvious application
• 1:00:08
  in retrospect and like and 2nd is that.
• 1:00:13
  On such a specialized product that cannot really find a good
• 1:00:18
  good summary which is interesting.
• 1:00:21
  Yeah, it's weird. It's weird because of maybe
• 1:00:24
  there's just too many providers and so the context or I don't
• 1:00:29
  know what they're doing behind the scenes with the deep
• 1:00:31
  research agents that they've got, but it's not really.
• 1:00:36
  At least when I did it a month ago, it wasn't working.
• 1:00:41
  So maybe it has changed since then.
• 1:00:44
  And I also again was using my Gemini's deep research.
• 1:00:46
  So maybe like the open AI is deep research.
• 1:00:49
  Significantly better than the Gemini 1 is it yeah, oh really
• 1:00:53
  way better. So like while Gemini is good for
• 1:00:57
  a quick summary like the the one from open AI like looks like
• 1:01:04
  there is some thought into it. There is a good thinking
• 1:01:07
  structural processes comes out of it.
• 1:01:09
  It's like the difference between, you know, like a senior
• 1:01:13
  analyst that just compiles sources and maybe like a
• 1:01:16
  consulting project manager or partner that actually think
• 1:01:20
  through the storyline. That's what with.
• 1:01:25
  A report also after the fact. You can ask for different
• 1:01:30
  formats to give you like a full report that you can tell him
• 1:01:35
  even hey I want a podcast out of it sty presentation.
• 1:01:41
  Actually I want to digest the information.
• 1:01:46
  That's that's interesting because this is well, if you
• 1:01:49
  want to ask deep research on the open AI side and send me the
• 1:01:53
  report it gives you, I would be happy because I I don't pay the
• 1:01:57
  200 bucks a month. Me the prompt, I'll I'll you can
• 1:02:01
  send me the prompt that you use for Gemini with Triton AI.
• 1:02:05
  You can send back the link to the result so you can see the
• 1:02:08
  link.