Latest / Agentic Conversations (formally mlops.community) / Design and Development Principles for LLMOps // Andy McMahon // #254

Transcript

Design and Development Principles for LLMOps // Andy McMahon // #254 · Agentic Conversations (formally mlops.community)

• 0:00
  So my name is Andy McMahon, Principal AI and ML OPS Engineer
• 0:05
  at Barclays, and I like a latte. What is up ML OPS community?
• 0:11
  We are back for another podcast. I'm your host Dimitrios and
• 0:15
  talking to Andy today. We went down so many rabbit
• 0:19
  holes. I had a blast.
• 0:21
  I highly respect this man. He is the person that I stole
• 0:27
  the definition of ML OPS. I think he puts it the most
• 0:31
  eloquently. We talk about it.
• 0:33
  We end his definition in this episode.
• 0:35
  You probably heard me say it a few times on this podcast.
• 0:39
  If you listened to any episodes before this because.
• 0:43
  Yeah. I shamelessly stole it.
• 0:45
  It's the best way of looking at ML OPS.
• 0:49
  He currently is working at Barclays Bank, which is a bank
• 0:56
  in the UK for anybody that doesn't know.
• 0:59
  And we went down rabbit holes on infrastructure, but we also went
• 1:04
  down rabbit holes on teams and how to function at a higher
• 1:08
  capacity, how to be a great leader and so many other things.
• 1:14
  I'll let you listen now and as always, if you enjoy, give it a
• 1:21
  like or drop in a comment, share it with one friend.
• 1:26
  And I don't have a song for you today to share with you.
• 1:30
  But while we rock into the conversation, but I will give
• 1:33
  you one thing for your recommender systems.
• 1:35
  And this was given to me by my lovely wife.
• 1:40
  I have to share it with the world.
• 1:43
  For anyone that likes tahin tahini, you know, sesame paste
• 1:47
  as some people call it, try black tahini.
• 1:52
  It is unbelievable. That's all we've got today.
• 1:55
  Let's get into the conversation with Andy, dude.
• 2:04
  So we're just talking about your book and how now it is Oxford
• 2:09
  material. It's on the curriculum for
• 2:11
  Oxford, right? Which is Oxford's no small feat.
• 2:15
  That's kind of one of the more popular universities in the
• 2:19
  world, I would say. Yeah, especially in the UK.
• 2:22
  Cambridge and Oxford are huge. But in the world, when you think
• 2:25
  about the top universities, I know that even me as an
• 2:29
  American, I knew about Oxford and and I wasn't that worldly
• 2:34
  before I moved abroad. And so that's really cool.
• 2:38
  Super congratulations to that. I mean, this is machine learning
• 2:41
  in Python. Is that the official title?
• 2:43
  Yeah, machine learning, engineering and Python.
• 2:46
  There we go. Yeah, it's the second edition.
• 2:48
  So it's been adopted as core material for the artificial
• 2:52
  intelligence Jenny I Cloud and ML OPS online course.
• 2:56
  So it's sort of a genuine professional development
• 3:00
  retraining, you know, a lot of a lot of existing engineers who
• 3:04
  want to upskill, but a lot of leaders and stuff.
• 3:06
  So I was a guest lecturer on the programme, but now they've
• 3:09
  adopted my book part of the sort of recommended reading and you
• 3:13
  get a copy of the ebook when you sign up to the course.
• 3:15
  So it's great. Also, then you you also
• 3:18
  mentioned that you're a guest lecturer somewhere else.
• 3:21
  Yeah, the University of Warsaw, so they have an EI for
• 3:24
  executives programme. So I work, I work there and
• 3:28
  present on two modules across sort of the ML development life
• 3:34
  cycle, Jenny I and LLM OPS sort of stuff.
• 3:38
  That's cool. So it's cool.
• 3:39
  Yeah, it's good. It's, it's different from the
• 3:41
  day job, you know, rather trying to build solutions and design
• 3:45
  and all those good things I like, but it's, it's, it's nice,
• 3:48
  it's nice to share your knowledge.
• 3:50
  And so the funny part was that you mentioned apparently 10 out
• 3:56
  of the 1st 50 open AI. Something like that, Yeah, I've
• 4:01
  probably missed know you've put me in the spot.
• 4:03
  I'm going to caveat and say I've legally misremembered.
• 4:05
  I was told that a significant proportion of initial openings,
• 4:10
  if you like, came through Warsaw.
• 4:12
  So Warsaw's got an excellent computing degree programme, I
• 4:15
  know that much. And yeah, they definitely had
• 4:18
  some of the early people in opening I came through.
• 4:21
  That's why Funny thing was basically those folks probably
• 4:27
  have left open AI already started their own thing and then
• 4:30
  got eaten up by big tech by or Aqua hired by big tech by now.
• 4:36
  So it is ridiculous how I'm seeing, I don't, I don't want to
• 4:43
  speak out of turn here because I imagine all these folks that are
• 4:46
  raising a tonne of money and trying as hard as they can to
• 4:50
  get a product out there in the Gen AI space, especially in the
• 4:53
  model space. And then they realise that
• 4:56
  things are a lot harder or something is not going to work.
• 5:00
  And they say, what's the escape hatch?
• 5:02
  Well, let's go and let's join big tech.
• 5:07
  And so it's that notion of failing upwards, but.
• 5:12
  Spicy. Man.
• 5:13
  That is spicy. I mean, I can't, I can't, I
• 5:17
  can't talk, right? I work for a bank.
• 5:21
  I sold out before I. Technically have a job so I
• 5:24
  don't know if I'm the best person to be talking shit
• 5:26
  either, but I look around and I'm like I could have grabbed
• 5:30
  the bag. I should have started a model
• 5:32
  company and raised 500 million and then after a few years of
• 5:37
  not releasing a product, been like now's the perfect time to
• 5:40
  go and join DeepMind. Well, don't do a model, but do
• 5:44
  do a do a rag company or something though.
• 5:47
  That's what is the next wave. You know, there'll be, there'll
• 5:49
  be another wave soon. Yeah.
• 5:50
  An ageing company, that ageing company.
• 5:53
  That's, that is the next one. That's absolutely, yeah, that is
• 5:56
  happening. That is happening everywhere.
• 5:57
  Do it, Can't miss it again man. That's my ticket to being a cog
• 6:03
  in the wheel at Big Tech, which is my lifelong dream.
• 6:09
  You want to fail outwards? Yeah, fail over that easy.
• 6:12
  That'd be great. OK, 20 seconds before we jump
• 6:14
  back into the show, we've got ACFP out right now for the Data
• 6:18
  Engineering for AI and ML virtual conference that's coming
• 6:22
  up on September 12th. If you think you've got
• 6:26
  something interesting to say around any of these topics, we
• 6:31
  would love to hear from you. Hit that link in the description
• 6:33
  and fill out the CFP. Some interesting topics that you
• 6:37
  might want to touch on could be ingestion, storage or analysis
• 6:42
  like data warehouse, reverse ETLSDBT techniques, etcetera,
• 6:45
  etcetera. Data for inference or training,
• 6:47
  AKA feature platforms if you're using them, how you using them,
• 6:50
  all that fun stuff. Data for ML, observability and
• 6:54
  anything fin OPS that has to do with the data platform.
• 6:57
  I love hearing about that. How you saving money?
• 6:59
  How you making money with your data?
• 7:02
  Let's get back into the show, dude.
• 7:05
  Well, OK, before we jump into what we came here to talk about,
• 7:09
  which is a little bit of ML OPS and what you like to call LLM
• 7:14
  OPS. I personally don't believe in
• 7:15
  that term because I have a vested interest obviously to not
• 7:20
  call it LLM OPS. I prefer the term AI
• 7:24
  infrastructure because I feel like it's more suited for what
• 7:28
  it is, but we can get into that in a second.
• 7:31
  What I wanted to ask you about is when you're given this AI for
• 7:34
  Executives course and you're talking about the life cycle,
• 7:37
  what is the life cycle? Yeah, it's really good.
• 7:41
  So I, I kind of agree with you. I think I used the term LLM OPS
• 7:46
  of necessity. I still view ML OPS as
• 7:49
  encompassing all of this and infrastructures are part of it,
• 7:53
  but there's lots of other things.
• 7:54
  But yeah, the life cycle is, I always think it's the software
• 7:58
  development life cycle with some extra pieces.
• 8:01
  And I've said this about ML OPS for a long time, right?
• 8:03
  And it was born of DevOps and all of these things still stand.
• 8:06
  It's just some extra pieces. So you still, you know, you
• 8:09
  still scope out your requirements, you still
• 8:12
  understand what you're going to do.
• 8:14
  You still go through sprints or scrum or however you're going to
• 8:17
  do it. But you go through a classic
• 8:19
  route to life of development, pre production or tests and
• 8:22
  through to production. As part of that you have to do
• 8:26
  testing and validation post deployment, you have to monitor
• 8:29
  and underpinning it all as infrastructure.
• 8:31
  You want to automate all those processes as much as possible.
• 8:35
  That's it in a nutshell. The challenges come because when
• 8:39
  it's not traditional software, it's machine learning or now
• 8:43
  Jenny I or AI, the whole question of you know, what is
• 8:48
  testing mean automatically has a much bigger answer.
• 8:51
  It's not. Just the pipeline.
• 8:54
  Exactly where does the data come from?
• 8:55
  Garbage in garbage out. What does monitoring mean?
• 8:58
  Yeah, all of those things have extra pieces fundamentally
• 9:02
  underpinning it. It's not it's not that different
• 9:04
  from software. And I think that's a big that's
• 9:07
  a big learning many organisations have had over the
• 9:10
  past few years. This sort of ML came on the
• 9:14
  scene and no Jenny I in the eyes coming on the scene.
• 9:18
  A lot of a lot of organisations may be raised to reinvent the
• 9:21
  wheel and then they actually discovered a lot of stuff could
• 9:24
  be leveraged that's out there. And then you could focus on the
• 9:26
  value added new pieces. And so so I think, I think, I
• 9:30
  think that's, that's a journey where we seem to be going
• 9:33
  through with Jenny I again, and I'm keen to accelerate that.
• 9:36
  I can see as sort of going back to the same place we were at
• 9:39
  with the 1st wave of data scientists are then deep
• 9:42
  learning, sort of starting to do that again, where we think it's
• 9:44
  all new, throw it all away. Yeah, you know, but actually
• 9:47
  it's just you're adding some pieces, you're adding some
• 9:50
  processes. Take a breather, take a beat,
• 9:53
  you feel. Like the and by the way, you are
• 9:58
  the person that I stole this from, which I still to this day
• 10:04
  will quote you every once in a while.
• 10:07
  ML OPS as a term. I like your definition of ML OPS
• 10:11
  the best out of any I've ever heard, which is going from north
• 10:16
  to 1 N plus one models. So you said it like, yeah, all
• 10:21
  right. Basically, the standardisation
• 10:25
  or the practise of ML OPS is when you're taking one model
• 10:29
  that is in production and scaling it up to N + 1 models.
• 10:34
  So it's it's exactly that. So I kind of, I likened it to
• 10:37
  proof by induction, this mathematical concept.
• 10:40
  But yeah, fundamentally it's like you say.
• 10:41
  So getting one model into production, the first model is
• 10:45
  always a big achievement for an organisation, A-Team company.
• 10:48
  They're always super excited. But that's sort of day zero, day
• 10:53
  one, day two is all I need to go back and do it again.
• 10:57
  And then what do I do? You know, I've got, I've got two
• 11:01
  models. Is the second one replacing the
• 11:04
  first one? Are they running in parallel?
• 11:06
  And obviously both of these are true.
• 11:08
  Eventually, eventually you have different products and services.
• 11:11
  They're all having champion challenger sort of logic going
• 11:14
  on. And that is exactly and that
• 11:17
  that totality of all that is is ML OPS to me, because it's it's
• 11:21
  very easy. What's not very easy, but I
• 11:23
  think it's easy to get one in production and monitor and
• 11:26
  maintain it. Just throw everyone on that when
• 11:29
  you start really thinking industrialising this process and
• 11:32
  you know, there's different analogies people talk about, you
• 11:35
  know, going through the sausage machine or whatever the conveyor
• 11:38
  belt. But but it's it's exactly that.
• 11:41
  And I think, I think like I said before, it's going from n = 1
• 11:44
  then equals two. And eventually you know how to
• 11:46
  manage what you've got, but you know how to increase the estate
• 11:49
  scalably. Yeah, that's the ideal state
• 11:52
  that you're at once Other piece on this, since you were
• 11:56
  mentioning Dev OPS and how and I fully agree, like ML OPS is
• 12:03
  heavily inspired from dev OPS. But I think what I've seen and I
• 12:09
  it's almost like every iteration it gets progressively worse in
• 12:13
  that thinking that DevOps is just tooling and it's just the
• 12:20
  tools that you use to accomplish these things.
• 12:23
  And then ML OPS, I saw it happen a lot and you probably saw it
• 12:26
  happen a tonne, especially in those basically 2020 to 2022 era
• 12:31
  where it was more conversations about the tooling and less about
• 12:35
  the processes and the cultural side of it.
• 12:38
  And IA 100% see that with the AI infrastructure space right now,
• 12:43
  where people just care about the tools and less about what is it
• 12:48
  that we're actually trying to do, what's the process, what is
• 12:52
  the culture and how we can make this be that N + 1 style of
• 12:58
  execution. That up up 100%.
• 13:02
  This is something I've seen for for years.
• 13:04
  As you said, it's still around, right?
• 13:06
  It's it's silver bullet thinking.
• 13:08
  It's sort of, you know, oh, if I just find the perfect tool or
• 13:11
  the perfect architecture, the perfect combination of tools,
• 13:15
  I'll be sorted. It doesn't work that way at all.
• 13:18
  I, I maintain I've said this in plenty of times and bullshit me
• 13:21
  for this, but you could do it any and all of this in EXO
• 13:25
  really, if you had a if you had an awesome process and really
• 13:29
  good people, right. And that's the point you're
• 13:31
  saying it's about processing culture first.
• 13:33
  Those are the sort of, and like you said, solving the real
• 13:35
  problems. So those are the 1st order
• 13:37
  problems and the sort of 2nd order or maybe a first order
• 13:41
  problem is, is data and you know, the underlying sort of the
• 13:46
  ecosystem. How, how rich is the soil you're
• 13:48
  sort of planting in 3rd or 4th order is tools.
• 13:53
  Absolutely. And like you can do so much with
• 13:54
  open source. You really, really can.
• 13:57
  And you, you should, I always encourage teams and people I
• 14:02
  speak to like push the boundaries of open source 1st
• 14:06
  and that then you then become what I call a kind of a more
• 14:09
  competent buyer or a more informed buyer because you, you
• 14:13
  can very much, you know, you get sales pitches and this is
• 14:15
  happening with the JDI stuff because it's so new.
• 14:18
  People blow you away with amazing decks and cool demos and
• 14:21
  everything's really slick. And you just think I need that.
• 14:24
  What you actually need is end products that work.
• 14:28
  You don't need that tool necessarily.
• 14:31
  So go and try it other ways 1st and then you find out where the
• 14:34
  gaps are because if you actually find out that you know, I can't
• 14:37
  get Jenny I working because provisioning AWS accounts is so
• 14:41
  hard for me. You don't need you don't need
• 14:44
  like the fanciest rag application as a as a service.
• 14:48
  Then you need something more in infrastructure state.
• 14:50
  You know, you can just just really, really do the
• 14:54
  groundwork, the homework, push the bones of what's available
• 14:56
  and open source and get back to open source if you're all doing
• 15:00
  that. Yeah, yeah, contribute back.
• 15:02
  That's, that's huge. That's so true.
• 15:05
  And even knowing and trying open source first will educate you as
• 15:12
  a buyer because you're going to know what features are actually
• 15:15
  valuable and what you wish that open source tool had or
• 15:19
  potentially was able to do that for some reason it's not able to
• 15:25
  do. And so then when you go on the
• 15:26
  market and you look for a tool, you're more capable of saying,
• 15:31
  let's find something with these requirements.
• 15:34
  Because if if everything is new in the beginning, you're going
• 15:38
  to not know what is really important for your use case.
• 15:42
  Yep, no, that, that that's it exactly.
• 15:45
  And the other thing as well is you, you really start
• 15:49
  understanding what the reality is.
• 15:52
  And this sort of links to what you're saying, but it's very
• 15:55
  easy from, you know, like a great example is like the demos
• 15:58
  that came out around, say, Devon and all the other stuff, you
• 16:01
  know, the auto rabbit auto developer.
• 16:03
  Yeah. And very quickly, I thought this
• 16:05
  was funny. Everyone said if they've
• 16:07
  automated development, why are they hiring developers?
• 16:10
  But that's that's a safe point. But like, that was a demo video.
• 16:14
  That was some, you know, snippets and all of this stuff,
• 16:17
  all, all of this hype. But really, if you're the people
• 16:20
  playing with, you know, Blank Chain or whatever it is and
• 16:23
  you're just, you know, all llama, you're playing around
• 16:26
  with this stuff, you go that would be incredibly difficult to
• 16:28
  do. And you sort of that's you
• 16:31
  becoming more informed right away to your point, because
• 16:33
  you're sort of you can smell, you can smell crap a bit easier
• 16:37
  and you understand where the sort of research activities are.
• 16:41
  So you know, for example, how hard it is to build agents that
• 16:46
  actually performed very well and robustly, right?
• 16:49
  So if someone comes in and says they've solved that problem, if
• 16:52
  you've played around with it, you're going to be, you're going
• 16:54
  to have a far higher bar placed on them before you.
• 16:58
  You give them your cash, which is which is so important, right?
• 17:00
  Because otherwise you're going to make crazy decisions.
• 17:03
  Yeah, which is easy to do. And expectations aren't going to
• 17:07
  be aligned. You're going to be thinking that
• 17:09
  you can do one thing and probably get a rude wake up call
• 17:12
  at some point in time. And the piece that you talk
• 17:16
  about too there is, I heard it put wonderfully yesterday.
• 17:21
  I think it's like a Hashicorp fundamental within the way that
• 17:26
  they build and it is something along the lines of work flows
• 17:33
  over technology. And so they really the whole
• 17:38
  philosophy there is saying all we care about is the end goal.
• 17:44
  However we get there, that can change a million different times
• 17:48
  and we can optimise it for the better as we find better tools
• 17:53
  out there or we recognise that we can build a piece of
• 17:56
  appointed solution that will service our needs better.
• 17:59
  But really the important thing is that end state that we are
• 18:05
  going for. So how we get there, it doesn't
• 18:08
  really matter. Yeah, yeah.
• 18:11
  And they, yeah, exactly the implementation details, yeah.
• 18:14
  Aren't the important piece necessarily so, so great.
• 18:17
  I get it kind of with I think about this in my head is when I
• 18:21
  see an architectural diagram, it's generally the case that I
• 18:24
  could swap all of the boxes for a competitor too, and it would
• 18:28
  still work exactly the same. And that's.
• 18:31
  Good. So I thought you were going to
• 18:32
  say I could swap all the boxes. Correct.
• 18:34
  Maybe I'm maybe going to have to retract my Excel statement,
• 18:36
  right. So do that to a point.
• 18:38
  But the point, the point roughly stands, right?
• 18:39
  Philosophically, yeah. But you know, if you're kind of,
• 18:42
  you know, if you're. Because if you're, and even at
• 18:44
  the level of the cloud provider or something, right, if I have
• 18:47
  to port from one to the other, functionally, they are mostly
• 18:51
  equivalent by necessity. And, and the increasingly a lot
• 18:55
  of tools are becoming interoperable.
• 18:57
  They're all, they're all sort of tending towards standard API
• 19:00
  structures and standard contracts.
• 19:02
  And that's all very positive. So it so it does mean that kind
• 19:04
  of, if you look at your architecture and the specific
• 19:09
  component that does like the ML pipeline piece, if you can, if
• 19:13
  you know you can confidently swap that out for one of the
• 19:16
  other tools, you've generally built the right workflow to the
• 19:20
  to the hashicor point. So I 100% agree with that.
• 19:23
  Where it becomes difficult, and this is me thinking with my
• 19:26
  banked sort of hat on a large, large existing organisations
• 19:31
  dance, Hey, big tech, they will have a lot of legacy solutions
• 19:34
  lying around. And sometimes that just means,
• 19:37
  you know, there is bespoke things that have to be built.
• 19:40
  But in general, the kind of the point of that still stands is
• 19:43
  like things should be interoperable.
• 19:46
  And that's how that's how now you can get best of breed
• 19:48
  architectures where you say I'm not going to go all in on one
• 19:51
  platform, I'm going to use multiple that all do things
• 19:56
  different things better. I'm going to have one thing for
• 19:58
  monitoring and one thing for scalable training and one thing
• 20:01
  for the engineering and that that's all very doable now.
• 20:04
  Yeah, I remember also. Back in the day I heard somebody
• 20:08
  talk about how they were hybrid cloud just to minimise the
• 20:12
  potential blast radius if they took down their whole AWS
• 20:16
  instance. At least they knew it would just
• 20:21
  be on AWS and they had their whole GCP instance that wouldn't
• 20:25
  be affected well. You you're laughing, right.
• 20:28
  But that that is that is policy in large organisations that that
• 20:32
  are not kind of a sort of technique of etcetera.
• 20:36
  That is the case. And so sometimes, like sometimes
• 20:39
  it is necessary if you like, because see a large financial
• 20:44
  institute, like a large bank, We're we're critical
• 20:47
  infrastructure for like the British economy.
• 20:50
  Yeah. So, so, so like it's good that
• 20:54
  we have people worrying about that in a sense.
• 20:57
  But but but it but it's true as well.
• 20:59
  Like you know it still stands what you said, the workflow
• 21:04
  should be able to work on Prem or in cloud.
• 21:06
  It's just different tools and components that do the specific
• 21:09
  tasks and you've. Inevitably been in a lot of
• 21:13
  rooms I can imagine where the ROI on switching tools or
• 21:20
  upgrading tools from legacy is being discussed.
• 21:24
  How do you generally go about that conversation?
• 21:27
  I think you're. Like you said, you're always
• 21:29
  focusing on the ROI and that that that becomes complex very
• 21:33
  quickly because you're not just thinking, you know, per unit
• 21:38
  cost of running queries or of storage or something.
• 21:41
  You're also thinking of total cost of ownership the entire
• 21:44
  life cycle. So I have to decommission this
• 21:46
  existing stuff. I have to spin up all this news.
• 21:49
  I have to maintain and run this new stuff after maybe up skill
• 21:54
  or train or new, new hire sort of comes into it as well.
• 21:59
  So it can become complex. But I think you always, you
• 22:03
  always have to think, yeah, what are the key components of the
• 22:06
  ROI calculation and what is the target state we want to aim for?
• 22:11
  And, and it's, it's sort of it's, it's always a balance
• 22:14
  between that short term benefit of maintaining things as they
• 22:19
  are versus, you know, you have to go through some pain to get
• 22:22
  to the real target state where the the benefits will drive
• 22:25
  through. But it's always complex.
• 22:27
  And sometimes the answer is not rip it up.
• 22:30
  Sometimes the answer is no, no, that that has to stay as is or
• 22:34
  it's not. It's not worth it.
• 22:35
  Yet is another answer. And when have you?
• 22:39
  Recognised that like how and because I can imagine there's
• 22:45
  it's like Hawks and doves in the government almost and it is
• 22:49
  sometimes. Yeah, it feels like there's.
• 22:51
  Probably a lot of a lot more Hawks than doves because
• 22:54
  engineers just like to get their hands dirty and the new stuff is
• 22:58
  always going to be cooler than the old stuff.
• 23:01
  So how have you like when have you thought about that just has
• 23:06
  to stay? And what were the arguments that
• 23:08
  won you over? Well, I think it's actually
• 23:11
  interesting you think there'll be more whole, I think in
• 23:14
  certain industries, heavily regulated industries, right,
• 23:17
  this, this applies maybe less in some jurisdictions, but like in
• 23:21
  Europe, yeah, if you're in finance and banking.
• 23:24
  Yeah, yeah, there's a lot of jobs.
• 23:26
  Yes, it's. Exactly, just by just by nature
• 23:28
  of the the org and that's, I think that's a that's a good
• 23:31
  thing in a way because like I said, we're, we're an important
• 23:34
  part of the economy. So we can't just go fail fast
• 23:38
  and break things, right. Doesn't apply when people need
• 23:40
  their money out of machines. And some cases it should apply
• 23:46
  and you should always innovate and move forward.
• 23:47
  That's important. But yeah, you don't, you can
• 23:49
  move at the pace of some big tech, right?
• 23:51
  That's just fundamentally true. The, the other aspect is that
• 23:57
  all related to that is like stability is really important.
• 23:59
  Trust is important. We are custodians of people's
• 24:02
  data. And it's super important data,
• 24:04
  right? It's, it's your, it's the stuff
• 24:06
  in your wallet. It's, it's your earnings.
• 24:08
  It's very, it's very personal information.
• 24:10
  That's all of these things about your life.
• 24:12
  So that's taken very seriously. So it's that that feeds through
• 24:15
  to there being many doves. There are, there are lots of
• 24:19
  cases where you put on a hot cat and you think this is crazy, we
• 24:22
  need to rip up. But I think the case is where
• 24:24
  it's prevailed. And I won't go into details
• 24:26
  necessarily, but there are just core pieces of banking
• 24:29
  infrastructure that have been around for decades.
• 24:32
  And I, I am not sure when they will be sort of totally holds
• 24:36
  and re revamped. And in a, in a way it's kind of
• 24:40
  like, you know, the systems keep working.
• 24:44
  So let's not, let's not poke it too much anyway, which, which
• 24:49
  may sound scary to some people, but actually it's, it's, it's
• 24:52
  good because they are well maintained.
• 24:54
  They are well looked after. They, they do have really strong
• 24:57
  kind of governance and programmes around them.
• 25:00
  But yeah, there's, there's actually, I would say it's more
• 25:02
  dovish than hawkish. The key challenge is bringing in
• 25:05
  a balance. So they innovate and you don't,
• 25:08
  you don't kind of get killed by your competitors, but it's a
• 25:11
  hard balance, right? And there are, there are like
• 25:14
  upcoming sort of fintechs and new starter banks and stuff that
• 25:20
  are challenger banks as they're called.
• 25:22
  They're a bit more nimble and lighter, but at the same time
• 25:25
  they're getting started out. I'm sure once they become the
• 25:28
  scale of like a huge bank, they'll feel.
• 25:31
  Different they'll have. To yeah, you, you know you can.
• 25:33
  You'll join. You'll join us.
• 25:36
  Live long enough to become the villain, as they say in The Dark
• 25:38
  Knight, right? Exactly.
• 25:40
  I, I remember we had think her name was Michelle from Lloyd's
• 25:45
  on here. And she was talking about how
• 25:47
  they were going through the migration to cloud and it was a
• 25:53
  $3 billion project. And it was almost, I think she
• 25:58
  said they budgeted like six or seven years for the full
• 26:04
  migration. And so that type of stuff, it's,
• 26:08
  it's like when I was in Portugal and I was getting a tour of
• 26:13
  Lisbon and someone told me, yeah, that building took 250
• 26:18
  years to be created and the, the 1st and 2nd generation didn't
• 26:24
  even live to see it be finished. And I'm kind of like looking at
• 26:29
  that Lloyd's. Building the Great Wall or
• 26:33
  something? Yeah, yeah, you're a bit, you're
• 26:34
  definitely a building cathedrals.
• 26:36
  That's sort of how I would think of it.
• 26:37
  But but that's again, like when you break it down, it's actually
• 26:42
  made of much smaller. When when you hear that, you
• 26:45
  think, why would I ever, why would I ever go somewhere where
• 26:48
  I do that? But it's not that it's not kind
• 26:50
  of it's not the case that you're just, you know, sitting around
• 26:54
  doing the tiniest iteration. What happens is more you divide
• 26:58
  and conquer, right. So you say, is one, one part of
• 27:00
  the bank or one set of data, we'll migrate first and then
• 27:03
  we'll derive lots of value and we'll build ML OPS and all the
• 27:05
  infrastructure we need for data science, etcetera there while
• 27:08
  the hydration is continuing in other parts of life.
• 27:10
  So I, I was kind of speaking to someone about this recently.
• 27:14
  When you're in an environment where the end to end is a bit
• 27:17
  slower or those kind of more steps, what happens is you
• 27:21
  rather than vertical scaling your horizontal scale to borrow
• 27:24
  a kind of infrastructure term, right?
• 27:26
  So what you do is you end up running lots of things in
• 27:29
  parallel and driving value that way.
• 27:31
  And then they they all start landing in a sort of
• 27:34
  consequential fashion and get you to where you need to be.
• 27:37
  Whereas there may be a more startup focused worlds.
• 27:39
  When I worked in the much smaller companies, you know,
• 27:41
  like a startup, it was more, you know, you had a job do it end to
• 27:44
  end, move on to the next one. Now it's more, we have slightly
• 27:48
  longer pieces of work, but let's horizontally stack them and
• 27:51
  spread our time between them and keep moving all of them forward.
• 27:55
  Yeah. So it's just, it's just
• 27:56
  different scale. But then when things land, it's
• 27:59
  massive impact, right? Millions of customers, people,
• 28:04
  you know, we're catching fraud better.
• 28:06
  We're literally protecting people's livelihoods and their
• 28:08
  money. You're, you know, when I
• 28:11
  remember working on projects before NatWest, it was like
• 28:14
  you're breaking up slave, slave rings that, you know, modern
• 28:18
  slavery and all these things. There's all these amazing things
• 28:20
  that come through banking technology, right?
• 28:23
  Yeah, it's easy to sort of this to miss, dismiss and forget, but
• 28:27
  it's like it's, it's critically important and it's, it's very
• 28:30
  real for people. It was the first time I worked
• 28:33
  in an organisation where everyone understood what I sort
• 28:36
  of did. I could say, you know that, you
• 28:37
  know, that retail app in your, your phone where you do your
• 28:40
  banking, we're protecting that and looking at the day and oh, I
• 28:43
  get that. Whereas before when I worked in
• 28:45
  say energy or logistics that I don't understand.
• 28:48
  What do you, what do you mean? What do you mean you're doing a
• 28:49
  routing problem? What's the diagrams and white
• 28:52
  boards? Whereas it's very, it's very
• 28:54
  visceral and it's really direct impact on customers, which I
• 28:57
  really like. That's why I've kind of stayed
• 28:59
  and financed these past few. All right, real quick.
• 29:01
  Question for you, are you a Microsoft Fabric user?
• 29:04
  Well if you are, you are in luck because we are introducing SAS
• 29:08
  Decision Builder. It's a decision intelligence
• 29:11
  solution that is so good it makes your models useful
• 29:16
  because, let's face it, your data means nothing unless you
• 29:18
  use it to drive business outcomes.
• 29:21
  It's something we say time and time again on this very show.
• 29:24
  But wait, what do you mean by nothing?
• 29:28
  Well, SAS Decision Builder integrates seamlessly with
• 29:34
  Microsoft Fabric to create effortless business intelligent
• 29:37
  flows. It's like having a team of
• 29:39
  geniuses you manage in your pocket without all that awkward
• 29:44
  small talk. With Decision Builder, you'll
• 29:46
  turn data into insights faster than brewing a double espresso.
• 29:51
  And you know how much we like coffee on this show?
• 29:53
  Visually construct decision strategies, align your business,
• 29:57
  and call external language models.
• 30:00
  Leverage Decision Builder to intuitively flex your data
• 30:04
  models and other capabilities at breakneck speeds.
• 30:08
  There's use cases for every industry, including finance,
• 30:12
  retail, education, and customer service.
• 30:14
  So stop making decisions in the dark.
• 30:17
  Turn the lights on with SAS Decision Builder.
• 30:20
  Yes, I did just make that joke. Watch your business shine, SAS
• 30:25
  Decision Builder, because your business deserves better than
• 30:28
  guesswork. Want to be one of the first to
• 30:31
  experience the fusion? Want to be one of the first to
• 30:35
  experience the future of decisions?
• 30:37
  Well, sign up now for our exclusive preview.
• 30:42
  Visit sas.com/fabric or click the link below.
• 30:48
  So talk to me. About how you look at the
• 30:54
  addition of. AI.
• 30:59
  To the ML work flows. And Yep, when you're thinking
• 31:03
  about a platform, are you thinking about the ML platform
• 31:08
  also servicing the AI needs or is it?
• 31:11
  No, you're thinking about these two are separate or the AI part
• 31:16
  is Greenfield and we can move faster if we just start from
• 31:19
  scratch, Yeah. What is that?
• 31:22
  Good. Good question.
• 31:23
  So the way I, I try and think of it more as a sort of you're
• 31:28
  providing an ecosystem in which the organisation can first of
• 31:34
  all play an experiments and then eventually build products and
• 31:37
  services. But you're sort of, it's not,
• 31:39
  it's not strict in my mind as like this is the platform for
• 31:42
  this, this the platform for this.
• 31:43
  So until you slice and dice it, but it's more you're creating an
• 31:47
  ecosystem of capabilities that people can tap into the sort of
• 31:51
  tools on the shelf. And then it's up to the business
• 31:54
  based on the requirements they have for customers, for
• 31:57
  colleagues, etcetera, what they need to build and then they can
• 32:00
  go to the shelf. So the key thing is building the
• 32:02
  right stuff to put on that shelf for them to use or enabling it.
• 32:06
  So why can I think about a lot? As you know, the traditional AML
• 32:09
  use cases are not going anywhere.
• 32:12
  They're not. They're going to be around
• 32:13
  forever, right? The thing that's going to
• 32:15
  outlast all of this is sequel, right?
• 32:17
  But let's not go there. You know, the longer that rule,
• 32:19
  I can't remember it, but the longer technology has been
• 32:21
  around, the longer it will be around.
• 32:23
  Yeah. So like you think like sequel
• 32:25
  has been around forever. It will be around forever.
• 32:27
  ML is catching up. It's going to.
• 32:30
  It's going to be there alone. And then.
• 32:32
  And then you've got AI. The way I think about it is,
• 32:36
  yeah, are you, are you servicing the capability?
• 32:38
  Are you giving people places where they can access the data,
• 32:42
  you know, do their exploratory analysis?
• 32:44
  Have you got development workflows where they can build
• 32:48
  machine learning modules and the pipelines that those set in?
• 32:51
  Can they orchestrate them and can they monitor?
• 32:54
  A lot of that thinking still applies to the AI world.
• 32:58
  The big difference now is obviously you don't own the
• 33:01
  models. So.
• 33:03
  The focus of the life cycle is a bit different.
• 33:05
  You're consuming models as a commodity or a service, like
• 33:09
  keep talking about models as a service and or, or as a
• 33:12
  commodity. So you're kind of you're just
• 33:14
  going and purchase the use of renting, the use of these models
• 33:17
  that someone else has built in most cases, the vast majority of
• 33:20
  cases. And that's just a different
• 33:22
  setup. So what it means is the
• 33:24
  traditional ML OPS life cycle was far more focused on, you
• 33:27
  know, how you're going to train, how you're going to track all of
• 33:31
  your experiments when you're training, how are you going to
• 33:33
  optimise the hyper parameters and how are you going to then
• 33:37
  obviously validate all of those runs in a sensible way that
• 33:41
  translates to production setting.
• 33:44
  Now you're not training. You might be fine tuning a
• 33:46
  little bit, but even then that's not more shoes cases.
• 33:49
  What you're probably doing is actually saying now I'm
• 33:52
  building, you know, a vector database, I'm chunking and
• 33:55
  indexing data in that database and I'm building my my chains or
• 34:00
  my flows around how I interact with the LLM to get the best,
• 34:03
  the best usage of that data. Totally.
• 34:06
  But you're still doing the monitoring piece, you're still
• 34:09
  doing evaluation and you're still going through those those
• 34:13
  sort of stage gates that you would before.
• 34:15
  They're just sort of different tools on the shelf again.
• 34:18
  So I kind of you as yeah, you're just, you're just building out
• 34:20
  this ecosystem and people can tap into it when and where they
• 34:22
  need. The kind of interesting case
• 34:24
  that I think hasn't came up yet, but well is when people want to
• 34:28
  do very hybrid solutions, they're both traditional ML and
• 34:32
  AI. So things like function calling
• 34:35
  and more agentic work flows, right?
• 34:37
  That's going to be really interesting because what you're
• 34:38
  going to have is like AI will be the key backbone and even part
• 34:42
  of the orchestration framework almost if it's doing function
• 34:45
  calling, but it may call in to your proprietary models that
• 34:48
  you've built. So I can imagine a world where
• 34:51
  we say to an LLM, run this analysis and use use my
• 34:55
  existing, you know, fraud detection models.
• 34:58
  And what it does is it hooks in your proprietary models, pulls
• 35:02
  in the answer and synthesises it with other models, comes up with
• 35:04
  an answer. Yeah.
• 35:06
  And you know, and they kind of, there's going to be a really
• 35:08
  interesting question there about how do you expose your models as
• 35:12
  services that can be consumed across the organisation?
• 35:15
  How are you going to, again, validate that?
• 35:18
  Because that becomes really complex very quickly.
• 35:20
  But that's the sort of thing I'm thinking towards by.
• 35:22
  Yeah, I think of it as a it's an ecosystem.
• 35:24
  So you could view it as all one platform or you could view it as
• 35:27
  just a collection of sort of buckets of technology.
• 35:30
  And the key thing is telling people to your point earlier was
• 35:33
  the workflow and process. For hooking these things.
• 35:36
  Together to build a solution, yeah.
• 35:39
  Making sure that's standardised so that all teams can access
• 35:43
  whatever they need to whenever they need to.
• 35:46
  And yeah, this and the idea of the hybrid AI plus traditional
• 35:51
  ML is fascinating to me too. I think about.
• 35:56
  Some. Things that I've seen, I think
• 35:59
  and I can't quite put my finger on how it works now, but I know
• 36:04
  that we have had people come on to the LLMS in production
• 36:10
  virtual conference and talk about recommender systems and
• 36:14
  using LLMS in the recommender systems.
• 36:17
  There's the very easy way to do that is just asking an LLM to
• 36:22
  recommend things or saying I'm X type of person recommend
• 36:26
  something to me. So that's.
• 36:29
  Like a. A very basic LLM recommender
• 36:32
  system type thing that doesn't account for recommender systems
• 36:38
  wanting to be very fast and the other pieces that most people
• 36:43
  aren't just looking at. I'm X type of person recommend
• 36:48
  where I should go on vacation. So there's like yesterday I was
• 36:52
  talking to a guy who was who was mentioning all these
• 36:56
  recommendations, job recommendations and all the
• 36:59
  features that they're using to create those recommender models.
• 37:03
  And then how they need to make sure that when they're serving
• 37:06
  it, it's super low latency and that you don't quite get with
• 37:10
  LLMS yet. Maybe can figure out a way to
• 37:14
  prune or make it super, super small models and get something.
• 37:17
  But I do think there is a world where we start to see more LLMS
• 37:23
  incorporated into recommender workflows.
• 37:26
  So I've heard people talk about how they use LLMS to get
• 37:32
  suggested features that might work.
• 37:36
  And. So that that's another cool one.
• 37:39
  It's a little bit before the training happens, right?
• 37:42
  Or it's a it's a little bit I would say like on the other
• 37:45
  side. Yeah, the.
• 37:47
  The serving. But but it's cool and the the
• 37:51
  last thing I'll say about what you're talking about with the
• 37:53
  ecosystem, which is a great notion of how to think about it
• 37:57
  instead of thinking, hey, there's this platform.
• 38:01
  That. You plug into here if you're an
• 38:04
  ML engineer and you plug into here if you want to do AI.
• 38:07
  It's more of you've got all these tools, you string them
• 38:11
  together however you want because you're the creative
• 38:13
  artist. I'm just giving you white paint,
• 38:16
  black paint, a few paint brushes, and then you make the
• 38:20
  masterpiece. And the fun part now, I think
• 38:24
  with AI and having these models exposed or having these models
• 38:29
  as a commodity as you were mentioning, is that you don't
• 38:32
  have to be an ML engineer, you don't have to be a data
• 38:35
  scientist to get the value, if you can figure out how to make
• 38:41
  it work from these models that are models as commodities,
• 38:45
  right? Yeah, exactly.
• 38:47
  So you're going to you're going to have a kind of stratified
• 38:49
  strategy, stratified strategy. But basically, yeah, you're
• 38:53
  going to have sort of one tier of applications is exactly that.
• 38:57
  It's sort of baked into some vendor offering or some existing
• 39:00
  solution you use, right. So Co pilots are popping up
• 39:03
  everywhere. Those those LLMS be baked into
• 39:06
  so many existing software products that we use day-to-day.
• 39:09
  So, you know, in my e-mail, in my word processor, in in my IDE
• 39:14
  for coding, there's going to be Co pilots there that you can
• 39:17
  they can use and like you say, I'm not building an application
• 39:20
  that does that. I'm just using it.
• 39:23
  Then there's sort of slightly lower level.
• 39:24
  There's things, you know, that are a bit more GUI driven, drag
• 39:28
  and drop, push some buttons and you know, like Microsoft Co
• 39:32
  pilot studio as an example, sort of it's big game toward you're
• 39:35
  still building something yourself, but it's, you don't
• 39:37
  really have to be a deep technologist to do that.
• 39:39
  And then there's the level of, you know what, no, we have to
• 39:42
  have a lot of control here, but to really own everything end to
• 39:45
  end, we have to, we want to build very bespoke components or
• 39:49
  string them together in a certain way.
• 39:50
  And that's when you get to like the EI and ML engineer.
• 39:54
  And I think I think that will that will work together as long
• 39:57
  as people are clear. But you said something very
• 40:00
  important earlier, which is about being driven by the actual
• 40:02
  needs of the organisation or of your customers.
• 40:07
  The danger I sort of worry about sometimes is we shouldn't just
• 40:11
  do Jenny I to do Jenny I just like before, we shouldn't have
• 40:14
  done deep learning to do deep learning.
• 40:16
  We shouldn't have done data science to do data science.
• 40:18
  We should really spend time as organisations really
• 40:23
  understanding, you know, where does this drive value?
• 40:25
  Because it will drive value. It'll absolutely drive value.
• 40:28
  It may not drive value to the tune of trillions of dollars and
• 40:30
  stuff that's been thrown around. I'm a bit sceptical of that, but
• 40:33
  it will drive value. It's just about being very clear
• 40:35
  on, you know, here's the need for that.
• 40:37
  And then there's an education piece that comes as well because
• 40:40
  now everyone is held of Jenny I, but many people have not still
• 40:44
  heard of traditional ML. So, you know, there's a lot of
• 40:47
  people may be thinking, oh, I I could use, I've seen things like
• 40:51
  people say I could use Jenny I to classify something.
• 40:55
  Technically, yes. However you could use logistic
• 40:58
  regression. Sure, you know, probably cheaper
• 41:01
  and faster. It'll be cheaper.
• 41:02
  Faster and far more controlled than again, in a regulated
• 41:05
  industry. Sometimes that's the kicker is
• 41:08
  like actually I have full control and auditability of
• 41:12
  this, whereas otherwise that's a black box and I know this thing
• 41:16
  is going to perform. It's far more deterministic,
• 41:19
  still statistical, but it's more deterministic than a genie I'm
• 41:22
  and I completely own it. You know, my little logistic
• 41:25
  regression is going to give me the same prediction today,
• 41:28
  tomorrow and next year based on the same inputs.
• 41:31
  Some genie I models won't if they're provided by third party
• 41:34
  providers and still doing pre training, etcetera.
• 41:36
  So those are important considerations.
• 41:38
  So it there's a kind of education piece that has to
• 41:40
  happen as well, where people were, you know, they they know
• 41:43
  that playing around with an LLM, they can do certain things.
• 41:46
  And then you say, actually, there are other ways to do this.
• 41:49
  The recommender examples brilliant.
• 41:51
  The naive way would scale, yet it'll be super expensive.
• 41:55
  It doesn't probably make sense in so many ways, but maybe you
• 41:58
  use a traditional recommendation engine and you get some of the
• 42:01
  first end recommendations and use an LM to synthesise a
• 42:04
  response is here and say if you're feeling this way, do this
• 42:07
  if you're feel you know there's kind of there's you can do lots
• 42:09
  of different combinations, but yeah, it's just that hard
• 42:12
  thinking still has to be done around what product should we
• 42:14
  actually build it? Yeah, that's.
• 42:17
  Why? That's exactly why I felt like
• 42:20
  the product managers are the ones who are really crucial at
• 42:26
  this point in time, Yes, because we've both seen so many products
• 42:34
  come out that are doing Gen AI just to do Gen AI.
• 42:37
  And do you see it and then you notice it.
• 42:39
  And I remember being at a hack a Thon a few months ago and people
• 42:44
  were telling me how they were doing this cool thing with the
• 42:48
  LLMS and it was an LLM hack a Thon.
• 42:50
  So of course they're trying to figure out ways to use LLMS. But
• 42:55
  then I asked at one point when I was, I was one of the judges and
• 42:59
  going around asking and hearing the pitches and I asked, so why
• 43:03
  can't you do this with just even just like regex?
• 43:09
  And that's true. They said, they literally said
• 43:12
  to me like, oh, that's the cool part.
• 43:14
  You can. I was like, well then just do
• 43:17
  it. Yeah, yeah.
• 43:18
  I can't remember who said this, but I wish I'd known if if
• 43:21
  someone knows, please tell Dimitrios.
• 43:22
  I know, but someone said this before they were like, yeah,
• 43:26
  you're using, you're using a literal supercomputer to
• 43:29
  reinvent regex. Like like think about what
• 43:32
  you're doing, like just learn some regex or learn some sequel.
• 43:35
  Now there's there are cases where you know, like text to
• 43:38
  sequel is an interesting one, right?
• 43:40
  I've seen a lot of debate about this and there's some people I
• 43:43
  follow and things are like, you should just learn sequel.
• 43:45
  But on the other hand, if you're a non-technical user and you
• 43:47
  need some information from a database, text to SQL is going
• 43:50
  to be super useful. Yeah.
• 43:52
  If you do know SQL, however, the sort of the precision control
• 43:55
  that gives you it may end up not being worth your time to go text
• 43:59
  to SQL. If you you know it's the, it's
• 44:00
  the same with the coding stuff. Like if you know nothing, like,
• 44:04
  you know, just get an LLM to write it and maybe it's broken
• 44:06
  and then ask it to fix it and you'll get somewhere and you'll
• 44:09
  learn something. But if you know a little bit
• 44:11
  you're not going to get, you're not going to get it to do
• 44:13
  everything for you. It's like having really helpful
• 44:16
  assistance Ranger, you're not going to get them to do
• 44:18
  absolutely everything because what's the point of view You're
• 44:21
  you're going to like offload tasks to them and delegate and
• 44:24
  manage. But I totally agree with your
• 44:26
  point on product. Really good product managers and
• 44:29
  people are thinking of product mindset more generally.
• 44:32
  That's that's going to be the key differentiator for teams and
• 44:36
  organisations moving forward. Yeah.
• 44:38
  Folks who understand what technology they now have at
• 44:41
  their fingertips, but they're not so hawkish as we were saying
• 44:45
  before, to just stuff it into everything and that that's worth
• 44:50
  its weight in gold. OK, dude.
• 44:52
  So going back to the idea of architecture and Gen AI and
• 44:57
  having this ecosystem of toys that you can play with and
• 45:01
  string together, I think the tools that you need are so
• 45:06
  similar but just a little bit different.
• 45:09
  So I don't know how bullish you are on trying to recycle tools
• 45:15
  from traditional ML into the Gen AMI space versus we probably
• 45:21
  should have a whole new tool for that.
• 45:23
  Yeah, I mean you, you definitely have new, new tools and new
• 45:27
  capabilities. It's just I sort of I put them
• 45:30
  on the shelf with all the other things as I was saying.
• 45:32
  So, so for example, you know, vector databases, if you're
• 45:37
  really, if you're going to go hard and build your own RAG
• 45:39
  stuff, which plenty of organisations will be doing,
• 45:43
  that wasn't really part of the classic data science ML engineer
• 45:46
  bag of tools unless you were in, it was if you were, we're
• 45:50
  talking about recommendation engine.
• 45:51
  They were using vector embeddings for a long time and
• 45:55
  not in the same way we do for RAG, but we're getting towards
• 45:57
  that, right? And that that was there.
• 45:59
  But in general, for most air science teams, they went sort of
• 46:03
  thinking about PG vector and Chroma and all these tools.
• 46:05
  They just went, now that is part of the language and part of the,
• 46:09
  you know, the toolkit they need and expect.
• 46:11
  So you need to enable that. The other thing is obviously
• 46:15
  just the providers of the models, they all have different
• 46:19
  sort of selves. So the different cloud providers
• 46:22
  the they have their models and they provide surrounding sort of
• 46:26
  ancillary products. They provide agent frameworks,
• 46:28
  they provide guardrails, they provide out-of-the-box databases
• 46:32
  and you just need to educate people and you know how much you
• 46:36
  want to consume that versus just consume the model and build some
• 46:39
  of the other stuff. And it, it's just a sort of,
• 46:41
  it's always a balance and it's always a, it's always a balance.
• 46:46
  It's just like when we migrate to the cloud, the sort of how
• 46:48
  far up the abstraction sort of ladder do you want to go?
• 46:53
  You know, you can go all the way to just consuming SAS.
• 46:56
  So I'm just going to use ChatGPT perplexity.
• 46:59
  I'm just going to use MMS 365 Co pilot, whatever.
• 47:02
  I'm never going to care about what's happening in the
• 47:04
  background. That's again, one tier, right?
• 47:07
  You go a bit lower and you say actually I'm going to build sort
• 47:09
  of my own front end and who can to an LLM provider.
• 47:12
  And then you eventually say I'm going to build my own Vector DBS
• 47:15
  and do my own chunking strategies.
• 47:16
  So then you get all the way down to, you know, some crazy people
• 47:19
  will be saying I'm going to compile Llama CPP and run it on
• 47:22
  a bare metal server that you have running in my room or
• 47:24
  something, you know, and that that's all fine.
• 47:27
  It's just being clear how far up and down that ladder you want to
• 47:30
  go, what the benefits are. So obviously you get more
• 47:33
  abstracted. It's easier to use, but often
• 47:35
  costs more and then you go down the ladder.
• 47:38
  It can be a bit cheaper, but you have more work to do.
• 47:40
  Yeah. And you need specific skills
• 47:42
  yeah, they all have the place, but it's just it's just sort of
• 47:45
  doing it. But yeah, the two the two the
• 47:46
  tools are definitely different and I can specifically.
• 47:51
  Was thinking about evaluation and how similar that is to
• 47:55
  monitoring, but how it's like not enough to where you can and
• 48:02
  when you're thinking about OK, now I I'm collecting a golden
• 48:06
  data set and training my model with that how similar that is to
• 48:12
  all right, I have my golden data set for what I'm going to
• 48:15
  evaluate the models output on Yeah, that.
• 48:18
  That, that's a very good point. So I was, I was talking about
• 48:21
  this to colleagues earlier, the sort of the challenges exactly
• 48:23
  like you say in traditional MO for supervised learning
• 48:27
  especially I had ground truths. I will get the ground truth
• 48:31
  eventually. I'll be able to calculate my
• 48:33
  performance if I do that in a schedule or event driven way,
• 48:37
  that's me doing monitoring point of time of values and monitoring
• 48:39
  were very similar for a lot of LLM or foundation mod or Jenny I
• 48:45
  applications. Ground truth is a slippery
• 48:48
  concept and that's not really what you're after right.
• 48:50
  If I if I'm if I'm preventing a chat boy experience, there's no
• 48:53
  such thing as ground truth. There might be for some specific
• 48:56
  aspects like if I do a rag thing and then I can start talking
• 48:59
  about retrieval precision and all of these metrics that that
• 49:04
  makes sense and I can apply some metrics blue and Rouge and all
• 49:07
  these these newer thing. But yeah, it's exactly that.
• 49:10
  And you have you have the kind of challenge of great truth.
• 49:12
  It's a bit of a slippery concept.
• 49:14
  How much do I need human evaluation of human in the loop,
• 49:17
  human on the loop? How scalable is that really?
• 49:20
  Yeah, how? And then there's obviously LLM
• 49:22
  as a judge using LLMS to evaluate other LLMS, which the
• 49:26
  first time people hear that they're like Inception, but
• 49:28
  actually makes a lot of sense because it's a sort of
• 49:32
  independent layer of verification.
• 49:34
  And you can have models that are trained specifically to look for
• 49:38
  certain things. Like Llama Guard was one, yeah.
• 49:40
  It was trained specifically to, you know, detect toxicity or, or
• 49:45
  things going outside of specific guard troops.
• 49:48
  And guardrails in general are often using LLMS though.
• 49:51
  So yeah, I think you're totally right.
• 49:53
  The, the whole evaluation and monitoring piece is, is
• 49:55
  definitely a new, a new challenge and we need to apply
• 49:59
  some of the new metrics that we have for specific use cases.
• 50:03
  We should also be thinking really hard with that product
• 50:06
  product mindset, what are the metrics we actually care about?
• 50:09
  So a great example I always think about is when I learned
• 50:12
  about containment rates for chat bots, right?
• 50:15
  So there's a metric you can calculate that basically says
• 50:18
  how long until you demand to speak to a human and that's your
• 50:22
  containment rate, right? No, but that, but that's, that's
• 50:25
  a good metric of performance. Yes, but I also wonder.
• 50:29
  Can you can you put right next to that the language of the
• 50:34
  remained? Like how sentiment.
• 50:36
  You could, you can live track sentiment, right?
• 50:38
  Nobody people do that, right. They they say how long do I
• 50:41
  contain them working with the ball?
• 50:43
  Do I successfully complete the task and what's the sentiment of
• 50:46
  the conversation? Yeah, either in summary or
• 50:48
  through time, if you're kind of if you're able to do that.
• 50:52
  But those are kind of examples of proxy metrics, but they're
• 50:55
  far more aligned to your, your business outcome or your your
• 50:58
  desired product outcome, right. They're not vanity metrics,
• 51:02
  which I've been fighting against my entire career.
• 51:05
  You know, you almost don't care about, you know, first.
• 51:08
  Well, you do care on some aspect, but it's like time to
• 51:11
  1st token or I go that I go that from .1 second to .09 seconds,
• 51:16
  right? Not that big a deal.
• 51:18
  However, everyone is dumping over my chatbot after three 3
• 51:22
  lines because the chatbot just doesn't have a clue what's going
• 51:25
  on. That's the metric you care
• 51:27
  about, right? I don't care that like the first
• 51:30
  token came really fast, but it was the rest of the tokens were
• 51:32
  absolute garbage. So you just need to be clear,
• 51:34
  you know, what's a vanity metric and what's what's a real metric
• 51:37
  that you're looking after. And that's the site.
• 51:39
  That's the sake where I know. But yeah, you absolutely can't
• 51:41
  take existing monitoring tools and just plug them in for LLMS.
• 51:45
  You have to have to think. You still build pipelines and
• 51:48
  you know, do the calculations and calculate these metrics, but
• 51:52
  it is different. So I do, I do very much take
• 51:54
  that point. It's not plug and play.
• 51:56
  Wait, what are? Some other vanity metrics
• 51:58
  because that is such a good one. The time to 1st token is
• 52:01
  something that you should be looking at further down the
• 52:05
  field. Makes 100% sense.
• 52:09
  I'm sure you have seen a lot of other ones.
• 52:13
  Yeah, so, well, I got this from the book Lean Staro by Eric
• 52:17
  Rice, which is a classic, Absolutely brilliant.
• 52:20
  You know, it's classic things like hits on a web page.
• 52:25
  You don't really care about hits on a web page if you're selling
• 52:27
  something, it's more how much product that they sell.
• 52:30
  Yeah, things like that. So when it, when it comes to ML
• 52:33
  models, like I don't care how many productions you run really.
• 52:38
  You know, I care about like, why were you running the production
• 52:40
  in the 1st place or to sell more mortgages or something.
• 52:44
  Well, how many more? Just just so you know, it's kind
• 52:46
  of it's just things, it's just asking that next question.
• 52:49
  You know, I think about things like the Toyota 5 YS, Toyota
• 52:53
  this it's five YS to diagnose issues.
• 52:55
  But sometimes I just like asking why lots of times anyway,
• 52:57
  because then you get to the real heart of the problem.
• 53:00
  So, yeah, things like that, things like no latency and all
• 53:04
  these things are important, right?
• 53:06
  Because if if things take minutes to load, that's a
• 53:08
  terrible experience. Yeah.
• 53:09
  But it's just making sure that's not the only thing you're driven
• 53:12
  by because once they're above a cent level is a vanity project
• 53:16
  making them better. It's more what's what's the core
• 53:18
  offering you're trying to you're trying to deliver like?
• 53:21
  The there's certain things that need to be within a bound, but
• 53:25
  when you hit that, you really need to be looking at the most
• 53:29
  important metrics and if those other things go outside of.
• 53:33
  Their. Bound.
• 53:35
  Then you should look at them. But for the most part, yeah, the
• 53:42
  how many mortgages are you selling?
• 53:44
  Is a very, very clear way of being able to say this is
• 53:49
  valuable or this is not. Exactly.
• 53:52
  And it and it, it speaks, speaks across the technology business
• 53:55
  divide, which still sort of exists.
• 53:57
  I wonder it's a bit more a bit more subtle, but relates to this
• 54:01
  is I'm a big fan of Dora metrics, so the DevOps kind of
• 54:04
  industry standards metrics. And so these are like, I won't
• 54:08
  remember them all, but change failure rate, how often does a
• 54:11
  change to production fail time from a change to production
• 54:15
  deployment frequency, all these things, right?
• 54:16
  So they're basically a set of four metrics that are the
• 54:19
  standards in industry for DevOps maturity.
• 54:23
  And it's really interesting because I brought in previous
• 54:26
  organisations, those for ML OPS and said they're important part
• 54:29
  of ML OPS, but you can play the vanity metric game there because
• 54:34
  in some organisations, you might say our change failure rate is
• 54:37
  0. What's your deployment frequency
• 54:41
  though? Or once a year, right?
• 54:44
  You know, so you can play games like that.
• 54:45
  And you say by much value, did you drive from the solutions
• 54:48
  about XX million or whatever? And you say, well, if we
• 54:52
  actually accepted some change failure rate and up to
• 54:55
  deployment frequency, we could generate 5 or 10X more value.
• 55:00
  And that's when you sort of, you know, that's the vanity metric
• 55:02
  game you can play. And I've seen that a lot,
• 55:04
  especially in more conservative industries.
• 55:06
  You can get fixated on the wrong thing.
• 55:09
  We must avoid, we must avoid this failure type or at all
• 55:12
  costs. Sometimes that's not the case.
• 55:14
  Sometimes the downsides are actually relatively small and
• 55:16
  you should move fast and just try things.
• 55:20
  Yeah. And it.
• 55:21
  Goes back to really deeply understanding what you're trying
• 55:27
  to do, what your end goal is and, and I really like how you
• 55:30
  pointed out there's that. Tech non.
• 55:33
  Tech divide like the business side of the house and the
• 55:35
  technology side of the house and if you can service the business
• 55:38
  side of the house. You're going to.
• 55:41
  Be in a great position because people are going to if, if you
• 55:46
  can speak in dollars and cents, people are going to like what
• 55:50
  you're doing. Yeah, but but I think it's
• 55:53
  incumbent on us to speak that language because why are we
• 55:56
  there as technologists? I'm not, you know, you're not,
• 56:00
  you're not a technologist in a large financial organisation or
• 56:03
  a large organisation of almost anything.
• 56:05
  You're not there because someone wants to pay you to have fun
• 56:08
  playing with tools, right? It's great.
• 56:09
  It's great if you can have that, right?
• 56:12
  I'm very lucky to have a career where that is true.
• 56:14
  I'm fundamentally there to help drive value for customers,
• 56:18
  stakeholders, shareholders, etcetera.
• 56:19
  That it's just the truth. But that works.
• 56:22
  Even if you're in the third sector, even if you're in
• 56:23
  charities, right? You're there to, you know, help
• 56:26
  your service users and help the people you help.
• 56:29
  You're not, you're not, you're not being paid to sit there
• 56:31
  because, you know, people like just people are really keen that
• 56:34
  Andy has a good time. That's not, that's not sort of,
• 56:37
  you know, why I come into work for a day and I think that's
• 56:39
  super, super important to remember.
• 56:40
  Right. Yeah.
• 56:42
  Keep that front of mind, put it on a post it and put it on your
• 56:45
  wall so that I never. Yeah.
• 56:49
  Well, we had this guy Nick on a few episodes ago and he was
• 56:53
  saying how he was listening to a leadership conversation or
• 56:59
  podcast, I think it was and how they were really excited
• 57:03
  because. They.
• 57:06
  Recognised an area that they could automate in their business
• 57:10
  and they said yeah and then we can get rid of all those
• 57:12
  expensive humans and so he made the point of.
• 57:17
  Somebody's. Always trying to figure out a
• 57:20
  way to not pay you. So you have to be figuring out a
• 57:24
  way to make sure you're getting and and creating more value than
• 57:30
  the person, the other opposite force that is trying to make you
• 57:35
  not get paid. Yeah.
• 57:37
  Yeah, exactly. You don't want to be a cost
• 57:39
  centre as an individual, you want to be a value generator,
• 57:42
  right. Are we?
• 57:42
  You always, I always think about that though, because some of
• 57:45
  the, some of the best teams I've worked in were very aligned to
• 57:48
  ROI for the team, you know, and having an ROI target.
• 57:51
  And I'm trying to think about that for myself.
• 57:53
  What's my ROI when I come in every day, you know, just am I
• 57:57
  delivering far more value than I'm taking out?
• 57:59
  And I let you know you like to think you are.
• 58:01
  And I think in most cases people are.
• 58:03
  That's why that's why we that's why jobs exist.
• 58:06
  But the, but the point is you can't, you can't just sort of
• 58:08
  assume it's, it's for granted to, to that point.
• 58:13
  You can't just sort of you're not there just to spin up cool
• 58:16
  demos. You're there to drive value for
• 58:19
  the organisation and teams you're part of.
• 58:21
  And that's just super important to bear in mind, But it's still
• 58:24
  a fun challenge. Like this is the thing.
• 58:25
  It's not people think that might take the fun out of it
• 58:28
  absolutely doesn't. I've always been driven by that.
• 58:30
  I don't want to just take care of for no reason.
• 58:33
  I want to be like there and go. You know, that that thing that
• 58:36
  you use everyday to your customer.
• 58:38
  We helped build that. I think that's that's really
• 58:41
  sort of exciting these. Teams that you speak of that
• 58:47
  were some of the best you've worked with, which were highly
• 58:52
  aligned on ROI. What do you think made them like
• 58:55
  that? What were some of the things
• 58:58
  that. If I'm on a.
• 59:00
  Team I could take back to my team and say, hey, we might want
• 59:04
  to think about this, yeah. Great leadership, I think.
• 59:11
  I think that's what does. Great leadership mean exactly.
• 59:14
  I'm going to tell you. I'm going to tell you, don't
• 59:15
  worry. So.
• 59:16
  I just think, I just think great leadership is someone who
• 59:20
  provides clarity, someone who provides stability, someone who
• 59:24
  provides an environment in which you feel psychologically safe.
• 59:28
  I'm a big fan of that, but also where you're safe to innovate
• 59:31
  and try things within the appropriate bones for the
• 59:34
  organisation. But as someone who provides air
• 59:36
  cover and backs you, it says you know and wants to listen to your
• 59:41
  ideas no matter what level you're at.
• 59:43
  I've been in organisations where like it's super horizontal,
• 59:47
  really large organisation, super horizontal.
• 59:49
  And that is excellent because that new data scientist you've
• 59:52
  just hired may have the best idea that anyone's ever had.
• 59:56
  So make sure you have an environment where they can say
• 59:57
  that. Also you, as I had this phrase
• 1:00:01
  before, hippos, highest paid, highest paid person's opinion,
• 1:00:04
  hippos. Hippos are bad, right?
• 1:00:06
  You should just look at the highest paid person and assume
• 1:00:08
  they know everything or the most senior person.
• 1:00:12
  I work on the basis that I know nothing.
• 1:00:13
  You know, I'll go ahead and I may be the most senior engineer
• 1:00:16
  in a room, but I just kind of always assume.
• 1:00:19
  All these people are smarter than me.
• 1:00:20
  Maybe that's my inherent imposter syndrome, but it's a
• 1:00:22
  really good tool because it means you listen, people come up
• 1:00:25
  with amazing ideas and this whole problem.
• 1:00:26
  So that's what good leadership is.
• 1:00:28
  I think alignment with, like we said, the organisation, the
• 1:00:32
  business which comes through leadership, but is also just
• 1:00:35
  should be part of your team culture.
• 1:00:36
  Like why are you there every day?
• 1:00:38
  You're there to, you know, and it could just be enable my
• 1:00:42
  colleagues have a mission, right?
• 1:00:44
  I'm there to. Some more mortgages, yeah, so
• 1:00:47
  we'll. Sell more mortgages.
• 1:00:47
  I like to think we're driving more value for our customers,
• 1:00:50
  right, OK, or or I'm trying to I'm trying to stop fin crime or
• 1:00:53
  I'm trying to, you know, whatever it is, whatever the
• 1:00:56
  mission is, be very clear on the mission and just really feel
• 1:00:59
  that through the team. And then I think get stuff done.
• 1:01:04
  I think Barack Obama or someone said that in their they're all
• 1:01:07
  biographer, right? And I just always think about
• 1:01:09
  that. Be the person who gets stuff
• 1:01:10
  done. Be the team who gets stuff done.
• 1:01:13
  It's just so critically important.
• 1:01:14
  It just ties in with everything we're saying.
• 1:01:17
  If you're the ones who sort of humming high and you think and
• 1:01:21
  you know, you come back and forward and you know that's not
• 1:01:24
  that's not where you want to be. You want to be the people who
• 1:01:26
  you say, yes, grab it by the horns, let's go.
• 1:01:30
  That's it. That's that's what made those
• 1:01:32
  teams successful. And speaking again on this,
• 1:01:36
  having the mission, is that something that you feel like you
• 1:01:39
  you've seen people continuously talk about when you're on these
• 1:01:44
  teams? Is it Yeah.
• 1:01:46
  Or is it just because, yeah, it's easy to forget, right?
• 1:01:49
  And I in positions where I've LED teams, I know that I've said
• 1:01:56
  it a few times, but it's incredible how you think you're
• 1:01:59
  saying it a lot. And then when you circle back,
• 1:02:03
  you recognise people have no idea.
• 1:02:06
  They're like, no, I never heard you say that.
• 1:02:08
  What are you said that or that's our mission.
• 1:02:10
  Wait, what? Yeah.
• 1:02:12
  Why are you saying our mission's about mortgages?
• 1:02:13
  Yeah, exactly. And so, no, absolutely.
• 1:02:16
  I, I, so when I, when I kind of have teams and stuff and play
• 1:02:20
  teams, I always try and bring it back to why are we doing this?
• 1:02:24
  Because I think it's important for people to hear and feel, you
• 1:02:27
  know, it helps people understand why their contribution is
• 1:02:29
  valuable. If you're saying is someone go
• 1:02:32
  fill in these, you know, 3 forms are going on board to this too
• 1:02:36
  or helpless team, you know, it's important to say and remember
• 1:02:41
  the reason we're doing that because if we can get these
• 1:02:43
  teams onto this platform, they will stop fin cream, it will
• 1:02:46
  stop more financial cream, right?
• 1:02:48
  Or, you know, it's kind of, I think it's important to embed
• 1:02:50
  that in. And I don't think it's if it's
• 1:02:52
  false or kind of disingenuous or corny to do that.
• 1:02:56
  I think, you know, it's just, it's just the way it is.
• 1:02:58
  Yeah, I think the mission coming back to that mission is always
• 1:03:00
  super important and, and recognising a job well done.
• 1:03:04
  I said be the person who gets stuff done.
• 1:03:05
  It's important to call out, you know, this was a good example.
• 1:03:09
  We should try and replicate that or this didn't go so well.
• 1:03:12
  How can we learn from it? I think that's simple as well.
• 1:03:14
  But yeah, I think I think mission is super important.
• 1:03:17
  If I, I kind of wake up and come into work every day and try and
• 1:03:19
  think, how can I enable all of them?
• 1:03:21
  I actually tell people my job is to make you look better.
• 1:03:24
  Nice. My job is to make you do better
• 1:03:26
  stuff. I want to enable you to go
• 1:03:28
  faster. I want to enable you to deploy
• 1:03:30
  more models. I want to enable you to build
• 1:03:32
  better products and services, right.
• 1:03:35
  That's my job and I love it and that's that's why I'm there.
• 1:03:38
  And that mission sort of really helps me focus on what's
• 1:03:41
  important. So, so last.
• 1:03:44
  Last one for you before we jump in that because dude so good on
• 1:03:50
  that enablement part. Yep.
• 1:03:53
  Do you look at any metrics around?
• 1:03:58
  And maybe. We'll cut this out because
• 1:04:00
  you're like, yeah, that's part of those dev OPS metrics that I
• 1:04:02
  was talking about earlier. But I think about the enabling.
• 1:04:07
  So there's the enabling enablement piece.
• 1:04:10
  So are you looking at metrics that are like time to production
• 1:04:17
  type metrics? Yeah, I through my career, I've
• 1:04:20
  always been obsessed with that. So time to time to value, I call
• 1:04:24
  it or I've had that called in different contexts, but but
• 1:04:27
  exactly that I think that's I think that's one of the key
• 1:04:29
  ones, right. So do you go from an idea on a
• 1:04:32
  white board or a sheet of paper or whatever to a service?
• 1:04:36
  And yet there's always a debate about when is the starting gun
• 1:04:38
  filed? And you know, when do we when do
• 1:04:41
  we say the clock started? But fine, just pick, pick, pick
• 1:04:44
  a definition. It's the first commit in the
• 1:04:46
  repo. It's the first beating we have
• 1:04:49
  whatever, but just pick a definition and exactly that.
• 1:04:52
  So how, how quickly can I get to production And then all those
• 1:04:56
  the Dora metrics that I've mentioned before also things
• 1:04:58
  again back to that value piece, right?
• 1:05:02
  What is the value of this portfolio of stuff?
• 1:05:05
  If we get to the end of like say two years from now or whatever
• 1:05:09
  and we say we've not really delivered much ROI, then we've
• 1:05:13
  kind of we've lost sake of that mission.
• 1:05:16
  And ROI can be, I think it's important ROI can be defined in
• 1:05:19
  some more qualitative ways, right.
• 1:05:20
  Some things are just core enablers you need in an
• 1:05:22
  organisation. So like you know, we need good
• 1:05:26
  HR technology, we need, we need lots of stuff, we need them just
• 1:05:30
  core IT infrastructure, etcetera.
• 1:05:32
  They are not necessarily things that drive up sales or cost here
• 1:05:36
  etcetera, but but they deliver good ROI.
• 1:05:39
  It's just, it's a bit more of a qualitative calculation.
• 1:05:42
  Yeah, look at, look at metrics like that.
• 1:05:44
  I want to look more, and I've done this before, look at
• 1:05:47
  metrics around, you know, adoption key tools and enablers.
• 1:05:51
  So if you build out say for example, some templates are
• 1:05:55
  exemplars for how to do specific things.
• 1:05:58
  So templates on how to do monitoring, some training, types
• 1:06:04
  of types of process, be really keen to track, you know, how
• 1:06:07
  many people are actually consuming and using them.
• 1:06:11
  And that justifies the value. And then what you can say is you
• 1:06:13
  can kind of bootstrap and say, do you know what 10 teams have
• 1:06:18
  used this and another 10 are asking for it.
• 1:06:21
  That tells me there's something here.
• 1:06:23
  And it tells you if you should invest more time in that versus,
• 1:06:26
  you know, doing this other site project actually only one person
• 1:06:29
  will ever use. And yes, it's a bit kind of blue
• 1:06:32
  sky and stuff. So that I'm very metrics driven
• 1:06:35
  and I try to be as much as possible and.
• 1:06:38
  That brings up a great point which I completely forgot about
• 1:06:41
  earlier that I wanted to ask you about.
• 1:06:44
  Now it's coming back to me on being able to distinguish or
• 1:06:50
  know which use cases are more valuable than others and
• 1:06:56
  recognising this model or use case, whatever service is
• 1:07:03
  driving millions for the business.
• 1:07:05
  So we really need to babysit it versus this is new, we don't
• 1:07:10
  have clear stuff around it. It's cool, but we don't
• 1:07:15
  necessarily have to do the same amount to it.
• 1:07:17
  And and I think that goes hand in hand with what you're just
• 1:07:20
  saying is how? If you have.
• 1:07:23
  Those those metrics about who's using it, what's it doing, how's
• 1:07:28
  it working? You can properly organise
• 1:07:33
  yourself to make sure that the right things are getting the
• 1:07:36
  right attention, yes. Absolutely.
• 1:07:39
  And and that whole thing about speaking the language of the
• 1:07:42
  business, like some of the most valuable people in that
• 1:07:45
  conversation are like business embedded professionals or maybe
• 1:07:49
  even not technical, or it could be the data scientists working
• 1:07:52
  most closely with the business, whatever.
• 1:07:53
  But the people who are closest to the actual value proposition
• 1:07:58
  always write the best business cases.
• 1:08:00
  And you just sort of as a technologist, help them and see
• 1:08:02
  what's possible and what's not. So if someone says, you know, I
• 1:08:05
  want to, I want to catch 100% of fraudulent transactions, you
• 1:08:09
  know, you can come in and go, that's probably not possible.
• 1:08:12
  But how valuable would it be if we cat we call 50% or 60% or 80%
• 1:08:17
  and you can start building a case around it.
• 1:08:18
  And you very quickly to your point, this actually, this is a
• 1:08:23
  very good return. Well, this is a critical
• 1:08:25
  enabler. And sometimes though you have to
• 1:08:28
  be a bit kind of, you know you have to you have to kind of take
• 1:08:31
  the risk your bets and just say this is a huge upside and low
• 1:08:35
  downside if it fails, but it potential is a huge upside if we
• 1:08:38
  can get this right. So we should try that.
• 1:08:41
  And you just do that, you manage it like a portfolio and an
• 1:08:44
  investment portfolio, if you like.
• 1:08:46
  Something that's an analogy, yeah.
• 1:08:48
  But I. Think that I think that's the
• 1:08:49
  way to do it. You know, you've got your sure
• 1:08:50
  bet, your sure bets, you've got your kind of medium risk ones
• 1:08:54
  that are a bit more valuable and then you've got your ones that
• 1:08:56
  are kind of blue sky put it all in black type thing.
• 1:09:00
  But, but, but the upside is huge and I think it's just, it's just
• 1:09:04
  really good leadership coming back to that point helps you
• 1:09:07
  because they help manage that, that whole risk investment
• 1:09:11
  decision. And the more you're in the sort
• 1:09:14
  of the industry, the more you can smell a good use case.
• 1:09:17
  I think you sort of develop a bit of a sense for it.
• 1:09:19
  You know, you can go this, this is going to run into lots of
• 1:09:22
  problems. I've done something like this
• 1:09:24
  before and we thought it was good on paper and it was a
• 1:09:26
  nightmare for these 10 reasons. And then there's other cases you
• 1:09:29
  go, I know that sounds difficult, but actually in
• 1:09:31
  practise, there's three steps and you're done.
• 1:09:35
  Let's just do it. Like the upside might not even
• 1:09:38
  be that great, but it's so easy to do.
• 1:09:39
  Let's do it. So there's, you know, there's
• 1:09:40
  just that's that's where it's good to have good people around
• 1:09:43
  to been through the the wringer of it.
• 1:09:46
  So good. Man.
• 1:09:47
  Well, Andy, it's been a pleasure talking to you.
• 1:09:49
  Dude, This is so much fun and I appreciate you coming back on
• 1:09:52
  here. And for anyone out there that
• 1:09:55
  has not read your book, if you want to be like the smart folks
• 1:09:59
  at Oxford, go and read it. Just like that.
• 1:10:02
  Thanks for having me, Demetrios is great.