Latest / Agentic Conversations (formally mlops.community) / AI Operations Without Fundamental Engineering Discipline // Nikhil Suresh // #250

Transcript

AI Operations Without Fundamental Engineering Discipline // Nikhil Suresh // #250 · Agentic Conversations (formally mlops.community)

• 0:00
  Nikhil Suresh I just go by Nick Suresh.
• 0:03
  So I work for a couple of people I won't name here the but the,
• 0:07
  the company I can't talk about without worrying about anything
• 0:09
  is I work at Hermit Technology. It's a consultancy I founded
• 0:12
  with some friends. It's just six of us.
• 0:16
  We have no employees or it's just six directors hanging out
• 0:20
  idea. I'm I'm a director there.
• 0:21
  We have a completely flat structure.
• 0:24
  And as far as coffee goes, it gets me way too jittery.
• 0:26
  I just drink a lot of a lot of straight sail on tea.
• 0:30
  Welcome back to the ML OPS Community podcast.
• 0:32
  I am your host, Demitrios, and today we're talking with Nick.
• 0:37
  This episode was all about AI projects, ML projects, data
• 0:42
  projects, and how you can have more success.
• 0:47
  What we can do to make sure that we have our feet on the ground
• 0:51
  when we are creating the projects so that we can ensure
• 0:56
  success. Nick has gone a little viral
• 1:01
  recently because of a blog post that he wrote, which is
• 1:04
  hilarious. And if you have not read it,
• 1:06
  we'll put a link to it in the show notes.
• 1:08
  It's all about if you say AI one more time I will pile dry view.
• 1:14
  Hopefully you read it. If not, go read it right now and
• 1:18
  then come back to this. It's well worth the read.
• 1:21
  It will make you laugh and we dive into the blog post a bit,
• 1:25
  but we also dive into these underlying themes of why he was
• 1:31
  talking about, what he talked about in the blog post, and
• 1:36
  where they came from. Hope you enjoy.
• 1:38
  Let's get into the episode. As always, if you like it, make
• 1:42
  sure to share it with just one friend so we can trigger that
• 1:47
  algorithm. I've got to start with,
• 2:00
  obviously we're going to talk about the article that you
• 2:04
  wrote, which I think hopefully most people have read by now.
• 2:07
  I've seen it on so many different slacks and so many
• 2:10
  people getting very energised about it.
• 2:13
  But it is basically, if you say AI one more time, I will fucking
• 2:18
  pile drive you. It makes me laugh so hard
• 2:24
  because it echoes so many sentiments.
• 2:26
  And as soon as I read it, I was like, oh, I, I should reach out
• 2:30
  to you and see if you want to come on the podcast.
• 2:32
  And so I did that and you responded like, yeah, I've been
• 2:37
  getting a lot of requests, but I would appreciate to come on a
• 2:40
  small podcast. And I was like, I don't know if
• 2:44
  that was a backhanded compliment or if I should feel honoured
• 2:48
  that you're coming on the podcast.
• 2:50
  So, Nick, it's great to have you here.
• 2:52
  Thank you so much. I'm going to apologise on air
• 2:54
  for that one. So what actually happened was
• 2:56
  I'm just, I'm just some guy, right?
• 2:59
  So that the issue is that when people said you want to come on
• 3:02
  the podcast, I'm like, no big podcast and want me.
• 3:04
  So I, I knew there had been one big one.
• 3:06
  I thought all the others were small.
• 3:07
  I, I got asked to do a clip for a documentary last week and I
• 3:12
  genuinely, I thought the guy asking me was asking me for a
• 3:16
  student project. And I was like, that would be a
• 3:18
  nice thing to do. And then two weeks later I found
• 3:21
  out he's like the director of the university in the US.
• 3:24
  Like I just have to calibrate everything up.
• 3:27
  You've been, you've been launched into a different
• 3:30
  stratosphere because of your eloquent use of words and
• 3:36
  writing. And the funniest thing, I
• 3:38
  literally laughed out loud when because basically, for those who
• 3:42
  haven't read your post, you kind of make a point and at the end
• 3:47
  of each paragraph, you in different words, say like, stop
• 3:52
  saying this and or else I'm going to kick your ass.
• 3:54
  And there was one. I can't even remember.
• 3:56
  I should pull it up, but it's like you, you put it into a
• 3:59
  formula. You were like, if you say AI
• 4:02
  again, you will meet these hands or something.
• 4:04
  Like that? That's it.
• 4:05
  That tried this bullshit? Catch these hands.
• 4:09
  Dude, I laughed out loud. It was the first time I laughed
• 4:12
  out loud reading something in a while.
• 4:14
  And so I'm glad to have you on here.
• 4:17
  I know that the quick history of what you've been up to and why I
• 4:21
  think you are able to speak about this is you've been
• 4:27
  through the gauntlet, like a lot of us have, on trying to
• 4:31
  implement machine learning or AI projects in companies.
• 4:34
  And you've seen them fail time and time again like we all have.
• 4:40
  Or you've seen leadership be like, hey, these guys are just
• 4:43
  like a cop. They're just a cost centre.
• 4:46
  There's no profit coming out of here.
• 4:47
  What the hell is this data scientist doing or what is this
• 4:50
  machine learning effort? We're not like seeing returns
• 4:52
  here. How do we justify it?
• 4:54
  Or even like just data engineers?
• 4:57
  I imagine so you want to give us the quick rundown on that.
• 5:01
  You don't have to say any specifics, but some of the.
• 5:04
  Shit these so I kind of at a high level.
• 5:07
  So the trajectory I, I think everyone goes through this.
• 5:09
  So I came from psychology. I moved over to data science
• 5:12
  because my stats background in, I was a stronger statistician in
• 5:15
  psychology than most people, which is that I then I went to
• 5:18
  meet real statisticians in data science and I'm like, is it next
• 5:22
  level? But you know, I did my best, I
• 5:24
  got good at it. And you go through that early
• 5:27
  phase where you're just so excited, you know, you're just
• 5:29
  like, I want to build, I want to do deep learning.
• 5:31
  That was the thing when I started studying in 2017, right?
• 5:34
  I want to build my own neural networks.
• 5:36
  I want to do super complicated stuff.
• 5:39
  I actually did some complicated stuff at university.
• 5:42
  It was much more classically classical statistics oriented.
• 5:45
  So it was a lot of information theory and expectation
• 5:49
  maximisation and all sorts of weird model fitting.
• 5:53
  Enter the workforce of like. This is where we'll do the deep
• 5:56
  learning and all I could. Find was people.
• 6:00
  Well, all I could find the the market was huge in Melbourne,
• 6:03
  right? I found a job in one day.
• 6:04
  I didn't even have a full work visa.
• 6:06
  They just like snapped me up and the pay was great, but no one
• 6:11
  even knew why they hired me, right?
• 6:13
  They would just land me at a company and I would be like, all
• 6:17
  right, like what do you need me to do?
• 6:18
  And they're like, ah, we got a spreadsheet.
• 6:19
  You know, could could you look at the spreadsheet?
• 6:22
  It has 10,000 rows. Is that big data?
• 6:26
  I feel seen. Yeah, be a bounce around for a
• 6:29
  bit. I, I ended up doing, I even
• 6:33
  found one place where we did deep learning, but within two
• 6:36
  days I'm like, this should just be like, this should just be XG
• 6:39
  boost, right? Like the, the model works, but
• 6:42
  it's degenerated into a keyword search.
• 6:45
  You just get your bigrams and you go like, it's fine.
• 6:48
  And I gradually came to the conclusion that like the market
• 6:51
  for specialist data scientists, it's actually relatively small
• 6:55
  in terms of number of use case. But the hype around the market
• 6:58
  has really over produced graduates and it's opened tonnes
• 7:02
  of roles, but only some of those roles are, are real.
• 7:05
  So that's kind of how I got to where I am.
• 7:08
  Like I moved to data engineering and it's not because I even that
• 7:11
  article, it's not because I don't respect machine learning,
• 7:13
  it's the thing I chose to study. It's just that I I couldn't
• 7:17
  stand people disrespecting the thing I did so much.
• 7:20
  Yeah, and so painful just seeing, like, projects being
• 7:26
  funded but with no clear objectives, no metrics, no, no
• 7:31
  clear like success. And then you're being blamed for
• 7:36
  it. I imagine it's like, why aren't
• 7:38
  you making this work? You're like, well, I don't
• 7:40
  really know what work means. Yeah, yeah, the a huge one is
• 7:46
  the business like doesn't even understand how a machine
• 7:50
  learning intervention would interact with their business.
• 7:54
  So, you know, I remember when I started work, the first thing I
• 7:58
  got asked to do was predict government funding for something
• 8:02
  the next year. And, you know, I was like, look,
• 8:05
  I could fit whatever you want to this.
• 8:07
  But I got, it's not a statistical process, right?
• 8:10
  Like I can't model the probability that a politician
• 8:13
  pulls funding next year. I can't get in his brain.
• 8:15
  You know, we only have like 10 years of data.
• 8:18
  It also is like, what would you do with this number?
• 8:22
  And the answer is they didn't know.
• 8:23
  They just wanted a line on a graph.
• 8:26
  And yeah, it makes it very hard. It makes, first, it makes you
• 8:32
  very feel insecure at your job because you, you know, you're
• 8:35
  not producing results. But it also means when you land
• 8:38
  in a job, you're not even in control.
• 8:40
  You're just hoping it is a job where they even conceptualise
• 8:45
  the role correctly before you got in.
• 8:47
  Because it doesn't matter how good you are machine learning,
• 8:49
  if they don't know how you would intervene and there's no
• 8:52
  mechanism and they don't want to do that work, it doesn't matter
• 8:55
  how good you are at it. Dude, you said something so
• 8:58
  important there, which I don't think enough people ask.
• 9:02
  And I had first heard it when we had this woman, Michelle, on
• 9:07
  from the Financial Times. And she was very strict about it
• 9:11
  in the Financial Times, like building the data culture
• 9:14
  because data engineers and data scientists and just like
• 9:18
  analysts, the data team was so overbooked or like asked
• 9:24
  questions of them all the time that didn't necessarily have a
• 9:30
  clear outcome or a reason for being asked.
• 9:33
  It was just like, oh, can we get this information?
• 9:35
  Let's ask the data team. And then next thing you know,
• 9:38
  they're flooded with questions. And she said that they needed to
• 9:44
  always have very clearly and almost like have an internal
• 9:49
  evangelist going around and making sure that before any
• 9:53
  questions went to the data team, there was someone that was like
• 9:57
  interfacing with the person trying to ask the question and
• 10:00
  saying, OK, so yeah, let's imagine that you got all that
• 10:03
  information, like whatever the data question is around how many
• 10:07
  people from this region churned in the last quarter or whatever.
• 10:12
  Now you have that information. What are you going to do with
• 10:15
  it? And if people couldn't give
• 10:19
  actionable insights on that, it was like, well, I don't think we
• 10:23
  need to really answer that question then if you're just
• 10:25
  like doing it to scratch an itch.
• 10:28
  And that is what happens. You see it play out time and
• 10:32
  time again where you have all these questions, you have all
• 10:35
  this stuff that's like, yeah, wouldn't it be great to know
• 10:38
  that? And then later you're like, oh,
• 10:41
  so we know it now. What are we doing with it?
• 10:45
  And it, it, the sad part is people's jobs, like you said,
• 10:49
  like there's jobs that rely on that.
• 10:51
  Yeah, yeah. I guess what you're saying is
• 10:54
  you almost need them to give you like a human decision tree
• 10:57
  before you give them a statistical one.
• 10:59
  Like unless, unless you show me like the forking path, I'm not
• 11:03
  giving you anything. And yeah, yeah, I said I I got
• 11:07
  out of data science around 2021. I moved to engineering because I
• 11:11
  found most businesses didn't have good infrastructure.
• 11:15
  But, and around that time in Melbourne, the jobs were drawing
• 11:18
  up. And if it wasn't for GBT 3
• 11:22
  coming out and, you know, it's a real field, so the jobs would
• 11:25
  have disappeared, but salaries would have taken a bit of a
• 11:28
  dive. And it's that we've gone the
• 11:31
  other way, which is businesses haven't figured out that that
• 11:35
  core issue, or at least yet. Most of them haven't.
• 11:38
  I know what tiny handful. There's one in the city that was
• 11:41
  talking about. They were asking like, hey, you
• 11:44
  know, could we hire you when we're ready for this big
• 11:46
  project? And I go, all right, so you want
• 11:48
  me as a data scientist, not an engineer.
• 11:50
  And they go, yeah, yeah, yeah. OK.
• 11:52
  When do you want me? And they had that plan.
• 11:54
  You know, they knew exactly what they wanted me to calculate and
• 11:56
  what they would do. So they're willing to pay tonnes
• 11:58
  of money, but they're like, we'll get back to you in two
• 12:00
  years because that's how long it would take Wow to put the
• 12:03
  infrastructure in place. And so I'll probably take that
• 12:06
  job in two years. But I've also like, you know,
• 12:09
  it's, it's hard once people are actually doing the math, the
• 12:12
  number of roles is much smaller. But the flip side is those roles
• 12:15
  are way more valuable. Yeah, if you can find a way to
• 12:21
  make it work or show the value, then exactly that it is
• 12:26
  immensely valuable. I've seen a lot of people go
• 12:30
  from the data science role to data engineering.
• 12:33
  Actually, right now I am reading this Fundamentals of Data
• 12:38
  Engineering by Joe Reese and he did that exact same thing.
• 12:42
  He was like, and he considers himself like a recovering data
• 12:45
  scientist because of it, because he's just like I saw data
• 12:50
  science is great, but what you really need is a strong data
• 12:54
  foundation. And so data engineering comes
• 12:57
  before AI. And you've probably seen the
• 13:00
  memes of like the company is just trying to jump straight to
• 13:04
  AI without having any data foundations.
• 13:08
  And so it is a strong, strong argument.
• 13:12
  I do feel like though, one point that you mentioned in the the
• 13:17
  blog post is that it's almost like there's been this
• 13:22
  resuscitation of the hype because of GPT 3.5 or ChatGPT
• 13:28
  that came out. And so now just as we were
• 13:31
  starting to like acclimate, like, Oh yeah, like data science
• 13:35
  roles may not be as valuable as those articles of like the
• 13:40
  sexiest job of the 21st century said that they were, or every
• 13:45
  company is AAI or ML company Now we are just thrown right back
• 13:52
  into it and everybody is expected to be doing AI.
• 13:56
  Yeah, Oh, oh. The first thing I'll say is
• 13:59
  Fundamentals of Data Engineering slaps.
• 14:01
  It's such a good book. It's so good, right?
• 14:03
  Yeah. Just really good carbon said
• 14:05
  stuff in there. But yeah, it's it's a broader
• 14:08
  point. Yeah.
• 14:08
  We're just back in this kind of crazy cycle.
• 14:11
  And I remember, you know, at before GPT three came out about
• 14:16
  a year beforehand, I felt like I was finally making progress.
• 14:20
  Like, you know, I could tell someone like, hey, we need to
• 14:23
  talk about model drift. And everyone was starting to get
• 14:26
  bored enough that they were like, all right, you know, I'll,
• 14:29
  I'll listen to this, this nerd talk about his thing.
• 14:32
  And the moment GVT three came out, it's like everyone's an AI
• 14:35
  expert and you can just you just look at their CVS for two
• 14:38
  seconds and it's like you've never shipped anything in your
• 14:40
  life. It's so obvious.
• 14:44
  And the hype, because there's so much attention going in
• 14:49
  different directions, like the attention dispersed.
• 14:51
  Now model drift is not important.
• 14:54
  Traditional ML, nobody cares about that anymore.
• 14:57
  It's all about how are we going to implement LMS into our
• 15:02
  workflow? How are we going to like power
• 15:04
  up our workforce? And the funny part is that I've
• 15:09
  heard stories of how hiring has gone on a freeze because like,
• 15:14
  they're like, Oh, well, we're going to be able to get more out
• 15:16
  of our employees since we're leveraging AI.
• 15:20
  And then now that it's been six months, a year of people really
• 15:25
  trying to leverage AI, they're like, actually we're not seeing
• 15:28
  that productivity gain. If anything, it's kind of taking
• 15:35
  a toll on our workforce because you have to go back and fix
• 15:39
  stuff that the AI fucks up. Yeah, yeah.
• 15:42
  Also something to think about is a big organisation, you know, a,
• 15:46
  a, a huge issue is the amount of pointless work that gets done
• 15:51
  right. It, it is a systemic issue.
• 15:52
  When you have a large company, you just have a lot of people
• 15:55
  who end up pushing emails in circles and it's very easy.
• 15:58
  I don't know if you're familiar with the David Graber, the The
• 16:01
  Old Curve is sociologist and anthropologist, but he has this
• 16:05
  book called Bullshit Jobs. I think it was originally
• 16:08
  Atlantic article. And it's funny because if you go
• 16:10
  to a company, you can go to people individually and
• 16:13
  sometimes you can find a whole department who are like, this
• 16:15
  department shouldn't exist. ChatGPT is not necessarily
• 16:21
  removing their work, it's making them.
• 16:24
  It's it's forcing them to generate more nonsense more
• 16:27
  quickly. Like I signs, I think about how
• 16:30
  many departments are just people.
• 16:33
  It's just ChatGPT emailing other people in circles, right?
• 16:36
  Like I prompt an e-mail, I sent it to a guy, he goes, summarises
• 16:40
  this in reply, and then we just do the back and forth for a
• 16:43
  while. It must be happening somewhere
• 16:46
  right? Like some government departments
• 16:48
  are just ChatGPT talking to itself.
• 16:52
  Yeah, and it's not fixing the underlying issue exactly that.
• 16:57
  Yeah, and there's a lot of factors with GPT.
• 17:01
  They're a little bit worse than I think the previous iteration
• 17:06
  in 2019 when people got excited about ML, they didn't even know
• 17:11
  what they were thinking, right. So if if they said we want AI
• 17:14
  and I go, all right, what are you picturing?
• 17:16
  It was nothing or Skynet. And I'm all right if that's but
• 17:20
  now the demos are so slick, but it's so hard to get a working
• 17:25
  product going that non technicians have this very vivid
• 17:28
  image. They're like, I'm going to talk
• 17:31
  and it's going to do everything. And and that's our way harder
• 17:35
  vision to push back against than like, oh, there's math you don't
• 17:37
  understand, right? It's so excited.
• 17:41
  Oh man, you're speaking my language, dude.
• 17:44
  OK, 20 seconds before we jump back into the show, we've got
• 17:47
  ACFP out right now for the Data Engineering for AI and ML
• 17:52
  virtual conference that's coming up on September 12th.
• 17:56
  If you think you've got something interesting to say
• 17:59
  around any of these topics, we would love to hear from you.
• 18:03
  Hit that link in the description and fill out the CFP.
• 18:06
  Some interesting topics that you might want to touch on could be
• 18:11
  ingestion, storage or analysis like data warehouse, reverse
• 18:14
  ETLSDBT techniques, etcetera, etcetera.
• 18:17
  Data for inference or training, AKA feature platforms if you're
• 18:20
  using them, how you using them, all that fun stuff.
• 18:22
  Data for ML, observability and anything fin OPS that has to do
• 18:27
  with the data platform. I love hearing about that.
• 18:30
  How you saving money, how you making money with your data?
• 18:34
  Let's get back into the show. And that's one thing that I
• 18:36
  really wanted to talk to you about was like, how have you
• 18:41
  found success in managing expectations, especially from
• 18:48
  the non-technical business stakeholders, especially like
• 18:52
  you? You've just been talking about
• 18:53
  how important it is for the data teams to really be aligned with
• 18:58
  the business side of the house so that the work that you do is
• 19:02
  valuable and it has strong business outcomes.
• 19:08
  How can you then translate that and manage expectations when
• 19:12
  somebody's like, oh, so we can just use AI for this, right?
• 19:18
  Yeah, it's, it's very hard. So I'll, I'll put some thoughts
• 19:22
  kind of groundwork on what we call in psychology, like theory
• 19:26
  of mind, which is, you know, imagining what is happening in
• 19:28
  someone's head, basically. And part of the issue is, and
• 19:32
  it's a problem across software engineering, people don't have a
• 19:35
  good model of what any software work is by default, right.
• 19:40
  So, for example, you know, I assume you don't have a medical
• 19:42
  degree. I don't have a medical degree.
• 19:45
  You have a reasonably good idea of what a surgeon is doing,
• 19:48
  right? If you, you couldn't do the
• 19:50
  surgery, but you understand like, well, that area needs to
• 19:53
  be sanitised. Leave him alone.
• 19:55
  He's a professional. Keep some blood bags on hand
• 19:57
  like you couldn't do it, but you get it.
• 19:58
  If you went into a surgery, you wouldn't be like, I don't
• 20:01
  understand what's happening. Whereas even pre AI you have to
• 20:07
  understand like to a non technician, it looks like you're
• 20:10
  writing a Word document in a weird language and that
• 20:14
  immediately causes issues, right?
• 20:15
  Because you you cannot. I like this term from Rich
• 20:19
  Hickey. Like a complex sort of
• 20:21
  simplifies have everything straight.
• 20:23
  And to complex is the tangle, you know, so really bad
• 20:26
  abstractions and bad. So there's like a big knotted
• 20:28
  spaghetti bowl. Word documents can't be
• 20:31
  complected. That is, you can write from the
• 20:33
  beginning to the end and this paragraph can be bad or
• 20:35
  whatever. It is not possible to write a
• 20:38
  sentence in a Word document so bad that the rest of the
• 20:41
  documents stops working right. Like that's not a that only
• 20:45
  happens when you're programming. So immediately I usually try to
• 20:48
  explain that kind of thing, but that needs a really long term
• 20:51
  relationship and a lot of patients and getting into AI
• 20:55
  like the LM thing, it's it's so difficult because it looks, it
• 20:59
  looks like a person talking. Yes.
• 21:01
  It's so, so difficult to explain to people.
• 21:04
  Like it's not quite a person talking.
• 21:07
  And all the demos, you know, salespeople, they're going out
• 21:10
  of their way to make you think it's a person because now you
• 21:13
  think you're buying a person for like $0.50 per API call or
• 21:17
  whatever. So it's, it's really about that
• 21:21
  disambiguation. I, I've been trying to find a
• 21:25
  prompt that really consistently breaks ChatGPT so I can like
• 21:29
  demo it. I wasn't sure what to use.
• 21:32
  I'm still playing around. I was thinking maybe something
• 21:36
  like asking it to play a game of chess because I've heard that it
• 21:40
  doesn't have anything approximating board state.
• 21:44
  So it just starts making moves that don't make sense.
• 21:46
  Like, that might work, but yeah, I'd have to play around.
• 21:50
  I seem someone's already done this.
• 21:51
  I should have looked it up instead of doing it from
• 21:53
  scratch. But really, you need to, I think
• 21:55
  I haven't succeeded with this yet.
• 21:57
  You just need to like, what's the Wizard of Oz thing?
• 22:00
  Show them the man behind the curtain.
• 22:02
  Like they're just. There's nothing here.
• 22:06
  Exactly, Yeah, Yeah. And for anybody out there that
• 22:09
  has figured out how to break ChatGPT consistently, let us
• 22:13
  know because that is a great way of doing it and just kind of
• 22:17
  like bringing people's expectations back to earth.
• 22:22
  And it is so hard for anybody who has used social media in the
• 22:30
  last year to not feel like, Oh my God, this is really
• 22:36
  incredible. And this is like the future at
• 22:39
  or the amount of stuff you hear on podcasts and how it's been
• 22:46
  projected to be something that is going to revolutionise the
• 22:53
  way that we work and live. And then I think there's us on
• 23:00
  the forefront of this and we're seeing advancements and we're
• 23:05
  seeing cool stuff happen, but it's not like what non-technical
• 23:10
  stakeholders will expect from it.
• 23:14
  And so I find that really hard. I, I know there's a lot of
• 23:18
  people that are not only struggling to translate that to
• 23:26
  the business side of the house, but also like, if you do manage
• 23:30
  miraculously to get something up and running and align with
• 23:34
  stakeholders and put it into production, how do you tell
• 23:39
  someone that we probably shouldn't trust this like a lot,
• 23:45
  you know, like we should lower our expectations on trust
• 23:49
  because this thing confidently lies a whole bunch of times.
• 23:56
  And so making sure that people are aligned on that part.
• 24:00
  There's so many like nuances of the productionizing of AI.
• 24:07
  And I really like how how you said it in your blog where it's
• 24:12
  like, maybe we don't need to even be thinking about this and
• 24:17
  it's a little bit of a waste of time.
• 24:19
  On most companies, you can buy something from a vendor that is
• 24:27
  toting how they're using AI, but really that vendor is probably
• 24:31
  just buying it from another vendor.
• 24:33
  And it is. And there's like you go, it's
• 24:36
  Russian dolls in a way. And down the line, somebody's
• 24:40
  using some kind of AI in there. But for the most part, you don't
• 24:46
  need to be the one that's trying to do the cutting edge shit.
• 24:49
  Yeah, Yeah. In fact, I don't even think it's
• 24:52
  good for engine. You know, normally in
• 24:53
  engineering cutting edge is good.
• 24:55
  But if you if you pull the the Russian nesting dolls out, you
• 24:59
  pull them all out. So many of these products at the
• 25:02
  end of the day like the smallest doll is just the line that says
• 25:06
  import open AI. That's not job security, right?
• 25:10
  Anyone can type open AI it, but if you're you're shooting XG
• 25:14
  boost like you actually have a specialisation, I'm not just
• 25:17
  going to pull some random guy and you know the I go for an
• 25:20
  improv theatre here in Melbourne and, and one day I turned up no
• 25:22
  one. There's an engineer, the only
• 25:23
  software engineer there. And I remember my really good
• 25:26
  friend Felipe comes up and he goes I've been thinking about
• 25:29
  getting into prompt engineering. You know, I've heard that's a
• 25:31
  thing and it's so hard to explain why people shouldn't do
• 25:36
  it because because that's a problem.
• 25:39
  You can do it. Literally anyone who like has a
• 25:44
  command of a language ChatGPT could speak could learn how to
• 25:48
  prompt engineer in like one month.
• 25:50
  Like it'll be fine. You do not want to build your
• 25:53
  career in something that people are going to catch up to in one
• 25:55
  month. Exactly.
• 25:58
  That's a really great point. And I feel like one thing that
• 26:03
  I've seen and I've been trying to workshop this in my head just
• 26:07
  to get my thoughts more clear on it.
• 26:09
  So I'll throw it out there right now and see what you got to say
• 26:12
  for it. Yeah, there are a lot of
• 26:16
  engineers that are spending a lot of time trying to figure out
• 26:20
  how to get AI to work right. Whether that is taking the
• 26:24
  really hard route and doing everything on Prem and open
• 26:28
  source and incorporating it into a product so that you can get
• 26:35
  whatever chatbot to be your companion.
• 26:39
  And that takes time, energy and resources from the engineering
• 26:44
  team that could be spending it on on other stuff.
• 26:47
  And then you've got like the marketers and the sales people
• 26:53
  who are using some low code, no code solution, putting together
• 27:00
  some pipelines, churning out a whole tonne of content or emails
• 27:07
  or whatever that is very useful for their job.
• 27:11
  So I've seen marketers who have increased their website traffic
• 27:16
  because just they spit out a whole lot of whole lot of blog
• 27:21
  posts, right? Or I've seen sales people who
• 27:24
  are able to really tailor their pitch to a certain person or a
• 27:29
  certain company, etcetera, etcetera.
• 27:31
  And this is or actually one way, sorry to go off on another
• 27:35
  tangent. I think the coolest way that
• 27:37
  I've seen a salesperson do something is that they get the
• 27:40
  data from people who are looking at their sales page and then
• 27:43
  they know which companies are looking at the sales page.
• 27:46
  And they can go and outbound to that company with a very
• 27:51
  tailored message on what other companies like that company have
• 27:57
  been doing and blah, blah, blah. And so they're doing all of that
• 28:02
  with AI, quote unquote AI, right?
• 28:05
  Yeah, yeah. Or tools that are using AI under
• 28:08
  the hood and they're seeing a lot of value from it.
• 28:11
  So that the thing, the point that I'm trying to make is like
• 28:14
  you have engineers over here that are spending time, energy
• 28:17
  and resources and really working hard to get the quintessential
• 28:22
  HR chat bot into production that at the end of the day is going
• 28:26
  to tell someone how many holiday they have left.
• 28:32
  Or you have the marketers and sales people over here who are
• 28:35
  making good money from their AI efforts using low code, no code,
• 28:41
  and they probably didn't think much about the actual
• 28:45
  implementation and how they're doing it.
• 28:47
  And so it's like, wait a minute, engineering spending all this
• 28:52
  time and effort on something that isn't giving us a return.
• 28:56
  And it's just like a hard engineering problem.
• 28:59
  Meanwhile, sales and marketing is actually using it and
• 29:02
  leveraging it and it's giving us money.
• 29:05
  And so I've been trying to like figure that out and how to
• 29:09
  eloquently put it for engineers to like not don't spend it, it
• 29:15
  is cool, but don't spend your time.
• 29:16
  Don't waste too much time on something that doesn't have that
• 29:20
  clear return. Yeah, and so I'm, I'm, I'm
• 29:25
  scheduled to give it my first talk very exciting in about a
• 29:28
  month. And it's based off a blog post I
• 29:30
  wrote. It's it's it's a bit of an old
• 29:31
  one, three months old, but it was called I called it does a
• 29:36
  does a frog have scorpion nature?
• 29:38
  And you know, that old fable love you got the frog at the
• 29:40
  scorpion. And, and the point I make a lot
• 29:43
  is that, you know, a lot of engineers are call it kind of
• 29:47
  like frogs meeting a pretty good nature.
• 29:50
  They want to get along with everyone.
• 29:51
  This you don't want to solve problems.
• 29:54
  And then they get halfway across the river and then they find out
• 29:56
  they've been laid off or they are forced to come back to the
• 29:59
  office or just do something they don't want to do.
• 30:01
  And they just sit there drowning, going like, why?
• 30:03
  You know, we were going to launch the project of the
• 30:05
  Scorpions like, LOL, I don't care, It's not my problem.
• 30:09
  And I really encourage, especially younger readers who
• 30:13
  reach out like to develop some idea of what's actually going
• 30:17
  through, like someone who's financially focused head.
• 30:20
  And if you listen to, I was linked to a podcast a week ago,
• 30:25
  which I declined to go on, but it was from a leader here in
• 30:28
  Australia. And they're talking about LLMS.
• 30:31
  And something really informative kind of cropped up because they,
• 30:34
  they don't work in the engineering space.
• 30:37
  So they do work in the engineering space.
• 30:39
  So when they talk about it, they're like, love engineers
• 30:41
  counterplace them, blah, blah, blah, really pro engineering.
• 30:44
  And then they talked about customer support and they said
• 30:46
  something like, yeah, we'll get rid of all those expensive
• 30:48
  humans. And it's like someone at every
• 30:52
  company, if they pay you a salary, wants to get rid of you
• 30:56
  because you're an expensive human.
• 30:58
  And engineers who don't really really think about whether they
• 31:03
  have made more money than they got like brought on for are at
• 31:08
  some level of risk. And even if you do make the
• 31:11
  money, I've I've seen the other case where people like earn
• 31:13
  triple their salary, but they don't talk about it and then
• 31:16
  they still get let go because like no one in finance knows
• 31:19
  what they did. Oh wow, yeah.
• 31:21
  So yeah, I, I have a friend, I think he he got hired by a
• 31:26
  company out in northern Australia, one of his state.
• 31:29
  He hasn't done too much work in the field, but he I think
• 31:33
  generated something like $1.5 million in his first year.
• 31:37
  But like no one in management knew so they cancelled the
• 31:40
  contract. Messed.
• 31:44
  Up. You know there's so many things
• 31:45
  you need to do for yourself that are just fixing the problem, but
• 31:48
  you have to start by fixing the problem, right?
• 31:50
  You can't not fix it. That's that's step one.
• 31:53
  Yeah. And that is one thing that it's
• 31:56
  so funny you mentioned that because I feel like in the
• 32:01
  community, that's one thing that I want to do or I almost look at
• 32:06
  is a signal for myself for if we are being successful.
• 32:14
  And I just want to give people like your friend the platform to
• 32:19
  be able to talk about these wins and, and understand that it's OK
• 32:24
  to kind of pat yourself on the back every once in a while and
• 32:27
  show others how you've done something, how you generated
• 32:33
  actual dollars for the company. And then that will give you the
• 32:39
  ability, hopefully when management is like, so what's
• 32:43
  that guy doing? Or what did what, what have they
• 32:45
  been up to? It's like, Oh yeah, they've
• 32:46
  been, it's very obvious what they've been doing because
• 32:50
  they've been giving talks at local meetups about their
• 32:55
  implementation or they've written A blog post about it or
• 32:58
  they've called on a podcast about it.
• 33:00
  And so I hear you where it's like a lot of engineers might
• 33:04
  not want to do that because it's almost, it almost feels kind of
• 33:08
  icky, right? It's like, I don't want to talk
• 33:11
  about how good I am or how well I did on this.
• 33:14
  But it is a necessity. Otherwise you may not be fully
• 33:20
  recognised for what you're doing.
• 33:22
  Yeah, and you know, there's two sides of that.
• 33:25
  Some engineers are just a little bit socially anxious.
• 33:29
  And to that I would say, you know, I I was until I was 18.
• 33:32
  You do enough reps, you don't stress out anymore.
• 33:34
  It's it's just like going to the gym.
• 33:36
  You just you just keep going and you don't mind if anyone thinks
• 33:39
  your squad sucks. Yeah.
• 33:42
  And the second one is, yeah. You know, I think something I
• 33:45
  love about engineering culture is I'm on Mastodon instead of
• 33:49
  Twitter. You know, I really like tech
• 33:50
  culture over kind of more generic corporate culture.
• 33:54
  And there's something I really like is they push back
• 33:57
  aggressively against what I call like a LinkedIn brain, you know,
• 34:01
  like there's a level of self congratulation that is I, I
• 34:05
  usually describe it as like masturbatory.
• 34:07
  You know, it's just kind of like, come on.
• 34:08
  Yeah, don't, don't do this in front of the children.
• 34:10
  This is this is horrible. And engineers push back against
• 34:16
  that, but they push back too far the other way.
• 34:18
  You know, you can, you can talk about what you did right in a,
• 34:21
  in a pretty honest way. You know, my career is doing
• 34:24
  decently now, and it was doing decently before the blog post
• 34:28
  blew up. I'm very honest when I talk to
• 34:30
  people. You know, I, I had lunch with
• 34:32
  them, someone right before coming on here a couple hours
• 34:35
  ago who was in a position to, you know, give me AI would say
• 34:39
  like a pretty good job. And I, I was honest.
• 34:42
  I was like, hey, yeah, look, I, you know, I'm smart and I work
• 34:45
  hard. I haven't seen like a
• 34:47
  successfully shipped project apart from this one thing, you
• 34:49
  know, like I'm batting, I'm batting like 20% in the ML
• 34:53
  space. And some of it wasn't my fault
• 34:56
  and I'm like, some of it was my fault.
• 34:58
  You know, I just, I ran out of mental health points before
• 35:00
  dragging the org across the line.
• 35:02
  Like technically I could have if I stopped sleeping or something.
• 35:06
  People are open to that because they're so used to everyone
• 35:08
  lying. You don't need to.
• 35:10
  I'm not a super Jesus. I mess stuff up all the time.
• 35:12
  I, I miss serious stuff up all the time.
• 35:14
  I guess it's not just the the easy ones that no one would mind
• 35:18
  like of a of a human and you can talk like that.
• 35:21
  It's OK. The ones no one's going to kill
• 35:23
  you for it, they'll they'll appreciate it.
• 35:25
  Exactly, that's the key. It brings that human aspect to
• 35:31
  it and people will very much appreciate it because we all
• 35:34
  know those blog posts where it's there's the meme, right?
• 35:36
  How to draw an owl, Like start with a few lines and then draw
• 35:40
  the owl and you're like what? That you skipped a few steps and
• 35:44
  the people if you don't talk about the hardship for what the
• 35:49
  challenges were, then it also is is a little bit bullshit.
• 35:53
  It, I mean, it comes off as insincere, like I've never seen
• 35:57
  an easy IT project. I don't think I've seen an easy
• 36:01
  any project ever, right? Just doesn't happen.
• 36:05
  Even like improv theatre, it's so it's unserious, right?
• 36:08
  It's just like we go figure out with a couple of friends.
• 36:11
  And even then I was, I was at a holiday recently and I had to
• 36:14
  miss three classes. And the, the T-shirt was like,
• 36:17
  you need to watch like 20 episodes of this TV show because
• 36:21
  it's the same format as like the improv piece.
• 36:23
  Like the things are just hard. And that's OK.
• 36:26
  You know it's and if you leave the hardship out, no one's going
• 36:29
  to believe. You yes, that is such a great
• 36:33
  quote. Like things are hard and that's
• 36:37
  OK. That's what almost makes it
• 36:39
  worth doing, because if it wasn't hard, we would call it
• 36:43
  pumped engineering. And so, and it's why it's so
• 36:47
  tempting, right? Because everyone's like, could I
• 36:50
  just do the easy thing? And I'm like, well, unless
• 36:53
  you're a grifter, you will only be paid for doing hard things.
• 36:56
  And you can study until they don't feel hard, but you're
• 37:00
  never going to start with them being easy, right?
• 37:02
  This is not not going to happen. I mean, here's a question for
• 37:04
  you. When you started this podcast,
• 37:07
  you must have gone through what I'm going through now and I'm
• 37:10
  bad at like the podcast editing arc where you learn about
• 37:15
  compression and just. Oh yeah.
• 37:18
  It's it's like nothing is easy. It's actually hard, right?
• 37:21
  It's really, it's a pain in the butt and it's just time you you
• 37:24
  put time in. Also, I think the hardest part
• 37:28
  about that, it wasn't I, I was really into video editing and
• 37:33
  audio editing before the podcast started.
• 37:35
  But the hardest part was you put hours into this and you're like,
• 37:39
  wow, you Polish it off so it's your best effort, you're putting
• 37:44
  your best foot forward. And then once you put it out to
• 37:47
  the world, you get like 5 views. And that was the hardest part
• 37:51
  where it's like, damn, I just put hours of my life that I can
• 37:54
  never get back into that and I got five people to listen to it.
• 38:01
  And it's not like that happened once.
• 38:04
  It was a continuous thing, you know, like for months that
• 38:07
  happened and you got to grind through it.
• 38:10
  That is really what it comes down to.
• 38:13
  I I'm a firm believer in it. Yeah, yeah.
• 38:17
  In fact, I think that's the only, the only failure about an
• 38:20
  engineer can really get into is that they put the hours in
• 38:23
  invisibly and that the nice thing about publishing a podcast
• 38:26
  is it's not invisible by definition.
• 38:29
  Yeah, it it could be awful, but as long as you keep working on
• 38:32
  it like it's out there, someone will eventually, even if it's
• 38:35
  your mom, like she'll get into it if it's her friends, which
• 38:40
  totally. Happened with my TikTok account
• 38:42
  the it's so funny my mom showed her friends and now I have them
• 38:46
  commenting on my TikTok and I'm like thanks I appreciate it.
• 38:51
  Not exactly the demographic I was going for.
• 38:54
  And I know they don't understand anything because I'm usually
• 38:57
  trying to give takes on machine learning and AI and all that.
• 39:01
  I know that they're most of it goes over their head when I
• 39:05
  start talking about recommender systems and the evaluation of
• 39:09
  LLMS and, but they still come in and they love it.
• 39:12
  And so yes, exactly. Yeah, yeah, yeah.
• 39:15
  A lot of people wrote in to me and they were like, I don't know
• 39:18
  anything you said in that blog post, but they were like, it
• 39:21
  sounds like I'm gonna keep my job.
• 39:23
  And it was kind of funny, you know?
• 39:24
  So I'm, I'm a subscriber. All right.
• 39:26
  Well, yeah. People just appreciate you being
• 39:29
  human. And it's, it's not a marketing
• 39:31
  strategy that gets talked about because it's scary, like
• 39:36
  vulnerable, you know, like I, I, I posted stuff about mental
• 39:39
  health before and it's, you know, pretty serious stuff, like
• 39:42
  stuff I've gone through. It's fine.
• 39:45
  People are fine with it. You know, it's Yeah.
• 39:49
  And I guess it's something I push a lot when I talk to
• 39:51
  engineers, which is, you know, you don't have to change who you
• 39:55
  are, but it's also OK to show people who you are, right?
• 40:01
  If you're super shy, don't like talking to people, you can you
• 40:03
  can find a deep down in your soul to give like one talk.
• 40:06
  Your friends will help you get through it.
• 40:08
  Everyone will think better of you for it.
• 40:09
  That even if you look super nervous, like people are
• 40:11
  supportive. The worst I've ever seen a
• 40:13
  comedian bombed was like a the Melbourne Comedy Festival two
• 40:16
  years ago. And but the person who bombed is
• 40:19
  clearly like freaking out. And like it, it was, they were
• 40:24
  not funny. They just didn't pull it off.
• 40:25
  But they got the loudest round of applause of anyone there
• 40:29
  because people just appreciate that they were going hard.
• 40:31
  But if that person had been really polished and bombed,
• 40:34
  like, there would have been no support, right?
• 40:37
  If, if, if you go to a Ted Talk in your suit and you try to come
• 40:41
  off all cool and you bomb there, you're done.
• 40:43
  Like Noah's helping you out? Oh, exactly.
• 40:49
  Yeah. I wanted to go back to another
• 40:51
  thing that you mentioned. And this is along the lines of
• 40:55
  being visible and being out there and making sure that
• 40:59
  people understand what you do well and understand who you are.
• 41:03
  Because I imagine you get a lot of people that reach out.
• 41:07
  I just as I do for like, hey, I'm, you know, trying to get a
• 41:11
  job or I want to do this and I 100% tell people you just start
• 41:17
  making content, let people know who you are and that will help
• 41:22
  you more than anything else. More than if you're going to
• 41:25
  spend 2 hours filling out applications, try and spend 2
• 41:29
  hours creating content and see if you can commit to doing that
• 41:36
  on a consistent basis and it doesn't matter.
• 41:40
  Like you don't have to be the best.
• 41:42
  That's what I really like. Try and hammer home.
• 41:44
  You don't have to be the best. You don't have to really like
• 41:49
  know everything before you start creating content.
• 41:53
  You just have to show the journey of you learning it and
• 41:57
  that is huge. Contents really worked for me,
• 42:02
  but I think the overwhelming thing, especially if you're
• 42:06
  young and entering the field is understanding that getting into
• 42:12
  the field doesn't require you to like be incredible cool.
• 42:18
  People will hire you off your potential as as long as you've
• 42:23
  put work into a thing that that's all it takes to have
• 42:26
  potential, right. You just tried to do something.
• 42:29
  If, if, if a junior engineer or somebody wants to be a junior
• 42:32
  rocked up to me and showed me like the worst Python I've seen
• 42:35
  in my life, but they clearly tried.
• 42:37
  I'd be like, all right, you're in.
• 42:38
  I'll give you a shot. And yeah, that, that goes a
• 42:42
  really long way. It's, it's different if you're
• 42:44
  like, it's probably like an Aegis thing here.
• 42:46
  If you're 50 or 60 and trying to make the swap, people aren't
• 42:49
  going to give you that same benefit of the doubt.
• 42:51
  And you've got to get a little bit more clever.
• 42:53
  It's doable. But you know, if you're between
• 42:55
  the age of 20 and 30, you could literally do anything as long as
• 42:57
  you seem enthusiastic. If you'll, you'll be fine.
• 43:01
  Yes, exactly. And if you're stoked on one
• 43:05
  specific thing and you've gone down the rabbit hole on it, even
• 43:09
  better. Like find a nice little niche of
• 43:13
  stuff that you can say like, yeah, I really like whatever it
• 43:17
  may be. Then that helps too.
• 43:21
  I think about maybe it's like, oh, I'm really into
• 43:24
  observability or I'm really into Rust.
• 43:27
  Like we've got a whole group on programming or LLMS and Rust or
• 43:32
  whatever. Whatever your choices, whatever
• 43:36
  floats your boat. I think that is a great way of
• 43:40
  looking at it too, and you can always change.
• 43:43
  Yeah, yeah. And, you know, it's not even
• 43:45
  engineering advice, right? This is advice on like, how to
• 43:48
  just build a good relationships with people.
• 43:51
  If you want to be a writer, just seem really excited about
• 43:53
  writing. Be really excited about writing.
• 43:56
  Yeah. Because so many people, I had so
• 43:57
  many emails and they're like, I appreciate where they're coming
• 44:00
  from, but they're like, you know, what Python book do I need
• 44:02
  to get into the field? And I'm like, it would be good
• 44:06
  if you read a book, but how many coffees have you lined up this
• 44:09
  week with people? That's the best you should be
• 44:12
  looking at. Yeah.
• 44:14
  Oh man, that's so true. And that is not that's for any
• 44:18
  stage of your career, right? Because it goes back to the
• 44:21
  point that you were talking about on making sure that you
• 44:26
  are aligned with the business objectives.
• 44:28
  And to do that, it's a very much a relationship game.
• 44:33
  We had Saul Rashidi on here a few episodes back and I asked
• 44:39
  her. She was chief data officer at a
• 44:43
  gigantic company. I think she said she had like
• 44:45
  900 people on her team. And yeah, huge, huge.
• 44:50
  That's huge for a data function like.
• 44:53
  I've never even heard of that. It's gigantic.
• 44:56
  And I was saying, you know what, what was the most important part
• 45:00
  of your role? What were you spending your time
• 45:02
  on? Like if you could give me a pie
• 45:04
  chart? Was it trying to figure out like
• 45:08
  the operations of the projects that were being implemented?
• 45:11
  Was it figuring out new tech that you may need?
• 45:15
  Was it relationship building or And she said like without a
• 45:21
  doubt, it's all the relationship like 90% it was relationships
• 45:25
  and just making sure that every one of the stakeholders was
• 45:29
  aligned the whole time and everything else like kind of
• 45:34
  fell into place after that. So it was, it's exactly what
• 45:39
  you're saying, man. And it kind of almost can go in
• 45:43
  that along the lines of what you were saying to where is because
• 45:49
  of that the politics of it, you get shit that this kind of
• 45:53
  bullshit. Yeah, yeah.
• 45:56
  And that that's a hard one because a lot of the time and
• 45:59
  people go like, hey, how do I fix this at this company?
• 46:02
  I, I just go, you don't you're, you're one person.
• 46:05
  You know, like there's those four people that say they're
• 46:08
  generative AI experts. They don't have another skill.
• 46:12
  You, you can't convince them to become a homeless, right?
• 46:15
  You're not going to talk them into that.
• 46:18
  Like you just have to accept some battles are are not
• 46:20
  necessarily winnable right now. I remember when I did that
• 46:25
  better offline episode, it ended with them asking what's your
• 46:29
  plan for dealing with the Gen AI hype?
• 46:32
  And I was, I was going to stay away from it, you know, like
• 46:34
  that, that looks like it's going to explode and I don't want it
• 46:36
  to take my career with it. But as as soon as it does
• 46:41
  explode, I'm probably going to get straight into the space and
• 46:43
  get into like the interesting bits.
• 46:46
  I just can't do it now because everyone tells me I I'd love to
• 46:48
  do it. It looks super cool.
• 46:50
  I just can't figure out who's serious about hiring and who's
• 46:54
  just kind of making it up. And I don't have time to like,
• 46:58
  work through 10 jobs until I find the one serious person.
• 47:03
  Wow, that is some wisdom right there.
• 47:06
  Thank you. Thank you.
• 47:06
  I should clarify for any listeners, I have no idea what
• 47:09
  I'm talking about, just making it up.
• 47:14
  Classic. No, I I do like that stance
• 47:18
  because you, you can set yourself up for failure if
• 47:23
  you're not picky or you're not smart about how you engage in
• 47:29
  this space right now. Yeah, yeah.
• 47:32
  A lot of it's just finding people who kind of have a little
• 47:35
  bit of mutual respect. The, the, the thing I always
• 47:37
  compare it to is, you know, when my electrician comes in and he's
• 47:42
  like, we got to open this wall up and move all this stuff.
• 47:46
  You know, I could be taken for a ride, but I just didn't.
• 47:48
  I don't know. I just go, all right, listen,
• 47:51
  you know, I, I trust you, Tony, whatever you want.
• 47:54
  Just fix the wall up. As far as I know, I'm not being
• 47:55
  scant, right? I haven't lost a bajillion
• 47:57
  dollars. Software engineering is the only
• 48:01
  thing where people have no idea what I do consistently try to
• 48:05
  like have an opinion. Like how nuts would it be if I
• 48:09
  don't know anything about electricity, right?
• 48:11
  It's a magic thing that makes the computer work.
• 48:13
  If I went to my electrician and I'm like, I don't think you need
• 48:15
  gloves, Tony, why can't you do it?
• 48:17
  Foster? You know, it's it's like you
• 48:19
  wouldn't even think about doing that.
• 48:21
  But people try it with software folks, but the people are.
• 48:25
  I just wait until I find someone who doesn't behave that way.
• 48:28
  And then I'm I'm going to work with you forever.
• 48:30
  You're never going to get rid of me.
• 48:36
  That is so good, dude. Well, this has been awesome,
• 48:39
  Nick. I appreciate you coming on here
• 48:41
  and chopping it up with me, and I really am stoked for your
• 48:46
  voice of reason in this space. Hopefully, you will continue to
• 48:50
  share your experiences and your learnings with the rest of us in
• 48:54
  the comedic way that you have been doing thus far.
• 48:58
  Duff, thanks. I I appreciate that.
• 49:00
  I will say this is the the coolest, chillest podcast I've
• 49:03
  been on so far. So that's that's tremendous.
• 49:08
  Well, we only have, you know, a few listeners so far I guess.
• 49:12
  That's what I'm sorry. I'm sorry, I didn't know.