Latest / Agentic Conversations (formally mlops.community) / MLOps meetup #7- Machine Learning and Open Banking with Alex Spanos of TrueLayer

Transcript

MLOps meetup #7- Machine Learning and Open Banking with Alex Spanos of TrueLayer · Agentic Conversations (formally mlops.community)

• 0:01
  Hello again everybody. We've got another special ml Ops
• 0:06
  Community podcast for you today. Our guest is Alex Panos lead
• 0:11
  data scientist at true layer want to mention one thing before
• 0:15
  we start and that is our ml Ops Community slack Channel.
• 0:20
  If you're not already in it, what are you doing?
• 0:23
  Come on, get in there. Share your voice on anything and
• 0:27
  everything machine learning and ml Ops.
• 0:29
  We'd love to hear it. All right.
• 0:31
  Hope you enjoy this episode. Take care.
• 0:35
  Let's go ahead and get started. Alex, thank you so much for
• 0:39
  coming. I appreciate you.
• 0:40
  Taking the time to chat with us. This is really exciting.
• 0:43
  Yes thanks for the invite so really looking forward to
• 0:46
  chatting with you. So for anybody that doesn't know
• 0:49
  what Alex does, you can see almost people, you can see on
• 0:55
  that slide that I was just sharing.
• 0:56
  He's lead data scientist that true layer.
• 0:59
  And today, we really wanted to talk about machine learning in
• 1:04
  open banking And I also wanted to talk about you wrote a great
• 1:08
  blog article on the hiring process of a machine learning
• 1:12
  engineer. And then I wanted to talk about
• 1:15
  the machine learning infrastructure or the ml Ops
• 1:19
  pipeline that you've created a true layer.
• 1:21
  But before we get into any of that stuff, let's go ahead and
• 1:25
  start with how did you get into Tech?
• 1:29
  okay, so I'm probably the only person in the audience who will
• 1:33
  know this, but the olsat mailing list What is your son mailing
• 1:39
  list? Okay, so it's interesting since
• 1:43
  back in 20, 2009 or something 2010.
• 1:47
  I was kind of fresh out of my degree in mathematics statistics
• 1:50
  and I was trying to figure out to be honest, what to do, which
• 1:55
  direction to go down. And there's this like mailing
• 1:58
  list I was in Greece at the time and there's this Royal
• 2:01
  statistical Society mailing list.
• 2:04
  That was good kind of posting you know job adverts.
• 2:08
  You know news about like statistics back then and then I
• 2:13
  saw like I was looking for, you know what where am I going to
• 2:16
  go? Like I didn't really have a
• 2:17
  clear Direction and these new like data science, Big Data kind
• 2:21
  of messages started coming through and really like I had no
• 2:25
  idea at the time and it really became interesting to me and I
• 2:29
  started researching online and it was about the time where this
• 2:32
  really like biggie Lotion was was happening like in the states
• 2:36
  and moving over to like to Europe and stuff.
• 2:39
  And Andy I think that really kind of just being part of like
• 2:43
  something that's so like involving and dynamic back then
• 2:47
  as contrasted to like the traditional statistics you know
• 2:50
  become an actuary or no no offense but like it or becoming
• 2:55
  an epidemiologist. Now this is really cool okay but
• 2:58
  you know it's a bit more was a bit more stagnant and like the
• 3:01
  tech so I became really, really Interesting.
• 3:04
  And like, being part of something, something big in
• 3:06
  like, you know. So, yeah.
• 3:10
  So then I decided like, okay, I probably don't have the right
• 3:13
  skills. I need to obscure little bit,
• 3:16
  and I had the opportunity to do a masters in the UK, where I
• 3:19
  focused a lot on the computational side of
• 3:22
  statistics, which okay, which led me to a Moore machine
• 3:26
  learning, right? And then, yeah, that's more or
• 3:29
  less, how than, after that? It's kind of unfolded naturally,
• 3:34
  be honest, perfect. And what has your trajectory
• 3:37
  been so far? You went to the UK, you started
• 3:40
  doing this, you started completing your studies and then
• 3:45
  you got a job in machine learning right away.
• 3:46
  Or how did that look? Oh no.
• 3:49
  So, I started out doing like, in Greece.
• 3:52
  I was doing like, School research on by medical projects.
• 3:56
  So the epidemiologist think I said was actually an option for
• 3:59
  me at the time. But then yeah, I went and did
• 4:03
  like my master's degree in here in Leeds.
• 4:07
  And this kind of opportunity internship opportunity.
• 4:10
  Came up with with this kind of Oil, and gas and oil and gas
• 4:15
  Services kind of company that essentially scanned the
• 4:19
  subsurface for kind of oil reservoirs.
• 4:23
  And there's a really nice opportunity to introduce machine
• 4:27
  learning techniques into that process.
• 4:29
  So, I did an internship there and it was a really good
• 4:32
  experience, like, in terms of technical kind of things, I
• 4:37
  learnt, and I decided to give it a shot.
• 4:41
  That to be honest that was not for me in the end, neither the
• 4:43
  industry or that much or like the the at the mindset I would
• 4:48
  say a little bit more like closed as an industry like very
• 4:52
  much intro intro, introverted rather than extroverted.
• 4:58
  So yeah, so a big opportunity came along to after a lot of
• 5:03
  like efforts It was one of the hardest things that I had to do.
• 5:08
  Like at that time, you know, it wasn't really established data
• 5:11
  science, wasn't really established like in 2014 here in
• 5:15
  the UK. So companies didn't know how to
• 5:18
  hire for data scientists, mostly right?
• 5:22
  People will remember that. Like you have these very this
• 5:25
  bucket lists of like skills. Like you need to be like master
• 5:30
  of algorithms, master of Statistics, Master of the
• 5:32
  Universe and it was it's hard to get a job as a hardest thing.
• 5:36
  I've ever done Wendy and this opportunity came along and it
• 5:40
  was with IBM, which was very kind of educational and a very
• 5:45
  good kind of transition into like doing data, science, work,
• 5:48
  formerly data science work. It was a bit more like on the
• 5:54
  Yahoo go a bit deeper into that. But after I BMI, I moved on to a
• 5:58
  startup called Black Swan data where I worked on like the core
• 6:03
  data science. Team building out algorithms for
• 6:07
  predicting kind of the next big thing in like fast-moving
• 6:11
  consumer goods using social data, that was that was
• 6:14
  interesting. And I move the trailer about two
• 6:17
  years ago. So, to be honest, it's a bit of
• 6:19
  a convoluted path but it's interesting.
• 6:23
  Yeah, completely and so, now jumping and looking at true
• 6:27
  layer and what what they do can you just give everyone a bit of
• 6:31
  background on what they are doing and how they're using
• 6:34
  machine learning. Okay, I'll start with the most
• 6:38
  like an interesting introduction.
• 6:41
  So PSD to your, this is a European regulation that came
• 6:46
  into false a few years ago. And I'm going to give the most
• 6:50
  like the worst intro to it ever. But essentially walk this this
• 6:56
  European regulation which is called open banking in the UK
• 7:00
  was all about kind of forcing financial institutions to open
• 7:05
  up their their systems for to enable kind of innovation in
• 7:13
  financial kind of services and financial technology.
• 7:16
  What does that mean? So the regulation forced Banks
• 7:21
  to create apis where applications can connect to and
• 7:26
  fetch data of users who consent for their data to be shared.
• 7:29
  So one of the best example, I can think of is like yolt, who
• 7:35
  are, who are kind of, you know, personal finance manager, you
• 7:39
  know, with y'all to connect your bank account and it fetches
• 7:42
  back, your your transactions and you it does a bunch of stuff on
• 7:46
  top of that, right? So to get At that connectivity
• 7:49
  before PSD to it was all about creating, like screen scraping
• 7:54
  like screen, scraping connections, which, which is not
• 7:59
  great for many reasons. The PST to force the banks to
• 8:03
  create dedicated apis for this. So it opened up a really kind of
• 8:07
  big for PS3 to was it the banks were keeping all this data for
• 8:12
  themselves and they weren't allowing anyone to get into it.
• 8:17
  If they what they would but it was a little bit so applications
• 8:21
  would have to build to essentially reverse engineer the
• 8:26
  login flows of when you log into open banking for example and
• 8:31
  that was not really scalable and not really like you know the
• 8:34
  bank changes the interface your whole connection is broken.
• 8:37
  Right? So PS and that's still the case
• 8:40
  in the US for example. But in the, in the in Europe, we
• 8:44
  have PST to. So the banks have apis for this
• 8:46
  We don't have to reverse engineer like he online banking
• 8:52
  system to get users data to incorporate into your
• 8:55
  application. So, yeah, they decide this one
• 8:58
  and payments is another very interesting application.
• 9:00
  Whereby, you can initiate payments directly from One bank
• 9:04
  account to go to another circumventing, essentially,
• 9:09
  MasterCard, visa and other kind of payment processing
• 9:12
  Technologies Okay, cool. And so then true layer is doing
• 9:18
  is doing what, in that space? Yes, it truly is on, kind of way
• 9:23
  on both aspects of PST to. So I have products in the data
• 9:27
  space and products in the payments based, I personally can
• 9:30
  speak more about the data space kind of work Stream.
• 9:36
  So essentially we're in API company, we solve like on the
• 9:42
  data side, the problem. Um, we kind of solve is, you
• 9:47
  know, let's say your company like a personal finance manager,
• 9:51
  right. And you have a bunch of users.
• 9:54
  They have different bank accounts for the use of the
• 9:58
  bank, with different providers. You have the choice of either
• 10:02
  building out and maintaining all these connections yourself or
• 10:05
  you go to a central place who manages all that complexity for
• 10:09
  you. So truly are kind of a bridge
• 10:12
  between the application and the financial institution and that
• 10:16
  way. Okay, nice.
• 10:18
  And now when what exactly are you doing in the machine
• 10:24
  learning space with with all of that information?
• 10:28
  Yeah. Yeah.
• 10:28
  So I'm onto the cool stuff. So, so fetching.
• 10:33
  The wrong information is one thing, right?
• 10:35
  So you can fetch a user's, you know, applications can fetch
• 10:40
  transactions of users. Upon consent from the users
• 10:42
  consent, do stuff with them, whatever.
• 10:47
  But but that's you can do like there's so much more one can do.
• 10:52
  So you can stop building entrance things stuff on top of
• 10:55
  the road transactions. So you could be adding more
• 10:59
  contextual information so you can for example identify What is
• 11:05
  the nature of this transaction? Is it kind of regular salary?
• 11:10
  Is it? If it's spending, is it
• 11:13
  shopping? Is it like utilities?
• 11:16
  And sorry, is it like, like paying for petrol?
• 11:20
  You know, you can do all these and all these interesting things
• 11:23
  about the nature of the transactions themselves.
• 11:25
  So that's more one layer. Then you can go one layer
• 11:28
  further. That's not a pun with the
• 11:30
  lecture later. I was just random, sorry, and
• 11:33
  you can build like, Insights products that are built on top
• 11:38
  of the road transactions and entirely.
• 11:39
  So you can say, oh, these are the aggregate behaviors for for
• 11:44
  this specific user. So you can say, you know, over
• 11:47
  time over category and this really makes it easier for
• 11:51
  applications who uses to use those insights directly rather
• 11:54
  than having to build them up themselves.
• 11:57
  And this is an area where machine learning.
• 11:59
  It's currently quite prominent actually, And how did you decide
• 12:06
  to use machine learning instead of just regular rules based
• 12:11
  algorithms? Yeah, so another interesting
• 12:14
  area. So when you, when you start out
• 12:17
  to conceptually, when you start out, solve the problem, you
• 12:22
  don't understand the you don't necessarily understand the
• 12:25
  complexity when we're talking about like data enabled
• 12:28
  products. So, you start out with the
• 12:31
  simplest approach to possible, right?
• 12:34
  And you see where that gets you you have to then measure and you
• 12:36
  have to make a judgment call whether it's worth going more
• 12:40
  complex or keeping it simple. So yeah, we started out with
• 12:44
  rules. We started out saying, okay, if
• 12:47
  this is a debit card transaction and it contains string,
• 12:55
  whatever, then it is complains of string, Uber, then it's kind
• 13:01
  of a taxi, you know, taxi kind of type of transaction, And I
• 13:08
  guess another thing you can do is you can identify the
• 13:10
  merchants or the entity name from the transaction, using a
• 13:15
  similar methodology. Okay, that works like a pack of
• 13:20
  that works for a certain amount of time.
• 13:23
  Assuming description transaction descriptions which we are using
• 13:27
  our kind of stable. And of course, they differ by
• 13:33
  provider and I'm be assuming that no other kind of logic
• 13:38
  falls apart. Then you can maybe identify up
• 13:42
  to 60% of the overall transaction flown through
• 13:46
  system, but that's not enough, right?
• 13:49
  You need to do more. So you have the option of
• 13:52
  building a bunch of more rules that are prone to change in
• 13:55
  break. Or you can do something
• 13:58
  different. And as I'm the audience, your
• 14:00
  audience will be very familiar with of much simpler ways to use
• 14:03
  supervised learning. Through through kind of human
• 14:07
  annotation. And We made that that judgment
• 14:11
  call we always, to be honest, it was always in the back of our
• 14:13
  minds but kind of commercial pressure to improve our product
• 14:17
  and personal kind of team drive to start.
• 14:22
  Making our methods more sophisticated LED towards making
• 14:26
  making the jump. I guess the jump itself was was
• 14:30
  more gradual. So we started out with a hybrid
• 14:34
  kind of rules based machine learning approach and I guess we
• 14:40
  are still using that approach and we're considering whether to
• 14:46
  keep it or whether to move to fully machine learning based and
• 14:51
  of kind of system. And the first question that pops
• 14:55
  out in my mind is, how long was it?
• 14:58
  And how much suffering was it? When you were going through this
• 15:02
  rules-based, and did you instinctively think?
• 15:05
  Let's just throw machine learning at it.
• 15:08
  Or was it something that it really had to go through the
• 15:13
  whole organization? And they had to recognize that
• 15:17
  this was the fix for the problem.
• 15:20
  I think like we were pretty autonomous like the end from
• 15:24
  like from a team point of view. You know, we were expected to
• 15:27
  get the expectation from the businesses to get this thing
• 15:31
  working as best as possible. So the buy-in wasn't wasn't
• 15:35
  really an issue in terms of like the team itself.
• 15:42
  So so I guess yeah, we it was just becoming a bit intractable
• 15:49
  to keep on building rules and And yeah.
• 15:53
  Just just, for example, reacting to cases where our
• 15:56
  classification was wrong. So you can imagine, like users
• 16:01
  of this classification service. We're saying, oh, this, this,
• 16:05
  this is a wrong category, what can you do about it?
• 16:08
  So you can either go write a new rule you know, with whatever
• 16:13
  that's chaos that might entail. Or we can actually leverage that
• 16:17
  feedback from the client in a loop kind of way in order to
• 16:21
  incrementally improve Your system and that's where machine
• 16:24
  learning. Like really, really shown as
• 16:28
  like the golden kind of solution there.
• 16:31
  Yeah, it starts to shine and now pivoting to infrastructure you
• 16:37
  guys decided, hey, we need to go into machine learning.
• 16:44
  We want to use this? How did how does your
• 16:47
  infrastructure look now? How did it evolve over time?
• 16:51
  Where did you start? Where did you come to?
• 16:54
  And why why Why? Yeah, yeah, so so truly I runs
• 17:01
  que Bonitas on AWS, this is our kind of core infrastructure and
• 17:08
  I guess the service, the team kind of started working out with
• 17:12
  given that it's our core programming language.
• 17:15
  At the moment was in see shop. So kind of the rules-based kind
• 17:23
  of classification engine was written in as a micro service
• 17:27
  tandiinsea shop. So I got to learn a bit of see
• 17:31
  shopping in the process, which I hadn't have any, personally an
• 17:35
  exposure to before and, and yes. So but when we're talking about
• 17:41
  introducing like machine learning, I guess your audience
• 17:45
  would agree that see shop is probably not the first option
• 17:48
  when would go with. And I guess it made a lot of
• 17:51
  sense to create car first. I'm not sure if it was a first
• 17:55
  but one of the first kind of python microservices in in
• 17:59
  production for this use case. So yeah so microservices they
• 18:06
  communicate through rest API and some a will.
• 18:12
  There's a mix but this might service in specific was written
• 18:16
  in flask. So yeah, it's a python flask
• 18:21
  microservice. Basically, we might have lost
• 18:25
  Demetrius well while we're waiting for Demetrius to jump on
• 18:31
  back on board here. Just really quickly.
• 18:34
  If you have questions, feel free, you can unmute yourself.
• 18:38
  You can also ask a question in a chat or just say, I have a
• 18:42
  question and we'll call on you that way.
• 18:47
  We actually have a question from Chris Harden, Chris, do you want
• 18:49
  to say your question out loud? Okay?
• 18:53
  When working in the financial services space decisions that
• 18:58
  are made on suitability of customers and credit scores need
• 19:00
  to be auditable down to the decision.
• 19:04
  Level, when using machine learning, do you avoid using
• 19:11
  Black Box decision-making? Or is there another method you
• 19:14
  use to overcome this? Yeah, so that's a great
• 19:18
  question, I guess. Our clients users in order to
• 19:26
  identify kind of the credit worthiness of their users.
• 19:30
  So many of our clients are in the credit space in the lending
• 19:34
  space. So as you can imagine, like for
• 19:39
  example, if we make an estimate on a user's income, which is
• 19:45
  then in turn used for making a credit decision on that you Zur.
• 19:52
  Then it is very important to be able to reproduce and replay the
• 19:57
  exact kind of machine learning model, kind of workflow that
• 20:04
  produced this estimate. So indeed the financial services
• 20:09
  industry is very tightly regulated.
• 20:10
  So, and added to that there needs to be a layer as you
• 20:17
  mentioned of interpretability. So clearly there is a big biased
• 20:21
  for us to use Is the simplest and most interpretable kind of
• 20:26
  algorithm available to us. So, we are averse to going more
• 20:32
  complex than required due to regulatory kind of environment.
• 20:37
  Don't know if that answers your question.
• 20:41
  Absolutely. Yeah.
• 20:42
  Thank you. Cool.
• 20:45
  So sorry I just got, I just dropped off for a minute and
• 20:49
  looks like you had some good questions going on.
• 20:52
  I see another question in the chat.
• 20:53
  We can go ahead and answer that real quick before we move
• 20:56
  forward. Thomas was asking apart from
• 21:00
  python flask do you build applications yourself with
• 21:03
  normal tools? Where do you use any other
• 21:05
  Library which is a core for the application.
• 21:10
  So I can only answer in this Sense on the python side.
• 21:14
  I wouldn't say I'm the specialist kind of back and
• 21:17
  engineer at ruler in terms from python microservices, I guess
• 21:24
  their apis and we tend to use a mix of different kind of
• 21:30
  libraries for that. We depending on whether we're
• 21:32
  using a synchronous or asynchronous apis, we tend to
• 21:37
  use either flask we He had experimented with Sonic in one
• 21:42
  occasion. And last year, fast API got apps
• 21:47
  was released and I've been extremely impressed with fast
• 21:50
  API as a whole. So yeah, I don't know.
• 21:54
  Again, if that answers your question, I would imagine it
• 21:57
  does. So let's let's continue.
• 22:01
  I'm not sure what I missed as far as the ml Ops pipeline that
• 22:06
  you, you put forth and the evolution of that.
• 22:10
  But I wanted to talk about just these the regulation and the
• 22:14
  reproducibility and how important you feel that is when
• 22:18
  it comes to the whole Pipeline and what you're doing with the
• 22:24
  data science. Yeah I guess we covered this a
• 22:27
  little bit before. So reproducibility is of very
• 22:30
  strong, like requirement in the space we operate.
• 22:35
  So yeah. So being able to replay our
• 22:37
  decisions like decisions of an algorithm like made six months
• 22:42
  ago, you know, meet me to need to be there like so can imagine
• 22:48
  like in production, you get a different, you know, host of
• 22:53
  models can be deployed over a period of time and having the
• 22:56
  ability to pinpoint which exact Code kind of produced which
• 22:59
  exact result, which data set was used, which parameters, we used
• 23:05
  all these are requirements for us to be able to replace that
• 23:07
  decision. And that's where I guess that
• 23:11
  science was was very useful for us, as it provided that missing
• 23:16
  element to kind of a traditional, get kind of version
• 23:21
  control system for being able to track that kind of provenance
• 23:26
  end-to-end. Yeah perfect.
• 23:30
  So I see in the in the chat there's also another question
• 23:33
  asked by Adam Butler and I also see yours Nick so we'll get to
• 23:36
  that in a second. Can you get explicit details or
• 23:41
  infer anything? And if you already answered
• 23:43
  this, let me know. But you get explicit details or
• 23:47
  infer. Anything about the items that
• 23:49
  somebody purchased during open banking.
• 23:54
  For instance, could you differentiate between somebody's
• 23:57
  Purchase at Sainsbury's, verse them, getting petrol at the
• 24:01
  Sainsbury's, petrol station. So, in this specific example, I
• 24:07
  guess it may be the case, so we the data we have to work with
• 24:12
  our details about the transaction itself.
• 24:15
  And these are the description is this is the amount.
• 24:19
  This is the timestamp, and, and some metadata regarding the
• 24:25
  nature of the transaction. So, how was the Ain't so how,
• 24:28
  you know, what was the kind of transaction method, and these
• 24:33
  are the basic information we have to work with.
• 24:35
  So, in this specific example, I seem to remember that
• 24:39
  Sainsbury's and Sainsbury's petrol station may have a
• 24:42
  different description. So I think this is something we
• 24:45
  can differentiate between, but I think the question itself is
• 24:48
  more about You know, for example, you have an Amazon
• 24:53
  transaction, right? Could be anything.
• 24:56
  So at this stage, we can't differentiate I've heard of like
• 25:04
  this in this may be a little bit off topic, but I've heard that
• 25:07
  monster is working with a company called flux who do like
• 25:11
  receipts. So they're incorporating like
• 25:14
  some receipt kind of management system which can tell you more
• 25:18
  granularly, what each transaction is, you know, the
• 25:22
  receipt attached to each transaction.
• 25:23
  And that in my mind sounds very interesting, but it's a
• 25:26
  capability. We don't currently have We kind
• 25:30
  of know, right? So if you see an Amazon
• 25:31
  transaction, five pounds, who knows what is, right?
• 25:37
  That being said, I think there are ways one can understand
• 25:43
  using with a large amount of annotated data.
• 25:47
  There are things that can be done in this space, but we're
• 25:50
  not there yet Amazon transaction, five pounds, it's
• 25:54
  probably a face mask or well for that paper.
• 25:59
  Sounds yeah. So getting back into it you
• 26:06
  mentioned. Well you mentioned to me before.
• 26:08
  I don't know if you said it here but you're using Prometheus and
• 26:11
  Griffon of for monitoring and can you give us some use cases
• 26:16
  on how you're monitoring and how you're setting up the dashboards
• 26:20
  when you have these and what you're focusing on?
• 26:25
  Yeah, we're API company. We run a micro services.
• 26:29
  So as you can imagine Prometheus is, its kind of the best way to
• 26:35
  monitor apis in production. We use Prometheus across our
• 26:41
  stock and kind of the machine learning services are no
• 26:46
  different. So for a classic service, there
• 26:51
  was a nice Library. We use goes called Prometheus
• 26:54
  flask exporter. I seem to remember and which is
• 26:58
  a nice kind of, you know, library that helps you
• 27:02
  instrument kind of the Prometheus metrics for, for
• 27:05
  specific python applications. So the way we use it is, yeah,
• 27:10
  it's a micro service. We report.
• 27:12
  Things like latency like error rate as is pretty standard but
• 27:17
  because we're talking about machine, a machine learning
• 27:19
  service. We also surface things like the
• 27:23
  distribution of inputs over time distribution of predictions of
• 27:30
  category frequencies over time and we use graph on out of the
• 27:35
  basically show. These all these things nicely in
• 27:37
  the dashboard that we look at every day and smile, or Not
• 27:43
  smile, we usually smile but but yeah.
• 27:49
  Cool. So as you look at the whole and
• 27:52
  the labs pipeline that you have set up, are there any places
• 27:56
  where you feel? There's a black hole or anything
• 27:59
  that you would like to fix you don't feel that you're doing
• 28:01
  well or now that time has passed.
• 28:04
  You feel like this could be done better.
• 28:08
  Well, that's a that's an interesting question.
• 28:10
  I think we're quite happy at the moment I guess the To be honest,
• 28:16
  as a data scientist. What I really want to know is
• 28:19
  like Is my is my model making the right call like on aggregate
• 28:27
  or not. And the way I currently
• 28:30
  understand this is through proxies the proxies I mentioned
• 28:33
  before. I guess what would be really
• 28:37
  amazing is this is me thinking out loud, right?
• 28:43
  If we had an automated way to kind of capture the accuracy, Of
• 28:49
  of this service like in the wild, through some
• 28:54
  programmatically like executed human annotation service on
• 28:58
  unlabeled and Soriano animus data, and integrate it into the
• 29:03
  graph. On a dashboard, that could be
• 29:05
  kind of a way to have everything in place, right?
• 29:08
  Because rather than proxies of the actual performance, you
• 29:10
  could have the actual like, you know, human reviewed
• 29:14
  performance, Maybe on a different time scale but still
• 29:17
  it would be in a place where you could actually no.
• 29:20
  See it though that's me. Talking out loud here.
• 29:24
  There you go. Luke is scribbling notes,
• 29:26
  frantically. I can see him there.
• 29:30
  So a good product idea. So there's some, there's some
• 29:37
  questions from Nick. I feel bad.
• 29:39
  I haven't been ignoring you Nick, I want to get to some of
• 29:41
  your questions. You've got some great ones.
• 29:43
  Let's start from this bottom, one in Prometheus anger, fauna.
• 29:47
  Are you limited by its matching to data being by regular
• 29:51
  Expressions as opposed to new Miracle by Data types.
• 29:57
  Yeah, so I guess I guess I see where this question is coming
• 30:03
  from. So in terms of like the
• 30:04
  rule-based, it's kind of hard to apply the same logic for the
• 30:10
  rules-based part of the of the system I guess in terms of like
• 30:17
  the rules, what we try and do is like we try and like look at the
• 30:23
  classification. Ocean like the coverage.
• 30:26
  So how many of the actual cases were successfully labeled by the
• 30:32
  rules based on and we surface this?
• 30:34
  But yeah, that's a good. That's a good shout.
• 30:37
  Actually, we use the common, the classic kind of API metrics in
• 30:44
  Prometheus for the rules based service, but for, you know, it
• 30:48
  doesn't exactly kind of fall into the same kind of monitoring
• 30:54
  Paradigm as the machine learning one at the moment.
• 30:59
  Perfect. So let's ask another one that we
• 31:04
  had in the chat here. Then we'll move on.
• 31:07
  Do you find the performance of python flask is?
• 31:10
  Okay. Are there limitations with that
• 31:12
  setup in production? Yeah, so, it's okay.
• 31:20
  Basically, obviously, it's a synchronous sorry.
• 31:25
  It is synchronous. So I guess there may be cases
• 31:31
  when we start scaling a lot. when we will probably have to
• 31:36
  move to to in a synchronous kind of and if we if our machine
• 31:40
  learning models become a little bit more sophisticated or not a
• 31:44
  rather sophisticated, that's a resource intensive, potentially
• 31:47
  adding a little bit of latency Than what we have Today Than,
• 31:53
  That Could Happen by making our kind of API bit more flexible.
• 31:58
  In terms of, you know, I want to throw the kitchen sink at this
• 32:02
  problem or I wanted liked version of the classification.
• 32:05
  That is kind of product Choice. Yeah.
• 32:08
  I mean, we might move to in a synchronous web server, but at
• 32:13
  the moment, it's working fine for us with the scale.
• 32:15
  We currently have Cruel. So, moving on to the blog post
• 32:22
  that you wrote about hiring in ml engineer, just as a bit of
• 32:28
  background information for everyone, can you explain what
• 32:30
  was going on? You, you were tasked with hiring
• 32:33
  someone as an ml engineer. Is that right?
• 32:37
  Yeah. So a lots of questions here so
• 32:40
  why animal engineer right? Well what is the exact skill
• 32:43
  set? We were after that were
• 32:46
  essentially looking for someone who is very able to take on like
• 32:51
  generic engineering tasks but who also have a strong
• 32:55
  appreciation of like the machine learning workflow end-to-end
• 33:00
  Okay, you're asking for everything here, a little bit
• 33:02
  which is always the the pitfall in hiring.
• 33:05
  So I think in the end we're looking for someone with very
• 33:08
  kind of generalist kind of programming skills that are
• 33:14
  modern and applicable to today's world.
• 33:18
  I can go a bit deeper into that but with a differentiation that
• 33:23
  not everyone has these end-to-end like machine learning
• 33:27
  and understanding of what machine learning prototyping is
• 33:31
  all about, right? So we're looking for essentially
• 33:33
  an engineer who Who gets machine-learning that's like, in
• 33:37
  the in the nutshell, what we were looking for.
• 33:41
  Cool. So, I just saw, we got another
• 33:43
  question from Sebastian, and it's more on this Pipeline.
• 33:47
  And so, I want to complete that before we jump into this hiring
• 33:51
  peace. And Sebastian is asking how much
• 33:54
  of this machine learning pipeline?
• 33:55
  Were you able to automate? That's a mmm.
• 34:03
  So I guess it's automated in the sense that you know keep any
• 34:10
  teas from the kind of engineering point of view.
• 34:13
  It's automated through que Bonita.
• 34:15
  He's like I guess from our side when we want to would be nice to
• 34:20
  be able to perhaps automate retraining of models once
• 34:27
  accuracy Falls below a specific threshold.
• 34:30
  That's not something we do today.
• 34:33
  I think that would be a really cool use case again, when we
• 34:35
  start getting scale and when we are in a place where we can, we
• 34:41
  can establish data pipelines that update training data, kind
• 34:46
  of more automatically. So yeah, so I guess if the
• 34:52
  question is around, you know, what happens when you want to
• 34:54
  deploy a new model? Yeah, sure.
• 34:56
  It's manual. But then I guess it's taken care
• 34:59
  of continuous like, Deployment continuous integration and the
• 35:04
  training is is that the moment automatic and the deployment is
• 35:07
  automatic through Humanities and get Ops through the gate, Ops
• 35:12
  kind of continuous deployment workflow that we use.
• 35:15
  I don't know if that answers the question.
• 35:19
  I'm just a thought here, Alex, it might be nice to share a
• 35:22
  little bit with the group here about how the GitHub Circle CIA
• 35:30
  dot science pipeline works, and kind of how that how you plug
• 35:33
  that together because I thought that was a really cool piece of
• 35:35
  engineering that you did. And I think that kind of answer
• 35:40
  when I spoke to the automation, question, kind of pushed again?
• 35:46
  Yeah, yeah. A big, a big.
• 35:47
  Shout out to my colleague, Luca Paolini, who was kind of who
• 35:50
  took the lead in creating that, that integration.
• 35:55
  So I guess at ruler we want to we're at the size where we want
• 36:01
  to stick with our existing kind of deployment patterns across
• 36:05
  all our services. So we were trying to find a way
• 36:08
  where we can like be able to deploy machine learning services
• 36:13
  in pretty much the same way as we deploy any service.
• 36:18
  So currently we use Circle C. I A trailer which which is our
• 36:24
  kind of build and deploy to And I guess we use dot science as
• 36:33
  discussed before like to give us that reproducibility and that
• 36:39
  kind of collaboration Dimension that we're looking for and
• 36:44
  essentially it was it was not that tricky at all in the end it
• 36:48
  was we used the dock science, kind of python Library we to to
• 36:54
  annotate our existing machine learning python workflow.
• 36:59
  So that happened at the application Level and when we
• 37:03
  deployed, we essentially trigger the bunch of circles.
• 37:07
  Yeah, jobs and build, build, build the surface.
• 37:11
  Then we had a train and save flow within certain CI.
• 37:16
  So of that, that's the most interesting bit.
• 37:18
  So the trend say flow, essentially, spawned off a job
• 37:22
  in our dedicated kind of dot science cluster on the dot
• 37:25
  science. Runner That fetch data from an
• 37:30
  S3 bucket run, the machine learning workflow.
• 37:36
  And essentially we could then review the accuracy and
• 37:41
  performance of that training process.
• 37:46
  And then there was a bit of a manual step where we had to go
• 37:48
  back to Circle CIA and manually approve, the next step, which is
• 37:52
  the deployment phase. And the after that, it was all
• 37:57
  in que Bonitas territory in terms of deployment.
• 38:00
  So it was pretty simple. I think, in the end, the moving
• 38:03
  parts are, you know, the SV S3. Buckets kind of the dot science,
• 38:08
  cluster, and circle CI. Hmm.
• 38:11
  And the middle orchestrating the the communication And the, I
• 38:17
  guess we've been chatting with, with DOT science about the
• 38:21
  chance to automate that flow even further.
• 38:24
  So I guess now there's the option to actually get a web
• 38:27
  hook back from from the dog science kind of job.
• 38:32
  Telling you whether, you know, the job is finished and whether
• 38:35
  the accuracy of the, a custom rule, you can say where the
• 38:37
  accuracy is above a certain kind of limit.
• 38:41
  Yeah. So that's the high level, what
• 38:43
  we're doing. Nice cool.
• 38:46
  So pivoting back now into this hiring and what was going on
• 38:52
  there. I thought it was really
• 38:53
  interesting in your blog. How you talked about the third
• 38:57
  wave of data scientists? Can you go into that a bit more?
• 39:02
  Yeah. Yeah.
• 39:03
  I think I'm in a position where I can answer this as well
• 39:07
  because I feel I've been part of like every kind of wave if you
• 39:12
  want to call it that. So I think this all starts to
• 39:17
  make sense after around 2011 or something especially like, okay,
• 39:24
  we always had like the Pioneers that Google Facebook Twitter
• 39:29
  whether we were like early adopters in early like like you
• 39:33
  know they adopted machine learning first and were like on
• 39:39
  the Forefront but I guess the rest of the world was a little
• 39:41
  bit using various proprietary kind of tools doing very similar
• 39:46
  work but was kind of all locked in and and pretty like not
• 39:50
  really it was an opaque kind of kind of world and I guess that's
• 39:54
  the first wave in a nutshell and then this big data like
• 40:01
  Revolution, And particularly where I think the second way is
• 40:06
  kicked in was jupyter. Notebooks So that that was just
• 40:10
  a profound moment in the dots either data science like the
• 40:15
  science like community and it just opened up like Python and
• 40:21
  like data science to to the whole world.
• 40:23
  It was like this, this is This brilliant moment and everyone
• 40:27
  should suddenly became. I would say including myself,
• 40:30
  everyone suddenly became a like a data scientist.
• 40:33
  And there was massive, like excitement, massive hype of the
• 40:37
  like possibilities that machine Earning and data science can
• 40:40
  bring to to Enterprise to business.
• 40:44
  And I think we were rolling in that kind of wave for maybe
• 40:50
  three or four years. But in the end, I think business
• 40:55
  is didn't really know how to extract business value.
• 40:58
  A lot of business, value out of out of kind of data science,
• 41:03
  jupyter notebook, data science, as it was back then.
• 41:07
  So you ended up with situations where yo you know it was just
• 41:10
  like shelf where PowerPoints things that were created in a
• 41:13
  vacuum and were never put in production and I think there was
• 41:16
  a there's a lot of that in business during the second wave
• 41:20
  so it was kind of a hike time but Find where?
• 41:24
  Business value wasn't optimal extracted and I think the lesson
• 41:28
  learned there was like the data scientists cannot just be people
• 41:32
  who know, build, you know, right, jupyter notebooks in a
• 41:35
  vacuum. They have to be involved in the
• 41:37
  end to end process of actually, prototyping something and then
• 41:41
  ensuring, that gets pushed into production.
• 41:44
  And I think there's a recognition amongst like
• 41:47
  businesses nowadays, That this is kind of, this is kind of a
• 41:52
  given. This is kind of a requirement
• 41:54
  nowadays for the data scientist or she'll engineer, whatever.
• 42:00
  Kind the kind of jobs. So, yeah, working in a vacuum
• 42:05
  and jupyter notebooks is not an option, I think anymore and the
• 42:08
  whole industry is coming to that realization.
• 42:11
  And I think that's the third wave of like the full stack kind
• 42:14
  of data scientist, great it now talk to us.
• 42:19
  Us a bit about this idea of the different mentalities between
• 42:24
  machine learning engineers and data scientists.
• 42:29
  So I think data scientists are should be used to fail a lot,
• 42:34
  right? So it's So yeah, so yeah.
• 42:37
  So I think like they've scientists are specialists in in
• 42:41
  failure and I mean that in a great way, I'm joking.
• 42:43
  So I think data science, I do see a distinction.
• 42:47
  So I think data scientists are basically Specialists and
• 42:51
  experimentation. Which means that you're
• 42:55
  constantly testing hypotheses constantly, you know, not
• 43:00
  validating your hypotheses and then you have this mindset, you
• 43:04
  know, you have to keep pushing on and in the end you might not
• 43:07
  even like get an answer, the answer that you want for, right?
• 43:10
  So in a way you're you need to really like tolerance to
• 43:14
  failure, but on the other hand, I think that.
• 43:17
  And that's this is just my point of view, right?
• 43:19
  I think machine learning engineer is kind of Intolerance
• 43:24
  like to does not should not tolerate failure.
• 43:27
  I think machine learning Engineers are more more about
• 43:30
  getting something like and deploying it and putting it out
• 43:34
  to the world, right? So this is a process that
• 43:36
  doesn't shouldn't go wrong. This should be a process that
• 43:38
  you should strive for Perfection.
• 43:40
  It's something that you can do machine learning Engineers,
• 43:44
  should be able to like create a model that gets you 80% of the
• 43:47
  way, but should be able to put it into production like And it
• 43:53
  should work like that. That is not a failure that the
• 43:55
  machine learning engineer can tolerate So in that case sense,
• 43:59
  I guess the highest level distinction is like focus on
• 44:03
  experimentation on the data scientist and focus on
• 44:07
  end-to-end like function functionality on the machine
• 44:12
  learning engineer site. And I think that I'm from, like
• 44:16
  data scientist in machine learning Engineers have come
• 44:18
  across in my career. I think it's difficult to be
• 44:21
  both like to Again, people argue that pretty much the same and I
• 44:29
  do kind of agree, but I think that mentality of like,
• 44:31
  experimentation of like very focused.
• 44:35
  Experimentation does not translate very well to machine
• 44:38
  learning engineers. And people may have a different
• 44:40
  opinion on that. Yeah, that's a great summary of
• 44:45
  it. So Thomas has a question here in
• 44:47
  the chat. He's asking, do you expect this
• 44:50
  distinct these distinctions were merged together in the near
• 44:54
  future or and there will be another kind of position.
• 44:58
  Well what do you predict and how it will evolve and say 10 years?
• 45:04
  And this was this kind of is is one of the questions I wanted to
• 45:07
  ask you to is, do you feel like there's a difference between a
• 45:10
  machine-learning engineer and then Ml Ops person is there and
• 45:16
  if there is anything, yeah. So that.
• 45:20
  Yeah, I think the, the names like nowadays.
• 45:23
  Yeah, I don't know how much sentence like names make
• 45:26
  anymore, to be honest. You know, you go into a company,
• 45:30
  you have a specific, you know, or maybe nonspecific, but you
• 45:33
  kind of work on stuff. And then you kind of decide
• 45:36
  whether you know where your machine learning engineering
• 45:39
  data scientist or, I don't know. So to be honest I think names
• 45:43
  will definitely evolved, but I also think, like, and you guys
• 45:49
  have been saying this to, I think, you know, the role is
• 45:51
  evolving and it's becoming a little bit more.
• 45:53
  Well defined. So like, the machine learning
• 45:56
  engineer think. Yeah, it's real.
• 45:57
  It's all about like end-to-end stuff.
• 46:00
  So so yeah. Now going to your machine
• 46:05
  learning engineer versus ml Ops I would hold my hands up there
• 46:09
  and say, I don't know. Yeah.
• 46:13
  Production-wise, I think the data scientist thing is not
• 46:16
  going to be around forever. I think like it's becoming a
• 46:19
  little bit. I think there's so much baggage
• 46:25
  associated with this kind of role name nowadays.
• 46:30
  There's a lot of damage as well like to the To what it means to
• 46:34
  be that scientists such an umbrella term, and I think,
• 46:37
  yeah, definitely will be broken down into the into things that
• 46:41
  make no sense. Where do you see it evolving
• 46:43
  into just more more specific terms?
• 46:48
  Yeah, I think, I think we'll probably go back to the more
• 46:53
  basic stuff like, So I think I think data scientist might might
• 47:00
  become like what it wants was so go a little bit, it will make
• 47:04
  sense for companies to do R&D and do a lot of like
• 47:09
  experimentation because it's science roll, right?
• 47:13
  I think things will move more towards the the engineering
• 47:18
  naming side. So I think like the machine
• 47:21
  learning or the, I don't know like data.
• 47:26
  They tension during like, kind of perturbation will become more
• 47:31
  prominent compared to data scientist.
• 47:35
  Yeah, I don't know. That's not a great answer but
• 47:38
  kind of. Yeah, it's perfect.
• 47:40
  It's perfect. And so, I'm wondering about this
• 47:45
  because you spoke about with, like a ml engineer needs to be
• 47:50
  able to work with this end-to-end and end-to-end.
• 47:54
  These days as a very big buzzword.
• 47:57
  And in your blog post you, you spoke about random buzzwords
• 48:01
  that companies will put out when they're hiring for an ml
• 48:05
  engineer. Can you go into that a bit more?
• 48:11
  You mean in terms of a buzz words who are like, yeah.
• 48:14
  And just, and how you feel about, what, what that entails
• 48:18
  because it's like companies will just throw a sheet of Buzz words
• 48:22
  that they want but they don't really know what they're
• 48:24
  actually looking for. Yeah, it's like the trophy they
• 48:28
  to scientist like phenomenon that we encountered like, you
• 48:33
  know, some guy and some, some person in the business like
• 48:38
  decides. Oh, I want a data scientist like
• 48:41
  I want to look cool to my boss. How do I do?
• 48:44
  I actually need that scientist. Who knows?
• 48:47
  What is the date scientists need.
• 48:48
  Oh, we need mongodb or we need the spark.
• 48:53
  They need. I don't know, like, IBM Watson.
• 48:56
  All they need I do and it's it ends up like being in these like
• 49:00
  job specs and you can really easily understand like whether
• 49:03
  the people you know whether the business is serious about who
• 49:06
  they want to hire just by looking at jobs back nowadays.
• 49:08
  And I do encourage people to really take Joe space, like with
• 49:12
  the critical. Oh, I like ask themselves does
• 49:16
  this really make sense? So, if I do join this company do
• 49:19
  they know what they expect from me?
• 49:21
  I think this is a real tell, but yeah, Buzz was really
• 49:26
  entertaining like a few years ago, especially when we're, it
• 49:29
  was all like the wild west. I wish I will.
• 49:33
  I have, I like kept like job, like specks and I could make
• 49:37
  maybe, you know, a little website.
• 49:38
  They're a little collection, but yeah.
• 49:41
  I vote buzzwords like avoid Shopping lists.
• 49:44
  Like if you're both and got specs like just keep it like
• 49:48
  keep it concise and keep it like logical.
• 49:52
  No unicorns. Okay there are some unicorns but
• 49:54
  you're not. Yeah you know I mean yeah yeah
• 49:57
  yeah completely and now it looks like we got about five minutes
• 50:01
  left. I want to make sure that
• 50:03
  everybody feels like they're questions.
• 50:05
  Got answered. So if anybody has any questions
• 50:08
  for Alex, please throw them in the chat or just unmute
• 50:11
  yourself. Feel free to say hi and talk and
• 50:17
  if not I'll wait, you know, a good five seconds and if not
• 50:20
  I'll ask my final question. I've got one of the main
• 50:24
  organizers. Here we go.
• 50:26
  Go for a Charles. Sweet.
• 50:29
  And so analogues to me, I always hear it and think of devops and
• 50:33
  obviously we had aged in the engineering community
• 50:36
  desperately trying to make it so that we have this really
• 50:39
  rigorous way of ensuring that the artifact that I've made on
• 50:42
  my machine is the same artifact that appears in Dev and then it
• 50:45
  business staging that it bears and product.
• 50:46
  In other words, I get no surprises.
• 50:48
  There's nothing different about what's going in prod and that's
• 50:49
  goes all the way through to like configuration as well.
• 50:51
  You want to make configuration changes As you push it through
• 50:53
  the pipeline like in the ml World it feels like that's a
• 50:57
  slightly different Paradigm. Always feels like that's the
• 50:59
  Paradigm. You don't want to follow because
• 51:00
  you want to be changing that that mlrs fact in production is
• 51:05
  DDC like a totally different approach to like operations
• 51:11
  deployment operations changes in production, to what we've
• 51:14
  developed in the world of devops in engineering.
• 51:18
  So I think the I guess the answer I think is that these
• 51:22
  things are merging. So the engineering community
• 51:25
  really learnt really faced these problems, like 20 years ago.
• 51:29
  Well maybe maybe before that, I'm not entirely.
• 51:31
  Sure. And I think we're just the past
• 51:35
  like few years, we've been trying to solve them really well
• 51:38
  in machine learning. I think we will arrive at
• 51:41
  similar, similar Paradigm and I think Version Control and
• 51:45
  provenance is obviously like like Central essential to that
• 51:49
  but But yeah, I guess this is a very high-level answer, but I
• 51:54
  think we will, we will be maturing to something that looks
• 51:58
  similar to what we have in engineering and I guess Luke is
• 52:01
  someone who's a bit of a folk leader as well on that subject.
• 52:06
  So feel free to jump in to look most famous.
• 52:08
  I have opinions on this which is just that most of these cases
• 52:13
  that I've seen as a you are not doing online learning.
• 52:18
  Most of these cases. I've seen a the model isn't
• 52:21
  Changing in production and you maybe retraining the model very
• 52:24
  frequently based on new data but your you should still have an
• 52:28
  immutable artifact. That is a Docker container with
• 52:32
  a model and a model server running in production and you
• 52:34
  should still version those and have full provenance from that
• 52:40
  model, back to where it came from the data, it was trained
• 52:43
  on, and so on. And in that sense, I think that
• 52:46
  it should be converging on exactly the same solution that
• 52:50
  devops Has achieved except there's, it's more complicated
• 52:53
  because there's more moving Parts like like data, versioning
• 52:57
  and parameters and metrics that need to be recorded and along
• 53:02
  the way. That's just yeah, that's my two
• 53:05
  pence. Nice, thank you.
• 53:09
  Luke awesome. Anybody else, got anything they
• 53:13
  want to chime in and ask any questions.
• 53:16
  My final question is qualities that you feel are important for
• 53:24
  machine learning Engineers, you kind of touched on this.
• 53:27
  But I just wanted to reiterate that Yeah, I mean I think I can
• 53:35
  speak for my own experience, like I work at a startup.
• 53:37
  So I mean you don't really know 100% what you'll be working on
• 53:42
  the next day, right? So in my mind, like, I think
• 53:46
  number one is like the ability to pick up new stuff like really
• 53:49
  quickly and I think that is a skill.
• 53:52
  So I think that's a skill that comes from constant like A
• 53:57
  passion for self-improvement, and the passion for trying new
• 53:59
  new stuff out like that. Your job might even not be
• 54:02
  giving to you, or your current environment might not be giving
• 54:04
  to you. That might include like
• 54:07
  contributing to open source and my contributing to might be
• 54:10
  sorry, that might be, you know, setting up a webcam with like, a
• 54:15
  deep learning model deployed on AWS that identifies.
• 54:19
  Like the Fox's of coming to your kind of garden and and eat your
• 54:25
  like plants. They don't think plans.
• 54:27
  But anyway. Yeah, you see I mean, so like
• 54:30
  it's this constant like strive for like learning new stuff.
• 54:34
  I think that's where that comes from.
• 54:36
  So I think that's and, like, machine learning.
• 54:38
  I think we have, we're blessed to this amazing Community
• 54:41
  nowadays, and I think having that passion that like,
• 54:44
  involvement in the community, even a voice in that community.
• 54:48
  So we know an attempt to put your thoughts out there.
• 54:51
  Log one would say, everyone does that, but really it's just shows
• 54:56
  like, no, you really want this. You really, you really serious
• 55:00
  about if you're not already in machine learning or getting into
• 55:05
  machine learning like it's so important.
• 55:08
  So, I think. Yeah, so generalist, I would say
• 55:11
  generally it's like Computing, like clouds some databases,
• 55:17
  obviously, like a passion for machine learning, but not to the
• 55:20
  extent that You're going to be implementing necessarily like
• 55:24
  the latest paper. It's all about I think, at least
• 55:28
  in a start-up environment. It's a lot about the generalist
• 55:30
  stuff, but that's just my where I come from.
• 55:36
  Awesome Alex, this has been so informative.
• 55:39
  I am so appreciative that you got on here and you talked with
• 55:43
  us if anyone wants to reach out to you.
• 55:45
  What's the best way to find you? Yeah, I'm really open like
• 55:49
  Twitter LinkedIn. I guess my details were there.
• 55:53
  Yeah, feel free to hit me up. I guess I have to say here that
• 55:56
  we're always hiring for like amazing, like back-end engineers
• 56:01
  and we're always interested in hearing about like machine
• 56:04
  learning engineers and product managers, but yeah, actually,
• 56:09
  we're hiring across the board. So feel free to reach out about
• 56:13
  any of those roles and check out.
• 56:15
  What should I do? It is folks Alex.
• 56:20
  I appreciate it. Everyone that came on and shared
• 56:24
  this time with us. Big, thank you.
• 56:26
  I'll be sending you a follow-up email with some links of
• 56:29
  interest and have a great rest of the week.
• 56:33
  Take care, everybody. Thank you.
• 56:35
  Bye everybody. Thanks Alex.
• 56:38
  All right guys. It is folks Alex.
• 56:20
  I appreciate it. Everyone that came on and shared
• 56:24
  this time with us. Big, thank you.
• 56:26
  I'll be sending you a follow-up email with some links of
• 56:29
  interest and have a great rest of the week.
• 56:33
  Take care, everybody. Thank you.
• 56:35
  Bye everybody. Thanks Alex.
• 56:38
  All right guys.