Latest / Agentic Conversations (formally mlops.community) / What is Data / ML Like on League? // Ian Schweer // MLOps Coffee Sessions #132

Transcript

What is Data / ML Like on League? // Ian Schweer // MLOps Coffee Sessions #132 · Agentic Conversations (formally mlops.community)

• 0:00
  Yeah. Hi.
• 0:01
  I mean I'm a senior software engineer at riot games working
• 0:05
  on the League of Legends data team.
• 0:08
  And I am a huge espresso guy. I love Columbia roasts, and I'm
• 0:14
  a bit of a coffee snob. So, find me at your local La
• 0:19
  coffee place. And let's have a coffee
• 0:20
  together. You know what?
• 0:23
  The first question is going to be, what's the first question,
• 0:29
  dude? What the hell?
• 0:30
  Is this? I thought you were joking.
• 0:31
  When you said that, you actually really are into the Mean Girls
• 0:35
  movie and then it turns out before we hit record, you are
• 0:38
  telling us all about a party. You organized all around me and
• 0:42
  girls and you do that every year.
• 0:43
  So this wasn't like a joke. You weren't lying about this.
• 0:46
  Yeah. And I'm very serious for like
• 0:49
  the last 10 years, I've had a mean, girls party every year on
• 0:52
  October 3rd and this year was the most extravagant.
• 0:56
  We got a house on the beach. There's probably Roughly like 25
• 1:01
  people there. So we had a bunch of games like
• 1:05
  we had Pictionary where prompts were like based on the movie
• 1:08
  there's like conversation starter cards based on like
• 1:12
  quotes from the movie, there was quotes that we hit around the
• 1:14
  house from the movie. It was like a guest book where
• 1:18
  you could take a Polaroid photo and like, write your own burn
• 1:21
  book entry for those of you familiar with the movie.
• 1:23
  So yeah, it was really over the top, a lot of fun.
• 1:26
  Yeah. You went all out, man.
• 1:28
  I am very surprised. And the level of dedication, you
• 1:32
  show do this. And what kind of people show up
• 1:35
  to this? What 25 people in l.a. did you
• 1:37
  call an indigo and are they all die hard fans like you?
• 1:41
  I mean, you know, any 20-something white woman is
• 1:46
  going to come to that and turns out l.a. is full of them.
• 1:53
  So, exactly, my people you had to cap it at 25 people.
• 1:58
  Huh. Well amazing today, I mean as
• 2:03
  much as I would love to chat with you about mean girls
• 2:06
  parties that you throw, we're not here to talk about that.
• 2:10
  We just got off of a incredible conversation with Ian, who is
• 2:15
  working at Riot games? As you all just heard.
• 2:17
  What were your takeaways man? What did you think about that?
• 2:20
  Yeah, you know, super interesting chatting with you
• 2:23
  and, you know, never really seen how am L might be used in sort
• 2:27
  of a gaming space and I'm not an avid.
• 2:29
  Bad gamer myself. So it's just like interesting to
• 2:32
  see some of the things that might be different, you know.
• 2:35
  Like in particular, there's like data sources.
• 2:37
  They have, that are kind of like, records of the game that
• 2:41
  they're able to use and it's like fully baked.
• 2:43
  Oh, that's so good. You don't have to go through
• 2:45
  this process of trying to aggregate a bunch of different
• 2:49
  data sources to like, you know, create a normalized view.
• 2:53
  Like a lot of the typical work, the data Engineers do.
• 2:55
  It's just like a nice snapshot of everything that happened and
• 2:58
  so that's a very interesting difference.
• 3:00
  Read some. Also just interesting to, you
• 3:02
  know, look into like the challenges of integrating with
• 3:06
  Legacy systems where it's like, hey, we have like this 10 year
• 3:08
  old C++ code base. I'm afraid to touch that, you
• 3:13
  know? So, yeah, so just very
• 3:16
  interesting to see how things are a little bit different, but
• 3:18
  also how there's a lot of overlaps and a lot of similarity
• 3:21
  with, like, all of the other work that I've been familiar
• 3:24
  with mmm-hmm. Yeah, it's first some context
• 3:27
  for everybody that's listening. Ian came on first to the pancake
• 3:31
  Stacks meet up that we do once a month and Skyler leads and he
• 3:35
  broke down the right game stack in so much detail.
• 3:39
  It was incredible. I loved it.
• 3:41
  And I loved it because he didn't just give us slides.
• 3:44
  He what he had like a mirror board and you could dig into any
• 3:48
  specific piece that you wanted. And so that was really cool
• 3:52
  because we got to kind of choose our own adventure and ask him
• 3:55
  about this stuff and After that I was yeah I was just really,
• 4:02
  really Blown Away by it and I thought well, let's try and get
• 4:05
  him onto the podcast. See if he'll do it.
• 4:07
  And we lucked out, he said he would do it.
• 4:10
  And also a little tidbit little side note.
• 4:14
  You guys went to the same College, what's this?
• 4:16
  You guys knew each other back in the day?
• 4:18
  Yeah, yeah, I think, you know, I think we only had like one class
• 4:21
  but I was highly involved in like all the clubs and
• 4:24
  universities. So we cross paths a lot, but
• 4:27
  yeah. Didn't we actually didn't really
• 4:29
  keep in touch and when I moved back to LA, I'll starting like
• 4:32
  the the local and lost meet up. I realized he was like working
• 4:35
  in the space. I had no idea and so like we
• 4:38
  reconnected so it's super cool to get them integrated in the
• 4:41
  community a lot. Not dude, so it's becoming like
• 4:46
  a six degrees of Skyler, pain type scenario.
• 4:50
  Now, anyone that is working in MLM tips.
• 4:53
  You can connect to Skyler, but if I at least Six degrees.
• 4:59
  I I love to meet people so they very I'm trying to I'm trying to
• 5:03
  get everybody's pain number down to one or two, you know?
• 5:08
  Incredible so that's it. We're going to jump into this
• 5:12
  conversation. Enjoy everyone.
• 5:14
  If you are listening and you enjoy the conversation, like
• 5:18
  always, I would ask Skyler and I both are politely asking you to
• 5:25
  hit the like button leave. Us a review.
• 5:28
  If you're listening on Spotify or apple, just anything you can
• 5:31
  do to let the algorithm know that this is awesome podcast and
• 5:36
  more people should listen to it. That would mean the world to us.
• 5:38
  All right, let's jump into it. Awesome.
• 5:46
  Great for all kinds of deep questions about life and the
• 5:49
  meaning of it definitely, dude, I didn't sign up for that kind
• 5:54
  of podcast. What the hell?
• 5:55
  Do you think this is Rick Ross here for?
• 5:59
  Oh yeah, Skyler, and I share a Skyler and I share a like, Keen
• 6:05
  awareness of mental health in like the engineering community.
• 6:09
  So, like I would not be disappointed if it It went that
• 6:12
  direction either. Wait, wait, unpack that a little
• 6:15
  bit more. Why why?
• 6:17
  I know Skylar? I know why Skylar does, but
• 6:21
  where's that come from from you? I mean, I think it's just like I
• 6:28
  I think Skyler, you also grew up in you grew up in like somewhere
• 6:32
  else like also middle California.
• 6:34
  Kind of desert land ish. Yeah, I come from similar area
• 6:40
  like, in the middle of California or middle age of
• 6:42
  California wasn't much going on. There's a lot of like Macho
• 6:46
  machismo expectations of you and like I guess like as you as you
• 6:52
  kind of get older, you maybe have to start reflecting on that
• 6:55
  about a lot more to understand like who you are think about.
• 6:58
  And like you know, for me like a lot of a lot of hiding under
• 7:03
  like all I'm going to build like super cool stuff.
• 7:06
  I'm gonna build like, you know, like I'm going to be like the
• 7:09
  best engineer. A lot of people have ever met
• 7:11
  like that kind of like needing to prove yourself but I think
• 7:14
  something that bowls that that we may be both shared just like
• 7:18
  gone through different paths. So that's like what where it
• 7:23
  comes from for me probably isn't too dissimilar from like the
• 7:27
  shape of Skyler story, but all the details are definitely very
• 7:30
  different and what was your way out of that?
• 7:36
  Or how did you cope with that? I mean, I went to, by the time I
• 7:41
  got by the time I went to like University and like, got out of
• 7:45
  got out of the deserts. That was pretty much what, what
• 7:49
  needed to start with me. So that and then like, you know,
• 7:52
  therapy and then just like, also just enjoying kind of enjoying
• 7:57
  the career as opposed to Trying to one-up everybody which is
• 8:02
  like a weird like, mental change that happens.
• 8:04
  That those are probably things that allowed me to like fully,
• 8:08
  leave the toxic stay behind and Venture, because it's
• 8:13
  interesting because like, I think that's one of the things
• 8:15
  that makes Riot like a really interesting place to work at,
• 8:18
  especially as an engineer. Just because like, they are very
• 8:23
  kind of keenly aware of, you know, like Riot has had a lot of
• 8:29
  history with like Somatic characters.
• 8:33
  So I think they're very keenly aware of like how you know, like
• 8:39
  a like a like a Bro e culture could burn people out or how
• 8:42
  like burnout culture can lead to like a bunch of other negative
• 8:45
  things so they tried to really kind of encourage all their
• 8:48
  employees to like take care of the mental health they have a
• 8:52
  lot of like extra benefits for like therapy or anything like
• 8:57
  that. So I think right?
• 9:00
  It is probably the most supportive company I've been at
• 9:03
  that. Like cares about like everyone's
• 9:07
  kind of like mental well-being, which is kind of a funny
• 9:11
  happenstance because I didn't go there intending on that.
• 9:13
  It's what they were because I was like, aw dude?
• 9:15
  Right super sick. Like they have free coffee, man.
• 9:18
  Let's after being there, you kind of get more of that.
• 9:25
  I think it's like quality of quality of life stuff.
• 9:29
  How much has your video game playing increased since going to
• 9:35
  work at riot? It's definitely gonna, it's not
• 9:42
  uncommon for people to like, play a game of League or t of T
• 9:46
  together on a Friday. Yeah, it's definitely not
• 9:51
  uncommon to go into the office and to see people playing games.
• 9:55
  We have like, you know, the PC bang on campus so you can like
• 9:59
  play games together. We have a bunch of like arcade
• 10:02
  games, and we have a like a console room of a bunch of old.
• 10:08
  Old consoles for those who like to play like older games and
• 10:14
  stuff. I'm I have been getting into
• 10:18
  like playing more retro games ish and like I use the word
• 10:22
  retro really really in quotes. A what does that mean like Duck
• 10:26
  Hunt? Know that, that I think that's
• 10:28
  like true true retro. But like, okay.
• 10:30
  There's like the the dot hack series on PlayStation 2, which
• 10:36
  is like a game. I didn't play except Like
• 10:38
  growing up my friends house, all the old final fantasies and the
• 10:41
  single digits Davis, ex, like the original one.
• 10:47
  Just like a lot of those kind of games have been given a lot more
• 10:51
  love to. Now that I have that, I see a
• 10:54
  lot of other people doing that at work, like, I've seen, I've
• 10:58
  come to work before, and I've seen dudes who just are playing,
• 11:00
  like, subnautica for like three days.
• 11:02
  They're, like, really into it. Like, oh my God, I gotta, I
• 11:06
  gotta do the thing. I got to build this super sick.
• 11:08
  Underwater base, everything else can happen later.
• 11:13
  Like it's it makes the I don't know it's very relaxing for me.
• 11:17
  Yeah I believe it goes back to what we were talking about man.
• 11:22
  So now jumping into your career and a little bit of the career
• 11:26
  progression, you went from software engineer to data
• 11:28
  engineer at Adobe, right? Yeah.
• 11:32
  I I started at I started at Adobe as, like a co-op kind of
• 11:41
  deal. They were doing it.
• 11:42
  Where, like, I was working 20 to 30 hours, while going to school
• 11:48
  and then, like fold summer internships during the summer.
• 11:53
  And I worked with a team, they're doing kind of.
• 11:58
  It was almost closer to like, Consulting kind of role.
• 12:02
  I would work a lot with Adobe customers.
• 12:04
  So, Build their like Solutions and that was all in the video
• 12:07
  tech space. I went to data engineering.
• 12:15
  Around like the 2018 Mark and I think back then and still kind
• 12:22
  of now, Adobe still kind of calls like some roles of a data
• 12:26
  engineer more like the traditional like sequel, kind of
• 12:30
  data warehousing roles and then there are data Engineers who are
• 12:34
  more focused on like Automation and that kind of stuff.
• 12:37
  I kind of got to follow in the latter half of that.
• 12:39
  So I was working on a lot of tooling for like actually a lot.
• 12:45
  Of the stuff that you hear, you know, like DBT trying to solve
• 12:49
  around like metrics layers, or like, even even some of the
• 12:53
  Realms of like observability data observability.
• 12:58
  And we were building like our own custom sack from the ground
• 13:02
  up on top of like zookeeper. So like we were building our own
• 13:05
  orchestration engine, we're building like Our own extensions
• 13:10
  on top of postgres, to be able to do like up certs and all this
• 13:14
  crazy stuff. So that team was going like
• 13:18
  super hard in the realm of like building the infrastructure kind
• 13:22
  of from the ground up a lot of it didn't work, I think, but it
• 13:27
  was it was good for me like early career because I got to
• 13:31
  like spend a lot of time thinking about like how to work
• 13:34
  station engines work. Like what what are the
• 13:36
  challenging things that airflow is doing?
• 13:38
  That I might not understand like going into it at first from
• 13:44
  there. I went to doordash and I think
• 13:46
  you guys interviewed he in maybe also on this podcast.
• 13:51
  Yep. So so there I was at doordash I
• 13:55
  was I was there for a little bit before he inserted and then he
• 13:58
  and started like around my eight month Mark I think and he over
• 14:04
  there it was just like kind of a like explode In the work I was
• 14:09
  doing like anything from like data privacy stuff with like
• 14:14
  CCPA and like getting all that kind of stuff out on the data
• 14:18
  warehouse Lair to like Cassandra replication.
• 14:22
  So it wasn't really a good answer for that at the time.
• 14:26
  So we were like building our own Flink streaming applications on
• 14:32
  top of like the bzm event streams.
• 14:35
  If you go back into like the bees iam Indra like in the
• 14:39
  Discord and stuff like that. I was like, I was in there when
• 14:42
  there were like a hundred people or something asking, like, how
• 14:45
  do you build the freaking jar? Like, how can I ship this?
• 14:49
  Yada yada yada because we were trying to solve that, that kind
• 14:51
  of stuff. So, I was working on that and
• 14:54
  just like a plethora of other stuff a lot of, like, container
• 14:57
  and devops you stuff, too. And then I came to, right?
• 15:01
  Kind of the start of the patan, pandemic.
• 15:03
  We back down to Southern California.
• 15:07
  Yeah. And then out here, Kind of a
• 15:09
  variety of kind of spent a little bit of time in kind of
• 15:13
  the game build space as the team I joined, when I initially came
• 15:16
  to ride, it was like a sorry game build.
• 15:19
  And then 1 over Z to data is kind of where my feel like I
• 15:23
  belong. That's like the that's like my
• 15:25
  SME but lots of funny things happen.
• 15:28
  When I was like going through the interviews for data like the
• 15:31
  role filled as I was doing the on-site interview.
• 15:35
  So they were like, well you passed the on-site, we see.
• 15:38
  I want you to come to Riot but the roles not here anymore.
• 15:41
  So you're going to go to that team as like heck, yeah, let's
• 15:45
  go. So awesome.
• 15:49
  So yeah, now I'm on the now. I'm on the data, I guess, like,
• 15:52
  we're kind of like data Ops or like ml Ops.
• 15:56
  So I guess some of the stuff I skimmed over there.
• 16:00
  When I was at doordash, was we were also, There is a series of
• 16:07
  doordash blog posts about a system called Sybil, which was,
• 16:13
  which is kind of like the features or that that doordash
• 16:15
  is built and way back in like the 2019 era, the search team
• 16:26
  kind of the ml team and data infrared.
• 16:27
  We all work like really closely doing a lot of their kind of ml
• 16:31
  stack. So they were doing.
• 16:33
  Out of the like, like pure algo development if you will or like
• 16:37
  training and stuff, but our team was like heavily involved in.
• 16:41
  Just like, how do you, how do you maintain your EMR
• 16:45
  infrastructure? How do you tie this in with air
• 16:47
  flow? How are you going to do live in
• 16:49
  friends? You know?
• 16:52
  And I think they were they were looking at the problem of like
• 16:54
  how do we pull features down from Snowflake without having
• 16:57
  like snowflake being in the You know, in the core path of a
• 17:02
  request. So a lot of those like kind of
• 17:06
  operations questions around machine learning which stuff we
• 17:09
  were. We were all kind of thinking
• 17:10
  about and I was able to get to be a part of, in a couple
• 17:15
  different, like small projects and stuff like that.
• 17:19
  But I guess because we had like really proficient leaders or
• 17:23
  people who have been thinking in this space a lot, maybe I like
• 17:25
  got to absorb a lot more of it. So a lot of that is like coming
• 17:29
  out in the work. We're doing in on ALDC where
• 17:33
  we're thinking a lot about like, however, doing live inference
• 17:37
  like in a game. Binary, how do we do live
• 17:39
  inference? You know, post-games how do we
• 17:43
  do like better data like data ingestion systems and like data
• 17:47
  Ops is kind of like another round.
• 17:48
  They're really kind of exploring.
• 17:52
  The answer is kind of a career path.
• 17:55
  Yeah. So it's really interesting how
• 17:56
  you describe how, you know, coming into Riot you started out
• 18:02
  in reliability and, you know, I think this is a big topic on
• 18:06
  people's minds topical today, because twimble con is happening
• 18:09
  today, and, And Todd Underwood, Neil Murphy, gave like a
• 18:12
  keynote, talking about reliable machine learning.
• 18:16
  Definitely a big topic, and I don't think there's a lot of
• 18:19
  expertise. So, just curious to hear a
• 18:20
  little bit about whether you feel whatever you learned and
• 18:24
  did sort of like working specifically on reliability.
• 18:28
  Is there any like lessons you were able to like apply as you
• 18:31
  like move more into like this data and ml Ops taste?
• 18:35
  Um, maybe, I mean, I think the, to be honest, the biggest thing
• 18:40
  I kind of took away from, like, my kind of stint in SRE is that
• 18:46
  like a lot of the problems we're seeing in ml Ops and in data
• 18:51
  engineering, like in the 2020s, 2019, s, they're all kind of the
• 18:59
  same shape. Like, I very much see, I very
• 19:04
  much. You like metric metric, or
• 19:06
  feature stores that do like point in time.
• 19:09
  Calculations kind of like feather talks about not being
• 19:14
  too dissimilar to solving the problem of slowly changing
• 19:17
  Dimensions, which was also, you know, like can also be solved by
• 19:22
  like, some of the have been solved by some of the bigger,
• 19:25
  like data data companies like a dream Airbnb or whatever.
• 19:30
  But they were solving it by just like snapshotting everything and
• 19:33
  using like sequel. Rose to be able to say, like,
• 19:36
  oh, you want to understand, like, you know, how many people
• 19:40
  rented houses in Venezuela and the data needs to look like it.
• 19:44
  Did on March, 31st will don't worry, use the SQL macro and
• 19:47
  we'll be able to do that for you.
• 19:51
  And a lot of those problems are like things that Google SRE.
• 19:56
  We're kind of thinking about just in, like, a much different
• 19:58
  context back when they were solving it.
• 20:02
  Like, I will not say that like, Like service configuration is
• 20:06
  anywhere near similar to like, like data or like training a
• 20:12
  feature, but, you know, they're, they're also thinking about like
• 20:15
  what would a config store? Be how could we pull config out,
• 20:18
  you know, as it was and on, you know, as it was on New Year's
• 20:23
  Day Before the incident happened, how can we track the
• 20:26
  changes between this configuration and that
• 20:28
  configuration? And I think like the other thing
• 20:33
  it's kind of shown me is the kind of this like don't standing
• 20:39
  on the backs of giants. Kind of thing is like really
• 20:42
  true in a lot of these bigger companies.
• 20:46
  I don't think I realized it when I was at doordash but you know
• 20:49
  we had you know we had our own like Flink infrastructure, we
• 20:54
  had our own Docker infrastructure, we had our own
• 20:56
  way to like build and deploy containers and like kind of
• 21:00
  Unfurl them and Jenkins jobs to do, cool optimizations or
• 21:04
  whatever. And similarly, as I'm sure maybe
• 21:09
  you guys have experienced, but like I companies like LinkedIn
• 21:12
  or like uber, you know, they'll have like, oh this is our
• 21:15
  custom, you know, spark mapper system that we built from
• 21:20
  scratch, but it secretly relies on like 200 other Uber
• 21:24
  Technologies that help to kind of unlock its ability.
• 21:30
  And like as we come over to Riot or as they come over to Ryan I
• 21:33
  just I see that kind of everywhere and it just changes
• 21:36
  the perspective on how I like how I build and design things,
• 21:39
  you know, like we're looking at just to like kind of give an
• 21:45
  example where we're looking at, we do these, our team builds
• 21:51
  these recommender systems, which are just kind of your classic
• 21:57
  collaborative filtering models. The, the hitch being we created
• 22:05
  at compile time and we actually embed it into the game binary.
• 22:10
  So these kind of features are like, stored in the binary code.
• 22:14
  So like if you feasibly if you're able to like unpack the
• 22:18
  binary and like exploded is like a hex dump, you would see some
• 22:22
  decimal number buried underneath that is like our feature or
• 22:26
  something. And we're trying to Ask
• 22:29
  ourselves like how could we actually like, you know, pull
• 22:33
  this out and like obdurate and compile time and read it from
• 22:37
  like a service or from some configuration, or whatever and
• 22:42
  Riot has like six or seven different ways to solve, this
• 22:46
  spewed, across all the different games, all with different levers
• 22:49
  that you might want to tweak and twist.
• 22:54
  And it's just like man like the the people who built you know
• 22:58
  like it it just it tells the story so differently from people
• 23:02
  like the you guys recently had the guy who worked on feather
• 23:06
  for like six years or something. Before we go to open source and
• 23:08
  it's like, man, the like the Inception of that was probably
• 23:13
  like hey we can like sit on top of LinkedIn service, a and
• 23:17
  Linkedin service, be and Linkedin service.
• 23:19
  See and like offer this super cool thing and then six years
• 23:23
  later it's like Kind of generic Cloud agnostic tool.
• 23:29
  And I think that's something that like I want to see coming
• 23:32
  out of the ml space is like you know we have all these companies
• 23:37
  who are built on top of like all these Super Cool Tools, like
• 23:40
  moving them into the open source.
• 23:42
  Moving them into like more holistic packages, I think
• 23:47
  that'd be great. Like, you know, a hypothetical
• 23:50
  would be like any of the stuff we might do around like pie
• 23:53
  tour. Ouch and game compilation if we
• 23:56
  were able to like open source that and show like hey you want
• 23:59
  to do like these crazy embedded models, like Riot has solved it
• 24:03
  and you can install the package of GitHub.
• 24:07
  That's kind of the stuff, I think that that's mostly what
• 24:10
  I've learned from like the SRE Rome, Just because that's how
• 24:12
  he's been doing this kind of stuff for like 15 years now,
• 24:15
  right? Like, there were companies based
• 24:16
  around what whereas sorry practices that are now just like
• 24:21
  commonplace. Yeah, yeah.
• 24:23
  Definitely. You know, this, a think like one
• 24:25
  of the things you kind of getting at is there's a lot of
• 24:28
  people who have solved very similar problems and like a
• 24:31
  slightly different domain. And one of the things you often
• 24:34
  hear criticism, I think of like the it.
• 24:37
  Ops space is that like, hey, you guys need to read like database
• 24:42
  literature. We've been doing this stuff for
• 24:43
  a long time, you know, in particular, if you haven't read
• 24:46
  anything from Shreya Shankar, I don't know if I'm pronouncing
• 24:50
  her name. Right?
• 24:51
  Hopefully I'm not butchering it, but she has some excellent
• 24:54
  writing which talks about this. She's doing a PhD and database
• 24:57
  Management Systems now and it's cool to see the connections.
• 25:00
  You can have their yeah yeah. The super cool and definitely
• 25:05
  agree on sort of the, you know, These tools.
• 25:08
  You, you hit the nail on the head with feather definitely was
• 25:10
  something that like started out built on top of all these
• 25:13
  LinkedIn systems and so you had a decouple it.
• 25:17
  Yeah I would love to maybe kind of like switch gears a bit and
• 25:20
  talk a little bit about what you're doing Riot.
• 25:24
  You gave a great presentation to pancakes tax and had like a nice
• 25:27
  mind map walked us through all the architecture.
• 25:30
  And it looked like you were doing a lot of cool stuff with a
• 25:32
  lot of different tools. It seems like you know, I would
• 25:37
  love to just like maybe talk a little bit about you know, what
• 25:41
  are the things that you've built out and Riot?
• 25:45
  What are the challenges you solved?
• 25:46
  And you know, maybe challenges that are still open for you.
• 25:51
  Yeah, I think I'll also go ahead and like kind of re summarized
• 25:54
  like a little bit of the architecture just for anyone
• 25:57
  listening who didn't get to see my as you guys call it awesome.
• 26:02
  As I thought of it I'll shit. What am I going to present kind
• 26:05
  of deal. Um, so yeah, I think so.
• 26:09
  Riot is a big company. We have a couple, we were like a
• 26:12
  games as a service company. We have a couple games out.
• 26:17
  League is one of them League of Legends.
• 26:20
  It's like a MOBA game I'm only explain this part.
• 26:24
  There was a little bit of feedback during the last during
• 26:27
  the session where I was like I don't want to jargon people a
• 26:30
  bit too much so like just in the basic mechanic of the game.
• 26:34
  It's like a 10 second 10 player, game 5 e 5u each like pick a
• 26:40
  champion that kind of falls into a specific role within your team
• 26:45
  composition. These are all kind of based off
• 26:48
  of kind of the MOBA archetype. So if you're at all familiar
• 26:53
  with mobas, you'll know what that means.
• 26:55
  If not just think about it as like, where you're playing on
• 26:58
  the map and kind of like the activities you typically do and
• 27:07
  league is built on top of its own like custom game engine.
• 27:11
  That was developed. Years ago is developed by our
• 27:15
  Founders, kind of as a fork of Dota 2 or maybe it was the OG
• 27:22
  DOTA unfortunately. I don't remember if you're
• 27:25
  really super interested, there's like a Netflix documentary on on
• 27:28
  Riot, you can go and watch that and it'll fill you in and all
• 27:31
  the nitty-gritty details. But for our purposes.
• 27:36
  So we have this, this kind of old game engine, that's gone
• 27:41
  through. Like a lot of changes and And
• 27:43
  kind of gone through, you know. Like some we're on like patch
• 27:49
  release 12. I think we're like 12 1912 20.
• 27:54
  Now, we do like releases every two weeks.
• 27:57
  So we've been doing this for a long time and we and and the
• 28:05
  the, the the game is like an entirely online game.
• 28:09
  So, we have a suite of Mike background services, that kind
• 28:11
  of help support. The game league is mostly a Java
• 28:16
  shop. So we've got this kind of web of
• 28:19
  java services, that help, you know, help a player kind of
• 28:23
  login. See the client buy stuff.
• 28:28
  Get match-made against other people, build your team's all
• 28:31
  that stuff. Get you in the game and collect.
• 28:35
  Kind of end of game data summary and of game data statistics and
• 28:38
  stuff. So, you can kind of see how well
• 28:40
  you've been doing against all your friends and all that good
• 28:42
  stuff. So, From there, we kind of the
• 28:51
  data Journey starts I guess so Java systems, C++ systems,
• 28:55
  probably all you really need to know about the game
• 28:57
  architecture. And from there we kind of
• 29:03
  collect data in a couple different spaces.
• 29:05
  We do your typical like DB scrapes and your typical like
• 29:10
  Services send kind of audit Telemetry out to a Kafka that we
• 29:14
  then ingest You Oughta. Hopefully that's not too
• 29:17
  dissimilar to what a lot of other people are kind of looking
• 29:20
  at. We do have one really
• 29:23
  interesting kind of data ingestion scheme which is this,
• 29:29
  which is a problem. We're also thinking about which
• 29:31
  is this, like end of game data, I do.
• 29:33
  I'm not sure if you guys have experienced this but, you know,
• 29:36
  at least it like doordash, there was never a data set that got
• 29:40
  generated at the end of a delivery that fully describes
• 29:44
  the delivery, we generally had to like Ping a bunch of services
• 29:48
  and scrape, a lot of databases to kind of like, reconstitute
• 29:51
  the story. Because of like how online video
• 29:57
  games work, at least ours in particular with service server
• 30:02
  side. Authoritative decision making
• 30:06
  everything gets logged out in the game server and to like this
• 30:09
  kind of nicely compact Json file.
• 30:12
  So we kind of get this like fully referentially complete
• 30:15
  data file, and we can then like, send out to other systems to do,
• 30:19
  like, just like, I guess really kind of interesting, like,
• 30:24
  Player level and game level like aggregations without even having
• 30:28
  to hit the data warehouse. So from there, you know, we're
• 30:33
  doing we're doing some interesting stuff there that I
• 30:36
  can talk about if there's anything that kind of pops up or
• 30:39
  Peaks interests but from there we push that data into our Hive
• 30:44
  data warehouse and We kind of have to hide a two warehouses.
• 30:49
  We have like an on-premise one that we manage and we're moving
• 30:51
  over to AWS glue for for kind of a lot of that management.
• 30:57
  And the kind of, I guess, kind of the space which most like ml
• 31:03
  practitioners are interested in is like once.
• 31:06
  Once the data is kind of landed in our data warehouse, we have,
• 31:11
  you know, kind of the typical series of like decision science
• 31:14
  jobs that compute like business, X and all that stuff.
• 31:18
  But we also have a series of I guess jobs you'd call them that
• 31:23
  are doing more like feature engineering or feature
• 31:26
  computations and sticking that into one of three different
• 31:31
  places. Sometimes it lands back in Hive.
• 31:34
  Sometimes it lands in S3 as like a tar.gz file that a service can
• 31:39
  kind of pick up and understand. We kind of do that, that's when
• 31:44
  trying to like detect people who have like opened up.
• 31:47
  The same account under, like, two different names or something
• 31:51
  would use this kind of S3 path to try to detect like, oh, hey,
• 31:54
  you were, we actually detected your second client.
• 31:56
  We're going to merge your accounts together or whatever.
• 32:01
  Or we actually go into kind of our own like rocks T, be
• 32:05
  implemented on top of S3 feature store, which just allows us to
• 32:09
  do. Like it just allows us to have
• 32:12
  more, I guess like isolation guarantees or maybe like item
• 32:18
  potent guarantees on top of data Transformations and that kind of
• 32:23
  data store Powers. Some of the models were doing
• 32:26
  around like are you feeding in game?
• 32:29
  Are you purposely playing bad? Do we need to ban you?
• 32:35
  And I guess kind of the more may be awkward or more different
• 32:40
  kind of approach from there is like, on occasion.
• 32:43
  We also take some of this data and some of these models and we
• 32:47
  serialize it or embed it into the game code.
• 32:51
  So then on the following patch, you'll get like the next version
• 32:54
  of our model, we do this with like, how we detect a player's
• 33:01
  Lane so do you know when they're playing a game that can pick one
• 33:04
  of five different? Lanes.
• 33:06
  But they don't tell us what it is explicitly.
• 33:08
  So we kind of have to infer it from your playing activity.
• 33:12
  So that kind of decision tree, we actually build way up in the
• 33:17
  kind of data warehouse Lair, stuff it into the game.
• 33:20
  Binary. So, yeah, we're kind of solving
• 33:25
  inference in, like three different spots like in game at
• 33:27
  service time or sometimes in like the decision, science
• 33:31
  space, And I guess a lot of the problems that I'm working on now
• 33:36
  are in that kind of game inference space.
• 33:39
  How can we kind of do more like game personalization and a lot
• 33:43
  around? Like, how do we do?
• 33:47
  How do you, how do you do, like democratization of features?
• 33:50
  How do you do monitoring of that stuff?
• 33:52
  We're still kind of at the beginning of that Journey.
• 33:57
  Hey there. I'm a Satanist and ever and I'm
• 33:59
  head of product a teachable hub and if you are into envelopes
• 34:03
  and want to hear more than just buzzwords, subscribe to the
• 34:06
  envelopes Community podcast and listen to the best practitioners
• 34:09
  in the field, sharing their wisdom. there's some you said,
• 34:17
  when when you were going through the Past.
• 34:26
  All right, let me start that one over.
• 34:28
  I'll spy trade it done because I cut off.
• 34:31
  Skyler, Skyler in. Sorry, I jumped right in.
• 34:34
  But basically there was so, there was something you said
• 34:37
  when we had the pancakes tax and I thought it was fascinating on,
• 34:41
  like, the whole real time element.
• 34:43
  And why you didn't want to do real-time, can you jog my memory
• 34:48
  on exactly what that was about. Sure.
• 34:52
  Yeah, I think um, so I kind of mentioned how will kind of
• 34:58
  compute some models and kind of then like embed them into the
• 35:02
  game. So like You know, if you imagine
• 35:08
  like, a decision tree, like just like, not a bags, decision tree.
• 35:13
  No, Ensemble methods here. Just like a single plain old
• 35:18
  decision tree that split by Information Gain or whatever.
• 35:21
  You have all of the splits. And ideally you'd be able to,
• 35:28
  you know, train on these splits and update them as you get new
• 35:32
  data and not have that be, you know, kind of like Every two
• 35:37
  weeks or every six months or something like ideally, you do
• 35:40
  it pretty often to kind of keep your accuracy pretty high. the
• 35:46
  challenge that we are kind of facing is like, There are a lot
• 35:52
  of challenges around like a video game, trying trying to do
• 35:57
  these kinds of things. So an example might be like,
• 36:02
  hey, in order for us to compute your decision tree, you know, we
• 36:06
  need to spawn a new thread and if we spawn a new thread, you
• 36:09
  may take too much frame computation time and then we'll
• 36:12
  see player hitches. So you don't want to do that.
• 36:15
  So you have to have like a hyper optimized kind of decision tree,
• 36:18
  which is already like I mean, if you're if you're following the
• 36:22
  research on things like jacks or any of these like computational
• 36:26
  engines, you can actually get pretty fast stuff, but it's
• 36:30
  still pretty difficult to do that within like a really fixed
• 36:32
  amount of time. We have a lot of additional
• 36:38
  layers of like network security that we kind of have to do.
• 36:42
  And I think this stems from the fact that like league is a
• 36:45
  pretty big game, a lot of people play it.
• 36:48
  We tend to get like a lot of people trying to DDOS us or a
• 36:52
  lot of people trying to kind of find their way into our
• 36:57
  infrastructure, kind of similar to the social engineering thing
• 37:01
  that happened at Uber recently. So there are a lot of a lot of
• 37:07
  extra steps taken to say like, hey game servers, can't even do
• 37:10
  you TCP connections. So like it's difficult to build
• 37:14
  a service that's like all UDP, only blah, blah blah to do that
• 37:18
  kind of stuff at the game, server level.
• 37:19
  Evil and then kind of the final bit is.
• 37:24
  So there's the technical reason of like rendering is hard when
• 37:27
  you're doing like these big massive mathematical
• 37:29
  computations while playing a game.
• 37:33
  There's a lot of like at a network kind of security stuff
• 37:37
  that is just like stuff. We haven't really thought about
• 37:39
  how to go around yet. We aren't really sure how to
• 37:41
  proceed kind of safely and ethically and like league is a
• 37:48
  older game. And there's a lot of historical,
• 37:52
  inertia, kind of going against some of these changes.
• 37:58
  And that's not to say, like any one person is saying like, oh
• 38:01
  no, don't put em on my game. Most of the time is very much
• 38:04
  the opposite but it's like how do you get into a you know,
• 38:09
  ten-year-old C++ code base with you know three different custom,
• 38:15
  unique pointer implementations and write your own.
• 38:19
  Jewel to do you know whatever inference you want to do kind of
• 38:24
  echo time. That's just like a really I
• 38:27
  guess it's just like a really challenging problem from the
• 38:30
  perspective of like how many ml Engineers write C++ Or like how
• 38:37
  many ml Engineers have an interest in like game engine
• 38:40
  architecture? I feel like those are just two
• 38:41
  very Niche subjects that tied together along with the fact
• 38:45
  that this code base is 9 or 10 years old, makes it really
• 38:48
  difficult to go in there and make changes Yeah, I can
• 38:52
  definitely imagine, you know, one of the challenges sometimes
• 38:55
  is like when you have a legacy system or something that already
• 38:58
  exists in your wanting to kind of like Port machine learning
• 39:01
  into it. You know, definitely already
• 39:04
  like operationalizing machine learning just in general is
• 39:07
  like, very challenging requires lots of skill sets.
• 39:09
  But if now also, you need to like understand, not, just C++,
• 39:13
  but specifically how Riot has been writing, C++, and their
• 39:16
  base, you know, that can definitely be daunting.
• 39:20
  So yeah, that definitely a lot of interesting stuff there.
• 39:25
  But yeah. So you know, very curious, you
• 39:29
  know, as you were explaining earlier you were talking about
• 39:32
  sort of your progression to start working where you are now
• 39:35
  and Riot and you've essentially kind of transition from war on
• 39:40
  like the software side to eventually.
• 39:42
  Now you're kind of like more on the data and in a lopsided and
• 39:46
  very curious to get your take on, you know, what is the maybe
• 39:49
  like Major differences on your sort of day-to-day
• 39:54
  responsibilities or like what is different between those rolls to
• 39:57
  you. I think it's like I think the
• 40:02
  the The best answer I've read for that is like the fact that
• 40:08
  your code doesn't matter as much as your data matters.
• 40:11
  Like that is like this weird kind of tendril creeping kind of
• 40:18
  epiphany that you have super out like no one really cares how
• 40:24
  crappy your Pious barcode is but if like the analysis that you
• 40:29
  offered is like really clean or like If the model that you've
• 40:35
  shipped has like a high level of accuracy and is delivering a lot
• 40:39
  of player value. No one really cares.
• 40:42
  If you you know imported scikit-learn and did some python
• 40:46
  code generation mumbo-jumbo, which is very much different
• 40:53
  from from being like a kind of software engineer working on a
• 40:58
  product. I didn't think that that divide
• 41:02
  was very big. I think if you asked me in 2020,
• 41:05
  I'd have been like, you know, Skylar I'm a software engineer.
• 41:09
  I just also think about data and there's no difference there, and
• 41:12
  I still like that answer philosophically.
• 41:15
  But like, when I was then kind of put into the realm of like,
• 41:20
  hey, you're a software engineer and you're working on our game
• 41:23
  build tooling and you're working on our reliability tooling, it
• 41:27
  was just completely different like, you know, I would build
• 41:30
  out stuff. And I, you know, I had to get
• 41:34
  the feature like working and it would all work great or
• 41:36
  whatever. It might be a little it's maybe
• 41:40
  the word I'm looking for the, it might be a little Rube Goldberg
• 41:44
  e, you know, like it's like, oh well soon Jenkins job triggers
• 41:47
  this, which triggers out with sugars that, but it's nice
• 41:50
  because I only wrote six lines of code here.
• 41:53
  Eight lines of code here or whatever and then one kind of
• 41:57
  reviewing that and like you know talking to other software
• 41:59
  engineers and be like dude just do this.
• 42:01
  All in Python, like right, like a 600 line python module, let's
• 42:05
  do this all in Python, so we can check it in to get and write
• 42:08
  unit tests and limited and all the seven is like God like this
• 42:13
  is dramatically different because I'm just very focused on
• 42:16
  like giving you the end product. But you know I think for like
• 42:19
  products software engineering, the product is assumed to be
• 42:24
  true. Like the game is already here in
• 42:27
  the game's not going to go away, you're not going to make it
• 42:29
  disappear tomorrow. Errol.
• 42:32
  So, the code, you write has to be maintainable and ready to go.
• 42:34
  And like, you need to look at the code AS gospel.
• 42:37
  Kind of, whereas everywhere in the data space, I've seen like
• 42:42
  it is that is never the first story, I think probably is you
• 42:47
  guys are also seeing I think as ml Ops and like machine learning
• 42:51
  is becoming more of I guess like a kind of software engineering
• 42:56
  trade. That's changing.
• 42:58
  Like, you are seeing people say like write unit tests for as
• 43:02
  much of your model code as possible, you know, mock your
• 43:06
  mock your training on like some kind of mock data use like
• 43:10
  observability practices to monitor.
• 43:12
  Your model is best possible. So I think a lot of that is
• 43:16
  changing but I think at the end of the day like you know in 2025
• 43:21
  or 2030, you're still going to see.
• 43:23
  Ml Engineers who are half talking about their code and
• 43:27
  mostly talking about their data. Yeah, yeah, it's interesting.
• 43:32
  You say that, you know, I think there's often a lot of competing
• 43:36
  opinions on this kind of topic around like a code quality and
• 43:39
  whatnot. There's a particular menu person
• 43:41
  in our community Lazlo that has like a lot of stronger than
• 43:44
  without. I knew you were going to mention
• 43:46
  code quality. Yeah, he's a lot of great blog
• 43:49
  posts on sort of like how he thinks you should think about
• 43:51
  code quality. And it's Interesting to see, see
• 43:54
  the tension almost in a cocoon of some people saying, like,
• 43:57
  that's not that important of people saying, it's kind of like
• 43:59
  Paramount, but curious to maybe to kind of contrast it with some
• 44:04
  of the things laszlo's said, I think a lot of folks that
• 44:07
  Community are familiar with you. Lots of wrote a blog post
• 44:10
  somewhat recently where he kind of made a separation between
• 44:15
  essentially code, that's for analysis versus code.
• 44:20
  That's, you know, for like production more or less.
• 44:24
  And he said, like, hey the code quality stuff, like just don't
• 44:27
  everything I'm saying, here does not apply to that first bucket
• 44:30
  of like analysis and whatnot. So, very curious to get your
• 44:34
  take on that because I think there was maybe some subtle
• 44:37
  Nuance of what you're saying, and but curious do you agree
• 44:39
  with that? Do you disagree with it?
• 44:41
  Where do you lie on that Spectrum?
• 44:44
  I feel like, I feel like the answer you want is like a hot.
• 44:46
  Take take down and I don't, I don't have that.
• 44:50
  I don't know, I would never want to put a hot ticket.
• 44:53
  Lazlo. Because he his hot take will be
• 44:57
  hotter than mine. I'm sure.
• 45:00
  Yeah, I think you know, this is something that me and kind of my
• 45:07
  to the my direct kind of coworker.
• 45:10
  His name is Serge. We are really thinking about
• 45:15
  this a lot, because we're thinking about it from the
• 45:19
  perspective of like, you know, I don't know if you guys have it
• 45:24
  and I'm not sure Laszlo has it, but I'm sure he probably speak
• 45:27
  to it as well, where they, I should say they.
• 45:31
  There is just not enough people who were like software Engineers
• 45:36
  working in the data space, like we just don't have enough people
• 45:39
  who are doing it. So like, the current reality is
• 45:42
  data, scientists are going to write things that go to
• 45:45
  production, and I kind of sucks. I think everyone doesn't want
• 45:49
  that. Like data scientists don't want
• 45:51
  that. One thing we can kind of do to
• 45:56
  make that a bit easier is to try to like right tools and like
• 45:59
  right guidelines on like how to structure a GitHub repo, how to
• 46:03
  write tests, how to use a Jenkins job.
• 46:06
  And we're thinking about that like a lot because we're trying
• 46:08
  to say like how can we, you know, hide the data abstractions
• 46:13
  as much as possible so that we can encourage a data scientist
• 46:18
  to be like, okay, go back to, you know, your You know, you
• 46:24
  know, you know, your data structures class or whatever and
• 46:27
  really think about the code that you're writing to make it like,
• 46:30
  really clean, or at least really efficient and like, make the
• 46:33
  trade-offs that you made clear, because this is going to be hit
• 46:38
  by like millions of players. If it's like in the line of like
• 46:42
  direct gameplay. and it's really like I think it's just like a
• 46:49
  really hard problem because it's like we're trying to take two
• 46:55
  very different approaches to something that is so maybe like
• 46:59
  super in development I don't think data science is really a
• 47:03
  concrete field yet like I think it's still kind of being built
• 47:08
  out so like I think I think you could very easily I guess my
• 47:13
  take is you could very easily see New kinds of data teams, one
• 47:17
  data team that takes code quality, extremely seriously,
• 47:21
  and still ships player value. And one that doesn't take code
• 47:25
  quality. Seriously, at all like it's for
• 47:27
  the birds and also ships player value.
• 47:30
  I think both of those are still like very viable paths and we
• 47:33
  don't know which one is better yet.
• 47:37
  I think as a software engineer, I am more inclined to, at least
• 47:42
  have a base level of quality, where it's like, hey, if you're
• 47:46
  afraid to least have me look at this code because, you know,
• 47:50
  it's that janky then, like, don't submit it.
• 47:53
  Like, let's, let's spend more time to fix it.
• 47:57
  But I'm also very excited to see people who are really passionate
• 48:00
  about, like, how to write better code in the ml space.
• 48:04
  Help us make tools to do that. Like, if Lazlo Builds a tool
• 48:08
  that helps us like hold people accountable to like, write
• 48:11
  better code than like I guarantee you I will be at all.
• 48:15
  The meetups will be at all the conference's.
• 48:17
  I will ship it in all the repos because I'm here for that
• 48:21
  reality. I just, I guess, maybe I'm like,
• 48:24
  deer in the headlights. I don't know.
• 48:26
  Which way is up and which way is right yet.
• 48:28
  Yeah, yeah. You know, like, I similarly like
• 48:31
  I keep waiting for Laszlo to convert that thought leadership
• 48:35
  into a product, you know, like yeah.
• 48:38
  I'm ready for it. I'm here for it.
• 48:40
  But yeah, I think he said a lot of interesting things that
• 48:42
  resonated like in particular, you know, we're in this
• 48:45
  situation that data scientists are writing production code.
• 48:48
  We don't have enough of these Engineers that kind of calls
• 48:50
  back to another guy and Community michiel.
• 48:53
  It was wrote this Banger article, we don't need data,
• 48:56
  scientists me data Engineers. They resonated with a lot of
• 48:59
  people and that like hey, we have like data scientists doing
• 49:02
  things that doesn't like really match their skill sets enough.
• 49:06
  So one thing I was since you were talking about like, hey the
• 49:08
  reality is like you're going to have data scientists that are
• 49:11
  essentially writing code that goes in production, very curious
• 49:14
  of like how you think about structuring that are like what
• 49:17
  are the tools techniques Etc used given this reality data
• 49:22
  scientists are going to be writing production code, what
• 49:24
  can we do to make that better better smoother process?
• 49:30
  Yeah, I think so. The I think there are probably a
• 49:36
  lot of different like Avenues you could go through to look at
• 49:39
  that. So I'll give you the more like
• 49:43
  this is the most direct answer I can give you because this is
• 49:45
  like the approach we're taking like as of today which is what
• 49:50
  we're trying to. If not eliminate severely,
• 49:56
  reduce the amount of notebooks in the world like jupyter,
• 50:01
  notebooks are a fantastic tool for Yes but like the moment
• 50:06
  you're thinking about like I'm going to write this you know,
• 50:09
  like little pie spark thing and it's going to be hit by a
• 50:12
  service then like we try to tear down the notebook, kind of
• 50:19
  artifice all together and we say like okay no like you're going
• 50:22
  to either a build a python package and you're going to ship
• 50:26
  the python package or you know, you're going to write a Java
• 50:29
  service or whatever and whatever.
• 50:33
  Were is in your notebook. Just kind of does not hit
• 50:35
  production. this is like, We thought that this was going to
• 50:42
  be like kind of a superhot, take I guess, I guess to give like
• 50:47
  maybe like more context as to why we did this.
• 50:50
  One of the most successful projects that LDC is shipped is
• 50:53
  something called the item recommender.
• 50:55
  So when you go and like play League of Legends and you like
• 51:00
  go and buy an item because you want to buy an item like every
• 51:05
  time you go back to base or like kind of a certain key key points
• 51:09
  you can tell. I'm not like a professional
• 51:11
  League player cuz I can't tell you when to buy an item but like
• 51:15
  You buy an atom when it's important to you and game.
• 51:18
  And when you bring up the item, I knew we re recommend like,
• 51:21
  hey, you're playing against these Champions, you are this
• 51:25
  Champion, you have this person on your team, you have this
• 51:28
  many, you know, Baron kills or whatever.
• 51:31
  You have this many Champion kills, we think this item would
• 51:33
  be a good fit for you. That product has been like very
• 51:38
  successful from the perspective that like players like it but
• 51:43
  also because like it's a very maintainable piece of code.
• 51:46
  Like we have modules, we have unit tests, we have linting, we
• 51:52
  have You know, we've treated it very much as like a small
• 51:59
  product that anyone can kind of come in and change, but we have
• 52:04
  like code review standards. We have, you know, like we this
• 52:09
  one of the only places I've seen it, but like we kind of enforce
• 52:11
  the use of python typing. So we kind of make use of a lot
• 52:17
  of the tools that Facebook built to kind of help us like, do more
• 52:20
  intelligent linting and like more intelligent finding of
• 52:24
  obscure typing issues on my All underneath your that might fall
• 52:28
  out of the room like fall out of scope of a data scientist or
• 52:32
  someone who just writes python to get the job done as opposed
• 52:35
  to like being a software engineer writing python.
• 52:40
  So we've seen that like okay, we've treated this model purely
• 52:44
  as a product and it is more maintainable.
• 52:48
  It is more like observable, it's more extensible and this didn't
• 52:53
  use a single notebook. This is Been a notebook list
• 52:56
  product the entire way. Maybe there's something to that
• 53:01
  and it's probably not the notebook as a piece of
• 53:03
  technology itself, but more like the workflow where the mindset
• 53:06
  that it puts you in as you're developing in it.
• 53:10
  So that's kind of the first thing we're doing is we're just
• 53:12
  trying to say like if you know this going to production don't
• 53:16
  write anything in a notebook and I guess that challenge has led
• 53:19
  to a kind of increasing quality in other places.
• 53:24
  The downside is it kind of For people who don't know how to do
• 53:27
  that stuff. You, you're adding a lot of
• 53:29
  extra work for them. So I think that's number one.
• 53:34
  And I think number two is like a frequent trying to treat models
• 53:41
  as like, I don't know if you guys have this but we definitely
• 53:45
  have a lot of the problems of like a model goes to production
• 53:48
  and it's not looked at again, until it's broken.
• 53:51
  And by Broken, we mean like no longer Doing the thing we want
• 53:56
  it to do but you know, along the way we don't know if the
• 54:01
  accuracy of the model went down, we don't know if like the
• 54:04
  underlying data distribution is shifted or whatever.
• 54:08
  We just know that now it's broken and is not working and
• 54:11
  people are mad at us and yelling.
• 54:16
  So we're also trying to like, kind of kind of structure ways
• 54:20
  for us to say like, hey, every two weeks, every month, go back,
• 54:23
  let's look at our models, let's kind of audit everything to
• 54:25
  understand that things are so working and like that can very
• 54:30
  easily become like a meeting that could have been an email.
• 54:33
  So like running that smoothly running, that cleanly is still
• 54:36
  something we're trying to figure out.
• 54:38
  I don't know if that kind of answers your question, but those
• 54:39
  are kind of the two ways we're trying to solve it, I guess.
• 54:42
  Interesting. Yeah, you know I think you You
• 54:45
  definitely unraveled your hot take in there, you know, with,
• 54:48
  you know, more and more of this notebook opinions.
• 54:53
  And yeah, that's definitely a topic in the community that gets
• 54:57
  a lot of people riled up and there's lots of proponents on
• 55:01
  either side. I, you know, maybe someday we
• 55:06
  can dive deeper and chat about this with some big notebook
• 55:11
  proponents but yeah, thanks for that.
• 55:14
  For those insights that a super awesome, you know, kind of like
• 55:19
  Switching gears and maybe, like, leaning more towards a wrap-up.
• 55:23
  You know, you've had sort of a, your kind of like well into your
• 55:26
  career now, you had a lot of different experiences.
• 55:30
  So very, very curious. Like if there was something you
• 55:33
  could change about your career up to now, would it be Oh, I
• 55:38
  would have took a networking class.
• 55:41
  I that, that may be a weird answer.
• 55:43
  But like, not actually understanding how computer
• 55:46
  networks were like Works has has bitten me in the ass a lot.
• 55:51
  I think that that that probably would have been the first thing
• 55:54
  at least from like a technical perspective from like a more
• 55:59
  personal perspective. I think I would have I think the
• 56:04
  one big change I would have would have done is like tried
• 56:10
  not to chase things as much and like try to have more open and
• 56:15
  honest relationships with people.
• 56:18
  I think like my first career, my first job at Adobe, I had like
• 56:23
  super incredible manager. Like even to this day I'm still
• 56:27
  like pulling out lessons at this guy.
• 56:28
  Is taught me just about like it is, it is kind of in the realm
• 56:32
  of how to be a good hang but like how to be a good engineer?
• 56:36
  How to be a good Tech lead, how to be a good senior engineer had
• 56:39
  to learn? Well, he taught me all that
• 56:41
  without maybe even explicitly telling me.
• 56:46
  But like, at the time, I was like so focused on, like I got
• 56:49
  to be writing code. I got to be, like, kind of the
• 56:52
  goat of, you know, some particular programming language,
• 56:56
  but I definitely wish I had, like, I don't know, maybe just
• 57:00
  calm down. Down and just like, hey man,
• 57:02
  enjoy this experience. Taking all the lessons like
• 57:06
  really bask in it for a little while but you know hindsight's
• 57:10
  20/20, right, like I think if I had done that, my career might
• 57:13
  have ended up differently. So yeah, definitely. if there's
• 57:19
  one best practice, you wish the data Engineers would take from
• 57:23
  software Engineers, what would it be Oh, that's a really good
• 57:30
  question. I think my answer would be for a
• 57:36
  composition over inheritance. I even now to the say, I think
• 57:42
  maybe you're kind of forced into it with like the abstractions
• 57:46
  that pie spark gives you but like the number of like code
• 57:50
  bases. I've gone into with a class
• 57:53
  named class ETL or class base ETL.
• 57:56
  The thing gets, Extended like 30 times that like, has to
• 58:00
  implement some base method that just passes or something.
• 58:05
  It's just add so much noise to like like dissecting the process
• 58:12
  and then on top of that, I think like that carries into I guess
• 58:17
  it carries into like the way the systems get designed like, hey,
• 58:22
  like, let's just make a new air flow job that can do this
• 58:25
  instead of maybe like picking the right system to do it or a
• 58:28
  like trying to integrate the right system into airflow.
• 58:31
  Let's just make a new air flow job and makes a new air flow.
• 58:34
  Operator. That does the thing is airflow
• 58:36
  is our new base class for everything.
• 58:40
  So I think that's probably my answer.
• 58:44
  Awesome, cool. And maybe to close things out,
• 58:48
  you work at a gaming company. We talked a little bit about
• 58:50
  games you play. But what's your favorite video
• 58:53
  game? Um hardest.
• 59:00
  What did it huh? The hardest question, we ask
• 59:03
  here. Yeah seriously dude, I It's
• 59:10
  gotta be, like, Legend of Zelda Majora's Mask.
• 59:14
  Like, I just love that game, two pieces here.
• 59:17
  Oh, man, I love it. I just love it to pieces.
• 59:21
  I like, every time I play it, I'm happy when we got the reap.
• 59:24
  I got the Remake on 3DS and I was like, it's just the
• 59:28
  Nostalgia, the, the emotion of the game.
• 59:31
  I love the dungeons. I just, yeah, I can't, I can't
• 59:34
  not answer that. Awesome.
• 59:38
  You're not gonna like it but I've never actually even played
• 59:41
  that game. So seems like I'm missing
• 59:43
  something here. Both of you rank that as your
• 59:46
  favorite. I think it's a lot of nostalgia
• 59:49
  of like, you know, being a very particular age and getting that
• 59:53
  gay. Like I feel like I was like the
• 59:55
  Peak Interest of video games, right?
• 59:58
  When that came out. Yeah it has a lot to do with
• 1:00:00
  just like how old I am and being in the cultural Zeitgeist at the
• 1:00:04
  time, but yeah. This is a very moving story like
• 1:00:10
  as an adult, it's still actually quite quite intense.
• 1:00:14
  So this is awesome man. Thank you so much for coming on
• 1:00:18
  here and sharing some of your wisdom with us and re taking us
• 1:00:23
  through everything that you're doing it Riot.
• 1:00:25
  And also going a bit more philosophical with us, that, you
• 1:00:30
  know, we have gone places. I did not expect us to go, but
• 1:00:34
  I'm very thankful that we did. Very nice.
• 1:00:39
  So excited me on guys. I know it's probably a bit
• 1:00:41
  different than like talking about feather or something, but
• 1:00:45
  I happy too happy to be in the same places those people.
• 1:00:49
  Yeah, it was awesome, man.