Latest / Agentic Conversations (formally mlops.community) / Real-Time Exactly-Once Event Processing with Apache Flink, Kafka, and Pinot //Jacob Tsafatinos // MLOps Coffee Sessions #97

Transcript

Real-Time Exactly-Once Event Processing with Apache Flink, Kafka, and Pinot //Jacob Tsafatinos // MLOps Coffee Sessions #97 · Agentic Conversations (formally mlops.community)

• 0:00
  Hey, this is Jacob. We're on the MLS Community
• 0:02
  podcast, aka the best podcast in the world.
• 0:04
  And today we're talking about a blog post that I wrote
• 0:18
  I promise you, I'm not in an assailant Asylum right now,
• 0:22
  although the walls through look padded, you're finally got rid
• 0:28
  of Demetrius, but over here, where he needed to be, they
• 0:32
  still let me podcast out of here, though.
• 0:34
  That's the worst thing that happened.
• 0:38
  So, no, I'm So right now we're about to do an in-person meet
• 0:43
  up, which is awesome. I got to go run and get the
• 0:45
  pizzas for the actual event. But we just got done with this
• 0:50
  podcast with Jacob, my fellow Greek brother.
• 0:54
  Who is he wrote the blog with a few of his teammates and he
• 0:58
  created the system around Uber's.
• 1:01
  Exactly. Once real-time processing engine
• 1:06
  or processing with Apache, Flink Kafka and P note.
• 1:10
  We also just put out a video that is like an animation video
• 1:15
  on that whole thing that was us trying to synthesize all of that
• 1:19
  blog post and data but we I mean it was cool, man.
• 1:25
  It was really, it was really cool.
• 1:28
  The guys freaking awesome guy was like, there were so many
• 1:34
  takeaways. I want to hear yours first
• 1:35
  though. Give it, give it to me, and then
• 1:37
  I'll play off of years. Yeah, no, absolutely.
• 1:39
  And I think that, I mean, his first off, I think it's super
• 1:42
  cool to meet an engineer. Who's also, you know, runs a
• 1:46
  very successful band which is, which is awesome.
• 1:48
  Just people from all walks of life, which is great.
• 1:50
  But I mean the system right, that he built is, I mean, you
• 1:53
  know, as an engineer. That's an incredibly complex
• 1:55
  system that they put together and at the scale that they're
• 1:58
  dealing with, you know, to Uber processing all of their event at
• 2:01
  events in near real time. I mean that's incredibly
• 2:04
  impressive. And I just thought I was like,
• 2:06
  really cool to hear sort of how much they were able to rely on
• 2:10
  open source. As the core piece, underlying
• 2:13
  everything right that they were using blink and Kafka and pinot.
• 2:18
  And just all these things, these these pieces of software that
• 2:21
  are fundamentally run my volunteers around the world.
• 2:25
  But they're powering, you know, this massive organizations
• 2:29
  entire event processing pipelines, which is, which is a
• 2:32
  testament hitting the power of Open Source.
• 2:34
  So I think that was one really, really cool thing to see.
• 2:37
  And then the other thing I liked was just, you know, In ml, we
• 2:41
  think a lot about this new trend of real-time systems.
• 2:44
  Yeah, other up-and-coming and I think it was just great to hear
• 2:50
  him talk through the importance of really just keeping customer
• 2:53
  in the loop. Right?
• 2:54
  And just understanding what value is being derived from the
• 2:57
  real-time systems. And why are we building the
• 3:00
  real? Why is the real time in the
• 3:01
  first place? Why can't we just rely on batch
• 3:03
  and being very proactive about those considerations from the
• 3:06
  get-go? It was so true how he was
• 3:10
  focusing so much. And he said it probably three
• 3:13
  times throughout the interview. Like, who are my customers?
• 3:16
  Who am I building this for? And is Real Time necessary for
• 3:20
  them. That was his big question that
• 3:23
  he went through. I loved when I asked him.
• 3:26
  The question about, let's imagine that you're getting
• 3:29
  pressure from your boss. You've never built any real time
• 3:32
  system. And your boss is telling you,
• 3:34
  you need to go out there and build it.
• 3:36
  Would you like Run away from that company.
• 3:39
  Get a new job where you can do data, sign stuff.
• 3:42
  Or would you sit through it? How do you look at that?
• 3:44
  And he was like, well, if it was me I'd just go out and learn it
• 3:48
  and build it and there was like a bit of a mic drop moment,
• 3:51
  like, oh damn, he was like, that's what I did with this.
• 3:55
  So that was so cool to see. I don't want to give too much
• 3:58
  away. Let's get into it.
• 4:01
  Everyone, if you are enjoying these podcast please give us a
• 4:05
  like And subscribe that would mean the world.
• 4:07
  World through us. You can also review the podcast
• 4:10
  and you can get bail notifications.
• 4:12
  Going out there to stay up-to-date on all of our most
• 4:16
  recent podcasts and that's about all we got.
• 4:20
  Let's jump into this conversation man.
• 4:25
  We're going to start it off with your band and all those guitars
• 4:28
  in the background, man. Let's just get it out because I
• 4:33
  feel like that's where we have to start being a musician
• 4:35
  myself. What's the band you're in?
• 4:39
  And how did it start? Give us a small little story
• 4:41
  around that. Yeah so I play in a band called
• 4:44
  good kid, I play lead guitar and it's started.
• 4:49
  We play like Indie indie rock music and it started in
• 4:51
  University. Actually we were all studying
• 4:53
  not everyone was studying computer science at the time but
• 4:55
  the majority of us were doing computer science.
• 4:59
  Our singer was like doing a bunch of stuff with a I are in
• 5:04
  the rest of us are pretty much like, mostly back in engineers.
• 5:07
  Yep, that's how we met. We decided to, like, we all were
• 5:10
  playing in bands, our whole lives.
• 5:11
  And so, we just started jamming one day.
• 5:13
  And then we're like, oh, this is great.
• 5:14
  And then we like added another guy.
• 5:16
  Then I got a guy for who I worked with like in a band from
• 5:19
  high school and we brought him in and then now he's he wasn't a
• 5:22
  programmer and he wasn't studying computer science but
• 5:24
  now he is as heard pretty much as a result of being in this
• 5:27
  band. So it's kind of funny.
• 5:29
  That's incredible. So, do you write lyrics about
• 5:33
  being a software engineer or is it more emotional art?
• 5:36
  Too sticky stuff. It's a mix.
• 5:39
  We have a song about like Doc. Well, kind of about Donkey Kong,
• 5:41
  we have a song about anime. We write, we take inspiration
• 5:45
  from like whatever is around us were consuming.
• 5:47
  Definitely we take inspiration from like the artwork consuming
• 5:49
  so whether that's like anime or a book or for video games or
• 5:53
  could be like, you know, our own emotional experiences.
• 5:57
  But yeah it's great. We just finished a US tour and
• 5:59
  we're about to announce another one actually today.
• 6:04
  Oh so by the time this comes out it will Alive will put a link to
• 6:08
  that in the description. Mihael has a very interesting
• 6:12
  question for you and music and being around you tourist.
• 6:17
  I see it here in the dock that you want to start.
• 6:19
  Let's start with that one. Sure.
• 6:21
  Yeah. I mean that's a fascinating
• 6:24
  background. Honestly like especially it's
• 6:26
  like you seems a girl. She really quite successful with
• 6:28
  the music. You know.
• 6:29
  It seems like it's more than just a hobby.
• 6:31
  I'm fascinated because we're, you know, here to talk about is
• 6:35
  music in some respect. But also Also, this really
• 6:38
  fascinating real-time system that you built a tuber
• 6:40
  previously. How do those connects?
• 6:44
  I mean, what is what have you maybe learned about real time
• 6:47
  systems by being in a band and vice versa?
• 6:51
  Hmm, I don't know if I would say they connect in, it's a little
• 6:57
  tough. I will say that this skills that
• 6:59
  require to be like in a band of five people were you have to
• 7:03
  work together over like a long period of time and across a
• 7:07
  Variety of like different problems are very similar to, I
• 7:11
  would say like a team at a company, you know, especially
• 7:14
  like a small team that you have to look kind of like figure
• 7:16
  things out as you go along, especially if you're doing like
• 7:18
  Greenfield stuff. Like the system that I built at
• 7:21
  a uber like we hadn't really built a system like that at the
• 7:24
  company before and a lot of those skill sets of like
• 7:27
  collaboration working with people.
• 7:29
  Making sure people feel heard all those things like our I
• 7:32
  would say like definitely have a lot of overlap and then in terms
• 7:35
  of like how the system by it helps the band.
• 7:38
  I wouldn't say there aren't there isn't much overlap in
• 7:39
  terms of like real-time systems, but it does give me a
• 7:42
  interesting. Understanding a much deeper,
• 7:45
  understanding of how these systems work when it releases
• 7:47
  something like Spotify. For example, gives a bunch of
• 7:49
  stats to bands if your spot. If you're an artist on Spotify,
• 7:51
  you have this, like Spotify for artist page and they'll tell you
• 7:54
  all these things about, like the performance of your songs, and
• 7:56
  all these things. And I can tell you right now,
• 7:58
  that's not using a real time system because it's all delayed
• 8:00
  and there's all these other things are getting that kind of
• 8:02
  understanding helps to be kind of evaluate something.
• 8:05
  So we have a manager for Apple at the band and he might say,
• 8:08
  like, oh we got a big bump in streams this day and I'm like,
• 8:10
  well actually that happened two days ago or that probably having
• 8:13
  like a day ago because like clearly this is like a batch
• 8:15
  system, you know, not kind of it.
• 8:17
  So there are some like little nuances and then if we're trying
• 8:19
  to figure out where that came from, if it came from, like, I
• 8:22
  don't know. Some like we've gotten bigger in
• 8:25
  the streaming Community like like twitch and stuff if it came
• 8:28
  from a twitch streamer using our music, it's it.
• 8:31
  We can now narrow our search down to like the day before
• 8:35
  instead of looking wasting our time.
• 8:36
  I'm looking for today so there's some like weird, some weird,
• 8:40
  some weird nuances there that like benefit each other.
• 8:44
  I would say I was hoping you were going to tell us how to
• 8:46
  game the Spotify algorithm because I don't think there's
• 8:51
  any way to game any algorithm. I think it's someone said this
• 8:55
  once, I can't remember who. And they said, like, any time
• 8:58
  you say the word algorithm, you can probably replace it with
• 9:01
  with like human psychology or something or like with audience
• 9:05
  and it really just is like if you make good things that people
• 9:07
  like then then the algorithms will pick it up.
• 9:11
  It sounds like one of the main wins I'm hearing for at least
• 9:16
  from the real-time tube and his been pretty much more pedantic
• 9:19
  about the systems that you use to actually do your music.
• 9:23
  And now you're that guy who's our well, actually, that's not
• 9:26
  how that works. So we should we should
• 9:28
  reconsider this or we're obviously a very data-driven
• 9:31
  been because we're all programmers.
• 9:33
  So So it's fun to be able to like geek out over all the like
• 9:37
  a like analytics that all these platforms, give you not just
• 9:39
  like Spotify, but Apple music or even like Twitter and Instagram.
• 9:43
  They there's so much data that you get as a someone on these
• 9:47
  platforms and so getting to like make sense of that in a way in a
• 9:51
  more nuanced way, I think is is definitely like a good overlap,
• 9:55
  but it's, I don't know if it's like two two related.
• 9:59
  I'll say that. Yeah, wait.
• 10:01
  But have you set up like the Mana?
• 10:04
  Like a data stack for all of your analytics.
• 10:09
  No, but only because I tried I did, I was like trying to work
• 10:12
  through all the different like apis that all these things
• 10:14
  provide, but but it was honestly just became like a big pain and
• 10:19
  there's like their services that do that for you.
• 10:21
  For like 20 bucks a month it is. And at this point I was like,
• 10:23
  okay, I have other things I want to like I should if I could
• 10:26
  write a song instead of spending my time doing this or even like
• 10:29
  I built a video game for the bend and I think that would be a
• 10:31
  better use of my time than making rebuilding something that
• 10:34
  It's for like a reasonable price.
• 10:36
  Oh, that's so good. That is so cool.
• 10:38
  So you need to be trying to get at like a Moneyball type of
• 10:42
  approach to you know literally again game the algorithm figure
• 10:47
  out who would do that. That's what I was hoping for.
• 10:49
  I mean I've got my songs on Spotify and what happened was
• 10:53
  really weird because I just put them out there and All of a
• 10:59
  sudden. Well, I forgot about him for,
• 11:02
  probably like three or four or five months after we like
• 11:04
  released our CD. And this was in 2015, and then I
• 11:09
  went on and I figured out, oh, I can see analytics and I can see
• 11:12
  how my music is doing. We released it with CD Baby and
• 11:18
  so you can see like they give you a dashboard of 0 on Apple
• 11:21
  music. It's like this on Spotify.
• 11:23
  It's like this. And so one day I went and I
• 11:27
  looked at the dashboard and I and we've randomly just got a
• 11:30
  bunch of traffic and it was like I, you know, no matter what we
• 11:35
  were doing on social media, it had no effect on our streams.
• 11:40
  Like we could try and pump something up and push like a new
• 11:43
  song. It wouldn't matter on Spotify,
• 11:45
  there was no correlation. And so I realized that all these
• 11:48
  people that are listening to us on Spotify, they have no idea.
• 11:51
  Like they don't follow us on social media, like they just
• 11:54
  randomly encountered, our music and then they put us on some
• 11:57
  playlists or something. And there we stand.
• 12:00
  And so out of nowhere, we have a lot of streamers from Mexico and
• 12:08
  Brazil that you it's just so random and so that's why I was
• 12:12
  wondering like hmm. Maybe if you've got some Secrets
• 12:15
  I could get into that would be, I think, I don't know.
• 12:18
  But this is what I was kind of saying.
• 12:19
  Like I think it just means that people in those areas like, five
• 12:22
  with your music, I don't know whether it's the type of music,
• 12:24
  or the style or whatever, but that's probably what it means.
• 12:28
  Like, the same. Nothing happened to us.
• 12:29
  Our first song we put out and it wasn't I think was like a month
• 12:32
  after we put it out on Spotify, got on like some playlists and
• 12:35
  or then I got like discover weekly and then again I think
• 12:38
  the algorithm way the algorithm works with the that stuff is
• 12:40
  like you know, if it gets on someone's discover weekly and
• 12:42
  enough people like it or they listen to the whole thing.
• 12:44
  Then it's going to get a more people discover weekly.
• 12:46
  And it's, this is why I say like I don't know if there's a way to
• 12:49
  game an algorithm other than just keep putting out good
• 12:52
  quality stuff and then I think like or things that yeah, I
• 12:56
  don't know. That's kind of at least how I
• 12:57
  see it is. Not saying that are It gets
• 12:59
  better quality than other people who aren't getting that
• 13:01
  traction. But I do think like, you know,
• 13:03
  we do put a lot of attention and Care into making sure.
• 13:05
  Even just like the production value is good or like our big
• 13:09
  decision when we started out was we were, we had enough money to
• 13:12
  either do like one song, well, like professionally or do like
• 13:15
  five songs, I'd like a crappy studio and we decided to do one
• 13:19
  song like we put all our money together into one song back when
• 13:22
  we had like no money and we were in like University and then it
• 13:26
  work, it kind of worked out. It's Evil man.
• 13:29
  So, let's jump into. I'm sure there's a lot of people
• 13:32
  that came here and they were not expecting to get this deep dive
• 13:36
  on music and the algorithm Spotify way to do things.
• 13:41
  Let's jump into this Uber blog that you wrote You've since
• 13:46
  moved on from Uber. But I think it is an incredible
• 13:49
  blog post that you and I ain't and a few others wrote.
• 13:54
  I want to just go into something like maybe we should talk first.
• 13:59
  The story of how this all came to be like, how I reached out to
• 14:02
  you. I told you we've got this video
• 14:05
  that we're producing. We're trying to base it on your
• 14:08
  blog post and then you were kind enough to look over it and you
• 14:12
  realize that we had it all wrong.
• 14:16
  So we had to edit it. We had to change everything
• 14:18
  around and now hopefully it is, right?
• 14:21
  It's coming out this Thursday, which is probably by the time
• 14:25
  this podcast comes out will be last Is day.
• 14:30
  But let's just talk a little bit about the bigger picture of what
• 14:33
  you all were trying to do the system that you built and then
• 14:36
  we can dive into specifics. Yeah, for sure.
• 14:39
  So, when I joined Uber the Toronto office, like, pretty
• 14:44
  much didn't exist. At least not for engineering.
• 14:46
  And so I was one of the founding members of the Toronto office.
• 14:49
  And as part of that founding team, we, they Uber made an
• 14:55
  interesting decision which I thought was cool was, which was
• 14:57
  to give like the Toronto office in.
• 14:59
  Project to be like, hey, like you know this is not an exact.
• 15:02
  We don't want to start the office with like existing
• 15:05
  projects that have like a lot of tech dead or a lot of like
• 15:08
  baggage or whatever we want to do, like something completely
• 15:10
  new that actually wasn't ads. That was like another, that was
• 15:12
  we were building a we're supposed to build grocery, then
• 15:15
  they acquired a company. It got we got pivoted over to
• 15:19
  ads which was another. Luckily, like we had a really
• 15:21
  amazing director. This guy, Nick Petty, and he
• 15:23
  fought for the Toronto office. Like, if we were going to lose
• 15:26
  grocery, he was like we want to be the team that builds as Is
• 15:29
  that a tuber? And so that's kind of it.
• 15:31
  Like, we were very small Scrappy team, there's a lot of work to
• 15:34
  do honestly. Like the fact, even the fact
• 15:37
  that I got to build this system was pretty rare.
• 15:39
  And I think lucky for the level I was at, I was Al Ford Uber at
• 15:43
  the time when I was like, starting this project it seems
• 15:46
  like and through this project, like, help me get promoted to
• 15:48
  senior but the scope and like responsibility of this system
• 15:53
  for an L4 is, I think pretty pretty rare, but yeah.
• 15:56
  In general, there's like multiple.
• 15:58
  All aspects of an ad system. You need, you need like a at
• 16:04
  auction. You need like a surface to
• 16:06
  display it. You need a place for like
• 16:08
  bidding. You need to Bill, Bill merchants
• 16:13
  or like Bill advertisers and then you would need a way to
• 16:16
  like, process all the events that get generated through ads
• 16:19
  to like, make it all work. And so, this system that we're
• 16:21
  going to talk about or the one that I wrote the blog post about
• 16:24
  is that like real time, exactly. Once I'd event processing
• 16:27
  system, Perfect. So where did we mess up?
• 16:32
  When we sent you the video? I don't remember exactly.
• 16:37
  It's not that like it's a pretty complex system L say that like
• 16:41
  and there's a lot of pieces that can seemingly be convoluted.
• 16:46
  If you don't like, have a deeper understanding of like uber
• 16:49
  infrastructure. I think you guys just like
• 16:52
  misinterpreted what exactly once meant or where, where it where
• 16:57
  it was being used. Used, which is totally
• 17:00
  understandable. Because you look at the diagram,
• 17:01
  it kind of looks like a big like spider web or something.
• 17:05
  I think it is, it's a very complex system and I can see how
• 17:09
  it's getting really difficult to build this.
• 17:11
  So it's super impressive that you guys put this together.
• 17:14
  It's one of the things that you talked about in your blog post
• 17:17
  was these are three different factors, you are trying to get
• 17:21
  rights in the in terms of the system, I kind of speed
• 17:24
  reliability and accuracy, right? Where the for the three things
• 17:27
  that you have put great emphasis This on.
• 17:29
  Yeah, I was a really good talk, a little bit about each of those
• 17:32
  considerations, what? Why each of them was important
• 17:35
  and how you how you emphasize them and maybe what were some of
• 17:39
  the trade-offs as you were going around building system if the
• 17:41
  guards teachers. Yeah, for sure.
• 17:44
  So in general like this is this system is responsible for
• 17:49
  processing. All the events that get
• 17:51
  generated from ads and it has a lot of dependencies like our if
• 17:55
  you if the way I like to think about a system sometimes the
• 17:58
  starting line Okay, who are clients the clients of this
• 18:00
  system are data, scientists engineers Ops people who are
• 18:05
  like running reports for for, like mcdot.
• 18:07
  Like, you know, I don't know if it's big dogs specifically, but
• 18:10
  if a big client is on ubereats running ads, they're not looking
• 18:13
  at their dashboard, they have some manual contact, who is
• 18:16
  doing it for them other like just manually, like automated
• 18:20
  systems, like a billing system, for example, as a client and
• 18:24
  then the advertisers themselves, like they get a dashboard of
• 18:26
  like the performance of their ads.
• 18:27
  So it's a lot, it's like, Much everyone is our client and and
• 18:32
  so. So different clients have
• 18:34
  different needs data. Scientists probably don't need
• 18:37
  the speed requirement. For example, like they can run
• 18:39
  their their reports with some delay.
• 18:42
  They usually do. Anyway but but billing for
• 18:46
  example, as like a system that needs to know how many ads did
• 18:50
  we show that needs to be fast or paste?
• 18:52
  They may not maybe even billing doesn't necessarily need to be
• 18:54
  fast but something like pacing or Updating budgets.
• 18:59
  So we know how much user has spent because we don't know how
• 19:03
  much time was spent until we get like a because we only charge
• 19:07
  you don't charge people for how many ads are shown.
• 19:09
  Well at least the way that overdid it.
• 19:10
  It's based on how many people clicked on the ad so you need
• 19:12
  that actual feedback, right? So you can imagine a world where
• 19:16
  like someone puts out an ad. They have 100 dollar budget and
• 19:20
  and they spent 10 bucks in their first day.
• 19:22
  And you maybe want to paste it over like a whole week.
• 19:25
  So you want to stop showing at but you need that feedback you
• 19:27
  need that feedback quickly because if you don't get You're
• 19:28
  just going to keep showing ads over and over, you're going to
• 19:30
  run out of the budget like pretty much immediately.
• 19:32
  So when we talk about like the different requirements, it's not
• 19:35
  that like I would say like the requirements, a lot of them come
• 19:38
  out come from having that many clients and trying to balance,
• 19:40
  like the needs of all of them or trying to make sure that they
• 19:42
  all get the needs that they have.
• 19:44
  So speed was a big one for. They're mostly for the debt, the
• 19:47
  downstream like, automated systems, like pacing and budget
• 19:50
  updates. It also has the extra but like,
• 19:52
  and some of them just give extra benefits, right?
• 19:54
  Like if it's fast then users get to see updated metrics
• 19:57
  immediately. Instead of like what we were
• 19:59
  talking about earlier with Spotify, where I know for a fact
• 20:02
  that like the my Spotify metrics are at least a day old.
• 20:05
  It'd be cool if I could see them in real time, right?
• 20:06
  And so that's what we just we get to make them better as
• 20:09
  product. And one that's actually kind of
• 20:11
  unique because I don't think a lot of ads systems use that for
• 20:13
  reporting so that speed reliability.
• 20:17
  I think that one's just kind of like standard like this system
• 20:20
  deals with money and anything that deals with money and like
• 20:23
  performance of like and directly like discussing like performance
• 20:27
  of Ads for customers. So, we deal with a customer
• 20:30
  data, we deal with money, it's got to be reliable.
• 20:34
  It's pretty much like non-negotiable and then
• 20:37
  accuracy, its kind of on a similar vein where, you know,
• 20:41
  we, again, we're dealing with with financial data.
• 20:44
  So if it's not accurate, it means that were either going to
• 20:46
  over charger. Undercharge.
• 20:47
  Merchants both are bad, undercharging boober loses
• 20:49
  money. Overcharging, we lose the
• 20:50
  customer, trust all that stuff. So that's kind of where those
• 20:53
  requirements come out. Typically, it's hard to get all
• 20:58
  three. Especially reliability and
• 20:59
  accuracy like at the same time and there so it's not that we
• 21:03
  didn't really make trade-offs to be honest.
• 21:05
  Like like there are there are trade-offs to make sometimes but
• 21:08
  we decided to try to like the goal of this system is to build
• 21:11
  something with without having to make those trade-offs.
• 21:13
  The trade-offs typic came in like extra work, you know, or
• 21:16
  came in like having to set up things in a way that was more
• 21:19
  complicated. So that's why I would say like
• 21:22
  the trade-offs are so there that was kind of my my next question.
• 21:28
  Going to be like, what were some of the biggest trade-offs?
• 21:30
  But it sounds like you just made.
• 21:33
  Sure to say these three, things are the most important to us.
• 21:37
  So we're going to pound the pavement until we get them,
• 21:41
  right? What were some of the challenges
• 21:44
  that you encountered as you were going through that Journey?
• 21:48
  Yeah, I mean so so like one, I guess like what we say, no
• 21:52
  tryouts. I guess the trade-off is
• 21:53
  complexity, right? Like this.
• 21:54
  This is a lot more complex than like what you could have done.
• 21:57
  We could have also just if we didn't have speed as a
• 21:59
  requirement. For example, we could have just
• 22:00
  made a batch system that's pretty easy to get reliable and
• 22:04
  and an accurate, right? Like that's like not that hard
• 22:08
  and you have then you just have that like one day delay of
• 22:10
  typical things like you ingest data.
• 22:12
  It goes to some data warehouse once it's there you run some
• 22:14
  like spark job or query that like that's a pretty standard
• 22:17
  practice with In the industry and then you're getting your
• 22:20
  like accuracy and reliability through that and less like
• 22:23
  assuming that your job infrastructure works and doesn't
• 22:26
  like fail randomly. But anyway, so the trade-offs we
• 22:30
  have was like the complexity of this system.
• 22:33
  And I would say like some of the some of the more interesting
• 22:39
  pieces are actually like, limitations of uber
• 22:43
  infrastructure, not necessarily like, and I think that's where
• 22:46
  the miscommunication came when you guys were looking at the
• 22:50
  Sorry, I think I misunderstood the question a little bit.
• 22:52
  Could you repeat it? Yeah.
• 22:55
  It was it was around more of the challenges like what some of the
• 23:00
  hardest parts for you had been when did you run into walls?
• 23:03
  What were some obstacles? Yeah, yeah, I think I've
• 23:07
  reliability was like, one of the hardest parts.
• 23:09
  Sorry, not reliability, accuracy was one of the hardest parts.
• 23:11
  Actually, depending on the time of the project, different pieces
• 23:14
  were harder at the beginning. Accuracy was really hard and
• 23:17
  that's because we were that and that's what motivated a lot of
• 23:20
  this exactly want stuff work that we did.
• 23:23
  But it wasn't so easy. Because like no no team at Uber
• 23:26
  had done exactly once before. So we had to like enable it in
• 23:28
  the casket. Topics in the Flink jobs, it's
• 23:31
  not that hard, like they're usually just like configuration
• 23:33
  changes, but this is kind of what I'm saying about like,
• 23:36
  having a deeper understanding of uber infrastructure.
• 23:40
  I think, pretty sure if I remember correctly, at the time,
• 23:42
  like, we couldn't enable it in certain types of Kafka topics.
• 23:45
  So, like uber has a concept of like local topics and aggregate
• 23:48
  topics and the aggregate topics which is like, you know, if Uber
• 23:52
  is a distributed system across Regions which it is for like, a
• 23:56
  whole bunch of reasons, you know, messages will first go
• 23:59
  into like the local topic. So some things in like the east
• 24:02
  of the state's, it'll go there for like if it got generated in
• 24:05
  the East and similarly for the west and then we have this
• 24:08
  concept of aggregate topics, which Aggregates the data
• 24:10
  between those the localized ones.
• 24:13
  So it would have been a lot easier if we could have put like
• 24:14
  an exactly. Once thing on the egg on the
• 24:16
  aggregate topics because then we have all the data, but that was
• 24:20
  like a limitation of Ubers infrastructure, we couldn't do
• 24:22
  this, we had to do it on all the local ones which means that I
• 24:24
  had to do a whole bunch of like extra extra work.
• 24:29
  Like we have a final job that literally sinks from the both
• 24:32
  regions. Like it.
• 24:33
  We have a fling job that that's pretty much his entire job is to
• 24:35
  sync data from from two different regions.
• 24:38
  Those are some unlike the complexities there than just a
• 24:42
  mother complexities with being the first team to do this.
• 24:44
  So, you know, things are never as advertised, especially when
• 24:47
  you're working at a big company. So, a team might say, like, oh,
• 24:50
  this works and then you find out, it doesn't actually work.
• 24:52
  So, one of our like Downstream Tendencies, which was Pinot had
• 24:58
  this feature up Sir that we were really leveraging and it's not
• 25:01
  that the observation didn't work but working with the Kafka team
• 25:04
  like the it all kind of again it's this is getting down to the
• 25:08
  nitty-gritty details but it all kind of depended on the on the
• 25:14
  in gesture to Kafka because the Absurd feature depends on every
• 25:17
  message of the same with the same primary key goes into the
• 25:19
  same Kafka partition. So the kepco partitioner that is
• 25:23
  used has to make Sure, that that that is true.
• 25:25
  And so we had like many times back and forth where like the
• 25:29
  partition or that, that was being used at over just like,
• 25:31
  wasn't doing it properly and therefore, like the feature
• 25:35
  wasn't working as expected. So when I did inspect my dad, I
• 25:38
  saw it's like a ton of duplicates and I was like, why
• 25:39
  are we getting all this duplicate data, this doesn't
• 25:41
  make any sense, pain through looking through it, like going,
• 25:46
  and we had to, like go in and literally change like the
• 25:47
  partition or logic and stuff like that.
• 25:49
  So, a lot of the complexities come actually from working
• 25:52
  really closely with the infrastructure teams.
• 25:54
  Metadata platform team member just like, yeah, trying to
• 25:58
  understand. Probably more than I any back in
• 26:01
  engineer, probably. Usually, understands of, like,
• 26:04
  how exactly everything is working at over.
• 26:07
  I think that, I mean, one of the design considerations, it seemed
• 26:10
  to be pretty important in the system overall, which I think
• 26:13
  you alluded to a little bit. Was this exactly once.
• 26:16
  Semantics, right, the idea that the event should only process
• 26:18
  the single time. Can you talk a little bit about
• 26:21
  why that was so crucial to the system as it was built?
• 26:24
  It's yeah. And I'll say that it's like, not
• 26:28
  exactly. Exactly.
• 26:29
  Once, like, there are pieces that are that are using exactly
• 26:32
  one. Semantics across like, certain
• 26:33
  jobs and also, there's that kind of fake it.
• 26:37
  I'm not fake it but like it, you still get the result.
• 26:39
  Like, I was mentioning like Pinot uses and up Sir feature.
• 26:41
  So even if we do duplicate it, we're never going to write to
• 26:44
  the database twice. But for example, like what the
• 26:48
  hive table that we use or like our data warehouse, there's no
• 26:51
  way for us to like, d duplicate on writing, so, YouTube people
• 26:55
  get on the query on the query side.
• 26:56
  So we know anyone who's acquiring that data knows to
• 26:58
  like use a use a primary key as as a gene duplication key when
• 27:02
  they're or is like an item potent c key or something like
• 27:05
  that, you know? So so exactly once was crucial,
• 27:12
  mostly for the reliability and the Act and the accuracy, Like,
• 27:17
  Flint jobs, do go down for a variety of reasons, all the time
• 27:20
  because whole system is built off the back of flank, right?
• 27:22
  And things happen. You might run out of memory.
• 27:25
  I don't know. Something like there's a million
• 27:26
  reasons for a wife. Like job could go down.
• 27:29
  And when that happens, it restarts, and when it restarts,
• 27:32
  it's going to reprocess all the events from like the previous
• 27:34
  check point from from when it like last saved its state.
• 27:39
  And if a reprocess is out of events, then you're going to
• 27:42
  overcharge Merchants. It's like the same.
• 27:43
  It's all the same idea. So, that's where exactly wants
• 27:45
  comes in. It's just like, it really helps
• 27:47
  with that act. It gives you those accuracy
• 27:48
  guarantees that you need and also gives you some security
• 27:52
  when things go wrong that you're not going to You're not it's not
• 27:56
  like it's tied necessarily to reliability like interesting.
• 27:59
  Make your systems more like it doesn't keep them up longer or
• 28:02
  keep them up in a more robust way.
• 28:04
  But when they do when bad things do happen you do have that
• 28:07
  confidence to know like okay we're knock at least we'd run.
• 28:10
  We're not going to process everything.
• 28:12
  So it gives you that like kind of peace of mind, you know, one
• 28:15
  of the I mean you mentioned it's not exactly exactly once and I
• 28:19
  thought that was really funny because it feels like you the
• 28:22
  title of the blog post is a real-time system.
• 28:24
  Mmm. But even in like a porter who
• 28:26
  saves like not exactly real time.
• 28:28
  I kind of think. Yeah, it is real time.
• 28:31
  It's just that like we put a delay there like it is all
• 28:33
  again, still technically real time.
• 28:35
  It's just like we have a specific delay because when
• 28:37
  you're building these systems, you can specify like checkpoint
• 28:39
  intervals and you can specify like what, what your window of
• 28:44
  so like is it not real time? If I say, like our window,
• 28:47
  length of like grabbing events is one minute.
• 28:49
  Like, I would say, I would argue, it's still real time.
• 28:51
  Even if that window was like 20 minutes, I would say, it's still
• 28:53
  real time. Because the processing is all
• 28:56
  happening in real time, you know?
• 28:58
  But yeah. But it's not like instant I
• 29:01
  could maybe I should have ordered it better.
• 29:02
  I should have said like it's not instantaneous.
• 29:03
  You know that might have been a better word.
• 29:06
  Well no. But I think I mean what you get
• 29:08
  at is this like interesting Nuance, right?
• 29:09
  Which is that real time is kind of this catch-all term and
• 29:12
  actually it seems like if you really dig into what it means,
• 29:15
  there's actually a lot of different gradations of
• 29:18
  real-time, right? Not just like, oh literally
• 29:20
  instantaneous versus a batch it takes a day to process, right?
• 29:24
  Actually there's a whole spectrum that you can you can
• 29:27
  live on and that's maybe. So, you know, in a sense your
• 29:30
  system is still real time by some definition of the word.
• 29:32
  It just may not be in like a microsecond real-time or
• 29:35
  something. Yeah, it's yeah.
• 29:36
  It's more about like the, the way the system operates rather,
• 29:40
  than rather than the time it takes in general and because
• 29:44
  even batch like batch you can do with like real time jobs, but it
• 29:47
  like, but what really I think batch gets maybe I'm someone all
• 29:51
  correct me if I'm wrong but my interpretation of like Batch or
• 29:55
  like the ethos of badges that, you know, you have data going
• 29:58
  into some data warehouse or some some database storage for like
• 30:02
  that's a time then you create a cut off and you say like OK at
• 30:05
  this. Now we're going to process X
• 30:07
  information within like these bounds and in the real time
• 30:09
  system that doesn't really happen.
• 30:11
  You're just as it's coming in your processing and that's kind
• 30:13
  of how I'm that's kind of how I understand the different the
• 30:16
  main difference if I were to think between the two of them.
• 30:19
  Yeah. I mean some people even say that
• 30:20
  bad just like I think a special case of group of real time.
• 30:23
  Yeah. So, alright, let's remove them
• 30:27
  one thing that I also constantly think about, and especially,
• 30:31
  when we're looking at this system and you built it from the
• 30:33
  ground up, what were some of those main?
• 30:37
  Because this is any engineer who's coming in to an Uber, or a
• 30:42
  Google, or Amazon, or whatever, they get those interview
• 30:45
  questions. Like, alright.
• 30:46
  So build me the system. And so how did it actually look
• 30:50
  when you were building the system?
• 30:51
  What were some of the main questions that you were asking?
• 30:54
  Was it around and how did you land on these?
• 30:58
  I guess you kind of talked about that already.
• 30:59
  How you landed on these three pillars?
• 31:02
  But what what questions were were there that made you think,
• 31:09
  all right, this is how we're going to choose these design
• 31:13
  decisions. Like you said, let's go with Top
• 31:15
  Gun Flink was that because numbers already using that is
• 31:19
  everything already there in a way you had to kind of piggyback
• 31:22
  on what was there? Like, what is That look like
• 31:25
  yeah, that's a good. That's a really good question.
• 31:29
  If I'm thinking about it, I think a lot of it, like I said,
• 31:32
  goes back to who your customers are or who your dependents, who
• 31:36
  your clients are who's dependent on your system.
• 31:39
  And that's typically how I approach any sort of like design
• 31:42
  or project as I think, like, who, who, who needs this in,
• 31:46
  like, whatever, whatever it is. And then try to work back the
• 31:49
  requirements from there. Our team was really strapped.
• 31:52
  We did like typically I would imagine that's the job of like a
• 31:53
  product manager but, you know, we didn't have one it.
• 31:56
  So so you kind of have to Yourself but even if you have
• 32:00
  one, it's still a good idea to do it as an engineer because
• 32:03
  like when this is very much like the there's one client is like
• 32:09
  an actual customer of as the, all the other ones are either
• 32:12
  Engineers or systems or like, those are my clients.
• 32:14
  So like, I don't even know if a product manager would have
• 32:15
  called that stuff out. So I usually start going there
• 32:20
  and then working my way back, and then once I know, like,
• 32:22
  okay, what are, what, what does this system need to do then?
• 32:26
  I start thinking, okay, like what is the best way to do that?
• 32:28
  And sometimes those are decisions can look, especially
• 32:30
  at a big company can be made for you.
• 32:32
  Something like half. Go, like uber is one of the
• 32:34
  biggest deployments of caffeine in the world.
• 32:36
  You're not going to use anything else for message queues.
• 32:38
  Like, you could, if you really, really push and fall for it, but
• 32:41
  it would be ridiculous. Like I would be the wrong
• 32:43
  decision 99% of the time. All of our tooling, all of our
• 32:46
  like, it's so easy. I do bird takes like clicks to
• 32:49
  set up a capture topic and then have all data that rights to a
• 32:51
  decoder, a data warehouse, for example, and they have like good
• 32:54
  in gestures setup from like Calcutta Pinot or other like do
• 32:58
  like didn't they? Is and stuff like that.
• 33:00
  So in general like you would have to be a you'd have to have
• 33:03
  like the best reason in the world to not use some of these
• 33:05
  things that said you know, this isn't use P note that wasn't
• 33:09
  made for me. This is Accused flank not
• 33:11
  necessarily made for me. There's other options there.
• 33:14
  I had a bit of experience that with link because the project I
• 33:17
  worked on previously for grocery was searched ingestion, so I
• 33:21
  built a. I was like I was responsible for
• 33:23
  ingesting, all of our data from like the grocery stores into a
• 33:27
  search index so that we could use it like a Is such a just for
• 33:30
  search on the app and I was using flick for that.
• 33:33
  So I had some experience with it.
• 33:34
  That was the first time I'd use Link at the time.
• 33:37
  So I had some experience with that.
• 33:40
  And so for me it was a mixed, the questions I was asking was
• 33:42
  kind of like, yeah, like what do our customers need?
• 33:46
  What is the, what are the best ways to do that?
• 33:48
  One of the best technologies that I have available to do them
• 33:51
  and then like doing an analysis of them.
• 33:52
  So even for Pino, I think it was like a decision between like
• 33:55
  Pinot Druid, click house, like there's multiple versions of
• 33:58
  those same Technologies. And for flank, as you know
• 34:01
  there's flankers Kafka streams their spark.
• 34:04
  There's there's other things you can you can use.
• 34:08
  But the end of the day, like once you land on like, okay,
• 34:10
  this system needs to be real time.
• 34:12
  Okay, we need some real time streaming streaming framework,
• 34:14
  right? Those are that's when those are
• 34:17
  the questions that you like, I don't know.
• 34:19
  To me it gets kind of simple and gets easier once you understand
• 34:22
  fully like what the what the problem is.
• 34:24
  Hi, this is my Alaric. I'm the founder of pun intended,
• 34:28
  Innovation and confetti. May I and this is a quick Public
• 34:31
  Service Announcement. If you're tired of having
• 34:33
  nightmares about the machine learning models, you have in
• 34:35
  production, then I encourage you to subscribe to the ml Ops
• 34:38
  Community podcast, it's where all the cool ml Ops kids hang
• 34:41
  out and learn how to tame their ml models.
• 34:44
  Yeah, I mean I think it's it's one of the most amazing things
• 34:47
  about the system is it seems like the whole thing is
• 34:50
  fundamentally built on open source, right?
• 34:51
  The whole system as a whole is this Testaments just OSS.
• 34:56
  And I mean I wonder What extent that you know what you think
• 35:01
  that says about just like the state of OSS and the fact that
• 35:05
  there's these systems are so robust that you can build.
• 35:07
  Literally one of the largest companies entire add event
• 35:11
  processing infrastructure on top of it and it seemed it be
• 35:13
  resilient to do that, right? Yeah, I mean it's amazing.
• 35:17
  Like I said, I know Uber as a company, really, really cares
• 35:20
  about that and like the people who work there especially on
• 35:22
  those teams that they, it's like fundamentals to them.
• 35:27
  So like we mentioned we're talking about that up sir.
• 35:29
  A feature that was contributed by an Uber engineer to pinot,
• 35:34
  and we got to use it immediately, right?
• 35:36
  So it's like mutually beneficial, of course, like, the
• 35:38
  whole world now gets to use this.
• 35:39
  Anyone who's using p. No, I guess not the whole world.
• 35:41
  That anyone who using Pinot gets used to substitute feature.
• 35:44
  And now I also get to use it as an engineer at Uber, and I got
• 35:47
  to work really closely with that engineer.
• 35:49
  He's amazing stuff and I think he got promoted recently but
• 35:52
  senior staff engineer, you Peng really really.
• 35:54
  He's one of the co-authors like he wrote the blurb on on absurd.
• 35:59
  And he's incredible. He's writing.
• 36:00
  Like he published his blogs all the time.
• 36:02
  He contribute to open source. He's like a big advocate for
• 36:05
  this stuff. I think it says a lot about the
• 36:08
  state of Open Source because like these tools can be
• 36:10
  leveraged by Massive companies, but it also says something about
• 36:12
  like the kinds of companies that give back because a lot don't,
• 36:17
  you know, like Amazon doesn't contribute crap, but open
• 36:20
  source, they just they just write a rapper region and
• 36:23
  standing on it. But you want to this changing
• 36:25
  it's an ongoing problem is changing.
• 36:27
  Yeah, maybe but in general, like there are MO is to actually like
• 36:30
  steel open source essentially and and put a rapper.
• 36:33
  Yeah. Sell it back to people, okay?
• 36:34
  Like it's actually a much more. They just consume consume
• 36:38
  consume from it without giving nearly as much back as they're
• 36:40
  consuming and as much as they're benefiting and then Uber is a
• 36:45
  good example of a company that like they leverage open source,
• 36:48
  like a credible, like we said, like this whole thing is built
• 36:50
  off the back of Open Source, but they give so much back to it.
• 36:53
  Like, you know, they give back updates for Cafe of the give
• 36:55
  back updates for Pinot for Flink, like they are, they are
• 36:58
  part of the Ecosystem. And I think that's a very
• 37:01
  powerful combination when you have when you have the, it can't
• 37:04
  I guess I was saying it can be like a good marriage of the two
• 37:07
  and when it is, it's really awesome.
• 37:09
  This is my first experience of it, to be honest.
• 37:11
  Like I had never worked at a place that like contributed so
• 37:13
  much, back to open source and where it was such a big, big
• 37:16
  piece of it. Yeah.
• 37:20
  Alright. So I've got one here that I
• 37:22
  wanted to ask that I think is, is interesting and it's a, when
• 37:27
  it comes to real time processing, In systems for
• 37:30
  analytical, use cases, such as like data, scientists analyzing
• 37:36
  data and a hive database. How important are having
• 37:40
  real-time systems for predictive use cases as well.
• 37:47
  Interesting. Like I mean, I think it it kind
• 37:53
  of depends on like every answer to anything, it depends on on
• 37:57
  what the system is like, what, what you're trying to predict
• 38:02
  and how that changing data affects the prediction.
• 38:05
  So, previously before I worked at, Uber actually worked at a
• 38:08
  company called Shopify, and we were working on a recommendation
• 38:11
  system for, for marketing, for merchants to be like, hey, you
• 38:16
  should run like an ad campaign for this because I don't know.
• 38:20
  We have like, Model that tells us like a product.
• 38:22
  You sell is trending or something right now.
• 38:25
  Now the I guess the question I would ask for that is like, you
• 38:29
  know, if new product data coming in at the like, Ms, changes the
• 38:35
  outcome of like that recommendation.
• 38:37
  Then it's a pretty important but it probably doesn't in those
• 38:40
  cases. Like, like, I'm sure there are
• 38:42
  some use cases, we could think about where like, every baby,
• 38:44
  like, I don't know trading. If you're building trading
• 38:47
  software and like the the ins and outs of like each individual
• 38:49
  trade could change, The prediction of like a move you
• 38:52
  should make, if you're trying to make a predictive predictive
• 38:54
  trading software something like that, I could see something like
• 38:56
  that being really important. But I think a large amount of
• 38:59
  use cases for these types of predictive systems, probably
• 39:02
  don't care about like the individual event coming in, if
• 39:06
  that makes sense. Yeah, I think it's kind of
• 39:09
  interesting idea because, you know, a lot of our listeners
• 39:12
  are, I'm a lobster practitioners people that some do built,
• 39:16
  purely analytical systems as data scientist, but many of them
• 39:20
  do. Tend to build machine learning
• 39:22
  systems that are predicted by Nature.
• 39:24
  So there is this question around yet, broadly.
• 39:26
  There's a trend maybe towards more real-time systems.
• 39:30
  But are they really necessary today for like a small
• 39:33
  organization, like maybe for Uber having a real-time system
• 39:35
  is crucial for a lot of these Downstream use cases.
• 39:38
  But it does an organization, that's a be smaller and smaller
• 39:41
  in stature earlier on, and it's in its life cycle, with regards
• 39:45
  to analytical and predictive use cases, do they still have value
• 39:48
  for something? That's Near real time, you know,
• 39:51
  and I wonder what your thoughts are on that.
• 39:54
  Yeah. It's I mean I've actually talked
• 39:56
  a lot about this and I thought a lot about it again.
• 39:58
  I just think it depends on your product like what are you
• 40:00
  building? What value are you getting out
• 40:02
  of that real time? Like I said, for us like most ad
• 40:05
  systems are not real time. Even a lot of what I did when I
• 40:07
  was building out, this system is that would go use, you know,
• 40:10
  luckily, because I play in good kid, I had an excuse to run ads
• 40:14
  to test out what other systems did.
• 40:16
  And I went through there like analytics, most of them are not
• 40:18
  giving you real time feedback. Back on like the performance of
• 40:20
  the app. And I understand why I like most
• 40:23
  advertisers probably aren't sitting there like clicking
• 40:25
  Refresh on their dashboard to see like how their campaign is
• 40:28
  performing. That said you know if you did
• 40:31
  have access at a know it could be like a chicken and egg thing,
• 40:34
  right? Like the more real-time data you
• 40:36
  have maybe the more ideas you can come up with four products
• 40:39
  built on top of that. For example, like if an ad could
• 40:41
  you interject Midway through an ad because you know it's not
• 40:44
  performing as well as another ad that you just read and maybe you
• 40:47
  could you could give a notification to your Like yo
• 40:50
  this is not working. Like the last one you did was is
• 40:53
  way better than this one. I've given the same point life
• 40:55
  cycle within its within its run, you should change something
• 40:58
  about a change, the title change, the description change
• 41:00
  the picture. Like, you know, that's not to
• 41:02
  say that there aren't things to do it to build off of, but it's
• 41:06
  a bit of a Chicken and the Egg. You're probably not going to
• 41:08
  think of those use cases unless you have the data to like action
• 41:11
  on. So, so at.
• 41:15
  Yeah, like I said, it's tough, I'm sure they're like, I said, I
• 41:16
  think the majority of the use cases, probably don't need it.
• 41:20
  But it's not to say that if you had it, you couldn't figure out
• 41:23
  cool ways to leverage it. You know that's fascinating to
• 41:26
  me like how it can open up new doors when you do have that
• 41:30
  potential and you can build new products on top of it.
• 41:33
  I think that's super cool to think about and just let the
• 41:36
  mind run wild. I actually could that be a
• 41:38
  different because that be a differentiator, right?
• 41:40
  Like if it is a use case like if doordash is running ads which
• 41:43
  I'm sure they probably are at this point, you know is are
• 41:46
  there ads in real time? Like I have no I don't
• 41:48
  personally know but like That could be one of the
• 41:50
  differentiators for a restaurant like, hey, I'd rather have like
• 41:53
  just small edges right? Like that make you a different
• 41:56
  like better, better competitor? Yeah.
• 41:59
  Chris that moat for sure. So there was actually this last
• 42:03
  week, I was talking to a guy who's coming from and this is a
• 42:09
  little bit of the same question, but with a different spin in
• 42:12
  story form. So bear with me here, the I was
• 42:16
  talking to this guy who just came into the community and And
• 42:19
  he was coming from a data science background.
• 42:22
  He just actually, he was coming from an electrical engineering
• 42:26
  background, got into data science.
• 42:27
  Now, he's working at this fintech as a data scientist but
• 42:30
  they don't have anyone that actually productionize has the M
• 42:33
  ml systems and so he's starting to move more towards ml Ops and
• 42:39
  his boss is really on his case about how things need to be real
• 42:44
  time and I was like, dude try and stay away from it as much as
• 42:49
  possible. For as long as you can because
• 42:51
  it adds so much more complexity and you coming from a data
• 42:54
  science, background all of a sudden having to set up a
• 42:57
  system. That is for real time is not
• 43:00
  going to be the most fun thing in the world.
• 43:03
  If you're looking at that and you're having to coach someone
• 43:07
  through that or you're the boss there again, I understand it
• 43:11
  depends and every use case is different.
• 43:14
  This use case was for fintech. So obviously I think there is a
• 43:18
  strong use case and In most fintech they want things real
• 43:22
  time. Probably for fraud detection, I
• 43:24
  would imagine how are you looking at that?
• 43:27
  Is it like this guy needs to go running from this company
• 43:32
  because or he needs to try and get the try and Champion for
• 43:37
  someone to be brought onto his team.
• 43:40
  That knows a little bit more about system design.
• 43:42
  Like, how do you think through that stuff?
• 43:46
  If it was me, I would probably just learn it and do it.
• 43:48
  That's that's how I would. That's how I would approach it
• 43:52
  because that's kind of what I did here.
• 43:54
  Like I didn't I had barely used. Like I said I used blink once
• 43:56
  before but I barely used like and I had to read it.
• 43:59
  I read an entire, I don't know if I have it but I read an
• 44:02
  entire book on flank to build this system, you know.
• 44:04
  Like I don't think I didn't have any experience with real-time
• 44:06
  systems before I built one and even this version like I built a
• 44:10
  shitty MVP that I didn't write about obviously but I built a
• 44:15
  really crappy MVP. First and then ivory-bill got to
• 44:18
  rebuild it all from scratch because like that's what it
• 44:21
  necessitated, so eight. But all that said, big obvious
• 44:26
  caveat is like is he in the environment with the people that
• 44:29
  will allow him to like learn while he builds it with the
• 44:32
  Forgiveness? Like when you do inevitably mess
• 44:34
  up because you are doing it for the first time, you know and
• 44:37
  liked is that the space for him to be able to grow?
• 44:39
  If he's in that space, then I would say, like, yeah, just
• 44:41
  learn it, learn to do it. I don't think there's anything
• 44:43
  that like, you can't really learn if you spend it.
• 44:46
  If you if you're motivated enough to do it it might just
• 44:50
  take more time because he hasn't done it but if he's got that
• 44:52
  support then like I think it'll it helps right.
• 44:55
  You level up your career quite literally but also like you know
• 44:58
  that does not just you know, how I did personally get promoted
• 45:01
  off this project but I didn't even if I didn't I would be so
• 45:04
  much better. Like in next when we were
• 45:05
  talking about interview questions.
• 45:06
  Like next time I'm in an interview I get to talk about
• 45:08
  this project when I'm interviewing accompanies.
• 45:11
  This is the project I talk about because it's the most complex,
• 45:13
  the most interesting and I think it's the most I think I've ever
• 45:17
  worked on even though like it's been a while since I've been
• 45:19
  there and it probably will be one of the more impressive
• 45:22
  projects I ever get to work on if that like if I'm being
• 45:24
  completely honest like I'm not sure I'll get to work on
• 45:27
  something this interesting for a while.
• 45:30
  So so I do think I would that that would be my advice to coach
• 45:36
  them. Like try to work with whoever
• 45:38
  his manager is whoever's putting that pressure, try to get them
• 45:40
  to understand. Like these things take time,
• 45:42
  push back on it. Try to get like timelines like
• 45:44
  appropriately scoped. And tell him like, and do enough
• 45:48
  research to understand like this is what it's going to take to
• 45:51
  build us. If you really want it, like, I
• 45:52
  need like six months, you know? And I need like two other
• 45:55
  people. And this is like, then we can
• 45:57
  get it done. Like, I didn't build this all by
• 45:59
  myself. Like, one of the other authors,
• 46:01
  or the two other authors on this were like back end Engineers who
• 46:04
  I worked with, like, closely on this.
• 46:06
  Like, you know, we had James Kwan, who built out, he built
• 46:10
  like a lot of the building infrastructure for ads, but he
• 46:13
  before he was on that, he was like, working with me, build it
• 46:16
  work. On these jobs and like, figuring
• 46:17
  out this stuff and Yuri bondarev, Luke is like an
• 46:20
  incredible engineer on ads as well, and he did so much, so
• 46:24
  much work around, even just like helping with the consistency of
• 46:29
  the system of the reliability of it.
• 46:32
  Yeah, I think that real time is an interesting thing in the ml
• 46:36
  community and I also have Clement anecdote.
• 46:39
  But also I think literally having a conversation that
• 46:41
  happened yesterday where you know nominally I believe the
• 46:45
  future will be increasingly real-time, right?
• 46:48
  They like it just makes so much more sense.
• 46:50
  We don't yet know all the things that we can do as you said and
• 46:53
  in sometimes we need to get there to start enabling all
• 46:55
  these new use cases. I think we will go there.
• 46:58
  I think it's generally reasonable to Assume that more
• 47:01
  data more, you know, systems are more up-to-date or quickly, more
• 47:05
  fresh, all will lead to better outcomes.
• 47:08
  And that seems pretty, you know, in controversial.
• 47:11
  However, I also spoke to someone who, you know, was working at a
• 47:15
  company where they were trying to basically help data
• 47:20
  scientist, productionize their systems and they spend six
• 47:23
  months targeting the real-time use case.
• 47:26
  And then after six months, they sort of pivot away from that
• 47:29
  because they just weren't seeing enough.
• 47:31
  I've seen enough examples of people solving these kinds of
• 47:33
  problems to build an entire company today.
• 47:36
  Around that, you know, I mean for and since then they've gone
• 47:39
  to more batch base predictive productionize in our systems.
• 47:43
  So I'm, you know, it's interesting sort of back and
• 47:46
  forth between his today when we need real time she would be
• 47:49
  thinking about building up that infrastructure today or is this
• 47:52
  the kind of more like a one to two year from now, sort of a
• 47:54
  problem. But this is kind of where, like,
• 47:56
  it's also a thing at scale, like I haven't mentioned it when I
• 47:58
  was talking about my clients, like the data scientists, Not
• 48:01
  the ones who needed real-time data.
• 48:03
  It was like the billing systems, it was the other systems, and
• 48:06
  just because of the nature of the system and being like, you
• 48:09
  know, honestly, like I can see a world where If this was a more
• 48:14
  mature team, like if it wasn't like, hey, we have one person to
• 48:17
  build this out. We have one person to build up
• 48:18
  like we literally had one guy had to do this system, which is
• 48:21
  me. We had one guy to do like even
• 48:23
  though people worked on it, it's not they were still working on
• 48:25
  their own things. Like the guy who was helping me
• 48:26
  was working on building a billing and we had the other guy
• 48:29
  working on like pacing like essentially at one point when
• 48:32
  all was said and done, we had one person per core service, I
• 48:36
  can see a world where like, if I had more resources or full team
• 48:40
  just for this aspect of it. Maybe I would have broken this
• 48:42
  out and I would have said, you know, Like, we don't need speed
• 48:45
  for data scientist, so they're gonna just do like a batch for
• 48:48
  their stuff. We're only going to focus on
• 48:50
  like the real-time aspect for billing and that's going to not
• 48:53
  even going to be related to life reporting and the whole
• 48:54
  reporting chain for like analytics for customers will
• 48:57
  just all be badging will be delayed and we'll only like, you
• 49:00
  know, this could be a lot more simplified.
• 49:02
  If it's just for the billing use case, it could just be like one
• 49:04
  flake job that processes clicks and sends them to this billing
• 49:07
  like service. You know, that would that would
• 49:09
  be pretty easy. I think one of the things one
• 49:13
  Like circumstantially, like being the only person working on
• 49:16
  this to me, it seemed easier to do it all all together, than to
• 49:20
  break it out into multiple things with different, like
• 49:22
  technologies that I'd have to learn like many different
• 49:25
  Technologies to do to do what essentially was like processing
• 49:29
  or the events for ads, where we're in my mind, it was easier
• 49:32
  to build out like one holistic view.
• 49:35
  It's not to say that this is like these fucking Jobs, live
• 49:38
  independently, a lot of in a lot of ways.
• 49:40
  So like it is still very extensible if we need to add new
• 49:43
  Use cases, we can easily add like you've linked jobs or
• 49:45
  whatever we can break them out but I could see a world where I
• 49:49
  would not have made that decision.
• 49:50
  If I had like way more resources and and time that's such a good
• 49:54
  answer. And it's such a interesting
• 49:57
  thing to think about because that constraint of you, having
• 50:01
  low resources at the end of the day, it helped you get something
• 50:05
  out and it helped you create something that was valuable with
• 50:09
  those within the constraint that you had and maybe If you spent
• 50:14
  more time and more energy and more effort on it, who knows
• 50:17
  where it would have gotten, right?
• 50:18
  It's in this land of an alternate reality that we never
• 50:22
  get to live. So yeah, I think, I think I
• 50:25
  asked one of the senior Engineers ones like dude, like I
• 50:28
  because I after we built it in and out and it was successful
• 50:30
  for a bit of like, did we really need to do this real time?
• 50:32
  Because I was just thinking about it and he was like, he was
• 50:35
  like, no but but it was our decision and we got to build it
• 50:38
  and it was more fun. And I was like, that's also true
• 50:40
  like, at the end of the day, like, you know, When you're
• 50:43
  really resource constrained and we didn't have like a lot of
• 50:46
  that we don't have a manager. For example, like we just had a
• 50:48
  director, we all report everyone in my team, reported directly to
• 50:51
  a director, you get to choose, you get to choose essentially,
• 50:54
  doing the director doesn't have time to like tell you what to
• 50:56
  do. And as an engineer you kind of
• 51:00
  get to choose what's like most fun, as long as it's not like
• 51:02
  getting in the way of your deliverables, like it, you can
• 51:05
  choose the thing that is like most interesting for you and
• 51:07
  that's kind of how we did it here.
• 51:09
  Yeah, so I've got one Last question for you, before we wrap
• 51:14
  this up and we used to call these War Stories but
• 51:20
  considering the current events that are happening.
• 51:22
  So close to my door, we're going to Rebrand them as ml.
• 51:27
  Oops. Do you have any any ml oops
• 51:31
  stories or what was your? What's one that jumps to mind?
• 51:34
  I'm not because I every engineer has a story.
• 51:38
  So what is one that you automatically think about?
• 51:42
  When I say Ml. Oops.
• 51:43
  And how can we learn from that? Yeah, I think.
• 51:52
  I mean, I've definitely push like bugs that?
• 51:54
  That have crazy. It packs them, like, bring down
• 51:57
  the whole system. I've done that.
• 51:59
  I'm sure I did that I didn't do it too much.
• 52:01
  I do /. Surprisingly actually, like I
• 52:02
  think I'm only did it with maybe like once or twice but I once
• 52:07
  pushed a bug in Amazon that like paste everyone in the organ that
• 52:11
  like was really, really, really sad.
• 52:14
  Well what the Lessons Learned there is just it's usually the
• 52:17
  same thing, which is like, how did we miss this in like the
• 52:20
  code review? What were the Issues like Debbie
• 52:23
  not have the right contacts for the not bright.
• 52:25
  Reviewers, not on it could have done more due diligence.
• 52:27
  Could we have set up better alerting or monitoring or tests?
• 52:30
  Why didn't we have a test to capture that specific thing?
• 52:34
  I find honestly the majority of these things, in my experience,
• 52:36
  come from me personally not having the context or like, the
• 52:39
  understanding of the system. And I think probably one of the
• 52:42
  reasons why it has didn't happen to me much.
• 52:44
  I do Burgers because I built the thing from scratch.
• 52:45
  So like I knew exactly how it worked like from the ins and
• 52:48
  outs and typically most of my troops were actually like
• 52:51
  infrastructure issues. Not things that I that I did.
• 52:55
  So I think that's kind of interesting words like when I
• 52:57
  think back of all the all my oops mistakes in the past.
• 53:01
  It usually came from some fundamental misunderstanding of
• 53:04
  how this system operated. And so my change would break
• 53:06
  something. Brilliant.
• 53:08
  Well, thanks so much for coming on here, man.
• 53:10
  And everyone listening check out this guy's band.
• 53:15
  Check out the new system design review, animation video that we
• 53:19
  put out and of course Read the original blog post and yeah, if
• 53:26
  somebody wants to reach out to you.
• 53:27
  What's the best way to get ahold of you?
• 53:28
  Jacob You can hit my email. It's a TS AF maaaaan at
• 53:34
  gmail.com or I'm on like Twitter and I think that's about it.
• 53:37
  I don't really use personally, I don't use anything.
• 53:39
  You can hit my band up. My band is on their own file.
• 53:41
  Yeah, I'm not personally very active on social media, but my
• 53:48
  band has so you had a good kid. Good kid band on on everything.
• 53:52
  There we go. Awesome.
• 53:54
  Thanks again man, that problems pleasure.
• 53:58
  Hey there, I'm a cinema star. Ava and I'm head of product, a
• 54:01
  teachable hub and if you are into envelopes and want to hear
• 54:04
  more than just buzzwords, subscribe to the MLS Community
• 54:07
  podcast and listen to the best practitioners in the field,
• 54:11
  sharing their wisdom.