Latest / Agentic Conversations (formally mlops.community) / Eliminating Garbage In/Garbage Out for Analytics and ML // Roy Hasson & Santona Tuli // #166

Transcript

Eliminating Garbage In/Garbage Out for Analytics and ML // Roy Hasson & Santona Tuli // #166 · Agentic Conversations (formally mlops.community)

0:00
Hi, I'm Shantan Azuli. I'm head of data at Absolver and
0:03
I have a morning ritual. Every morning I do a pour over
0:07
coffee for myself. Takes about 10 minutes and it
0:11
frees me freeze my mind. I buy local coffee.
0:16
So now that I live in Northern Virginia, I get coffee from this
0:20
local roaster called Vigilante Coffee.
0:23
They're they're quite good. And yeah, when I move I always
0:27
try out what's around there and and switch to something new.
0:31
Ohh hi. Hi.
0:33
My name is Roy Hassan. I'm the head of product at
0:34
Upsolver. My I don't actually drink
0:37
coffee, I drink tea. I'm a big fan of teas, different
0:40
kind of teas. My favourite is a a matcha tea.
0:43
So I actually get my matcha tea from a layered Hamilton
0:47
Superfoods. It has much of course it has
0:50
some functional mushrooms. I drink that every single In
0:58
addition to that I drink. I drink Chai tea every
1:01
afternoon. As well.
1:02
So that's my my daily ritual. Welcome, welcome, Welcome to the
1:07
ML OPS Community Podcast. I am your host Dimitri OS, and
1:11
today I'm flying solo as a host. But we've got 2 guests.
1:15
To double the fun, they're both working at Upsolver, and it was
1:20
an absolute pleasure to chat with them.
1:23
Let's dive in real quick. Just get a little background on
1:26
who these people are that we're talking to before I give you my
1:29
takeaway and we jump into the conversation.
1:32
All right, so Santona is a PhD, and she began her journey into
1:36
data through the world of physics, banging particles
1:39
together at CERN to detect rare particles, which is absolutely
1:45
awesome. I think I remember 1.
1:47
Time. I read in the newspaper about
1:49
how France was gonna bang some particles together and we may
1:53
blow up because I'm in Europe and so watch out for that.
1:56
But I don't know if San Antonio had a part in that or not.
2:00
I just remember that day. Now that I told this story, I
2:03
realised it was not as good as I.
2:09
All that ride with no destination.
2:11
There we go. This is what you get to expect
2:13
from me, folks. I'm on my own today.
2:15
I got no one to save me. Anyway, Santona Win and she
2:20
started working in Python when she realised that hey, I really
2:26
like data orchestration and she started programming data
2:30
orchestration tools like Airflow, helping improve the
2:35
developer experience and I believe she went and she worked
2:38
with the managed air flow company called.
2:43
Astronomer. That's right, Astronomer.
2:45
And currently she is at Upsolver, and she's got a bit of
2:50
a product mindset. She leads data engineering and
2:54
science, so data engineering and data science, and she is driving
3:00
developer research and engagement for the declarative
3:04
workflow authoring framework in SQL.
3:08
I think Roy was kind on me and he gave me a much less detailed
3:13
intro about himself or background.
3:15
So Roy is Head of product at Upsolver.
3:17
He's helping companies deliver high quality data to their
3:22
analytics and ML tools. Previously, before he was at
3:26
Upsolver, he led product management for AWS Glue and AWS
3:31
Blake Formation. This conversation I loved
3:35
because both Santona and Roy bring a product mindset to what
3:39
they're doing. And we dove deep into that.
3:42
We talked a lot about how you can be more successful coming
3:46
from that product mindset. They mentioned how you want to
3:51
talk to the end users, whoever it is that you're building for,
3:54
and you want to ask them. And it sounds simple.
3:57
It's like, yeah, of course you do that, but how many of us
4:00
actually do this if we're building platforms like we're
4:03
building an ML? Platform for our company or
4:06
we're building just any kind of data platform.
4:09
Do we go out and talk to the users who are using these
4:12
platforms? Do we go ask them every day, how
4:15
can I make it better? What can I do to meet your
4:18
needs? What are you struggling with?
4:20
This is the product mindset that both of our guests today were
4:25
advocating for and it was cool to see.
4:29
I mean, I. Think that everyone is going to
4:33
need a little more product. Fundamentals in the way that
4:37
they see the world, especially as more and more of these.
4:43
Hard problems that we thought could only be solved by using
4:48
some complicated machine learning.
4:51
They're starting to become less and less hard because you can
4:55
ping in API. That's just a large language
4:57
model. Maybe I'm oversimplifying and I
5:01
am going to be the first to say I don't know how true that is
5:04
going to be, but. I still think no matter what
5:08
whether or not that is true, a product mindset and product
5:12
fundamentals are going to be useful for you along your
5:15
journey. The other thing that Santona
5:19
mentioned that I wanted to point out was how she talked about how
5:23
at her job she is so cross functional and.
5:29
How important this idea of dog footing is?
5:32
So if you are building for others, you wanna ask them what
5:35
they're pains are, but you also want to work with the tools that
5:38
you're creating to see what is painful so you can have a little
5:42
bit more empathy for those users.
5:44
So let's dive into this conversation and hear it from
5:48
them specifically. But before we do that, I must
5:50
mention that this is a podcast that loves to be shared.
5:56
It's like your aunts Facebook post.
6:00
They just love to be shared by the family and they get about
6:03
two likes. But you know what?
6:05
Might as well share it. If you found some value in it,
6:09
please go ahead, send it to one of your friends.
6:12
Let them know that they got to listen to this.
6:14
It is an absolute joy that I get to do this every week.
6:18
I get to come and sit here and be with you all and interview
6:21
such bright minds and hear the way that they see the world, the
6:25
way that they build product, the way that they're thinking about
6:28
machine learning and ML OPS. Ohh, I love it and it is thanks
6:33
to you that that is begin with the back story on how this
6:46
happened, how we are here with both of you because I think it
6:50
is worth mentioning that I was hosting the Dracon conference or
6:55
I guess that's a little repetitive Dracon conference,
6:58
but I was hosting the Data Reliability Engineering
7:03
Conference and Roy, you were there, you were watching and
7:08
soaking in. All of the insights as was I,
7:11
while I was hosting and we were talking, we sparked up a
7:15
conversation and I realised that you have a lot of knowledge and
7:20
wisdom that is worth sharing. And so I invited you on here and
7:23
you said that's great, but it would be even better if I
7:26
brought San Antonio with me because she's doing all kinds of
7:29
incredible stuff with machine learning.
7:31
So this is the tag team if you will.
7:34
Really you are more in the data engineering space and Tony,
7:37
you're on the machine learning engineering.
7:39
Side but I get the feeling you probably have a lot of overlap
7:43
and I'm excited to explore that overlap.
7:46
I'm excited to explore what you all have been doing.
7:48
And then I'm also very excited to talk about what the different
7:52
solutions are and these buzzwords.
7:55
What is hype? What is not hype in your eyes
7:58
these days because there is so much noise, whether it is just
8:01
around data engineering concepts or if we God forbid we start
8:07
talking about AI and how much noise is in bath.
8:10
Field. So yeah, it's probably worth
8:13
just doing a little bit of a round on who you all are, what
8:18
you're doing, how you came to be, where you're at, and Royal
8:21
have you kick us off, please. Yeah, Sure, sure.
8:24
So I'm I'm currently the the head of product at a company
8:28
called Absolver. We're a data analytics product
8:32
that focuses on making high quality data ingestion easy for
8:37
companies and for users. I started in my data.
8:41
Career working for a WS back in 2016.
8:45
A lot of hands on working directly with with customers to
8:49
build their data infrastructure, right.
8:52
The data platforms, all you know AWS services, but building data
8:56
lakes, building data warehouses, all the infrastructure that kind
9:00
of goes with it. And then throughout my years in
9:02
AWS I kind of switch roles. I was doing some go to market
9:05
stuff for analytics for a while and then I became a product
9:08
manager. So as a product manager for AWS
9:10
Glue and then. Gonna be a slave formation, so
9:13
data processing and in data security.
9:16
And then I switched over I left and I went to to Absolver.
9:19
So I have a lot of kind of hands on experience helping people
9:23
write spark code, you know debug hive, hive QL code, you know
9:28
debug clusters and issues like that and.
9:31
Just a lot of hands on experience and and building and
9:34
dealing with these things. When I'm participating on on
9:36
events like Dragon and things like that, you know you hear a
9:39
lot of high level talk about these things and to me it's
9:43
always like well what's the next level down, right, Like what how
9:46
do you actually implement and what does that mean?
9:48
So that's kind of what gets me excited.
9:50
That is where I would love to go today in this conversation.
9:54
But before we do, Santona, for those who are not already
9:58
familiar with you, and if they aren't, I highly recommend
10:01
giving you a follow on LinkedIn. Because you share so much
10:04
insight on there, can you give us your story?
10:06
Absolutely. Thank you so much.
10:07
Demetrius kind as always, as you said, I have traditionally been
10:13
doing a little bit more mill than analytics or data
10:16
engineering. So I started through particle
10:18
physics where I mean it's really more end to end data science, ML
10:23
is used to extract the signals, but there's just so much more
10:26
that goes into it. There's capturing the data while
10:29
it's being produced at really fast rate, pipelining them
10:33
feature. Engineering, heavy feature
10:34
engineering. And then you go to the mill.
10:36
And I think that's true everywhere, right?
10:38
You can't find the Mulus case that doesn't involve just a lot
10:42
of feature engineering. And then from there, I worked as
10:45
an LP engineer, mill engineer for an LP product.
10:50
So I owned the ML service that we had.
10:53
So it was a lot of fun, a lot of like dev OPS type work as well
10:57
in addition to what you would normally or you might naively
11:00
think of as ML. And then I did some, I didn't do
11:03
some analytics. Work at Astronomer which was the
11:06
last place I was at before Absolver writing pipelines and
11:10
Airflow boats for Halifax use cases and e-mail use cases and
11:14
then I switched over here to Upsolver with Roy.
11:18
And yeah we do data. We're a data engineering tool.
11:22
We help you in just real time streaming or batch data into
11:27
your warehouse or lake. We also have a lake that does in
11:30
place transformations very optimised and it's just it's a
11:33
it's a cool tool that I wanted to.
11:35
To help explore. First question come in hot out
11:38
the gate is as a product and machine learning.
11:43
Profiles that we have on this call right now or this
11:47
conversation, How do you all find is optimal to work
11:51
together? So I think there's maybe two
11:54
parts to this question. I think the the first part is
11:57
how do I and Chandana work together to kind of.
12:02
Evangelise dot dot idea and then the second part is how do we you
12:05
know the product teams the data teams in the MLS teams work
12:08
together. So I think from from how do we
12:10
like myself and and Chantana work together.
12:13
You know Shantanu is the one that spends a lot of time out in
12:16
the community. I I do as well but she spends a
12:19
lot of time with the community data engineers, ML engineers
12:22
operators business leaders etcetera etcetera to kind of
12:26
understand what some of the challenges are right.
12:29
We all talk tech and we all can can read slides but when the
12:32
rubber meets the road right there's a bunch of.
12:35
Issues that come up that sometimes people just feel like
12:39
they're just. Part of the problem and I don't
12:42
really need to describe them because I'm just going to deal
12:44
with them, right. Like did a quality BB one of
12:47
them where like, well, just the way things have been, like I
12:51
don't like why am I talking about this, but we want to hear
12:54
those things, right? We want to get that information
12:56
out and then you know, something that does a really good job it
12:59
kind of extracting those pain points, those paper cuts.
13:03
And then together we sit and we try to understand well what are
13:06
the best ways that we can think about solving these things.
13:09
You know, as a technology on a market that's already addressing
13:12
those those things. Is there new technology that we
13:16
should develop? How do we incorporate those
13:18
things into our products so we can then help our customers
13:21
solve these paper cuts? Because the simpler we make life
13:24
for our users, ultimately the better it's going to be.
13:27
The second part of how do companies work together, right?
13:32
I think there's. And there's a shift in companies
13:36
mentality to become more product oriented.
13:40
Initially we were data-driven companies, now we want to become
13:43
more product oriented companies. It's really easy to say that
13:47
it's really easy to put that label, but in reality it's
13:50
actually really hard to implement because having a
13:52
product mindset is not just, you know thinking I'm going to
13:55
package this into a product. Like you have to think about you
13:59
know what are my users need, how am I going to solve those
14:02
problems, what technology exists, what?
14:04
You know what's the best way to solve these problems, not just
14:07
to solve them, right. So I think that's the piece that
14:11
we're still trying to educate, not just us, right.
14:14
I think the community or the the, the industry is really
14:17
trying to learn better of how to apply product best practise or
14:22
product thinking to more parts of the business.
14:25
The teams that are building the application that's facing the
14:28
users understand product better. They have, they do surveys, they
14:32
do studies, they understand, they get feedback, they measure.
14:35
Everything in these study that and they figure out how to make
14:38
things better. The internal teams don't
14:40
necessarily do that, especially on the data side.
14:43
Data side has always been. We're we're basically the end,
14:47
right. All the data flows downstream to
14:50
the analytics dam, right? And then when you get there,
14:53
somebody tries to do something with it, but there's very rare.
14:57
Really a conversation with the end consumers to say, well, does
15:00
this meet your needs? What can I do better?
15:03
How can I improve this? That's what we always see, like
15:05
finger pointing the business saying, yeah, these guys aren't
15:08
moving fast enough, they need to do more.
15:10
For us, these things always broken.
15:13
You know, how do we get them to move faster?
15:14
You know what? They're not going to move
15:15
faster. So I'm just going to do it on my
15:17
own, right? That's not the way we solve
15:19
things, right? We gotta communicate.
15:21
We got to work together. So we're definitely seeing that
15:24
conversation starting to to become more front and centre.
15:28
Which is really awesome to see and we're obviously
15:31
participating that and trying to learn from that and see what can
15:34
we alley on the technology side, on on a tool side to make
15:38
people's life easier. And then of course there's
15:41
people, processes that you can't really, you know, build into a
15:44
tool. Straight most straightforward
15:47
way I can think of of answering that question is by expressing
15:50
how much I love the fact that my role is cross functional and
15:53
just how cross functional it is. So its product and its data and
15:58
it's engineering. And it's also user research,
16:00
market research, like all of these things like have to work
16:03
together in order for me to do my job well and beyond.
16:07
It extends much beyond me like I feel like in the in the tooling
16:11
space, right? Like both at my previous company
16:13
and this company like if you want to.
16:17
Build for other data scientists and data engineers a useful
16:21
product that's also easy to use like user-friendly.
16:25
You have to do the, you have to use the, we call it dogfooding,
16:29
right. If you use the product, you have
16:31
to use other products out there, play around with it.
16:34
And I mean that's another part of it that I love is I just get
16:37
to play with tools. So like my the the number of
16:41
tools that I know you know how how to use is much larger than
16:43
it would be if I was on a team where we were using one stack.
16:46
So just that that. Exposure and and talking to
16:49
people and learning so much is is what I love about this this
16:52
role. And do you feel like that is
16:54
something that? Is.
16:57
Common across different teams in and especially when you're
17:01
looking at the machine learning engineer position.
17:03
Because there are so many stakeholders.
17:06
You do need to be talking to so many different people and
17:10
looking at so many different tools because one stakeholder
17:13
may optimise for one thing and another may be looking at
17:17
something else. Yeah, I would definitely say so.
17:20
And that's I think the other angle of the data product
17:23
discussion today is like what about building data products or?
17:28
Small products right? As opposed to being a product
17:30
manager or or a data scientist for a data tool.
17:34
And yes the the degree of overlap there in my experience
17:40
is is huge and like good data scientists and good machine
17:45
learning engineers will have a really thorough understanding
17:49
of. Whatever they're building as a
17:51
product right? And and it's it's I've, I've
17:54
I've found that it's a skill that you can develop with time.
17:57
So certainly it's it's a it's a learnable skill but not not only
18:02
understanding. It's not just about talking to
18:05
the different stakeholders and figuring out what everyone
18:07
wants, right. It's that you could have scope
18:11
creep but like things could get really large.
18:13
But then distilling that down and figuring out, strategizing
18:16
around what to build, what not to build and how to like address
18:19
as many of those. Stakeholder needs and use cases.
18:23
By building one thing to all of those, like thinking hard about
18:27
solving a problem, I think is a common thread.
18:29
It doesn't matter if it's a millwork or data work or product
18:32
work, it's really just problem solving.
18:35
I was going to just add to that. Things aren't gonna really hit
18:38
it on the head right? And we're we're kind of throwing
18:41
the word product around a lot. But many times in these in these
18:45
organisations, there really isn't a clear definition of what
18:48
does that mean. But there's a lot of things that
18:51
go into. Building a product and building
18:53
a sustainable, reliable product. And those are the things that we
18:57
really need to help companies unpack.
19:00
Because remember, like within a startup, you know, you have ML
19:04
teams, things like that, like people are building a product, a
19:06
product company, they they know, but in in enterprises, you know
19:10
there's a product team that's out there on the side and
19:13
there's a lot of internal infrastructure teams that can
19:16
say the word product, but don't necessarily know what that means
19:19
and what that entails. And it's really important to
19:22
sort of. Still that down into what does
19:25
that actually mean right? Like how do you build a road map
19:28
for your thing that you building.
19:30
What infrastructure are you going to build to be able to
19:32
serve that to your your end consumers?
19:35
What kind of guarantees and reliability are you going to
19:38
give them, right? Is there API spec right?
19:41
Is this thing going to be 99.9% available and or not what is
19:47
your support look like right. If something breaks you know
19:50
what is that going to look like right.
19:51
So there's a bunch of things that go into.
19:53
Making something a product that's beyond just.
19:58
You know, have a road map and and treat it as a product like
20:01
what does that mean, right. So I I think a lot of people are
20:05
still trying to understand what that means and we're sort of
20:09
saying the words, but we're not actually doing the actions and
20:12
I'm seeing like in our tooling where we're trying to build
20:16
more. Automation around this, like
20:20
I'll tell you, like an absorber, for example, when we when a user
20:23
uses it to deliver a data set to an end consumer, right.
20:27
Whether it's for analytics, OML. We try to bundle a lot of the
20:31
capabilities that you'd need around data management on
20:36
consistently updating the data, making sure that stuff.
20:39
You know if if your source system breaks or stops
20:42
delivering data for some reason, like your pipelines don't break
20:45
and all of a sudden you have to fix it, right?
20:47
Like we automate that recovery from from those failures.
20:51
And those are those are infrastructure things that you
20:53
have to build to have a reliable data product.
20:56
Now if you're not using a tool then you got to do it all on
20:59
your own. But I I really see like our
21:01
tooling in the in the industry need to get better at
21:05
simplifying this because the the teams that are building products
21:09
aren't infrastructure experts, right?
21:11
They don't have those tools, they don't have those
21:13
capabilities, right, Like an application team will have a
21:17
team that does infrastructure, they will have a team that
21:19
handles you know the back end servers in the databases like
21:23
they're designed, they're structured to deliver a product
21:26
but an ML. Team is not designed to deliver
21:29
a product. They're designed to deliver a
21:31
model, right and they're responsible to go spin up
21:34
Kubernetes or whatever with an API front end and deliver, you
21:38
know highly scalable inference endpoint that.
21:43
Maybe they don't know how to do right.
21:45
They can technically put it up, but they don't know how to, you
21:47
know, manage it as a product. And I think that's that's where
21:50
the disconnect is in a lot of the conversations.
21:53
There's one thing I wanted to add offer a slightly maybe
21:59
different view than Roy, which is it's so interesting to me
22:04
when thinking about product for external end users as opposed to
22:10
thinking about data and ML products for internal usage like
22:13
it's. There are things that transfer
22:16
and then there are things that don't.
22:17
And so like you know I kind of want to hear from Roy what you
22:20
think about you know you just said we kind of say product but
22:24
it's it's not always the same product like concepts and ideas.
22:29
But I feel like there is something to be said for you
22:32
know it's a different scale, right.
22:34
If you're often, I mean if you're building an ML
22:39
application let's say that you want you know your total
22:41
addressable market is is very large, you know everyone could.
22:45
Find some potential use use for your for your dual versus one.
22:49
You're building a prediction that's going to help let's say
22:53
your customer success team or something like that with their
22:56
work like that is and ML based data product that is internal
23:01
and it's a much smaller user base, it's a it's a very narrow
23:04
use case. So what are the things that you
23:08
think we absolutely should bring in the same sort of product
23:11
concepts into that internal product versus what really
23:15
doesn't? That makes sense to bring.
23:17
Scale is the one that is always the first thing the conversation
23:20
and and you're right in in, in some cases, if it's externally
23:24
facing, usually that scale is really high.
23:26
It's internal like you know like you just get the support team.
23:30
That scale isn't the problem, right, but it's the reliability
23:35
right. It's the availability of that
23:36
service. If that service goes down, if
23:38
your ML prediction service for your support team goes down and
23:41
they can't use it, is it a deal breaker as the world going to
23:44
end? Probably not.
23:46
But if you're gonna call it a product, if you're gonna
23:49
actually treat it that way, then treat it that way.
23:51
If today your company has two support engineers, tomorrow you
23:55
have 50 support engineers. Well, if you built it to support
23:59
too, with some, you know, easy kind of Python script that's
24:03
running on your local machine, it's not going to work for 50
24:05
users anymore. If tomorrow somebody says, oh
24:07
wow, this is really awesome, I want to embed this in
24:10
dashboards, right? So it's not an individual user
24:13
logging in and and and clicking a thing.
24:15
Now it's embedded in dashboards and those dashboards are
24:18
automatically. Refreshed every 10 minutes.
24:22
Now your scale just went way U right without even thinking
24:25
about it. So I think the scale is is
24:27
definitely one that can kind of like come and go like it.
24:31
It really depends. But the other ones are
24:33
reliability like what's the uptime of this thing?
24:36
Like how can I depend on this, what's the quality of this?
24:39
Like who is, who is in charge of that that model or that data set
24:43
that if it looks weird or it doesn't look correct or maybe
24:47
there's some missing values like who am I going to call?
24:49
Like who is going to fix it if I have any?
24:52
Any additions to it, if I have road map changes that I want to
24:55
make to that, right? Like I gotta have a process, I
24:58
gotta have a way for somebody to come in and say hey I I want you
25:01
to add this, so I want you to to change that.
25:03
The other thing that that you often see is.
25:08
You know, feedback. Like how do I learn from what's
25:11
going on? Like I can throw a model over
25:13
the fence and have people use it, but if I don't have any
25:16
mechanisms to collect feedback, I don't know if it's good or
25:20
it's not. I only know if people use it or
25:22
don't use it. What's to stop using it?
25:24
Do I just abandon it? Like I don't know what happened?
25:26
Right. So feedback is huge and it's
25:29
something that I don't see a lot in internal tools.
25:32
It's usually based on emails and asking people, but it's not
25:36
based on sort of like collecting live feedback.
25:39
Can we do in product right, you know how many, how many millions
25:41
of events we are collecting from all over our product just to
25:45
understand the usage. Those are things that you know
25:48
we talk about it, but they're usually are a little bit harder
25:50
to do and folks don't typically implement them especially for
25:54
internal facing sort of products.
26:00
I'm Emmanuel Mason, Machine Learning engineer at Stripe and
26:04
author of Building Machine Learning Powered Applications.
26:07
And If You Don't Want Your Machine.
26:08
Learning models to explode? Well, you should subscribe to
26:12
this podcast. How many times have we heard ohh
26:27
this is the company tool and it's so painful to use.
26:31
Brilliant. That's like the downstream
26:33
effect of not having those mechanisms in place and.
26:38
And sometimes you have to use those tools.
26:40
There's no other tool. Right.
26:41
Yeah, exactly. To the dismay of the whole team.
26:46
That's right. Stoked on it.
26:47
But they have to. And so I totally feel you on
26:50
that. That's the perfect example.
26:52
So like if if that if you hate that tool so much but you treat
26:56
it as a product, then you'll know right away that you're
26:58
using it. Aren't happy right?
26:59
If you don't treat it as a product then.
27:02
You know, eventually you'll hear all the emails and you'll you'll
27:04
see why. You don't know why.
27:05
Especially like we tend to get attached to our tools sometimes
27:09
too. I think especially if you're
27:12
building like, you know, developer Productivity Tools,
27:14
right internally and they say this is the best way to do this,
27:18
let's get everyone of the company using it.
27:20
But you know, maybe it's a bad, bad UI and you haven't.
27:24
You don't have any way of telling that people are like
27:26
you're actually going behind back doors to avoid using your
27:30
tool. This is a great segue into
27:33
another. Idea that I was thinking about
27:36
as you were talking, which is around building for data
27:41
engineers versus building for data scientists and how
27:45
different those profiles seem to be.
27:47
I mean you do have the occasional crossover data
27:51
scientist who understands Kubernetes and understands data
27:55
pipelines and understands everything that you need to know
27:59
when it comes to like DevOps and data engineering and they are
28:03
that Unicorn that you want to hold on to and never let them
28:05
go. But for the most part, a lot of
28:09
the data scientists that come into the fields have backgrounds
28:12
in economics or physics, and they maybe are messing around
28:17
with a bit of Python, but they're not necessarily so
28:22
advanced when it comes to the coding side of things.
28:24
And I know there's a lot of people that are very vocal about
28:27
this and trying to help. Data scientists become more used
28:32
to using Git or more used to using clean code fundamentals,
28:37
but as we are talking about this product idea and building
28:41
products, especially if it's like you're building an internal
28:44
platform for your company. And you need to service not only
28:49
data engineers, machine learning engineers, but also data
28:51
scientists. What are some things that
28:55
automatically come to mind? I am going to be a little bit
28:58
controversial. Ohh, I like that.
29:02
So I've had the. Maybe I've been especially
29:04
fortunate in this, but I've had the fortune of working with
29:08
folks across various organisations being more
29:12
technical, and I have actually found in in a couple of cases
29:18
that we were not treating our user base with a.
29:24
Like we, we underestimated what they were they would have
29:28
wanted. So I mean it's it's all about,
29:30
you know, figuring out what your users actually want, right?
29:32
That's about it. So you can, I guess.
29:34
My point really is that you can go both ways.
29:35
You can end up building something that's too complex for
29:39
for someone to use for their workflows and then it's useless,
29:42
but also if you build something that is.
29:46
You know doesn't because the trade off is usually it's either
29:49
complex or it doesn't do everything right that that's
29:52
usually where the trade off happens.
29:53
So if you sort of try to make everything like too streamlined
29:57
or more point and click then you end up in the situations where
30:01
it's not a flexible tool. Personally too I would rather
30:04
like learn how you know any any given tool or or library or
30:10
whatever works and like get it to do what I wanted to do then
30:14
be be restricted. So I mean that's that's one of
30:16
my biggest things with like. Tools for instance, and the data
30:19
visualisation tools in particular.
30:21
I would much rather generate a a plot in plotly or something like
30:25
that. Then you know, make a pre made.
30:31
Sankey Chart or something like that.
30:32
So that's. That's because that's how I see
30:37
it. I like, I do like to have the
30:38
power user in mind when thinking about about products or who
30:42
we're building for. But I want to temper that by
30:46
also saying that you know when you're building product they say
30:48
like it's 8020, right? Like make sure that you.
30:52
Build something that most people are going to get value from and
30:56
then you can think about, like how do I bring in more people
31:00
into the discourse? But yeah, so that's that's my
31:03
semi controversial answer. I'll just, I'll just add to
31:07
that, I think you know. I agree with Ashanti was saying
31:11
that the way I the way I see it, or the way I sort of explain it
31:16
is. You know your product, the
31:19
product that you're building, needs to target a specific set
31:23
of audience, right? Specific target user, right.
31:27
You're never going to build a product that targets everybody,
31:29
right, Because you're gonna make everybody unhappy.
31:31
So you need to sort of focus on who is your target audience,
31:35
what do you need to give them to make their experience the best
31:40
and allow them to work the fastest and the most convenient
31:44
way that they know how. So for example, I may build a
31:47
product that's targeted. For data engineers, right?
31:50
Because I want a data engineer to have an easy way to move data
31:54
from a source system to find data quality expectations on
31:58
that data and then deliver it to your target system like
32:00
Snowflake or Redshift or Delay or whatever, right?
32:03
So I want to make that experience as easy as possible,
32:06
so I will build my product to focus on that now.
32:10
I don't want to alienate the other users like I'm a machine
32:13
learning engineer may say, hey, I want to read data from
32:17
postgres this particular table, and I want to store it in a data
32:20
lake in CSV files because I would then want to build my
32:23
tensorflow model on top of that. Great.
32:26
Now do I need to make my user experience or my my product fit
32:32
them? You know, they're not quite my
32:35
target audience. I you know, I can try to do
32:38
that, but what I the way I look at it is focus on the target
32:41
audience for your user experience and then allow your
32:45
product to be extensible, right? Have those APIs, have the SDK,
32:49
have the CLI, have the hooks that users can come in and say,
32:53
hey, I'm an ML engineer, actually I feel very comfortable
32:57
inside of a Jupiter notebook. I don't want to use the, you
33:01
know, your your UI, I just want to plug into it.
33:05
Right. I want to use the engine, but I
33:07
want to use my own, my own interface, and I think that's
33:10
the most powerful way to do things.
33:12
Now that we've done, that's the way we're sort of thinking about
33:14
things. Designing AW too is like you
33:17
have the core design for your target audience, but the API's
33:20
are there, the integrations and there.
33:22
So if you want to plug in something else that's easier for
33:25
you could do it. Because at the end of the day,
33:27
where you're going to make people more sufficient is by
33:30
letting them pick the tools that they know already and just run
33:34
with it. Like having the forcing them
33:36
into a new user. In the face or a new tool and
33:38
having to teach them how to do it.
33:41
OK, yeah, it may work, but it's just going to take a long time,
33:44
right? People aren't going to always be
33:46
happy. I already have all my libraries
33:48
and all my tools in in this other system.
33:50
Like, how do I get them over here?
33:52
Allow me to do it. Allow me to bring my tool and
33:55
plug into your engine. And that's really the big
33:58
benefit of of a data platform, right?
34:00
A platform is not about solving everybody's problems, about
34:04
providing an interface to the core engine that you can plug in
34:08
different systems and make it easy for everybody to be
34:11
effective. Love that.
34:14
Yeah, I think with good products it's almost like little Easter
34:18
eggs that you can find if you're like a power user, right?
34:22
You know, like 1 easy thing that comes to mind is like you might
34:27
have a preconfigured default Kubernetes pod that is going to
34:33
launch for any user. But then you you let people
34:35
configure their own, you know what they what they want in
34:38
their pod as well as as a as a secondary thing like.
34:41
So the person who knows how to do that, great, but also the
34:45
person who doesn't know. Going to get something that
34:47
should be fairly optimised for for simple use cases.
34:50
So changing gears a little bit, Roy, you've mentioned before we
34:54
hit record about how you wanted to discuss what value we are
35:00
solving for when it comes to these users and we're creating
35:04
solutions, but what are we creating these solutions for?
35:08
Can we dive into that a little bit and maybe you can kick off
35:11
that discussion for us? Yeah, I mean it's a it's, it's
35:15
it's a broad statement. So let me maybe try to focus it
35:18
a little bit. I'll tell you in particular kind
35:22
of like what what we're trying to do at Absolver.
35:26
We originally build a data pipeline solution, right?
35:29
So users could write SQL, it builds a data pipeline for them
35:33
to move data from point A to point B.
35:35
They can do transformations in the middle, et cetera, et
35:37
cetera. Sort of the a lot of the, the
35:41
focus in the industry has sort of moved on, not moved on, but
35:46
sort of shifted towards identifying and improving the
35:50
quality of data. So like we helped users build
35:55
the pipeline right, the the, the infrastructure to to deliver
35:59
data, but we didn't really pay attention to the data inside of
36:03
it like that was sort of up to you right And as we start having
36:07
more and more of those conversations with with
36:09
customers. And then potential customers,
36:12
we've heard the, we've heard the challenge over and over again
36:15
like yeah, OK, you can move data into the warehouse, yes, you can
36:19
get in there, but it's kind of garbage in, garbage out, like
36:21
how do we then address it now we can run tests and we can do all
36:25
these things but. And they're a bit reactive.
36:29
They're after the fact. If I run the test, the data is
36:32
already in a warehouse, people already depending on it, the
36:35
model is already trying to run on it, and it's already
36:37
breaking. So what do we do now?
36:39
The test failed. OK, fantastic.
36:41
Now I got to go find where the problem is and fix it.
36:44
But you already loaded the data in, right?
36:46
So. We sort of took a step back and
36:49
we said, OK, you know, we can just be a dumb pipe and load
36:52
stuff in or we can double click on these problems and find
36:56
solutions for our users that will actually help them solve
37:01
the real problem. Like how do we change the
37:03
outcome, right? Just loading data in a pipeline
37:06
does not change the outcome. Loading reliable data or loading
37:10
high quality data in a reliable way changes the outcome because
37:14
now you can start building trust.
37:16
Right on that data where before it wasn't reliable, like if
37:20
something showed up, great. If some rows were missing, I
37:24
don't really know that, right, until, you know, I run my report
37:27
on my dashboard. All that's on my bar graph is a
37:29
little smaller than it used to be.
37:30
Like what happened, right. I can't trust it.
37:34
And we see this in in, in our users where they're like, you
37:37
know what, I don't really trust this.
37:39
I'm just going to unload the data from the warehouse into
37:41
Excel and I'm going to do it on my own, right, Because I can
37:45
trust myself. So that's sort of what.
37:47
And talking about where? We say there's a there's a data
37:51
problem, right? There's data quality problem and
37:54
then quickly unpacking what that actually means and finding those
37:58
aer cuts that allow you to actually solve the problem.
38:02
So like one of the biggest paper cuts that we hear, two of them
38:05
are how do I know when my schema changes?
38:08
And the second one is how do I know how fresh my data is?
38:12
Sounds fairly simple, right? But these these problems always
38:16
come up in every conversation that we're having is the first
38:20
thing is. How do I know when somebody adds
38:22
a column or drops a column or changes a column?
38:24
And the second one is, how do I know when was the last time my
38:27
table was updated? Like this is super, super
38:30
simple, right? But it seems to be a problem.
38:33
So if these are really painful paper cuts, like why don't we go
38:38
solve them? So that's those are some of the
38:40
problems that we then, you know, Shantanu and myself and
38:44
engineering team kind of dove into and say, OK, let let's take
38:48
the big problem of data quality, unpack it and then.
38:51
Find those little things, those paper cuts, that we can go soft.
38:55
So good. So yeah, just thinking about the
38:58
bigger picture and thinking about the user experience from a
39:01
holistic point of view as opposed to just this is my space
39:05
that I've carved out and I'm going to just focus on this
39:08
space, right, thinking about what is the end result.
39:13
And in I think in marketing they have this term jobs to be done.
39:17
Have you heard of that one before?
39:18
Yeah, yeah. It's kind of like that.
39:20
Hey, I just want to, I just want the end result.
39:22
On the picture, I I've always heard it said as I want a
39:26
picture on my wall, I don't want a hammer and a nail and a
39:30
handyman to put that picture on the wall, right?
39:33
I just want the picture on the wall.
39:34
Let's get to that. How can I get to that the most
39:36
efficiently, whether it's with a drill or a hammer or a handyman
39:40
coming and helping me? That's right.
39:42
That's right. And and then so sorry to just
39:44
double back on what you just said, like I think that's that's
39:46
a perfect example of what we see on LinkedIn these days.
39:50
And I love LinkedIn. I love all the people who post.
39:53
Up there. But that's an award.
39:55
You see, you see these architectural diagrams.
39:58
You see these, these these long posts about, you know, the
40:01
handyman and the hammers and the nails and the wall and the, you
40:05
know, the they'd measure all that kind of stuff like.
40:09
Yeah, technically it's fantastic.
40:10
But I I just wanna. I just wanna know when was the
40:14
last time my table was updated? Like just just, yeah, make that
40:17
easy for me, right. Give me that solution and please
40:19
don't make me buy a whole new system just to tell me that be
40:21
the bit of information and that that's another thing that kind
40:24
of judge me a little crazy in our industry these days is like
40:27
if there's a problem, there's a product for it.
40:30
Like I don't need 15 different products to solve these little
40:33
paper cuts, right? You know how do I, how do I
40:37
consolidate the whole idea of us And I I I wrote something like
40:40
that. On LinkedIn, I think recently,
40:42
you know, as a vendor, my job is to help solve problems and make
40:47
users life easier. But it's also my job to help you
40:51
simplify your your end to end architecture, right?
40:55
The more products I'm adding like it's good for me, but it's
40:58
not good for you, right? Because now you have 15
41:01
different products, you got to manage 15 different vendors you
41:03
got to deal with and pay money to support and blah blah blah
41:06
and all those things like just make it simpler if you can
41:08
simplify it, let's simplify it. Yeah, the tooling sprawl is
41:12
real. And it's not only real in data
41:15
engineering. It's the same with machine
41:17
learning engineering. It's everywhere.
41:19
And a lot of times I think with machine learning engineering and
41:22
just mill OPS in general, what you get is there's different
41:26
tools that have. A certain value prop, and that's
41:31
what they message as their main value prop.
41:33
But they also do five other things.
41:35
And then there's another tool that has a main value prop, but
41:38
it does those same five other things.
41:40
And so if you want those two main value props, you're going
41:45
to have to decide what are you going to do those five other
41:47
things, Which tool are you going to use for those five other?
41:50
Things right. And they might not be good at
41:52
those other things, they might just say they do it or they
41:55
have, you know, basic capability of it, but they're not, may not
41:57
be good for that. It may not be the right tool for
41:59
that. Yeah.
42:00
So you gotta get another tool for those five, five other tools
42:04
for those other things. So that's that's definitely has
42:06
a bit of a pain and also navigating the space to know
42:10
which tools do what. And I know at least for me it
42:16
was even worse I think a year ago where and now I think it
42:20
probably moved more when it comes to the large language
42:24
model sphere. It is very much like this now
42:28
where you're trying to navigate through these websites and you
42:31
can't really tell what the different tools do because it
42:35
all sounds the same and you just sprinkle a little bit of
42:39
marketing jargon on it and then talk about I my favourite thing
42:44
used to be to. Find websites that would quote
42:48
the 80% of machine learning models, never make it into
42:52
production, and it's like whoever made-up that statistic
42:56
has got to be so happy because they're the most quoted by every
43:00
tool out there. And so these days I don't think
43:03
I see it as much because all the machine learning models are
43:08
large and they're all in production now and it's just how
43:12
reliable they are. That is another story.
43:14
I I think. I think it's also really
43:15
important to. You know, for all of us to to.
43:20
Clearly distinguish and differentiate between.
43:23
This is fun and I want to go play with it and I want to check
43:25
it out to this is the day job. And this is to be reliable.
43:30
This is to be accurate. This needs to be, you know,
43:33
stable, playing with a lot of cool stuff and then quickly
43:36
trying to bring it to your day job.
43:37
Is that let's just store it in here and add it.
43:39
Like, no, Like, this is a business you're running a
43:42
business. Like you need to be.
43:44
Like you can just bring toys into your into your day job,
43:47
right in your free time. Go for it.
43:49
Go nuts, right. Have fun, learn all this stuff,
43:52
but draw a clear distinction clear line between when do you
43:56
shift stuff over. And I think that's where.
43:59
Some of the maturity around these models and and these tools
44:03
really need to come in and like I can play with them.
44:06
But until these things are mature, these things have a lot
44:09
of best practises built around them.
44:11
There's some good case studies of companies using them at scale
44:15
and going through all the pains and tribulations of using those
44:18
tools. Don't bring them into your into
44:21
your day job unless you have a big team that you're comfortable
44:25
testing this and you're comfortable doing this.
44:28
But most companies don't have that like most companies.
44:30
Have an army of data engineers who can go play with all these
44:33
new tools, and if Flink breaks for some reason, they can go
44:37
write some code, contribute it back to the open source, and fix
44:40
the problem. Now many people have that right,
44:43
yeah. So you got you gotta be careful
44:45
when you bring you know these tools into your into your day
44:47
job. Yeah, you gotta look at it more
44:49
like a Skunk Works programme, yeah, And recognising this is
44:52
our D this is not core to our business.
44:58
So. I wanted to ask a question here
45:03
because. We mentioned also before hitting
45:07
on or before we started recording about things that you
45:11
would have loved to have learned five years ago and you five
45:16
years ago talking to yourself. What are some things that you
45:20
could tell that past self that would help accelerate their
45:25
growth and. Their knowledge in this field.
45:30
I think I would have. So five years ago I was
45:33
finishing up my PhD or getting close to finishing up my PhD and
45:40
I I think at the time hadn't fully committed to the idea of
45:47
doing data science outside of academia.
45:50
I was considering different options.
45:53
So you know it's going to be very different answer from from
45:56
Roy's answer, but. I would have liked to know more
46:04
about what options were out there with a analytical problem
46:10
solving mindset. I I mean, I think unfortunately
46:13
ended up in an area that's a very good fit for me and I
46:17
really enjoyed doing the work. But yeah, that that's what would
46:20
have been relevant to me. So I I feel like that's, that's
46:23
what I'm speaking to right now, right, Grad students who are
46:26
about to figure out what's next. So that's a part of that that's
46:29
relevant. And then also I mean I I learned
46:34
this with time and kind of proved it to myself.
46:38
But it also would have been nice if someone said, hey, a lot of
46:41
the work that you've done is super transferable to what
46:46
you're going to be doing five years from now.
46:49
You know you're just like statistics courses that you've
46:52
had to take and you know figuring out confidence
46:55
intervals and stuff like that. It's it's all it's all useful
46:58
impactful and and people are going to you know respect your.
47:01
Data analysis, because you have that background.
47:04
How about you? Really, five years ago?
47:06
What would you have liked to have known?
47:09
Or what are some things that you feel like would have helped you
47:11
accelerate your learnings? Or even three years ago, Two
47:16
years ago. It doesn't necessarily need to
47:17
be because five years ago I think.
47:20
Maybe it's too far. Back Well, yeah.
47:23
I think it would have helped me. Three or so years ago is.
47:30
Sort of understanding. Understanding better and how to
47:35
navigate. Umm, sort of the strong opinions
47:40
that people have about? Solutions Technology.
47:46
Approaches things like that. You know I feel like in in in
47:53
the technology space, especially in data engineering and ML like
47:57
there's there's it's almost like religion, like different people
48:00
have different religions within this this industry like the ETL
48:03
versus ELT, like it's such a big religion like you either one or
48:06
the other. Data modelling There's people
48:10
who live and die by data modelling there's others say is
48:12
that this is, this is. Pointless, right?
48:16
Not saying one is better than the other.
48:17
All I'm saying is that I think. For me it would have been kind
48:21
of better to to be prepared that there is this.
48:26
You know, this, this, this preference and this sort of like
48:28
religion between them because in my mind, it's technology, right?
48:33
I mean, it's multiple ways to solve the same problem.
48:36
You know, choose which one. If you want to do something a
48:39
certain way, that's fine, that's your choice, right?
48:41
But people are very sort of like of a mindset like this is the
48:46
only way to solve the problem is the best win is a millionaire
48:49
and you can argue all those points till you're blue in the
48:52
face, right? I mean, you're always going to
48:54
make the same argument. This one is scalable.
48:55
No, this one scalable. This was performing.
48:57
No, this was performing right. So just kind of knowing how to
49:01
navigate that a little bit smoother, You know, it doesn't
49:06
trigger a lot of these emotional emotions from different people
49:11
that I think that would that makes things a little bit
49:14
easier. It's very easy to spark those
49:16
conversations and to get like these exchanging views and and
49:19
get people walled up and and crazy about that stuff.
49:22
I don't know how productive that is to be honest.
49:26
So maybe if I had to summarise, it's kind of knowing how to
49:28
navigate personal opinions and preferences around technologies
49:33
and and ways of doing things. Kind of would have helped me a
49:37
little bit better to kind of connect with with more people
49:40
and maybe to be more effective in in the way I communicate.
49:45
Well, thank you both for coming on here.
49:48
I think we're going to end it there.
49:49
That was a perfect finishing point and.
49:53
Both of you. For anyone that wants to follow
49:56
you all, we're going to leave all of the links in the
49:58
description. Anyone who wants to continue the
50:00
conversation can go and check that out and reach out to you
50:03
all. Awesome.
50:14
No, I will not recommend your podcast as the lead engineer TP
50:19
Bank. I'm definitely not wanting other
50:22
engineers to have all this relevant information and I will
50:25
not give my name. Which is Lena?
50:28
So definitely I do not recommend this podcast.