Latest / Agentic Conversations (formally mlops.community) / Fixing Your ML Data Blind Spots // Yash Sheth // MLOps Coffee Sessions #102

Transcript

Fixing Your ML Data Blind Spots // Yash Sheth // MLOps Coffee Sessions #102 · Agentic Conversations (formally mlops.community)

0:00
Hi, I'm guy said, and I like a double-shot cappuccino with
0:03
low-fat milk. It's one of those mornings.
0:07
And I mean, normally, I'm in your time zone.
0:10
But today, I am not, as you can see, by the clothes, hanging up
0:15
beside me, actually do diamond. So, I'm in New York right now
0:19
and I'm in a hotel room, that is no bigger than a shoebox.
0:24
And the thing is so expensive for the like, Five nights that
0:30
I'm here. I don't even want to talk about
0:33
how much I'm paying you walk into the room and you're already
0:37
in the bathroom, which you have to Traverse through the bathroom
0:40
and then you just get to the bed, which is like in the
0:43
windowsill. So it's fun in New York but it's
0:46
good location and I get to enjoy the hot Miss of New York this
0:52
week and also biking around. I've been biking like crazy but
0:55
anyways today he even helps me upstairs.
0:58
So we went, yeah, on Wednesday. There was one and it was
1:01
incredible. I got to meet a bunch of people
1:03
that I normally just talk to you through the screen.
1:06
Super cool. To see how the ml Ops Community
1:10
is starting to meet in person again and we're always looking
1:14
in case you want to start organizing something around your
1:19
city or town. Let me know because we're making
1:22
it happen. We've got like a Playbook that
1:25
you can follow and we can just run with it.
1:28
And so Anyway, London one would be to the London.
1:32
One was great Amsterdam. One is awesome.
1:34
Berlin were, we've got some stuff happening.
1:38
Especially, I'm looking for people in Denver.
1:40
I was just talking to a guy in Denver and he was saying that
1:42
there's a lot of talent in Denver and a lot of like demand
1:46
and desire for these things to happen.
1:48
So anyway, today we talked with Yash Chef or she's I can't
1:54
pronounce names as you all know, probably by.
1:56
Now, if you've heard me introduce anyone over the last
2:00
Years. And he is the co-founder and VP
2:04
of engineering at Galileo and Galileo happened to sponsor this
2:09
episode of the community podcast.
2:13
So if you are on the market for an unstructured data helper,
2:19
this is the tool for you. They've got a really cool blog
2:25
post that we will link to also if you want to dig a little bit
2:28
deeper into what they're doing. We posted that on the envelopes
2:31
Community blog. Yash is an OG man, like he's
2:35
been in the game. Especially the unstructured data
2:38
game for over 10 years. He spent the last 10 years at
2:41
Google where I was doing all kinds of stuff around automatic
2:44
speech recognition he was on the and leading, the core speech
2:50
recognition platform team. So the guy who knows his stuff.
2:53
What were your takeaways? Adam?
2:55
Yeah, obviously. Deep sort of knowledge and
2:58
talent. There was really cool to speak
2:59
to For me, actually, there is the funny bit around.
3:04
Someone mentioned recently that you might need calculus to do
3:07
machine learning and deep learning.
3:09
And yes she said was really interesting and he talked about
3:11
how back in the day you would need to really understand how
3:15
the compiler worked for to go to write software, right?
3:18
And that's kind of gone away like I don't write.
3:21
So he point you to maybe sign posted a little bit that that
3:25
may be the way deep learning is going to go and these tools will
3:28
become more abstract and more general purpose.
3:31
So that for me was really quite cool to see how maybe some More
3:34
experienced Bryce or teams in the world are starting to think
3:38
about this stuff as a bit more off the shelf, configure
3:41
overcome that kind of stuff. I also liked his heart, take on
3:45
synthetic data and how it just what he's seen as it causes more
3:51
harm than good if you're not careful.
3:53
And so I and you also had experiences with that to write
3:57
back in the day though. How long ago was seven?
4:00
Yeah, white back way back. I mean there's probably been
4:02
some updates since then but But you still gotta be careful if
4:07
you're using that synthetic data.
4:08
If you are using it, reach out to us, let us know how it's been
4:11
going. And and if you love it, I
4:13
definitely want to hear from you because synthetic data.
4:16
As you mentioned before to it is a bit of a buzz word type thing
4:20
it's getting more popular. Yeah.
4:23
So alright, let's get into this podcast with Yash and definitely
4:27
check out Galileo because these guys are doing some awesome
4:31
stuff. That's it.
4:35
First question, obvious question.
4:38
I mean, you have an incredible background.
4:42
What got you started down this path, what made you go into, not
4:48
only machine learning, but then why into the unstructured data
4:52
field and can you give us the decisions that were made along
4:56
the way? So yeah, I've been working on
4:59
speech recognition at Google for the past nine years and You
5:04
know, along the way what we've learned is that in order to
5:10
build world-class models that can you know, that that power,
5:13
you know, 20 plus products at Google across 80 languages, one
5:18
of the key things that the team got it right.
5:20
Was the ability to effectively understand the data sets and
5:24
proactively fix gaps in the model before launching anything
5:29
in production. So and and along the way.
5:34
Well, you know, we also developed a key rigor to sort of
5:40
understand data better and develop the tools for it.
5:44
So, you know, that's that's my background and where I am, you
5:50
know, where I've been working on and what got me into machine
5:53
learning, is sort of a step further out in.
5:57
I was in grad school, and machine learning, you know, was
6:01
was still very nascent, as you know, data.
6:04
Gets worse over scares in Academia and, you know, we did
6:08
machine learning in Matlab in our and, you know, the
6:12
sophisticated tools that are there today weren't available
6:15
back then, but it still was a very interesting field.
6:18
And you know, I was always fascinated about how math can
6:23
be, you know, purposefully position in into algorithms to,
6:28
to understand features, from from data.
6:32
So and and Fast-forwarding to now, I think.
6:36
But what got us excited about about Galileo, and about the
6:43
need for data Centric techniques.
6:46
In an unstructured data, especially is the need.
6:51
I mean, the, the need for the, the required tools for the data
6:55
scientists to be able to effectively understand data
6:58
today. So what was the question?
7:03
Where does it fit in? Unlike the workflow then and
7:06
when like would eyes are saying fledgling data science team,
7:12
start thinking actually wins it going to be the best bang for
7:14
buck for me and stuff like that. Yeah, so I think before before
7:19
we dive into that I think I just want to highlight the the
7:24
different challenges in the unstructured data workflow and
7:27
maybe that might you know lead us into sort of understanding
7:32
when we need to apply these techniques.
7:34
Inaudible flow. So I mean as we know that over
7:39
90% of the world's information is unstructured today and it's
7:42
growing incredibly fast at you know there are estimates that is
7:46
growing at 55 to 65 percent every year.
7:49
And with the with the proliferation of deep learning
7:52
techniques more and more businesses are starting to tap
7:55
into this data to build modern day.
7:58
I powered applications. Right?
7:59
So I think a lot of the, the key Challenges today are coming
8:07
from, not only the fact that there are sophisticated
8:11
algorithms that work, well, on a variety of modeling tasks, on
8:17
unstructured data and they generalize well across domains.
8:20
But with unstructured data itself, the challenge is that
8:26
come in are firstly, understanding that data and
8:29
creating your data set? That's, you know, that's not
8:33
only You know, cumbersome, but it's not just about sampling
8:38
data equally across the labels that you want to identify.
8:41
Because there are a lot of nuances within unstructured
8:44
data. Like, you know, different
8:46
sources. The data are coming from
8:49
different modalities of the data may be, you know, there are for
8:54
example, within speech in my experience, there are tons of
8:57
speakers with tons of accents. So it's not just about what is
9:00
being spoken. But how and in what environment
9:04
And and and so, once you understand this, you can curate
9:08
your data set better. But when you actually train your
9:11
model, it's very important to understand where the model is
9:15
making mistakes. So and in order to really
9:20
understand where the model is making mistakes today, folks are
9:25
using ad hoc scripts and looking at data dumps, you know,
9:30
individually and trying to understand.
9:32
Oh, why is this sample bad for my model?
9:35
And and so, you know, in the past six months at Galileo,
9:38
we've spoken to over 100 ml teams working with unstructured
9:41
data. And this is where they spend the
9:45
most amount of time. If that's in the post training
9:48
phase where, you know, they spent and this is sort of
9:53
anecdotal evidence. And the overwhelming statistic
9:56
is that they spend more than 50% of their time today, just trying
10:00
to understand issues in their data sets.
10:03
Lastly, once you once you manage to launch your model and
10:06
production, since unstructured data is tied to free from
10:10
content created at an astonishing Pace in the world,
10:14
things are changing really fast that the model is seen, and we
10:18
need to constantly keep on proactively keep an eye on out
10:23
on what is changing and how can we make our data sets more
10:28
relevant and and keep the model performance high in.
10:32
Action. So, now that we understand the,
10:36
the entire ml lifecycle for unstructured data, I think the
10:40
key challenges there are in the pre-training phase for creating
10:44
a data set in the post training, phase for fixing your your data
10:49
set to overcome or blind spots. And the last part is in the
10:53
post-production phase to keep your models relevant.
10:57
So I am really interested in digging into these different
11:01
life cycle and and like you kind of gave a high level of each,
11:06
one of these, can we go a little deeper down the rabbit hole on
11:11
this and and specifically like I'm thinking that, you know,
11:18
when you're looking at things like unstructured data, I know
11:24
people have talked about how testing can be a burden and how
11:29
how that happens to and so Wondering.
11:35
As you're looking at this life cycle and you're seeing all
11:38
these pains, let's like, dig into the different pains a
11:42
little bit more. And and then hopefully talk a
11:46
little bit about testing. Yeah, for sure.
11:49
So, I guess I'll just take it in the same sort of ml workflow
11:56
cycle that I described earlier. And and so initially when you're
12:00
creating a data set, as I said, like you we will notice that In
12:04
the unstructured data, realm effective data scientists
12:09
generally have a very strong intuition about the domain and
12:13
the data, and it's more. So for unstructured data, the
12:18
reason is that you need to really understand the nuances
12:22
about this, the data sources and, and the different, you
12:25
know, as I explained different sources of data are different,
12:29
maybe it's long form data and short form data that is as one
12:33
of the examples. Different characteristics of the
12:38
data. Maybe there is noise around the
12:40
data and you want data with different kinds of noise in it,
12:45
to make your model robust. And so there are tons of nuances
12:50
that come in with the data itself at the post at the
12:55
pre-training phase when you're curating a data set and it takes
13:00
a few alterations and as well as you know, a lot of Different Q.
13:06
I would say creative thinking around, you know, getting your
13:10
data data set to a point where it produces effective results
13:14
from the model. But now next on the, on the post
13:19
training phase as we, you know, as we all know, that is of deep
13:24
learning can can almost be like a black box and more and more.
13:30
You know, nowadays that deep learning techniques are, you
13:32
know, packaged in libraries They are being treated as Black Box,
13:37
black boxes versus you know I guess you know a decade ago
13:42
folks who have had, you know, phds in data science.
13:46
They have had a lot of experience.
13:47
They were writing their own neural networks and entertaining
13:51
them that there was a magnitude of like, like extra
13:56
understanding of the math that's going under the hood.
14:00
Whereas, I mean, I'm not saying that it's required now.
14:03
In fact, You know, that have been, you know, libraries and
14:08
even like even course like deep learning courses like, you know,
14:13
and Rings data-centric a II course where you know deep
14:18
learning has been made in like made very, very intuitive that
14:21
someone with is sort of just like like basic algebra.
14:24
Knowledge can can build deep learning models today but what
14:31
happens when a model Miss predicts on Something and I
14:37
think, you know, earlier earlier and even more so you know, a
14:43
couple of years before data scientist would focus on the
14:47
model to fix their the the issues in there in the post
14:51
training phase and so. And that was that was that was
14:56
valid because the model architectures and the algorithms
14:59
then were not generalizable enough.
15:02
But today as I noted earlier, We have you know, algorithms that
15:06
can be applied to a wide variety of tasks effectively.
15:11
So there's less of a focus on improving the model and you can
15:16
pretty much fix, you know a large amount of issues in your
15:21
model like bugs in your model by fixing your data set.
15:26
So there is there's an even more importance on actually fixing
15:32
biases in your data unknown. Like trying to look for unknown
15:37
unknowns that, you know, that that is causing your model to
15:41
miss predict. Because, you know, if you treat
15:44
the model as a black box, it's like garbage in garbage out
15:47
there is, you know, the the, the sort of the input that is what
15:52
you need to fix first. So that's the sort of that
15:54
supposed to any phase and the going into specifics.
15:58
Some of the errors that we see in unstructured data, especially
16:04
is, I mean Firstly, sampling errors.
16:06
As I, as I mentioned, you're like, firstly, you're missing
16:09
out on a few nuances in the data set and you need to include more
16:14
of that or augment your data with noise with, you know,
16:17
different form like different forms of your input.
16:21
Secondly, unstructured data is heavily dependent on the human
16:26
annotations. Even if it's, you know, even if
16:28
they are, you know, partially semi-supervised.
16:31
There is always a human component.
16:33
Somewhere are associated. Your data set, and that alone,
16:37
you know, comes with its own. I would say error potential
16:43
because there are, you know, The annotation need to be done at
16:47
scale and you know, of course, you know, there are human errors
16:51
introduced there and and thirdly, there's sort of data
16:54
freshness and relevance. And that's the piece I talked
16:57
about aware, you know, you need to firstly.
17:00
Make sure you test set is relevant and And then make sure
17:05
that your model is performing very well on that.
17:07
If it's not then you need to again, go back to your training
17:10
set and make sure it has the the necessary data data data samples
17:15
to avoid confusion. So that's the supposed to any
17:19
phase and this is where most people like most data.
17:22
Scientists spend most of their time.
17:25
And and lastly I think I'll be I'll be short here because we
17:28
can go deeper if we'd like, but I think everyone nowadays is
17:32
talking about the concept of drift and more.
17:34
So The structured data side because, you know, you're
17:37
looking at, you know, District distribution differences in your
17:42
feature, set in your training, in your training data.
17:45
But I mean, there's also a very real.
17:50
I would say concept of like semantic drift in your
17:54
unstructured data and that's and that's even harder to predict
17:58
these days. So it's very, very important to
18:01
keep your datasets refreshed. In production and you can only
18:06
do that if effectively at scale because these data sets are
18:09
huge. You cannot just resample the
18:11
whole world every month and relabel them, it's costly.
18:15
And so you need to have an effective sort of almost
18:20
workflow or an automation built around it to be able to scale.
18:25
So one thing that comes up when I'm hearing what you're saying
18:30
is really, it feels like you're going for And what you're
18:36
talking about like as the landscape has changed it.
18:43
There's this data scientist or machine learning engineer, that
18:47
may or may not be the only one at their company that is now
18:51
tasked with putting into production these models and they
18:56
are really the sole responsibility and it I bring it
19:03
up. It's probably recency bias.
19:04
Just because I was just talking to someone earlier about how
19:08
they are the machine learning engineer.
19:10
They need to do that and they also need to train the models,
19:13
and they need to do the data scientists part of the work too.
19:18
And I was thinking that it sounds a little bit like this.
19:22
And I hate using this this term because it's very cliche, but
19:26
it's like the democratization of machine learning.
19:31
And so I just threw that one. Out there.
19:35
I know that it is not the best term because we it's been
19:39
overused but I'm wondering how like what's your vision around
19:43
this? Do you feel like those teams and
19:47
what you're trying to build is trying to speak more to those
19:51
teams to help them, especially if they're a little bit smaller
19:55
and then they can get things done much faster.
20:00
That's a great question Demetrius.
20:02
And I think I can, I can speak to the differences between small
20:08
teams and large teams because like I was at Google on a large
20:12
team, whereas the customers that we're talking to, many of them
20:16
are these small teams that are almost like a SWAT team of like
20:21
two or three people, you know, do I carrying the burden of of a
20:27
ion in the within the company? And that's Um, I think that the
20:34
need for rigorous data analysis and understanding is is
20:40
universal, right? The challenge is that small
20:44
teams have is firstly, an insert is sometimes lack of experience
20:51
where you know it's hard to find folks.
20:54
Again, I said, like, since we need domain, expertise on
20:57
unstructured data, it's hard to find and Hire folks who have
21:02
years and years of experience in this on in this domain, this
21:06
data for training such models because all these also the
21:11
democratization is relatively recent so.
21:17
And and so that's that's the challenge that small teams face
21:21
and it's since they're small. They're also just working on one
21:24
model, like one task at a time. And then it's almost like a, we
21:30
On one model for a quarter or two.
21:33
And then once which launched, we go, to the next model, another
21:37
challenge that small teams faces.
21:40
They often tend to, you know, just launched and forget about
21:45
the model that they've launched in production.
21:47
And and by forget, I mean like they might revisit it but it's
21:51
in a year's time because they have limited bandwidth to really
21:55
maintain that model. And so both of these things
22:00
combined, And I think, you know there's there's a need for data
22:04
specific, tooling that can almost be there.
22:07
The data scientists assistant to helping them quickly, understand
22:12
issues in their models. They don't have to run days and
22:15
weeks worth of experiments to get these insights.
22:20
And often times, it can be proactively given by the tool
22:23
instead of them jumping and shifting Focus.
22:26
So if they've already started working on, let's say another
22:29
model. And you know, this, there are
22:32
tools that give them insights hate like your model is, is
22:36
getting weaker on this kind of data.
22:40
That's a proactive nudge to to Reno.
22:43
Just ask them to really act on it and then even like these.
22:47
These tools need to also be actionable.
22:50
And so that's the that's the key part here.
22:53
Where sometimes, you know with the again the The Limited
22:57
expertise or The Limited domain But he's, there isn't an
23:03
actionable tasks for the data scientist to do quickly in order
23:09
to fix it. Like, they'll have to read a
23:11
bunch of papers and, and explore the domain better to see.
23:15
How can I act upon this and that the tools should almost give
23:20
them insights, that are quickly actionable to fix the model, the
23:23
Gap. In the data set on, on the flip
23:28
side with large teeth, Seems I think there are a bunch of
23:34
people building a bunch of models but the the end there is
23:40
a bunch of experimentation as well and but it's very ad hoc
23:46
every everyone runs their own ad hoc Python scripts or SQL.
23:51
Scripts on the data looks at data dumps on their own and then
23:56
comes up with ad hoc metrics and insights that, they only they
24:00
found. Out.
24:01
And that causes a sort of like a uniform.
24:06
Look at a uniformity problem. We're not.
24:09
Everyone is doing the same set of analyses and how can we relax
24:13
like this? No code review on those scripts
24:16
happening. And so that's that's one of the
24:19
things there as well as the the post-production step.
24:25
We're like we we should be more actively tracking models in
24:29
production, That's valid even with large teams.
24:33
So do you feel like what you were talking about there at with
24:38
the small teams, is a specific unstructured data problem,
24:44
because I feel like there maybe it sounds a little bit like it
24:49
could be applied across the board.
24:53
Yeah, good question. I mean, so we have the other
24:56
end, you're right. Structured, like structured
25:00
data. Data in specially deep learning
25:02
applications with structured data also can benefit from these
25:06
tools. The only reason why we're
25:09
tackling unstructured data first is because you can like the data
25:14
scientists can quickly come up with some basic distribution
25:18
level insights on structured data.
25:21
Whereas on the unstructured data is still very ad hoc, we don't
25:25
know what metrics to track. We don't know how to understand
25:29
the data in its Domain-specific. So there needs to be a
25:33
standardization of tools there and they can also apply to
25:39
structure data. The whole part being like
25:42
proactive and being an assistant to the data scientist that still
25:46
is necessary but the dire need is on the unstructured data
25:51
side. Wow.
25:53
The interesting point around like the tools almost need to be
25:55
unified across because I was always going to ask you like
25:58
always feel the Structured data as a bucket is very vague and
26:04
very Broad and like actually like audio data is so different.
26:09
This has Galileo handle that like his.
26:12
Yeah. And it's funny you bring that up
26:16
because I, you know, in my experience I've also seen like
26:19
the structured data and unstructured data world of ml,
26:24
as almost being like a left brain, right brain problem where
26:27
they do. They both don't talk to each
26:29
other. And there's a There's a, there's
26:31
a deep divide between the two exactly.
26:33
And once creative and the other one is analytical.
26:37
I like that. Yeah.
26:39
But I think the unifying factor is the model architectures now.
26:43
I mean, we share those, you know, on the structure data
26:46
side, there are, you know, extra boost models that work very well
26:50
on structure data. But more, and more teams are now
26:54
trying to create embedding some structure data.
26:57
And, you know, of course, train deep learning Deals with
27:00
structured data. And this is what.
27:05
And as this this sort of trend grows on in the structure data
27:10
World, there is an opportunity to unify the tools because when
27:15
the model becomes the unifying Factor the tools that apply to
27:20
that those models can be shared and that leaves me work on
27:27
around in a circle to where we talked a little bit about the
27:29
democracy. Democratization of data science.
27:32
And I just someone recently posted something saying, you
27:36
just don't need to know calculus to be a data scientist like
27:40
LinkedIn thing and like, that's an interesting point actually
27:45
because what are you doing? Do you need to know calculus to
27:49
be able to use the tools, apply them and get value.
27:52
And what I just wonder where you sit on that kind of spectrum,
27:55
what's your opinion there? That's yeah.
28:00
And I'm and I think firmly on the, on the opinion that you do
28:04
not need to know calculus to to, you know, do deep learning.
28:09
I mean there are their tasks within deep learning and even
28:14
like reinforcement learning, there are still nascent.
28:16
And if you're working in those fields, then I'll say like you,
28:20
you definitely would need to understand the math in within
28:24
the model, but you almost sort of can like and you have to
28:29
there's a distinction I think between like an AI researcher on
28:34
the modeling side, whereas an applied ml engineer and if
28:38
you're applying a if you're if you're building AI applications,
28:43
you do not need to know the math that goes underneath the within
28:48
the model, but we can use the gate, keep everyone out.
28:50
Like it's not. I mean, I think I have the
28:56
analogy here about software and this keeps coming in my mind.
29:00
Mind, very often nowadays because, you know, ml models
29:04
are. I mean, in many cases, replacing
29:07
heuristics within that were built within software and as we,
29:12
I mean, you can think of like, in the early days of, you know,
29:14
like Fortran. And we're like, you had to
29:17
really understand how the compiler worked in order to code
29:22
nowadays. Any, you like folks don't need
29:24
to know how the compiler works for, in order to build software,
29:28
and I think it's the same analogy here with more. where
29:32
the thing of the compiler as like the the model that's built
29:36
underneath that strained underneath with using the the
29:39
recipe that you're training with and then if you don't, even if
29:44
you don't understand that, you can still apply that tool to
29:48
your application, That's a really cool that he is the queen
29:54
of a mobster torque and this is your daily dose of analogs.
29:58
Well, if you are dead serious about the mo UPS, you have to
30:03
immerse yourself in the amount of content.
30:06
The best way to do it is to subscribe to the email Ops
30:10
Community podcast. All great people are here on
30:13
this podcast, so good luck and keep learning.
30:18
Yeah, that is very nice. To jump in and talk a little bit
30:22
about synthetic data. And the role that you see it
30:26
playing in the unstructured data world and how it works with
30:31
Galileo and what that looks like.
30:34
Yeah, that's synthetic data is then and the need for it is is
30:40
growing is I mean I would say like you know, fairly fairly
30:45
rapidly but I think that it comes with its own challenges.
30:50
What we've seen, you know, in in our experience, be no dealing
30:54
with large datasets synthetic data, can often the come to bite
30:58
you on the, on your foot because it really Ties to the heuristics
31:05
that were used to generate the synthetic data.
31:08
Like, you know, we've we've been General like, you know, we in my
31:13
experience, let's say we've generated synthetic speech where
31:17
we use Text-to-Speech models to generate the audio itself but
31:22
then you're trying to almost like your biasing the model
31:25
towards the algorithm that you wrote to generate that speech in
31:29
the first place, also on the on like, you know, with my My
31:35
co-founder Vikram was on Google Cloud working as a product lead
31:39
there. They were doing Vision models
31:42
for n and they were doing like document parsing models and if
31:45
you were to generate like fake W-2s and fake driver's licenses
31:50
they you know there's just nuances there that were in fact
31:54
bringing the model performance down because you know there's
31:58
obviously some this obviously some And in the synthetic data
32:05
that's just you know forcing the model to learn the wrong things.
32:11
So and you can also go wrong in other ways where you do too much
32:14
synthetic data, it needs to be a good mix of synthetic data with
32:18
real data in order for your model to really benefit from it.
32:22
Oh yes, a really interesting point because it's quite, like,
32:25
trendy at the moment is quite Hot Topic synthetic later.
32:28
Yeah, Ally. And I've never had a good
32:30
experience with you. I tried to write a
32:32
semi-supervised learner like about Six, seven years ago and
32:35
it was a disaster to the eyes of effort.
32:37
Didn't really work. So and I just wanted to act like
32:43
Dimitri. Second question on, how does
32:45
Galileo deal with it? And I think, you know, with some
32:49
of our customers using synthetic data gallery has been like an
32:54
essential tool in their workflow because they are almost like
32:58
they almost get quality checks from in synthetic data for free.
33:03
Where Gala? You can just point out hey this
33:06
is the cohort of samples that are pulling your model
33:08
performance down, you can remove them or fix them, within within
33:13
Galileo itself and they're like, if your synthetic data is bad,
33:17
if it's pulling the model performance down, within seconds
33:20
of training your model, you will just see that result up front
33:24
instead of having to dig in and find out, or this issue is
33:27
actually coming from my synthetic data.
33:30
So that's interesting. Adam I interrupted.
33:35
Yeah, that's cool. So that kind of leads into what
33:38
I was going to say. Yeah I didn't have ever had a
33:40
good experience with it and so I was going to ask like do you
33:45
have any other tips for full data science team that have an
33:49
application? That is unstructured data.
33:51
They just don't have a big data set.
33:54
Are there any other tips and tricks or is it like stuffing?
33:58
See kind of thing. Your I mean not I think, now we
34:01
can go and like, we're going into domain specific
34:03
applications, the tips. And tricks are very, very
34:05
specific to The Domain. For example, it's very easy to
34:09
augment, you know, real samples on in the vision space, you can
34:15
have different coloration. Like, let's say if it's an image
34:17
of a car, you can have different colors of that.
34:20
Car automatically, you can have different backgrounds, you can
34:23
have different noises. You can have people in the like,
34:26
in, in the picture. You can, you can very
34:28
effectively augment those things.
34:30
Yeah, but if it's like on on the NLP side and if it's on the text
34:35
you know, Text corpus side. It's harder to augment that and
34:40
speech sand is, you know, comes is somewhere in the middle where
34:43
you can add different noises to that, but you can't really
34:47
change the voice completely because the voice
34:49
characteristics are what you're learning from.
34:51
But and you know, we generally tend to augment the data with
34:55
different noise, samples on the Text corpus though, even though
34:59
you cannot augment it. You can get a bunch of dummies
35:03
samples from like, you know, like public.
35:05
There's there are more public data sets around text as well.
35:09
And, you know, free Corpus is like, Wikipedia and news
35:12
articles and what not. So you can be a little more
35:15
creative there around getting that data but it's, it's still
35:21
very risky. It's still harder to do in the
35:23
NLP space than Vision. Yeah.
35:25
So so now, let's talk a little bit about like, The different
35:36
kinds of users that you feel benefit the most from Galileo
35:41
and how you've been seeing them? Not only like what has the
35:47
progression looked like and what are some things that people are
35:51
stoked about? Yeah, so yeah, our customers
35:56
today range from like, you know small startups with small teams
36:00
to you in Fortune 500 companies and currently I think some of
36:06
the verticals that Galileo is being used when within our
36:10
conversational a eye contact center or customer experience
36:14
Solutions Financial Services and Healthcare.
36:18
And I think the the progression that we've seen is, you know, as
36:24
we have built out the product we focused on the the the post
36:31
training phase, a lot where like you're trying to find find out.
36:35
Errors in your data. As you know, you know, once
36:37
you've once your training and hydrating on the model and
36:40
that's as I said like that's that's been like sort of the
36:43
biggest piece of the pie in terms of the time spent on
36:47
fixing the data set and that's where uses gravitate the most.
36:52
But more advanced teams that have models in production are
36:57
now starting to build Out Auto are, like, Active Learning
37:00
pipelines around the whole, you know?
37:05
Auction phase where we Galileo, helps them identify which new
37:09
samples that they should. I annotate and add to the data
37:13
set to keep their models relevant.
37:15
So, there's a, there's a, there's an almost like an
37:19
automation play as well, where, you know, users are excited
37:23
about because once they launch this model, they hook up,
37:27
Galileo with it and then they don't they are almost alerted
37:30
proactively on when they should kick off our refresh cycle.
37:35
Instead of having to keep it keep tabs on monitoring it in
37:40
real time. So you need me to I think if I'm
37:45
say I'm a machining engineer, Right?
37:47
In a reasonable size company, may be a small team though and
37:51
I'm absolutely stoked about the thought of Galileo and it solves
37:54
all my problems, how do I how do I sell it to my boss or like how
37:58
do I sell it to leadership? What's the almost a business
38:01
case for dummies? Like a lot of techies aren't
38:05
very good at actually communicating yanking that cap.
38:07
What's the? This is gold and silver
38:09
advertisements for you. It.
38:11
It's it's so funny. You say that Adam because I was
38:17
just talking to somebody yesterday, actually, this
38:20
morning, big difference. But and they were they were
38:24
talking about how you know for them to get something like by in
38:29
they need to not only sell it to their boss but solid to their
38:32
bosses boss and they have to be bought in but a lot of the
38:36
companies that you see out there they don't and especially I
38:39
think there's a little bit of a gap.
38:40
Gap in the mo UPS world. We don't really have, though,
38:44
that literature that makes it something.
38:47
And the guy I was talking to this morning was like, I need
38:50
one slide because my higher-ups, they're not going to want to get
38:54
into very much time with this. They just want to see what is
38:59
it. Give me that one, slide, give me
39:01
the benefits and why we need it and then get it out of here.
39:04
They're going to talk for maybe those couple minutes and then
39:08
they'll be done with it and they probably will be looking at.
39:10
Phones. The people that they're pitching
39:13
to like you got to Captivate them.
39:15
So yeah. Yeah.
39:16
How does that, how have you thought through that problem and
39:20
what are you doing about it? As a great question.
39:23
Yeah, I think that the most important thing that resonates
39:27
with, I mean, there's a there's, this is a great question because
39:31
often times like the buyer who's the, you know, who's the VP of,
39:38
you know, engineering or VP of AI there a little bit far, far
39:44
away from the data scientist who's the the actual beneficiary
39:48
of this tool. And what we've seen work really
39:54
well is for the buyer to sort of be data Centric and and
40:01
understand the importance of focusing on the data sometimes
40:06
you know, they don't and and I've been in that position.
40:11
I've been leading teams and you don't understand how much time
40:16
is actually being spent in this. This process of fixing like
40:21
this, we call it data, detective, work and and and it's
40:26
not only the time, I think the cusp like some of our users who
40:30
are using Galileo, still getting 10 to 15 percent relative
40:34
improvements on their accuracy by fixing the insights.
40:38
That Galileo surfaces. In fact, You know, we we're
40:42
getting amazing feedback where data scientists are saying we
40:46
found insights about the data that we never did in the past,
40:50
you know, you know over even when we're working on this data
40:54
set for a year. So it's like you know, it's just
40:58
because you don't run all like all possible sets of experiments
41:02
and and the algorithms that we've developed go deeper deep
41:08
into the model layers as the model is training.
41:11
So you need that level of deep rigorous experimentation to
41:15
surface these insights and and that is what Galileo brings out
41:19
of the box. And it's and that's where, you
41:22
know, leaders who understand the importance of data Centric, Ai
41:29
and application of AI being even more eccentric than module
41:32
Centric. That's, that's the thing that
41:35
leaders need to understand to really see the value in Galileo.
41:41
You say the last part again the applications of data sent.
41:45
Yeah so yeah the so most teams and this is also this is even
41:53
you know more true for smaller teams but most teams are there
41:58
are applying algorithms on models on their data sets, which
42:05
is, which is also saying that, you know, they're they're
42:09
basically tuning some hyper. M for the model and then
42:12
training their model on their data set in, if you're doing
42:16
that, if you're not building your own neural, network
42:19
architectures or or tweaking the architectures, for your use
42:23
case, which is more than 90% of the the cases.
42:27
Now, in terms of the teams that are applying machine learning,
42:33
you, you do need to be data-centric in your, in the
42:38
amount. And in the, I would say the
42:40
Importance. You you have for your ml
42:43
workflow. For example, like if I am a
42:46
leader, I'm allocating time or I'm allocating Budget on like
42:51
where my team should focus. I think having a key data
42:55
Centric. Focus is is a key way to succeed
42:59
with working with unstructured data is especially so obviously
43:06
in a really talented saying loads of great experience works
43:09
and some great people You got any horror stories for us or War
43:13
Stories? Any kind of like biggest don't
43:16
moment or anything like that? Well I mean I think there are
43:23
there were a fair share of I would say embarrassing Fiasco's
43:30
in production and this is you know this is where all the
43:33
learning comes from. I think.
43:40
I mean, they were there were cases where like, you know, I'm
43:43
not going to sort of name names. But there are celebrities and
43:46
prominent figures in the world. Where, you know, we we try to
43:50
make, you know, make recognizing their their names and their sort
43:57
of their names better. I mean for example, a famous
44:00
query that we worked on was the how tall is Barack Obama and
44:03
then we use it for all the demos but, you know, they're Barack
44:07
Obama was actually like an easier word to Nice.
44:10
But there were a bunch of things and this is, you know, stemming
44:13
from the fact that the world keeps changing like the like the
44:19
person who is the president of the United States changes every
44:21
four years, you know, the the winning football team changes
44:27
every year win Super Bowl. And so you got to, you got to be
44:33
mindful of that and make sure your models can recognize those
44:36
things. Covid was another Ample where,
44:40
you know, everyone was going into lockdown and like data
44:43
centers have to scramble working from home.
44:46
But also updating their models to, to understand covid related
44:50
terms to, to serve users best in production.
44:55
So that's, that's sort of the reactionary or reactive aspect
45:02
of working with unstructured data.
45:04
Deep learning, that's there today.
45:06
If you have these tools that Almost a the proactive signal or
45:14
give you a proactive signal around these difficulties.
45:17
You won't have to face these horror stories and production.
45:22
Hour to finish with a few rapid fire questions.
45:25
If that's all right with you. First one is, it seems fun.
45:32
What was the last book you read? I think I then the trade off by
45:41
Kevin money. I have here and now he's eating
45:45
it. Okay, cool.
45:46
What was it about? It's I think it's about sort of
45:52
the, I mean, the, the tension between sort of like working on
45:57
quality and convenience, I mean, in your, in a day today is sort
46:02
of engineering workflow. You often times prefer
46:06
convenience over quality you would cut, cut corners and, and,
46:12
and, and it's fair and fair to do in some cases, but I think we
46:15
need to, especially, I think, you know, if you're, if you're
46:18
Leading a team. If you're if you're part of a
46:21
team even and a data centers are an engineer, it's a very good
46:25
sort of weed. Think about how to be cognizant
46:29
of how much corner-cutting you're doing basically.
46:33
That sounds amazing. Well, that's that.
46:40
Tell us about the last bug, you smashed.
46:47
Interesting. I mean. the last bug I smashed
46:57
was, I mean, I think it's is, I mean, interesting thing with in
47:01
Galileo where we were looking at I mean it was a strange sort of.
47:14
Not I would say like not a bug within the algorithm but it like
47:17
it was a bug within the the UI that interacted with the API.
47:22
And it was another I don't I normally work on data science.
47:27
I mean I'm managing a team of of cross-functional folks as well
47:34
as ml ml engineers. And the interesting thing was
47:37
like, fixing an issue with the visualization, I think we've
47:42
done a lot of Hard work, within Galileo, to surface to run
47:46
powerful clustering, algorithms on in the embedding space, and
47:50
surface different visualizations that help you highlight things
47:55
within the embedding space. And I think that the bar guy,
47:58
the last bug I fix was like making that embedding space
48:02
visualization with clustered by a specific Insight, that we will
48:08
be provide better. Nice.
48:13
All right. So you are now a co-founder of a
48:19
company Galileo, and you probably have talked a lot about
48:24
marketing. Was the most ridiculous thing
48:26
you almost did on the marketing side.
48:33
That's a good question. I mean, I I think the the
48:42
ridiculous thing I did on the marketing side is kind of like
48:48
not focus on marketing. It's it's I think then that
48:53
tends to be, you know, in the early stages of company, like
48:56
Founders don't really think about marketing and I think I
48:59
would say like, in hindsight to like almost start thinking about
49:03
marketing sooner rather than later in your journey folks
49:07
think that oh, the, you know, the you Almost like bootstrap,
49:11
the company to up to, for a product Market fit, you know,
49:15
with founder, LED sales and and that should be enough.
49:19
But I think every company needs to think about how we, how we
49:24
can improve or create mindshare about the company itself.
49:28
And we've been thinking a lot about about that, especially
49:32
because Galileo is in this space where we're creating a category
49:37
of tools here. Then there are no tools.
49:40
In the data scientists workflow today that help us almost like
49:45
we call it ml data intelligence and we you know we there is a
49:50
need for ML data intelligence now more than ever in the ml
49:55
workflow. So I love that word ml data
49:59
intelligence. That's, that's a marketing
50:01
thing. That that's what we should name
50:06
this this podcast. I like that a lot.
50:10
Last one for you. What piece of technology are you
50:14
bullish on right now? That might surprise people.
50:21
Hmm, I'm quite bullish on Quantum Computing.
50:24
I mean I think you know every like I I know that we're still
50:28
very very early in that Journey but I think we can, you know,
50:33
leverage, you know multiple orders of magnitude more of
50:37
computing power. I mean as we all know that less
50:39
than Moore's Law is flattening now and you know, we're trying
50:44
to do is sort of things with distributed computing and you
50:49
know, making sort of accelerated.
50:51
There's and that to almost like keep the Moors keep Moore's Law
50:55
going. It's it's almost like we're
50:59
hacking Moore's Law, right? Like with being able to run
51:04
certain pieces of our software faster in and so yeah I think I
51:10
mean like I'm extremely you know, excited about the work
51:15
that's happening in the quantum Computing world, and and very
51:18
very sort of bullish on on that. On the outcome of that.
51:22
Nice. Well man this has been a
51:26
pleasure I really appreciate you coming and sitting down with us.
51:29
Yes. And teaching me a little bit
51:31
about this unstructured data world that I don't get to talk
51:36
about enough with people or I always love talking about it and
51:41
we'll go ahead and end it right there.
51:42
Thanks again. Yes yeah thanks for having me
51:45
and you know love the discussion we had and you know hopefully
51:53
Hopefully we start thinking more about like unstructured data as
51:58
we start applying it more in the real world.