Latest / Agentic Conversations (formally mlops.community) / Real-time Machine Learning with Chip Huyen // MLOps Coffee Sessions #133

Transcript

Real-time Machine Learning with Chip Huyen // MLOps Coffee Sessions #133 · Agentic Conversations (formally mlops.community)

0:00
Hi, I'm chip. I'm the CEO of clip on AI and I
0:03
like my coffee with Autumn milk and no sugar and no, you are a
0:08
cappuccino sometimes. Just give you sugar, I just
0:11
don't get it. But I feel great pain.
0:16
This new my man you're in Martha's Vineyard right now.
0:20
Yes, sir. And is it an actual Vineyard?
0:24
Did you figure that one out? I have not spent time cracking
0:27
that conundrum quite yet. And that is quite yet.
0:32
No. Maybe by the end of this trip,
0:34
maybe maybe we'll see what I have time to get to nice.
0:38
Well, this I mean we say this all the time like this was a
0:42
legendary podcast, this was incredible.
0:45
But I think this time we outdid ourselves because we got none
0:50
other than chip Yuan on the podcast finally.
0:54
I've been trying to get her on the podcast for probably like a
0:58
year and a half now. And the Stars aligned dude, what
1:02
did you think? I mean it was Chip is one of the
1:06
most notable ml experts out there.
1:09
Her thoughtful blog posts, not to mention her Sanford classes
1:13
and the various programs that she teaches, and it was just
1:16
awesome to interview her. I mean, I got to say way to go
1:19
with the persistence and one thing I realized.
1:23
So I thought everybody knew chip, I would expect if you're
1:25
in mlr to, you know, chip. I asked mederic Shout out to
1:30
him. He's organizing the Luxembourg,
1:33
local community that we're getting started and I said, hey
1:36
man, I'm about to interview chip you have any questions because
1:39
he was mentioning that he listens to the podcast and he
1:44
told me who's chip, I was like, hey once you don't know, it's a
1:49
beautiful thing to discover. It's a beautiful thing to
1:51
discover an expert. You didn't know about learn more
1:53
about it. Yep.
1:55
So for those who do not know, chip obviously did everything
1:59
the vision who But she also was working at snorkel before she
2:03
set out and now she is the CEO of a company called Clay Pot Ai
2:10
and this conversation was brilliant man and there were so
2:13
many cool ways that she looks at things.
2:17
For me, I will say my favorite was probably when she talked
2:21
about how she likes to learn and how she likes to force herself.
2:24
Basically, put herself into situations that she knows will
2:28
make her learn And it reminded me of this video.
2:32
I saw from this guy named Johnny Harris.
2:35
I don't know if you saw that video, it's like these forcing
2:37
functions and how you can basically like supercharge your
2:41
learning and it is by doing that, like, by putting yourself
2:46
into a situation that, you know, you either have some kind of
2:53
responsibility to others to learn or there's this like,
2:58
what's the word I'm looking for. It's like accountability on you
3:04
so you have to learn and yeah, that was awesome.
3:08
What about you? What were some takeaways?
3:09
I know we talked to Tony about streaming.
3:11
That's her big thing. These days, you know, thing from
3:13
that's exactly what I was about to say.
3:15
Yeah, just to sort of reiterate chip, is a leading expert on
3:19
machine learning and building machine Learning Systems.
3:21
She's the author of the recent O'Reilly book designing machine
3:24
Learning Systems, she's taught courses at Stanford.
3:27
She worked at snorkel. Snorkel as Demetrios Jesus said,
3:31
speaker at multiple notable conferences, including scale, a
3:35
eyes transform X conference and she is an expert on streaming.
3:39
That's what her company focuses on.
3:41
And my favorite part of the podcast was when she walked
3:44
through, what streaming machine learning is like, and her point
3:50
that it's not that hard when you really think about it.
3:52
It's not that big of a mentor, your two card.
3:58
I think it was like this is Simple in theory but maybe it's
4:02
more complicated in practice and that's exactly where her
4:05
perspective is what we just need the tools and that's what I'm
4:08
going to go to, which I appreciate it.
4:10
I think her walkthrough of the streaming Paradigm was very
4:13
helpful and is probably the highlight of the episode for me
4:16
because of you know how technical got and how much it
4:18
helped me understand streaming. And for those who are interested
4:21
in learning more, I would recommend checking out her
4:23
website and seeing the recent post that she wrote about
4:26
streaming in a direction is streaming for data scientists.
4:29
Yes, yes. And we could probably will try
4:33
and get chip to give us some free books that we can give out
4:36
to people. So if you want to leave us a
4:38
review or leave us a comment on YouTube and let us know that you
4:43
want some free books. Then we'll see if O'Reilly will
4:46
give us some otherwise we will buy them ourselves with all that
4:50
good sponsor moaning we got coming through and man, let's
4:54
just get into it because it was fun for anybody that is
4:58
listening, you know. It'd be super helpful for us is
5:01
if you shared this episode with a friend that you think would be
5:05
interested. And as we mentioned before,
5:08
leave a review and give us a like all that good stuff.
5:12
It makes a world of difference to the algorithm and we want as
5:16
many people to see this as possible.
5:20
All right, let's jump into it. I want to ask the first
5:31
question. And the first question I want to
5:34
ask is, let's how am I going to phrase this?
5:39
All right. So, Chip you have a Discord
5:43
server and a lot of people hit me up and they asked me about
5:49
stuff that's going on in your Discord server because they
5:52
think that it's called the ml Ops community and then we also
5:57
have something called the ml Ops community.
5:58
So can we just settle this forever ever?
6:02
Right now. Can we call it something else?
6:04
Your Discord server? Can we just figure out a
6:07
different name for it and then we will run with that so that
6:10
people don't have to hit me up. My to me a red Kanye, West
6:14
wanted to treat Martin what music, you know, if you let
6:17
using Community is Mo off. So, I don't know, I think I will
6:19
just call it a mouse committee because you were too lazy to
6:22
name it. It's very casual.
6:25
We didn't mean to like it's not a good like an organization.
6:27
It's not registered anywhere. It's just like, we just like way
6:32
too lazy. To come up with a name.
6:35
So he has name suggestions like even have any good names and
6:39
yes, and we have to declare that every, every few months, every
6:43
like what should we name our community that was just like, no
6:47
idea. I'm just saying it because
6:52
sometimes I'll get hit up by people.
6:53
Like, oh, there's this cool thing that Eugene hands doing in
6:57
the ml Ops Community. Where can I sign up this great?
7:00
Like I would love for you to Onion to be doing something with
7:04
us, but I think you're a little bit confused here.
7:07
Eugene just did one thing with us and then I haven't heard from
7:11
him since. So there's a new name.
7:14
We got to figure out a new name and we'll be taking name
7:21
suggestions. We have a discussion like every
7:23
month. There we go.
7:26
So it wasn't people have For Hoosiers gather on this chord.
7:35
We agree. It's not it's not really like
7:38
and it's not really and organizations make sense.
7:43
Well that being said it is awesome what you're doing and I
7:47
think you're sharing a ton of cool stuff in that.
7:50
Discord for anybody out there that is not involved in it.
7:53
Go check it out. Were hopefully going to separate
7:57
the names and so we call it, the Discord ml Ops or the slack ml.
8:03
Dobbs, we gotta we gotta figure out some and we're open to
8:06
suggestions and so tweet at us, tweet it chip.
8:09
Next question I have is you are like the incredible, probably
8:17
the most prolific evangelists out there when it comes to ml
8:21
Ox, you spend a ton of time. I would imagine just creating
8:27
cool content. Your content is so deep and so
8:31
thoughtful. And so One question, we got from
8:35
the ml Ops Community from the slack ml Ops Community is.
8:42
How do you go about evangelizing, ml Ops to your
8:47
collaborators and customers? Do you have any advice that will
8:51
help others out there? Be good at this task.
8:56
Okay, you'll feel it. You set me up for this is like
8:59
okay I don't think I'm like the top evangelist.
9:01
I think I just so many amazing people like you mentioned even
9:04
guarantee quiz. So much content.
9:06
There's so many that you create incredible content.
9:09
You have you like you consistent?
9:11
You I feel like a more like a casual like I don't I don't do.
9:17
I do I do enjoy learning ladies that says on the infield for me,
9:23
I don't think of it as like eventually ISM for me.
9:25
It's more I that's a topic I want to learned and it just
9:29
happens that for me on one good word for my children is to try
9:33
to talk to other people about it and try to summarize and yes I
9:39
thought whenever I write a little bit topic, I first talked
9:43
to people and then I make note and I try to organize and in the
9:47
process of wait a second, that doesn't make sense.
9:50
So in then, I get feedback from other people.
9:52
And then we'll publish it. I get a lot more if you'd like.
9:55
So it's just a way to learn. Yeah, yeah, that's that's for
9:59
super helpful. I mean there are a lot of people
10:01
out there that are that are writing and authoring content,
10:03
but one of the, what always strikes me as Demetrio says, is
10:06
how deep you go into pulling together, what it is, you pull
10:10
together and how much structure you place on very novel topics,
10:16
right? Or very, you know, Cutting Edge
10:19
topics, for example, your pieces on streaming and Mel or how to
10:23
approach ml systems design. And, you know, I'm kind of
10:27
curious, you know, in the process of your own learning.
10:30
Let's say, you know, you're learning about a new topic at
10:32
what point do you go about pulling together these posts?
10:35
How do you do it? What's your system, right?
10:37
I think people like hearing these kinds of things, I'm kind
10:40
of curious if you could share with us.
10:43
So oh, this is interesting. So I do think it's like there is
10:48
student okay? So like let's be frank, I really
10:51
like. Tension right, there are four.
10:54
It wasn't like I haven't written anything for so long she was
10:58
poor something because UK this is attention but you want the
11:01
person you need to fight it. So for me, like if sometimes
11:05
anyway, goes for your few months ago, has his ideas like a
11:07
popular topic, but I didn't feel genuine to me because I, okay
11:11
it's not. Doesn't feel like it's a topic
11:13
that I'm actually interested in, but I write it because I feel
11:18
like, oh, this because I think other people interested in it
11:21
and then I Did Amanda by not publishing it.
11:25
Because so so for me like there was a process.
11:26
I first have to be very interested in the topic itself
11:30
so like screaming and I was super cool compiler.
11:34
It was super cool and I do think it's the best time to grab our
11:38
topic is when you're learning about it because then you have
11:42
this like big analyze. So we've got these are still
11:45
make sense. You might not also like a lot to
11:47
you seem confusing to me - would be confusing to other people as
11:51
well. So so I tried moving know.
11:54
So I think what my boss has. Another idea for another topic.
11:56
So like good bike much appreciated and then it's time
11:59
to hunt down people that I can do, or the best in the topic.
12:03
He asked my friends to, like, I guess it's, didn't you just do a
12:08
lot of, like, brainstorming? Like I just go to friends like,
12:11
hey, what do you think about this?
12:12
Like, personally even want to talk to you, too.
12:14
And I got him like, hey, have you heard about that contract
12:16
recently? Because I'm thinking about a
12:18
schema, so evolutions. And if they eat rice as they
12:22
could, Pick up and take when I talk about it, then was okay.
12:25
You know what? I'm actually thinking about
12:26
making some load about it, I can I share with you and they were
12:29
like, yeah, sure because it gives it interested when they
12:31
would all like they're going to hate.
12:33
You should always be my scary. Hey, you should talk to other
12:35
people might see. So person A or B or C because
12:38
they also similar field and that is a paucity to take me like
12:43
months to write about something we could.
12:45
I could, I would imagine it would take that long to really
12:49
understand the topic. I love that.
12:53
So it's almost like a collaboration and it is a
12:56
collective type of learning and also the idea of going out and
13:04
writing about a topic as you're learning.
13:06
It is so true because you see things with those fresh eyes and
13:11
others who are getting into it and diving in.
13:16
They probably like you said have the same questions as they are
13:19
trying to navigate the space and It's like you are making a
13:23
roadmap for them and you're carving out a trail in the
13:26
jungle. Whatever.
13:28
It's too much to expect a chinon.
13:34
The result you get scared? I took me.
13:36
I feel like it was like, oh we're right.
13:38
And I want to be a road map for other people.
13:40
It's like there's a lot of responsibility and I just don't
13:45
agree. I do that.
13:46
It would never finish because it would never be perfect.
13:51
Interesting. So just keeping it very low-key
13:54
and realizing that okay, maybe these are notes, maybe this will
13:58
evolve into a blog post. Maybe it will evolve into a
14:01
whole book, but you don't have expectations when you start out
14:07
and you set off on the Journey of what it will become that.
14:12
Being said, you did write a book.
14:14
So I imagine at some point you had to make the decision that,
14:20
okay. This is, there's Enough here.
14:22
I know enough about this that I think I could put it in a book.
14:24
How did that go? Obviously the right to say
14:29
something? Now, it's just saying, good
14:31
point, good point Demetrius, so you can call and, you know, same
14:33
question about your courses, right?
14:35
I saw, you know, you written books, you've taught courses at
14:37
Stanford multiple, you know, how did that, how did those sort of
14:40
more structured? Maybe less exploratory sort of
14:46
projects come about. Yeah, so I think is is also
14:52
books or courses are soldiers way children.
14:55
I'm not sure if I ever thought. So I don't have any tangential
14:58
but I'm not sure if I ever thought about like what would be
15:02
the most important things for you to optimize for as a person?
15:06
So so I think it had been having this conversation with myself.
15:09
I guess what it was, I'm going to be the most important for me
15:12
is to like personal development, but if you treat, I do is just
15:16
so I feel like everything I want to do is like you see, myself
15:19
progress and and I think a lot of these stem from like what
15:25
could put me installation when I'm forced to learn the most?
15:30
So the first cause I told was not really, it wasn't meant to
15:34
be of course. So it was back in summer 2016
15:38
and you're like, tens of flow was just just came out and it
15:41
was very exciting and it did it easier for my internship and it
15:45
was great and it was like only once you learn more.
15:47
It's only went back to school and it talk to my professors and
15:51
was like, Hey, can you like dish?
15:52
Of course on like this we'll probably wasn't really much to
15:54
learn about it and they were, like, I don't have time for it.
15:58
And so somebody just gave talking to people to people
16:00
like, hey, you want to form a book group, like study group on
16:03
tensorflow and somebody give me an idea.
16:05
Like, hey, you know, what does is like idea like
16:07
student-initiated course where you can actually propose of
16:09
course, who stand for instead, what less you like, give you a
16:12
classroom so we can. So it started out as like a
16:15
workgroup like a way for us. To study together and it won't
16:19
into like a course because somebody needed to have it done
16:23
at reading group. I went to, I suppose you read a
16:26
paper together Visual and doesn't have a to like, prepare
16:30
everything about that paper and present.
16:33
So I guess like that was like, how is that became?
16:36
So I think it's just as a very natural progression.
16:41
I didn't think, I don't think I was qualified to take me out.
16:45
Now we're gonna find out But I think the key for me was I
16:49
having a lot of people who were willing to help maybe because I
16:55
think maybe there's and come from a place where a nice very
16:58
okay to ask for help. I know it's hard to friends here
17:02
and sometimes was like this is very weird thing about something
17:04
that says don't really ask people to have them with their
17:08
questions and I don't know why I really appreciate the the
17:12
transparency and honesty about, you know, how prepared you might
17:17
feel. and how how much of a Learning Journey is for you,
17:20
because I think it's valuable for others to hear that that you
17:24
don't necessarily have to feel like an expert to develop
17:26
expertise and that's often part of the journey is to allow that
17:31
Curiosity to let you figure things out and letting something
17:36
become a forcing function for you to have to grow more because
17:43
you basically set yourself up and you're saying okay how can I
17:48
like you said, how can I put myself in situations?
17:50
Where I Going to learn the most that is a really great way of
17:54
explaining something. That I also feel like is it's
17:59
something I have and I like to do and maybe I don't necessarily
18:04
consciously think about it all the time but I really appreciate
18:09
the way you put that into words and you say okay this is
18:12
something that I know if I go do this, it's going to make me have
18:16
to learn. It's going to make me have to
18:18
become better at whatever it may be that I may I'm trying to I
18:22
set a goal for something and so you're in a way just I don't
18:28
want to say it's like assuring the success but it is helping
18:32
making that that become more of a How can I put this?
18:41
You are in a way, assuring the outcome and so, I like that,
18:48
you're doing that. Now, when it comes to What
18:53
you've talked about a lot online, I want to dig into some
18:58
of this because I think there's the whole streaming ecosystem is
19:03
pretty crazy to understand right now.
19:05
And real time is something you've talked about a lot.
19:09
There's probably like the easy question that I wanted to get
19:13
into which is Is real time. When is it not needed to use
19:20
real time? And I think that, I see a lot of
19:25
people quoting and I've seen one of your pieces of work.
19:30
That's talking about how machine learning is going real time.
19:33
Now, a lot of stuff doesn't need that extra complexity, right?
19:37
So, when would you want that extra complexity and when would
19:41
you not So that's the first assumption here.
19:45
That streaming means that your complexity.
19:48
I do think that like, me technology is going to get
19:50
simpler all the time. So I'm not sure it's gonna be
19:53
like, even more. They tried complexity.
19:55
And the question is, what in, what use cases, we don't need
19:59
shrimp of your tongue is the question is like, when do you
20:03
not want to access data as it comes like personal, you want to
20:08
go to? I don't know, like become an
20:09
archaeologist and dig up something from my tennis and
20:13
years. Ago.
20:13
Then you probably don't need streaming data, right?
20:15
Because the data is, there is like a Subway?
20:18
It is not going anywhere. Yeah.
20:20
Yeah. So but breaking like is just
20:22
informations. Right?
20:23
I think I think we I do. See that like as we tend to
20:30
happen we a lot of people like have access to think fast but
20:34
before we go spray is like you send mail like snail mail and
20:38
they were like and how I want email and then even in there was
20:41
too slow. I want something my very very
20:44
Like instant. And I do think that people tend
20:46
to write writings that are faster and the same type of
20:49
data. I mean, it's real time.
20:51
Should I just like yesterday just happening right now,
20:54
instead of having to wait and even the delighted in seeing big
20:56
dealing with a few hours a day. But anyway, if you can access
21:01
right now isn't wait. Wait.
21:02
So you mentioned how the assumption is that streaming
21:09
adds extra complexity? Yeah, would you say right now
21:13
because And then you also said that down the line.
21:16
It's not going to be like that, but right now, do you feel like
21:19
we're out of state, where we can safely say, streaming and going
21:23
real time, doesn't add extra complexity or, is that something
21:27
that needs a few more years or a few more months to actually be
21:31
able to say that? I do see that abductions goes
21:38
hand-in-hand with availability of tool of 20 so we do see is a
21:43
companies for someone. I say hey we have this problems
21:46
and we look around. There's no good tool to address
21:49
that then they are now gonna move forward because it's too
21:52
much effort for them to do something in house, right?
21:55
But we do see that maybe if there are tools and people said,
21:58
okay, now I can just liking stuff be consulted and if it
22:01
doesn't work then I do see that my push a dog.
22:05
More and it's like, it's a back will break.
22:08
So I do think it's like sometimes that's what is a
22:12
reason. So we we start a company and in
22:14
the space and the questions we did have was a virtually the
22:18
market, but that we do do things as like we can also, okay?
22:24
I'm not saying that it's a very, very present, a chance.
22:27
Just like, maybe if we can build something that is extremely easy
22:31
to use, the maybe we can't make it not true.
22:35
Early anymore. I like that.
22:38
I like the way you put that, you know, a tool or tools help Drive
22:43
adoption right. I think that's so it sort of,
22:45
you know, one of your contentions there, and one of
22:50
the things I'm thinking about, as you say that is, you know,
22:52
you wrote sort of, a famous blog post about how, you know, there
22:55
are 200. Plus machine learning tools that
22:57
are out there. You know, you've worked a tool
23:00
company and snorkel. You're now running a tool
23:02
company, you have a lot of expertise in terms of just like
23:06
thinking in terms of machine learning.
23:07
Them's, tell us some of the things that you think a lot of
23:11
tulle focused companies get wrong right now, what are some
23:14
of the things that are not being done?
23:15
Well, some of the mistakes that are being made to should be
23:17
avoided as your friend, everyone in a single like podcast.
23:24
Just curious, just curious. You don't have to say any names.
23:29
I think many like not what other countries are doing wrong.
23:34
I do things that a lot of them are incredible.
23:37
And I do think is like as I'm starting a company and trying to
23:44
build a tool. I appreciate the process a lot
23:47
more. Is it get me a new light, you
23:49
appreciated? Well, regular time.
23:51
You look at it. It is so easy.
23:53
Like just I didn't seem so easy and I went as company like, I
23:57
mean if you didn't really appreciate like how the tools my
24:00
God options and I how the design process, but now I just like
24:03
appreciate that. So I feel like every company out
24:06
there still I'm like really amazing.
24:08
Islam is a still alive that that is like already.
24:11
Like amazing. I think it's chair that I do.
24:13
A lot of like our mistake I was so because I'm a first-time
24:18
powder, I've seen it at other companies are not just like the
24:22
company will get before but like I was of did help like advice
24:25
like few other companies and also they just won't personal
24:31
devices. Like clay pot is originally we
24:35
as engineer I did. Think about owned and not
24:40
engineer stuff. That is very boring present.
24:43
One thing I realized is like getting this selling your
24:46
company who's not building to a product but you're building a
24:49
company and if appreciate how much work it is like outside
24:54
product because it's, for example, okay, we need people
24:57
and was like, oh, wait a second. We do, but has a high in process
25:00
that we've ever had like the because it's very easy to
25:03
complete a, how broken the hiring process of other
25:06
companies or But sometimes you just like, when you had to build
25:09
us a really ugly, what do what do we do?
25:12
They just like, how do we make sure that this hiring process
25:14
better than other hiring processes?
25:16
Are we complained about before? That's what I'm saying.
25:20
We go to product is itself, very easy to hate, talk to users, get
25:26
feedback and Eatery. Is it sounds so easy.
25:29
But even talk to 100 people Singletary, 100 things.
25:32
And how do you know who to listen to?
25:34
I think the hard part is not and you talk to people but I should
25:36
decide what. To take into account and I even
25:39
if he listens and then how do you like translate that into
25:44
engineering my stone. So, so I think that is a lot of
25:50
money because he has a balance between the opinions and data.
25:55
I think sometimes we say okay we need more data to make decisions
25:58
but you don't have like forever to wait for for data.
26:02
So just need to like make decisions and then you have all
26:05
the guys. So that was like, how weird how
26:07
Do I know that this is a right decisions?
26:10
It's yeah. So I feel like it's tough.
26:13
It's a tough World. Hey there.
26:18
Emma sitting standing over and I'll have a product, a teachable
26:21
hub. And if you're into envelopes and
26:23
want to hear more than just buzzwords, subscribe to the
26:26
envelopes Community podcast and listen to the best practitioners
26:29
in the field, sharing their wisdom.
26:34
Yeah, the you're building culture.
26:37
You're building a product. You're building a company.
26:40
It is, there's so many things to keep in mind and also how you're
26:44
incentivizing people. Like, if you go out and have a
26:47
sales team, you are the incentives, you give the sales
26:51
team are going to make them. It's going to create certain
26:54
actions for them, that could be different.
26:57
If you had different incentives, or same goes with marketing.
27:00
And so it's fascinating to think about and I am going on question
27:09
there actually that I wanted to dive into Interviews famous
27:16
piece about ml interviews, you know, guides to them.
27:20
You know, it's something I actually looked at When I Was
27:22
preparing for machine learning interviews, for thank you for
27:24
writing it. What are some of the things that
27:28
okay, I'm not going to ask you what people do wrong but what
27:30
are the some of the things that you think that you focus more on
27:36
than maybe others are. Some of the things got some
27:37
areas where you may have reinvented the interview
27:39
process, you know, yourself and running your own company.
27:42
That was that you're on the other side of the table.
27:45
So I don't think that the intake process is very personal which
27:50
is very with personal to both the candidate and interviewer.
27:55
So so ears, like so makeover and I we come from very different
27:58
backgrounds. So he read some treatment about
28:01
Netflix. He's very experienced, the
28:05
passage of your tongue of like senior, very experienced
28:07
Engineers. Right?
28:09
And me, I'm like, I went to college, I graduate college and
28:12
work for a few years and it's not a cop.
28:14
Never reach my staff. Level me Shilling Engineers.
28:17
So what does that means? Is that like for me I have this
28:19
personal belief that, okay, if you're young, you're smart,
28:25
you're motivated. You to get things done.
28:27
Right. So, that was my personal belief,
28:30
but then, at the same time, there's a lot of, like, fire,
28:33
like more complex, technical details.
28:36
It just like, yes, you can. You young?
28:38
You're smart and motivated. You can learn because I how long
28:41
you're taking a jeweler and it is startup.
28:43
You just sometimes just don't have time to wait forever, but
28:46
person to learn. So that was a difficult decision
28:49
right now. Beginning.
28:50
It was like, yes, I want. I don't care how many eggs, for
28:54
how many years of experience someone else.
28:56
As long as a person is smart and motivated.
28:58
I could carry a lot about that, but then I was like, wait, but
29:02
yes. But like do we like so?
29:04
So we think we need to balance out our belief.
29:08
Yeah, finding someone. If they're already there or if
29:11
you, they need to Train to get there.
29:15
And so I didn't make a bet on under the person like every
29:23
higher is a bad. Yeah, and then like luckily at a
29:30
start-up I guess you can edit the team quickly, right?
29:33
But it still is painful. Not only on the deep emotional
29:37
level, I mean like you can if you do make a, if you feel like
29:43
somebody is not right for the team, you can change that
29:48
Dynamic and it's not, okay, okay?
29:52
I think it's very hard because I as a TV bring on someone Do you
29:57
even if you omit them like a yes, we bring it on your own
30:01
because we want to build a company with you.
30:04
You are one of us. Right.
30:06
And people Joy because they feel like they joining this course
30:10
together. And if you let go someone like,
30:13
okay, so if you are 100% company and you let go of one person,
30:17
just one percent ivara 10% company and recommendations 10%.
30:22
And it's very clear. It's very obvious.
30:24
Like it's not something that I okay the - Ben doesn't show up
30:28
in their ways notice. It's like everyone's a notice
30:31
and its really let go of one person.
30:33
It's not just a person MSR Wonder, am I next?
30:36
Because it feel like because we work so closely with each other,
30:39
you know? It's just so worried isn't
30:40
personal connection. It's not, it's not easy to it,
30:43
did your team? Yeah, I think I feel like I I
30:47
also so I see that side of things.
30:49
And I also see the other side when I've noticed, if I'm
30:53
working with someone on my team and they aren't really putting
30:57
in the amount of work that I'm putting in.
30:59
If I am, I love the company that I'm working for and I am putting
31:04
in these Blood, Sweat, and Tears.
31:05
And then I look to the left of me and I see someone that is not
31:10
as invested as I am. If they get edited out of the
31:13
team and they did they get let go.
31:16
I'm almost, it's not that I'm happy for that.
31:20
But I do feel like okay, let's bring on someone that is more of
31:25
at the same level that I'm at because otherwise if one person
31:30
is able to do that it in a way, feels like you can sometimes not
31:36
all the time. Obviously have the team be
31:39
dragged down. So I understand the sentiment
31:44
and you think that's like for a start-up, you have a student
31:46
amount of like failure budget. So yes, even if another person
31:51
that doesn't work out is a hiring figure is like you made
31:55
the wrong bet. It doesn't feel so, yes, you can
31:58
correct that and you can fix that but it cows to work with a
32:02
number of mistakes. You can make Yeah, I think that
32:08
that's a very mature perspective, you know, I think
32:10
these kinds of topics there. It's an art, not a science,
32:14
those science can certainly help us understand how to do art
32:17
better. And I think I enjoy hearing it
32:23
from your perspective, you know, having been on both sides of the
32:25
table is sort of, you know, both somebody that's taught about
32:27
like a lot about interviewees. And now, as an interviewer, it's
32:31
interesting to hear that shift in thinking that you've had, but
32:35
without pivoting too hard, I really did want to ask you a
32:38
little bit about your book which recently came out.
32:42
Designing machine, Learning Systems, Great Book, very
32:45
comprehensive. Thanks for writing it.
32:47
Thank you. My question to you, is, which
32:52
topic do you wish you had gotten deeper into, or do you feel like
32:57
there's now a lot more to write about now that you have the
33:00
perspective of hindsight? Oh man.
33:05
I feel like if you asked me today to be a list of topics,
33:08
you ask me the month. Twitter is a good topic changes
33:14
so fast, it's not do things as they are.
33:24
I wish I had gone way to streaming.
33:26
I will have actually I think she was a cop-out answer.
33:31
Yeah I think I do think this. A lot more concept that I wish
33:37
it had gone to it by until like what, right?
33:42
Don't talk about my agency and the book in the book, I
33:44
discovered a lot of latency but another concept I think is very
33:48
important is stillness and it's not something you can shoot off,
33:52
mic latency and Stillness. So for example, Taking us all so
33:58
late. So what is that?
33:59
They didn't really say come from?
34:01
So let them see if you're usually defined as the time when
34:04
you receive on the time, you receive a prediction to the
34:07
time. I'm sorry.
34:08
Receiver position, request to the time.
34:10
You return a prediction, right? So either Versa bouquet like,
34:14
best predictions you precomputed predictions.
34:17
So, like the time it takes to say, okay?
34:18
This is how we take the decisions of habitation, fetch
34:22
the pre-computer prediction so that latency but as a cause of
34:25
vibrations, pretty stale. Right.
34:27
So now I people I do another hey, how about you predictions
34:30
on my but that using bash features.
34:33
So in this case the latency is like the time it takes to
34:36
retrieve the pre-computer features and then the time plus
34:40
the time it takes you to complete the predictions based
34:43
on the precomputed features. So that is a.
34:46
So now suppression is fresh but features is still.
34:50
So now you polite go and it's like how do we make the features
34:53
less still. So so people that do things,
34:56
maybe they come. It features an appearance like
34:58
in the batch process. Maybe you compute these features
35:01
that we every day but instrument, you can still
35:03
compute features like on a Cadence, right?
35:06
So, so I say you can compute like for like every time it is
35:09
every minute. So so and as a petition and
35:11
especially this prediction, I like features value.
35:15
So the question I think this is one thing that makes a bear like
35:19
very like the division between streaming and Bash really weird
35:22
because I don't bash, you can also do it on a Cadence and
35:26
streaming it. And he's on occasions, I'm not
35:28
socially. What, like, what?
35:29
Like, a beach 3 battery is like make its shrimp or bad because
35:34
I'm doing Michael bash, very fast or I can do a live
35:37
streaming like pretty slow. But another thing streaming and
35:40
give it to give you a give you is that maybe you don't do
35:44
compute future value and arrogance, say you won't be
35:47
something like driver grading so you can, you know, the devil
35:52
reading depends on user reading, the reading, the user gives the
35:55
driver and so you can see Hey, every time there's a user user
36:00
gives really into a driver, then it was just in the update that
36:04
driver reading value. So now there's like, almost no
36:06
still less because of British am, if you meet the governor
36:09
readings, especially this Valley.
36:11
So now like this less still for the future because the cause of
36:14
my behavior computer constantly like a lot, but you might not
36:18
every time, you know, you can fit a lot more frequently and
36:22
say every five minutes every 10 minutes doesn't make sense.
36:26
Yeah, does it, does it sit It's a fascinating introduction to
36:31
some of the technical challenges that start to come up when
36:33
you're thinking about, you know, the streaming Paradigm, right?
36:40
And what, The kind of question that I have to, you know, I know
36:45
you wrote a recent post for, you know streaming for data
36:48
scientists. You've been very active about,
36:51
you know, trying to explain what streaming looks like how it
36:57
works and what it's applications are.
36:59
I'm kind of curious, where do you feel like companies or
37:05
Engineers still struggle to grasp or to cover the people?
37:09
What is the the final thing? I think that people still don't
37:13
About streaming that you're still looking to explain or do
37:16
you feel if they understood they really get what you get about
37:20
streaming. I can is trimming at the course.
37:25
Very, very simple idea. This is like a pant all along
37:29
right now. You have an event coming in a
37:31
panda and you process that lock. So I think it's a it's like it's
37:37
just very simple but before so I don't think any conceptually it
37:41
shouldn't be difficult. I think like this, a lot of
37:45
so-so I think the the challenges could be so so like when we use
37:53
some tooling and we get used to it, that it's hard for her to
37:57
move away from that the system, right?
38:01
So, say, like I would say, for example, like do you spandex, do
38:05
you like these apples? Just lie.
38:09
Yeah. You always have these pedals all
38:10
the time? Yes.
38:12
Also, practice is great. Yeah.
38:14
It was sitting this sort of thing for pandas that make it
38:16
hard for people to move away. Was a pendant Brandon.
38:20
You understand like say say so Independence right?
38:23
You may have a CSV file. If we come see us, we never did
38:25
a free and if you want to compute something like the
38:28
average value of a column, then you just like you spend a strike
38:33
something that guy out. Oh, so you're not a shot at me.
38:36
Okay. Now, I hope it's not a pop quiz
38:38
because the pretty wrong. So, so that's also something a
38:42
kind as couple three things. The first is asking myself, the
38:46
schema is describe what the shape.
38:48
The data like here is a, The Columns reading is a float,
38:53
you're the driver ID is an intent or something like that,
38:58
right? So the schema describe the shape
39:00
of the data, these are the data source which is a little data
39:04
field is a schema, it comes from a CSV file and there's another
39:09
is a conclusion engine, which is my pandas combustion engine.
39:13
And now I have you can relate maybe, bandages, slows, and it
39:16
will switch it out and it dusk more time.
39:18
Yeah, right. But I very similar interface,
39:21
so, so now I have three different thing.
39:23
So now I want to go too late for pandas.
39:26
Do you once you read in the data?
39:29
The data is like doesn't easily change until amazing your menu
39:33
at the - yeah. I don't want to read into CSV
39:35
file. Like that is solid upgrade,
39:37
right? You can manipulate that but they
39:39
that doesn't like mature eyes. Doesn't like a tawdry, add a new
39:44
transactions without your knowledge.
39:46
Thank you. You make that happen.
39:48
So that I think it's like because it's just stationary you
39:51
don't really care about schemas in fact that this is zero scheme
39:55
a chicken. So say like you have this data
39:57
frame, we have on the road, you like reading should be float.
40:01
But then if you add another row and it makes that really like
40:06
shrink there's nothing connects like, yay.
40:08
Go for it. I do do do something.
40:10
Like, it's not going to stop you and you see like that and work.
40:13
So it is stationary but it's not going to work as a data.
40:16
Keep on changing. Right?
40:19
Or why is it comes to energy? Is very few.
40:21
People don't use my eyes and don't think the competition
40:23
engine. But you just like I'm not sure
40:25
if I would prepare, don't tell me like, I kinda like, you know,
40:28
it's not something. I'm not sure maybe just me or
40:31
you or think about that? No, no.
40:34
I think you're right. Yeah, so so so I do think it's
40:37
like because of that but like when you start to streaming now,
40:41
it's different. Now sharing it is not stationary
40:43
anymore but it keeps on coming you don't, it is indeed a comes
40:47
in Don't even knowledge because it's to our understanding data
40:50
and because I needed all schema checking and also because of
40:55
that, you might have two different compression engine
40:57
because now the day my you mazey Flink instead of like Ray or a
41:02
spark. So, so, so I think it's just
41:05
like maybe you can have some more abstractions like
41:08
decoupling on the concept may be scoping things easier for people
41:12
to Graphs it not sure, hmm. Okay I'm going to say make it
41:18
easier to graph it on the saying is that if we have a better
41:23
understanding about how things are what things are happening
41:26
under the hood, the end. Yes.
41:29
I feel like swimming next thing I would like instead of like
41:31
performing an auto fix seriously, if I think was a CSV
41:34
file keep on like daring new data was, it is hospital and
41:38
having new cold and of course, is new considerations that we
41:44
Mr. Teacher account. I love it.
41:47
Yeah, so I'm going to change gears real fast.
41:51
You mentioned in a random tweet that you mentioned actually go
41:57
into. What was that?
42:02
I think it was helpful. I think it was.
42:04
I think it was super helpful. I think what you laid out in
42:07
terms of like, how the problem, An analog to the problem in a
42:15
similar. The analog to the streaming
42:18
problem and how it looks different in the pandas context.
42:20
I thought that was really helpful, so that that was
42:24
totally. That was a great, there's a
42:25
great great, great explanation, but I think this picture is just
42:29
had a different direction. You wanted to take it.
42:32
Well yeah, speaking about that. I like the idea of how streaming
42:36
it is a simple idea or a simple concept but Then once you are
42:42
addicted to a certain tool and the way that you're used to
42:46
using it, and now you want to go put streaming on top of that.
42:50
That's where it can be a little bit more of a headache to try
42:54
and figure out. What, what do we need when we're
42:57
using. Like I had I need my panda fix
43:00
and so if I'm going to add streaming on top of this, what
43:03
does that actually look like? So but yeah, I wanted to jump
43:08
and change gears a little bit because As I saw, I think it was
43:13
a tweet a random tweet or maybe a LinkedIn post of yours.
43:16
And you mentioned, how China is very Advanced when it comes to
43:21
ml Ops and I'm fascinated by what is happening there.
43:24
I have not been able to find anything or anyone that is
43:30
willing to talk about what they're doing in China and I
43:34
want to know. Can you like unpack?
43:37
What made you say that? And and why There's a
43:41
disclaimer. That is a pretty much of
43:44
personal opinion. I have not done conducted a
43:47
nationwide survey. So I do think it's a lot of
43:54
interesting things happened outside the west and I do think
43:58
maybe I'm coming from Vietnam. I do think there's a lot of like
44:01
western-centric media viewport and apply to a lot of things,
44:06
entertainment politics, but also technology.
44:10
So we do see that Michael A lot of countries that have very
44:12
exciting things going on. I just don't really get enough
44:15
coverage. Yeah, so totally yeah.
44:20
So I do think that's like just my personal anecdote.
44:25
I do talk to, I guess it's just like, I just have the impression
44:29
it's hard because we, I know that I got a lot of like, so I
44:35
just funny and reason anecdotal. So in Vietnam in our language,
44:40
we recycle I'm very ink. Her sake in, not very contrite,
44:47
not very precise imprecise. So you setting me up is like, oh
44:52
yeah, a few days in a few days to be like, whenever I love your
44:55
house like tomorrow afternoon, it could be like 7 p.m. and it's
44:58
okay. So so I think that sub is but
45:02
night when I started liking me. Yeah.
45:03
I think as we get very, very precise.
45:06
So that's against it's a good. I guess that is like, that's a
45:09
good thing to learn to like you, but especially when I want to
45:12
convey an idea. Or that try to represent a lot
45:15
of like it is hard to make a generalization.
45:18
But I think I did it just like very personal.
45:20
I get talked to a bunch of companies and they just have the
45:22
impressions as like they Way faster.
45:27
Like a lot of things out. It seems like my friends here
45:30
the different committees. They always so hard to do.
45:33
I was just like, don't really want to go into detail like what
45:36
you much to go detail. I didn't give us the impression
45:38
when I was talking to, like, Engineers the same roles
45:43
companies in China. So, so it could be very much
45:47
like selection by us. But I did look at the sample
45:50
survey. So I guess I'm so bad at
45:52
comparing the day I index in the US and China and also Europe.
45:56
And I think people want cannibal is like the number of research
46:00
paper top conference but also like citations and in that case
46:03
seems like the u.s. like at least a couple of years ago.
46:07
It seems like they was doing pretty well, didn't consider so
46:09
whether but when you talk about like is another, Dimensions is
46:14
adoptions and like initio Doctrine is seemed to me like
46:18
channels seem to have a way longer adoptions and I thought
46:22
about it percentage of company. Is it uses the The Nu AI for to
46:26
a of time because percentage-wise is that survey
46:29
show that mrs. Has higher in terms of like
46:34
industrial. Yeah, actually funny enough that
46:38
you mentioned that I was just reading in today the state of a
46:40
I report that came out from that are Street Capital and he was
46:46
mentioning how yeah. The adoption has been huge
46:48
especially with computer vision and the applications around And
46:54
that. So that's cool to see.
46:56
And this is in China. I want to mention too well, so I
47:02
don't think maybe US government has any program.
47:04
Shouldn't Nationwide? Come division, special
47:07
recognitions. Do we?
47:10
Yeah, I don't I think that would be very hard to get the public
47:15
in the US on board with a nationwide computer vision.
47:18
Facial recognition type of plan whereas in China it's a
47:23
different story. So the other thing that I was
47:28
going to ask, you got to tell us this story because for anybody
47:32
that's listening, chip has an incredible like hundred things
47:36
bucket list type of blog post, I guess is what it would be on
47:40
your website. And first of all, I think it's
47:42
really cool how you are being very transparent about all of
47:47
your goals and you have some just like top-notch goals there.
47:52
I it inspired me one. Them was like, get an ice cream
47:56
named after you which I find incredible.
47:59
I think there would be so many different flavors that you could
48:02
be. It could be real time, chocolate
48:03
chips, even if we wanted to call it that and you can't Yeah,
48:13
exactly. So Rocky Road to real-time,
48:16
that's it, like Rocky Road to real-time.
48:19
That's the best. Bernie sander ice cream flavor
48:24
and does not. The sounds were interesting, but
48:28
it was not my Top Choice. Yeah.
48:32
So just yeah, the thinking around that to have an ice cream
48:38
flavor that you want named after you is, Is super cool and I
48:42
would have never thought about that for my bucket list.
48:45
I'm also very vanilla when it comes to my bucket list.
48:47
I try and think about things that I want to do before I die.
48:50
But honestly like I've done a lot in my life and I'm pretty
48:53
happy and so I can't get that creative on it, but you have one
48:59
thing on there that I want to mention, because I find it.
49:03
So cool. What's the story around being on
49:08
stage with Flo Rida? My Friends, the show it was so
49:21
it was Europe conference. I think I can 2017.
49:25
When the AI weather is cake hype and Intel apparently has Florida
49:31
introducing his a new chip, which is very rare.
49:34
I want to know what was going on in Florida's head, but he went
49:38
to this conference, but they have this, okay, guess what?
49:42
This gets wasn't name. Of the, the parties at the table
49:45
until true. And it's introducing a chip.
49:52
Yes. So the two apart it in your
49:54
ethical or Ida introducing chip yeah do it to us there's a
50:01
gradient flow but we'd like flow spelling fluoride that's really
50:06
good, that's pretty good. Honestly, gradient Flow Rider so
50:11
so that was a story. I think it was there just there
50:13
and I think it was very loud so for us like this, As part of me
50:18
was like, okay and they just like got me on stage and then
50:22
the next day, I went to the dimension.
50:32
My writing that. Yeah.
50:38
So you said earlier that you were writing something but it
50:42
didn't really vibrate with what your it Wasn't your Soul's
50:47
Calling or you didn't vibrate with what you were writing and
50:50
you had to discard it. What was it that you wrote?
50:52
And then discarded, let me find it somewhere.
50:57
Or what was it about? Was it about crypto?
50:59
Was it your new crypto mining? So I think my yeah I do a lot
51:07
of. Did you publish everything?
51:09
You write. No, I'm so much at all on the
51:12
back. Yeah.
51:13
What are some people should be thankful.
51:17
I don't publish everything. All right.
51:20
But what was the percentage of things that you're right?
51:22
Like that's gonna publish 50. Maybe answer 30.
51:28
That's pretty high, I think I was like.
51:36
So I got to set a lower higher bar for myself.
51:38
My bar right now is quite low and I'm just putting out,
51:41
whatever is good. I think I think for me I do a
51:48
lot of free free, right? So it's just like, it's just
51:51
some topic comes to mind and then I never liked it that Mom,
51:54
I said oh my God. This is so interesting.
51:57
I must I write about it and then you know the law that time is
52:00
passed, you got into some other interesting topic and this topic
52:02
didn't seem interesting anymore and you just like did not
52:06
publish it. But I think I have this is like
52:10
chart elsewhere and I did the kid like the secret to write is
52:15
just music publishing before you get more of it.
52:19
So yes, I did because it took me or take me so long to write
52:24
about topic. I just sometimes I just don't
52:27
Skateboard of it before I finished it.
52:30
Yeah, that's so true. And then if you go through and
52:34
you edit it many times then by the end of it, it's also with
52:38
music, I find that where I'll be working on a song, working on a
52:42
song. And my friend that I worked with
52:45
who is a producer, he would always tell me albums, don't get
52:49
finished, they get abandoned. And that is really how it works
52:53
in music because you just at the end of the day you're just like,
52:56
all right, I don't even care anymore.
52:57
More. I can't hear the difference is.
52:59
I don't know what, you know, if you put a little bit more
53:03
compression on that snare drum. If it makes the mix sound
53:06
better, there's just these Minor Details that you go into because
53:09
you're so deep in the forest. And at the end of the day, the
53:13
average listener isn't going to hear the difference.
53:15
And so you just say let's put it out.
53:18
So I fully agree with that. Get it out there before you get
53:22
bored because you have some forcing functions, right?
53:25
It's like, yes, even though you didn't not very Excited about
53:28
any more. I just need to like push it out
53:30
there because you made a contract with someone or
53:32
resource. Oh yeah.
53:36
So speaking of this, where are some places that you learn?
53:43
I imagine right now, you still are learning and everyday, is it
53:49
just learning with potential customers or people that you're
53:52
interviewing out in the field? And I'm guessing that your
53:56
learnings are also a little bit different than what they were a
54:01
few years ago. But where are some places that
54:04
you're learning? Right now. so, It's also depends
54:12
on on the topic. So right now a lot of my work.
54:15
So I need to learn about order stuff video time machine
54:18
learning and like everything around it.
54:20
So a lot of how I do have people that talk it regularly and if I
54:27
bounce off ideas with and put his treatment generous with
54:31
their home and I always feel bad because I was like, hey can we
54:35
catch up quickly? Because I have some ideas and
54:38
want to talk with you about and they when I say ideas and We
54:40
feel like, oh my God I never do anything for you in return.
54:43
So yeah, so yeah. So a lots of a lot of people I
54:48
think over the years I think and so it went to like contact
54:51
people as well. So I feel like if somebody
54:54
writes something just I find it. Oh that is new.
54:56
I did just like is so exciting. They want to learn more.
55:00
They can be reached out to them and try to fight way to just
55:05
talk to that person. Yeah.
55:08
So I think This is the only thing nice about like.
55:14
Logging my writings and stuff because I feel like if they have
55:17
read my post reports and they are more likely to say yes to me
55:20
today. Yeah.
55:21
So I was like, yes, like young, like I was feel give something
55:26
my wrists and my friends is like, if they want to like,
55:29
write a nice, not the reward is not really getting followers
55:34
because frankly, I'm not sure like what force if she hoped but
55:39
like it's like the I don't know what you're interested in and
55:42
then I was like yes, that person.
55:43
Think about his topic. I know what to talks is persons
55:47
about. Okay, socially awkward.
55:49
Like, you may go to a person and I don't know what they're
55:51
interested in. I don't know what to talk to
55:53
that person about like, I don't know, but even know.
55:56
It's like, oh yes. They say right about this and I
55:59
can talk about this. So they could be talk to people
56:05
and I don't envy. Sorry, I'm reading this book
56:08
say, I really like is like, complex adaptive systems.
56:12
It's something very Random thing about like should go into it,
56:16
it's okay, okay? Don't get me started talking
56:19
about this. I can talk about it for hours or
56:21
so that's that is my weak spot expert.
56:26
Well we'll leave that link for anybody that wants to go into it
56:30
and then they can reach out to you and talk to you about it.
56:32
Last question, I've got for you before we bring this home, it
56:36
was one of the most up voted questions when I asked the ml
56:39
Ops Community, what they would like to ask you and it Is around
56:45
your journey on being in and out of Academia.
56:48
And where do you ultimately see yourself?
56:51
What is that going to be near his idea to be professors at
56:56
some institutions? I wouldn't say it's necessarily
57:00
Professor. But yeah, someone who teaches or
57:05
someone, who writes papers. That's what I would think
57:09
Academia or does research. Yang a good research as fighting
57:15
answers to questions, that people may not know the answers
57:19
to before and I think a lot about thing of research is like
57:22
going to produce something. Like maybe you have the answer
57:24
is you write it up as BP revealed as the answers.
57:28
Correct. I do think it's like for what
57:31
we're working on right now is in a pretty new territory.
57:34
So we are like looking for answers to a lot of questions.
57:38
We might not write papers. But we do some sort of like the
57:44
knowledge exchange with other people who care about the same
57:47
topic. So I don't think this is had
57:50
Academia but it's a Brazilian of knowledge.
57:53
Excellent tip. This has been awesome.
57:56
Thank you so much. It's been a long time coming.
57:58
I am so thankful that you came on here and you shared your
58:03
wisdom with us. It's so cool.
58:05
Thank you so much. Thanks for having me and sorry
58:08
about the name confusion. We I really didn't mean to be
58:11
like, We are again and ultimate is really just the like a fart
58:16
like it was really just like we just couldn't think of any new.
58:20
Yeah it's hard. Yeah.
58:23
Well hopefully you know I was I was joking about it and but if
58:28
you did yeah we'll figure out different names somehow.
58:33
Maybe we'll change our name so that people can figure that out.
58:12
We are again and ultimate is really just the like a fart like
58:17
it was really just like we just couldn't think of any new.
58:20
Yeah it's hard. Yeah.
58:23
Well hopefully you know I was I was joking about it and but if
58:28
you did yeah we'll figure out different names somehow.
58:33
Maybe we'll change our name so that people can figure that out.