Latest / Agentic Conversations (formally mlops.community) / The Role of Infrastructure in ML // Niels Bantilan // #197

Transcript

The Role of Infrastructure in ML // Niels Bantilan // #197 · Agentic Conversations (formally mlops.community)

0:00
I am Neil's Bantilan. I'm the Chief ML engineer at
0:03
Union AI and I take my coffee. I make an espresso every morning
0:10
with some soy, basically soy latte, so it's good.
0:17
Welcome back everyone. This is the ML OPS Community
0:20
Podcast. I am your host, Dimitrios.
0:23
And today we've got such a wonderful conversation.
0:26
This conversation is for those real machine learning engineers,
0:33
the people that have been sick of me talking about LMS for the
0:36
past six months. This is the conversation for
0:39
you. Real ML OPS is to be heard in
0:43
the chat with Neils and what a chat it was.
0:48
My Lord. He is such an awesome human
0:51
being. I appreciate chatting with him
0:54
every time we get together. I mean, we did the engineering
0:57
labs AKA hackathons last year with Union and Neils was the
1:02
leader of that whole thing and it was a tonne of fun.
1:07
But in today's conversation, we talked through his creation of
1:12
Pandera, a data quality tool that he has, since Spoiler Alert
1:18
merged with Flight. We also got into using machine
1:24
learning at the infrastructure layer.
1:26
So not like talking about text to Terraform with LMS, more
1:34
talking about predictive analytics and predictive machine
1:39
learning when it comes to running your machine learning
1:43
workloads. So got a little meta in there
1:46
and then we slowly slithered on down and we had to chat a little
1:54
bit about the LM sphere and what's going on with the Open AI
1:58
day. That just happened.
2:01
And later on we talked all about how he looks at these different
2:07
maturity levels and how he thinks about data lineage and
2:13
really reproducibility when it comes to ML.
2:16
And last but not least, I loved that he talked to me about how
2:21
he sees the ML stack and how it will compare to the LLM stack,
2:32
and how he feels that there's still needs to be this whole
2:36
machine learning stack that probably in his eyes is a bit
2:41
under. It's like a little bit further
2:44
down than what you get when you're just using an API from
2:49
Open AI. It's like what is Open AI using?
2:53
That is the XML stack that we talked about today, not what are
2:58
the people that are building on Open AI API.
3:01
No, what is Open AI using? Or what are other companies that
3:05
are doing machine learning using and not getting caught up in the
3:09
hype one bit. Kneels was incredible.
3:12
I loved chatting with him. I hope you do too.
3:15
And for everyone that does not know Neils, because you did not
3:18
take part in our virtual hackathon last year, his mission
3:22
is to help data scientists and machine learning practitioners
3:26
be more productive. So that's why we got him on
3:29
here, man. He's doing work with Union at
3:32
the moment. You've probably heard the
3:35
conversation that we had with Katon, the CEO of Union.
3:39
And if you have not, I would highly encourage you to go check
3:42
that out right now. We'll leave a link in the
3:44
description. Last but not least, a huge shout
3:48
out to Union because they have been sponsors of the community
3:52
since like 2020. I gotta send some love their
3:56
way. It is incredible to see how much
3:58
they're growing and how much love they're giving us.
4:01
So right back at you, Union, Thank you for all your support.
4:05
If anyone is looking for some orchestration layer tooling, go
4:10
check out Flight. You can get that open source,
4:13
They've got an awesome community around it.
4:16
Or if you just want to go and give them your money right away,
4:19
check out Union. That's it.
4:21
I'll see you all in a bit, dude. We, I think we got to start with
4:32
Pandera. What is that and what happened?
4:35
What's the story there, man? Give us the background on it.
4:39
Yeah, let's do it. Yeah, Pandero was one of those,
4:42
is one of those projects that I sort of didn't expect would gain
4:48
any traction whatsoever. It was, you know, it's not the
4:53
most popular thing, but in terms of like within the the scope of
4:57
the project it's, you know, pretty well used and well known
5:01
at this point. But it really literally was a
5:05
scratching my own edge. I was working with data frames a
5:08
lot pandas, you know, I I wasn't doing anything super big with
5:13
the data set size wise, but encountered all sorts of data
5:18
type errors. Date, date, time formatting,
5:22
negative numbers where there shouldn't be null values where
5:25
there shouldn't be. So because pandas and other data
5:30
frames right now they're it's kind of changing a little bit
5:34
with polars, but more on that later.
5:37
But yeah, with pandas it's powerful, it's dynamic,
5:42
dynamically typed, and it does some surprising things sometimes
5:47
when you have like a long chain of transformations and you can
5:53
easily get lost and not know what the data looks like
5:56
anymore. And Pandera was this tool I
5:59
built for myself to just help with that OI can.
6:03
In my code I can just see OK, there's a function that outputs
6:08
a data frame that follows this schema and as like OK, what does
6:13
the schema look like now? I could just go to the schema
6:17
definition. I can see here the column names,
6:20
here the data types. These are the kind of check kind
6:24
of constraints and range values and things like this.
6:28
So I can just see it at at a glance.
6:30
And before that I would have to load the data into memory, do
6:36
some profiling just like do a dot describe and you know, use
6:40
pandas profiling and all this. Stuff.
6:43
You know, which I think is still a useful activity activity.
6:47
You still need to like grock your data somehow to write the
6:51
schema in the first place, but it's just a way for you to
6:55
encode that and then put it in your code base.
6:58
Dude, it's so it's so cool because it's one of those things
7:01
that you probably were like, alright, this is mine, I'm just
7:06
gonna, you know, put it out there in case one or two other
7:10
people feel like they would find use in it and then all of a
7:15
sudden it kind of, I mean, relatively blows up, right?
7:20
Yeah, slow. Yeah.
7:22
I would say modestly slow growth right over over the span of what
7:28
it's been six, 5-6 years now. And like every time I go to a
7:35
conference to talk about it, there's like a little bump and
7:38
then it kind of flattens out and goes, it goes to like the the
7:42
kind of ambient organic growth and then another conference and
7:45
then another confer. So kind of the story, I'm still
7:49
working on the story to be honest.
7:51
It's like sure it's the core is data validation, but now this
7:54
scope has sort of expanded and now we're working on a Polar's
7:59
integration. I worked with the Quantum Black
8:03
team at McKinsey to implement a Pyspark integration.
8:07
So that's out. That took like two months, yeah.
8:12
But to be clear, this is not your day job.
8:16
If you are not getting paid, this is full moonlighting right?
8:19
And you never thought about creating a company out of this?
8:23
I did and we actually. So maybe a slight correction is
8:28
that I kind of I merged Pandera into Union.
8:31
Ohh OK. So that's fairly recent news.
8:35
We didn't make it too big of a deal out of it, but to answer
8:40
your question, yeah, that that was on my mind.
8:43
Even when I joined Union was OK. I have this project and the the
8:49
company building thing came across my mind but whichever way
8:55
like I didn't want to go through the VC route and not have a
8:58
clear business model right. So I wanted to kind of think
9:02
through it and every which way I would think of it and having
9:06
talked to some customers, users I guess, but I would say
9:10
customer would be someone using it in production at a company,
9:14
right, not like a hobbyist. Like I would talk to them and I
9:17
would ask you know, So what? Like I have XY ideas to
9:24
productize this and I'll monetize this and every single
9:27
time I heard. Like I don't want you to touch
9:30
my data. You know, I I want Pandera in my
9:35
cloud generating error messages and error reports that I control
9:42
and like, OK, maybe we can send you some metadata and you can
9:45
visualise it and I'm like, OK, that's an interesting platform
9:51
idea, but generating reports is not.
9:55
That didn't seem like a fun thing for me to build right?
9:57
Like creating a web UI, would like visualisations on error
10:02
reports like code come you know for data, right?
10:06
Not where you were putting on, not where you were put on this
10:09
earth to do, I'm guessing. Yeah, it it just wasn't exciting
10:13
to me. So every which way I thought of
10:16
it, like the cool stuff that I wanted to do was kind of
10:19
tracking data, data quality overtime for like a specific
10:25
data set. And like the only way really to
10:30
do this effectively is if you have a product that people
10:35
install, hint, hint, flight, right.
10:38
The people install on their cloud that they own that they
10:41
feel secure about and you generate these reports that you
10:46
can then you know as you're collecting data shift or like
10:52
how your data evolves overtime and like the nature of your
10:55
errors, you can sort of like take that and do some more
10:59
useful things with it. You know, so you can think of
11:02
like yeah, data set shift things, things like problems
11:06
like this. So it just seemed like I talked
11:09
to Caitlin, you know and basically we merged Pandera into
11:14
Union like earlier this year, maybe end of last year, but so
11:19
now it's it's not an it's kind of my full time job now, but I'm
11:24
still sort of like one of the sole maintainers of it.
11:26
So you know we're we're ramping up, we're a startup, we're gonna
11:30
invest more in it very, very soon actually.
11:33
But right now it's me working with like our open source
11:38
contributors. You were Aqua hired, but exactly
11:42
free Aqua hires. You're already working for them?
11:45
Yeah, then you just figured out a way to make that work, and
11:48
that's so cool. Talk to me about the integration
11:52
with Polaris, more polar sort. Bowlers, yeah, yeah.
11:57
I put a little bit of a French twang, Larry, right there.
11:59
Yeah, you see that? It's a global.
12:01
You know, school Laura is no yeah.
12:04
Talk to me about the integration with pollers.
12:08
So that suggests started. There's a lot of interest in the
12:12
community. Umm, I think it's the issue with
12:15
the mouse. Thumbs up right now.
12:17
Yeah, So the way this has worked in the past is either I've
12:22
started, I've gotten the ball rolling and then I kind of like
12:26
gather A-Team of interested people and then they'll work on
12:30
it. So that's how this is turning
12:32
out. Other times like some person
12:34
like the kind of identic style syntax with data frame model
12:39
that was completely done by someone and he just Jeff Zank
12:44
have to give him credit each time because he was like the
12:47
first major contributor to the project one of the first.
12:51
So yeah just for the Polaris integration it's really fun.
12:55
I've never, I hadn't actually never used polars, but.
12:59
I don't know. I just kind of seen it and I saw
13:01
like the performance, you know differences and like the syntax
13:05
looked pretty cool. So when I started playing around
13:08
with the integration, it's it's an interesting programming model
13:13
that's substantially different from pandas in a way, but it's
13:18
familiar enough where I think you know a pandas user can kind
13:21
of hop in learn a few new abstractions like lazy lazy
13:26
evaluation. You know the lazy frame is a
13:28
super powerful so. What is that exactly?
13:33
So unlike a data frame, you say you know pandas dataframe dot
13:39
assign X = 2 and you get a new column X = 2 like and when you
13:45
execute that code it's going to add the new column.
13:49
But with lazy frames, it's essentially you're building like
13:53
a graph of transformations that aren't evaluated immediately,
13:59
but when you call date dataframe collect it's it actually
14:05
compiles your transformations and makes it more efficient,
14:09
which is one of the reasons why polars is like much more
14:12
performant in many cases. What a great name too.
14:16
It makes total sense after you explained it.
14:19
It's like of course the lazy lazy friend dude.
14:22
There is something that I wanted to mention.
14:24
Going back to this data quality piece and tracking it over time
14:28
and something that we were talking about before we hit
14:31
record was like, ohh man, isn't there ways that we can start
14:36
injecting ML into our infrastructure?
14:41
And it feels like if you have this data on how your data is
14:47
changing and transforming or the errors that you're getting
14:50
overtime, that gives you a bit more to play with on on your
14:57
side. And maybe it's, I don't know
14:58
whose job it would be if it's like you as Pandera slash flight
15:04
are going to take on this challenge or if it should be us
15:07
as the users that have all this data.
15:10
And we're going to take on this challenge to just have it be
15:14
more like we can do cool stuff with this extra data that we're
15:20
getting. Because now we have overtime
15:25
we're able to see what's going on and then maybe we can figure
15:29
out a way to leverage that. Yeah, I think in terms of
15:35
responsibility, it is on the orchestrator layer to collect
15:42
that data in some structured way, semi structured way.
15:46
That again, I don't know. It'll probably be kind of like a
15:50
plugin or some kind of side thing that the orchestrator does
15:55
to take that patterns and that data.
15:58
For example. Like this particular workflow
16:02
tends to error out a lot with particular sets of inputs or
16:06
data set sizes, or this other one like has these weird kind of
16:13
memory spikes, right? And you don't really.
16:18
Maybe in those simple cases need ML to really surface that it.
16:22
This is like an analytics thing. You like collect it, you alley
16:26
some set of like deterministic rules on it, and then you
16:29
surface that to the user. So I think that's that's
16:33
probably kind of like MVP for this kind of abstraction on top
16:38
of an orchestrator to to give the user of that orchestrator a
16:42
lot more visibility into what's happening.
16:45
But I can imagine much more complex things that you know.
16:51
You want to summarise errors, you want to do other things and
16:55
surface patterns that are not easily captured with kind of IFL
16:59
statements. In that case, I think the first
17:04
step is just to collect the data and then there's a tonne of
17:07
research, I think. I don't know, this is not really
17:10
the shiniest or sexiest thing maybe.
17:13
I don't know if people are going to work on it, but if there is,
17:18
if there is a business value to it, IE like less cost or you
17:22
know some kind of savings, then there there should be enough
17:25
incentive for like Union or another company similar to it to
17:29
work on it. Well, I guess at the end of the
17:32
day, it's like if you can better predict the future on the jobs
17:36
or the workloads that you're about to run, that has a
17:41
benefit, right? And there's some value in that
17:44
if you can save yourself time in any way possible because those,
17:51
especially if you're starting to deal with GPUs, it could be why
17:57
costly if all of a sudden you're you're not saturating the GPUs
18:02
like you want to and you didn't realise that beforehand and so
18:07
now all of a sudden your job is not being fully utilised.
18:11
It would be great to know that, like, before I queue that up,
18:15
that's about to happen and you get those warning messages.
18:18
Like, hey, by the way, we've seen stuff like this in the past
18:22
from you and it didn't turn out too hot.
18:24
Or your friend over here has been doing this all afternoon
18:28
and it's just not working for him or whatever it may be,
18:32
figuring out how to better predict the future in a way.
18:36
And that's why it's funny because when we were talking
18:38
about it and you were like, yeah, how do we inject ML into
18:42
these different infrastructure problems or just into the
18:48
infrastructure layer? I was like, what do you mean
18:50
like text to infrastructure? And it's like no, even cooler
18:54
than that would be the same way that you predict or that you can
19:00
try and predict what someone will like when they buy five
19:04
items on Amazon or when they watch 5 movies on Netflix.
19:08
Can you predict what will happen next if you have all this
19:11
information and all this data about the workloads that they're
19:14
running? Yeah, exactly.
19:17
The current state of things is that it's kind of, you know, you
19:23
lick your finger and put it to the wind and you're like, OK,
19:25
what do I need in terms of resources for like a gig of data
19:30
with a 1 billion parameter model or something like you can do
19:35
some back of the envelope calculations of what exactly
19:38
exactly you'll need, but the rubber meets the road when you
19:42
you have to execute it. You have to know if it fails or
19:45
succeeds. And this is all this is kind of
19:48
like a reinforcement learning signal, right?
19:50
It's like you have a binary success failure signal, you have
19:55
some inputs and you know you have a pretty structured data.
20:02
Structured kind of grammar for how you specify the resources,
20:06
et cetera, et cetera. Not suggesting you apply an RL
20:09
to this, right? But I'm just saying it's
20:11
classification, the simpler framing of it.
20:14
But this is something that we're interested in because for on the
20:22
flight and more I guess specifically in the Union side,
20:25
we have built these tools for you to visualise utilisation.
20:29
You can set your limits and quotas and we kind of show, OK,
20:33
here's your quota, here's what you're using.
20:35
You know, like you could stand to increase, you know your batch
20:41
size for example, but that from a practitioner perspective it's
20:46
all it's. It's like I just have to push
20:49
the button. You have to like run the thing,
20:51
fiddle around with it and then see what works.
20:54
So once you have enough users and you have enough scale to be
21:00
able to say to collect all of this these mappings from like
21:05
what workflow code resource specifications to whether it
21:10
succeeded or failed on a particular set of hardware, then
21:14
you can start doing some interesting, at least at the
21:17
research level, seeing if there are any patterns there at all.
21:20
And I'm I'm guessing there will be.
21:22
I mean it's it's pretty obvious to me, like if you have a string
21:27
in your code that says code llama 7 billion, well your
21:32
model. If it's betrayed on enough data,
21:34
it should be able to say, OK, so I see a 7 billion thing there,
21:39
like I'm going to need at least 7 billion, you know, like worth
21:45
of memory. And that should map them to some
21:48
specific hardware, et cetera, et cetera.
21:51
We're not there yet, obviously, but that's something we're
21:56
thinking about, that I'm thinking about.
21:58
And you know, we're we're collecting like everything in
22:01
from a metadata layer, so we won't necessarily have access to
22:06
the code of, you know, users or customers, but it is an
22:12
interesting application area that I don't read about that
22:16
much. And it feels like your feet are
22:18
on the ground because a lot of times I think you get people on
22:23
Twitter or people that have become influencers talking about
22:27
how AI's gonna change everything.
22:29
And this is just the start, But it's like, OK, here's an actual,
22:33
like, real use case that could be quite useful.
22:37
Or maybe not. I think that's the fun part is
22:40
like, let's go validate if A, this is possible, which it seems
22:44
like it is, just back in the napkin type thing, and then B,
22:49
if it actually adds that much value to where people are like,
22:53
Oh my God, we need this, right? And so I think the interesting
22:59
piece is, yeah, how you do it, how you make sure to get all the
23:04
information you need. But you you're mentioning like,
23:08
alright, we're not gonna have access to a lot of this stuff,
23:11
so then it it becomes an even harder problem.
23:14
I would imagine from your side like you gotta kind of
23:17
guesstimate things or just fill in the blank.
23:21
Yeah. I mean I think probably from A
23:26
features perspective, the probably the most important
23:29
thing if we're thinking just ML workloads or maybe data
23:33
workloads is data set size and model size.
23:36
I imagine that will kind of be the bulk of the predictive
23:39
power. And then there will be some
23:41
other features there that would have kind of the the margins
23:45
adjust the what the predictions are.
23:47
But yeah, I know stories from people who worked at Meta, their
23:52
internal platform FB Learner, hadn't features like this,
23:56
right? But it it was more algorithmic.
23:58
It was like try this and then if that fails, double it.
24:02
And if that succeeds? Like cut it in half.
24:04
You know, you know, cut it by 75% and if that fails, like
24:09
increase it slightly, you know, so that it may turn out that's
24:14
the best we got. Yeah, but it's interesting.
24:18
I mean, ML is is all about empiricism, right?
24:22
It's like you just gotta try it. Yeah, yeah, yeah, totally.
24:26
It is. It is funny because it's like,
24:28
man, like you were saying, it's not the sexiest of things, but
24:31
I'm having a great time nerding out about this because it's
24:35
like, whoa, that would be. That's a totally open area.
24:39
And it feels like that. That can definitely happen.
24:43
It's not like it's out of our grasp or our abilities right
24:48
now, right? I think if you're telling me or
24:51
if you're talking to me more about like this idea that I was
24:54
telling you like the text to Infra, which I know there are
24:57
people that are like generative DevOps.
25:00
Yeah, there's all generative DevOps movement.
25:03
Have you explored that corner of the Internet at all?
25:07
Not, not deeply. I've.
25:10
I've seen the term and I've seen like the this kind of the demo
25:16
of you say I want such and such infrastructure and it like
25:20
generates A Terraform file or something and it makes sense,
25:24
like that application does make sense to me I think.
25:29
So yeah, I'm curious to see where that goes.
25:33
Yeah, exactly. I think that's one though that
25:36
I'm much more sceptical of, I guess just by nature, because of
25:42
the times that I struggle just with ChatGPT in English, right?
25:50
And English is it was trained way more on English I'd get the
25:54
feeling than on Terraform, and so I wonder how well that does
26:00
and if it doesn't just end up causing more headaches than it's
26:05
worth, right? Like you're like yeah, check out
26:08
this cool thing. I can just like you talk to it
26:11
and then it'll give me Terraform and you end up talking to it and
26:14
all this like code. If it's like half English, half
26:18
code type things that you have to know to specify this and
26:22
that. And at the end of the day you're
26:23
like, wouldn't it just be easier to get a really good plug in on
26:27
them and do it yourself? Yeah, I think that's it's the
26:33
sign. It's the sign of a a a boom in a
26:36
space when you just try everything on it and then move
26:41
five and five, three to five years, we'll have like as much
26:45
smaller set of things that turned out to be useful.
26:50
Yeah, DevOps, LLM kind of generative DevOps is
26:53
interesting, but as you as you point out, it's the the risk
26:59
there is like complacency almost.
27:02
Yeah, I mean, I think of doing right, like the the story of the
27:08
thinking machines and how there is like this whole war kind of
27:11
like between humans and machines, not in the Terminator
27:14
sense, right? But it's like in the original
27:17
books. It was about how we outsourced
27:21
reasoning to these thinking machines and that's why I don't.
27:27
I'm not anti these new developments, but it I think
27:32
there is there's something to we get things get normalised and we
27:38
get lazy and I will just accept a generated piece of code.
27:43
Maybe not me, because like I'll look over it, but I can see
27:48
myself if I use it enough and it's like accurate enough that
27:52
like I'll just completely outsource my judgement to this
27:56
thing. And even though like the
27:58
creators of it are like, ohh always, always put your wheel on
28:01
the car right? So it's like, always look, read
28:04
your code, but that's not how humans work.
28:07
And so yeah, it's it's a kind of a balance of you know,
28:14
leveraging it for useful things and then design like how are we
28:19
designing these things so that we mitigate the risk in this
28:23
case of you know, outsourcing judgement and reasoning.
28:27
Yeah, I mean you just it's basically pushing it to the
28:30
limit. Like, how much can I do this
28:33
until it comes back and bites me in the ass?
28:36
Yeah. I'm gonna keep accepting these
28:39
suggestions until then. Ohh like I just shipped that to
28:44
prod and nobody caught that there was this huge
28:47
vulnerability or nobody noticed that like I crashed everything.
28:52
Yeah, and the thing that concerns me too is the way
28:56
people are thinking these days are.
28:58
I'm sure this is not everyone, but like a thread that I've been
29:02
seeing is like ohh, LMS have introduced some new problems.
29:07
Well, let's solve that with looms, right?
29:09
So like, let's have a review bot looks, let's have a bot that
29:13
writes the code, and then we'll have another bot that's
29:16
specially trained to review that code.
29:18
And. But there's this infinite
29:20
regress to. Like at the end of the day,
29:24
you're either. You either have a have like a
29:28
person behind the curtain kind of like needing to vet these
29:32
things, or you just completely let it run and see what happens,
29:38
which that that's not. I don't see that being a pattern
29:43
for serious companies anytime soon.
29:46
It's funny to the that idea just always reminds me of the exhibit
29:51
meme. Like yo dog.
29:52
I heard you like cars. Yeah.
29:55
We put a car in your car, It's like I heard you like lol.
29:58
So we made an LM that reviews your LMS on your L so it's so
30:05
good. But dude, I mean, not to get too
30:07
caught up in the LM hype, but I know that yesterday it was
30:13
basically every LM lovers dream because Open AI had their dev
30:21
day. And so how do you feel like what
30:25
you're doing right now and the work you're doing in the
30:28
orchestration layer, like fits into all of this with the stuff
30:33
that just came out yesterday? And we'll time, we'll timestamp
30:38
this as recording it on November 7th, right?
30:42
So a day after day. Yeah, I reviewed sort of the
30:47
presentation on what was released, I think in the context
30:51
of my work and the work at that union.
30:55
In a way, I think it's sort of orthogonal.
30:59
Like the people we talk to, they are not generating tokens.
31:06
Umm. They are processing video for a
31:13
very specific use case that maybe in 100 years, 50 years, 25
31:19
years. Islam can kind of code those
31:21
problems, but I think still there will be a, you know,
31:27
software 1.0 layer, right? That is like the deterministic
31:31
stuff like the good stuff of like.
31:33
Just good old programming that just like works.
31:37
And it's item potent. If you want it to be item, item
31:40
potent and it's like, you know, so there's this fundamental
31:43
layer that I see myself kind of operating in.
31:48
This is Skyler. I lead machine learning at
31:50
Health Rhythms. If you want to stay on top of
31:53
everything happening in ML OPS, subscribe to this podcast.
31:57
Now. Now.
32:00
Now. Now.
32:04
Now. Now.
32:08
Now. And from my perspective, the
32:13
Open AI Dev Day was sort of like, to make an analogy, it's
32:17
like the a store with the iPhone, right?
32:18
It's like here's like software. I don't know how many point OHS
32:23
it is, but it's like here's this like new compute platform that
32:26
operates on tokens. And LLM is like a virtual
32:29
machine that you give a tokens and if it can understand those
32:33
tokens, it will spit out some more tokens, right?
32:37
And those can be instructions that are function calls to other
32:40
things, you know. So it's a very interesting like
32:42
paradigm, but I still, I think for the foreseeable future there
32:48
will still be people making survival models, people training
32:51
models on tabular data, computer vision like image and video
32:57
data, audio data, like text data where all you need is like a
33:03
sentiment prediction or something.
33:05
And maybe you can use all for that.
33:07
But you know, they are building kind of these bespoke systems
33:11
that maybe speaking too soon here.
33:14
But yeah, I think for the foreseeable future you will
33:18
still kind of need this. You'll still need humans in this
33:23
kind of like lower layer where you're not like delegating a lot
33:28
of the work to a machine learning model, cause that
33:32
that's kind of the thesis, right?
33:34
It's like if a with a powerful enough LLM, you can have it
33:39
code, you can have it be your CEO, you can have it do
33:41
everything right. But that's that's kind of the
33:45
vision they're selling, which is interesting.
33:50
I think people will create a lot of interesting applications on
33:54
top of it, but in the context of of my work, it's, as I said,
34:00
orthogonal. I think like there is still the
34:04
question of open source versus closed.
34:06
And like, I think their narrative is a lot more centred
34:11
on this kind of AGI vision and things like this, whereas I
34:16
prefer to work in this less speculative space.
34:20
And that kind of means, hey, what does a customer need today
34:24
or in a week? Yeah, I think that's why we get
34:29
along so well, man. Sometimes when I hear people
34:32
talk and say the word AGI, I kind of mini vomit in my mouth.
34:37
I just like, get the fuck. Out.
34:41
Yeah, so it's but at the end of the day they're they're like
34:45
using Ray, right. So you know, yeah, I guess shout
34:48
out to Ray. But like Open AI is still using
34:51
the tech stack that we all know and love to build their their
34:55
stuff, you know, so. Their AGI?
34:58
Yeah, that's true. Then then AGI will like, have
35:02
will will know numpy and like love numpy because that that's
35:06
that's like the ancestor of it's whatever the age.
35:11
That's so funny. AGI will run on Quadrant and
35:14
Ray, and so if you ever want to fuck with it, just merge some
35:20
bugs into those open source projects.
35:24
That's so funny. And I mean that is true.
35:27
It's like there is this base layer.
35:29
You go a little bit deeper down the stack, almost just like when
35:35
we were talking ML OPS from like 2020 to 2023, before the boom,
35:43
there was this stack and this lifecycle that machine learning
35:49
played in, right? But then you could go down the
35:51
stack a little bit more and all of a sudden you're encountering
35:55
Kubernetes and you're encountering, hey, are we, how
35:58
are we networking these nodes and what are we doing?
36:01
And it's more like the Sr. DevOps type persona.
36:04
Yep, that interfaces with the ML engineer, but they're not.
36:09
It's orthogonal, right? It's a little bit further down
36:11
the stack, and then you can even continue down the stack, right?
36:15
And as far as you want. But the there is.
36:19
Yeah, you almost you through something on top which now has
36:26
its own stack. Almost.
36:29
Now, when people interface with LMS, there's a LLM stack that
36:34
you have, and under those LM's are still.
36:38
There's still a whole stack to be utilised and used there, and
36:44
so I wonder about how the separation there then becomes.
36:51
Because like the orchestration layer for what you're doing is
36:56
one thing and I imagine you're still encountering limbs on the
37:01
daily basis, right? Like people are.
37:03
Maybe it's open source, Maybe I would.
37:07
Imagine, yeah. A lot of open source, if they're
37:10
using flight and they're adding it as part of these different
37:14
workflows or what they're doing like, how does that look?
37:18
Because you have a stack that is the LLM.
37:23
LLM also stack, but then you also have these millot workloads
37:29
or the orchestration that you're doing.
37:31
Yeah, so. I I'll.
37:35
I'm. Kind of.
37:35
I've dragged the Kool-aid. Basically the the flight.
37:41
Kind of philosophy that. When it comes to machine
37:44
learning. Models and MLM's are just quote
37:48
unquote ML models, right? AI is kind of like a branding
37:53
kind of thing as far as I'm concerned, but ML models?
37:58
It's like a trifecta. Of model.
38:01
Architecture. Data set and infrastructure,
38:04
right? And I think.
38:06
It all. I feel like the evolution of
38:09
this space, in this, the evolution of the space
38:12
infrastructure was less kind of emphasised and I'm sure there
38:16
are organisations where it was. But like, at least from my
38:19
perspective, right, you start with, it runs on my laptop.
38:23
We don't. We don't.
38:25
We're not training billion parameter models.
38:27
It's like, here's K learn Xgboost, here's some tables.
38:32
I'm just going to train on it and it works on my laptop,
38:34
right? So the Explorer.
38:37
And it will. Work on pretty much everyone's
38:39
laptop except for the times where your virtual environments
38:41
don't wake up. Hence the tooling around, you
38:44
know. So we're already getting into
38:46
the infrastructure a little bit where like the whole it works on
38:49
my laptop main kind of came from but as.
38:53
As we're. Scaling the models.
38:54
As we're scaling the data sets now, it's now infrastructure in
39:00
this trifecta is becoming more and more important, and I
39:03
remember. The first.
39:04
Time I encountered infrastructure as code like
39:06
terraform. Core like a WS cloud.
39:09
Formation it was pretty cool because all I do is but write
39:14
some Jason or yaml and they're they're problems there I guess
39:17
as well. But like it's declarative, I say
39:21
what I want and like there is process that figures out how to
39:26
like do all these things with what I've declared.
39:31
So. You know life is good when
39:34
you're an ML engineer at a fairly new kind of ML practise
39:37
at an organisation. You're like.
39:40
Do a query, SQL query. Now you have your little data
39:43
set, you play around with it, OK, it looks pretty good on the
39:46
test set. So then you like spin up a
39:49
little Flask or fast API server and then that's how you deploy
39:52
it. But at some point you will
39:55
encounter all of these issues and that's what these point
39:58
solutions are now like, you know, experiment tracking
40:02
weights and biases, Mill flow, scalable data transformations
40:06
like you have Dask, Ray, Pyspark, you have experiment,
40:11
tracking, model, observability, data observability, data
40:15
quality. But how do you like put all
40:18
these things together? And that's where an
40:21
orchestration layer is really helpful.
40:23
Then I would say with LLM OPS it adds a few more pieces to like
40:29
the software kind of system. But it's fundamentally it's it's
40:35
like you're adding some information retrieval with BRAG,
40:39
you're adding a few little pieces that we actually know
40:42
pretty well and these are pretty mature spaces.
40:45
But now you just you know think about vectors and you know
40:49
similar cosine similarity or other kind of like distance
40:52
metrics instead of like, I don't know, keyword matching or key
40:56
phrase matching. So there there's.
40:58
Like a? Different spin on it, but at the
41:00
end of the day. Then this is what we're seeing.
41:04
Is the more and more mature and organisation is and the more and
41:08
more pain points they've felt in this journey, the IT correlates
41:13
positively with their uptake of flight because they're like ohh
41:17
Kubernetes. OK, it's like maybe not the
41:20
easiest thing to work with unless you're like an expert,
41:23
but like I'm gonna invest because I can see the value of
41:29
lineage and reproducibility. So I think LM.
41:35
OPS. Specifically will undergo
41:38
similar sort of growing pains because right now it's like
41:42
pretty Wild West and sort of the standards are being created and
41:47
you were borrowing from different spaces.
41:49
But I don't think these kind of production requirements will go
41:55
away, like reproducibility for example, or resource kind of
41:58
isolation and things like this. It is a fascinating 1 to think.
42:04
About reproducibility when it comes to using these LM's,
42:10
because that is basically all thrown out the window, yeah.
42:16
Yeah, exactly. It's.
42:18
I mean, I think interestingly enough, they did show in the dev
42:21
day yesterday. Um.
42:24
What is that like? You can pass in a seed parameter
42:27
now so you know you you do get some item potency there.
42:31
Yeah, so. The but that kind of kind.
42:33
Of like, case in point, right? There has been the demand to be
42:36
like, hey, I want the same output if I gave it the same
42:39
input, yeah exactly. So and it's funny too, like this
42:44
idea that you're talking about on how the.
42:48
Companies that are the. Most mature are the ones that
42:51
have been through and gotten their bruises and their
42:53
scratches. They're like, OK, I see the
42:56
value here and I understand, yeah, Kubernetes is like I'm
43:02
probably on it and I understand the workload that it needs and
43:08
then light it gets me there because it has that bulletproof
43:14
Ness and it has that like it's a workhorse and it will do.
43:19
It will scale with me because I'm already at this certain
43:22
scale and you're not looking to again, you're not looking to the
43:27
future of like this high flying, hey, are we going to have AI
43:33
take over the world or it's going to take over our company?
43:37
It's just like, no, I got to do this stuff.
43:39
I said that I would complete this task by the end of this
43:44
Sprint and so I need it done now.
43:47
You know I can't be spending messing around with seed
43:50
parameters and figuring out how I can reproduce this, right?
43:54
Yeah, exactly. They're.
43:56
They're the constraints, right, of time and resources.
44:00
Where in the longer time horizon, you know, maybe to play
44:04
devil's advocate of my and maybe your general feeling is like,
44:08
yes, long enough time horizon, we will have amazing things that
44:12
we'll do, things that we can't even imagine.
44:15
But my time horizon is in a month.
44:18
So like, like these, these sort of more aspirational visions
44:26
are, I think, useful to have. But practicality and and
44:31
pragmatism kind of like is a one of the values that I, I sort of
44:38
have on top of over things that kind of get me excited you know
44:43
and and it's it's important. I think you need to kind of a
44:46
balance of inspiration and just like being real.
44:50
Yeah, that's I think the. Saying that that.
44:54
Kind of confirms my stance on this is just like the open
44:58
source flight users, the Union Cloud customers we're bringing
45:02
on, they all have very thorny like.
45:07
Workloads. And very specific things that I
45:11
don't think you can really automate.
45:13
The things we can help with are all of the things that I just
45:15
mentioned, right? Like your your job is already
45:18
complicated enough. I just want to make sure once
45:21
I've gotten it a working state that it's gonna work next year.
45:27
Yeah. What are some?
45:28
Of the things. That what are some of these
45:30
challenges that you're seeing I'm I'm.
45:32
Gonna try to be careful. Here because we we haven't
45:35
really announced too much, but some of these challenges are
45:40
around iteration, iteration, speed and making it feel like as
45:47
a data scientist or mill engineer, I can kind of
45:50
seamlessly go from my local laptop to whatever resource that
45:55
I want. Not like some opinionated
45:57
platform, but could be spark, could be data bricks, could be
46:00
ray, could be union or flight cluster, could be something
46:04
else, right? So we are solving for these
46:07
problems and trying to make flight and flight kit the SDK
46:12
just more useful by itself just to make.
46:16
It feel as if. Hey, I'm I'm on my laptop and
46:20
whatever I can do on my laptop will run on it.
46:23
But I can delegate to these other cloud.
46:27
You know these other products as needed based on where my company
46:31
is at and based on where my companies tooling and like what
46:35
has been approved as like the yeah the vendor, right.
46:38
So there are all these like real world constraints and at an
46:41
organisational level that. You may not want to.
46:45
Buy into something too opinionated and that's that's
46:50
sort of like one of our operating philosophies is yes
46:54
flight is opinionated but it's there is escape hatches and
47:00
things you can do to customise if you need like all these other
47:04
kind of tools in your tool belt that you're already familiar
47:07
with. So sorry if that's it's is that
47:11
if that's a little vague but we are solving for this sort of
47:14
like. Problem of scale.
47:17
And like developer experience, where it feels like you're just
47:21
working on your laptop. Yeah, it makes.
47:24
Sense it's like this idea of not letting flight get in the way of
47:29
you wanting to do something. Right now I have access to a B&C
47:35
resources. I'm going to leverage flight to
47:37
do that. Not like AH Flight only allows
47:41
me to do A but not be in C, right?
47:46
Because it's so opinionated and it will only have that be my
47:51
choice. And so you get those, those
47:54
integrations and you get that flexibility.
47:57
So that that makes complete sense.
47:59
I could see that, yeah. And and a related.
48:02
Problem is you know, like say. I'm iterating on.
48:06
Something and I'm like kicking off all these executions, Um.
48:12
On local and on flight as. Well, and I'm producing all
48:15
these artefacts, right? So that's data, it's models,
48:18
it's metadata, yeah. And today Flight is good at
48:23
encapsulating data. Lineage is like, OK, I have my
48:26
workflow and I have all these nodes and when I execute it,
48:30
flight will keep track of that in that kind of workflow in
48:35
isolation. But what if, like, this workflow
48:38
produces a data set, and then I have a completely separate
48:40
workflow here that consumes it, right?
48:43
Ohh, that produces a model. I don't want to like strongly
48:46
couple those two, but I do want to track that artefact, yeah?
48:51
As it's travelling through the system in various different ways
48:55
so that's another problem we're solving for.
48:58
Well, we'll announce it when it's when it's ready, but it's
49:02
it's it's a pretty exciting development that's gonna happen
49:07
pretty soon. So, well, it goes back to that.
49:09
Reproducibility. And it's funny, man, because
49:12
back like when I started this community, the company that I
49:16
was working for was an Milos company, right?
49:19
And the main thing that we did was this reproducibility.
49:23
So wherever your data was, wherever your code was, we were
49:27
able to track it and make sure that whatever happened to the
49:31
data, you knew exactly what was happening to it and where it
49:34
came from and what it got partitioned into.
49:37
Whatever that may be. However it was moulded, nobody
49:42
fucking wanted it so. The company went out of
49:45
business. Shit, yeah, that's why.
49:47
That's why. I'm doing the community.
49:49
Yeah, that's the. Plot twist.
49:51
So I hope, I think and my theory is that A, it was way too early,
49:59
right? Like this was 2019 that we were
50:02
trying to sell this product to people, and people had a lot
50:05
bigger issues in 2019 when they were trying to do machine
50:08
learning. And then B, even if you did have
50:13
people that were at this stage that they were ready for it,
50:19
they probably had been building their own tool or they were just
50:22
big enough to where it was, like, the maturity wasn't there.
50:28
You know, Like, they didn't care that much about it because they
50:31
hadn't felt these pains of years of pain.
50:36
It's like maybe they had just gotten to that maturity level.
50:40
And so they kind of didn't know that it was actually a pain.
50:45
And now we're whatever, three years later, almost four years
50:48
later, people are probably waking up to the fact that like,
50:52
oh, it doesn't need to be like this.
50:54
Yeah. No I I.
50:55
Totally understand that kind of story and journey of, you know,
51:00
the people you were, you know, selling to people you were
51:03
talking with. And it's it's unfortunate,
51:08
right? Because the first and foremost
51:11
the saying that people want is for this their system to work.
51:14
Like I just want it to work. I don't care.
51:17
Just pile on the tech debt. I'll just use my tech debt
51:20
credit card as much as I can to even get something that.
51:25
For you know, ideally it provides business value.
51:29
Or less ideally, like get your leadership team excited and
51:32
executives and investors kind of hyped up, right that that's kind
51:36
of like less of a. Ideal case.
51:40
But you know, whatever, like political acceptance or and like
51:44
you know, like there there is the the that angle to machine
51:49
learning where if you're not an ML first company, it's like oh,
51:53
you have to prove yourself right.
51:54
So like to prove yourself you need to get that.
51:58
Working model. That whatever metric metric you
52:02
choose, like, works right? It's accessible.
52:05
And then you think about all these.
52:07
Things like, Ohh, I have a massive like, credit card debt.
52:13
Yeah, here and. I need to fix.
52:15
It and whatever platform and tech stack you choose to and
52:20
that first phase of your journey.
52:23
That you're kind of like you're either.
52:25
Stuck with it unless that platform adds these features or
52:30
you have to migrate, right. And so that's it's, it's sort
52:35
of, this is one of my big learnings, sort of like over the
52:39
past what three years is like there's this longer time horizon
52:45
process of what I just described, right?
52:48
Whenever I talk to someone new now that's like I I need to, I
52:52
should like make a blog post on this or whatever, but like there
52:55
should be like a map, right? And you should be able to like
52:57
pinpoint this person is right here.
53:01
They don't care. They're not gonna buy right?
53:03
Or they're like here and they're they're thinking about like what
53:08
are the roadblocks now for me to adopt your product?
53:11
Yeah, yeah, we're on this if. There's like a spectrum of what
53:16
where they're at. Where are they in that whole
53:20
spectrum of like, I don't care. I'm just in that first phase
53:24
trying to get something up and running, trying to get
53:26
leadership bought in. Or you're at the last phase of
53:29
like, yeah, this has been a real thorn in my side for the last, I
53:33
don't know how long. And I'm, I want to get it out of
53:37
my side and I'll do anything, like, just tell me what do I
53:40
need to do, Right. Yeah.
53:41
And so I would love to see that blog post.
53:44
Maybe we could put it on the envelope community blog, if you
53:47
ever write it, because it does feel like something that would
53:50
be quite valuable to everyone in the community and something that
53:54
we've thought about a lot. You know, like we always are
53:56
philosophising on maturity levels.
54:00
And I think that is one thing that is quite interesting for
54:04
people. I know we've had some, we've had
54:07
some awesome guests on here who have talked like Maria Basak
54:10
came on here and they talked about and they're from
54:13
marvellous envelops. They talked about how they did
54:17
an internal ML maturity study for their company, where they
54:24
went around to all the different teams and they said, hey, we've
54:27
got this form, fill it out just to know what are you using, How
54:31
are you using it? How do you like it?
54:32
What are you using for observability?
54:34
Are you like, yes, no, maybe this is the tool we're using.
54:37
What are you using for orchestration?
54:40
Here we go, boom, boom, boom. And then they were like, OK,
54:43
well, if we're going to build something centralised, we're
54:47
going to leverage what we already have that's already gone
54:50
through procurement and we already like are paying the
54:53
credit card, right. And we're, we've got it on
54:57
whatever, some kind of retainer. Can we just up our usage of that
55:00
or add it into the current centralised platform?
55:04
So I thought that was a really nice way of doing it.
55:07
And then they wrote a blog post on, here's all the questions we
55:10
asked, Here's how we looked at the maturity of these different
55:12
teams and how we thought about creating that centralised
55:16
platform for our company. And so there's things like that,
55:21
like maturity, especially as quickly as we're evolving and as
55:25
quickly as the tooling is evolving, we still need to think
55:29
about, like what does maturity look like now in 2023 or 2024
55:34
versus, yeah, like maybe in 2020 it was like, has your team read
55:40
the high interest, credit card debt, machine learning?
55:43
Have they read that paper? Whoa, you're super mature.
55:45
Like, everybody's heard of that. OK, cool.
55:49
So now it's a whole different ball game that we're in.
55:54
Yeah, so this. This hypothetical blog post
55:58
right I think would be useful to both like orchestrate our teams
56:04
like or like Mill Products teams to figure out like to qualify
56:08
their sales funnel but also for organisations looking to buy
56:12
solutions right So and it's you know as much as I would like to
56:18
just constantly be selling flight right it's it's sort of
56:21
like leave it up to whatever like local conditions
56:28
organisation and then like the the team is is subject to at a
56:34
particular time then like let's make then let them make the
56:37
choice right if containerization is not super important and like
56:43
right there should. Be a road map.
56:45
For you to make those decisions. But you.
56:50
Know at the same time I am. I do think that there are just
56:53
like some fundamental designs that will probably stand the
56:58
test of time. And you know, I've kind of
57:02
already mentioned them, right? But maybe an analogy would be
57:05
like with programming languages, right?
57:06
Like we're learning, the types are pretty good.
57:10
You know, like having types in as a first class feature in your
57:14
programming languages. Like maybe you do the way of
57:18
Python or you add it on later and it's not like really baked
57:22
in properly or like really like fundamentally baked, baked in.
57:25
But like, there are some features in programming
57:28
languages that are just really nice to have and just make us a
57:33
lot more productive. And I think similar requirements
57:37
and feature sets will come out for for like the Milos Olmo
57:42
space. Yeah, yeah.
57:46
It's that congregating around design and systems and like, OK,
57:52
maybe there is everyone's a special snowflake in some way or
57:57
another other, but there are certain things that are treated
58:02
as first class citizens and you you start to see that and you
58:06
start to realise that if I'm not treating these things as first
58:09
class citizens then I'm going to make my life hard.
58:14
Yeah, so potentially that's like the second part of your blog
58:18
post. You're gonna write like, what
58:19
are the first class citizens now?
58:21
Like, Yeah, it it'll be a. Challenge because I'll I'll need
58:26
to sort of do some more research and get out of my little flight
58:29
bubble and like, you know, soul searching, do some.
58:32
Searching. Yeah, exactly, dude.
58:34
You know what I was thinking about you.
58:36
Know, I get the feeling you probably think like this, but
58:42
living in like declarative worlds and seeing how easy that
58:46
is when it comes to working with computers, do you notice you,
58:50
like, try and do that in real life too, for when you work with
58:56
other people, it's like, hey, here's the, you know, like, I'm
58:59
just declaring what I want and you figure out how to go and do
59:02
that. That's interesting.
59:05
I can't. I mean.
59:06
When you say that I I can't kind of pinpoint a specific thing,
59:10
but it does kind of jive with. Sort of myself.
59:16
One of my. Kind of values of of, sort of.
59:22
Working with people. Right.
59:23
It's like. I don't like to be micromanaged.
59:26
That is the imperative. Imperative programming is
59:29
micromanaging the computer, right?
59:31
Declarative flow programming, like really under the hood there
59:35
is some set of instructions that are carrying out your
59:38
declaration right? So it's maybe like a false
59:41
actual false binary, but anyway a like yeah, when?
59:46
When I'm working with. People or I have a.
59:50
Manager and we're trying to get stuff done.
59:53
It's not like here are all the steps.
59:55
It's like this is the outcome. This is the output.
1:00:00
You know, do the thing yourself, figure it out and you know,
1:00:04
execute it however you want. And yeah, I guess it's that's a
1:00:09
very interesting actually point I need to think about that more.
1:00:13
Sure, I just noticed it because. In my life I started trying to
1:00:18
do it and I realised like how much more energy it is to try
1:00:22
and break things down step by step and then when you got to
1:00:25
debug, like why did this process not work out or what's going on
1:00:29
here, It's just time consuming. Although you can offload parts
1:00:35
of what you're trying to do, you still aren't necessarily able to
1:00:40
just be like here's what I want to happen, no, go make it
1:00:44
happen. And that's where I think when
1:00:48
you find someone that you can trust, that will do that.
1:00:52
That's. An incredible.
1:00:54
Incredible. You know, it's just like, ohh
1:00:56
sweet, like, here's what I want to happen and then I fully trust
1:01:00
that you're going to be able to come back to me.
1:01:02
How? Over a certain amount of time,
1:01:04
whether it's a week or a month or whatever, however big the
1:01:07
project is and you're going to be like, OK, so here's that
1:01:12
outcome you were looking for. Here's how I did it.
1:01:14
And maybe there's things along the way where you need that
1:01:18
input from me, but it's not necessarily like I'm telling you
1:01:23
how to do it. It's just like, hey, I've come
1:01:25
to this crossroads and maybe you have some wisdom about how this
1:01:29
should be done next. And so I'm really, like,
1:01:33
realising in my life I'm trying to find more of that declarative
1:01:37
person that I can work with so that I'm not going crazy and
1:01:40
running around like a chicken with my head cut off on all
1:01:43
these inputs. Like, oh, this is how it goes.
1:01:47
Like step one, Step 2, step three-step 300, you know, yeah
1:01:52
and I. I think the risk of that is that
1:01:54
those instructions may not fit in someone else's head.
1:01:57
Yeah, right. Or the edge cases.
1:02:00
Like just like in programming, you know if the same shit in
1:02:05
real life where all of a sudden you think it's going to go one
1:02:08
way. And then because I've done it, I
1:02:10
have a list of tasks right? Of things, procedures on how
1:02:17
this podcast is going to work right?
1:02:19
And or maybe it's how we post A blog post on the analyse
1:02:23
community blog or these things where there's standard operating
1:02:27
procedure. But then next thing you know
1:02:30
you've got somebody that maybe when they fill out the form that
1:02:34
I sent you beforehand, they don't fill out certain parts of
1:02:38
the form. And then OK, well what's the,
1:02:41
what's that then what's the edge case around that?
1:02:43
How are we going to deal with that part of it.
1:02:46
And so you have to go and figure out what you do there and then
1:02:49
next thing you know, ohh, the person doesn't, the guest
1:02:53
doesn't show up or the guest like shows up late or tries to
1:02:56
reschedule. OK, now we got to figure out
1:02:58
there's just so many different things that you have to work
1:03:01
through. It's like, I don't want to have
1:03:03
to work through each one of these edge cases.
1:03:05
I just want to figure out, or I want to let you figure that out
1:03:09
and trust that you're going to be able to figure it out, Yeah.
1:03:13
And maybe to to bring it back to, you know, not not to flight
1:03:20
this topic too. Much, right?
1:03:21
But this? Is like you know chain of
1:03:22
thought like this is this is there are like interesting
1:03:27
mappings to like the LM space here right?
1:03:30
Because when you give an instruction, you either expect
1:03:36
the model to just get to the result immediately or you have
1:03:40
it kind of reason and try to like output more useful tokens
1:03:45
for it. In its context its working
1:03:47
memory to like kind of help it in its task.
1:03:52
And that's like. The process of breaking things.
1:03:54
Down right. Like when when you provide that
1:03:58
generative or, sorry, when you provide that kind of declarative
1:04:02
instruction, the person executing that instruction will
1:04:05
still need to do things imperatively.
1:04:07
Like, really, At the end of the day, so.
1:04:12
I mean it's it's. Interesting that like we are
1:04:16
creating these systems that have are like simulacra or like kind
1:04:20
of simulate things that we can sort of relate to and I think.
1:04:26
That's why that's like. I would say that's a big part of
1:04:30
their power. Is there our ability?
1:04:34
They've they've kind. Of they have.
1:04:36
Kind of hacked us into like empathising or like
1:04:40
anthropomorphizing, these systems that are just token
1:04:44
token crunchers. Yeah, really dumb.
1:04:47
Autocomplete, as some say, or really smart autocomplete some
1:04:51
would say. Dude, well this is awesome man.
1:04:54
You want a philosophical conversation?
1:04:56
I think we went a little philosophical there at the end.
1:05:00
That's it. I think we'll end it.
1:05:01
There. That's the bod, bro.
1:05:04
That's awesome. Thanks, Dimitrios.
1:05:10
Yeah, hey everyone, my name is Aparna Founder.
1:05:12
Of Verizon, and The best way to stay up to date with Emma Lops
1:05:15
is by subscribing to this podcast.