Latest / Agentic Conversations (formally mlops.community) / Alignment is Real // Shiva Bhattacharjee // #260

Transcript

Alignment is Real // Shiva Bhattacharjee // #260 · Agentic Conversations (formally mlops.community)

0:00
I am the CTO of True Law. We build, you know, bespoke AI
0:09
solutions for law firms. And and as far as I take my
0:15
coffee black, no sugar, no milk, I think just the way I feel.
0:20
I do feel like, you know, I'm from Calcutta.
0:24
We, we grew up drinking tea and we, we don't drink tea with like
0:30
milk and sugar and all of those added.
0:32
I think like the, the actual flavour of tea comes from, you
0:36
know, like Darjeeling tea. That's what I think the same
0:40
thing with coffee is also it's like where the milk and all of
0:44
these, and I mean, current American coffee is like recipe
0:48
for diabetes. I feel like, I mean versus, you
0:51
know, things like black no sugar, coffee is, is is that's
0:55
what that's for me. What is happening?
0:59
Good people of Earth? This is another Enelops
1:03
community podcast. We're talking with Shiv and what
1:07
a leader this guy is. He has done so much in the
1:12
engineering world. I feel honoured to have gotten
1:15
to speak with him. Before we get into the
1:18
conversation, I want to play a little song for you.
1:20
Get that recommendation engine that you have spiced up just a
1:27
bit. Maybe you can consider this
1:30
destroying your algorithm. Maybe you can consider it
1:33
upgrading it. That all depends on you.
1:39
The only problem that I have with this song right here that
1:42
we're about to play, which is called Glimpse by Virelli.
1:50
Only problem here is that it is too short.
1:56
I put this. Song on a.
2:00
Playlist that I called the sound of Angels wings because
2:08
literally that's what it feels like to me.
2:10
I listen to this Correct me if I'm wrong, but you
3:03
guys are using DS Π, right? Yes.
3:07
So you're, I think I put something out and I said, if
3:11
anybody is using DS Π in production, please get a hold of
3:15
me because I would love to talk to you.
3:17
And you reached out and it was like, hey, what's going on?
3:22
Let's talk now break it down for me.
3:25
What? What exactly are is going on and
3:28
how have you found it? So, you know, we were
3:33
experimenting with prompting as usual.
3:37
I think, you know, as, as things were coming along, aside from
3:39
zero shot prompting and we, we played around with Lang chain at
3:45
the time. And there's always this notion,
3:49
or I mean, I think I resonated a lot with Omar's notion that, you
3:54
know, this prompts are brittle and if you change them here and
3:59
there, this sort of breaks. And at the time when the LLMS
4:03
were, you know, the GPT revisions were much faster or we
4:07
wanted to experiment with other LLMS, there was definitely a lot
4:12
of select changes to the prompt we needed to make in order to
4:17
select at this, you know, the result now things have over time
4:20
has improved. I'm sure now there's like
4:23
structured outputs that Open AI is doing.
4:27
But, you know, at that time, this was not so easy.
4:31
And there was a lot of these kind of like changes to the
4:33
prompt that we had to continuously make.
4:36
And that's sort of like, you know, resonated quite a bit.
4:39
I mean, I, I worked at data bricks before.
4:41
So I had, I had matte in my sort of like LinkedIn, you know,
4:47
feed. And I think from there I kind of
4:50
learned about DSPY or DSPI. I think that's what what they
4:55
call it now. Always called it.
4:58
I guess that's. I was calling it before so no
5:02
shade. However you want to call it.
5:06
And then, and I mean, I think I, I went through couple of these
5:10
videos that Omar has put out and I think they resonated very
5:14
strongly, especially, you know, in the, in the context of what
5:17
we were doing that it's, it's modular.
5:21
It's the idea, the areas that, you know, you sort of like the
5:25
optimizers make a lot of sense in terms of, you know, how you
5:30
can actually iteratively improve this.
5:34
And you have you still have to kind of go through, I mean, and
5:37
we can talk about that later, but you know, you still have to
5:39
go through this limitation. I think the optimizer still take
5:43
a bunch of examples to sort of like optimise on And so, but but
5:47
in cases where you are doing that and where you have those
5:50
limited examples that you can show to sort of like mimic what
5:54
you're looking for, I think that this is a better, you know, that
5:59
that sort of makes a lot of sense.
6:01
I also really like the sort of, you know, the pie aspect of DS
6:05
pie is that it's very like pie torch.
6:09
It's very modular in nature. So from a programming point of
6:14
view, I think it was very modular and we can go a little
6:18
bit over the sort of like stack on the retrieval side that we
6:21
have done. And it was easy to use different
6:24
rerankers. It was easy to use those things.
6:27
So. So I think that that's why we'll
6:31
be stuck with this PYI mean, we also use this for, you know,
6:35
generating data, synthetic data in some cases, but nice.
6:40
But yeah. Yeah, We can talk about the
6:43
synthetic data in just a second. But the burning question that I
6:46
have is, aren't you afraid that DS Π is basically a research
6:54
project and it is not necessarily something that you
7:01
want to put that much faith into?
7:04
Yeah. I mean, you know, like The thing
7:07
is that we obviously have made changes.
7:10
So the way this works is like, you know, you clone the rebuff
7:13
yourself and you're making changes along the way because
7:15
you obviously when our upstream change is going to get pushed
7:20
in. I mean, that way actually the
7:22
the community has been pretty good.
7:23
I mean, we have pushed one or two changes that that got
7:26
accepted very quickly. But yeah, I mean, at the end of
7:31
the day, I mean, you know, it's Python code.
7:33
You're looking at it and and kind of like seeing seeing the
7:36
same. And that is also true for the
7:38
land change of the world. But there were certain, I mean,
7:43
that was the one of the good things.
7:45
I also fell in the spy because of its modular nature and how it
7:54
works. You can kind of write your own
7:57
module also and and kind of like see how how it works versus
8:06
again, I maybe I made mistakes when I was doing this kind of
8:10
changes in Lang chain. I felt like you, you kind of had
8:16
thing your chain had to had a had to go through a lot of
8:19
layers to kind of make the same sort of changes versus in DSP
8:25
like we were using our own reranker.
8:28
It was easy to sort of like use that as a different module and
8:32
then plug it into the framework whereby you're still calling
8:35
this DSP wide retriever. I mean, the, the, the object
8:38
hierarchy and the calling is just better and modular in, in
8:42
my opinion, in generally like, I mean, you know, there's always
8:46
that thing where whereby you know what you're doing.
8:51
I mean, first of all, this the the thing, the nice thing is
8:54
that, you know, this is this is code that is just running on
8:56
your servers bunch like our use cases.
9:00
It's not it's, it's really latency bound rather than
9:04
anything else. And you know, you, you can do,
9:10
you can enable tracing and basically see the sort of like
9:14
calls that you're making. Actually one of the things that
9:16
we have done, which is which has been quite helpful is that
9:20
because because of the chaining and the hops that are happening
9:24
and that sort of like query rewriting that happens, we kind
9:29
of have exposed that intermediary step, step to the
9:32
end user. So you know, they get a good
9:35
sense of like what is happening and how the query is becoming,
9:39
you know, more contextualised with the content that they're
9:43
searching from, from where they started.
9:46
So I think in some of those cases that sort of that
9:49
visibility and observability actually has helped in also the
9:53
product in terms of what we have done.
9:58
Yeah, it's fascinating to me because DS Pi is the one open
10:04
source tool I think out there that does not have a big well
10:09
funded VC behind it and a company that has been that just
10:15
got all that money from the VCI think.
10:17
And so you see that there's an incredible community and there's
10:22
a lot of people and a lot of energy around it.
10:25
But I know that some folks get a little scared because there
10:30
isn't a company that is in a way giving it that stamp of approval
10:35
and saying like, yeah, we're going to be the shepherd of this
10:39
open source tool. Yeah, I mean, I mean, that's a
10:44
very interesting point. Yeah.
10:45
I mean, I think you were right. Like, you know why why Lang
10:48
chain has funding and you know that got got got started and not
10:52
DS Pi. I think that's a very good.
10:54
I mean, there's definitely, you know, the first mover advantage
10:57
of, of these things and you know, how much active
11:00
development goes on and, and, and stuff like that.
11:04
I think with with DSPI and my so like gut feeling here is that
11:12
because there's this subtleties around improvements and the
11:17
iterative influence, it happens like the prompting that happens
11:20
over in the back. That's probably not as exposed
11:25
as in in Lank chain. I mean, in Lank chain you are
11:29
still sort of like explicitly calling out, you know,
11:33
particular like chain of thought or something like this very
11:36
explicitly. And again, you mean these things
11:41
have, you know, they've become more modular and more abstract
11:46
over time. So it's, it's been a while since
11:48
I have used Lang chain, but the DSP when we first looked at it,
11:52
there was a, the learning curve was, was a little bit, I would
11:58
say steeper than than Lang chain.
12:03
And I don't know if that that hasn't that causes adoption
12:07
issue or you know, like that's, that's kind of where, but
12:10
ultimately, you know, it is a very fascinating thing that, you
12:14
know, these are equivalent systems.
12:18
They they still sort of like suffer from the same like, you
12:22
know, this is ultimately prompt engineering that that is
12:24
happening. So you know, there are
12:26
limitations around that. But but it is a good point.
12:30
I think the main thing in my opinion would be typically I've
12:35
seen a way to productionize things is you kind of have to
12:41
provide a service. You know, like you, you
12:44
basically, if you were like taking this thing and you needed
12:48
like a server or some state management that you needed to
12:53
manage yourself. And that's kind of when this,
12:56
you know, there there's a company that comes behind and
12:59
says, you know, we are going to take on all the operational
13:01
headache of you doing this and that becomes a legitimate
13:05
service. And that's when you see this
13:06
with Lang Smith and stuff like that, wherever you know, you're
13:09
doing this drag and drop and observability of those.
13:12
And those are becoming kind of the part of it.
13:15
I'm sure if someone invested the time they can, they can make an
13:20
argument of the same sort of suite of things that, you know,
13:24
LAN Chen has done on the DSPY side of the world.
13:28
But yeah, I mean, if you just think of it as a framework and
13:33
an SDK and you know, like in a library that you're kind of like
13:37
using, then most people will be like, kind of like running it.
13:43
All this code and server is stuff is running on their mind
13:47
environment. What what is the point of this?
13:49
But yeah, but yeah. So I think you have to sort of
13:52
like always associate a service with this which may or may not,
13:56
you know, like you have to think through like what what that
13:58
would be and that probably would justify, you know, so like
14:01
building out a company well. OK.
14:04
So getting back to what you're working on, you have a Rag use
14:10
case, right? And you mentioned that sometimes
14:12
DS Pi is the interesting route and then other times you fine
14:17
tune. And I would imagine other times
14:19
you just you don't do much and it comes.
14:24
It's like naive rag as they call it.
14:27
Or can you break down these different scenarios and what you
14:29
found to work best when? So one thing you know, we have
14:34
focused on the legal domain. So we our, we, we build like
14:39
bespoke solutions for effectively lawyers and law
14:43
firms. And here, you know, precision
14:48
and quality matters Trump's over latency a lot.
14:52
So they want still like search results to be quick, but it
14:58
doesn't have to be instantaneous in terms of like, you know, your
15:02
Google search experience, right? Yeah.
15:04
So we always had to focus on quality.
15:08
And that way also this demographic of users are not
15:16
like the best prompt engineers. I mean, they're not thinking in
15:19
terms of, you know what, what is the best way to write a prompt.
15:23
And I think what we have found out is sort of like the way the
15:28
questions are posed by a typical lawyer.
15:32
And it's very important to contextualise that question in
15:37
order to able to do a better retrieval.
15:41
And that's kind of where this query rewriting helps a lot
15:45
where and that's kind of what we are using DSPYDSPY for.
15:50
And then we, it's very parameterizable, which is means,
15:53
you know, you can determine the number of hops you want to do
15:57
and you know, like what is the depth of retrieval you want to
16:03
achieve? Then you know, how many
16:05
documents you want to see in your top K and all of those
16:08
things. And these are again, like are
16:11
kind of important to the lawyer. You know, they're making a call
16:14
between recall and precision in terms of the number of data that
16:18
you want to get back. So and, and because we could
16:25
play around with the, the latency we don't have to, we
16:28
don't again like I think they understand like the more hops
16:32
people are doing, you are going to get a little bit better
16:35
quality with, with this approach, but at the expense of,
16:39
you know, the results taking a little bit longer time.
16:43
And this is especially true in kind of like in a very domain
16:47
specific question answering stuff for the fine tuning work.
16:52
We, we have done some work around embedding models to
16:58
because that's one of the other metrics around times of like you
17:02
know, when you're doing the search and the retrieval the
17:07
typical way, if the embedding model is just trained on the
17:12
corpus of data that you're searching on and the queries
17:16
that you're doing, it's just a better typical match.
17:19
And again and domain specific things this, this tends to work
17:23
a bit better. But then there is also the
17:28
question around the generation part of it, whereby in a certain
17:35
certain firms have a particular way of seeing the answer or you
17:39
know, like they expect a certain, because typically this
17:43
would have been done by junior associates where you, you
17:45
effectively would have told other folks to do this search
17:49
and come back and then and they have a certain way of presenting
17:52
this data. So I think that alignment of
17:56
generation is something where we have also used fine-tuned for
17:59
fine tuning things for and you know, typical RAG.
18:03
I mean, again, I think when we started this, we found a bunch
18:06
of like off the shelf RAG approaches and again on domain
18:12
specific things. And for the lawyers, they, they
18:15
typically don't, I mean, in our experiment, they didn't perform
18:19
that well. We always had to do a few
18:23
different things, either extracting metadata first or you
18:27
know, basically make the retrieval process much more
18:30
contextualised in different ways before we aggregated the data.
18:34
If you just were to use your typical like, you know, just use
18:39
an embedding and your regular embedding and just get the
18:41
retrieval, the quality was not that great.
18:46
Well, the so there's a few things that I would love to know
18:50
about, specifically on the fine tuning of the embedding models.
18:53
I think you can do that for relatively cheap these days,
18:57
right? Yeah, I, I think, you know, the,
19:00
the main thing here is because the embedding model itself is
19:07
doesn't have to be very big that you, you can, you can actually
19:11
have just the encoder part of these models and they typically
19:18
are, you know, the dimensions doesn't have to be like very,
19:21
very large. And the main thing there in an
19:27
embedding model is to kind of figure out the training data
19:29
that's to like, you know, you're basically giving it some
19:34
contrastive data and that that the generation of that
19:37
contrastive data is the hard part based on the percent
19:40
that's. Kind of where it's like it's
19:42
cheap money wise but it is expensive in resources.
19:47
It's expensive in figuring out like how to generate that
19:50
contrast of data. I think that's the harder part.
19:53
But yeah, I think we in general, even training and all of these
19:56
things now, these are becoming extremely commoditized.
20:00
So we, we actually don't focus on the the training
20:04
infrastructure per SE. We we have an orchestrator layer
20:08
that's all like is very agnostic to where things could be
20:11
trained, but. But in general, like I think the
20:17
all of these are, you know, I think from when we started to
20:20
even now like the the drop in price is significant and I think
20:23
that will just continue to grow. I mean even when she PT4O mini
20:30
is is a very powerful model and this training that is very
20:34
cheap. Yeah.
20:36
So you know, what I would love to break down is you've worked
20:41
at a whole slew of incredible companies.
20:45
You're now CTO at True Law, and you knew coming into it, we're
20:51
going to be setting up infrastructure that is itself
20:56
very prone to change. And technology in general is
21:01
prone to change. But with AI and LLMS, this is
21:04
like taking that to the Max. It's very volatile, right?
21:08
Because as you said, things get cheap.
21:11
A model that is super powerful, all of a sudden it becomes just
21:16
a commodity overnight almost it feels like.
21:18
So as you're going through and you're setting up the stack and
21:22
you're thinking about which pieces to value, which pieces to
21:27
try to make future compatible, or making bets on what's going
21:34
to come down in price at. How are you thinking through all
21:39
of these stages? Since you have that bird's eye
21:43
view and you're the one who's ultimately the guy in charge on
21:48
the technology side and you decide with your teams what gets
21:52
implemented. Can you walk me through your
21:55
decision making process and how you think about that?
21:58
Yeah, that's a very good question.
21:59
I think like, you know, in general, like obviously, you
22:03
know, one of the starting points is sort of money constraint.
22:07
Like as a startup, we are have to be scrappy.
22:10
And so like when to build versus buy is always a decision we have
22:14
to make. And then, you know, you have to
22:17
also juggle with the fact that we cannot take six months to
22:21
build a feature because, you know, we we have to go through
22:25
that quick iterative cycle of development.
22:28
And, you know, I think very soon into this approach, we realised
22:33
that building foundational model is very, very difficult.
22:39
And you know, it, it is just not a, even if it is like
22:43
technically feasible for us to learn and do this.
22:47
The sort of like resources needed is just, you know, very,
22:51
I think. And then at that time, you know,
22:53
we, we started focusing on more, you know, fine tuning approaches
22:58
and, and kind of like building, I think Matei and Omar has the
23:01
paper around like this compound systems, which is like, you
23:04
know, connect building sort of like a system of small language
23:10
models that can orchestrate and, you know, like doing bunch of
23:13
these things. And the corollary to this is
23:15
like, you know, your brain and you're, you're doing different
23:19
parts of the brain or kind of like doing different things.
23:22
And then this is sort of like coordination around, I mean,
23:25
there's a 2 train of thought. It's like one large model doing
23:28
it or a bunch of smaller things, coordinating and doing things.
23:31
And I think we kind of took this approach of the compound system
23:34
where which I think it was more feasible in terms of the
23:38
infrastructure. I think that's very good in the
23:41
side. Like we understood that, you
23:44
know, in order to get that scale, the price point right,
23:48
you need like economy of scale to do certain things, right.
23:53
And it's very hard for us to so like build a like a training
23:57
infrastructure just for the few models that we are training on
24:02
or the volume at which we are training on.
24:04
So from a very get go, we understood that we have to
24:07
leverage this sort of training services or, you know, model
24:12
training services that are available and in and you know,
24:15
my previous experiences at at Confluent where we were, you
24:19
know, doing this orchestration for like provisioning
24:23
orchestration for for Confluence servers or KAFA servers and
24:27
stuff. Kind of that got me some
24:30
insights into building this as a as a way of an orchestration
24:34
service which does this in terms of the training.
24:38
And then there's an and then there's so like data generation
24:42
pipeline. So if you, if you think of it
24:44
like the way that we have focused and ever sort of like
24:48
all of our IP is around this, you know, data generation and,
24:53
and how are we fine tuning things and incorporating into
24:58
the models. And in that way, it is not too
25:01
dissimilar from my previous experiences of what I have
25:06
worked on, which is sort of like this data management and, and
25:11
you know, orchestrating that data flow.
25:15
But the way we have always built our stack is we haven't, we have
25:20
always leveraged other SAS providers in terms of sort of
25:26
using their infrastructure to train this.
25:30
And this has been and this has both been a blessing and and
25:35
sort of like cost wise it has been quite efficient partly
25:38
because we give the options on which infrastructure teams could
25:43
be trained. Of course, we can train it on
25:45
our cloud, we can train on the customers in Azure environment
25:53
also. So that flexibility is actually
25:56
quite powerful. You said something there that I
26:00
wanna double click on, which is around how you used to be
26:05
dealing with data flows and now you're still dealing with data
26:09
flows, it's just the data has changed a little bit.
26:12
So I imagine you used to be dealing with event click data,
26:16
that kind of data flowing around or or purchase, this person
26:20
purchased this user ID, purchased something etcetera,
26:24
etcetera. Now you're dealing with prompts.
26:27
I am assuming and outputs of those prompts and how you can
26:30
best bring the output back into the fold to make sure that
26:35
you're constantly levelling up the the pipeline.
26:39
Is that what I understood? Yeah, Yeah.
26:44
I mean, in essence, at the end of the day, whether you're doing
26:46
prompting or fine tuning approaches this like input and
26:50
output, you know, to the way you're talking to the LLMS And
26:54
in general, like, yeah, I mean, I think in, in, in confluent and
26:58
when you're dealing with, I was in the data governance team also
27:02
for a few, few years. It was all about sort of like,
27:08
you know, the flow of data. And then and then that
27:11
particular case is flow of metadata that you have to be
27:14
worried about like, you know, from where it is originating
27:17
into how it is distributed to everywhere else.
27:21
Here. This is a similar thing where
27:24
you know you're getting, you're generating data and you kind of
27:29
like aggregating this data. Of course, this is all
27:31
unstructured data. So the benefits of of BLM is
27:34
like you're trying to get a decent sense of what this is.
27:39
But at at the end of the day, it is sort of like incorporating
27:43
the feedback or moving this data or constructing the select
27:47
training set on what this is. So it yet of course everything
27:53
is data and what you're removing with the context of data changes
27:56
quite a bit. But the sort of like
27:58
infrastructure that you need in a keeping versions of it and,
28:01
and, and things like that and being redundant and being able
28:04
to replay back. These are all sort of sort of
28:08
the same building ethos and engineering ethos around like
28:13
how you do that and that has a lot of similarity.
28:17
So there's stuff that you decided to buy and not build. 1
28:22
obvious one is the LMS, which I think that is in hindsight a
28:29
very good choice in knowing how much it cost to train them and
28:32
seeing how quickly they are going down in price.
28:35
That makes total sense. Are there things that you are a
28:40
little bit more surprised that you went to for the buying
28:45
option or you you are happy I guess is how I could frame it
28:49
that you bought instead of built or vice versa, you're happy that
28:53
you built instead of bought. Yeah, I think so.
28:57
You know when we first started we actually built I think with
29:03
every micro service. Of course our thing is a micro
29:06
service architecture and we needed a communication mechanism
29:10
between, you know how they will talk.
29:14
So you need some sort of mechanics between, you know,
29:18
this communication mechanics between your micro services and
29:22
you know, we have leveraged given from confluent like I
29:28
errors like confluent clouds to but ultimately we build a
29:31
messaging system between these services to handle a bunch of
29:35
asynchronous stuff. So again, as I said, like most
29:38
of these work that are dealing with LLM is latency bouncer.
29:42
So the asynchronous communication had to be, you
29:45
know, part of the building stack from whatever we built.
29:50
And that served us pretty well. And until, you know, we were
29:56
getting production use cases of like doing massive inferencing
30:01
for very, you know, thousands of emails or like very large scale
30:08
inferencing, which will take hours to run and, and, and
30:12
things like that. And in those cases, we are
30:15
actually using temporal on this is, I don't know if you know
30:18
that this is like we call it durable workflows started from
30:24
like Uber in terms of like how they manage.
30:28
I mean, this is ultimately again workflow management sort of a
30:31
thing. And we kind of made a good
30:36
decision because I think I was talking to the developers, the
30:39
first pass of, you know, doing this.
30:41
We, we kind of used our existing infrastructure to do that.
30:46
And again, the sort of like loopholes that there is the kind
30:53
of like guarding you need to do against like retry how many to
30:57
how much to retry when things get interrupted.
31:01
All of those other things that you need to take care of versus
31:05
sort of like using, you know, a company that's effectively like
31:10
this is the, you know, first of all, we're we're a small group
31:13
of engineers and then kind of like using them to kind of
31:18
handle this. Durable workflows have worked
31:20
out very well for us in terms of when running this very long,
31:25
running inferences or doing training, getting notified if
31:30
things get interrupted and getting handled.
31:33
Of course you have to write code to sort of like get it to use.
31:36
But I think we haven't had infrastructure problems related
31:40
to that thing, which I'm sure you know, it would have taken us
31:43
at least a couple of months to kind of perfect that sort of
31:47
infrastructure for ourselves even.
31:50
And then again, it will be very custom to just our use case and
31:53
not very generic. So every time we probably needed
31:56
to add and make any changes to the state machine, we would have
32:00
to have gone back and you know, like make those changes and
32:04
tested and whatnot versus I think using Temporal has been
32:09
very useful for us like because I think we kind of like you get,
32:13
it's a workflow engine. We have to build of course the
32:15
state machine that does that. But writing that state machine
32:18
logic is much more simpler, much more additive in nature.
32:21
And then depending on them to execute that is it's much easier
32:25
for us or has been very easier for us.
32:30
So one thing that I noticed about you is that you've worked
32:36
in many different areas of the stack.
32:41
We could say I and I think it's more even vertically and
32:46
horizontally I could say, and I don't know if you agree with
32:49
this, but knowing your background, you've gone very low
32:52
level. You've also gone from the data
32:56
side or the front like front end, back end to DevOps C now
33:01
LLMSMLOPC, how do you see things now?
33:08
Is there stuff that you can you can look at and say because I
33:16
know this, I can draw parallels with certain parts of the stack
33:23
or the LLM side? I know you mentioned earlier
33:27
that there's data flows. Now it's just LLMS and AI data
33:33
flows and all of this metadata is the important thing.
33:37
And you've came, you came from Confluence.
33:39
So you had that data flow kind of in your blood.
33:43
I know you were also, you were working in, in many different
33:47
awesome places, one of which Apple.
33:49
And so you got to kind of see the gambit.
33:53
Have you noticed different things that shouldn't
33:56
necessarily be related, but that you were able to now relate
34:00
because of this? Why this breadth of experience?
34:06
Yeah, I mean, you know, I've been very fortunate.
34:09
I think in in certain cases, you know, things have clicked in
34:15
unexpected ways. For instance, I, I clearly
34:18
remember I was at Riverbed and we were doing, you know, file
34:24
systems there at least a little bit at that time was building a,
34:28
a file system, a deduplicated file system to do so, like
34:32
compression on your primary file system.
34:35
And one approach there was that because these teams are
34:42
ultimately are your performance is ultimately, you know, like
34:46
how well your data reads and writes are happening on the disc
34:52
and they get blocked. And you know, how, how do you
34:56
sort of like paralyse as much as you can, but that is so like how
35:01
you know, how, how's your pipeline into those data reads
35:04
that happens. And in riverbed we remember, we
35:09
use this very macro based C to kind of get ourselves this sort
35:17
of like version of async IO, if you think of it, but at a much
35:21
deeper level. But they use this notion around,
35:26
you know, used, I forget what was called our threats.
35:31
If I remember it's a riverbed threats.
35:33
That's that, that's the idea there.
35:35
But you know, it was all like macro level way of doing it.
35:39
And you have to kind of get used to this framework.
35:41
It, it wasn't like CC when it was written in C where, you
35:45
know, it's kind of understand like when things would unwind
35:49
and, and and stuff. Surprisingly, when I went to
35:55
Apple and I was kind of explaining what I had worked on
35:58
in, in, in this war, you know, in the riverbed side of the and
36:05
I, I think they were there. I mean, I didn't even know at
36:08
that time that they were working on, you know, the just Grand
36:10
Central dispatch, which is their sort of like approach towards
36:13
multi threading libraries. And suddenly they were like, oh,
36:17
like this makes sense. Like you have, you must have
36:20
like a lot of context around this.
36:21
And we got started working on it.
36:24
And so it was like, you know, one of the two maintainers of
36:27
GCD at the time, but that sort of like, you know, that
36:31
correlationships to those those are unexpected things that
36:34
happened with it. It's also, I also feel like that
36:38
gives you great insights into, you know, in one stack, what
36:43
you're working in different companies, but you know, they,
36:45
they could be related to sort of like this other stuff, which
36:49
seems completely unrelated. But you know, at the core,
36:52
they're also like connected. I remember at Arista, I remember
36:58
Arista, I think Adam Sweeney or I think you know, very senior
37:05
principal or I think ABP at that time, he told me about, he was
37:10
asking me questions around like circular buffers and times and,
37:14
and other things. And then I, I think he mentioned
37:16
like all of computer science could be boiled down into like
37:21
this 7 or 8 algorithmic questions.
37:23
And if you can, you know, you're good at this, you can kind of
37:26
like have a pretty good grasp at most of these, you know, system
37:30
level things. And I think that has been true.
37:32
Like, you know, if you know, buffering and you know how to
37:35
deal with memory overflow, like, you know, out of core algorithms
37:38
and and things like that, they have a lot of, I mean, you find
37:44
that application in several different places.
37:46
And I think that has definitely helped.
37:49
But yeah, I think, you know, going back to your original
37:53
question, like has the breadth helped in understanding?
37:57
It does, I think for sure there's still areas of, you
38:02
know, GPU learning and how, how the GPU works.
38:06
I, I don't have a very good depth of, I never really worked
38:09
on those kind of things. But I think in principle when I
38:13
read some of these papers, it's sort of all up.
38:15
I mean, again, it's, you know, short circuiting, you know,
38:19
certain things, caching certain results for quick access.
38:23
I mean, they are all so like consistent approaches to, you
38:29
know, solving performance related issues that has been
38:33
sort of like universal. Maybe one last thing I should
38:38
just add, like even a data breaks when I was there, I think
38:41
one of the very thing that got repeated all the time was that,
38:48
you know, Spark in general is is an amalgamation of all these
38:55
different concepts. You know, like let's do caching
38:58
properly, let's do distribution of data, like separate the data
39:02
separately and you know, like let's do work on immutable
39:05
immutability and, and things like that.
39:07
So I think, and I think that it was repeated quite a bit like,
39:11
you know, like, are we doing it something very revolutionary?
39:15
I mean, but not in anyone particular angle, but if you
39:19
like aggregate all of those features, Spark as a system
39:24
seems very complete in terms of being able to have implemented
39:28
all of these sort of like best practises.
39:30
And I think that was one that was that kind of stuck with me.
39:34
I think that was true. Like if you look at any
39:37
particular feature of it, it was, it was kind of like will,
39:41
it had basically kind of gotten from the experience of, you
39:45
know, what are the issues with other systems and then and kind
39:49
of like addressed all of those principles.
39:52
In essence. I think that was part of of its
39:55
popularity. And and of course the the data
39:58
frame API was very good to access with the declarative idea
40:02
around it. But yes, so many great
40:05
teachings. Awesome dude.
40:07
We'll end it here.