Latest / Agentic Conversations (formally mlops.community) / LLMs in Focus: From One-Size Fits All to Verticalized Solutions // Venky Ganti & Laurel Orr // #196

Transcript

LLMs in Focus: From One-Size Fits All to Verticalized Solutions // Venky Ganti & Laurel Orr // #196 · Agentic Conversations (formally mlops.community)

0:01
Venki Ganti, Head Product and Engineering at Tumblr station.
0:06
I take my coffee in the South Indian what is called a philtre
0:10
coffee style with chicory routine at a mixed in it.
0:13
So that's how I like my coffee. My name is is Laurel.
0:17
I'm a kind of principal engineer.
0:19
I do a lot of ML at at number station and I will drink almost
0:23
any coffee, but some of my favourite is actually camping
0:25
coffee where it's unbelievably strong and almost steep like tea
0:29
and you usually have it when you're camping or backpacking
0:32
and there is nothing quite like it so.
0:35
Yes, welcome back to another edition of the Envelopes
0:40
Community Podcast. I am your host, Dmitri Ouse.
0:43
And today we were talking with none other than Laurel and
0:47
Benji. And I've heard that I do not do
0:52
justice introducing our guests on these little rants that I go
0:56
on before we actually get into the conversation, So let me tell
1:02
you about our guest today. Laurel is a principal engineer
1:08
at number station and before that, she was doing all kinds of
1:13
cool stuff with Chris Ray, the famous Chris Ray at Stanford.
1:17
Benki brings over 2 decades of experience in software
1:22
engineering and technical leadership to number stations
1:25
and now he is the SVP of Product and Engineering.
1:31
Before that, he was CEO and Co Founder of Mesh Dynamics, the
1:37
API test automation company, and it got acquired by Google in
1:42
2021. Prior to that, he was CTO of
1:47
Allation Elation. Forgive me if I pronounce that
1:52
incorrectly. Venki Elation was the Enterprise
1:57
Data Catalogue company and C helped create the new Data
2:02
catalogue product category. And in this conversation with
2:05
these two bright individuals, we talked all about open source AI
2:09
and what some of the downsides are of one-size-fits-all LLMS
2:16
and why they're not ideal. How to make this trade off
2:20
between the size of the LLM you're using and the performance
2:24
that you're looking for? We also how can we not have
2:27
someone on from number station and not talk about text to
2:31
sequel. I feel like that was a lot of
2:33
the conversation because wow, text to sequel baby, it's a
2:38
reality. We also talked about what it
2:42
means to host and manage private LLMS, what different challenges
2:48
are and the verticals, how you can look at problems with
2:55
especially text to SQL. We went over that quite a bit as
2:58
I mentioned. And so we talked about some of
3:00
the challenges that you may have and also where you can get
3:03
unequivocal gains, what you can do to make sure that you set
3:07
yourself up for success. So let's get into it with good
3:11
old Benki and Laurel. Hope you enjoy and of course if
3:16
you do, we would love a review. I'll see you on the other
3:47
I'm not a expert on Laurel Canyon, but I've definitely
3:50
heard of it and my name is not Laurel and I did not live in
3:55
California. However, your name is Laurel and
3:58
you did live in California. You never heard of it?
4:01
Yes, although to to to my credit, none of my friends have
4:05
ever mentioned Laurel Canyon to me either.
4:06
So not only do I not am I not aware of it, but no one I know
4:09
is apparently aware of it. So collectively we have failed.
4:11
But you know, at least we failed together.
4:14
Yeah, even I haven't heard of it, for whatever it's worth,
4:17
yeah. Really.
4:18
Oh, man. So.
4:20
Well, I guess now we know Laurel Canyon.
4:23
I'm going to do a little bit deeper diving into it because I
4:27
think that it's a cool place, or at least from what I've heard of
4:31
it, it was a cool place, especially back in the day with
4:34
all the music and artistic stuff going on there.
4:37
And Speaking of artistic stuff, why don't we just jump right
4:43
into this whole fun AI boom that you all are on the cutting edge
4:48
of. I mean you're both working at
4:50
number station. We've had, I think, Laurel, you
4:52
came on to the last LMS in production conference that we
4:56
did and before that we had Inez. And so we're, I like to say that
5:01
you're almost like friends of the Pod, even though nobody from
5:05
number stations has been on the pod yet.
5:07
I do feel like I get a front row seat into seeing what you're
5:10
doing, but it's probably worth maybe Venki, can you give us a
5:14
breakdown of what exactly Number station is doing and then how
5:18
you came like how you found yourself at number Station?
5:23
Yeah, that's an interesting story as to how I came about.
5:26
But first let me back up and say what number station is all
5:29
about. I know Chris, Ray and Chris for
5:33
a couple of years now. I was at Google doing an
5:37
interesting incubation project there into in the commerce
5:41
space. And then I was, I was teaching
5:45
to get back into the data space. I was talking to Christian Chris
5:50
Burger and one thing led to another.
5:54
Starting around April or so, we started seriously thinking, oh,
5:57
you need to get back into the Zero to 1 space.
6:00
It's not like that's where what I love doing and I I like the
6:06
data space. That's what I realised.
6:07
And so after talking to other founders in SN and and a few
6:12
others at Number Station, I decided to join, join the team
6:16
And it's super exciting because given all the talent that Number
6:20
Station has around machine learning, LLM and technology and
6:24
all of those things. So it's it's it's clear that we
6:28
can actually make a dent in how elements can be used for various
6:33
problems around data management, data insights and so on.
6:37
So that's what excited me and I'm, I love coming back to the
6:43
startups. So that's what that's what
6:45
brought me back and coming back to what number station is all
6:50
about. So we are building private data
6:52
LLMS for enterprises and these private data like you all know,
6:58
they become the platform for enabling various applications on
7:02
top, one of the key applications that we are targeting is around
7:06
conversational BI on data, right.
7:10
So what this does is enables business users to ask questions
7:14
of their data the way we ask questions from ChatGPT for
7:17
general information. So business users can actually
7:21
ask questions from their data and get responses back and
7:25
accurate ones because these are private data elements which
7:28
understand the enterprise context and so on.
7:30
So that's one of the key applications and there are a few
7:33
other applications that we are hearing from our customers
7:35
around entity resolution data wrangling and that's one of the
7:38
first papers that came out of Stanford.
7:41
Ennis and Laurel are co-authors of that one.
7:43
So these guys have sort of led the feel in that space.
7:47
Excellent. So Laurel, give us your story on
7:51
how you ended up here. Yeah actually kind of an
7:55
interesting moment in my career where I wasn't actually planning
8:01
at all and joining a start up and I remember actually telling
8:03
myself when I was in grad school doing data management work that
8:06
I would never do in a start up because it's far too risky and
8:11
but yeah I ended up doing a post doc with with Chris Ray at the
8:15
Stanford lab worked very closely with with with one of the Co
8:19
founders Sen for for many years. He taught me pretty much
8:22
everything I know about machine learning and was kind of on the
8:27
verge of going off into academia potentially was a little
8:30
dragging my feet some you know there was a sense of hesitation
8:35
but you know had joined negotiation really part time
8:37
just kind of having fun see what they're up to.
8:39
You know did some of the work on on entity kind of entity
8:42
resolution and then started you know didn't did to work on on
8:46
these large models and was fairly convinced that they were
8:49
sort of going to change how we build applications and interact
8:52
with our data. Really wanted to explore this
8:55
further. Didn't really know how it would
8:57
fit into the academic side of my life at the time.
8:59
And then I remember being on one of the all hand this meeting
9:03
that that the CEO Auburger gives and he kind of made this
9:06
declaration. He was like we are a language
9:08
model a large language model company like this is our future
9:11
like where we're going to bet heavy on these things.
9:13
And it was like right at this moment where I didn't know what
9:15
I was going to do and took that as a sign.
9:17
And so then signed up completely full time and have been there
9:20
ever since. And just I'm so kind of
9:22
passionate and excited about what these models can do for for
9:26
engineers, for developers and kind of just want to be at the
9:29
forefront of understanding how they work and have.
9:32
Yeah, it's been, it's, it's been a lot of fun.
9:36
Yeah, no regrets. So then key, I think there is
9:40
something that you mentioned that number stations does, that
9:43
I am starting to see more of and I would love to centre the
9:48
conversation around this for the next couple minutes which is
9:52
it's not all just ChatGPT like use cases where you go and talk
9:58
with a chat bot. But you said, oh, customers are
10:01
coming to us and they're saying that they want data wrangling
10:05
and how does that even look? And the when you say that, I
10:08
think, wait a minute, is AI just going to be able to do my data
10:13
pipelines for me? Is that what you're trying to
10:16
tell me, or are we still pretty far?
10:18
From that, so there are multiple multiple stages in data
10:23
transformation and data wrangling before data is ready
10:26
for all the analytics work that happens over a data warehouses.
10:31
So one is the transformation from a source system into the
10:34
target data warehouse and that involves schema transformations,
10:39
data transformations and some data wrangling and data cleaning
10:42
there. So data wrangling could be
10:45
either. Oh, I'm I need to match the
10:47
customer request like entity resolution.
10:49
That's one aspect of it. Other aspect is around
10:51
sanitising column names. So I have state columns
10:54
abbreviated or fully expanded in some regions, in some other
11:00
regions that are abbreviated. So that kind of standardisation
11:03
needs to happen and null values, all of these have to be filled
11:07
in. So this is all manually
11:10
intensive. So and there are various
11:13
problems that people have to address, the data engineers and
11:17
and data scientists have to address in this space.
11:19
O the place where we are seeing significant traction is around
11:23
data entity resolution, matching records which which are which,
11:29
which represent the same entity, but they're represented
11:32
differently in the database right from because they're
11:34
coming from different sources. So that's one significant area
11:37
that we're seeing a lot of customers like insurance
11:40
customers, one of our customers is there and another one in the
11:43
real estate industry is this. These kinds of customers are
11:46
coming in and asking us oh we need help here have rule based
11:50
systems and these are taking a lot of time to develop and
11:53
maintain. So instead can we use something
11:55
like a large language model and make it efficient as well as
12:03
accurate enough so that we can put it in production, right.
12:05
So that's that's the place where we are seeing a lot of traction
12:09
in the in the data cleaning, data wrangling space here.
12:12
And and coming back to one other point that you guys are we at
12:15
the point where we don't need any any data engineers anymore
12:18
to do this. We're quite far from that.
12:21
So we are still assisting these engineers to do the work, both
12:25
in the transformation. So I need to take this source
12:29
data and map it to the target schema.
12:31
There's still a bunch of manual engineering effort that is
12:34
involved. At some point we'll have
12:36
Copilot's kind of a thing playing a role there, but it is
12:39
still a significant engineering involvement that is necessary.
12:42
And same here in the data cleaning space too.
12:44
It's not just LLMS will come and do it.
12:47
We need some tuning from the engineers.
12:48
We need the domain experts to guide the LLN to do the right
12:52
thing, so the engineers will still play a significant role in
12:55
this. These elements will start
12:57
assisting these engineers be more productive and more
13:00
accurate and more efficient in what they do.
13:03
Yeah. Anything what I'm kind of about
13:06
the belief that chat isn't always the right interface for
13:09
for all the tasks that people do.
13:11
Like it's it's great for some things and like is amazing to
13:14
like hook people and get them really engaged and like great
13:16
for some nice sort of quick one off demos.
13:19
But the one that you did show us at especially like like like
13:22
Github's Copilot, if you're familiar it's like the coding
13:25
assistant. What I think is my personal take
13:27
is like probably one of the most successful applications of these
13:29
models is that they're built to be Co pilots.
13:31
Like they're not there to replace people, right.
13:33
Like, they're very much there to help you go faster.
13:36
But they don't. Yeah, they're they're not there
13:38
to completely wipe away what what you do.
13:40
I mean, it's just there to to facilitate your process.
13:44
Yeah, 100%. So one thing that I have heard
13:47
people talk about, and I know anyone who has been a machine
13:52
learning engineer or a data engineer for probably more than
13:55
a year or two years has seen some hairy sequel being written.
14:00
And whether that's like a Sequel query of like over, I don't
14:07
know, 102 hundred, 300 characters long, you've probably
14:11
seen way more than anybody and you've seen like the nastiest of
14:16
Sequel queries being written. One thing that I've heard people
14:19
say is that when it comes to Sequel and how we are piping
14:23
around this data or using this data, manipulating this data,
14:28
it's very much like context specific, but not just context
14:33
specific. It's like very much like the
14:36
engineer or the person who is writing it has to write it.
14:41
And there is a little bit of that like the bus problem where
14:45
if they get hit by a bus and then tomorrow they don't come in
14:47
and you have to figure out what is going on with this SQL query,
14:51
it's going to take you a while. I get the feeling that you have
14:56
thought about that quite deeply and especially when it comes to
14:59
LLMS being able to interpret different SQL queries and
15:03
understand what's going on. How have you seen it match up?
15:07
Like, does it get it right away? Is it just how do you know that
15:11
it's not just hallucinating something?
15:13
All of those questions that I'm sure you're thinking about day
15:16
in and day out. Yeah, yeah, no, no.
15:18
Texas sequel, you know, like came from a data management
15:22
background and that has been like a classic problem in the
15:25
whole in the community for, for, for decades.
15:27
Like, can we get this translation to work?
15:29
And we're actually at a point now where these models can
15:33
understand a lot of basic sequel syntax.
15:35
And so we're actually hitting into these much more interesting
15:37
hard problems around kind of business knowledge and context.
15:41
They're like, I don't even think we were able to even start to
15:44
grasp that a couple of years ago.
15:45
So, so, So, yeah, I mean these models again are trained over
15:51
pub movement for the most part, very public public data sets you
15:54
could scrape from the web, right.
15:55
So all kinds of SQL queries and GitHub logs and they're messy
15:59
and people have different patterns and different different
16:02
styles of writing queries and you know, different databases
16:04
that they're running over. But these models are kind of
16:06
jackable trades. And so they, they, they're very
16:09
good at kind of learning general, general sequel, right.
16:11
So if you go and like this, you know, I think a classic example
16:14
for chat to BT, you go and you ask a query over very kind of
16:18
canonical, canonical database like.
16:20
And there's some databases that are just, they're armed, they're
16:22
everywhere. Like customers like products, as
16:25
you'll see. And you'll be like, oh, like
16:26
look, it's getting it right. Like it's understanding
16:28
everything. But then the minute you get
16:30
right and that's, you know, and and that's true.
16:32
Like it is especially for again, things that are that are common,
16:36
but businesses have hundreds of tables that are there, you know,
16:40
very kind of crazy columns that are named like nothing but 5
16:43
letters and no one knows what it means except for two engineers
16:46
who built it. And and you know, it's been set
16:49
into all these pipelines. And so it's, it's the minute you
16:53
really touch real customer, real customer data, the models like
16:57
they guess and they're not, it's a smart guess.
16:59
Like it's, it's, it's as if you put me in front of their data
17:02
and asked me to answer it, I'd be like, well, I don't know like
17:04
this seems logical, but it's that as you really just said,
17:08
like they they will not be able to handle the complexity of what
17:11
the engineers, but what's in their knowledge 'cause it just
17:13
hasn't seen any of the company's data, hasn't seen any of their
17:17
query logs, has no idea what's going on and there's no way it
17:19
could cause never was ever trained on that data, right.
17:22
Like, it's a kind of a fundamental limitation of these
17:24
public kind of third party models.
17:26
Umm, yeah, more from a certain. There's other challenges too in
17:30
terms of like wrong contacts and all the stuff that you could
17:34
have seen a lot kind of been been talked about as well.
17:36
But the the business knowledge is probably the core challenge
17:39
to to overcome. Yeah, and and so there are two
17:43
leaps that I think we need to make to get these LLMS to work
17:47
and this complex SQL and and complex business knowledge.
17:51
In fact, right one is being able to customise the LLM to the data
17:56
scenarios. It could be text to SQL, or
18:00
understanding SQL, outputting SQL correctly and also
18:04
understanding what kinds of data is there in that particular
18:09
database or data warehouse, the context around the data when you
18:12
apply for entity resolution. So there's a there's one leap of
18:16
around focusing the LLMS, verticalising the LLMS on that
18:19
data related task 'cause that's one.
18:22
The second is you talk, you go into an enterprise, you go into
18:25
a customer environment and you look at then like Laurel said,
18:28
you look at the entire schema and the kinds of values that are
18:32
there in the database, what kinds of queries are happening
18:35
on that database. So you need to understand all of
18:37
that context and how people are using the data in that
18:41
particular database. What kinds of queries are they
18:43
writing? What is the purpose of each of
18:45
the query. So understanding that, making
18:47
the elements understand those and then asking the questions
18:52
which will actually result in the complex business logic that
18:56
makes it a lot more accurate. And those are the two kinds of
18:58
leaps that we need to make actually, to get the get the
19:02
answers correct. And if you look at the sequel,
19:04
it's there are certain pieces which make the sequel really,
19:07
really long. One is it's like all the case
19:12
statements, all of those things, right?
19:14
So how do you get the LMS to understand and output those?
19:18
So there are pieces of information that can be input as
19:21
knowledge to the LLM and they get the output needed, yeah?
19:25
So it kind of feels like we should go in this direction of
19:30
one-size-fits-all LLMS versus like very verticalized and
19:34
specific smaller I imagine LLMS and how you all are looking at
19:40
it and especially Venki as you're building product or
19:43
Laurel as you're like engineering around this.
19:46
What are the considerations that you're looking at?
19:49
And are you trying to go smaller as I feel like that's the trend?
19:56
Is that let's just be as big as we need to be, but no bigger?
20:02
Yeah. Yeah, yeah, no apps.
20:04
Absolutely. And it's been, yeah, a great
20:07
trend to see because there was that kind of race to see how big
20:10
these malls could get and then people realised that maybe maybe
20:13
you don't need to be. And the real, in my mind, the
20:15
real insight here is that the minute you go into a
20:18
verticalized specific space like text to sequel, if you think
20:22
about it, these big, you know, big models, you know, that are
20:25
on these third party models, they know a little bit about
20:28
everything, right? You can go and talk to them
20:29
about recipes, you, you can go get sequel back, They can, you
20:33
know, help you summarise documents, right.
20:35
Amazing capabilities. But if I'm just care about
20:37
sequel, like I don't need 95% of that information that it's like
20:41
storage, right? Like a lot of that information
20:43
is being you know, compressed and stored these models.
20:46
And so it just also makes a lot of sense that like, hey, look at
20:49
my vertical eyes, I don't need that.
20:51
I can really go smaller while maintaining the capabilities.
20:53
And the minute you're smaller, you're in a space where like the
20:57
we're really at this nice middle ground right now where we're at
21:00
a size like you can host a main team locally, right?
21:03
I mean with with some hardware, but like you don't need like I
21:06
can host one of these models on a single kind of single single
21:11
GP for the most part. Like a very easy, not easy to
21:14
set up, but a more reasonably kind of priced piece of
21:18
hardware. Like I don't need distributed
21:20
systems and all that kind of crazy kind of a lot of, you
21:23
know, you can actually maintain it locally.
21:25
And so this allows enterprises to go and they can start to own
21:28
the models, they can start to really customise them on their
21:32
data because it's in, it's there, it's in their BPC, they
21:34
can host it and then this will really work because we're in a
21:38
fairly narrow task, right. And so you have this really nice
21:41
trade off where like the more verticalized you go, the smaller
21:44
you can go and the more you kind of really take advantage of of
21:47
all the kind of smaller models that we can we can play with
21:49
right now. Yeah, and and so.
21:52
One is yes, small enough just or big enough just to understand
21:57
and ask answer the questions that people are interested in.
22:00
That's one thing. The 2nd is yeah like Lauren
22:03
mentioned the cost of hosting these and and and us being able
22:06
to host these in an enterprise VPC easily.
22:09
So these are what the the verticalization and reducing the
22:13
size of the models helps us. And the 2nd is we can also make
22:17
it a little more accurate because we feed that context or
22:20
the specific scenarios that we're going to ask the LLM.
22:24
And another important one is actually around reducing the
22:28
hallucinations right. So one of the things that we can
22:32
so as a chat GP, there are very one-size-fits-all anyone can ask
22:37
any kind of question and the model has to respond and the but
22:41
in the case of data questions so when we get a question first of
22:44
all we can say oh is this related to the data kind of
22:47
question. If you're asking how is the
22:48
weather out today, we can we have the liberty to say sorry I
22:51
cannot answer that right. So in in the data context.
22:55
So first of all, we need to understand, is this a data
22:57
related question? Is it something meaningful that
23:00
as a model we expect it to answer back?
23:02
So these kinds of what we call the guardrails around questions
23:06
and saying, no, I'm not going to answer that.
23:08
This is that's one thing. The 2nd is that you're asking a
23:11
question, There's not enough information for me to answer it
23:14
accurately, right. So those kinds of things are
23:16
what we can train these LLMS, these verticalized LLMS because
23:21
we know the context, we know the kind of questions that the
23:24
people will be asking and the kinds of responses we have.
23:27
We are made to give it back to the give back to the users from
23:31
a product perspective. And so all of those things are
23:34
what are enabled by verticalizing, reducing the size
23:37
of the footprint, so and and adding additional capabilities
23:40
as necessary one thing. That I can imagine people have
23:46
thought about and I would love to hear your opinions on when it
23:49
comes to the trade-offs of how do I know whether or not to make
23:55
it 2 models that interact well together or just one bigger
24:00
model. And what do you look at when
24:03
you're you're thinking about that and you're building models
24:06
for people? Like should we just make this a
24:09
smaller model and potentially like fine tuning it on a
24:15
different type of model or our own open source model?
24:19
Or should we try and make it a bit more bloated?
24:24
But we know that it's going to catch all of these edge cases
24:27
and we have confidence that it's going to be a better experience,
24:32
even though it's going to have that overhead of being a bigger
24:35
model and I say two models, but we can like extrapolate that out
24:41
to end models, right? Yeah, that's actually a really
24:45
interesting question. And I think my immediate
24:47
reaction would be when you try to deal with one model that can
24:52
do multiple things. I mean, if you're just taking it
24:55
kind of off the shelf and not tweaking it like that, can can
25:00
sometimes, you know, can sometimes work 'cause it is,
25:02
it's kind of slightly more general purpose model.
25:03
But my experience has been if you really want to customise it
25:06
to your data and start to train it, it's actually a lot easier
25:09
from the engineer's perspective to have slightly better tight
25:12
task isolation. At least to start Because
25:14
scoping the problem and because at the end of the day like this
25:18
the problem of getting training data hasn't hasn't gone away
25:21
right like it is, it is still the my opinion.
25:25
The single most important factor to getting a high quality model
25:27
is the data that comes in and like this has been true for
25:30
machine learning and deep learning for years and so the
25:32
more well scoped the task, the easier it is at least from the
25:36
engineer side to go and figure out how to how to turn on the
25:39
model. So from the from that
25:42
perspective like a little bit more of this kind of ecosystem
25:44
of the model makes a lot of sense, right, 'cause then you
25:46
also get a nice separation where like hey look if one components
25:49
performing well and another one isn't, it's much easier to go
25:52
and just tweak and turn this one on rather than having to kind of
25:56
deal with this potentially correlated kind of complicated
26:00
larger model that like potentially you try to change
26:03
its behaviour for one task and you end up degrading forms in
26:05
another task, right. That that can get kind of it's
26:09
just harder to to to figure out and maintain.
26:13
But again, you're right that the system's complexity of multiple
26:16
smaller models potentially is, is is more where you know we
26:20
don't want to go back to these kind of really siloed pipelines
26:22
like we had before. We'd like to kind of stay in the
26:25
still general purpose kind of model ecosystem.
26:28
So it would also be very task dependent as well and kind of
26:31
what what task you want. If you want a model that can
26:34
also be able to like you know, chat body style things, I can
26:38
ask and answer questions about your entire, you know, all the
26:42
tasks you know, then that kind of thing feels also more like 1
26:45
general purpose model. So I'd say it really depends on
26:47
the task that you care about and the infrastructure that you're
26:50
dealing with. And if you need to, you know,
26:52
fine tune or not. But there's no clear like answer
26:55
I don't think either. Yeah, yeah, I mean.
26:59
Just to add to that, it's it's safe to say that we are still
27:02
exploring the boundaries of these things like which which
27:06
are general purpose verticalized LLMS in the data space versus or
27:11
we need to find very small specific models for certain
27:16
kinds of tasks. So multiple criteria come into
27:20
the play here. One is, is a human asking the
27:23
question or is it a system that's asking the question?
27:27
If it's a system that's asking the question in the case of
27:29
entity resolution and so on, well, these can happen millions
27:32
of times in a in a day or in a few hours, right.
27:36
So then we need the system this, this LLMS or these models to
27:40
respond much more efficiently and at a low cost O we need
27:44
really fine tune small models that can be deployed in
27:47
roduction at scale and so on O that's one kind of task which
27:52
Laurel was saying oh this is a task specific small model that
27:56
we need to fine tune for that specific entity resolution task
28:00
not just generic entity resolution, entity resolution on
28:03
that particular customer data and then customer environment
28:07
even, right. So that's the level of fine
28:09
tuning that needs to happen. So we need training data for all
28:12
of those things and and some human help to actually guide
28:15
the, the building of the small models.
28:18
So that's one. But if you like, if you take
28:20
tasks such as SQL or answering a or give me the OR an aggregation
28:26
query like give me the sales on the US West Coast for the small
28:31
and medium business segment in my in my target customer base,
28:35
right. So if you ask such a question,
28:38
it's a human driven question, people are asking for insights
28:41
and that's something that we solve at our customers like a
28:43
realistic customer and insurance customer.
28:45
All of these guys want to these business users want to ask these
28:48
questions and get responses back.
28:50
So these are human questions and there's the rate at which this
28:54
comment is not that that high. So we we can keep the model big
28:59
and the response latency enough for the humans to deal with.
29:04
And that's one kind of it asks and so the models can be bigger
29:08
there and then these same kind of models can potentially
29:12
leverage be used for data transformation task.
29:15
Oh, I have a source schema, can I use that to transform it into
29:19
the target schema? So maybe the same model that we
29:22
learn for the business domain can also be extended or transfer
29:27
learned for this schema mapping kind of a scenario which happens
29:31
once in a while, right? So there are how related are
29:34
these tasks. In that case the transfer
29:36
learning can come in and we can use a model to do both the tasks
29:40
versus one model where there is a benefit of having information
29:45
from one specific task to be captured carried over to the
29:49
other task as well. You have to.
29:51
Immerse yourself in the ML OPS content.
29:54
The best way to do it is to subscribe to the E-mail OPS
29:58
Community Podcast soon. Good luck and keep learning so.
30:10
All of these have to be thought through carefully and I think
30:13
we're just scratching the surface there.
30:16
Makes 100% sense. And especially one thing that I
30:21
think we could probably jump into now is like the end sequel
30:26
foundational models that you all put out because there are
30:29
different sizes of these, right? It's not just one size or it is
30:34
there. I can't remember what.
30:35
What is it? Laurel 8B and A.
30:40
What are the other ones? Yeah, yeah, we have.
30:41
Some, some that are like sub 1 billion and then up to seven 7
30:47
billion based on the Llama one of the Llama two models that was
30:50
released like four or five months ago.
30:52
Yeah, something I can't remember.
30:53
I felt like years ago, but yeah. Exactly.
30:57
And and so you mentioned how the hardest part on this is not
31:03
necessarily what model am I going to use.
31:07
It's like what data am I going to use to add to this model to
31:13
make this model effective and useful.
31:16
And so when you're working with people like, what are some
31:20
challenges that you've seen on the data side, especially when
31:22
it comes to this idea of like using SQL as your data source.
31:29
And I know Venki you mentioned there's these two sides of the
31:34
the problem a little while ago where it's like, how are people
31:38
interacting with the data and then what does the data look
31:40
like, What are the schemas look like?
31:41
So there's there's those two sides of this problem when it
31:45
comes to the context and how you want this model to interact.
31:49
But as far as like training a model for a customer that is
31:55
able to do these things, like what kind of data are we talking
31:59
that we need? And is it just fine tuning or
32:02
are you going and you're saying, you know what, we're going to
32:04
train a whole foundational model too?
32:07
And maybe what are the different sizes or how much data do you
32:10
need at each one of those steps? Can we answer that last question
32:17
first? The more the more you fine tune
32:20
and there's some kind of ML tricks around how you fine tune.
32:23
And like there's some kind of fancier techniques where you
32:26
kind of update subsets of the weights, like the less you
32:29
change, roughly speaking, the less data you need from from the
32:34
base model. You know that's that's
32:36
oversimplification, but and so obviously you're training
32:40
completely from scratch. You can need a lot more data
32:42
from a company and it's gonna be you know more expensive and and
32:46
and and all that. So so there's kind of that that
32:50
spectrum there and and models right now are good enough that I
32:54
would question, it's it's a little unclear.
32:57
I mean unless you're like a really new domain where like all
33:01
all you know in essence like how you talk about things is
33:03
fundamentally different. I think most of the the focus is
33:07
on fine tuning or earliest kind of continually to update models
33:11
that already exist just 'cause there's so much cost and effort
33:14
that goes into that kind of initial trading round that it's
33:17
it's not always it's worth it for companies to kind of repay
33:20
that cost. Now in terms of you're focusing
33:23
on this kind of fine tuning, like I don't want to actually
33:25
customise, you know give a customer a high quality text to
33:28
SQL model for them. The real challenge is, is that
33:32
for the most part the data that we need doesn't exist, right.
33:35
Like they've never, you know the dream scenario which doesn't
33:38
happen is that oh they've like tried to do this Texas sequel
33:42
for years and have massive terabytes of the data lying
33:45
around. But like, no, this is that's
33:47
never going to happen. And so really the like and maybe
33:52
you can elaborate more. Two of the challenges that we've
33:54
dealt with a lot is this idea of how do we walk through the
33:57
process with a customer of saying, hey, look, you typically
34:00
have a bunch of log data, a bunch of dumps of data.
34:02
They actually have a tonne of data out there.
34:04
So how do you help them curate it and then work with them to go
34:09
build kind of, you know, we really need to build, you know,
34:12
kind of evaluation set so that they know what the quality is
34:15
and we know. So at the end of the day, like
34:16
when numbers are good, we're all happy.
34:18
And that's really important so that customers know what they're
34:21
getting and how do you get the data labelled and processed,
34:25
right. Like there is a slew of models
34:26
you can actually use to help with this, get to kind of be
34:28
careful which ones you use and how you do it.
34:31
And then privacy also comes into play a lot, right?
34:32
There's some data that it's difficult to share, but it's
34:36
really like a valuable mind. And so there's kind of a whole
34:39
back and forth that happens around you know how how, how
34:42
easy or hard it is to get access to to to data that matters.
34:46
And so it's it's not an easy, it's not an easy thing and we're
34:49
actively figuring it out. But again, it's probably one of
34:52
the most important things is making sure that the data that
34:54
you're going to walk through with the you know work with the
34:56
customer to get high quality data and understand what the,
34:59
what the goals are. And Justice to add to that, from
35:04
a customer perspective, I think that the two things that they
35:07
care about, let's say are realistic customers or the
35:09
insurance customers they care about, Is it accurate of the
35:13
response that I'm getting, are they accurate enough and can I
35:16
actually get it time to value for that?
35:19
OK. These the number station comes
35:21
into our environment and from that point till the time when
35:26
business users or entity resolution jobs can actually
35:29
start using those models, how much, how long does it take?
35:33
Are we talking once, are we talking like typical bi products
35:37
out there which take months for anyone to actually prepare the
35:40
layers so that the BI tools can sit on top?
35:43
Are we talking days to weeks, right.
35:45
So our promise to this customers is yeah, we'll we'll be up and
35:49
running in days to weeks. And the place where we actually
35:53
innovate here is, is 2 two points.
35:55
One is we take A1 size fits all like LAMA model.
36:00
We verticalize it to the data domains based on the public
36:03
information that we have on GitHub or on the web and so on
36:05
and also Tableau web books, those kinds of things.
36:08
So we verticalize that and we already have an off the shelf
36:13
number station verticalized model which understands what
36:17
data context means and what kinds of responses it needs to
36:20
give, right. So that model is already there
36:23
at the base. Now we take that and we go into
36:25
the enterprise environment and and going back to my violation
36:29
there, So what is the context of the data?
36:31
It's a schema. It's the queries that have
36:33
running over the schemas and it's the reports that have been
36:36
constructed over these schemas, right.
36:38
So all of this define how people are using that database in the
36:42
customer. Now we need to give that context
36:44
to this number station based data LLM and it's called it a
36:48
pre training phase which may take means which may take a few
36:52
weeks and after that it's ready to be used for that specific
36:56
context and the specific enterprise customer, right.
36:59
So that's that's the way we approach it and it reduces the
37:03
time to value to to weeks rather than like months, right.
37:07
So that's the place and then like Laurel said, we find
37:10
continue fine tuning after that. So this context is not static,
37:14
it's continues to evolve. So we need to continue the
37:18
tuning of this private data and that we've built for that
37:21
specific enterprise. So that's the ongoing activity
37:24
that we need to do. OK, cool.
37:26
So I'm going to change gears and go in a bit of a different
37:29
direction because since I have both of you on here and I know
37:35
you're thinking about the LM space and the foundation of
37:38
model space daily, maybe hourly potentially, I would love to
37:45
know where you think things are going and what is going to
37:50
change in the next 6 to 12 months.
37:55
Yeah, no, I I am very excited about this.
38:00
I think I mean you and you're seeing a little bit of this with
38:04
I was actually like just chatting with a friend yesterday
38:07
about some of the new like Apple Mac books that just got released
38:09
that have you know like like listening to like you find
38:12
memory but but basically I think what we're seeing is we're going
38:15
to start to have a lot more not to say commodity hardware, but
38:19
it's going to be a lot easier for companies and organisations
38:21
and even individuals to host and potentially even train local
38:26
models on laptops on desktops like.
38:29
So you know the, the, you know you're still going to need
38:32
really expensive hardware to like really train these things
38:34
from scratch. But I think we're going to see
38:36
this era of potentially you know personalization and and
38:40
customization of these models. We just don't see right now
38:43
where you can again have a model that stays and lives on your
38:46
laptop the entire time and that's a that's a pretty
38:49
exciting scenario, right. You can get customised to your
38:52
own for example for engineers like customised to how you code
38:55
rather than just generic you know coding patterns in general.
38:58
And so I think that's one trend that we're going to see a lot of
39:01
is these kind of smaller models that that do a lot that do it
39:05
all locally and that's that's also going to be very exciting.
39:07
I also think we're going to start to and we're seeing a lot
39:10
of work on the academic space slowly start to tease this
39:13
apart. But a lot of the like it was my
39:16
personal take, but a lot of the advantage opening I had put into
39:20
the arguably still has a little bit of is the quality of their
39:23
data, how they were getting their training data.
39:26
You know there's been some articles about you know the the
39:28
massive amount of data labels that they have to go and kind of
39:31
curate these these very high quality training data sets for
39:34
these chat models. And I think that Moat is already
39:38
closing and I think will be closed in the next 6 to 9
39:40
months. I think we're going to start to
39:41
crack open these kind of quote unquote data recipes so that
39:45
it's going to be a lot. So in essence you're going to
39:47
get open source models, these cross fingers crossed that are
39:51
very much as performant as escalators, private third party
39:54
ones which again opens the door for a lot of training and tuning
39:58
and and and things. So just really democratising the
40:00
access to them. So there's a kind of two trends
40:02
I see. I mean I may be wrong but we'll
40:05
we'll we'll see what happens. Wait.
40:08
On that last point on the data, recipes being democratised or
40:14
basically Open AI not having that edge as much because the of
40:19
the data that they have, you don't think.
40:21
I I would have thought it's going the other way where since
40:24
they have so many users and they have so much feedback happening,
40:28
they are and they're getting just so much input daily, they
40:33
are continuing to curate that data set that they have that I
40:37
mean. True, I think I do think they
40:39
run into some issues where they can't always use the the the
40:42
user data directly anymore. Like it it, you know, I think
40:45
they can't, I mean who knows what they were doing originally.
40:47
But you know, I don't think they're allowed to directly
40:50
train on on, or at least you can certainly opt out I believe of
40:53
training directly along. I think it's certainly a useful
40:56
signal, but I think also some some limitations are there.
40:59
But yeah, I mean it's so, I mean yeah, you just sure.
41:02
But I also like, I just think if you look at the meta, the Llama
41:06
models for example kind of, I mean I think in my mind they
41:11
prove that that's not, that is not necessarily necessarily
41:14
true. Again, I do think there's like a
41:16
lot of polishing that goes from like 90% of the way there to
41:21
like 98%, which is a lot of this like chat Polish and that
41:26
certainly Open AI has has a lot of advantage on.
41:30
But I'm not, I'm still very unconvinced that it won't, that
41:34
we won't figure out 'cause I I still don't think they have 100%
41:37
no either cause a lot of it's like, do you think about the
41:39
web? A tonne of web queries are kind
41:41
of messy and not really useful and something just kind of
41:45
capture what what is useful. So, so yeah, they certainly have
41:49
a lot of customers and users but also you can imagine there's a
41:51
tonne of other people using things, using things open as
41:54
well. So we'll we'll we'll see.
41:56
I'm not I'm not convinced that that is that that's actually the
42:00
the the case but but maybe we'll we'll see.
42:04
Yeah, I think. My my view on the on this one is
42:07
it's both will progress advance both the open AI the data
42:12
creation will continue to advance and and use whatever
42:15
usage they have and then gain benefit from it.
42:19
And along the same lines there is demand for open source models
42:22
and and to advance both advance the technology itself.
42:26
So there'll be a lot of research going on and contributions from
42:28
the open source to advance that class of models.
42:31
The I'm sure pretty soon we'll see how to either gather this
42:37
kind of curated data tooling around open source tooling so
42:39
other people can contribute to enhances open source models.
42:43
Perhaps there will be research that that will say oh this is
42:46
how we can actually create high quality data from noisy
42:49
information and so on. So there will be a lot of people
42:52
that will be contributing into this into this both sides of the
42:56
coin whether it's proprietary open AI like models or or open
43:00
source meta LAMA kind of models, right.
43:04
So both will advance from the technology perspective and from
43:08
the customer side. I think the demand for private
43:12
models that I don't want, I don't want to send all my
43:15
contacts and IP for a customer, an enterprise customer or even a
43:19
SMB customer, Their data is their IP, right.
43:21
So that's what they own and their value is embedded in that
43:26
one. So the demand for private LLNS
43:29
that they can deploy in house and not lose their IP to someone
43:34
else outside, which is common across all vendors is is going
43:39
to increase of that demand will drive a certain kind of
43:42
development like like Laurel said, being able to host small
43:46
models infrastructure will be driven to be able to host these
43:49
kinds of private elements with a large or small in VPC.
43:54
So that demand will continue to grow in the next 6 to 9 months
43:59
and we're already seeing that, I mean among our customer base.
44:01
So we want private models and that's where we derive we, we
44:05
deliver the value there. Yeah, and there's.
44:07
One more trend actually I want to would I?
44:10
Yeah, I kind of want to move from a technical standpoint, but
44:14
also actually some really interesting implications that we
44:17
can pull it off is we're getting into the space with much with
44:19
models that have much longer context length and like you're
44:23
seeing with some of the meta Llamas or metal almost, umm, so
44:27
the the code models, they have up to 100,000 tokens and roughly
44:31
you can roughly think of a token as a word.
44:33
It's not quite that way, but close enough.
44:36
And so there is this kind of hypothesis about if you had a
44:40
model with enough with a massive, with, you know,
44:42
millions of contact lanes, which we're not there yet but soon
44:45
maybe do you even need to fine tune anymore if you can
44:49
basically take a training data set and insert it into the pro.
44:52
So there's some interesting injury artefacts with these long
44:55
context models where potentially we can need less and less
44:57
training, which is I don't really know exactly what that
45:00
would imply, but kind of an interesting scenario that we're
45:02
also touching on. So you're going to see also more
45:05
long context models come out because it alleviates the burden
45:07
of having to to tune. You can put more context in
45:10
there and and customise the models more more easily without
45:13
having to to update them, yeah. Have you played around with
45:17
Claude and like the 100,000 context window?
45:21
Because I remember when it came out, a lot of people were like,
45:24
well, there goes the whole vector store, that's we don't
45:28
need vector stores anymore. Because of that, it feels like
45:32
vector stores haven't really gone anywhere and the 100,000
45:37
context window, It is nice, but it's not necessarily better for
45:45
most use cases. And it goes back to that
45:47
tradeoff that we were talking about earlier almost it's it's a
45:50
parallel I would say on when do you use a bigger model and when
45:54
do you use a smaller like very focused model.
45:57
And so now we're adding like a new spice to the mix of when do
46:01
you use like a bigger context versus when do you use a
46:04
fine-tuned. And so there's all these knobs
46:07
that we're getting to turn now. And because we get to turn these
46:10
knobs and say, well, this one, I want to be a very super focused
46:15
model that is on a certain flavour of SQL or I want this
46:19
coding model to be on Rust, only Rust or things like that versus,
46:26
oh, I want this to be all of SQL or all of coding, et cetera, et
46:31
cetera. And I want it to have a gigantic
46:32
context window so I can just throw my whole GitHub like repo
46:39
on there and I'm good to go. So it is fascinating to think
46:43
about, like what those implications will have down the
46:47
line and how we'll be balancing them.
46:51
I think a good. Parallel to consider, here is
46:52
how the database technology evolution happened.
46:55
Right. So there's one kind of database,
46:58
one Tye of workload and soon people started realising we can
47:01
do analytics on top of it and the way it got the Ola kind of
47:05
engines and then slowly the Cloud engine, Snowflake and
47:08
column stores preceding those, All of those that evolution
47:12
happen and the and the and the segmentation of the workload.
47:15
So OLTP workload, analytics workload and then you have this
47:19
unstructured databases data with unstructured documents and so on
47:22
like Mongo DB, right. So all of this workload
47:25
expansion happens and then the technology evolves to catch up
47:29
with all of these things. So similarly today we have LLMS
47:32
is 1 size fits all or one kind of LLMS and we are thinking of
47:36
workloads as one kind of workload, but the workloads are
47:39
also multifaceted here multi different types of workloads and
47:42
you'll have different either LLNS or finer models or
47:46
fine-tuned models, small models coming out of those LLMS which
47:49
will attach address these kind of workloads.
47:52
So we'll see a similar expansion of the types of LLMS, types of
47:58
fine tuning. All of those things will evolve
47:59
soon. Oh.
48:02
I love that analogy there, and I hadn't thought about that.
48:05
And it makes complete sense on the more that we explore, the
48:12
more workloads we're going to see and the more flavours of
48:14
these LLMS we're going to see. And it's very obvious, I think
48:18
for all of us and anyone else who's been playing around with
48:21
LL Miss that it's not going to just be GPT 4 that we do
48:26
everything with. GBT 4 May be what we test with
48:31
to see if there's a viable solution and see if our thoughts
48:35
or our MVP actually makes sense. And as soon as we see that, OK,
48:39
this is, this is kind of good, we could try and get there or
48:42
yeah, this does pretty well. Then you go and you figure out
48:46
how to really again turn those knobs and make it very, very
48:53
tailored to your specific use case.
48:55
I think there was a. Blog post by Chris Ray at some
48:57
point the last mile of the AI or some such thing, which area was
49:02
like oh the Foundational models LLM will get us 8090% of the
49:06
way. But the last mile accuracy which
49:09
is demanded by many enterprise applications, that's where you
49:13
need to actually invest effort into pre training those things,
49:16
the new verticalized model, the custom model and then fine
49:19
tuning those to the specific tasks that that that accuracy
49:22
demands, those kinds of steps and evaluation and also the time
49:27
it takes like going back to your cloud example of 100,000 tokens,
49:31
yes, that's good, but how long does it take to do the inference
49:33
right. So some of these scenarios you
49:36
need to respond within less than a second.
49:39
In that case you cannot really exploit that 100,000 token
49:42
limit. You need to have a smaller
49:44
context line. So, so all of those kinds of
49:48
workloads, scenarios that you have, all of those, if it's a
49:51
bad job, I don't care about how long each one calls and it
49:54
takes, then yeah, it may be a different, different
49:58
application. Yeah, I've heard about this idea
50:03
of the AI Gateway. I'm not sure if you you've
50:06
played around or or heard people talking about that too, where
50:10
it's like depending on the use case and this workload, you have
50:14
some kind of gateway that you hit first and then it will route
50:19
it to whatever workload you need instead of saying here's a
50:24
verticalized workload. It's very much like, OK, I
50:27
understand that this needs to be a larger context window and then
50:31
it will route it to that. I also like I was speaking to
50:35
the creator of DSPI and I'm not sure if you're familiar with
50:41
that. It's like an orchestration
50:43
layer. And the way that they think
50:46
about bringing all these pipelines into effect, I think
50:52
is is also very novel and it has a way of abstracting it away, a
50:59
way of abstraction that is very useful for the end user.
51:04
And so the last thing I wanted to touch on because you
51:09
mentioned this last mile is around the way that you do
51:14
evaluation. And Laurel, you mentioned, hey,
51:17
we have to make sure that we're evaluating things and and we're
51:21
getting a good result when we are giving customers different
51:25
text to Sequel models. Have you curated or is it on a
51:31
case by case basis, like how do you do evaluation and what are
51:34
you looking at? Like, walk me through that whole
51:37
process because I know that's something that a lot of people
51:40
are struggling with right now too.
51:41
I mean. Typically these tasks that
51:43
people really are caring about now are multifaceted, right?
51:46
Like it's it's they both want some kind of free form text
51:49
result and then often times that'll lead into maybe some
51:52
kind of some some kind of more classical like classification
51:56
tasks or like action tasks where the do something or call an API
52:00
or things like that. And so you know when you have
52:02
something that's a little bit more well defined like a
52:05
classification task, you know it's it's a little bit easier to
52:07
measure, right, because you're just dealing with sort of more
52:10
traditional standard labels And so obviously you want to capture
52:13
capture all of that. It gets really tricky when
52:16
you're dealing with with the outputs of free form text,
52:19
right. There's there's metrics that
52:20
people have to like compare you know for example summarization,
52:24
right. Like you know, there are some
52:25
some ways that people can measure roughly measure the kind
52:28
of quality of it, but the standard metrics are very are
52:33
not great, right. They're they're they're they're
52:35
fuzzy. And so in terms of that like the
52:38
the best kind, I mean we're still learning and figuring out
52:41
how to how to do it. I would say the end of the day
52:44
as long as there is typically there's some actionable output
52:47
that the user cares about like for text to sequel it's really
52:50
hard to actually compare 2 sequels.
52:52
You can but it there's like you know you could take a sequel
52:55
query and rewrite it 25 different ways and you're doing
52:58
with parsers and and stuff which again those tools exist.
53:01
But at the end of the day what they really care about is does
53:03
the output table match right. And so like you can kind of
53:07
start to move if anything just treating the model output as an
53:10
intermediate to like the actual final result of the user cares
53:13
about. And so it's it's not the best
53:15
way to it's it's not I, you know it's it's it's noisy but at
53:18
least it starts to help you measure some sense of is the
53:22
model doing what you want. But that's something that you
53:24
need to interact talk like talk with the customers like what do
53:26
they really want out of it is that they really care about the
53:29
text. They care about the actions that
53:31
can happen from it right. And so it's it's a it there's
53:33
not A1 size if it's all evaluation either like it's very
53:36
custom depending on on what you're doing.
53:39
But is is is one of the most important first steps of any
53:43
kind of project on this route for sure and for a.
53:46
Certain class of Sequel, you can automate that comparison.
53:49
So it helps to constrain this Sequel output to a certain like
53:54
subclass of all general sequels. Awesome.
54:00
Well, you know, this has been incredible.
54:03
I thank you so much for coming on here.
54:05
I'm a huge fan of what you're doing, especially with like N
54:09
Sequel and everything at Number Station.
54:12
It's always a pleasure to get to chat with you all.
54:15
And so I encourage anyone who is trying to figure things out and
54:21
if they like anything that they heard here to hit you up and
54:26
then I think we'll end it there unless there's anything that you
54:30
wanted to say. Any questions that I didn't ask
54:33
that you wish I would have asked?
54:36
No, lots of questions, but yeah, thank you.
54:38
Really, really enjoyed being on the on the party.
54:40
Yeah. Thanks, Dermatos.
54:42
Yeah. Thank you.
54:47
Hey, this is Mike. Dub, also co-founder and CEO of
54:50
Tecton ML OPS Community, is the best way to stay in the loop on
54:54
the latest ML OPS news and best practises.
54:57
It's also a great way to connect with experts and get support
55:00
from an amazingly helpful community.
55:02
Subscribe and stay in the loop.