Latest / Agentic Conversations (formally mlops.community) / The Semantic Layer and AI Agents // David Jayatillake // #343

Transcript

The Semantic Layer and AI Agents // David Jayatillake // #343 · Agentic Conversations (formally mlops.community)

0:00
Yeah, I mean, I mean, that's why I believe in using a semantic
0:04
layer in the way anyway, right? I don't want them LM to write me
0:08
raw sequel every time having to do it once like when it's
0:13
building the semantic layer and then you reviewing it and
0:17
approving it and then it reuses it over and over again.
0:19
That's fine, right? 2 days ago I opened up one of my
0:29
e-mail newsletters that I subscribe to.
0:31
It's called Exploding Topics or Exploding Trends.
0:35
I can't remember what the name is, but in it, you know, what
0:39
was hilarious was that one of the trends that is more and more
0:45
popular by the day is something called the semantics layer.
0:49
So I saw that and normally it's like talking about, oh, this
0:53
trend of CBD gummies or whatever, like magnesium
0:58
gummies, that's a new trend now. Or hair loss supplements type
1:04
stuff. Those are trending and they just
1:06
look at Google search results and I imagine now they're taking
1:11
other rich data sources to look at.
1:14
But the thing that blew my mind was I go and I look at it and I
1:19
see semantic layer is now a trending topic.
1:23
And I laughed because I was like, all right, cool, I'm
1:25
talking to you. And I feel like you know all
1:29
about that. Yeah, that's definitely
1:33
something I I've spent a lot of time thinking about.
1:36
And it's definitely the semantic layer that I talk about.
1:38
It's like some other semantic layer.
1:40
Yeah. Oh yeah.
1:41
No, it's for sure. That would be hilarious if it's
1:45
like, no double entendre, it actually means two things and
1:49
it's not the same semantic layer.
1:51
Yeah, because I I had to double click on it and think like, is
1:54
this the same semantic layer? And the way they dumbed it down,
1:58
I thought was kind of nice. And I'm now going to start
2:00
saying that it was like, this is how the nerds make sure that the
2:05
data is readable for the non nerds type thing.
2:10
I was like, yes, I'm totally in. Yeah, that's a fun way to
2:16
describe it. But you've been doing a lot of
2:19
work in the semantic layer and just in data in general.
2:23
Can you give me the background on when you first got your?
2:28
You found that the semantic layer was your true love.
2:33
I it's not that I wouldn't say it's my true love.
2:35
I just, I just realized that it's needed.
2:39
And I think I, and the thing that is funny is I keep seeing
2:43
people say, oh, no, we don't need it.
2:45
And I'm thinking everyone uses 1.
2:49
You just don't necessarily know you use one because that if you
2:53
don't think you need one because you don't need the software,
2:56
that means you're the semantic player.
2:57
You're the human semantic guy. And I've spent big parts of my
3:01
career being that human semantic.
3:03
So I, I know that there's always one.
3:06
It's just, is it in someone's head or is it codified
3:10
somewhere? So inefficient for me in
3:13
someone's head, but it makes you feel very powerful.
3:18
Yeah, it doesn't make you feel useful.
3:19
Like I remember being a junior analyst and all these different
3:24
senior stakeholders would come up to me and say, hey, can I
3:27
know this or what's this? And I, because I knew the data
3:30
model, I could write any, you know, any reasonable sequel
3:34
query that could answer their questions.
3:36
And so, yeah, that was cool. But you know, it like one week I
3:40
did 77 of those, like, because I had tickets to do them.
3:43
And I was like, I think this could be automated.
3:46
I don't think a human needs to do this every time.
3:51
So how have you seen semantic layers done well and semantic
3:55
layers done poorly? So, and there's a lot of context
4:01
about this. So one of the things I've
4:04
already written about a bit is I don't really like semantic
4:09
layers that are kind of baked into BI tools.
4:12
And I think for the readers, sorry, the listeners of this
4:16
podcast, that would probably resound with them because they,
4:19
you know, that that logic gets locked into some like
4:24
proprietary platform like. And so we've seen this with
4:26
tools like Looker and even with tools like Power BI where you
4:32
define your semantics in their proprietary DSL.
4:36
So it could be like, look, it's not usually YAML, but like with
4:40
Power BI, it looks a bit more like SQL.
4:42
And so then that you've put that logic there and it can't be used
4:47
anywhere else. Now if you're a small company
4:49
and you're only use for that semantic layer is reporting and
4:54
you're only going to report with that tool, maybe that's fine.
4:57
And actually maybe that's perfect because then the
5:01
semantic layer and the thing using it are developed together,
5:05
designed to work together perfectly as much as humanly
5:08
possible. And it works well.
5:10
And we've seen that's part of the success of why BI tools and
5:13
semantic layers succeed. But if as soon as that's not
5:20
true, like as soon as you want to use that semantic layer for
5:23
something else, like maybe you want to pull metrics into like
5:27
an ML pipeline or into a data application.
5:30
Or maybe you have multiple BI tools or multiple ways of
5:35
consuming data and you know you've got people using Tableau,
5:40
Power BI, Excel, whatever else to access the same data.
5:45
If then that logic is locked into one platform, then you'd
5:51
have to duplicate the logic across every other platform.
5:55
Or you just maintain that separately and then you have
5:58
different versions of the truth. Either way, you end up with
6:00
different versions of the truth, and then you've kind of defeated
6:04
the object of or part of the object of having a semantic
6:07
layer. Now that resonates 100% with me.
6:11
It's it's like we spend so much time, time trying to avoid
6:16
vendor lock in with technology. And this is something that
6:20
potentially we don't necessarily think about on the vendor lock
6:24
inside where it's like, oh, if the only place that we can
6:28
translate the data is within this technology, It's not only
6:33
that we are committing to this tech way more, we are now like
6:42
beholden to everything that they choose to do.
6:47
And as we grow, we're going to feel it more and more.
6:50
And you're just, at the end of the day, building that technical
6:53
debt. 100% And so if you know that that's coming and you, you
7:01
know, as a especially as a team, that's maybe a better resource.
7:04
I just think it's a very bad idea.
7:06
Like you, you can make better choices.
7:09
And then I'm assuming the way to do it is to abstract it out and
7:14
make sure that you don't have any semantic layers within the
7:19
specific tools. Yes, exactly.
7:21
And so you so where I used to work was a company called Cube.
7:25
Cube looked after an open source semantic layer.
7:28
That semantic layer is hosted elsewhere and therefore, you
7:33
know, you could have, you know, like React apps using that
7:39
semantic layer, you could have BI tools using that semantic
7:42
layer, you could have ML pipelines using that semantic
7:46
layer. You could have AI applications
7:49
using that semantic layer as well.
7:51
And so that, that is the principle I, I think is correct
7:56
in terms of the way to deploy this.
7:59
But yeah, it, it, it, it's not without its drawbacks.
8:02
You know, there's very little incentive for the BI tools to
8:05
integrate nicely with that semantic layer because they like
8:09
the lock in it gives, makes their customers stickier.
8:13
So you in order for that to work, you end up either having
8:17
to build that integration yourself as the semantic layer.
8:20
So like if it's an open source BI tool or you have to pretend
8:25
to be someone else's API. So like with Microsoft, you can
8:28
pretend to be a DAX or MDXAPI and then those tools like Excel
8:33
and Power BI can use, you can pretend that you're their
8:36
system. And yeah, so it's, there are
8:40
drawbacks as well, but it's, it definitely stops you being
8:42
locked in. And you know, you, you, you
8:45
don't want it. You know, open source is
8:47
important as well, right? Because fundamentally, if
8:50
someone doesn't want to pay for a cube's paid offering, they can
8:52
just use the open source one. And you know, that's, that
8:57
that's freedom as well. Like that's important for for a
9:00
team. Now I want to get into some of
9:03
the work that you've done and the writing you've done around
9:06
agents and like headless BI and semantic layers and where that
9:12
all fits in, in almost like this new paradigm.
9:15
But I also want to just close the loop on this and ask how in
9:21
your mind do semantic layers and data modeling fit together?
9:27
Yeah. So they fit together very
9:29
tightly, but they're not the same.
9:31
So with data modeling, it's about arranging your business
9:38
data or company organization data so that it cleanly and
9:44
clearly describes the activities and entities in your company.
9:49
So you can have, so you should have entity tables that describe
9:54
any given entity, whether that's like a customer, an order, or a
9:57
product, and then possibly further models that describe the
10:03
state change of those entities. And then you also have event
10:06
type tables or fact type tables as some people call them, which
10:09
describe interactions between the entities or creations of the
10:13
entities by others. And that's like what I'd call
10:18
data modelling. And there's different methods.
10:20
So, you know, there's quite famous ones like Kimball and
10:23
then like bit more obscure, but becoming like more used like
10:28
data Vault and things like that. And I'm not like beholden to any
10:33
of those, like, I think, you know, choose which one makes
10:36
sense given the situation. I'm not even against like what
10:40
one big table some people call it.
10:42
It's where you have like, and I think this is quite popular
10:45
sometimes in like the data science where I just have like a
10:47
very big wide table with which just has lots of different
10:52
features on it. And, and I think there's a
10:54
natural reason why that is because of how that fits into
10:57
pipelines, but I'm not against that as well.
11:00
Sometimes it makes the most sense.
11:01
And if it's not hurting you, go for it.
11:03
But yeah, they that's, that's what I consider data modeling.
11:07
And then what the semantic layer is, the semantic layer describes
11:12
that data model in terms of, oh, here are all those things in
11:15
that data model. How do they join together?
11:17
How do you use them? How do you aggregate them?
11:20
Which ones are like things to group by?
11:23
So you've got dimensions. And So what that then allows is
11:27
a very simple API which is quite close to natural language,
11:31
whether that's in REST or graph QL or something else.
11:34
And it what what that allows is someone to say, well, here's the
11:38
menu of things in the semantic layer.
11:39
I don't need to worry about that data model or how it fits
11:42
together or how to create or how to write SQL.
11:44
I can just ask for the things I want.
11:47
It will compile my request of which is a list of items, into
11:51
quite complex SQL sometimes and run it and give me back my
11:56
answer and my data. And that's that's kind of like
12:00
the difference between data modeling and semantic layers.
12:03
They they're very tightly linked, and good data modeling
12:07
makes using a semantic layer a hell of a lot easier.
12:10
But they're not the same thing exactly.
12:14
So it's almost like start with data modeling and then get to
12:18
the semantic layer. Exactly.
12:20
Yeah, But you can do them like you can go top down as well,
12:24
right, Because actually the whole point of a data model is
12:27
to be queried. And so if you're querying it,
12:30
you're querying it through a semantic layer, whether, like I
12:33
said before, whether that's human or bought software.
12:36
And so if you think about how you want that data model to be
12:40
queried, that really dictates how that data model should be
12:42
built. Yeah, it's not.
12:46
It feels a little bit like the more you query it, the more
12:53
you're going to see the best ways to create the data model.
13:00
Almost like the more that you take a certain path, that path
13:04
is going to get well defined. When you're walking up the
13:08
mountain, it's very clear that here's one trail, there's
13:13
another trail there, there's another trail, and they can get
13:16
you to different parts of the mountain.
13:18
And I think this is where data engineering goes a bit wrong
13:22
sometimes is where they just start.
13:24
And they just start, OK, we're going to extract this data and
13:27
you don't really know what you can do with it.
13:29
And then you just start processing it.
13:31
And then only like you get, they get to halfway and oh, I kind of
13:35
need to know what I'm going to do with this before before I can
13:38
continue. And yeah, that would have been
13:42
much easier. And you could have done less
13:43
work upfront if you actually figured out firstly what you
13:46
were going to do with it and how it was going to be you.
13:48
Might not have needed all that data.
13:50
Yeah. Well now, how does how do agents
13:54
fit into all of this? So I think agents can fit in in
13:58
a bunch of ways. I think agents, if you think
14:03
about like data pipelines for a for a start, a lot of what
14:09
companies do there is really wrote like just setting up a
14:14
connector. And this is what a lot of data
14:16
engineering teams spend a lot of time doing is looking after
14:19
connectors and you know, extractions from databases,
14:24
third party systems, which are databases in a wrapper, right?
14:28
And they just spend a lot of time dealing with that because
14:32
they kind of reinvent the wheel every time they go about it.
14:35
So either they're setting up five Tran connections for the
14:40
15th time or they're building their own custom connector,
14:44
which is again, something someone's already done for sure.
14:47
And actually if you had an agent, I think agents would be,
14:51
I haven't, I haven't tried this, but I know other people are
14:54
working on it or have built things like this.
14:57
Building an agent that could build connectors is very much
15:00
achievable because there is no difference between how one
15:04
company another company should just build that basic connector.
15:07
Because most of the time people don't pick certain fields, they
15:10
just take everything from that source and dump it into like raw
15:15
storage. This is that the ELT way of
15:17
working which seems to be the norm today.
15:20
Not a lot of thought goes into it.
15:22
No. So why, why does you know you
15:24
should be able to ask an agent, hey, connect to my sales force
15:28
and pull my data here and into this like S3 bucket, right?
15:33
That's that's why shouldn't an agent be able to do that?
15:37
And why does data engineer have to?
15:39
That's it. I just think that can be
15:40
automated. Then from there, I think there's
15:45
transformation. So like typically people use DBT
15:49
or SQL Mesh or something like that today to do that.
15:52
And I think even there a lot of transformation work is right as
15:55
well. Like you think about what it
15:58
takes to get from raw to staging data like that is almost always
16:03
the same. It's like deduplication, it's
16:05
casting and dealing with null values, all that stuff that is
16:10
known, right? It's a recipe to follow.
16:13
We could definitely have an agent do that stuff.
16:16
And then then it gets a bit harder where I think you're
16:19
doing things like identity resolution and you're starting
16:23
to build those models to describe the business world.
16:27
Now, if you think about that as something that could be
16:31
anything, I think that's a bit hard for an agent.
16:35
But if you think that actually most businesses should be asking
16:37
the same kinds of questions and doing the same things, then that
16:42
actually dictates that their data models are quite similar.
16:44
And then you're just conforming different sources of data to the
16:48
same kind of data model. And so if that latter latter way
16:52
of thinking is true, then I think you can have agents look
16:56
after transformation as well. And so once you've got that data
17:00
model, and now this bit I have done, building a semantic layer
17:04
from a data model is very easy. Also in using an agent, I've
17:07
built agents to do that. And even from quite bad data
17:14
models where you've just given it like a query that someone
17:18
would run to get an answer, but the query exposes the joins and
17:23
the entities of the data model, it can learn from that and build
17:26
a semantic layer. So I know that's possible, and I
17:28
know then having an agent query that semantic layer as possible.
17:33
I built, you know, a startup before, yeah.
17:37
So if I'm understanding this correctly, the way that you're
17:45
looking at it is instead of or. Let me, let me try and compile
17:53
this in my mind because I, I think you said something pretty
17:58
important there. The way that you're looking at
18:02
business is there's going to be a certain amount of questions
18:07
that every business is going to have and they're going to want
18:11
data that can answer those questions.
18:15
So if you can know what those questions are, you can have an
18:22
agent compile the data around it.
18:25
Yeah, exactly. So you know what those questions
18:28
are, you know this therefore that dictates the semantic layer
18:31
you need because the question is like, you know, this is a basic
18:35
question, but like showing me revenue by this channel and by
18:38
those products filtered by this, right?
18:40
That when you have a question like that, that dictates a
18:44
metric and a bunch of dimensions and then some number of
18:47
entities. So those are the things that
18:49
need to then be in the semantic layer to answer the question.
18:53
And that's very closely linked to what should be in the data
18:55
model. OK.
18:58
That makes sense. I think I guess the yeah, the
19:02
key piece there is giving the agent that glossary of what you
19:09
mean, what those key terms are when you say XYZ, what does that
19:14
actually mean for this company and specifically in the data?
19:20
Yeah. On the other piece, I think
19:25
there is something worth saying when you were talking about how
19:29
it's like the connector side of things, maybe the hard part on
19:34
that as I was thinking through, all right, grab my Salesforce
19:39
data and then just dump it in here or wherever, whatever the
19:43
connector may be, the permissions issue is probably
19:47
going to be the biggest piece. Like 100%.
19:50
That's probably where the the connector is going to want to
19:55
make sure that you have the right permissions for the data
19:58
that you're getting and that agent that creates that
20:02
connection or that agent that pulls that data automatically is
20:06
going to need to be equipped with that right permission.
20:10
Well, potentially, or you could kind of guide someone and, and
20:16
I've seen this in ELT tools like 5 tram where they have a very
20:19
long step by step guide. Here's how you create a role,
20:23
here's how you assign it. Here's the permissions you need
20:26
to give to the role. OK, bring, get that token, get
20:31
the secret and then give me those things, right.
20:36
And so that that's, that's what needs to happen, right?
20:41
It's probably difficult for an agent to then go and touch those
20:43
systems and get those things right.
20:45
That's probably difficult and probably breaks a bunch of
20:49
security rules. Whereas you just, I think that
20:54
bit is a bit any human could do, right?
20:56
Any, any human who has access that system as an admin can do
20:59
that. And it's once, right?
21:01
That's not a huge amount of work.
21:04
Yeah, Yeah, I can see that. Then on the other side of it
21:10
saying, OK, now that we have this semantic layer and we can
21:14
create or we can have the agents link up and create the semantic
21:20
layers. Is that what I understood you
21:22
said? Yeah.
21:24
So once you've done that, because it because it's like
21:28
Henry Ford model of like data production, right.
21:31
And so everyone's doing a little piece.
21:34
There's the agent doing the connector and running the
21:38
extractions. There's the agent doing the
21:40
first bits of transformation. There's maybe an agent doing
21:43
identity resolution and some other data enrichment.
21:47
Then there's the agent building the exposed data model for the
21:51
semantic layer, the agent that builds the semantic layer and
21:53
maintains it, and then the agent that queries it.
21:56
So you just try and get this collection of specialists.
22:03
Yeah, I wonder, have you tried to do this just without creating
22:10
a specialist agent? I've done the latter behalf of
22:19
that with agents. I haven't tried to necessarily
22:23
do it all with one. It's possible, yeah.
22:29
Especially like if you think about build the data model,
22:31
build the semantic there, that last two steps could, you know,
22:35
those could easily be joined. But then where you're actually
22:40
doing something quite different, like where you're doing the
22:43
extractor and just saving the data, the difference between
22:48
that and then doing any transformation and building span
22:52
today, those are quite different activities.
22:54
And like having an agent that could do all of it feels like
22:58
you're asking too much. Maybe you could, I don't know.
23:02
Yeah. Or maybe it's that you're just
23:05
spending more time on the tools that the agent has accessibility
23:10
to it. As opposed to saying I'm going
23:12
to build an agent that is specifically for this.
23:15
I'm going to build tools that the agent can call upon and I'm
23:18
going to make sure that those tools can accomplish these
23:22
tasks. As you went through and you
23:26
built your agents, what were some of the hard pieces around,
23:33
like building an agent for a semantic layer?
23:39
The hard pieces there were context.
23:43
So if you have great context, you can build a very good
23:46
semantic layer and and actually a good data model as well.
23:50
The so you, it's about feeding that context into the LLM in a
23:57
way that's concise and rich and fairly complete.
24:03
And you know, I've noticed that you kind of wanted to be as
24:06
short as humanly possible. Otherwise, you know, it gets
24:10
like attention drift or whatever you want to call it.
24:14
So that those, those are the, the tricky things.
24:18
And actually I found using something like called Code or
24:22
Cursor or Windsor to help me build those like basically
24:27
prompt stuffing systems like they're, they're actually very
24:30
good at helping you build those. The prompt like template or
24:35
definition. Well, it's not only just a yeah,
24:38
so the template, but then also the the the pieces which pick
24:43
what goes into the prompt. Almost like these variables that
24:48
you're adding in in the different scenarios.
24:51
Yes, and like pausing things like knowing like what you
24:56
should pause and what's OK to leave as is like.
24:59
So SQL for example is something where I experimented with.
25:03
Can I just leave it as it is and let the LLM read it?
25:07
Or is it better for me to pause it into some kind of structure
25:11
like Jason or something where I can describe the joins?
25:15
So I used a library called SQL, Got to pause SQL into that kind
25:20
of format, which makes it very explicit and I found there was
25:23
left from doing that, yeah. That's a nice little secret.
25:29
So basically parsing the sequel gave you gave the LLM better
25:36
understanding of what you were trying to do with that sequel.
25:38
Exactly. It makes me all the less
25:44
confident in the LLM sequel output.
25:51
Yeah. I mean, I mean, that's why I
25:53
believe in using a semantic layer in the way anyway, right.
25:57
I don't want them LLM to write me raw sequel every time.
26:01
Having it do it once like when it's building the semantic layer
26:05
and then you reviewing it and approving it and then it reuses
26:09
it over and over again. That's fine, right?
26:13
Yeah. It's just that long tail of
26:15
questions. And I guess that's kind of where
26:17
you go back to. Well, look, there's probably a
26:20
bit of a 8020 principle here where you have certain amount of
26:24
questions that are always going to be asked and then you got
26:29
that long tail. And it reminds me of a guest
26:34
that we had on here, Donae, who created a data analyst agent.
26:38
And she was talking about how the majority of the productivity
26:44
gains that they saw when they gave their analysts access to
26:50
this agent was on the, the like middle questions.
26:58
So not the Super, super easy ones and not the really, really
27:02
hard ones, but those ones in the middle is where you really saw
27:06
that. Wow.
27:07
I used to have to spend a lot of time putting something together
27:11
here, but now I can ask it. And the way that they did is you
27:15
just in Slack asked in natural language and then the the agent
27:22
would go and check the database and it only had access to one
27:25
database, which so if it was the marketing channel, there was the
27:28
marketing agent that could go and access the marketing
27:32
database. And that was always that.
27:35
That kind of stuck with me how it is really attacking those
27:43
that middle ground, almost like that bell curve of there's
27:46
probably a lot of questions that are very hard that are those
27:50
long tail questions that you still need to sit down and think
27:54
through and spend time with. And then you have your shower
27:58
moment of, oh, that's how it is. But for those other ones that
28:03
are are, like you were saying, very rope, let's try and get as
28:08
much automation in there as possible.
28:09
Yeah, it's interesting. What would you say, Donae, what
28:13
you said about the easy questions?
28:16
And I understand why someone would say that because the easy
28:19
questions are like a 5 minute ping an analyst on Slack and
28:22
they'll do it. So you don't feel like it's a
28:24
lot of value and it's maybe not in the actual time of answering
28:28
those questions. But I think in the terms of like
28:31
context switching and disruption, it probably is still
28:34
really valuable to answer those easy questions.
28:36
Yeah. Well, I, I think the way, if I'm
28:39
remembering it correctly, she was talking about was that it
28:43
wasn't as helpful for people who had absolutely zero data
28:50
knowledge, right. So it's not like the the random
28:58
janitor at the company could jump into that Slack channel and
29:01
be like, hey, and they would know all the jargon and they
29:04
would know how to ask the question correctly.
29:06
So those weren't the folks that were getting help.
29:10
And so it wasn't entirely self-serve in that way, but and
29:15
then on the other side of it, it wasn't like the Super hard
29:19
questions that you need real in depth thought about and what
29:23
kind of played in that middle layer, I guess.
29:25
So maybe I didn't explain it correctly that first time.
29:29
Yeah, you're talking about like it's almost like this concierge
29:31
where there are people out there who just don't think in a
29:34
constantive way and they don't need to to do their jobs.
29:37
And then they come to data and I don't, they don't know how to
29:40
conjugate the question that would be easily answerable.
29:45
Yeah. And I guess like there is an
29:46
element of like coaching you could do.
29:48
It's like, OK, well, these are the things you can know about
29:51
and these are the sorts of questions you could ask.
29:54
And then I think you can kind of guide someone say, OK, I'm going
29:57
to ask a question in that kind of format.
29:59
And then, yeah, but yes, yeah, I can see.
30:04
When you say that, it reminds me of way back in the day.
30:07
I had someone on here from the Financial Times and they talked
30:13
about how they, and this was pre LL Ms. era and agent analyst era
30:21
and their data team was just getting rocked because everyone
30:24
was coming to the data team as they do with these small simple
30:27
questions and or big questions and very important questions.
30:32
But the team was understaffed and overworked.
30:38
So what they decided to do, instead of trying to do like a
30:42
self-serve motion where you now can come and ask your own
30:45
questions, they said, we're going to try and get like a
30:51
human API in a way that you ask this person only all the data
30:58
questions. And that person is going to then
31:00
work with you on that question and come back and ask you, OK,
31:05
so like, if you had this information, what would you do
31:09
with it? What kind of actions are you
31:11
going to be able to take there? And what's what is that going to
31:17
solve getting that information? What is it?
31:19
And a lot of times folks would after talking through it with
31:22
somebody, they would realize like, actually, I don't know if
31:27
I want that kind of data. I really want this data.
31:32
And they would get to the crux of the problem or they would
31:34
just forget about it all together.
31:36
Which on the other hand, is like a great argument as to why
31:41
that's a horrible idea. Because it's sometimes you just
31:43
want to know. You just want to say like, I'm
31:46
curious about this, if we could know this data.
31:50
Then later, two weeks down the line, you realize like these two
31:54
data points connect. Yeah, with those abstract
31:57
questions as well, What I'm seeing is, is that these agents
32:02
are just like a jumping off point.
32:04
And so ChatGPT and Claude like desktop are being rolled out in
32:11
companies like Enterprise Scale and people are beginning to use
32:16
them kind of like how they used to use Excel.
32:19
And So what we're seeing is, and I've seen this be asked, being
32:24
asked by C-Suite to data teams is we want MCP access to our
32:29
data in ChatGPT or Claude. And, and what they'll do, what
32:33
they all, what they want to do and they have done before by
32:36
taking CSVS is I'm going to ask like questions of this data and
32:41
manipulate it using ChatGPT. Then I'm going to get some other
32:45
data. And then I'm going to like draw
32:48
like my conclusions based on using those, just like you would
32:51
in Excel, right? And that's what we're seeing.
32:53
So I think those abstract questions, they can't be
32:57
answered by an agent entirely, but their agent can provide the
33:01
inputs to those. And it's just like today, like
33:05
where you get those people and they ask the data team, oh, can
33:09
you pull this stage for me? Can you pull this stage for me?
33:11
And then they take the different things and do what they want
33:13
with it. Yeah, man, how fascinating is
33:17
that? Just thinking about how much
33:19
unstructured data is out there because of every interaction,
33:25
every touch point with ChatGPT in the enterprise that now is
33:29
getting rolled out and they're talking with ChatGPT and kind of
33:35
creating these hypothesis and all right, cool.
33:37
I think this, you know, this data makes sense.
33:41
Or check out the CSV file. Tell me what you can see or let
33:44
me see what I see. And then if you're trying to
33:47
access that as a company wide MCP server, that sounds a little
33:53
bit like a mess. Yeah.
33:55
And I think this is again, why the semantic layer is important
33:59
because if that MCP is sitting on a semantic layer and we're
34:04
seeing vendors do this, which is good, then when they're when
34:08
they're taking that data into ChatGPT, at least it's like from
34:12
a governed and known like way of calculation.
34:18
So that's great. So I've seen this all over the
34:21
place, like data bricks, thought spot, DBT, cube, they're they're
34:26
all offering this. Because the idea is you don't
34:31
want folks just bringing in random CSV.
34:34
Files. No, no, you'd rather they were
34:36
taking it from like an enterprise like semantic layer
34:40
where people have agreed on some kind of truth.
34:44
So then when you are trying to do some data stuff on all of the
34:51
ChatGPT conversations, you understand?
34:55
Wow, it seems like people continuously are asking about X
34:59
data point. Why is that?
35:01
Let's dig in. Yeah, that would be a great bit
35:04
of a metadata to have, right? How are people using the
35:08
semantic layer in ChatGPT and what are they joining together
35:11
that I didn't know they wanted to join together?
35:14
Well, I wonder I I'm not familiar enough with ChatGPT
35:18
Enterprise to know if you can get that kind of access, if you
35:22
can know everything that your employees are talking to ChatGPT
35:27
about. I'm not sure either.
35:29
I would imagine yes, right. I imagine that's something that
35:32
they would want to log because of security reasons.
35:35
So I imagine that exists. Yeah, it feels a little bit
35:42
personal sometimes, but that just shows you like why you
35:46
shouldn't be. You should use your own instance
35:48
of ChatGPT, not your company's instance.
35:50
Yeah, unless it's something you genuinely don't care about or
35:53
office work. Right, so.
35:54
Yeah, yeah, exactly. Hopefully it just stays work
35:57
related, but it reminds me of back in the day when folks would
36:01
be like, yeah, company can read your emails and you're like, oh
36:07
wow, I got to be careful there. But then I realized, like, I
36:10
have so many emails, all right, who's going to go through and
36:14
look at my emails? Well, that's the thing right?
36:17
They unless it hits some obvious rule right?
36:21
Although now though like they can just get an LM tweet it.
36:24
Yeah, yeah, give me the user sentiment here.
36:27
Yeah. Is this a sackable offence?
36:31
Yeah, yeah. I'm just looking for a reason to
36:34
fire this guy completely. But but I do think that all of
36:42
that metadata there around the way that the whole company is
36:50
using an LM provider, it's almost like it feels like
36:56
there's a lot of products that could go with that, that I have
37:01
no idea what they are right now. But that just made me get
37:03
excited of like, wow, you could probably offer a lot of data
37:07
products around the ChatGPT Enterprise version if you did
37:12
have access to all of the data. And that's what I think is still
37:16
earlier everyone's given their employees this because their
37:19
employees were using it anyway and it's just a massive security
37:22
problem. But now they're like, well, what
37:25
more can we do there? And you need to connect it to
37:28
your enterprise stores, whether doesn't have to be a semantic,
37:32
therefore like tabular data, it could be, you know, Notion or
37:36
whatever it might be so that people can pull the stuff
37:40
together and play with it, which is what they were doing before,
37:42
right? Just in manual tools.
37:46
Wow. Yeah.
37:48
I'm, I'm trying to think like what things could be valuable if
37:52
you knew everything that your company was talking to ChatGPT
37:56
about and how that could. Because at the end of the day,
37:59
you would hope that the most important data gets surface to
38:04
the right people. Yeah, I mean, I, I there there's
38:09
like even indirect things you can know because let's say
38:12
company like takes a bet on some like initiative and that
38:17
actually the, the most of the staff don't really think it's
38:20
going to work, but you'll see them kind of pulling.
38:24
They won't say they they won't just say something as obvious as
38:28
like, oh, I don't think this is going to work.
38:29
They wouldn't say that to church GPT, but they will start pulling
38:33
information about oh, you know, they'll say, oh, is this is this
38:38
like feature going to work because have we got this thing
38:41
or is this going to have product market fit?
38:44
Oh, let's see what's in the data.
38:46
You'll see all these different things where people are like
38:48
kicking the tires on this idea how?
38:52
Do I respectfully tell my boss I don't think this is a good idea?
38:57
Well, maybe ChatGPT should tell them you know all your
39:00
employees, They they don't really trust this.
39:02
Yeah, 100% that. Yeah, you'll get almost like
39:07
trends or insights. It, it makes me think of how we
39:11
started this conversation with the exploding topics.
39:15
And if you knew throughout your 10,000 person enterprise the
39:21
different conversations that are happening with the LLMS and you
39:24
could see there's certain trends that are happening, then I don't
39:29
know, maybe this is very Big Brother in a way, or maybe it
39:34
can help you be a better leader. But damn, I imagine sometimes if
39:43
I had that I wouldn't want to know, I think.
39:45
It could be. But I don't know, maybe there's
39:50
other good products that you could use from that, right?
39:56
Like if you know all that output and you know all of the.
40:01
Because right now what we're thinking about the way that we
40:04
usually think about using LLMS is, is almost like one person to
40:09
one LLM instance and like 1 chat window and then you can start a
40:14
new chat or whatever. And sometimes.
40:17
We'll share it with people or sometimes we'll take that output
40:21
and either we pipe it somewhere with a connector or we copy
40:25
paste it. Yeah.
40:28
But looking at that output on more of a like hive mind type of
40:36
level is totally different. Well, I mean, there's even
40:41
things like the concept of memories that they're trying to
40:43
bring in. And like Sam Altman has said,
40:46
like, I think even like he's begun to accept that there's
40:49
like a limit to this technology. Like everyone is kind of
40:52
expected. But what he's saying is now with
40:56
GPT 6, there's not going to be such a gap between 5:00 and
40:59
6:00. And also the thing that he's
41:01
focusing mostly on is memories. And if you think about memories,
41:04
yes, it's between conversations, but it can also be between
41:08
people. So if you like get to some kind
41:11
of conclusion with ChatGPT in your organization and you've got
41:16
a good result, if the next time someone else in your company
41:20
tries to do something similar, it's got that memory to draw
41:24
upon and say, OK, maybe I can do use this either verbatim or like
41:29
modified. Yeah, that is always funny to me
41:32
because it is memory is in a way.
41:34
It's like one part evals and one part like search and you have
41:43
you have to be able to find that information and it has to
41:46
actually be relevant. And then you also have to know
41:49
that that information was either good or bad.
41:52
It it like got the person the outcome that they were looking
41:55
for or it did not? Yeah.
42:00
And I think about that from the like from a data point of view,
42:02
that's what for that agent people ask data questions from.
42:05
It's like, isn't there a social part of that where like someone,
42:10
oh, you know, Maria from marketing asked like which state
42:14
is performing the best? And, and like that's a
42:17
subjective question. And through her chat, it starts
42:20
off with just revenue and then she's like, no, I want EBITDA or
42:23
something and I want it by, you know, states which, where we
42:28
have business and where, you know, we have decent coverage of
42:32
products, right? And so through that through that
42:35
conversation, she's got an answer she likes.
42:38
And then the next time someone comes along, like maybe it's the
42:40
CEO and says, I want to see like which state is performing well
42:45
and why? And then it should I I feel like
42:49
the system should offer, hey, Maria from marketing asked this
42:54
question yesterday and she got to like an answer she was happy
42:57
with. Do you want to use the same way?
43:00
And like they can say, oh, OK, so this is the way that
43:03
marketing are thinking about it. Yeah, that sounds good.
43:06
Or maybe not, you know, maybe. No, that's not how I want to
43:10
look at it. Yeah, it's, it's almost like, Oh
43:14
well, marketing, when they say the best state, they're looking
43:17
at MQLS that are generated and then finance is looking at the
43:22
bottom line or whatever it may be.
43:24
And so you can ask the same question and that tends to
43:30
happen, right. We're very vague in the
43:31
beginning, or we try to be descriptive, as descriptive as
43:35
possible, but we still forget about things.
43:38
And it's not until you have a few back and forth turns that
43:43
you get to the answer that you're looking for.
43:46
And so that could be a a great way to leverage memory and also
43:50
a great way to leverage the data in a way like you're not going
43:57
to be able to get that easily unless you have that semantics
44:01
layer that we're coming back to. Yeah.
44:04
And I think it makes things more transparent as well, because
44:07
like let's say marketing have filtered that to yeah, states
44:11
where they have business, states where they have good product
44:13
coverage, states where they're actually doing marketing
44:16
activity and finance comments are saying, no, I don't want all
44:19
those filters. I want, I want everything
44:21
because they don't care. Like they they just want to see
44:23
the whole lot. And the CEO is like, well, I do
44:27
want to see the ones where we operate in because why do I want
44:29
to look at the things I've chosen not to operate in, right?
44:31
So it's like you've got these different versions of the truth.
44:34
And I remember this like I used to work at an online grocer,
44:37
like very early on in my data career.
44:40
And you had trading, marketing and supply chain all had
44:45
different concepts of revenue. Like marketing was everything
44:49
sold, trading was everything. Like that got out the door and
44:53
supply chain was everything that didn't get returned, right?
44:56
So you thought you have these like 3 different definitions of
44:59
the same thing and they're right.
45:00
They're right to be different because those are the like how
45:03
those people think about the world.
45:05
But without that kind of semantic layer that makes it
45:07
easy to see that they're choosing that difference is
45:12
really hard. That goes back to the idea that
45:20
I had of yeah, like a glossary on, you have to know when you
45:27
say revenue and you're saying it, it's a marketing person
45:30
that's saying it. They mean revenue in this way
45:33
and that it it's so it's like, yeah, is it memory?
45:38
Is it glossary, whatever. It doesn't necessarily matter.
45:42
But what does matter is that like no two job roles look at
45:48
the data the same. Yeah, well, dude, what?
45:52
Is there anything that I didn't ask you that you want to talk
45:56
about? No, I mean, I guess like one
45:58
thing I'm interested in, like you think about your audience,
46:01
like how do they think about like this topic this like
46:05
semantic goes, because sometimes I think about feature stores,
46:08
which they're much more interested in than semantic
46:10
goes. And I think they're very, very
46:11
similar that I only see one difference between like a cube
46:15
and a feast, right? And the difference is like a
46:18
temporal difference in that there's a lot of concern in a
46:22
feature store about being able to evaluate something at a
46:24
certain point in time, whereas that's not really a major
46:27
concern for a semantic layer. But I was just wondering like,
46:31
yeah, how do they think about it?
46:32
What but you? Yeah, but that's like one of the
46:35
most important. Pieces it is, absolutely.
46:37
Right. Yeah.
46:39
Because without that, the machine learning model really
46:44
predict, yeah, what it it wants to predict.
46:48
But I do see the point on like if you take that piece out, even
46:54
though it is a key piece, key piece, it's kind of the same
46:58
thing. If you're talking about batch, I
47:00
guess if you're talking about real time stuff, then it's a
47:04
whole different. Semantic class can handle real
47:07
time now, Yeah, yeah. OK, then, yeah, I mean, I'm just
47:14
trying to think like the features you're, I mean, yeah,
47:20
let's walk through that in a way.
47:23
The features you're trying to describe something or someone,
47:27
an entity, and you're trying to find the most relevant pieces of
47:33
that entity that you can feed to a model so that it will help you
47:37
make predictions. And a semantic layer is
47:44
descriptions of an entity. Yeah.
47:46
Usually served to a human, though not a model.
47:49
Yeah, exact. And I guess it is trying to be a
47:52
lot more robust, like you don't need to give a whole 360 view in
47:59
a feature store. You just want the features that
48:01
you care about. You don't necessarily want
48:05
everything. Yeah.
48:06
Well, maybe it's not always obvious that there's features in
48:08
the feature store, like looking at that feature, why is that
48:11
feature important? But it is.
48:13
It's really important and whereas in the semantic layer I
48:16
know it's plain very obvious, like why you should want
48:20
anything. Well, it's almost like you, if
48:24
you're doing the feature creation and correctly, you want
48:29
a semantic layer for the features.
48:31
Also, you don't want to just have some random feature in
48:35
there and then be like, well, what is this feature?
48:37
Who created this? What's going on?
48:39
And. So what's E 79?
48:42
How did we get this? And then all you see in the
48:45
comments are do not ever delete this.
48:51
Yeah, but I do. I like that.
48:55
It just feels like semantic layers, Yeah, to me are more
48:58
static and features stores are more in motion.
49:01
I don't know why. That's like a very not clear way
49:06
of putting it. Yeah, I think that's that's true
49:10
in some ways, but less than others, because I feel like
49:12
definitions of the same thing will change over time in a
49:17
semantic layer. Whereas in a feature store,
49:20
you'll just replace E 79 with E80, like when you want to have
49:23
a new better version of it. And then some things will keep
49:25
using the old one and some things will use the new one.
49:28
And that shouldn't happen in a semantic layer situation.
49:32
Yeah, you need that canonical source of truth, right?
49:35
That's the whole reason that you're creating that semantic
49:37
layer. Yeah, exactly.
49:42
But the yeah, 'cause features, you can be very like generous
49:48
with feature creation in a way. You don't have to limit yourself
49:53
or try to create the one definition of whatever it is
50:00
that you're looking at like you do in a semantic layer.
50:05
I do like that though thinking about it because it it's like
50:07
they're not that far off but they are far enough to where I
50:12
guess they weren't different names.
50:15
Yeah, they weren't different names, they weren't being
50:17
different tools for sure. But yeah, there's so, so many
50:22
similarities.