Latest / Agentic Conversations (formally mlops.community) / Bridging the Gap Between AI and Business Data // Deepti Srivastava // #325

Transcript

Bridging the Gap Between AI and Business Data // Deepti Srivastava // #325 · Agentic Conversations (formally mlops.community)

0:00
What if you could put a box right in between all of your
0:04
data systems on one side, all of the data like databases, data
0:08
warehouses, whatever data systems, and your LLM systems on
0:12
the other side and just have straight lines.
0:15
Oh, you need data from Salesforce?
0:16
We'll go get it for you. You need data from Postgres, or
0:19
we'll just go get it from you. You need data from Snowflake or
0:21
Data Brakes? We'll just go get it from you,
0:23
for you. My name is Deepti Srivastav.
0:31
I am the CEO, founder of Snow Leopard AI, which is a
0:35
intelligent data retrieval platform that we're building.
0:39
And I take my coffee with milk and sugar, and you can come at
0:46
me for that. You just said to me that you
0:51
back in the pandemic days went to some virtual online meet up
0:56
that we did and it was on ML flow.
0:59
And I think I know exactly the one that you are talking about
1:02
because it ended up being one of the most watched episodes later
1:08
when we put it on YouTube. And I really think it's because
1:12
it hit a nerve and the episode was the difference between Cube
1:17
flow and ML. Flow.
1:19
Oh, interesting. Yes, maybe.
1:21
And it's super confusing, especially back in those days in
1:24
2020, folks didn't really understand the either one or how
1:29
when to use one and when to use the other.
1:32
And Byron came on and he explained it like, look, cube
1:37
flow is a sledgehammer and MO flow is like a little pickaxe.
1:41
So you, they're both great tools.
1:45
It just depends on what you're trying to get done.
1:47
And so I, I thought that was, that was awesome.
1:50
And, and it's really cool to hear that all the way back in
1:54
those days, you had seen the ML OPS community.
1:58
When I wasn't sort of when I wasn't as excited or when I
2:03
wasn't in it, actually, I was excited about the community, but
2:06
I wasn't even in it at the time. And I was like, oh, this is
2:08
interesting. At some point I should look it
2:09
up and lo and behold, like what is it?
2:12
Five years later, 4 1/2 here we are.
2:14
So that's very exciting. Full circle.
2:17
And so the the thing that I think is is really cool as you
2:23
were saying, yeah, back in those days even we were talking about
2:26
this problem that you've been hearing and talking with folks
2:31
about a lot. Can you just break down what the
2:35
problem is and then we can get into different thesis is around
2:38
it? Yeah.
2:40
So there are many ways to talk about the problem, but the crux
2:44
of it is in this new world of LLMS, like we expect AI to like
2:52
be this trillion dollar like new industry, new platform shift.
2:57
By the way, I bought into it. Like This is why I'm here with
3:00
Snow Leopard, right? Like, I bought into it,
3:03
whatever. 2:22-ish when the, when Chachi really broke.
3:09
Yeah, when the hype happened, I was like, 'cause I'm not into
3:11
hype. I'm a so, so backing up.
3:13
I'm a distributed systems person.
3:14
I've been in infrastructure my entire life.
3:16
We are anti hype. Just, you know, we, we are like,
3:19
we are the opposite of early adopters.
3:21
Let's just put it because we care about production and five
3:25
lines of availability and all that stuff.
3:26
And you can't really just be like, Oh, yeah, let's try the
3:28
new thing and like, see everything.
3:31
Yeah, so. So I come from that.
3:33
And I think that's important because as I said, I don't
3:36
believe in hype. But when I saw it and played
3:39
with it, I was just like, no, this is truly a platform shift.
3:42
I also did a minor in AI back in the day, and so in undergrad and
3:46
I was like, this is hype. So the issue is, I mean, this is
3:53
truly a platform shift. Al alams enable so many things
3:57
that we didn't even think was possible.
4:00
But, and this is the crux of what I believe the problem is,
4:03
right? Like for LLMS to be the platform
4:09
shift that they are, for Gen. AI, for AI applications to truly
4:14
change the way people live, work, behave, you need to
4:18
connect them to the crown jewels of any business or any sort of
4:24
user situation, right? And that is operational data.
4:30
And by operational data, I mean structured data.
4:32
So I mean sequel, no sequel API based information, right.
4:39
Today there is a huge chasm still between operational data
4:46
on the one side and LLM and LLM based apps on the other side.
4:50
And that's precisely what Snow Leopard and I set out to do when
4:55
we when we when I started Snow Leopard.
4:58
So it's AIA, 100% agree with this idea of for AI to live up
5:06
to the hype, we're going to need to plug into all of this data.
5:09
And I, I don't think that's necessarily the most spicy take.
5:13
Everyone is saying that and they're almost saying it like
5:18
incessantly on all of these different social networks.
5:22
We got to have, you want to have your own data and you want to
5:25
have your own models. That's kind of the theory that
5:28
we see. And on the other hand, though,
5:32
what I do think is, is fascinating here with what
5:35
you're saying is we need to be able to connect it to data that
5:42
right now we are not necessarily connecting it to unless it is.
5:47
We've seen those use cases where it's like agents that are text
5:51
to SQL agents or you have your data analyst.
5:54
And so you're using LLMS to query your database, that kind
5:57
of thing. But I feel like what we are
6:02
seeing more often than not is a whole new building of Jenny I
6:12
apps that are off to the side of your structured data.
6:17
So it's taking, it's connecting the Gen.
6:20
AI and the LMS to your Notion or to your docs or trying to make
6:24
JIRA work with it. And it's not really thinking
6:26
about connecting it to the databases or the AP is that
6:29
you're talking about. Yes, that's right.
6:32
So you're right that everybody talks about your data should be
6:36
ingested, right And like we should connect it etcetera.
6:42
But I think where exactly at your point where the focus has
6:44
been, has been like chatting with your PDF, right?
6:47
And and that's cool. It, it is something that was,
6:50
you know, wasn't easily possible before this.
6:52
But I think you, you mentioned something very important, which
6:57
is that building apps to the side of your stack of your
7:01
critical path is never going to really, you know, generate those
7:04
new use cases, user value, business value, etcetera.
7:08
And so you have to make it part of your critical path in order
7:13
for it really to, to your point, live up to the hype, right?
7:17
So today everything is around the fringes, right?
7:20
One of the things that, you know, when I started talking to,
7:22
so I validated the problem before I even started Snow
7:24
Leopard. I know, sort of counter
7:26
intuitive, but just that's just how I am.
7:28
But when I started talking to people like VP's a platform or
7:32
like head of AI at like all these companies, enterprise
7:37
companies, SM BS, you know, people in my network, all of
7:41
them were like, you know, we don't have a way good way of, of
7:46
putting it in our sort of true stack, right?
7:50
And there's a lot of talk to your point about how there's a
7:53
new stack emerging, like the anytime there's a new stack,
7:58
like that's outside of what the existing stack is, to be honest,
8:01
right? And in my experience of 20 years
8:04
of doing this, building data infrastructure and, and systems
8:08
for enterprises, right, production grade data systems,
8:14
you have to like everything exists, like the tech stack
8:17
exists in an ecosystem, It exists in a context.
8:21
And if you want to derive value or create new value, whether
8:24
it's for users or for business, right, you actually have to be
8:27
part of that unique ecosystem. You have to fit into it.
8:31
You can't just be like this cool new thing and, and adopt us
8:33
here, but like the rest of your whole world is here.
8:39
I, I think that's really the, the key, right?
8:41
Like, and we can get into this. But I do believe just one more
8:45
thing here, that while people say that, you know, you should
8:48
connect your, your LLMS and your agents and your assistants to
8:53
this, this data, I just fundamentally believe that the
8:57
way that people are going about it isn't going to solve the
9:00
problem. It's funny because I instantly
9:05
think of this visual of I live in Germany and I was driving
9:10
down the street the other day and what do you know, there's a
9:13
caravan of motorcycles with side cars on them.
9:19
And it feels like what we're doing with Jenny I is the
9:22
sidecar to the motorcycle, which is the production system.
9:27
Yeah, yes. And you know, I let's, let's get
9:33
into the spicy takes. I have quite a few, but there we
9:36
go. But I think the way people do it
9:42
today is, you know, rag, which is retrieval augmented
9:45
generation, which the concept sounds good, right?
9:49
There is the other thing which you're saying, which you know,
9:51
you train your own LLMS and you do all this stuff right.
9:56
Yeah. Both those options actually
9:58
don't get you to where you need to be, right?
10:02
So like spicy take #1 is that, you know, doing ETL and putting
10:12
all your data into a lake, house, ocean, like whatever,
10:18
right isn't actually solving the problem, right?
10:23
Spicy take #2 is. And this is the more important
10:25
spicy take, which is that, and we're talking about intelligence
10:29
will solve this. Agents will become self, self
10:31
aware, etcetera, and like we'll go out and do stuff and that's
10:34
cool. I'm not against that, right?
10:36
But the problem is the most intelligent machines today,
10:41
human beings don't make the right decision regardless of how
10:46
smart they are, regardless of how cool their reasoning is,
10:49
right? Like how good they are at
10:51
reasoning at, at like all of that stuff.
10:53
The intelligence today doesn't know how to make good decisions
10:58
with bad data. Bad data is equal to stale data
11:01
also, by the way, right? Like if I ask you if you've been
11:05
in a coma for six months and I ask you, right, who is the
11:09
president of the United States or who is the Prime Minister of
11:11
another country? Until I give you that
11:14
information, until you Google it, perplexity at open AI, at
11:18
whatever, you don't know that answer.
11:20
You cannot make the decision, right?
11:23
So if, if intelligence today, best intelligence today can't
11:27
make that right, can't make right decisions without the
11:30
right data at the right time. Those words are important, right
11:33
data, right place, right time, right until you do that, like
11:38
there is no way that artificial intelligence is going to be able
11:41
to make the right decisions, right?
11:43
Like we just have to reasoning will not solve it all.
11:46
Is is the point I'm trying to make.
11:47
I know this is a this is a counterculture take, especially
11:52
for this audience, but but I'm really honestly trying to help.
11:56
I'm honestly trying to make that intelligence more intelligent,
11:59
right? Like more useful.
12:01
Yeah. And I, I don't necessarily think
12:03
that it is so counter intuitive to say that reasoning is not
12:09
going to do it because the thing that we've seen time and time
12:13
again is the more context you provide to a model, the higher
12:20
likelihood that you're going to get to either the task being
12:26
done or the answer that you're looking for.
12:29
And so again, if you're not able to provide it the correct data
12:34
in the form of the context, even if you are providing it a lot of
12:38
data, but it's not the right stuff, then you're going to it's
12:41
going to be a shitty hat answer or outcome in that regard.
12:45
We talk a lot about hallucination and yes, there are
12:48
inherently in the way the alarms are built, right that can cause
12:52
hallucination. I agree with that.
12:53
But honestly a lot of the hallucination is because you're
12:56
not getting the right data. It's just trying to answer like
12:59
human beings hallucinate. If if we get the wrong
13:02
information at the wrong time, we will hallucinate, right.
13:06
And the other thing I will say here is you were saying, right,
13:09
to give you like the more context you give it, there is an
13:12
inflection point on which if you give it too much data, it will
13:16
also come to the wrong conclusion, right?
13:18
Because it doesn't know what to focus on, which is very
13:20
intuitive as human beings, right?
13:21
Like if you throw a bunch of data at me and I or information
13:25
at me and I don't know where to focus.
13:27
I mean, I may pick up the wrong thing, right, and start going
13:30
down the wrong rabbit hole. So again, these are like very
13:35
sort of to your point, they're not sexy things, but they are
13:38
very important, in my opinion, again, as a boring
13:41
infrastructure person to enable all the cool stuff to be built
13:45
right. Like we're, we're still nibbling
13:47
at the edges and until we like really like bring the right data
13:52
from the right place at the right time to these systems,
13:56
whether they're assistance agents, the next big thing right
14:00
in, in reasoning and NAI, like we're not going to really, we're
14:04
only scratching the surface here.
14:05
Yeah, it's true. We had Donae on here probably
14:10
three months ago now, and she was talking about how difficult
14:15
it was to get an agent to say no or to say that it did not have
14:21
the right information that it needed or that it did not
14:24
understand the task that you are asking for.
14:27
And it plays right into what you're talking about with
14:29
hallucinating because you ask it to do something.
14:33
And then if it doesn't fully understand or there's a
14:36
fuzziness to the terms that you're using, then it comes back
14:41
with one interpretation that may or may not be what you are
14:45
trying to get. And then you're like, this just
14:47
hallucinated the shit out of the answer.
14:49
And so you have to figure out a how to troubleshoot that.
14:57
And what they did is they said we are going to create a
15:00
glossary of all of the terms that we're using.
15:04
And we need to make sure that there's no vagueness in any of
15:08
these terms, the definitions of these words and these key terms
15:12
that we are creating, they can't be fuzzy.
15:17
It can't be vague at all, which when a human reads it, they
15:21
think, oh, that's fine. And then when you go and you try
15:23
and evaluate the output on it, you see.
15:26
Oh, yeah, I guess it didn't really understand what I meant
15:30
by create a good summary. It's like, what is good?
15:34
That's subjective. Right.
15:36
And I like, I think there's two things here that I, I believe
15:41
are key, right? 1 is like, imagine, like just
15:45
think about how much work they had to do and it still wasn't
15:48
producing the right answers, right?
15:49
Because they are inherently sort of so.
15:54
So here's the other reason that I think what we're doing is
15:58
interesting to me because like LLMS are actually really good at
16:02
this sort of fuzzy interpretation of stuff, right?
16:04
Like they're actually really good at summarization, like, you
16:08
know, classification, those kinds of things that inherently
16:11
require a little bit of like extrapolation, if you will.
16:14
And but they're not good at point.
16:16
Look up point solutions specific yes, no answers in all cases
16:20
right now. Yes, you can, you can tune them,
16:23
you can do reinforcement learning and they'll, they'll
16:25
get closer and closer and closer to what you wanted to do, right?
16:29
But but not everybody has the time, money, expertise to do
16:33
that, first of all. And secondly, they're actually
16:35
really good if you just give them again, I'm going to keep
16:39
going back to this, the right information for them to make the
16:42
the decision, right. And this is where I think I
16:45
think honestly, LLM get a bad rap in some way because because
16:50
ultimately everybody's using Rag right to do it.
16:52
And the way Rag is done today is ETL a bunch of data into a
16:57
vector store, right? And then use the vector store to
17:01
build context in at query time. But vector stores themselves are
17:06
fuzzy matching systems. They are not point look up
17:09
systems. They are not precise answer
17:12
systems, right? They're meant to do what they're
17:15
doing, which is be fuzzy matching, right?
17:17
Like whether you spell Demetrius with an, with an S or AZ,
17:21
they're going to come up with the same like they may come up
17:24
with both answers because they, you know, to them they're like,
17:26
oh, these are similar. You wanted similarity search
17:28
like here, these two are similar.
17:30
Now you figure it out, right? And so like we're taking first
17:35
of all, all the, all the information that truly can help
17:38
make this happen is, you know, the crown jewels, as I keep
17:42
saying are in your databases, data warehouses, in your like
17:46
Salesforce, HubSpot, those kinds of API based systems, right?
17:50
Which if you extract them, you change their nature, you lose
17:55
business context, and you put it in the vector store, right?
17:59
So now they've actually lost all the structure, the information
18:02
that was very carefully designed to be in those systems.
18:05
Like you've completely lost it, right?
18:07
It's in some BLOB, in some essential fuzzy matching system,
18:11
right? And then you pick it up from
18:12
there. So not only is it stale, not
18:14
only does it suffer from all the ETL and reverse ETL problems, it
18:18
also suffers from you took it out from a context in which it
18:22
exists. You put it in a different
18:24
context with a bunch of other data, right?
18:26
And then you serve it to the LLM and and you know, then, then
18:29
you're like, OK, the definition of good aside, it won't just
18:32
know whether to and how to answer the question, right.
18:37
So I think there is something to be said for fine tuning around
18:41
like give me a yes or no answer. But you know, the way the models
18:45
are going, you if you just tell them not to like, like if you
18:47
don't know, just tell me. Don't know.
18:48
Like they will tell you that today, like the the models are
18:51
getting smarter in this regard, right?
18:55
So, so then what you need to do is no matter again, no matter
18:57
how smart they get, like without them knowing what you're asking
19:01
about, they can't give you the answer.
19:04
And I do like this almost. I look at it as compression loss
19:11
when you're taking it, your ETL, ING it and doing all of this
19:14
stuff. And I, and I also, it makes me
19:16
think about something that I've been pondering for a while,
19:20
which is you don't hear as much about RAG these days because
19:24
agents are all the rage. Yes, it's the next wave of hype.
19:28
Of hype. Exactly.
19:29
And and you just connect them to your MCP server and you're good.
19:34
We're. All happy about the outcome then
19:37
and it changes the world and the way we work.
19:40
But the thing that I constantly going back to when it comes to
19:45
RAG or when it comes to how we are doing things and we have
19:50
been doing things with LLMS to try and create products that are
19:54
valuable products with them and and new products, inherently new
19:59
products. It's not like we're grabbing an
20:00
LLM and sticking it into a fraud detection system that we've
20:05
traditionally used ML for. And with RAG I, I always kind of
20:11
laughed because I would think. Are we just over engineering the
20:16
shit out of this? Are we just straight up we're
20:20
going and we're creating all of these different hacks?
20:23
And so in the beginning it's naive rag and then it goes to
20:26
advanced rag and then we're like, no, we got to do graph
20:29
rag. Because at each stage, it feels
20:32
to me like we get to the stage, we think this is going to help
20:36
us create a better system and have more reliability, have more
20:41
consistency in the output. And then at some point you see
20:45
the outcome and you recognize, I mean, it's good, yeah, it's
20:50
good. But is it what we really need?
20:52
Is it, well, maybe let's try the new technique.
20:55
Maybe there's a better technique out of there, but and or maybe
20:59
we can engineer this a little bit more to try to get better
21:04
results. And it's, it's inherent because
21:08
of the stage that the models were at.
21:11
I do think that with the new reasoning models, potentially
21:15
we're going to see these huge leaps due to that.
21:19
But that always it, it always makes me laugh like, oh, it
21:25
feels like we're having to do a lot of bending over backwards to
21:30
make this actually work. And I also would laugh because
21:37
all of that work that we're doing, all that bending over
21:40
backwards to create some chat bot that can tell you about your
21:45
company's. HR policy.
21:47
HR policy is not worth it and that is very clear.
21:52
I am so happy that you brought this up because happy.
21:57
But just like it just resonates with me.
21:59
That's, that's sort of what I mean, because yeah, I, I have
22:04
been in that world for so long, forget AI, right?
22:07
Like just this world of like, oh, we need this data now.
22:11
So now we're going to create this complex pipeline to go from
22:14
point A to point B. But oh, we forgot about this N +
22:17
1 silo, right? And now we have to like create a
22:20
whole new sort of pipeline, right?
22:24
And like a whole new way to do ETL, etcetera.
22:27
And that takes months, first of all, if not right, many months,
22:31
it takes a few months for sure. And then, oh, by the way, we
22:35
changed the dashboard or the use case or now in the AI world, the
22:40
question a little bit, which requires a mobilization of
22:46
entire teams to like rejig the whole thing.
22:49
Like, you're right. I'm so sad to see these like
22:53
complicated wires, pipes going everywhere.
22:57
Like if you look at a data architecture like a real data
22:59
architecture diagram, not one put forth like to.
23:02
Present. Some kind of decision maker like
23:04
it's like this, it's squiggly lines.
23:06
I know there's a podcast, you can't see me like gesticulating
23:08
like crazy, but with squiggly lines.
23:12
And no, sometimes you can't even tell why the line exists, where
23:15
it goes to what's going on, right?
23:16
It's just so over engineered, which honestly, my heart goes
23:20
out to the developers that are like trying to maintain these
23:23
systems and build them and like enhance them, right?
23:26
Because you need an answer to a new report instantaneously.
23:29
And the way these systems are crafted precisely for the, for
23:33
many of the reasons that you called out, right, like the the
23:35
models were not as good, the data systems were not as good,
23:38
they're not as scalable, they're not as like sort of any to any
23:41
connections. So yeah, what if we just took an
23:45
eraser? That's literally what I tried to
23:47
do when I was initially coming up with the architecture first
23:49
or whatever. I was just like, erase, just
23:51
erase. What if you could put a box
23:55
right in between all of your data systems on one side, all of
23:58
the data like databases, data warehouses, whatever data
24:02
systems and your LLM systems on the other side and just have
24:06
straight lines. Oh, you need data from
24:08
Salesforce? We'll go get it for you.
24:09
You need data from Postgres or we'll just go get it from you.
24:12
You need data from Snowflake or data breaks?
24:14
We'll just go get it from you for you, right?
24:16
Like no need for this craziness because like in my experience,
24:23
again, having seen, you know, Google, Oracle, you know, even
24:27
an observable, like just looking at how people are building
24:31
applications on top of these stacks, they spend developers
24:36
like data analysts, business analysts, all, all of that.
24:39
The whole data engineers are spending upwards of 70 to 80% of
24:43
their time just doing this part, which means you're not spending
24:46
as much time building those new value creation applications, new
24:52
business creation, new like, you know, user value opportunities
24:57
because you're spending all your time doing this.
24:59
So what if you just didn't have to do that and you could spend
25:02
time like coming up with the new creative ways of doing stuff?
25:06
I was talking to a data engineering friend of mine a few
25:09
weeks ago and he mentioned the predicament that he is in right
25:13
now, which is for the last three years their company made this
25:17
big push to self-serve analytics.
25:20
And so that was incredible. You know, the whole digital
25:25
transformation thing and he's data engineer.
25:28
And so he said. So now I've come into this
25:30
company and we've got over 11,000 data assets that the
25:36
engineering team has to keep and or has to service.
25:43
So he's saying we're kind of rethinking this whole self-serve
25:48
thing and wondering what can we blow up or what can we take the
25:53
eraser to and what actually is valuable or it crosses a certain
26:00
threshold of value and is still being used.
26:04
Because a lot of these are built by someone.
26:06
That person leaves, then there's no knowledge sharing on how
26:10
those dashboards are actually created.
26:13
It breaks one day and boom, that person's out of there.
26:17
And you know, you don't really get it again.
26:19
And why would you if it's self-serve?
26:21
So it's almost like, well, I didn't really even like that
26:24
dashboard that much. So I'm going to create a new
26:26
one. And then I, I have a better way
26:28
of doing it right, of course. And so you get into those
26:32
scenarios. I I do wonder though, in this
26:35
world that you're mapping out where you have the abstraction
26:39
on top of all the the data sources, what would keep it from
26:45
also being in that sprawl? Because the developers are not
26:51
building that sprawl, right? Like I think there's a couple of
26:53
things here. 1 is we're not saying it's magic, but we're
26:58
saying instead of each developer and data engineer team,
27:01
etcetera, doing this over and over again, like we just, we do
27:06
it in a generic way for them so that they're not having to
27:09
engineer pipelines at all or maintain them or like build them
27:13
at all. So the what we're saying
27:15
actually, so this is again, I think a spicy take.
27:18
Maybe I just think it's the like, you have to imagine the
27:21
world differently and then see if you can get there in a way,
27:24
right? Because a lot of this sprawling
27:27
situation comes from, frankly, the data world I've been in.
27:31
And like, you know, data grew faster than the systems that
27:34
could support it, right? So then there was all this like,
27:36
put it in a lake, then put it in an ocean, put it in a house,
27:40
right? All those things happened.
27:41
And to be honest with you, like ETL is helpful, especially in
27:45
the sort of business analytics, like historical trend analysis,
27:48
those kinds of things. But now that people had a place
27:51
to do that, right, like round peg, square hole, everything
27:54
goes in there and everything needs to get out of there,
27:56
right? Like that's one way to go about
27:58
it, which causes this. Like do we have this like Ant
28:02
silo? Have we put it and put it into
28:04
this lake house ocean? Like if we haven't, then we have
28:06
to go build it, right? And then to your point, like
28:09
there are these all these random dashboards.
28:11
But what we're trying to say is specifically what I'm trying to
28:15
to do here is what if you didn't have to move the data?
28:21
What if you could just go get the data right from the source
28:25
directly when you ask a question?
28:28
So there are no pipelines. It's not that we are building
28:31
the pipelines. There are no pipelines.
28:33
There is just connection to the data source and you go fetch
28:37
data from there when you need it.
28:39
It's kind of like when I ask you a question and you go look up
28:42
your favorite search engine and find the answer.
28:45
You're not keeping all that data, downloading it like
28:47
sifting through it, filtering. None of that is happening.
28:50
You're just saying, hey, like, you know, what's my date of
28:53
birth? Oh, go to the database that you
28:56
like, you know, some Social Security database or something
28:58
and go get it. Yeah.
29:00
So how do you deal with things such as OK I need a lot of data
29:06
points on, I need to know how many customers did XYZ, but in
29:11
the source data there you need to apply a few different kinds
29:17
of transformations on top of that to get that answer.
29:21
I mean, that's a classic like data warehousing problem.
29:25
And so those data warehouses are already built.
29:27
I'm just saying you don't need to build new pipelines and new
29:30
data sources and new like connections, right?
29:33
So if you're doing something that is important to your
29:36
business, which is where the original data warehousing
29:38
concept comes from, right, we can go fetch it from that data
29:42
warehouse. But what we can also do, and
29:45
this is what's done through complex software or complex
29:48
pipelining today, we can also do things like or or we should be
29:52
able to do things like, hey, you know, for example, if you ask
29:56
me, I say this software like I just, I don't care about
30:00
Terminator happening, right? I care about where my order is,
30:03
if I ordered something online, for example, right?
30:06
And. Speaking of which, I ordered
30:08
some fucking coffee the other day.
30:10
Turns out for some reason it was in Google my address for my old
30:16
house in Spain. So now somebody in Spain has got
30:19
some nice ass coffee. That's hilarious.
30:22
By like the 10th day I'm like, where is that coffee?
30:25
I could really use it right now. But anyway, sorry I I.
30:28
But that's a really good example, like somewhere
30:31
something should have updated your data.
30:33
Like. It didn't happen, right, because
30:36
some pipeline somewhere failed ultimately.
30:39
But the point is like we're saying, what if you had to, for,
30:44
for you to look up your order, like just ask, right, your
30:47
favorite assistant, where is my coffee?
30:51
It needs to do 2 things. You need to look up where your
30:54
coffee order was made, right? Whether it was Nespresso or some
30:58
other fancy stuff, right? And then where was it shipped?
31:01
Which means now you're looking at basically a posters database
31:04
or some sort of CRM system, right?
31:08
And then you're also looking at your tracking system like UPS or
31:12
something like that. That means you were supposed to
31:14
do a join between two data sources that are outside.
31:18
So, so data bases, data warehouses, data lakes are all
31:22
great at joining within. And if you need to join two
31:25
sources today, you have to dump the data in one place so that
31:27
they can do a cross store join, right?
31:29
But what I'm saying is you just need what you actually need to
31:32
do. What a human would do is look up
31:34
the, you know, your tracking number basically from your order
31:39
management system and then look up the tracking history in the
31:43
delivery system. Snow Leopard can can do that.
31:47
Like the aim with Snow Leopard is you can pull from both of
31:49
those places and give you an answer.
31:51
So you don't have to build any pipeline.
31:53
You don't have to like create any new way of answering this
31:56
question. If you're building an assistant
31:59
or an agentic flow, you just ask Snow Leopard the question and
32:03
Snow Leopard said, oh, I need to.
32:05
So it's doing intelligent routing.
32:07
It's saying, oh, I need to go to these two different sources,
32:09
right? And then it's doing intelligent
32:12
querying, query building, which is like, oh, for UPS, I need to
32:16
write this kind of query because then API and for, you know,
32:19
Salesforce, I need to write a SOCO query.
32:21
And for Postgres I need to like a PostgreSQL query, right?
32:24
Which is different from a Snowflake Sequel query, which is
32:27
different from a database query, database query.
32:30
So it built that in real time. Then it went and just fetched
32:34
directly from those sources. So that means the data is fresh,
32:36
which means if you know, if the coffee was delivered in to Spain
32:40
in the last half an hour, you should be able to get that
32:43
information instead of it's still in transit.
32:45
Then you go back and look at it next day because you know, data
32:48
dumping is still that's sort of what we're that's the world we
32:52
are talking about. That's the world we're
32:54
imagining. And that's sort of where, you
32:57
know, I've talked to recently to see to you at so far as I talked
33:01
to like, you know, head of data and AI in Asia for PNG, like the
33:07
stuff you were talking about, they're dealing with that same
33:09
stuff, right? Like 8000 silos of data, which
33:13
dashboard is powering what? Right.
33:15
Like let's just pare it down because we need to figure out
33:17
how to maintain it, right. And like our data engineers are,
33:20
are drowning in that our our data sort of data scientists are
33:26
drowning and, and all of that. So yeah.
33:29
Yeah, and we didn't even touch on the data quality part where
33:33
it's like, oh, something changed upstream and now the data is
33:37
absolutely worthless because of the way that these dashboards
33:41
have been built. One thing that I want to know
33:44
about this vision of how things work, and do dashboards still
33:50
play a part in this world? Or is it just questions and I
33:55
have to know what kind of question I'm asking before I can
33:59
get the answer? Because I think sometimes when I
34:02
look at a dashboard it provokes questions inside of me.
34:08
I think this is a very interesting philosophical
34:10
question and I think there's different camps on this.
34:16
Again, as an infrastructure person, I don't like predicting
34:18
the future, but you know my work at Observable, which is the data
34:22
visualization platform, like we are all visual people, so right?
34:29
Like data points on a graph is much easier to grok than like
34:34
1000 data points. Like we just know that.
34:36
So are dashboards going away? No, I don't think so.
34:40
I think data visualization will always exist because the easiest
34:42
way for humans to grok information.
34:45
But what if you could augment that with asking all the
34:48
questions that came up right in your mind when you looked at a
34:51
dashboard could be answered instantaneously because you
34:54
could go get the data if the data exists, right?
34:56
So you didn't have to. I will give you an example.
34:58
When I was looking at churn for my products at Google, right?
35:03
I have to first because first of all, I didn't have, I didn't
35:07
have the right dashboards. I had to build the dashboard.
35:09
So I came up with sort of a solution for what kind of data
35:12
we need, etcetera. Then I had to talk to my
35:15
business analyst friend who was by the way, stretched thin
35:18
because he was actually, you know, looking after 10 different
35:22
products. I had to queue into his thing,
35:24
into his workflow like, hey, I have, I want to build this churn
35:28
dashboard needs to pick data from here, here, here, right?
35:31
Can you build me that right now? Once, once he built it for me,
35:35
like 3 weeks later, I looked at it and I'm like, oh, I actually
35:38
need to know like churn by geographies.
35:42
So churn in Southeast Asia, churn in the US, churn in Canada
35:45
markets, in fact, one level deeper churn in, you know,
35:49
industry as well, right? So each time that answer
35:52
question came up, it was a few weeks to get that answer.
35:56
Either he had to build that into the into the data warehouse, or
36:01
he had to take the time to pull it out and give it to me.
36:04
But what if I could just, you know, once you have the
36:06
dashboard, but I forgot to just chat with, let's say it was big
36:09
queried, right, That where all the data existed, I could just
36:11
chat with my big query and I would also be able to like join
36:14
it to like, you know, all the Salesforce data that had the
36:18
latest customer information. What if I could just do that and
36:20
the system would take care of that for me, right?
36:23
So we're talking about eagentic flows here too, by the way,
36:27
right? Like EAGENTIC flows have the
36:29
same problem and spicy take #3 is while MCP is amazing and and
36:36
has it's a great start, open source start to the to the
36:41
connector problem, it doesn't solve it right.
36:45
It also had, I believe it has all the same problems that
36:48
previous generations of solutions have had, which is
36:52
that it? It's not really tackling the
36:55
hardest part of the problem. And the hardest part of the
36:57
problem always is intelligent routing and more importantly,
37:05
business logic. No.
37:06
I mentioned before we had Donae on here and she had built a data
37:12
analyst agent. And the way that they did it to
37:16
keep the agent in line, we could say is that they had different
37:23
Slack channels for different agents with access to certain
37:29
databases. So you would have a marketing
37:32
Slack channel that had access to the marketing databases and the
37:36
marketing type of queries were happening in there.
37:41
And so that was almost the way that they were able to hard
37:45
code. Hey agent, you speak this
37:49
dialect of Sequel. You have access to this type of
37:54
database right here. And here is your glossary, as we
37:57
mentioned before. So if any of these key terms
38:01
come up, you know what they are and you know where to get them
38:04
and grab that information. And so they were able to do cool
38:09
stuff and the, the agent could then go and create sequel
38:13
statements and pretty much do a lot of the menial work.
38:21
We could say I, I think I remember them saying it was like
38:24
a barbell. The majority of the questions
38:27
that were asked were those questions that are not super
38:33
complex, but they're not something that someone who has
38:38
no idea of data and is not a data analyst would be asking.
38:43
It's like those, those kind of middle of the road questions.
38:47
And anyway, this all this whole story is because I would love to
38:52
know how are you making sure that the agents or your system
38:59
speaks the right language to the right database?
39:03
Yes, that is a wonderful question.
39:08
In fact, everything you just described is sort of how people
39:11
are trying to do it right? The hard coding piece is so like
39:16
that's just how it's done today. Really.
39:17
You have to hard code and you have to separate out because
39:20
otherwise mixing questions that go to my sequel dialect versus
39:26
going to snowflake dialect, even though they're all sequel and
39:28
they all follow NC sequel standards will confuse the heck
39:33
out of everything. The system, entire system,
39:35
including LLM, right? This is actually why text to
39:38
sequel doesn't work either, right?
39:40
Apart from the fact that it hallucinates and all that stuff,
39:42
It's just which dialect and what is allowed in which dialect.
39:47
And, and this works better for open source databases cause the
39:50
LLMS have gone through that. But for closed source databases,
39:52
it's like a nightmare. I know because we have been
39:56
doing it right. So what we're doing is we're
40:02
using known data infrastructure techniques to do things that are
40:10
essentially systems problems. We are not trying to shove
40:13
systems problems. Into the reasoning AI, you know,
40:18
predictive world, as I said earlier, right, like LMS are
40:22
great at classification, at summarization and, and those
40:26
types of tasks, right? Deterministic systems are great
40:31
at if you if you code them the right way, they're great at, you
40:39
know, point look ups and and being able to do precise and
40:43
deterministic work. When you are building a sequel
40:47
query for the right dialect, you are doing precise and
40:50
deterministic work. So we are using essentially data
40:56
retrieval techniques, not in IR, but data retrieval techniques.
41:00
So we are building sequel, for example, if you're talking about
41:04
sequel, right, like we're building the sequel for the
41:08
right dialect, right, based on where that query needs to go to.
41:13
And we're doing it in a deterministic way, which means
41:16
we're using like data or or sort of query building techniques.
41:21
Query building techniques exist like databases use query
41:24
builders, right, data warehouses using use query builders, A
41:27
bunch of data retrieval systems built use query builders.
41:30
We are doing it using those techniques and like using
41:36
existing sort of methodology to like for example, Orms are great
41:42
at doing this, right. So we use sort of that existing
41:45
methodology for building like a Postgres sequel query versus a
41:50
Snowflake query. We're doing this today.
41:52
We have a a design partner that has data in sort of one of the
41:57
proprietary data warehouses and we just build like queries for
42:02
that right on the fly, right. So where do you need to do sort
42:07
of that intelligence in retrieval?
42:10
Like, oh, you know, based on this question and based on my
42:13
understanding of business logic, we know that the query needs to
42:18
go to Postgres, and also there's a separate query that needs to
42:22
go to, let's say Big Query. Interesting.
42:26
We build both of the. Once we know that right, then
42:29
the rest of it is deterministic, right?
42:31
Like the rest of it is. Oh, build a Postgres sequel
42:35
query. Build a big query, Sequel query.
42:38
So we do that and that that is just sort of like we're using
42:41
this is what I think is exciting is we're using data system and
42:45
deterministic programming techniques for that piece and
42:48
we're using, you know, AI and agentic work for the sort of
42:55
intelligence in the routing and intelligence and what data we
42:59
need to fetch. The understanding.
43:01
So if I'm understanding this correctly, it is AI gets to a
43:06
certain point, it takes the query and then it says, all
43:10
right, cool, to answer that I'm going to need to use these tools
43:16
and then boom, when it fires it off to the tools, then it steps
43:19
back. It's not actually creating the
43:22
sequel. And I think that is a really
43:28
good way of doing it because you're taking a lot of the
43:32
responsibility off of the LLM. And anyone who's dealt with LLMS
43:35
know that the less responsibility you give them,
43:39
the better, unless you do something where it's very siloed
43:43
and it is only you. As I was saying, donate created
43:48
where you're a market, you're in the marketing channel.
43:50
I know this agent only has access to the marketing
43:53
database, it's going directly there.
43:56
They opted to let the LLM do the Sequel query, but they also have
44:02
a lot of calls to the LLM to like double check the query and
44:07
do this. So it gets back to that
44:09
hackiness a little bit like, all right, cool.
44:11
Yeah, we do that. And then when something gets
44:14
returned, they're also double checking it there to make sure,
44:18
hey, judging by this question, is this Sequel query and the
44:23
data retrieved doesn't make sense.
44:25
And so you'll have like a critiquing agent that is is
44:28
double checking all of that. But in your case, you're saying
44:31
the LLM is only used to understand the query and then
44:38
fire off the different tools that it needs.
44:41
Right. And that's sort of like how you
44:44
would describe it in the sort of, you know, these days the MCP
44:47
agentic world, right. But we're take, we're not just
44:49
firing off a tool call, we're taking responsibility for how
44:53
that tool is created and the accuracy of that tool as well,
44:57
right. So when we ran our benchmarks,
44:59
for example, internally we did it with an older sort of rag ish
45:04
system, which is how most of these things work.
45:07
Like we had a 9998% accuracy. This is without us doing any
45:12
fine tuning of our own intelligence.
45:15
And the the system we were comparing it had like 6%
45:18
accuracy for this type of SQL based look up right?
45:20
Because either doesn't know how to build the right SQL or it's
45:23
not getting the right data or it's hallucinating.
45:25
All of those things just don't like even what you would do.
45:30
But but that aside, I think back to your use case that Donae was
45:34
was building, right. What we are doing for sort of
45:38
our design partner here is like that Slack channel can actually
45:41
be used by sales and marketing people, right.
45:46
What if you could have that? Because turns out there's a lot
45:49
of overlap in the kinds of questions that sales and
45:52
marketing people have around, you know, the, the pipeline
45:57
piece and all the way to the sale and post sale piece, right
46:00
around customer data and around it, like all that kind of stuff.
46:02
So if both of them can ask questions, then we didn't have
46:07
to and and you didn't have to hard code anything, right?
46:10
Then you actually open up this whole new world where they could
46:13
probably even talk to each other and come up with better
46:16
campaigns and better end to end work flows for, for having
46:19
making a sale happen, right. So I think what we're what I'm
46:23
imagining is in this world of ad hoc questioning and ad hoc
46:30
information needs, why not make the information like retrieval
46:38
ad hoc? Why not make the data retrieval
46:41
ad hoc, right? So what we're doing is we're
46:43
having like a natural, you know, there's in the data world,
46:46
right? There's all this like single API
46:48
to fetch everything, which has never worked because, right, you
46:51
just talked about one of the many problems, which is which
46:53
sequel do you build at the end of this single API?
46:56
But what if the API is natural language, which is what it's
47:00
become? But then it translates to the
47:02
precise like other end what you're calling tools, right?
47:07
Like it translates to the precise dialect that those tools
47:10
talk. So the tool is not doing any
47:13
guessing right. The the API is not doing any
47:17
guessing right. You use the intelligence for
47:20
what needs to be used for the understanding and the oh, I
47:23
think it needs to go here and do this, and then you leave the
47:27
rest of it to the parts that know how to do it, which again
47:31
leads to better accuracy. I didn't quite understand how
47:36
are you differentiating on that tool call part and taking
47:44
ownership for the outcome that all right, we're going into
47:49
whatever database, whether it be air table or your CRM of choice
47:55
and we're ensuring that it's going to gather that data
47:59
properly or I think you you said something like we authenticated
48:04
or we authorized that it, it will be correct.
48:07
Oh yeah, we're just taking responsibility for the accuracy,
48:10
I think. Right.
48:11
So, so think so Snow Leopard is a three-part sort of workflow.
48:16
The first part to be more more like, let's dig into it, right.
48:19
Like the first part is just what you said, like you have a
48:22
question. We do natural language
48:23
understanding, no rocket science here, right, mostly AI here.
48:28
And then the next part is we are combining data infrastructure
48:34
logic, your business logic to figure out which API call needs
48:39
to be made. This is our proprietary sort of
48:41
way of doing or or rather this is the built intelligence we're
48:45
building, right? And then once we figure out or
48:49
it needs to go fetch user ID, you know, in your case for
48:53
example, it needs to fetch your name, your address and the order
48:56
number right from one place, right?
48:59
Which is your, let's say, CRM system.
49:01
And it needs to fetch the order information and shipping
49:06
information from the delivery system, right?
49:10
Once we know those two things, this is what the intelligence is
49:13
telling us, then building the right query like we're not doing
49:16
text to sequel, right? So there, there is no fuzziness
49:18
there. We're literally saying build the
49:21
sequel statement, which is like it's a known thing, like build
49:25
the sequel statement directly for this particular dialect and
49:30
we go build it, right? So we're basically using the
49:33
like regular programming, like systems programming to do that.
49:36
And then we go run it on that system.
49:38
So, so you know, if you run APG Sequel query on APG Sequel
49:43
database, it will give you the information, right?
49:47
That's known. And then yes, we can run it
49:49
through an evaluation system and all that stuff, right?
49:51
Which is again, these are known things that people have done,
49:55
right? But the, there is no like, oh,
49:57
did it get the right data? Like did it, you know, we had
50:00
this column, but does that column exist?
50:02
Like, I don't know, like that's what you do when you do text to
50:05
sequel or, or sort of these agentic flows, right?
50:07
The tools have to, you hand off a lot of responsibility to the
50:11
tool and the tool developer has to take on the responsibility,
50:15
which and you don't know who the tool developer is, right?
50:17
Like #1 number two different tool developers.
50:21
Like so MCP servers for post risk can be built by anyone,
50:25
right? And so because there's no like
50:28
you have to build it this way, MCP, the protocol and MCP, the
50:32
sort of open source in framework doesn't tell you that you have
50:36
to do it in this way, which means different people can
50:38
interpret that framework and build different MCP servers,
50:44
right? So either you just like with any
50:47
other open source software, either you take somebody's MCPC
50:50
server, then you like harden it, you test it, you fix things in
50:55
it the way you want them to be fixed, right?
50:57
Or you just rely on them to do the right thing, which anybody's
51:01
that been in the open source world, like right, it's just the
51:05
last 20% is sort of where all the 80% of the work goes, right
51:09
in hardening it. And the one additional thing
51:11
here is that we are taking the sort of responsibility of
51:14
understanding business logic that you tell us, right?
51:18
Which the NCP server is just a connector, right?
51:22
So it's just going to do if you give me like a SQL query, I'll
51:26
go run it on on Postgres. But like which SQL query to run?
51:29
Is that the right SQL query to run?
51:31
Like, does it understand whether column order ID with a capital O
51:35
is different from column order number with a small O like it?
51:39
It goes down to that level of confusion and complexity, right?
51:42
This is what we're trying to do. Yeah, the downfall of the MCP
51:48
servers are very much that and I appreciate you calling that out
51:51
real fast because it is very easy to put up AMCP server and
52:00
that's like the blessing and a curse.
52:02
Yes, 100%. That's that's probably why it is
52:06
so incredibly popular right now, because folks can stand up at
52:11
MCP server in a few hours. That's right.
52:14
But as you said, if you don't know and really know intimately
52:21
the insurance and outs of your Postgres database, you throw up
52:25
a server with a lot of tools, and those are your opinions on
52:29
how the tools work, you could get yourself into trouble.
52:33
Or if I come and I grab your MCP server and I think, all right,
52:38
cool, it's a Postgres MCP server, I'm going to just throw
52:42
this into my Postgres instance. Then.
52:45
Yeah, there's, there's a little bit of wire crossing that can
52:48
happen. Yeah, and I, I agree with you.
52:50
Like I think it is a blessing and the course, right?
52:51
Like I, I'm not trying to poo poo on it.
52:53
I'm just trying to point out because people attach themselves
52:56
emotionally to the next big thing, to the next hype and then
53:00
they get super disappointed, right?
53:01
And I and I want people to just understand and go wise wide
53:05
open, right? Like it's actually great for
53:07
Snow Leopard if there's MCP service, because if one gets
53:09
popular, it actually helps us because we don't have to build a
53:12
connector then. So it accelerates, you know,
53:15
user delivery for us honestly, right.
53:18
But I just want people to like know that there are sort of, you
53:25
know, there are blind sides to this that you want to be aware
53:28
of. You want to go in eyes wide
53:29
open. Like it's fun to play with.
53:31
But in my experience, building like those high availability,
53:34
high reliability production systems requires a lot more
53:38
than, you know, putting something together in a few
53:41
hours, right? Like that's not the the hard
53:43
part. The hard part is, again, going
53:47
from POC to production, right? Like this is still a problem, I
53:49
think with AI systems from the, you know, engineering leaders,
53:54
CT OS that I talked to that, you know, we built this cool thing,
53:58
we couldn't put it in production because reliability and accuracy
54:03
and, you know, all performance isn't even a thing right?
54:06
Right now. It's just reliability and
54:08
accuracy. Like I needed to answer the
54:10
question when I needed to answer the question in a reasonably
54:12
accurate way. Yeah, we're, we're OK with
54:16
waiting 15 minutes to get this answer back as long as it's the
54:21
right answer. And so, yeah, well, anything
54:26
else you want to touch on before we go?
54:30
I just think it's a really exciting world, right, that we
54:32
live in right now. I think the fact that there is
54:34
MCP and the fact that there is all this discussion around tools
54:38
and, and things like that means that people are finally really
54:41
starting to like, understand the thing that I was saying.
54:44
Like I went blue in the face. If you look at my LinkedIn from
54:47
last year going like, but you need your, your structured data
54:51
systems like you need them. And so I'm personally really
54:54
excited that people are starting to notice this and is starting
54:58
to pay attention to this problem because that is what I believe
55:00
will cause the true wave of AI to like make people's lives
55:05
better. But yeah, I'm really excited to
55:08
talk to the people who are facing this problem.
55:12
I want to, you know, even Danae, for example, it's been really
55:14
exciting to hear from you about what, you know, they've been
55:16
doing there because I just want to understand honestly, how
55:19
people are tackling this. Do they have this problem?
55:21
How are they tackling it? Right?
55:23
Can we help? Like it's a really exciting time
55:25
to be in this space. And I keep saying the last thing
55:28
I'll say here is, you know, I, I've been telling my my system's
55:32
friends like, hey, come over on the AI side.
55:34
It's OK. I know it's non deterministic
55:36
and I know you hate that. Come on.
55:39
It's, it's fun out here. And I really, I feel like some
55:42
people are sitting up to pay attention, which I'm, which I'm
55:45
excited about because I want the AI world and the systems world
55:48
to come together because that's how we're going to, we're going
55:50
to solve these problems. Yeah, I'll have to introduce you
55:54
to donate and for anybody that wants to check it out, it's Snow
55:57
Leopard dot AI and I, I love what you're doing.
56:01
I think it is super cool and I'm excited to see how it progresses
56:06
and live in a world where you have infinite connectors to all
56:10
of these different databases. So I can I actually have a use
56:14
case right now that I'm thinking about where I was asking myself
56:17
so many different questions yesterday and I have to go and
56:23
gather the data between air table and Salesforce and I am
56:28
really bad at Salesforce. And so it took me way longer
56:31
than it should have. And and then you get into, oh, I
56:34
don't know if I have access to that data.
56:36
And that's not going to be solved by Snow Leopard, I'm
56:39
sure. But I have to go and talk to
56:40
somebody and say, hey, I need this for my report, blah, blah,
56:43
blah. So yeah, data fun.
56:46
Data fun. Excited for a world where Snow
56:49
Leopard can hopefully get rid of the majority of that pain.
56:53
Yeah, thank you. It was so fun to talk to you.
56:55
Dimitrios, thank you so much for having me.
56:56
And and yeah, this is this is the kind of thing that I think
56:59
people should be like talking about the very exciting.