Latest / Agentic Conversations (formally mlops.community) / Talk to Your Data: The SQL Data Analyst

Transcript

Talk to Your Data: The SQL Data Analyst · Agentic Conversations (formally mlops.community)

0:00
We are back for another edition of the Process ML OPS Community
0:04
Limited series, all around using agents in production.
0:07
And this episode we breakdown the Tokan Data analyst, which is
0:11
one of the agents that actually made it to production.
0:15
So Paul and I talk about the inspiration behind it first, and
0:18
then we bring in donate to talk about the technical aspects and
0:22
where they hit challenges. Again, it's very rare for me to
0:27
see companies that are this trans parent about the technical
0:31
difficulties of putting something into production.
0:34
And so I got to give a huge shout out to Process and thank
0:37
you for making this happen and being this transparent.
0:40
Let's get into this episode. Welcome back to another episode
0:49
of the Pros SAIML OPS Agent and Production podcast.
0:53
The mini series, I like it. What are we going to talk about?
0:56
Today, data analysts and everything that that means,
0:59
because I'm sure when we say it, nobody understands it.
1:02
It is an internal project that made it through the great filter
1:05
of your projects that have had success.
1:09
So break it down. What is it?
1:12
Well, what we talked about last time is that we've, you know,
1:15
we're moving down this sort of evolution of agents and getting
1:18
them into production in ways that can add value.
1:20
And you spoke about the graveyard of AI agents.
1:24
The data analyst is one of those that is not at the graveyard.
1:28
It's live and kicking. It's in production, answering a
1:30
ton of questions. There's a lot of experimentation
1:32
going on. And the data analyst is the
1:34
agent that essentially helps anybody answer questions about
1:39
data that sits in any of the companies in the process group
1:43
using an agentic framework. And it's it uses all of the
1:50
agentic tools that we talked about earlier.
1:51
So the ability to go out and fetch data because it can talk
1:55
to the database, it understands the the the fields and the
1:57
columns. And in essence, it's like
1:59
talking to a real data analyst on Slack.
2:02
That's what what it is. And we're going to bring Donae
2:04
on later to talk about the technical insurance and outs of
2:08
this. But before we do that, let's
2:09
talk more about the inspiration, the actual tool, what it means
2:15
on the business side to have this and a little less
2:19
technical. Somebody wants to just jump to
2:21
the technical part. They can skip ahead right now,
2:24
but we're going to set the scene for what this actually means and
2:28
who's using it. What successes have you found
2:31
from it? Yeah, the inspiration is at the
2:34
end comes back to you know how who we are and what we do as at
2:38
the group at process. So we're about 30,000 employees
2:41
as spread across various tech companies where people
2:45
continuously try to innovate, make decisions in the product,
2:48
try to understand, you know, what the customers are asking,
2:52
where their orders went and so on.
2:53
And that often requires going into a database and, you know,
2:57
fetching the right information to be able to make a decision,
3:00
respond to a customer, dispatch an order and so on.
3:04
And so we've got thousands of people whose full time job data
3:07
analysts are to help others in the organization, enable them to
3:11
make decisions. Yeah.
3:13
And as we found the, you know, the our token data analyst was
3:19
actually able to answer those questions with high fidelity,
3:21
high precision. We wanted to roll that out to
3:24
basically beef up everybody's ability to ask and talk to data,
3:29
ask any question and talk to the.
3:30
Data and you say, so there's thousands of data analysts, I
3:33
presume they know how to go in and ask questions of the data
3:38
already. So is it only them that are
3:42
getting use of this data analyst agent or are other people able
3:48
to now get more familiar with the data and answer questions?
3:51
No, we, we see it as a addition to any data analyst team.
3:54
So you've got existing teams of data analysts and the token data
3:57
analysts agent joins a team of data analysts and is able to
4:01
answer a whole bunch of questions with them for end
4:05
users. And it's part of the team.
4:08
So for example, when people go in and ask question to the data
4:11
analyst, it's in a Slack channel that the other data analysts are
4:15
also there and would maybe typically be the first point of
4:18
context. Now the data analyst, the Tokan
4:20
data analyst is the first point of contact.
4:22
It can answer the question. There's other data analysts in
4:25
these teams that can see the answer.
4:27
Make sure that you know, if there's issues with the answer,
4:30
can't retrieve it or it's incorrect, they can flag that to
4:33
the end users. So it's very much in addition to
4:36
the existing data analyst teams. Of course, the token data
4:39
analyst agent is always available, so it answers very
4:41
quickly. We've seen that that's a huge
4:43
benefit. It can't always answer
4:45
everything, but because it's it's fast, it's got high
4:49
accuracy. It's something that has been
4:52
able to give people a lot more speed and finding the right
4:55
information they need. Yeah, it's almost like the
4:58
support burden of people coming to data analysts and and maybe
5:04
it's other data analysts coming to a another data analyst to
5:08
say, hey, I'm trying to get deeper on something that you're
5:13
very familiar with. Can you give me these answers,
5:17
right. That support burden has now been
5:20
almost like filtered out a bit by the data analyst because you
5:25
do not need to go to another human as your first point of
5:28
contact. You can see if the agent can
5:31
answer it. And then if it can, cool, I got
5:34
what I need, I'm going to go and do it my thing.
5:37
But otherwise, if it can't, then you can escalate it up to a
5:39
human. And I think that's one of the
5:41
pieces that we're seeing time and time again on these use
5:44
cases is the support use case for agents to be able to be that
5:49
first line of contact. And if they can't answer this or
5:55
get the ticket closed, then you can escalate it up to a human.
5:59
And so it relieves a bit of of that first line.
6:03
So those simple questions, right.
6:05
Yeah. Let me give you a real example
6:07
of how it's being used. So let's let's take ifood,
6:09
right? It is a leading global food
6:14
delivery player and based out of Brazil and they do about a
6:18
billion, more than a billion orders every year.
6:20
And there's a lot of people in the in the organization who are
6:23
trying to understand, you know, what's happening.
6:25
So how many orders has this restaurant done or what are we
6:30
seeing in this particular zip code?
6:32
Or a customer operator that's trying to figure out what
6:35
happened to a specific order with a certain order ID that
6:39
needs to figure out what's going on.
6:41
That ought to get the answer to those questions.
6:44
You need to be able to interact with the data, but these are
6:46
often repeat questions. Hey, tell me what's the status
6:49
of this order or like I said, you know, I'm going to visit
6:52
this, this partner, this restaurant, you know, how have
6:56
they been doing over the last three months?
6:59
Those are the kinds of things that the token data analyst data
7:02
analyst is able to, you know, do really, really well because it
7:06
understands the data. They're fairly simple queries.
7:09
There's still a lot of very complex queries that the human
7:11
data analyst obviously need, need to need to help with.
7:16
But it's sort of, as you said, the first layer of high
7:18
frequency questions that that we know we can answer well, that
7:24
the data analyst can support. These people with it aren't able
7:28
to write sequel queries themselves, or don't even have
7:30
access to the databases to do so.
7:32
Yeah, that's a great point. And you talk about I food being
7:37
one of the companies that you release this to.
7:41
Did you release it to everyone in like the process group?
7:44
And so you have anyone who wants to can now use the data analyst
7:50
or is it something that you rolled out to specific companies
7:53
in the portfolio? Yeah, we, we have it's, it's not
7:57
sort of a flip of the switch that you just connected and it
7:59
works. There's a, a process that takes,
8:02
you know, a couple of days to onboard a token data analyst on
8:05
a specific use case. So we, it's available to onboard
8:10
for everybody and we've made it fairly seamless to do that
8:13
onboarding. It's very quick and easy to do
8:16
so, but it still requires effort because of course there's some
8:18
integrations, you know, we, we can connect with Snowflake and
8:21
with data breaks and with Big Query and many of the other
8:24
commonly used data stores. But that still needs to happen,
8:28
right? That's an integration that needs
8:30
to take place, and it's a. Specific instance like you have
8:34
the I food data analyst instance so that you're not grabbing data
8:38
analyst stuff from another this. Is all, I mean this is all
8:41
logically separate and you know that that is sort of by design
8:45
how token and it's agents work. So it's per company.
8:50
But then even within a company, so when it goes to OLX, which is
8:53
you know one of our global classifieds players or global
8:57
users in the food delivery space, we're all using the state
9:00
analysts. They need to then go and figure
9:03
out, OK, this is a marketing use case and the data analyst needs
9:06
to be connected to marketing data or it needs to be connected
9:09
to the logistics data set which actually tracks the status, real
9:12
time of orders. And so that integration needs to
9:15
take place. And then when once that's done,
9:18
then they can use anybody can ask questions about that data
9:21
set. And we see that, you know, when
9:23
people do that, there's two things that happened. 1 is the
9:26
time to get the answer reduces. We measured it's about 74%
9:31
reduction 70. 4% reduction, yeah.
9:35
So now, no, that's huge. That's huge.
9:38
I mean for for folks who are I. Was I thought it was like 20%,
9:41
yeah, even 20% would be a success, right?
9:44
But 74 percent, first of all, how did you measure that, right?
9:48
We got to go deeper into these numbers because that is huge.
9:51
Yeah. I mean, there's so when when
9:53
somebody has a question, they typically post it to one of
9:56
these Slack channels and then they need to wait for somebody
9:59
who's available. And then that person needs to go
10:01
spend some time, write the query, check it in return and
10:03
respond, yeah, we're only measuring here the actual time
10:07
it takes for somebody to go and once they're already working it
10:09
that basically the. OK, yeah.
10:12
So, so query submitted, yeah. And then response submitted type
10:16
of thing. OK.
10:17
Yeah. So.
10:18
And that's one piece. The other piece is the fact that
10:24
we see many, many more questions being answered as a whole on
10:29
these data sets. So we saw very quickly after
10:32
connecting token data analysts, a certain, you know, for
10:35
example, the marketing data set, a logistics data set, those
10:38
became the most used data sets in the company because now
10:42
anybody can ask a question to them.
10:44
So we have, you know, a doubling or tripling of the number of
10:47
insights we call that that we can actually generate based on
10:50
this data because it doesn't, there's no bottleneck or limit
10:55
to the number of humans who can run queries.
10:57
In fact, now anybody can go into that Channel and ask a question.
11:00
So those are the two interesting effects, time reduction, but
11:03
also net new insights generated as a total for for a company.
11:08
A lot of people, almost everybody in an organization
11:12
wants to be able to quickly get information and, and you know,
11:15
some organizations have really spent a lot of time structuring
11:18
how to make sure that they democratize data access rights
11:21
are the the popular term, but give an example at I food, they
11:26
have a term called bad who. So it's an acronym for business
11:30
analysts, data heavy users. And these are the people that
11:34
are basically going to need to interact with data very
11:40
regularly, continuously, daily in their jobs to be able to do
11:43
their job. Often it's in a supporting role,
11:44
but not always. And maybe part of the sales
11:46
team, the logistics team. And they realized that this was
11:51
an important skill they need to have as an organization to be
11:54
able to move fast. And so they create a whole
11:56
program. They gave this name to the to
11:58
this court of people that they gave access to a certain set of
12:00
tools. And it's exactly that
12:04
constituency that would benefit a lot from this kind of a tool
12:09
because they they are interfacing with the data all
12:12
the time. They know it intimately well and
12:14
they've got a lot of repeat questions and a lot of
12:17
organizations aren't as structured around this as As for
12:20
example, ifood. But we see that almost every
12:22
organization in the group has that need and has some sort of
12:27
a, you know, group of people who are continuously leaning on data
12:31
to kind of asking. Questions.
12:32
And if it wasn't with took on, they would be pissing off the
12:36
data team. Probably, Probably, yeah.
12:38
If they were asking things of the data teams, it reminds me of
12:42
a conversation I had with like a big media company back in the
12:45
day, right? And they said that they were
12:48
inundated with data questions continuously and so much so that
12:53
they couldn't get their work done.
12:55
Eventually what they did to fix this problem, because it was
12:58
before the agent world and any notion of being able to have
13:03
self-serve, they said, we're going to send out data advocates
13:09
into the company. And we are going to pair them
13:13
with each person and make sure that that person knows when they
13:18
ask a question of the last marketing campaign data, they
13:22
have to have a next action. So they can't just ask these
13:28
random questions that later they're not going to action
13:31
because those are what we're taking up more time for the data
13:35
team to answer and service. And if it didn't actually do
13:39
anything because oh, the marketing team now knows that
13:42
this was better than that. And, and these like small
13:44
insights that you recognize, they are very powerful because
13:49
they might just like percolate in your mind for a while and
13:52
then boom, you have a stroke of insight in the shower.
13:54
And later it's like, it's all because I had those random
13:58
queries, but for this team, they didn't have the resources to do
14:02
that. So isn't.
14:03
That silly because you want everybody to have a good
14:04
intuition of where the business is going and how we're doing.
14:07
And there's, you know, for us often we're managing by, you
14:10
know, certain types of North Star metrics and there's not
14:14
always a next action immediately.
14:17
You just want to understand where are we?
14:18
Are we on track? How does this stuff I do
14:20
contribute to that? And sure, in some cases there's
14:22
a very direct action, but sort of this feels like constraining
14:25
anybody to understand exactly so.
14:29
But I think we're here to talk about, you know, the technical
14:34
aspects, so. Might as well get into it.
14:36
Yeah, why don't we bring in Donae, who's?
14:39
Our expert, our resident expert, exactly.
14:41
We just heard the breakdown on like the why, the inspiration,
14:45
but you get to tell us what's behind the scenes, What happened
14:49
to make this actually go into production and what were some of
14:52
these headaches and what would you do differently, hopefully if
14:56
you knew then what you know now. But maybe we can just start with
15:00
like, what does it look like behind the scenes when someone
15:03
asks a question of the data? Then what?
15:06
Yeah. So the SQL Analyst, I mean it's
15:08
it is an agent. So if you're familiar with agent
15:10
frameworks, it's kind of running on those same fundamental
15:13
concepts of, you know, you have the LLM, which is the brain,
15:16
you've got the tools which are doing the actual work.
15:19
And then you've got the history, which is maintaining context,
15:22
which allows the agent to continue a conversation with
15:26
context and like kind of understanding its history to
15:30
like sort of more the details is because it is a specialized
15:34
agent, it also has specialized tools.
15:36
So I think when we talk about like chatbot agents, often it's
15:39
like, you know, web search, there's some like file handling
15:43
and these are not necessarily what the SQL agent is going to
15:47
need to do its job. So it's got specialized tools
15:50
which are specifically for it to understand context on the table,
15:54
metadata, schema, you know, data types, what the tables are used
15:58
for and where they fit into, like the broader business
16:01
context. And then I think probably the
16:04
most fundamental 1 is like the ability to execute SQL queries.
16:08
But it doesn't have specific databases to call as a tool.
16:13
Or how does it know where the data lives and which database to
16:17
go to? So we've built it in a way that
16:20
kind of to the agents perspective is a generalized
16:24
execute ability to execute sequel.
16:26
So it's an execute sequel tool. And then under the hood we're
16:29
building integrations one at a time for the different databases
16:33
you could connect to. So we, when you can set up your
16:37
agent, because this is because it is based on company's data,
16:41
they do kind of set up the agent with their context.
16:43
And part of what they set up the context is what Sequel connector
16:47
are you using and what dialect are you using.
16:49
And that's part of what the agent knows as part of its
16:53
context. And it will then execute Sequel
16:56
and let it know, OK, I need you to use the Redshift 1.
16:58
I need you to use the Snowflake one.
17:00
And this is the dialect and the version of that that I'm using.
17:05
Oh, interesting. So, so basically there's there's
17:06
almost like a knowledge base that says if it is a question
17:11
around marketing, you go to this database and this is the kind of
17:14
database. So here's the SQL that you
17:15
should be using. Yeah, it's not at this point,
17:19
like it's a bit more fixed, a bit more deterministic.
17:22
So you set up a Sequel agent for your marketing database.
17:25
And so when you set that up, you say, my marketing database is
17:28
using Postgres and it's using this dialect of Sequel.
17:31
And then anytime a question is kind of asked, asked on that use
17:36
case, the agent only kind of gets that information.
17:38
And that's where the Slack channels come in because that's
17:40
how we're managing the agent, knowing are you on the marketing
17:44
database or are you on the logistics database or are you on
17:47
the groceries database? Because it's a a specific Slack
17:50
channel for each. Oh OK, so then you don't even
17:54
have to have the agent calling different databases.
17:57
It's just that it knows I'm a marketing database agent or
18:02
doesn't necessarily know that, but that is how you architected
18:06
it. That makes.
18:06
Sense so the agent, from the agent's perspective, all it
18:08
knows is I write sequel in, you know, T sequel and this is the
18:14
database I have access to. And under the hood, we take from
18:17
those pieces of information, OK, you've got to go to, you know,
18:20
this particular executor in order to actually run the query.
18:24
One thing that I wanted to know about was are you using specific
18:30
text to sequel models, like fine-tuned in certain ways or is
18:34
it just the throwing the model, throwing it at whatever open AI
18:39
model? Yeah.
18:40
So right now it's just purely on the open AI model when we POCD
18:46
it, the the efforts around, you know, hosting your own model
18:50
like that, it wasn't necessarily worth it as well as the
18:52
additional cost because we were working with open AI generally
18:55
for this kind of work. So the POC, it made sense to
18:58
kind of use the quickest, you know, solution.
19:01
And because what we found is like even like a generalized
19:05
model like that, they're really good at generating valid SQL.
19:09
You don't need a specialist at generating SQL.
19:11
They're very, very good at that, but they're not very good at is
19:13
reasoning over the context around the table metadata.
19:17
And you don't need a specialized SQL agent for that.
19:19
You need a good reasoner. So it made sense to kind of
19:21
stick with models that were good at reasoning rather than models
19:24
that were specialized at SQL. Also, because I think we, you
19:28
know, we do a lot of model fine tuning and so on for other use
19:31
cases, we had a pretty good idea what it would take.
19:33
But at looking at the token data analysts initially we wanted to
19:37
make sure that it was usable and it needed to have certain
19:39
accuracy and reliability from a user perspective.
19:42
And generating the sequel was not the the bottleneck there.
19:45
There's a lot of other reasons sometimes it wouldn't answer.
19:48
Yes, but that's what we need it. Wasn't the text to sequel
19:51
capabilities? Yeah.
19:52
What were the reasons that it wouldn't answer if it wasn't the
19:55
sequel generation? Ironically, it wasn't so much
19:58
that it wouldn't answer it that it would continue to answer when
20:01
it shouldn't. Too much confidence.
20:04
So much confidence in these large language models.
20:07
I think one of the, it's maybe one of the first places where we
20:11
really had to sort of start thinking about prompt fine
20:14
tuning, you know, trying to set up the persona, trying to
20:18
instruct this thing away from answering questions that it
20:21
didn't know the answer to. Because these agents get like a
20:24
bit of business context and they get the user question.
20:27
And from that they are then sort of expected to evaluate the user
20:32
question for clarity. Do I have enough information to
20:35
answer the question? And very often, especially at
20:39
the beginning, it was almost impossible for it to say no.
20:43
And because. It always thought it had enough
20:44
context. And he would always sort of
20:47
assume right, it was doing. And I think this is a large
20:49
language model problem in general is where there is
20:53
ambiguity or there is space for assumption, they will do that.
20:57
And so it becomes quite unpredictable in these cases to
20:59
know what it will do, and generally what it will do is.
21:03
Even if you were prompting it like crazy to say, make sure you
21:06
ask the question this or. That we had some use cases when
21:09
we were building where we had like requirements on the amount
21:13
of information it needed to have and he would still try it on he.
21:16
Would hallucinate the requirements and of course I got
21:18
that requirement. I mean, it was one, it's
21:20
actually one of the scariest failure cases because if the
21:24
model is powerful enough in its reasoning capabilities and it
21:26
has enough examples, it will generate an answer.
21:29
It's plausible. So it's like, yeah, this, this
21:32
took 15 minutes to deliver. And it's like, yeah, that, that
21:35
sounds like, it's actually sounded like, right.
21:37
But it had, there's actually no query ran, right.
21:41
And so it's a it was a very difficult failure case to spot.
21:44
Forcing it to do it was very, very hard.
21:47
So this is one of those places where we had to find another way
21:50
to solve the problem because LLMS were not sufficient,
21:52
prompting was insufficient. And so we've kind of introduced
21:55
this idea of like a pre-check, like you ask in isolation, does
22:01
this question have enough? So the agent's job at this point
22:04
is not to generate SQL. It's job is to say, do I have
22:06
enough information? And you isolate that task and
22:10
then it's, it starts to behave a bit better.
22:12
And for that purpose, we've also built like a clarity data set,
22:15
which is a data set where we give the question and the
22:18
context. And yeah, the, the, the, the,
22:22
the test is, is there enough information here for you to
22:25
answer the question? And we test the models on per
22:27
LLM on like how good is this model at doubting itself?
22:31
And that's that's really been a big part of building the trust
22:35
in the sequel agent because the answers are plausible.
22:38
And if it's not a user who's, you know, kind of into the data
22:42
enough to see that problem. It's the perfect storm.
22:44
They quote numbers that aren't necessarily true.
22:46
And then the analysts who do know they, they lose trust in it
22:50
because it's kind of their assistant.
22:53
If they don't trust it, they won't use it and they won't give
22:55
that to people to use. And because they own it, they're
22:57
the ones that have to add people to those Slack channels that let
23:00
them query the the the actual agent.
23:03
At the sake of going on a little tangent not related to the data
23:06
analyst, I want to talk about how you are breaking down the
23:12
queries that are coming through or the the agents on the back
23:15
end that are like modular in this sense.
23:17
Because what you talked about there is this verification agent
23:22
or I don't know if you have a specific name for it.
23:25
It was. So I think in the original
23:28
version of the token analysts, we just called it like a like a
23:32
clarity check and it was, it was part of the original flow.
23:37
So as the user question comes in.
23:39
So it's not it's not really a part of the agentic framework in
23:42
the typical sense. It's part of this how the
23:44
architecture of the system is built and the architecture of
23:46
the system has this sort of pre step before it's fed to the
23:50
agent. So it goes into a single task
23:54
which is not agent specific, it's just an LLM call and then
23:57
that comes back and based on that decision either the agent
24:00
is kicked off or it goes back to the user.
24:02
Yeah, because it feels like that can be very generalized across
24:07
all of you use cases. Anytime you're interacting with
24:09
an agent, you should have that as your first step just to make
24:12
sure, like do we have enough information?
24:15
You have to give it the context that it needs to get.
24:17
But then if you build that once, you can now have that as like a
24:22
modular piece that you stick into all agents.
24:25
Yeah. And I think that's one of the
24:27
like cool things that kind of the SQL Agent has taught us
24:30
about general agents. And I think as we go into more
24:32
specific agents, because they have their task and they
24:36
understand we, when we build them, we understand this is what
24:39
we want the agent to do, building the step that goes, OK,
24:43
let's just double check, let's set up the question correctly
24:46
for this task. That's like a super useful step
24:49
to implement for any kind of agent, to kind of enforce this
24:52
uncertainty until LLM's get a little bit more human.
24:57
A little less confident now. So that was one big piece.
25:01
What were other parts on like how it fell down?
25:04
Sequel generation isn't the hard part, as you said, clarifying if
25:09
it has the enough information. I know.
25:12
I think you mentioned before that you had to create context
25:17
for the business and that was really difficult because if I
25:21
say, if I'm in marketing and I say an MQL, what does that mean?
25:25
An SQL, what does that mean? And we know, yeah, OK, MQL is a
25:28
marketing qualified lead, but what isn't a marketing qualified
25:32
lead? Yeah.
25:34
So that was, I think that's the remains the biggest bottleneck
25:39
to getting a new use cases set up.
25:42
And it is the the biggest task that any analyst who wants to
25:46
use this has to go through is setting up the context.
25:49
So when we talk about context, generally we're talking about
25:51
the schema and the schema includes table descriptions and
25:55
columns, column types and column descriptions.
25:58
There's a lot of information that's needed.
26:00
And many companies that have like quite good data platforms,
26:03
IFID was one of them, have a lot of this available.
26:06
They have, you know, very sophisticated data on their
26:09
actual tables. And even once we could kind of
26:13
plug into that, we found that when it's written for humans, it
26:17
doesn't mean it's written for agents.
26:19
So. That's an insight.
26:20
Wait, pull on that thread a little bit more.
26:23
What does that mean? It's written for humans versus
26:25
agents. Well, because the metadata for
26:27
these tables is still written by someone in marketing for other
26:32
marketing data analysts. So there's still this kind of
26:36
expected understanding of the abbreviations.
26:39
There's an expected understanding of how the data
26:43
connects to each other. I think we had like 1 funny
26:45
issue where the table someone was asking for like location
26:48
data or like based on locations and the instructions you know,
26:54
still spoke about like you have to use the country, you have to
26:57
use the country for this and someone asked for like for the
27:00
city. So other examples like financial
27:03
metrics, right, somebody would ask like what's the contribution
27:06
margin or what's GM one like gross margin one, which are all
27:11
very, very strictly defined terms in the certain context.
27:15
But the agent wouldn't know without that being, you know,
27:18
actually said, OK, gross margin 1 equals this cost or this
27:23
revenue minus this cost and so on.
27:25
And so having that available to the agent at time of then
27:30
interpreting a user's question, you know, is important because
27:32
it isn't just, you know, there's a data catalogue that describes
27:35
gross margin 1 is this, you actually need a formula, right?
27:38
And so those are the kinds of things I think we were missing
27:41
typically in the data catalogues were naturally always sort of
27:45
there's a lot of implicit company knowledge already
27:47
embedded in the way these things were described at the column
27:50
level or table schema level that the agents still didn't really
27:53
automatically be able to know or understand.
27:55
How do you just continuing on the idea of agent versus human
28:00
descriptions, how do you now think about like what needs to
28:04
happen so that the analyst gets all the data it needs or it is
28:10
more agentic in everything that you're building out?
28:13
So we, I mean, because we tended to build these especially at the
28:17
beginning quite closely with the the use cases that were.
28:21
So the teams that we're actually using the analyst, we learned a
28:23
lot about like how you need to phrase this information so that
28:26
it is clear and there's very little space for ambiguity.
28:30
And so when we have new use cases, we have, you know, like
28:33
recommendations on how you structure and instruct it.
28:37
Yeah. So it's just, I think the big
28:39
thing we've learned is that you we have to learn how to phrase
28:42
statements without ambiguity. And there's things where you're
28:46
like, oh, but it's obvious, but it's, it's stupid things like,
28:50
you know, if you have sufficient information to us, that's clear.
28:54
But what does sufficient information actually mean?
28:57
And you run into these kinds of barriers all the time,
29:00
especially when you're kind of describing what this is useful
29:03
for. And so that's kind of what we've
29:06
had to sort of learn and and the like the information we try to
29:08
give as as we're setting up new ones.
29:11
So basically anything subjective, you got to get that
29:13
out of there. Anything way, especially, yeah,
29:16
if it's subjective or if there's an assumption of knowledge, an
29:20
assumption of context or history, you have to get that
29:23
out or you have to explicitly put it in.
29:26
I mean, and when we did these onboarding use cases, it was
29:29
often like, well, imagine you've got you hired a new data analyst
29:32
and this is day one. They also wouldn't know a lot of
29:35
these things. So what are the kinds of
29:37
information you need to provide to them and you know, making
29:41
sure that this stuff is embedded into the system, that it doesn't
29:44
behave like a day one data analyst, but a hopefully year
29:47
one data analyst or maybe even more down the road.
29:49
Yeah. Well, now let's talk about how
29:52
the architecture looks, because it is calling the database right
29:58
away. Like what is it?
30:00
What does it actually go into? You said that you have the
30:02
verification step and then it generates the sequel.
30:06
Or does it call a tool that has a function where the sequel is
30:10
already generated? And this is a hot topic of
30:13
debate that I love asking people because it's like, well, do you
30:16
have thousands of tools or thousands of function calls?
30:19
Or do you let the LLM generate that sequel, which may or may
30:23
not be the best sequel that's generated?
30:26
Yeah. So we, we went through a couple
30:29
of cycles with this as when we were POC.
30:33
Of course you're working with one kind of understanding of the
30:35
problem and as you release it that that changes.
30:37
So the big thing is where do you generate sequel?
30:41
I initially we did generated as a separate tool.
30:44
So the tool was sort of like generate sequel and it would be
30:48
given the context generally like the business rules and the user
30:50
question, it would come back with the sequel and then that
30:52
would be sent over to a an additional tool which would
30:55
actually run the execution. But the kind of problem as
30:59
context and the problems we were working on got harder and harder
31:01
with that is sharing enough information to the tool that
31:05
generates the sequel for it to generate sequel that is valid.
31:09
So at the moment we generate sequel as part of the main agent
31:14
is like responsibility. And that thing is passed through
31:18
as like a parameter. But it's, it's one of those
31:21
things where you are sort of having to sort of weigh up the
31:25
the the consequences of that. Because also what we then found
31:28
was if that happens, it generates sequel as part of its
31:32
main flow, which is used in a function is if it breaks, you
31:36
have to go back to the main agent flow and kind of start
31:39
from square 1. And often that's not necessary.
31:43
You know, you talk about like an analyst on day one.
31:46
And I think it's a very good way to think of this analyst is like
31:48
a trainee. So, and I think even even very
31:52
practice, sequel analysts are going to date functions.
31:56
You always have to look it up. It doesn't matter how long
31:58
you've been doing it. If you've had to swap between
32:00
date functions once in dialects, you're going to confuse them.
32:03
And that's one of those things that this agent frequently makes
32:06
mistakes on is I don't know which date function to use, even
32:09
though I know what dialect I'm in.
32:12
And it's a waste, right? So you go to execute sequel
32:14
function, execute breaks, and I've got to go back to the main
32:16
flow to fix what is probably like, you know, 3 characters
32:20
that are wrong. So we've also as as we've
32:24
learned, made these tools more sophisticated and allowed this,
32:27
the tool that's getting sequel to also update the sequel and
32:32
that, you know, so there again, like where is the sequel being
32:35
generated? Well, actually it's being
32:36
generated in two places. And that, and that's helpful
32:40
because it's really like, how much context do I need to do
32:43
this particular task? And when you're generating the
32:46
initial query, you need all of it.
32:47
But when you're fixing a date function or a small error syntax
32:50
issues, you don't need the full context, you just need the error
32:52
message. It's almost like you checkpoint
32:54
it. Yeah, and you don't have to go
32:56
back to square 1. You just come back to where
32:59
there's the problem. Yeah, exactly.
33:01
Yeah. And I want to hear a bit more
33:04
about the evolution and the design decisions that you didn't
33:07
take and why. I think one of the big things we
33:11
get asked about very often is are we using Rag and are we
33:15
using embeddings for the task of selecting relevant schema?
33:20
So getting relevant schema for the Sequel Analyst is the
33:24
biggest step. If you think about like going
33:28
through like your table documentation, finding the
33:30
correct table and then finding the relevant columns for your
33:33
question, that's the biggest part you're going to have to do
33:35
in this whole task. And that's again, for the agent,
33:37
that's going to be the biggest thing it has to do correctly.
33:40
And documentation can get quite big.
33:42
We've got use cases that have, you know, 18 plus tables and we
33:46
can't feed all of that information to the agent in one
33:50
go. Especially when we first started
33:51
building, the context just weren't long enough for us to do
33:53
that, which is why the tool was built initially.
33:57
And that process we do kind of in plain text right now.
34:02
So we store the the metadata in a structured way and that gets
34:07
fed into an LLM call along with the question.
34:10
And we through like a series of steps, filter out the relevant
34:14
tables and the relevant columns and the relevant information and
34:17
then share as little as possible back to the agent.
34:20
There is of course the option for, you know, we could have
34:22
embedded this information and embedded the user question and
34:25
kind of done it as that sort of process.
34:28
Part of the reason we didn't was just the complexity of a
34:31
solution like that because we then have to continuously be
34:34
updating those embeddings because they change quite
34:36
frequently, especially at the beginning when the analysts are
34:39
setting it up, because they're making context testing it,
34:43
changing context, testing it. And if you're doing this via
34:46
embeddings, it just gets incredibly high maintenance.
34:49
We didn't want that to start with.
34:50
And I think what we've found when we've tried other knowledge
34:53
based projects is when you're using embeddings, it's quite
34:56
difficult to embed the content in a way that will get your
35:02
question to retrieve the correct information.
35:04
And that comes against to this idea of clarity.
35:06
Like questions are short, they don't have a lot of context.
35:09
And in a table, what's being embedded like the gist of it is
35:13
quite different to what's in a question.
35:15
And so it just created such a big problem that didn't need to
35:19
be solved for us at the time. Maybe as projects get bigger, if
35:24
we start looking at like, better ways to structure questions so
35:28
that they more accurately match what an embedding for the table
35:31
would look like, that would work.
35:32
But at least initially, like it just it.
35:35
It's so much easier to also to rationalize over plain text than
35:38
it is over embeddings when it starts making mistakes
35:40
especially. Yeah, and you don't have to
35:43
figure out the pains of rag and is where are we chunking?
35:50
How are we chunking this? There's so many things where RAG
35:54
can fall over, especially if you're doing naive RAG.
35:56
And then you're like, do I need to make my RAG more advanced
35:59
here, or is that where the problem is?
36:01
And so you're adding this complexity that it sounds like
36:03
you didn't need. Yeah.
36:04
And it's just another thing you have to evaluate, right?
36:06
So now besides evaluating, can I retrieve relevant context as
36:09
evaluating is the embedding correct or not correct but
36:13
appropriate for the project. And it just, it created so much
36:16
more complexity than I think is needed for this kind of project.
36:22
Yeah, it starts in the sprawl. Really.
36:23
Yeah, because you're trying to solve one problem, right?
36:26
So what's the best tool for that?
36:28
Not the most sophisticated tool for that, Yeah.
36:32
So accuracy is like a really hard thing with a project like
36:35
this, or I think just with like SQL in general, there's a lot of
36:40
ways the same query can be written.
36:42
So you can't really go, OK, well, let's just make sure that
36:45
they match because of certain like dates factors.
36:49
You also can't just check the outputs are exactly the same
36:51
because if you ask it and the test sets not appropriately set
36:54
up, you can't just use the numbers.
36:57
So it's very difficult to measure accuracy.
37:00
And we're very reliant on the analysts who are setting this up
37:03
to kind of evaluate is it appropriate.
37:06
So initially when we were building, we tried to push
37:09
really hard to have like fixed test sets with like a defined
37:13
question, the expected SQL query and the expected output.
37:17
It's quite difficult to get people to commit to those kind
37:20
of those kind of like hard requirements, but it was a good
37:23
place to start. Also just when we were making
37:25
changes to to check that everything was working.
37:29
But I think part of what we had to what we learned when we were
37:31
doing those kinds of evaluations was again, kind of refining the
37:36
problem down to a smaller space as possible.
37:38
So when analysts set up the project, they set up or kind of
37:42
I think we call them like use cases.
37:44
So it's like the use case for collecting orders for the last
37:48
month based on, you know, said parameter and they'll say, OK,
37:52
we're 100% sure that the the agent can answer these types of
37:55
questions. And so they release that as
37:57
being a question that people can ask regularly and kind of in
38:00
that field, it can do that use case.
38:02
And then use cases have like levels of complexity as well.
38:05
So the harder A use case is, the harder it is to kind of get the
38:08
agent to reliably answer it. And so we've classed the use
38:12
cases into like difficulty and then we've classed them also
38:14
into, is this a releasable use case for users or is this
38:17
something that's still under development?
38:19
And so we also have like every analyst has two versions of
38:22
their agent available. They've got what we call the
38:23
prod and the dev. And the dev 1 is ones where
38:26
they're constantly adding new tables, they're changing context
38:29
and they're running many, many queries to test it.
38:31
And only when they're happy with context, they release it to
38:33
prod. And prod is where they're adding
38:35
the users that they would normally be serving or the other
38:37
analysts that would need the agent because that's where
38:41
they're trusting those use cases to be, correct.
38:44
We don't get a lot of hard numbers from the analysts on in
38:48
terms of like, you know, we've got 6 use cases and you know,
38:51
80% of them are at 100% accuracy.
38:54
We don't generally talk in metrics.
38:56
I think again, if you think about it as an analyst, they use
38:58
it, they talk to it. And when they're like, I trust
39:00
the agent understands, they kind of go, OK, you can now be
39:04
released. Well, you're, you're saying
39:06
something that's pretty cool there is how closely integrated
39:10
the development is with the analysts.
39:12
Yeah. And I think that's part of why
39:15
this has worked as well as it has is because we didn't build a
39:19
general sequel analyst and say, OK, there's the analyst, go
39:24
forth and you know, build context.
39:27
We worked very closely with the people who were going to use the
39:30
analyst and they kind of built it for them, for them.
39:34
And because they had built it specifically for them, they knew
39:37
and they trusted it and they were invested in it.
39:39
And so they pushed it. And that's a huge part, I think
39:42
of of why they it succeeded the way it did.
39:46
And yeah, it's, it's a very important process because again,
39:49
this, they have that knowledge, they have the domain information
39:52
that we don't have. The agent can't infer.
39:55
And they're the only ones who know how to share it in a way
39:59
that, you know, is, is clear. And where we helped was sort of
40:03
this, how do we get rid of the ambiguity in your descriptions?
40:06
You know, where do we deal with the fact that there's
40:08
assumptions here? And where do we put those
40:09
assumptions in explicit texts? You know, so the agent has
40:12
access to it, but we did have to work very closely with them.
40:16
And now it's a bit more mature. And because the teams we work
40:19
with share information with each other, you know, like I food,
40:22
you know, there's loads of data teams they share with each
40:24
other. They can kind of run on their
40:27
own, but that first step you, the, the analysts had to be
40:30
very, very involved in how they build the context.
40:34
And I think this approach is to highlight, you know, why this is
40:37
to you. Why is this so successful?
40:38
We, we've seen and of course thought ourselves about other
40:41
things that we've seen fail, right?
40:43
So I think 2 common ones that others have tried that haven't
40:47
sort of seen this uptake 1 is the kind of really boiled down
40:52
version of we've got a workflow that can generate a SQL query on
40:57
our data for our data, right. And so it's basically an LLMA
41:01
prompt which has some access to the business context and its
41:03
output is given a you know, human readable query, a SQL
41:08
query. But that didn't solve really
41:11
like a lot of teams have built these in house solution didn't
41:13
really solve the problem of where these thousands of other
41:16
people who can't verify where that answer is correct to run
41:20
it. And so plus an actual data
41:23
analyst doesn't need help with those simple queries.
41:25
They need help with the complex joins and the ones they actually
41:28
spend a lot of time on. So that's one Ave. where sort of
41:31
these home grown solutions build your own, you know, sequel query
41:35
writer actually don't really solve the big problem.
41:38
The other one is there are solutions out there that claim
41:40
they can entirely substitute a data analyst, right?
41:43
They can basically say, well, listen, we are just a completely
41:46
no code, you know, data talk to my data solution and there's no,
41:51
no data analyst involved. And our solution, the token data
41:55
analyst is sort of a mix in between the two because you're
41:57
interacting with a token data analyst in a Slack channel,
42:00
public Slack channel, or at least a shared Slack channel
42:02
with data analysts. And that only happens after the
42:05
data analyst has curated, honed and developed Co developed this
42:11
system to actually give sufficiently reliable answers.
42:15
And the data analyst really becomes the kind of orchestrator
42:18
supervisor of this actual agent. And, and I think that's probably
42:23
why we're at a point now where there's real usage because these
42:28
other two where you don't really solve for the actual, how this
42:30
thing gets used in production, because document agent
42:33
production. If you don't think about the
42:34
users and expectations and who else needs to contribute to this
42:36
thing to be reliable, then there's less likely for it to be
42:40
really. Adopted.
42:41
What about the complexity? Have you seen there's a red line
42:44
where it just tops out on questions that are super complex
42:48
and the agents can't get to it? Yeah, we've definitely seen and
42:53
I mean, this is something we're still talking to some teams
42:55
about because the complexity kind of has generated this.
43:00
There's a ceiling. Yeah, there is a ceiling.
43:02
So it, I maybe I give it too much character, but I kind of
43:09
think of the agent kind of always trying to solve it in the
43:11
simplest way possible. And so it was quite difficult to
43:15
convince it to sort of do very complex queries where it's doing
43:18
very complicated joins. You know there's like CT, ES and
43:21
you know multiple step queries. Yeah, No, it's basically it's
43:26
always goes to the simplest solution, which is if you go to
43:29
the complex things, it just that there's sort of a natural
43:32
ceiling where I think users still a substantial constituent
43:38
of our users constituency, our users, they actually think like
43:41
it's, it's still not good enough because they want to ask it much
43:45
more thing, many more complex things.
43:47
And it defaults to the simplest route, which is, you know, if
43:50
you look at the pyramid, it's, you know, it's a healthy share
43:52
of the questions coming in. But there's still a lot of other
43:55
more complex questions that it could and probably soon will be
43:59
able to answer but doesn't yet. And users I think find that
44:02
disappointing or frustrating sometimes.
44:05
But I think that's why it's why we also position this as it's
44:08
not going to replace a data analyst.
44:10
A data analyst is going to be able to do these very complex
44:12
queries. And it's also not going to be
44:15
necessarily for someone who knows nothing about the data and
44:19
has zero kind of insights into sort of what the information
44:24
they need, like what the technical requirements are.
44:26
It's really that middle ground where it's people who need some
44:30
assistance with the data. They have a little bit of
44:31
understanding what they're looking for and it can do that,
44:35
right? And and the where the value is
44:37
is that that analyst who can do the complicated queries doesn't
44:40
have to do it anymore. So there's this offloading or
44:44
like kind of first line where we can say, OK, try the analyst and
44:48
you can get your answer from there.
44:49
And and the then, yeah, the analysts who kind of would
44:52
normally be doing that, they can focus on those complicated
44:54
questions that and also their actual work, right.
44:57
So the insights they're trying to generate for like, you know,
45:00
for business users, for like campaign, like campaigns or like
45:03
conferences and that sort of thing.
45:05
Whereas, you know, like the sort of in between users, they can
45:10
use this to catch whatever insights they want.
45:12
And that means they can ask many more questions and they can make
45:14
more data-driven decisions than they could before just by virtue
45:17
of this thing always being online.
45:18
Yeah. There might be a couple of other
45:20
things that are, I think, interesting on the technical
45:22
front that we learned at the beginning, for example, because
45:25
it's got a validation step and it can sort of self correct and
45:30
sometimes it can go back and fetch other columns at the
45:34
beginning. One of the simple things we had
45:35
to solve for is it gets got stuck in loops, right?
45:38
So that's one thing like how do you get this thing that some
45:40
point just say I'm I'm done, right?
45:43
Like I don't know or I do know. Another thing was how do we make
45:46
sure that we have that integration set up that's agent
45:49
compatible? Because you're actually running
45:51
queries on a database like you don't want to do select star,
45:55
right? And because these are like I
45:56
said, I food the billion orders a year, right?
46:00
So if you just like start on the order database like, you should
46:03
have some kind of guardrails on what it should and should what
46:06
queries should. Yeah, the agent could get really
46:08
expensive really fast. Or drop table, obviously there's
46:11
no, you know, right access or that kind of stuff should never
46:14
be able to do that. So that those things, you know,
46:17
we had to solve, right Then you had to make sure that some of
46:21
the queries, because when people start to use a lot would
46:23
basically require that much memory that they would shut down
46:29
the database or overload and nothing would work anymore.
46:32
We had to deal with the fact that, you know, at some point
46:34
people wanted to visualize the data, right?
46:37
So, OK, well, we had a code execution tool.
46:40
So we, you know, connected that, but it needed to be in certain
46:43
standards. So how do you make sure that
46:44
the, you know, the charts are produced are correct?
46:48
We had issues on again I'm training problems are in for the
46:51
time like real hard learned. Lessons are real problems.
46:54
Other things that we we started with well like what data should
46:58
it actually return to the user. Sometimes users wanted a, you
47:02
know, a set of orders, give me a sample of orders that this they
47:06
want to CSV file back. So how do you sort of extract
47:09
what is allowable, what, you know, how do you return that in
47:12
Slack or in other environments to the user?
47:16
You know, these are, oh, other things when we went through
47:19
generations of models, right? So we've gone through various
47:23
generations of models that were doing parts of the flow.
47:27
And as we were upgrading, you know, you know, frequently on,
47:32
on, on, on a new generation, new version, the whole thing would,
47:35
you know, unexplainably start to behave very differently.
47:39
Yeah, let's stop working. But also sometimes it'd be very
47:41
different. And so you just convinced a
47:43
court of analysts to be able to phrase to phrase their questions
47:46
or the tool would, you know, look for the the column names
47:49
and search said wait, and then that wouldn't work.
47:51
Running prompt evals to solve that as you were going to
47:55
upgrade because you have all these questions I imagine from
47:58
the analysts. Did you run them through?
48:01
How did you like know if or or did you just plug in the new
48:05
model and then it was like, uh, it does a?
48:07
Little bit of both. Yeah.
48:09
Well, yeah, it is a little bit of both.
48:11
So when we did the initial transfer to try and move from
48:14
the existing model to one of the newer ones, we had some test
48:18
sets from the original use cases and we just saw massive changes
48:21
in behaviour. And it just the, the, the prompt
48:26
had been truly optimized for the model that it had been built on.
48:29
And the context that the, the the analysts had been building
48:33
and changing and, you know, updating for the agent like that
48:37
had all been optimized for this prompt.
48:38
And it was one of the first times I had seen that you can
48:41
also we don't make machine learning models anymore, but you
48:44
can still over. Over engineer, over fit the
48:48
prompt, yeah. Yeah, yeah, you can over fit the
48:50
prompt. Like that is the new risk of
48:52
like when you're working with these kinds of things as you can
48:54
over fit the prompt. So yeah, when we moved models,
48:57
we ran some of the tests we had and we saw like huge changes in
49:01
behaviour. And so we sort of had to to roll
49:04
back to an old model. And it's kind of one of those
49:07
problems. We're still trying to figure out
49:09
how to make these things sort of more agnostic to the model that
49:13
they're being given. I think as models get smarter,
49:16
maybe that'll be a little bit easier, but at the moment, yeah,
49:18
it's just the model is the model.
49:21
I think this is also space where, you know, we're now we're
49:24
recording this early 2025 when we're just seeing this next wave
49:28
of reasoning models come in there.
49:32
You know, it feels like that's a big opportunity, especially on
49:35
these kinds of fairly mechanical tasks that are very rational,
49:40
right? We're not doing create write me
49:42
creative marketing, you know, piece on no, no, we're actually
49:45
asking it to go very logically through a step of actions
49:49
pulling right certain sets of data to get to an answer.
49:51
And I think these reasoning models actually will offer for
49:55
this particular agent a ton of potential to do that well
49:59
because of this rationale, intermediate rationale
50:01
generation, the the, the inference compute that you can
50:05
sort of scale. And I think we'll soon be able
50:08
to fine tune on these kinds of tasks.
50:12
We'll probably see a leap in performance and probably a
50:17
lifting of the ceiling of what these models can do to actually
50:20
be able to answer questions using, you know, data sets,
50:23
running queries and so on. That's my prediction on the the
50:27
agent for. That's the next one.
50:30
So where's this all going? Where do you see this heading in
50:33
the future? Yeah, I think that's one of the
50:36
things I'm excited to see is, you know, especially if we're
50:38
thinking about this like pre validation step because the
50:41
reasoning will help it to understand to to reason.
50:45
Do I have enough information? And also, and this is one of the
50:48
things I'm excited is like, can we do this?
50:50
Use this pre step to generate the question in a more
50:53
structured way, which would again, remove the assumptions,
50:56
remove the gaps in knowledge, remove the ambiguities.
51:00
And that would then help down the line as well because you
51:02
would only need to have that version of the question, not the
51:04
original 1. So the reasoning 1 is I think
51:06
that's going to be where we see some finally see some big jumps.
51:09
It's and I wonder because it's funny how when you put the
51:13
context, you don't know if you've given it enough context
51:18
until you run it and you see that didn't work.
51:21
But if you could just give it to the reasoning model and say
51:24
like, is this enough context for you?
51:26
That would be easier than running the whole thing and
51:30
seeing if it fails and passes. And we could also see to some
51:33
degree what it's what it knows, what it's assuming.
51:37
And so we can diagnose more easily where the disconnect is
51:40
between what it's being given and what it's expected to do.
51:43
Yeah, because right now we're really like reliance on like
51:47
going through it and kind of guessing or, you know, it's
51:50
they're educated guesses after this long, but it's still kind
51:53
of a guess of why has the 70 billion parameter model done
51:57
this? It's a bit of a.
51:59
And we see the ability for us to like take these like that's a
52:02
deep sea car. One is doing it basically
52:04
distills its reasoning into like a smaller Quinn or Llama model.
52:09
I think that's a perfect that application of here, you know,
52:12
for for SQL generation. You can look at query and then
52:16
SQL answer pairs that we've generated in the past, you know,
52:20
stitch rationales together, figure out why did you do that.
52:23
And obviously that's the label, the training data set you need
52:25
to then use to fine tune this model.
52:27
And then it should be able to apply that reasoning about the
52:31
data sets and the kinds of questions coming to, to do the
52:34
downstream future task much better.
52:36
Yeah. So I think that's certainly
52:38
something that we'll be looking into as.
52:40
Long as the data schema doesn't change and everything stays
52:44
perfect, right? Sure, but it should.
52:46
Get also a lot more business context, yeah.
52:49
And so maybe the data schema will change, but it will be able
52:53
to do it much better than the current version of these, you
52:56
know, data analysts, which definitely when the schema
52:58
changes, you know, now it has, yeah, it makes a false
53:01
assumption, so. And they will, they can act as
53:04
assistance, right? So one of the things we want to
53:07
work on but haven't quite managed to do in a scalable way
53:12
yet is let the, you know, do some kind of evaluation of the
53:16
schema as it exists and say, OK, where's the point out the
53:19
ambiguities, point out where there's not an information check
53:21
where there's terminology that's not clear.
53:25
And if you could give that to a reasoning agent again with the
53:28
context of like question answer pairs it, it could you know
53:31
suggest OK, you need to change this which for the onboarding
53:34
process also would ease the process significantly for the
53:37
the analysts.