Latest / Agentic Conversations (formally mlops.community) / AI Data Engineers - Data Engineering After AI // Vikram Chennai // #309

Transcript

AI Data Engineers - Data Engineering After AI // Vikram Chennai // #309 · Agentic Conversations (formally mlops.community)

0:00
My name is Vikram, I'm building Ardent AI, I'm the founder and
0:06
CEO, and I like drinking lattes. Welcome back to the ML OPS
0:11
Community Podcast. I'm your host Demetrios, and
0:14
today my man Vikram is doing some wild shit around bringing
0:16
LLMS to the data engineering pipeline use case.
0:21
I love this because it is merging these two worlds and
0:26
hopefully saving some time for those data engineers that, you
0:30
know, are constantly under the gun.
0:35
Let's talk with him about how he's doing it and what
0:37
challenges he has had building his product.
0:42
And as always, if you enjoy this episode, share it with a friend.
0:45
Or if you really want to do something helpful, go ahead and
0:48
leave us a review. Give us some stars on Spotify.
0:51
It doesn't even have to be 5 of them.
0:54
You know, I'm OK with four, but you can also comment now on
1:00
Spotify itself, which will trigger the algorithm, give us
1:04
more engagement. Let's jump into this
1:06
conversation. But before we do, if you are one
1:11
of those folks that is listening to this on Spotify or on these
1:18
different podcast players, I've got a treat for you.
1:23
The next musical recommendation comes from someone that just
1:29
joined the community and told me about Cosmo Sheldrake.
1:34
Never heard of him and I am now in love.
1:36
Legend has. It at the Moss goes on the north
1:52
When the sun beams come move upon say them with daddy.
1:54
We'll let you can see that the world's winds round on an axis
1:57
of 23°. But how do you have the zombie
2:01
of the rabbits in the moon or the cow?
2:03
The thumbness was traveling soon for she.
2:04
The mountains are sitting on the tune and swap the songs is
2:16
rollers are riding on the moon. Where you at, by the way?
2:45
Are you in San Francisco right now?
2:47
In San Francisco. You've been doing meetups.
2:51
Not too much. I actually moved here like 6
2:55
months ago and so I started working out of some Co working
2:59
spaces and then sort of got like networked into the community.
3:03
So there's not as many meetups anymore.
3:05
I did a lot when I got here. I went to a lot of hackathons.
3:09
Great place to meet people. But after that, once you sort of
3:12
got, you know, a group of people, then it's sort of
3:14
naturally expands who you meet. And then it's not as much go
3:18
out, meet people yourself. It's more just like they just
3:20
enter orbit. Enter orbit.
3:24
I like it. The other thing that I was
3:28
thinking about who are the folks that you talk to as like users
3:34
of your product? So for U.S., data engineers are
3:39
the primary users, which might seem a little counterintuitive,
3:45
especially since we're building AI data engineers.
3:49
But what we found is data engineers are really the people
3:52
that understand how to get things done in their stack.
3:57
So they know which pipelining tool to use, the little things
4:01
that can go right or wrong. And so when they're in control
4:04
of a tool that can do data engineering for them, then you
4:10
just get way more done way faster.
4:12
So they've been our our best users so far and we've seen them
4:16
just do incredible things. So I want to look under the hood
4:21
at how you are creating what you're creating, but you kind of
4:26
glossed over something real right there and we need to dig
4:30
into it. What is an AI data engineer?
4:34
Yeah. So it's an AI agent that's
4:37
connected to your stack that can perform data engineering tasks
4:40
like building pipelines or doing schema migrations with the
4:45
sentence. So it's very similar to
4:47
something like Devon, except for verticalized and focused on data
4:51
engineering only and really integrating deeply with that
4:56
stack. So an example of that would be
4:59
you have Airflow set up with GitHub and that's how you
5:03
control the code for your data pipelines.
5:05
And you want to build a new data pipeline.
5:07
Normally you'd have to go clone that repo, you know, find
5:11
wherever you're storing your, your Dags, your data pipelines,
5:15
and then go write all the code and then check the databases and
5:18
then figure out what to write and you know, the structure of
5:21
everything, the schemas and all that mapping.
5:23
Or if you're calling from an API endpoint, you have to understand
5:26
what that endpoint will drop. So you have to do all this work
5:30
and then you can push a pipeline up and then test if it works for
5:33
us. Our agent has a terminal, so it
5:36
has a computer attached and it can do all of that for you.
5:40
So you just tell it, hey, I want a pipeline that does this.
5:43
It'll go look at the web, look at that API endpoint, figure
5:46
out, oh, how does this thing output?
5:49
And then build the entire pipeline for you, push it to
5:52
your GitHub repo as APR. And so you don't have to do any
5:55
of the work. You just say, hey, go do this
5:57
thing, it'll do it for you. And this is primarily with data
6:01
pipelines. Yes.
6:04
Are you also doing stuff within databases like the fun of trying
6:10
to query the databases? Or is it just the maybe you're
6:13
transforming or you're doing like?
6:16
How complex have you seen this get?
6:19
Yeah. So we have the context into
6:22
databases. So when we have users set up,
6:26
they set up with whatever pipelining service.
6:28
So if it's Airflow, if it's Dagster or Prefector, whatever
6:32
the hell they're using. But then you also have context
6:35
over over all of their databases too.
6:38
So you can connect Postgres or or connect Snowflake or
6:42
Databrick SQL or Mongo DB or whatever you use.
6:45
And so our agent actually has context over what that looks
6:50
like. And it allows you, we're
6:52
focusing specifically on pipelines.
6:54
But when it writes that code, then it'll know the schema.
6:57
So if there's a table you want to drop things into, it's not
6:59
trying to guess or make up a new table.
7:02
It knows you already have these three tables.
7:05
It'll make the most sense to drop the new information into
7:07
table one. And here's how we're going to
7:09
transform it to do that. One thing that I've heard a lot
7:14
of folks talk about is this type of coding with AI, or dare I say
7:23
vibe coding is very useful when things are quite simple, but
7:31
then it kind of falls flat once there is a lot of complexity.
7:37
And when you're at the enterprise level and you're
7:40
looking at a stack that goes back 20 years, it gets very,
7:45
very hard to have AI be able to really help out in the coding
7:54
and code generation side of the house.
7:57
Have you seen something similar where if you're trying to do 123
8:03
step pipelines, it's really good at that.
8:07
But as soon as you start bringing in this very messy data
8:12
from all over the place and it's being transformed 10 times
8:16
before, or you're asking it to be transformed 10 times so that
8:20
you can get the exact data you're looking for, it falls
8:24
flat. So we've seen a little bit of
8:27
that for sure, but I think that 2 main ways we've tried to
8:31
combat it and I think that it's kind of a general principle too.
8:35
One is managing context. So for something like we were
8:40
talking to a few companies, some enterprise companies, and they
8:42
had like 15,000 pipelines. Now obviously you can't feed
8:46
that all into a contacts window. It there's no way, one, the
8:49
contacts window doesn't scale to that.
8:51
And two, even if it did, it would get incredibly confused on
8:54
what you're asking for. So there's a a portion of
8:58
generating really, really, really good context, which
9:02
pretty much is trying to simplify the problem down for
9:04
the agent. So that's a huge portion of what
9:08
you're trying to do. And I think the other part is
9:10
specified training. So for more generalized coding
9:13
agents, I think one of the things they struggle with is
9:16
they're not designed for specific task flows.
9:19
They're trying to do everything at once.
9:21
And so because of how language models work, they're probability
9:26
engines. So if your probability
9:27
distribution is literally everything, you're going to
9:31
struggle to do the specific stuff.
9:34
But if you design products around specific verticals,
9:37
specific context, and specific outcomes, then you'll get a
9:41
much, much higher quality result.
9:43
But I think that the problem of like vibe coding and getting
9:48
errors at that scale is something that actually won't
9:50
ever disappear. But I think that's where you
9:52
have to build specialized infrastructure to make sure that
9:55
you can solve those problems. So one thing that we're
9:59
exploring right now in our next build is creating staging
10:03
environments for everything that they connect by default and
10:06
trying to make them really lightweight and ephemeral.
10:09
So let's say you make a change in your database, instead of it
10:13
going directly to whatever database you have connected,
10:16
assuming that maybe you don't have a staging environment, or
10:19
if you do have a staging environment, it'll just say,
10:21
here's the change that we're going to make.
10:23
This is what it looks like. But that also allows the agent
10:26
to go reflect on that and say, hey, this is how the system
10:29
would change. Is this correct?
10:32
And so even if it screws it up on time one or time 2 or time 3
10:35
or whatever, even 10 times, it's not getting committed to your
10:39
actual code base or to your database or whatever.
10:42
And it has the ability to reflect on that.
10:45
And so it can then correct itself.
10:47
And the accuracy over a bunch of different trials goes up a lot.
10:53
Wow, OK, now are you also creating some kind of an
11:00
evaluation feedback loop so that you understand what is
11:06
successful and what is not? So we've actually created a a
11:10
benchmark because there really doesn't exist a good one for
11:12
data engineering. Well, I was going to even say,
11:15
it may have to be on a case by case basis or have you found it
11:20
can be a bit more generalized with just data pipeline
11:24
evaluation benchmark? Yeah, it, I think it actually
11:28
can be a lot more generalized, but I think that's specifically
11:31
because of how data engineering works.
11:33
It's not exactly general. It's because the tools that
11:35
people use are pretty standard, like they're using Airflow or
11:39
Prefect or Dagster or Databricks or whatever.
11:42
You can actually name every single one of the popular tools.
11:46
And because of how important data is to companies, it's very,
11:50
very unlikely that people are going to use some kind of
11:54
unknown tool that no one's really battle tested.
11:57
Because if you have an error with your database and for some
12:00
reason they've, you know, coded it really, really badly and it
12:03
deletes everything, you're screwed.
12:04
Like you're never going to take that risk.
12:06
So, so people always, you know, tend to go with the, the
12:11
standards and that creates a nice framework for us to be able
12:14
to test in or we can say as long as it can operate, you know,
12:18
these tools really well. And we'll give it, you know,
12:21
hundreds and hundreds of tasks to optimize over.
12:23
And it's getting better at that. And we're pretty confident that
12:26
the majority of people will be served well with it.
12:30
OK. And you mentioned in the staging
12:34
environment that allows the agent to almost game out what
12:39
would happen if we made what the first agent is recommending?
12:46
It's it's almost like seeing into the future in a way and
12:48
then reflecting upon if that is the right move or not.
12:53
Can you talk a little bit more about what type of validation
12:57
checks you have? And is it all through agents, or
12:59
is there also some determinism involved in that too?
13:03
So it's mostly through the agent itself, but we can also pull in
13:09
like data quality checks and other tests that you have
13:11
running. So our goal is to replicate your
13:14
actual environment as much as possible.
13:16
So it writes A pipeline, then it runs whatever tests, if you have
13:19
them, we'll bring them in. The agent itself will be able to
13:22
see into error logs and all that kind of stuff to understand,
13:25
hey, did I write code with bad syntax or hallucinated
13:28
something? Or does the output look the way
13:31
that's expected and it takes all of that and then it can just
13:34
sort of loop until it gets things right?
13:37
That's really cool. A lot of times folks talk about
13:41
how hard it is to get right when they are talking with agents,
13:47
the ability for the agent to ask questions if it does not have
13:53
enough context. How have you gone about fixing
13:56
that problem? Because I could say like, yeah,
13:58
set me up an Airflow pipeline and use my database and the
14:04
agent might go and just set up some random Airflow pipeline
14:08
with some random database and say, here's what you asked for.
14:11
And or it could hallucinate something when it doesn't have
14:14
the right context and the right ideas, right?
14:17
So what are you doing to make sure that right off the bat the
14:22
agent doesn't go start working unless it has the right amount
14:26
of information? Yeah.
14:28
So one thing we actually added in our task flow and actually
14:32
specifically train on is if tasks are possible.
14:37
So one of our test cases, like a very simple one was we're going
14:41
to not feed it in Postgres credentials and we're just going
14:45
to tell it go build something in Postgres like add this table and
14:49
see what it does. And at the beginning it like
14:50
tried to do it exactly like you said.
14:53
And we, you know, essentially trained that behavior out.
14:56
So we added a check step in sort of the planning phase of it
15:00
where it'll try to determine if the task is possible or
15:03
impossible. And so we just tried to train
15:06
that out where it will tell you that's not possible.
15:08
You should probably not do that. And that's another LLM call.
15:13
It's part of the original 1. So the flow that we have set up
15:18
is you make a request, the agent will gather contact, search the
15:22
web, all this stuff. And then it'll give you a plan
15:25
of like, here's what I'm going to do, are you OK with this?
15:27
And you can go back and forth and just make edits.
15:30
And so if it's made a mistake, that's your opportunity to look
15:33
in detail about everything it's going to do and say, like, that
15:36
step is wrong or that step is wrong, or actually I wanted this
15:39
table or something like that, and then let it go off on its
15:43
own. But in that phase, there's a
15:45
check there that tries to say, is it impossible or not?
15:49
And if it is, then, well, it'll say please revise that.
15:54
Yeah. How do you think about really
16:00
big tasks versus trying to break up the tasks and make them
16:05
smaller? And when you're designing your
16:07
agents, trying to ideally have smaller tasks so that you have
16:12
more dependability. I think that breaking it down is
16:15
definitely the way to go. I just don't think you can get
16:20
enough from a larger task. And in US breaking it down in
16:23
the planning step, so it won't just say, hey, I'm going to
16:26
build you a pipeline, it'll say here are the steps that I'm
16:29
going to execute. Is this right?
16:31
It allows a lot more fine grained control over what you're
16:34
going to do. And yeah, you get a lot more
16:36
accuracy out of it. But it also helps on the user
16:38
side because they can see exactly what little steps are
16:42
going on. And so they also have control.
16:45
And so especially if our users are data engineers, they very
16:49
much understand what needs to be done.
16:52
And so they can really make the edits like, hey, Step 3 is
16:55
wrong. Like, or it's a little bit off
16:57
or you inferred the wrong table. Let's correct that.
17:01
And so it doesn't put all of the, the onus on the agent, at
17:04
least not yet. And there's always going to be a
17:06
little bit of a mismatch, right? Because there's no way you can
17:09
perfectly replicate everything into your brain and just dump
17:12
that into the into the context. Are you using the GitHub repos
17:22
of the organizations as context also to help when creating the
17:27
different pipelines? Yeah.
17:29
So the primary way people tend to store pipelines is through
17:33
some sort of version control software, and that's usually
17:36
GitHub. And so, yeah, the agent will
17:38
connect in sort of scan and understand that repo, look for
17:42
specific folders that are important, such as if you have a
17:45
DAGS folder, which is Airflow's name for data pipelines and all
17:50
of the, all of the, the files in there.
17:52
So it'll know that's where I write my pipeline code or that's
17:55
where I, you know, extract from. But you're not specifically
18:04
trying to vectorize all of this stuff that's in there.
18:08
You're you're just taking it at face value.
18:11
Like you don't need to over complicate things by saying,
18:15
well, let's vectorize everything that we have.
18:18
Let's throw it into a vector DV and then hopefully it can give
18:22
us better retrieval and also it will know better semantically,
18:27
like what is being asked. None of that actually matters.
18:32
Are you talking about for the code?
18:34
Yeah, or any of the stuff that you're giving it as context.
18:39
So we do actually do a bit of rag and retrieval, and the main
18:44
place we do that is pretty much on the all the context layer.
18:48
So if you ask about Postgres and Mongo DB, what we're going to do
18:52
is try to RIP out as much unnecessary context as possible.
18:55
So it's actually less of a problem of giving it the right
18:57
context. What you want to do, at least
19:00
for data engineering agents or I think coding agents in general,
19:03
is give it as much as possible because we have this this policy
19:07
of like achievable outcomes. So if you don't give something
19:13
like database credentials, I don't give you my Postgres
19:15
credentials, there's no way you're going to ever get into
19:18
that database and understand what's there in any conceivable
19:21
universe. And So what we try to do is give
19:25
the agent just enough, but remove all the unnecessary
19:28
stuff. So for example, if it does a
19:31
retrieval search and says, OK, we don't need data bricks.
19:33
We don't need any of this other stuff.
19:35
But we give it just enough so that if it did need data bricks,
19:39
then it can start pulling that context or make that part of the
19:42
the planning steps of, hey, I need to pull this thing, I need
19:45
to pull that thing, then it can still do it.
19:47
So we do add sort of a search layer on top, but it's more to
19:50
get rid of unnecessary information.
19:52
Like you think 15,000 pipelines, there's no way, right?
19:55
So you do need to index all of that and understand for a user
19:59
query what to pull it and what not to.
20:01
So if I'm understanding this correctly, it's more about how
20:06
do I tell the agent where to look if a certain question is
20:13
asked? I have that information there in
20:15
the context and it knows. Oh cool airflow.
20:19
I go and look in this area and then it gives me more
20:23
information about how to deal with airflow.
20:26
Yeah, exactly. So it's a mix of it'll pulled
20:30
stuff directly. So for the pipelines and the
20:33
code, we do index that so that it can pull in OK, Pipeline 3
20:37
and it looks like this. And here's all the code and go
20:39
edit that code and here's how it's stored in GitHub.
20:41
Go do that. But let's say it wanted
20:43
information that we've decided from that vector search is not
20:47
relevant. We still have enough information
20:50
so then it can go and find its way to it.
20:53
I see so that in case it comes up later on down the line then
20:58
it it's not just like what is data breaks.
21:00
Yes, exactly. And you're in a bit of trouble
21:06
and that it's scenario that makes a ton of sense.
21:09
Now, one thing that we talked about I think probably two
21:13
months ago with a team that was creating like a data analyst
21:18
agent, they made sure that the data analyst agents were
21:26
connected directly to different databases.
21:29
And if the agent was for marketing, it was only given
21:36
scope for the specific marketing databases.
21:40
And all of that analysis that takes place in like almost this
21:46
walled garden. Have you seen that to work or is
21:50
it? Is it something different?
21:53
I think for certain companies that make sense, especially for
21:58
larger enterprises, I think there's a good chance a lot of
22:02
that comes from security. If I had it my way and I had
22:06
access to everything I wanted, I would I would just be like, OK,
22:10
give me as much context as possible.
22:11
I'll index literally everything you own.
22:14
And you know, I designed some sort of custom embeddings on top
22:18
of that, which is some of what we do to make sure it's really
22:21
accurate on everything. So I think there's like a trade
22:26
off there where I think more context is generally better
22:29
actually in this case. But you again you want to keep
22:32
it pretty thin, right? Like you don't want everything
22:36
about everything. You want just enough about
22:39
everything, especially if you have sort of these cross
22:42
contacts flows. Now with pipelines, that's
22:46
something like sort of the call out structure where you have a
22:51
pipeline that'll trigger a data processing service to go do all
22:54
the heavy lifting. So you're not processing
22:56
millions or billions of rows of data in your airflow instance,
23:00
which will blow up like you're putting it off the data bricks
23:03
or someone else. So you kind of want that
23:05
context, right? So I, it makes a lot of sense,
23:08
especially at the enterprise scale to have it sort of guarded
23:12
off like that. And it may improve the agent
23:14
too, but I tend to go for bringing lots of context and
23:20
just keep that understanding so that when you have those flows
23:24
to sort of peek out, you know what what's happening.
23:27
Along the lines of processing data in the wrong place or with
23:32
the wrong tool that's going to make something blow up or be
23:35
really expensive after the fact. Do you have alert set up or do
23:40
you have some kind of way to estimate?
23:42
As you said, we have that staging environment.
23:45
We can estimate if this is going to work which is 1 vector, but
23:49
then I imagine another vector is.
23:50
How much will this cost? Yeah.
23:53
So what we've actually found especially for existing
23:57
customers is they actually kind of know what their cost will be
24:02
because a lot of times the work is not, I don't know what I'm
24:05
doing, tell me how to do it better.
24:08
It's I know what I need to do but I wish there was 10 of me
24:12
like I just don't want to take. So long to yeah, yeah, it's like
24:15
cumbersome. The work is very cumbersome.
24:18
Exactly. And so they off board that work
24:21
to our agent like hey, this pipeline is taking 10 minutes
24:24
and it needs to be a minute. Now I know how to do that
24:27
probably I really don't want to do that.
24:29
So can this thing just auto scale the pipeline out?
24:33
OK, cool. It will do that and they can
24:35
give it specifics on how to do that.
24:36
So we haven't seen that as much. But yeah, you will be able to
24:39
pull that stuff from the staging environment.
24:41
And somewhere we're looking to go more down the line is, you
24:45
know, being able to auto optimize sort of at that level
24:49
of like, hey, we can save you 20% on your bill if you just
24:53
changed everything like this. And our agent knows how your
24:56
costs are set up and all this stuff.
25:00
The other thing that I was thinking about on this is do you
25:03
primarily interact with the agents through Slack or is it
25:09
via web GUI? What is it?
25:11
So it's mainly through a web app and they can, yeah, they can
25:16
just push whatever changes to the agent.
25:18
So it's just a chat interface. And then we've got sort of the
25:21
the terminal style, so you can see what it's doing as it's
25:24
doing it. And then you have a bunch of
25:26
options on do you want to make it more copilot or full agent,
25:29
that kind of thing. So you've seen most of the
25:32
interaction go through there. But we also built out an SDK and
25:35
API so you can actually build the agent reactively into your
25:39
flows. So for example, a really good
25:41
example is like at 3:00 AM, you have a data pipeline fail and
25:45
you don't want to get up for that because that's ridiculous.
25:48
And so instead of having to wake up and then figure out what's
25:51
going wrong and you know, write some more code or if it's really
25:54
simple, just restart the pipeline.
25:56
Like why are you doing that at 3 AMI want to sleep?
25:58
You can put in code directly into your like error handler or
26:03
something and it will auto trigger the agent to run and you
26:06
can put in whatever text you want.
26:08
So it's the same thing you would do in the web UI, but now you
26:12
can do it in your code. And that allows those flows to
26:15
be, hey, there's an error at 3:00 AM, the agent has gone
26:19
while you're still trying to get up and shake off the sleep like
26:22
it's going and doing all the work.
26:24
And it said, hey, I found the fix here.
26:27
It is like here's APR to change, you know, whatever bug is going
26:30
on or you just need to restart the pipeline.
26:33
I'm ready to do that. Would you like to do that?
26:36
Just a yes or no. So it's not actually having full
26:39
autonomy like you were saying before, you're trying to get to
26:42
that point, but at this moment in time, it still gets that
26:46
human intervention or the green light from some kind of a user.
26:52
Yeah, I, I honestly don't know if we'll ever get to full
26:55
autonomy. And that's mainly because like,
26:58
even if you placed it in the role of, OK, we're hiring a
27:01
junior dev, would you really want them to push to prod?
27:05
Just like it's fine push to prod, just go, there's some
27:09
situations we're fine. Yeah, you probably would want
27:11
that, right? But you know, maybe you might
27:14
have some breaking changes that are a little bigger than the
27:16
initial one and you don't really want those pushed.
27:18
So I think allowing people to retain control is pretty
27:22
important, especially with agents, where they are, where
27:26
they're, I think we're at the point where they're useful, but
27:28
they're far, far from perfect. And so it's not really a good
27:34
idea in my opinion to just say do everything for me and I'm
27:38
just going to close my eyes and let you do everything.
27:41
Yeah, the idea of cognitive load with the agents and if at the
27:51
end of the day they help you take off certain amounts of
27:56
cognitive load, then that's awesome.
27:58
The better question I think would probably be around the
28:00
idea of cognitive load and how you are looking at not adding
28:07
extra burden to that end user. Because a lot of times we've
28:11
probably both felt it. We've had interactions with AI
28:15
that are not great and we come away from it thinking, damn,
28:22
that would have been faster if I just did it myself.
28:25
Yeah. I think the best thing we can do
28:28
in that scenario is just make the product better, train it
28:31
better, get it more accurate over scenarios.
28:34
I think one of the other benefits of building sort of
28:37
workflow oriented products is you get a lot of feedback from
28:40
your customers. So when something screws up for
28:43
them, you get a trace of like here are the steps and here's
28:47
everything that blew up along the way.
28:49
And so as more people use it, the better you can make it for
28:52
everyone because now you just have, you know, it goes from
28:55
thousands to hundreds of thousands to millions of flows
28:59
coming in of OK, this happened and here's the evaluation.
29:02
Yes or no, yes or no. And you can use that to just
29:05
make it great. So I think the best thing to do
29:07
honestly, is just make it better.
29:09
And then, you know, building those sort of staging bits and
29:12
the context bits that allows the agent to actually do good work
29:17
because I think that's the only real solution to it, right?
29:20
Like you can Band-Aid fix it and say, well, it won't affect
29:23
anything real, but the real thing is you want it to do good
29:26
work. That's why you're buying it.
29:27
That's why you're doing anything with any of these tools.
29:31
And so I think that's like the only real solution.
29:35
It is nice that you play in the field of the suggestions or the
29:43
actions that the agent takes. Has very clear evaluation
29:48
metrics. It's it works, it runs like data
29:52
is flowing or it's not. And on other agentic tasks, I
29:57
think what I've heard from some really awesome people is the
30:02
closer you can get to like runs does not run, the better it's
30:08
going to be for an agent because you can clearly decide if it was
30:12
a success or if it wasn't. I totally agree with that.
30:16
And that's actually part of how we train in sort of the the
30:21
benchmark training set, whatever you want to call it.
30:25
So it's like an eval and train set there.
30:27
We actually do have deterministic outcomes.
30:29
So we have everything from super simple stuff like just testing
30:34
if the agent even understands the database, right?
30:36
Like put something in database, add a name and an age as a
30:41
record into, you know, our Postgres database.
30:43
Super easy, go do that or make this table and then you
30:47
validate, did those things actually exist?
30:49
Are they there? And you add layers on top of
30:51
that of like, how long did it take?
30:53
And you pull all of that back. And so when you're optimizing,
30:57
you're not just optimizing on the simplest, like, you know,
31:01
maybe an LLM to evaluate the code.
31:03
Did this look right? Are there errors?
31:05
It's like we actually saw the data end up where it needs to
31:08
go. We saw how long it took, we saw
31:11
fast. The actual query ran like this
31:13
is too slow, this is too fast. And especially when you're
31:16
training, that's really useful feedback, right?
31:20
Yeah, we had my buddy Willem on here, probably, I don't know
31:27
when, a couple months ago and he was talking about how for them,
31:32
they're doing something very similar, but for Sr.
31:35
ES and root causing problems and triaging things.
31:39
And they set up a knowledge graph with Slack messages that
31:44
are happening. They also set it up with code
31:47
and they set it up with I think JIRA so that you could have a
31:53
bit more of a picture when something goes wrong for that
31:58
product. It makes a lot of sense.
32:00
I wonder if you have thought about trying to do something
32:05
similar or if you're thinking that the way and the ability for
32:11
you to get the context that needs to be in that agents
32:16
context window is enough right now with how you're doing it.
32:20
I think right now where we focused is all the like, this is
32:27
what it needs to work. So all of the the coding pieces,
32:31
you know, pulling in GitHub, pulling in database context,
32:35
that's what makes it work. I think that next level of and
32:40
here's all your Slack and JIRA and this and that is just, it
32:44
gets better because essentially what people are doing right now
32:47
is they're taking that contacts that's out there already putting
32:50
it into their head and then putting it down into the prompt.
32:54
And so people are kind of transferring themselves, but it
32:57
would probably be something that we're looking at.
32:59
So we had a few customers ask for a doc attach or they could
33:04
just put in like a documentation because they'd apparently been
33:08
storing all their their contacts and all their practices in one
33:11
giant document. And we heard multiple people do
33:13
that where they're like, this is our master document on how to
33:16
write everything. I think they were using it for
33:18
like a combination of training new hires, but also just a
33:22
document like this is how we do things at this company,
33:25
especially for data engineering. And so they wanted to be able to
33:28
attach that as like a permanent fixture of like this is how
33:32
you're supposed to do everything.
33:33
So we've gone down the road a little bit, but I definitely
33:36
think it's the right approach of trying to pull in even more.
33:39
Again, you get into like the context balance issue, but I
33:42
think if done right and done, you know, the, the gold standard
33:45
there's like if it's done perfectly, then you do gain a
33:49
lot more than I think you'll ever lose.
33:52
Speaking of that, doc attach I've.
33:55
It's not the first time I've heard it and it's in a little
33:58
bit different of a scenario, but the whole idea is how can we
34:03
create some kind of a glossary for the agent to understand us
34:11
when we talk about terms that are native to our company or
34:15
when we want things done that the way that our company does
34:19
them? Because maybe it's different in
34:21
every company. Or maybe what we mean when we
34:25
say I need to go get MQLS from the database.
34:29
What is an MQL? It's not necessarily labeled as
34:33
an MQL in the column so or so. It's not like it is a clear cut
34:39
and dry thing and the agent needs to know either how to
34:43
create that sequel statement to figure out what an MQL is, or
34:47
they need to understand what that actually means and which
34:51
column that relates to. No, absolutely agree with that.
34:56
Actually, I think one of our core principles is that we're
34:59
not going to make you migrate anything.
35:01
And so that doesn't that's not just about like databases or you
35:05
know, sort of that code level stuff.
35:07
It's like we want to work the way you work.
35:09
We don't want to tell you how to do things.
35:12
And I think there have been products, especially in the past
35:14
before the whole AIAI wave is they've kind of tried to make
35:20
decisions for you because there was a real trade off, right?
35:24
And they were like, here's a better way of doing things.
35:26
You'll save a ton of time. But now we're in this place
35:28
where, you know, maybe the switching might be a little bit
35:33
easier or you can sort of have cross contacts tools, but still
35:37
migrations changing the way you do things is like the worst
35:40
thing you can ask someone to do, in my opinion.
35:42
If you say, hey, switch your database, they're going to look
35:45
at you like, are you crazy? Like for, for your AI is great,
35:49
but not, we're not switching our database for that.
35:51
Like, please leave. So we very much found that,
35:55
yeah, working the way people do is like, that's, that's how you
35:59
do it. That's how you do it, right?
36:01
And you're essentially allowing them, you know, even if you try
36:04
to improve it a little bit, you're saying while you're doing
36:06
this and you could do it better in maybe this slight way, but
36:10
it's not like we're going to come in and tell you what to do.
36:13
And actually you have to, you know, RIP out your pipeline
36:16
service and use our custom pipeline tool that we've built
36:19
with AI in it. Like, no, just use your stuff.
36:22
This thing will drop in. It'll solve your problems.
36:27
And along those lines, do you have something that is like
36:32
common asks or common patterns, common requests of the agents
36:38
that you've codified and you figured out, OK, this pipeline
36:45
is being requested like twice a day or twice an hour.
36:50
And so maybe we can just make that A1 button click instead of
36:53
having to have someone prompted every time.
36:58
We haven't seen as much of that directly.
37:01
I actually think that's a little bit different than the
37:03
philosophy we're going after because usually for pipelines,
37:08
first they're set up once and then they're managed.
37:11
So it's rarely like recreate this over and over and more of
37:16
we've created it and now we have to make sure it doesn't break
37:18
and make sure everything else doesn't break at the same time.
37:22
So a lot of that and then I think a lot of our approach is
37:25
also the thing that LMS are great at is being non
37:29
deterministic and solving a wide array of problems.
37:32
And so even if we see a pattern in there that might be easier as
37:36
like a one click, just do this. I don't think that's actually a
37:40
good practice to add to the agent because you're trying to
37:43
direct it yourself versus train on data.
37:47
So what should be happening is if you see that happen a lot and
37:51
you keep adding that as training feedback, it should get really
37:54
good at it. So you know, they might have to
37:57
prompt and ask for it, though usually they're not going to ask
38:00
to recreate a pipeline 6 times, but it'll just get good at those
38:05
kinds of tasks, right? Or if you want, you can just use
38:08
the API. And if you really want to
38:10
recreate a pipeline 6 times in a row, you have an API, you have
38:13
an SDK. Spread a for loop and it'll do
38:15
it six times in a row. I'm thinking about the Uber
38:19
prompt engineering Toolkit because we just had a talk on it
38:23
last week for the AI in production conference that we
38:26
did and they were talking about how they will surface good
38:30
prompts, quote UN quote, good prompts like prompts.
38:34
They people can create prompt templates, we could say, and so
38:38
that maybe it's not exactly the same thing that's being asked,
38:44
but you have the meat and potatoes of your prompt already
38:50
ready. You click on that and then it's
38:52
it's there for you and you change a few things or
38:57
extrapolating that out. I was also thinking about
39:00
another talk that we had from Linus Lee who was working at
39:04
Notion and his whole thing was with Notion AI, we just want you
39:07
to be able to click and get what you need done through clicks
39:14
without having to have that cognitive load of trying to
39:19
figure out what it is exactly that I need.
39:22
Because there are like 6 things that when it comes to Notion at
39:26
least, and I understand it's a completely different scenario
39:29
for you when it's in Notion. Maybe you want to elaborate on
39:33
to something, you want to summarize, you want to write
39:37
better, clean up the grammar. And so they give you that type
39:40
of AI feature just with a few clicks.
39:43
I think our version of that would probably be like user
39:46
specific prompt suggestions or like chat suggestions.
39:50
So if you are requesting for a pipeline or working with XYZ
39:55
pipeline a lot, then it will be able to learn from that and give
39:59
you sort of almost like search suggestions.
40:01
The same way in Google or any service, it'll say, hey, were
40:04
you thinking about asking this or this or this?
40:07
And then you know, sort of go that way.
40:09
That'd probably be the best version of it for us.
40:12
But it would definitely help, you know, people don't have to
40:15
ask for the same thing. Yeah, like 6 times You'll it'll
40:18
start to learn. Like maybe you do want to talk
40:20
about pipelines because that's all you've been talking about.
40:24
Simple humans, what are you doing no.
40:28
Talk to me about pricing because I know this can be a headache
40:32
for founders in this space specifically because like the
40:38
traditional way of saying it's seat based pricing can get
40:42
really not useful or not profitable for a company.
40:48
If everything is on usage based pricing then you run the risk of
40:54
the end user thinking twice before they use the product.
41:01
It's like, if this is going to cost me like a buck or two
41:05
bucks, maybe I should do it myself.
41:07
I, I mean, hopefully people value their time much more than
41:10
$2.00, but I know that I've been in that situation and I think do
41:15
I want to spend the $2.00 right now?
41:17
I don't know. So how, how do you look at
41:20
pricing? How are you currently doing it?
41:21
And what do you think about like as you've learned from customers
41:26
and talking to customers? So usually when we, when we sign
41:29
new customers, it's usually like a flat subscription fee that we
41:33
give them. So and they get a credit
41:35
allocation for it. So usually we evaluate what
41:38
their needs are and then we essentially come up with like a,
41:42
you know, we have a scaling of here's how many credits is to,
41:46
you know, whatever dollar amount.
41:48
And then we usually offer them a subscription on that, especially
41:51
because, you know, usually they want to either build new
41:53
pipelines or maintain existing ones.
41:55
And so they want to remove that sort of work.
41:58
And a credit is like a token or a credit.
42:01
Is it's a mixture of token and tokens and compute.
42:05
And so it's all the resources that the agents are using to
42:08
solve the task and then they get billed onto your account.
42:12
But with the way we price, you know, you'll have like 2000 or
42:16
3000 credits or something on your subscription for your
42:20
company. And so like spend away, like
42:24
those are yours to spend. Like please drop it to 0, you
42:27
know, and if you want more on top of that, then we offer sort
42:31
of a, a token basis on top of that.
42:35
So it's like, hey, you've used up all your credits for this
42:37
month. If you want re up here send
42:41
Bitcoin to this wallet address then good.
42:46
Yeah. So not yes, not exactly the
42:48
Bitcoin, not that far, but yes. Oh yeah.
42:53
Wouldn't it be nice though? Wouldn't it?
42:56
That's hilarious. The but the yeah, that makes
43:00
sense. And that is one of the pricing
43:02
patterns that I've seen because it will, it will help folks.
43:06
So you're kind of estimating as you're looking at how nasty
43:12
their data pipelines are or how many they have.
43:15
If it's that 15,001, you're going to give them a bigger
43:18
quota or think that they're going to use a bigger quota, so
43:22
you're going to give them an estimate that's a bigger than if
43:26
it's just one data engineer with a few pipelines they need to
43:30
run. Exactly.
43:31
And I and our primary goal like whenever we're on those calls is
43:34
not to say like, you know, it's not really you have three seats
43:38
and they're it's like, OK, what's your problem?
43:40
How do we solve it right? And it's making sure you have
43:43
the resource allocation to actually get your job done.
43:45
Like what we don't want is the hassle of someone coming in and
43:49
saying, well, OK, we, you know, we got you know what, let's say
43:53
we had a different model, 100 credits, and it didn't really do
43:56
the thing. It's like, yeah, that's because
43:58
it's the task you asked was not suited for that exact, you know,
44:03
scenario. Like we didn't.
44:05
Everything isn't set up right for you to be able to do that
44:09
the way you want it. Or we've set up a, a run that
44:12
runs every three days and does data quality checks or just
44:15
checks for new changes and stuff.
44:17
And, you know, is there anything wrong?
44:19
Like whatever you want the agent to do, right?
44:21
We've set it up in the SDK and like, it just stopped running.
44:24
It's like, well, yeah, because that's not what your needs
44:27
actually were. So I think it helps a lot on the
44:30
customer side where they just have to worry about getting
44:33
their job done and that's it. That's the only thing you have
44:36
to think of. And then as things start to
44:38
scale, you know, then you're like, OK, well, you can re up
44:41
credits or maybe we can change to a, you know, a, a, a
44:45
different sort of subscription or something like that.
44:48
But at least for now, plus a little bit in the future, they
44:52
just don't have to worry. Yeah, you mentioned also
44:57
compute. Why is compute involved in
44:59
there? Are you abstracting away the
45:01
pipelines themselves too? Are you like adding the compute
45:06
or are you doing the compute yourself?
45:08
So the compute comes from the agent actually running thing
45:12
like running code to get your job done.
45:15
So the fundamental way it works is it is a coding agent.
45:18
And what our thesis is that seems to be working is that
45:23
coding is like coding is a language that we've already
45:27
created to interact with every service out there.
45:30
So why would we try to rewrite it like it's already there in a
45:34
box? You want to interact with data
45:36
bricks, They have an SDK and an API.
45:38
You want to interact with Postgres like they have it for
45:40
you. Why would you try to rewrite
45:42
that? And so the way the agent worked
45:45
is it will actually just code the way that a human would to
45:48
get the job done. So if you have a GitHub repo set
45:51
up, it'll clone the repo and make the changes in the right
45:54
place for for DAG files for Airflow and then push you APR
45:59
for that. Or if it's just make a change in
46:01
my mongo DB database, it'll write the code to get that done
46:04
and then push that. And so all of that requires
46:06
compute to run that VM that the agent is given, and so that gets
46:11
passed on. Oh I see.
46:14
So it's not only the LLM calls, it is also the VM and everything
46:20
that's happening in that sandbox.
46:24
Awesome, Now what? Before we go, man, I feel like I
46:31
want to make sure that I get to ask you everything because this
46:33
stuff is super fascinating to me.
46:34
And I love the fact that you're taking like this AI first
46:38
approach for the data engineers because Lord knows every day is
46:42
hug your data engineer day. And they go through so much crap
46:46
and get so much thrown upon them that this is a tool that I
46:52
imagine they welcome with open arms.
46:56
And I guess the other piece of it is like, I imagine you
47:03
probably for fun or out of passion of building this
47:07
product, have looked at a lot of logs or looked at a lot of stuff
47:14
that the agents have been able to complete.
47:16
What is one run or something that an agent did that surprised
47:22
you that it actually was able to pull it off?
47:25
We did a bunch of testing to have it right spark code and
47:28
data bricks and to call that orchestrator pattern of have
47:33
something in Airflow trigger something out of data bricks.
47:35
And like a lot of companies do this where they use Airflow
47:39
purely as an orchestrator, which is generally a good pattern.
47:43
And we just had a, it was like a very simple like data frame
47:48
calculation that we wanted to do.
47:50
But it was a fact that it was able to use multiple services at
47:53
the same time. One shot the code properly, like
47:56
process the data elsewhere and then sort of pull that all
47:59
together in like an Airflow pipeline that didn't error out.
48:02
That's like, there's no way that thing just works, right?
48:04
Like that's insane. So I think that's, that was
48:09
probably the biggest thing where, you know, it wasn't just
48:11
the complexity of write the pipeline.
48:13
It was like, and you have to call out to a different data
48:15
processing service, which means you need to understand what are
48:18
the clusters that you're running and like all this stuff that
48:21
needs to go in. Then you also need to write the
48:23
Spark code properly to make sure that doesn't error.
48:26
And it like did it all. And I was like, oh, OK, guess
48:30
we're on to something like this is, wow, this is impressive.
48:36
It worked once and you're like don't touch anything, nobody
48:39
move. We need to pray to the LLM gods
48:43
right now that it will work again.
48:46
Yeah, I, I remember, I immediately just texted my
48:49
friend. There's like, there's no way
48:50
this thing just work. Like this is insane.
48:52
And then you know it, it's pretty cool when you have those
48:55
moments like you're running through training loops and it's
48:57
just crashing, crashing, crashing, and then it just
48:59
starts working. And then it works more and more
49:01
and more and it's just not failing anymore.
49:03
And you're just looking at that thing like, holy crap, like,
49:06
wow, this is working. Like this is unreal.
49:10
Like you think three years ago you'd have bots pretty much that
49:14
you could just say go do this super complex task.
49:17
You also need a code. You also need to understand like
49:19
an entire data bricks environment.
49:21
Oh, and you, you know, you should get it right in one or
49:23
two tries. Like that's just unreal that
49:26
it's happening.