Latest / Agentic Conversations (formally mlops.community) / Hard Learned Lessons from Over a Decade in AI

Transcript

Hard Learned Lessons from Over a Decade in AI · Agentic Conversations (formally mlops.community)

0:00
That feels like it was a 5 minute convo and we're a whole
0:02
hour back. Dude, it was a great one, huh?
0:06
Action. We did it.
0:09
We're here. Cool.
0:12
It feels like sometimes we get caught up in this like new Gen.
0:15
AI world and and we forget like half the Internet is still
0:20
powered and half of the ML is still making like the majority
0:24
of the money for folks. I would say most of the value
0:28
that people get from like decisions that come from
0:31
machines is not coming from LLMS. Right now, not yet, and
0:36
it's still debatable if it ever will.
0:39
For sure. You get to see from this spot
0:42
where you're talking to customers all the time who are
0:45
actually getting value from stuff and they have to cross
0:49
almost like this chasm. So it's kind of interesting to
0:52
think about right now like all the LLM stuff, where does it fit
0:56
in the like adoption curve or sophisticate, you know, all
1:01
these like you see the bell chart, the bell curve, right and
1:05
technology adoption curve. And you know, we've been around
1:07
for a while at Tecton and we we've had to think a lot about
1:13
that adoption curve. So just for some background,
1:17
well, maybe I'll give some background for everybody, just
1:19
kind of like how, who, who I am first.
1:22
So, so I'm Mike, I'm one of the Co founders of Tecton.
1:25
I'm our CEO. Before working at Tecton, I led
1:29
the Michelangelo team at Uber, which is the, the infrastructure
1:33
that powers all of the, the AI and machine learning and, and
1:38
kind of like decisioning that happens at Uber.
1:40
And so that's like real time decisions, slow decisions, but
1:45
very importantly, things that are in production.
1:47
And before that I worked on the the ads decisioning system at
1:51
Google. So like, which ads do you see
1:53
when you go into when you type a search right?
1:56
And also very like production oriented like it doesn't nothing
2:00
matters if you don't actually put it in front of users.
2:02
And that's how they print money. And yeah, and it's like, and
2:05
it's a really good example of this stuff really matters for
2:08
some businesses. There's some businesses where
2:11
there are a whole and we should come back to this, but some
2:12
businesses there are a whole business model evolves or
2:15
requires really smart decisions to be made automatically in the
2:19
spirit of, hey, let's really figure out how to get as much as
2:21
possible to production because that's where you kind of unlock
2:23
the value. What was getting in the way to
2:26
get to production at, at Uber? Well, there was a lot of things
2:31
at first, we didn't have a way to train models.
2:33
We didn't have a way to serve models, stuff like that.
2:36
But I had, you know, my first like month or so there, I had a
2:39
spreadsheet that I, that I built that tried to catalog like all
2:42
the places where, you know, there's hundreds of data
2:45
scientists, all the places where like people kind of wanted to
2:47
use machine learning in some way, right?
2:50
And this is back in the like late teens, 20 teens.
2:55
And what we found is that there's a lot of projects that
2:57
were super, super valuable, a lot of projects that were kind
3:00
of just like random experiments that were like, I don't know if
3:04
we should spend any time on this kind of thing.
3:05
And then we started learning some other dimensions that we
3:08
felt like we're really important to like, to catalog and
3:11
categorize to allow us to figure out how to prioritize them.
3:14
So like, does this team have it shit together?
3:17
Like, do they even know what they're doing in the first
3:19
place? That was the the official title.
3:23
That was the column pattern, right?
3:26
Do they have their shit together?
3:27
But also like, do they have enough people to, you know, if
3:30
we help them build this thing, do they have enough people to
3:33
like take it and take over ownership of it afterwards?
3:36
Because we were the central platform, right?
3:38
And so one of the things we found, so, so you know, we had
3:41
this big list of all these projects.
3:43
And so we went through this process of like helping them
3:46
out. We'd start with, you know, start
3:47
with surge pricing and then work with the ETA team and the fraud
3:50
detection team, stuff like that. And that was good.
3:52
We're, you know, knocking things out, getting things to
3:54
production. When we'd find a problem, we'd
3:56
fill the gap added to the platform, stuff like that.
3:59
Along the way we, we saw, we got like the obvious stuff, built
4:02
the model serving model training.
4:04
That's the stuff you hear about all the time.
4:05
The thing that was surprisingly like, we didn't really know
4:09
people want to talk about this that much at the time, but was a
4:12
major blocker that we practically spent a ton of time
4:15
on in every individual project was just the data pipelines.
4:19
Like there's just data engineering problem, right?
4:22
And back at this time, the industry was at this stage where
4:26
it was like, we just spent all this time, this investment we're
4:29
figuring out, it was like a really big deal to figure out
4:31
how do you get all your data together and like record all the
4:33
data is like just coming off the big data thing.
4:36
How do you bring all the data together so you can do something
4:38
with it, right? And now it was like, hey, let's
4:41
do something with it. And So what people were doing at
4:44
the time was like, let's just have a dashboard.
4:46
Let's you know, look, let's let's do descriptive things,
4:49
diagnostic things. There's a problem.
4:50
Let's figure out like what happened, but now people were
4:54
like, let's get more value, let's go to more predictive,
4:57
prescriptive things, right? And that's more like
4:59
forward-looking. And so doing that requires, you
5:02
know, a whole new set of technology, but also it requires
5:05
that strong link to the data. And so we were building those
5:07
data pipelines again and again. And we, we realized, hey, we're
5:12
doing the same thing in every single project.
5:14
Let's let's centralized this, let's automate it, just bring it
5:18
into the central ML platform. And that's what we call the
5:21
feature store. So that was kind of like the
5:22
genesis of actually Tecton because we built this feature
5:26
store. And then, you know, that wasn't
5:28
really a term at the time, but that kind of became this
5:30
inflection points in AI adoption at Uber rather.
5:34
And it was kind of this Cambrian explosion of.
5:37
Did it make it more self-serve? Super self-serve because you
5:40
didn't need us to build data pipelines for you, right?
5:42
And give you like a way to configure which data do you want
5:44
transform to which way and it'll be available to your model in
5:47
real time. And you can build high quality
5:48
training sets so you can build your models.
5:50
Dude, and you know what? There is a few different blogs
5:53
that I've recently read on the Uber ML and Data Engineering
5:56
blog, and it is such a breath of fresh air because it's all about
6:01
this predictive ML use cases and it showcases how wild you can
6:08
get when you have this baked into your culture.
6:11
It's first of all, I think with Uber, it's just in there and
6:14
folks are learning it. Even if you're not on the data
6:17
team, you can, if you want, learn a whole ton about data,
6:22
MLAI, all of that stuff. It's all about democratization
6:26
there. That was our like mission on the
6:28
team and it was kind of weird because on my literally my first
6:31
week on the job, they were like, OK, you're going to go present
6:34
the company's machine learning strategy to the CEO.
6:37
There's. No pressure.
6:39
I don't even know anything about this company of like 5%.
6:42
So that was kind of a like a, a funny meeting that, you know, it
6:46
went well, but it was like, you guys didn't really set me up for
6:48
success on this one. But but yeah, bringing all that
6:52
up because there's just like, you know, that was the stage
6:54
where it was like, how do we do ML, right?
6:56
And Uber was definitely at the front of things where we were
6:59
trying to figure out like not just how do we do it, but like
7:03
we actually want this business to be driven by it in some way.
7:07
It's it's not like a random side thing.
7:09
This was like, we want to power pricing with this thing.
7:12
We want to power fraud detection with this thing.
7:14
There's a ton of fraud. Well, and that those feel like
7:16
the big buckets like rocks and, and then you have the sandwich
7:21
is where when I was reading these blog posts, it was so
7:24
refreshing to see that even when someone signs on to Uber now,
7:29
they get different flows on what you've been identified as.
7:34
And so it's very customized on the flow.
7:37
Just when you're onboarding or when you're opening the app,
7:40
you're seeing a different thing. And that to me felt like it
7:45
wouldn't be possible unless you have that democratization for
7:48
the MLAI personalization piece of it. 100% And so that
7:54
democratization was all about like, let's make this possible
7:56
for a lot of people, right? And let's try to make it like
7:59
kind of easy. It's not like it was trivial.
8:02
And you know, anybody can go build the highest quality, like
8:06
the world's best quality thing in like 20 minutes kind of
8:09
thing. But it was like if we have a way
8:11
to get value, we put some people on it.
8:13
We had smart people on the team, they could figure it out and
8:15
build something pretty good. Right.
8:16
And you did say something else that you recognize, and I think
8:20
I read this in another one of their blogs, like the From
8:23
Predictive to Generative blog post that they did.
8:27
They talk about how different models and different use cases
8:31
have different Slas. And so if it's an experimental
8:34
AI project, you're not going to get the same kind of love from
8:38
the team if that shit hits the fan at 3:00 AM as if it is the
8:42
surge pricing model, which, you know, is driving so much value.
8:45
Absolutely and we've seen that with our customers today, like
8:48
they have so, so in going through this journey, we just to
8:52
like finish the thing and I'll complete that thought.
8:55
As you know, we recognize the value of all this data, like
8:59
bridging the gap between the data you have and like how do
9:01
you use it in some way for automated decisions.
9:04
And, and at the time, I was running an ML platform meet up
9:07
in the city here in San Francisco, and it was like
9:10
companies like Facebook and Twitter and like those, like the
9:13
obvious kind of companies, all the platform teams would get
9:16
together and just show what they're working on.
9:17
And everyone was kind of working on something like tangential.
9:20
And it was obvious that this is like a thing a lot of people are
9:22
going to need. So we started Tecton to
9:24
basically build the best version of that and bring it to
9:27
everybody, right. So we feel like we've, you know,
9:29
we kind of created this category feature store.
9:32
It's kind of morphed into a feature platform.
9:34
And, you know, we're, we're the leader of it today.
9:37
And that's what we do. I say that like nobody thinks as
9:40
much about feature problems, features, data pipelines for ML
9:44
as I do. You've been.
9:45
Doing it for the last what, 7-8 years?
9:47
Something like that. Well, I started working on this
9:49
stuff at Google in 2013, so it's been over.
9:52
A decade. Yeah, it's been a long time.
9:55
So This is why we could talk about these adoption curves,
9:58
right? A lot of ML projects back in the
10:00
day were very experimental. They were back in the day.
10:04
I'm talking about like even 2018-2019.
10:07
People were like, we would love to find a way to figure out how
10:10
to use machine learning in our business, but we don't really
10:12
know what to do with this. Can you help us today?
10:16
It's like we know this is possible and we just got to get
10:19
this stuff into production. Here's exactly what we need.
10:23
And so that means that these types of projects have a much
10:26
higher expectation of making it to production, right?
10:29
So the like kind of perception and the ROI calculation for this
10:33
stuff is pretty different. It's like it's like a lower
10:36
risk. It's like we're confident we can
10:37
do this and we kind of like know what the value is going to be.
10:41
We know we can reduce fraud. If we reduce fraud by 1%, it's
10:44
going to give us $10 million a year or, you know, whatever the
10:47
equivalent is. We get X percent more click
10:49
throughs on the website, blah, blah, blah, right.
10:51
And then in the LLM world, it's a little bit different.
10:54
It's like we don't really know if we can do this.
10:56
It's kind of like a Skunk Works project.
10:57
We don't even know if this technology can do the thing that
11:00
we aspire for it to do, and it's with the.
11:01
Reliability that we need with the.
11:03
Forget even about like the enterprise like readiness part
11:07
of it. It's just literally like we
11:08
don't even know if the thing, something we want to do is
11:10
possible. So there's a lot of those kind
11:11
of projects because people don't really understand what's fully
11:13
possible. Well, it does feel like we've
11:15
got a very clear understanding now of where predictive ML can
11:22
add a ton of value. There's use cases now, whereas
11:26
maybe in 2018 those use cases weren't as clear.
11:29
We were like 100% could we do stuff?
11:30
And when I like when we started Tecton, I remember in one of our
11:34
like investor conversations, they go, hey, what use cases are
11:37
you going to be good at? And then I was like, I don't
11:40
know, Like I don't even know all the things people use machine
11:43
learning for. So I can't tell you even which
11:45
ones are going to be the big ones.
11:46
Like I can tell you what we did at Uber.
11:48
But today it's like it's very well understood where the value
11:53
of like automated high quality automated decisions comes from.
11:57
And you know, I'm just saying that as a way to convey the
12:00
general, the more broad concept of machine learning, because
12:02
sometimes that also includes like rules based decisioning to
12:06
every one of these use cases goes through like a little bit
12:10
of its own maturity journey where if you're just going to
12:12
get started, the most basic thing is you start with an if
12:15
statement. You know, if this, if this
12:17
person is in this country, show them this.
12:19
If this person's in this country, show them this, right?
12:22
And then you can like kind of make it a little bit more
12:24
complex. Well, but if it's night time, do
12:26
this and you kind of grow this like business logic over time,
12:30
sometimes you'll adopt like a rules engine.
12:32
And this is what a lot of certain types of use cases like
12:35
in the financial world, they go really heavy on rules engines.
12:39
And this is basically just like super fancy like if statements
12:42
and case statements and stuff like that.
12:44
Then at some point they go, OK, there's things like just some
12:47
really brittle mess and there's better ways to do it.
12:50
Let's just like train a model and put a model in there.
12:52
And so, you know, the rules engine basically is a model that
12:57
you're building a model, but it's like a hand coded model in
12:59
some sense. And so there's that like kind of
13:02
maturity journey, but the the use cases where the value or the
13:06
value is accumulated is there's a there's a couple of them,
13:11
right? There's so we particularly focus
13:13
on this real time or fast decision world.
13:15
But you can think of like, you know, a lot of companies are are
13:19
basically at their core. You can look at, you know, you
13:21
can look at anything from different like different lenses,
13:24
but a lot of, especially in like the financial world, they're
13:27
basically just decisioning businesses.
13:29
Like if you go to like any fintech company, everything
13:33
related to that company is just about how do you make like, I
13:37
guess like four kinds of decisions really well, we need
13:40
to figure out how to acquire customers.
13:41
Well, so let's be really good at doing like marketing, like
13:44
automated marketing, right? We need to estimate risk for the
13:48
customer. So given that this person wants
13:51
a loan, you know, how much credit should we give them or
13:54
how should we give them a credit card or how much should we
13:56
underwrite their insurance or whatever the financial product
13:58
is that we're estimating risk for?
14:01
The third category could be fraud detection is this person
14:04
who they say they are, right? And so that's like a really huge
14:07
area. And I hear about the fraud use
14:09
case. I just in like the cartoon where
14:12
dollar signs go into my eyes. Yeah.
14:14
I mean, that's a, that's a really like large dollar sign
14:16
one, but, and it's one that anyone who touches money has to
14:19
deal with. But it's like a, a very large
14:22
category and not just the like how much resources do I allocate
14:28
to it, but how large of an impact does this set of
14:31
decisions have on my actual business performance?
14:35
Like if you're the CEO of Coinbase, you care a lot about
14:38
your fraud detection system. It's not like a random thing,
14:41
right? And so you have the acquisition,
14:43
you have the risk estimation, underwriting for your customers,
14:48
you know, fraud detection, and then you have something that
14:51
every company kind of deals with, but they don't always
14:53
think about it as automated decisions, which is just
14:55
operational things. How do we make our customer
14:57
support team like better enabled?
14:59
How do we help them do support our customers faster?
15:02
You know, there's like a 5B or maybe like a category 5, which
15:06
is like personalization, but that's a little bit broken into
15:09
like the acquisition stuff. Like how do we make our products
15:12
something that like customers like and want to use kind of
15:15
thing. And so a lot of companies really
15:18
what they're, they have to get good at is building these
15:21
decisioning systems like you want to have, if you're the CEO
15:25
of some neo bank kind of thing. Really you're thinking about
15:28
like, I need to make my business and my team really good at, at
15:33
delivering high quality decisions reliably quickly, like
15:38
quickly if that's like what it needs, right?
15:40
Like that's what the product surface requires and you want to
15:44
do it accurately so you, you know, get the value.
15:46
And accurately doesn't always mean like detect, for example,
15:50
in the fraud detection, like let's catch as many fraudsters
15:54
as possible to reject them. But the because of the decision
15:57
science behind it, like the ideal rate for fraud is not 0%
16:01
because if you're rejecting every fraudster, you're
16:02
rejecting too many good people, right?
16:05
And so having a different like threshold there where you let in
16:10
a little bit more couple more fraudsters, but you let in a lot
16:14
more good customers can actually make a really big impact to the
16:17
bottom line, the bottom line of your business.
16:19
I didn't even think this is something we hear from our
16:20
customers all the time. Well, and that's again going
16:23
back to the maturity of now that we've been doing fraud.
16:27
Well, it's not like all of a sudden we woke up and said we're
16:31
doing fraud now. It's been a constant cat and
16:34
mouse game. For sure, but but it's doing
16:37
become more sophisticated all the time too and.
16:39
Doing it with ML. Doing it with ML, but the
16:41
fraudsters are also using ML, right?
16:45
And so we have customers who are confident that their opponents,
16:51
the fraudsters are actually training ML models to like
16:56
estimate or kind of imitate their anti fraud models so they
17:00
can work around it, right. So it's like it is a constant
17:03
cat and mouse game. But that's where when you talk
17:05
about the use case maturity, which problems are people using
17:09
machine learning for versus not using machine learning for,
17:12
There's these categories of problems where it's getting very
17:15
sophisticated and fraud is one of those.
17:18
And certainly how sophisticated a use case is correlates to how
17:24
many dollars can be saved or gained by it.
17:27
You know, how can it affect the business?
17:29
So I would put like fraud in that, in that bucket.
17:31
The risk stuff, everything from like credit decisioning, loan
17:34
decisioning, under insurance underwriting, that's kind of all
17:38
something, you know, recommendations is a much
17:41
broader topic. I'm sure you talked to a bunch
17:43
of people who work on recommendations in one way.
17:46
You saw there's a bunch of like really good blog posts that give
17:50
like an overview of the recommendation, how to build a
17:52
recommender. I think Eugene wrote a cool blog
17:54
post about this, but it's like really broad.
17:57
There's like a lot of different ways to do that.
17:58
But if you go to like fraud, it's kind of like we're trying
18:01
to do the same thing again and again.
18:03
We just need to do it really well so you can go a lot deeper.
18:06
Yeah, because there's a lot of variety that you can get with
18:08
the recommender system. And that ground truth with the
18:12
recommender is debatable, I guess because you don't know if
18:18
we didn't show them this, would they have still bought it or
18:20
whatever the that recommendation could be with the fraud, it's
18:24
probably not as debatable. You can tell.
18:27
You can figure it out. Was this fraud?
18:28
Yep. You find out.
18:30
You don't always find out immediately because you find out
18:32
was there a credit card rejection or chargeback later
18:35
on. But you can you get the data
18:37
because, you know when you lost money.
18:38
Yeah. So that's like that.
18:41
That's a real thing. But people get to that point of,
18:43
you know, because every company also has to make the decision of
18:46
like, which of these decisions do we need to own in house?
18:52
Like do we need to build a system for, do we need to have a
18:54
team that knows how to build these models, etcetera.
18:57
Or which of these things should we just like, you know, buy some
18:59
API and kind of, you know, like just outsource this decision to
19:04
someone else. And so there's different paths.
19:07
Some companies say we're never, never going to buy any other,
19:11
you know, rely on anyone else. And they just kind of slowly get
19:14
their way to like building their more their thing being more
19:18
sophisticated over time. And the people who do this are
19:21
the folks for whom their proprietary data is particularly
19:25
valuable, right? It's particularly predictive of
19:27
fraud, right? And then there's people who are
19:29
like, look, I basically I'm doing the same thing as the next
19:32
guy, right? And I just want to, you know,
19:35
there's another company that's like fraud detection as a
19:37
service and I'm just going to use that as my API And you know,
19:42
like if it I'm going to, you know, it'll shoot me up to 80%
19:46
good, but it won't get me to 100% good.
19:48
And what we're seeing now with the sophistication as you get
19:50
farther in the bell curve is people are are mixing both of
19:54
them. We're building in house, but
19:56
we're using the AP is AS or the external services as extra
20:01
signals. Like a gut check.
20:03
Well, kind of like a but just an input to the model, like what is
20:06
the other model that that's the outsource model thing.
20:09
Cool. That will give me like the sense
20:11
of think of it as like there's a credit check model,
20:14
creditworthiness model somewhere that someone built based trained
20:17
on, you know, all of the people across America, right, That's
20:20
great. But you know, if you're fintech
20:24
company, maybe you maybe you service a very specific socio
20:28
economic like demographic. And so the the kind of like
20:34
national level model is not even relevant for you.
20:36
It's like an input, but it's not exactly.
20:38
And you have a bunch of data specific to those people.
20:40
So you think, hey, I can't really use these other, these
20:44
other things aren't as predictive as they could be
20:46
because they don't have my, they're not tuned for my
20:49
population, my customers. And I have all this like
20:51
proprietary data that I've collected for my customers that
20:54
I should be able to use, right? So people are combining the best
20:56
of both worlds. And that's like what happens
20:58
when you get even farther along in the maturity, in the maturity
21:01
curve, right? So bringing it all, you know,
21:03
back to like the very beginning of this, that's what we're
21:07
seeing in predictive ML People are still moving forward and
21:10
they're and it's a cat and mouse game.
21:11
So you can't just like chill out and do not there's no fraud team
21:15
that just has their fraud model and then they're just like,
21:17
cool, that's done. Let's move on to the.
21:19
Next, we're going to go figure some of those out.
21:20
Yeah, it's, it's like a, it's like a thing that that the CEO
21:24
reports in like earnings calls. It's a thing that is like a lot
21:28
of people are being hired to work on this stuff because it's
21:30
very valuable, right? And there's a whole like
21:31
category of these decisions. But on the other side of the
21:34
maturity curve, it's these these less problem and immediate
21:40
dollar sign driven projects, right.
21:46
Doesn't mean there's not value there, but it's but it's that
21:48
the ROI may not be as as obvious or well understood upfront.
21:54
The impact may not be as immediately like observable,
21:58
like like sometimes projects are like we want to run this, we
22:02
think it'll be good, but we we're not going to be able to
22:04
like see the impact immediately exploratory.
22:07
Yeah. And sometimes and then the
22:08
feasibility of the project, it could be like a question mark
22:11
and that's, you know, that's just categorizes or describes a
22:14
lot of like newer LLM type projects.
22:17
This is new cool technology and it's not even super, super new
22:21
anymore. People are getting this stuff
22:23
into production, but it's not at the like, you know, you're a
22:27
banks fraud team level of sophistication.
22:30
And so you need different things for different levels of like of
22:33
of maturity through that. Journey empowering the business
22:36
and how much it's powering the business.
22:38
And going back to what we were talking about with the Uber and
22:41
saying we've got almost these different buckets of models that
22:45
we support because we know how valuable they are.
22:49
And we can say very clearly this model gets all the love and care
22:56
possible because this is what the business is built on, 100%
23:00
this model. Yeah, We've had to get really
23:04
good at learning which category does a different use case fit
23:10
into and being able to talk to our customers and help them even
23:13
figure that out because they don't know like Uber, you know,
23:15
they're way out there in the in the maturity curve, right.
23:18
And so they have all of this experience of like, well, what
23:22
are these different categories and how should we operationalize
23:24
different levels of support, etcetera.
23:27
But if you're like a bank, you're an e-commerce company,
23:32
you got three models, you don't really know.
23:35
Maybe everything is like mission critical.
23:37
You think of it as mission critical.
23:39
So one of the things we do at Tecton is, you know, people come
23:42
to us with these problems where they're like, look, we're just
23:44
trying to like figure it out and, and we help them figure out
23:47
like, what do you actually need, right?
23:49
You, you're building a recommender system.
23:52
How important is this to your business and how critical is it?
23:57
What are the Slas that you want to take on as a business?
24:01
And if the problem is 1 where, well, you know, this actually
24:06
doesn't matter if this thing fails and we can just go press
24:08
retry, like who cares? It's we'll press retry.
24:10
We're going to use these predictions like next week
24:12
anyway. Or the predictions are just
24:14
going to go into like a slide deck where we show like some
24:17
forecast or something. Then you don't need to treat
24:19
this like you guys don't want to build all the systems that are
24:23
going to and the course and take on the corresponding cost and
24:26
overhead that comes with treating it like it's a mission
24:28
critical thing for your business.
24:30
It's just like just be fine with pressing like retry sets of use
24:34
cases that we spend a lot of time on and tend to be like
24:37
really correlated with value are when when people have decisions
24:41
that are automated at a velocity that humans can't be involved in
24:49
and at a speed that humans can't be involved in.
24:51
So like the fraud one's a good example, but also like
24:54
recommendations, real time pricing, a lot of the real time
24:57
decisions that happened in like the live customer flow where you
25:00
can't actually like have a person there doing the thing.
25:03
Yeah. Checking.
25:04
Is this what it says it is, yeah.
25:07
Or should we be serving this to this person?
25:09
Yeah. And a lot of those use cases are
25:11
like, as we were talking about before, they're like critical to
25:14
the business. Like one of our customers is one
25:16
of the biggest insurance companies in America and they
25:19
use us to do all of the logic, all of the the reasoning when
25:24
someone's signing up for insurance and they are super,
25:27
super careful about reliability. They're like, you know, one of
25:31
America's, you know, favorite companies kind of thing.
25:34
They can't ruin their. Brand, they can't go down right
25:36
and it's like a super high trust thing.
25:38
And and so when you look at how those those teams operate, they
25:42
go slower than other like kind of cool technology Bay Area
25:45
companies, but they do that intentionally.
25:47
It's not like because they're not good at technology, but it's
25:51
just that they're checking every single box along the way to have
25:55
full reliability, disaster recovery, resilience, all of
25:58
that kind of stuff. So they minimize the chance that
26:01
something bad happens because they have to be there for their
26:03
customers and it's that much more valuable.
26:04
It's an enterprise use case, yes.
26:07
You know what I was also going to ask you is how have you seen
26:10
the best teams translate the value that they are doing on the
26:16
data and ML teams to the? Business.
26:20
Oh, that's a good question. And you mean do you mean with
26:23
respect to like how do they report it up within the company?
26:26
Yeah. And you can always say like,
26:28
hey, we made or saved 10 million bucks and that's great.
26:32
But I think a lot of times it's much more nuanced than that,
26:35
right? And yeah, and sometimes you
26:38
can't be that clear on it. Or if you are, you're fudging
26:42
numbers, you know. What though I think like, it
26:44
tends to be the use cases that get funding are the ones where
26:50
it's more clear. Because before you come to, like
26:53
a cool vendor who can help you do something, you have to know,
26:56
like, is this even being prioritized in the company?
26:59
Back to the fraud example briefly.
27:01
Like that's always a priority. And it's never like, we're not
27:04
sure if we should work on this kind of thing, right?
27:06
Yeah. But there's a lot of projects
27:08
that are like, well, we think we could do this, but we don't
27:10
really know like it's value. How do they get?
27:13
Scary those ones are are actually scary because you're 1
27:17
reorg away from like that that project not existing yeah right.
27:20
Or one layoff away or whatever it is, right?
27:22
And so, so there's a couple of ways that we kind of like help
27:27
our customers. One thing we do by the way, is
27:28
we have, we have just like a, a value framework that we work
27:32
with them on. So it's like we come in and help
27:34
you ask the right questions. So we can write down like with
27:38
you, put the business case together with you so you can
27:41
show it to like, whoever's upstairs the right that value
27:45
and like, hey, This is why we should spend this money.
27:47
This is why we should work on this project in the 1st place,
27:49
right? But there's a couple of, it all
27:52
comes down to either we're gonna make more money, we're gonna
27:55
lose less money, right? It's gonna cost less, or we're
27:58
gonna reduce risk. Things are gonna be less risky.
28:01
The place that something like Tecton typically helps with is
28:04
we help you go faster as a team, right?
28:07
So you're, you need a lot of a lot of these use cases also
28:10
really depends on or it's very important for them to be able to
28:13
react to the new hacker or the new fraud vector, whatever,
28:16
right. But go from idea to in
28:18
production much quicker. That velocity.
28:20
Yeah. And that helps with like that
28:22
reactivity, that market reactivity I was just referring
28:24
to. But also, just like, you know,
28:25
we try to take something that took six months and make it
28:27
possible in six hours. And so how much, how many more
28:30
things can you do in that year as a team?
28:34
I think if if you're spending in just this is like the the logic
28:38
that a data, a data leader should be thinking about.
28:40
It's like I have, I'm spending just raw or fake numbers,
28:44
$100,000 for this person. They can do 2 cycles this year
28:48
or they can do, you know, one cycle every day.
28:50
How much more value am I going to get out of that person,
28:53
right? So that's like the speed
28:54
dimension, the second dimension is just accuracy.
28:57
Like your models will get better, right?
28:59
One of the things we do is we help you build types of
29:01
features, different signals into your models that you just
29:03
weren't able to build before. And so you're getting more of
29:07
the good information from your data going into your models and
29:10
your models are becoming more accurate, right?
29:13
So it's like new signals, but it's also a lot of companies
29:15
that go look, we have like all of this data, all of these cool
29:18
signals in different places. But the way we architected this
29:20
thing, we can only really use like we have to choose.
29:22
We got to use, it's going to be a batch, It's going to use only
29:25
the real time. All these data don't come
29:27
together. And so like, how do we use it
29:29
all together into in one thing? So we help people use all of
29:32
their data in every decision. So that's the accuracy
29:36
dimension. And then a third is just like
29:38
the reliability and scale and stuff like that.
29:41
Like as people are going through that curve and these use cases
29:44
become more important, you're moving away from being
29:47
comfortable or OK with having a hacked together duct tape thing
29:51
and you're moving to a world where you actually need that
29:54
level of resilience. And you have a bunch of
29:58
customers who've either had major outages with their
30:04
existing systems that literally cost them 10s of millions of
30:07
dollars. So there's a cost component to
30:09
it or their ongoing cost from their system.
30:11
It's just implemented in a really like inefficient way.
30:14
Like they recalculate every single thing every time they do
30:17
a prediction or whatever. And you don't need to do that
30:20
like you should. The right way to do that is with
30:23
incremental compute. And there's like nice ways to do
30:25
that. Like we can help you.
30:25
Actually, we had Rohit on here a month or two ago and he was
30:29
talking about that like you can save a lot of money just by
30:32
tuning your data preferences just a little bit, you know, and
30:37
recognizing do we need to checkpoint this data, how fresh
30:40
do we want this data? And if you are OK with it going
30:43
from an hour to a day, that can potentially save you a ton of
30:47
cash. Yep, Yep, 100%.
30:49
And so we, so those are kind of like some dimensions of value.
30:53
But you know, if you're working, if you're listening to this and
30:57
you're working on some, you know, data science or machine
30:59
learning project, the way it's been motivated to you is you've
31:05
probably heard of one of those things like we're doing this
31:08
because we've got to make the model more accurate.
31:10
And so there tends to be a primary bottleneck.
31:13
But the underlying thing is actually we care about all of
31:15
these things, but there's one thing that's more painful.
31:17
For us, well, and making the model more accurate is one piece
31:21
of it, but then you're thinking like, what does that enable?
31:26
If the model is more accurate, what does that mean?
31:28
For sure. And that depends on the use
31:30
case, right? And it's really hard for
31:33
platform teams to do this at scale.
31:35
So like if you're the ML OPS guy or the ML engineer on your
31:41
company's like platform, ML platform, there's, there's
31:45
actually not. I haven't, I haven't found it.
31:47
I would love to hear if someone else has it, but I've never met
31:50
a platform team and I struggled with this at Uber.
31:52
Who's got a really good way to collect, automatically collect
31:57
and like and like track the impact of their help to all
32:01
these, all of their internal customers?
32:03
Right. The value they're bringing to
32:04
those data scientists or whoever's using their platform,
32:07
that is a great call. And it's hard, right?
32:10
And you, the way you'd like to do it, say, well, what's the
32:13
common currency across all these?
32:15
It's dollars. So why don't we just ask all of
32:17
our customers, the teams that depend on our stuff, like how
32:20
much money is the same to you? How much money did you make?
32:22
How much money is the speed up worth to you?
32:25
Cool, let's just get a dollar value for all of these and then
32:27
just like bring them together. And you know, that's kind of the
32:32
best you can do, but it's still shitty answer, right?
32:34
So that's that's why it's in, it's a shitty answer because you
32:37
know, things change and the people who you're talking to
32:40
aren't even usually good at like giving you those answers.
32:43
And so you have a bunch of like kind of half answers and then
32:45
you got to aggregate and then they're also out of date and you
32:48
got to kind of aggregate it around them or aggregate them
32:51
together. And so it's not a very high
32:53
quality signal that you can present.
32:54
Also, you may have a great product that's bringing a ton of
32:58
value to the data scientist, but if the data scientist is working
33:01
on the wrong project, yeah, yeah, and that's not making
33:04
money. Yeah.
33:05
But that's like that's actually up to, you know, the leaders,
33:09
the the, the business leader. There's so there's someone who's
33:12
the head of risk, there's someone who's the head of
33:14
acquisitions who has to make sure that their team is working
33:17
on the right problems and, and it's OK.
33:19
You know, sometimes you work on experiments and like they fail
33:22
or whatever, but you hope you get it and it's OK to fail, but
33:24
you hope you get it right, like on average.
33:26
So you're net adding value. But you need those leaders to
33:29
like get it so they can have the right mindset about investing
33:32
the right amount, not infinitely, but investing the
33:35
right amount in this stuff to enable the team to be
33:36
successful. I do like the idea of speed of
33:39
iteration and being able to shorten that because that feels
33:42
like something that you can always use as an anchor and say,
33:45
look, this person, whatever we're paying them, it's like we
33:49
have two of them now because of the platform.
33:51
Yeah, yeah. What what some of the things we
33:54
like learn about our customers along the way, we help them try
33:57
to figure it out. It's we try.
33:59
We have a very like, hey, we're your partner, we're here to help
34:01
you kind of approach is like how many people are working on this
34:04
thing today? And if you didn't have to do
34:06
this stuff, like how many people would need to work on this?
34:08
What else could they do? Let's talk about what's the
34:11
before and after. What would this enable?
34:13
Yeah, exactly. We like write that down.
34:15
Let's be, let's be really crisp about that.
34:17
So, but it comes to like it's, that's a thing that we see
34:22
people thinking about more and more now, because we've kind of
34:26
gone through this like just actually as Tecton, we've gone
34:29
through this journey of in the early days, all this stuff was
34:32
basically impossible. You can never use streaming data
34:35
or real time data. That was like the initial thing.
34:37
Let's help people use their fast data and their decision.
34:40
And so I think of like Tecton as having gone through these three
34:43
phases, where the first phase is let's make this stuff possible
34:47
for people. And so our first customers were
34:49
people who were like, hey, we really just want to use this
34:52
real time data. We have all the streaming events
34:54
and we got to make our models better.
34:56
We don't really know how to do, it's impossible for us to do
34:58
right now. Can we use your thing and like
35:00
make it possible? And that was great.
35:02
And we can put in the work. The platform's excellent.
35:06
We'll help you guys do stuff that was never possible before.
35:08
You unlock a lot of value. The second kind of stage was
35:12
cool. I know this is like technically
35:14
possible, but like, I work in a team and I'm in name your
35:19
Fortune 100 company, right? I'm in a company with a bunch of
35:22
like special rules. We have compliance.
35:25
We have weird politics. Stakeholders.
35:28
A lot of stakeholders we need to like this needs to work for our
35:30
organization, like we need to be able to share these things
35:34
across. We need to have, I need to be
35:36
able to report up to my boss, like how much money we're
35:38
spending on this thing, have visibility controls, all of that
35:41
kind of stuff. And that was really a, a thing
35:44
that was a bottleneck for a lot of like larger companies and,
35:48
and like teams that need to collaborate is preventing them
35:52
from being able to use like modern ML tooling technology.
35:56
So that's kind of like the second kind of set of things
36:00
that we worked on as a company. I think we're just kind of like
36:03
definitely coming out of that zone where we feel really good
36:06
about like the top. If you're, if you're a Fortune
36:09
100 company, there's no reason you shouldn't be using Tecton
36:11
for your streaming and real time decisioning, right?
36:14
Then the third thing now is really interesting where if you
36:16
think about like, OK, now, if you're someone, if you're
36:19
someone who's working on these problems at any like big
36:22
company, you should be not blocked technically, you should
36:27
be not blocked from like the organizational red tape
36:29
perspective. So what's the gap between what
36:33
you're doing today and what you could be doing, right?
36:36
Well, just think about it. You're probably not the best
36:38
machine learning engineer in the world.
36:40
There's going to be someone who's a better machine learning
36:42
engineer. What would that person be able
36:43
to accomplish that you couldn't accomplish, right?
36:46
They maybe they go faster, maybe they're smarter than they come
36:48
up with like better designs, better features.
36:52
They can do the thing in a different way, right?
36:55
And our goal is to help every one of those people be the best
36:59
ML. Engineer, now it makes sense
37:00
because I saw what you guys released with helping with the
37:04
features and so this feature creation being able to almost
37:08
consult AI, which is pretty meta in a way.
37:11
Yeah, Yeah, it is. It's really cool.
37:12
So we launched the AI copilot and that's four people who are
37:16
building AI, right. So, you know, you hear about how
37:20
do I use AI to help me write code?
37:24
How do I use AI to help me like write my essay or whatever.
37:28
But, but a lot of these decisions are not a lot of the
37:32
decisions in like in an organization are not driven by
37:35
an LLM, but they have, they're still very, for whatever reason,
37:39
there's many reasons that are that make a lot of sense, but
37:42
they're, they're driven by structured decisions, by
37:44
predictive models, right? But that's great that that's a
37:48
predictive ML model, but does that mean that it needs to be
37:51
hand tuned, hand built, stuff like that?
37:55
What like, why would you expect that?
37:57
Like you are the best guy, you're going to build the best
37:59
model, right? So what we're building is a
38:02
system to like a copilot to help every ML engineer and every data
38:06
scientist build the best possible models for for their
38:11
unique circumstance. And that can be as simple as
38:13
like, hey, right now the features that you use in your
38:16
model are things that how do you come up with the features?
38:19
You guys just gotta think about them.
38:21
You just gotta invent. Them well, and I remember you
38:23
telling me back in the day, some of the best data scientists you
38:25
knew always had this intimate understanding of the data and
38:30
they would have to spend so much time with the data to get that
38:34
where it was almost second nature.
38:35
And now that is a perfect use case for AI because it can just
38:39
ingest all that data and give you those types of well, have
38:43
you thought about this? So yeah, there's two kinds of
38:45
personas here, right? There's two.
38:46
Like by persona, I mean, if you go to a bank or you go to like,
38:52
you know, big company, there's you're going to see two people
38:55
and one person A sitting in a chair, person B sitting beside
38:58
them. And they have different skill
38:59
sets. The first guy's really good at
39:02
ML stuff, doesn't really know what's going on with the
39:04
problem. He's not like, hey, I've been
39:06
dealing with credit card default or credit card chargebacks for
39:10
10 years, right? The second person is like, look,
39:12
I've been doing credit card chargebacks for 10 years.
39:16
I'm pretty good at sequel, but I'm not like an expert at
39:19
machine learning. Some companies have like the
39:21
person who can do both of those, but that's really hard to build
39:24
a whole team, right? And so they're not like
39:27
completely impossible to find, but you know, it's really, it's
39:31
really tough to find them. And even then they're not the
39:33
best in the world either of those things, right?
39:35
And So what we're trying to help people do is if you have subject
39:39
matter expertise, allow that subject matter expert to impact
39:44
your production system directly, right?
39:46
So they can convey their intuition to the system.
39:49
So you know what? Like new users who sign up from
39:53
this channel, there's always something fishy about them.
39:55
I never really knew exactly like how to catch it, but like,
39:58
that's where you should go look and figure it out.
40:00
And this system should help you figure out the right signals.
40:03
Right and sketchy ass channel that's.
40:05
So funny, they have the intuition about what's which
40:07
alleys are sketching and which aren't, right?
40:10
That's so true, and it's so in their head.
40:13
It's not like they can document all they want, but nobody reads
40:16
that documentation. Right.
40:17
And, and, and then you have like the other people who are like,
40:21
you know, to take the metaphor continue, it's like they're
40:23
really good at driving a car, but they don't know the city
40:25
kind of thing, right? They don't know anything about
40:27
the alleys. And so every ally is like, I
40:29
don't know, I'll go down it, right.
40:31
And that's also a waste of time. And so like an AI copilot can
40:36
help automatically understand what's going on in your data.
40:39
So we can guide that person in the right direction.
40:42
And So what this adds up to is like, we're going to help
40:45
automatically come up with feature ideas, automatically
40:48
author, like literally build those features.
40:50
And in Tecton, when you write like a small snippet of feature,
40:53
a small snippet of transformation code, that
40:55
becomes a fully productionized feature from right from the
40:57
beginning. So it's all about like
40:59
production pipelines for important decisioning systems.
41:03
And so that stuff's always, all the hard part is solved.
41:06
Now let's get the AI to just help people, like write the
41:09
stuff, write the right things, and write it quickly.
41:11
You've got the ability to just say, yeah, cool, I want to
41:16
create some production ice features with that.
41:19
But how are you making sure that they actually do what they say
41:24
they do? Yeah.
41:24
And what you want them to do, there's some feedback loop there
41:27
where you're evaluating. It we let people evaluate on
41:31
their own. A lot of use cases have pretty
41:35
complex systems to evaluate and they've been building it for
41:40
years and stuff like that. So we can allow you, we'll give
41:43
you the feature data back. And this is like, we're the best
41:47
at making awesome, fast, point in time correct training data.
41:50
So you can then figure it all out.
41:52
And then you can say, hey, like actually this feature sucked.
41:54
Let's just delete it. Right, because you know what, I
41:56
forgot that you plug into a predictive ML system.
42:02
So all that evaluation is very mature as opposed to you're
42:07
plugging into some chat bot and you have to figure out that
42:10
evaluation right? Exactly.
42:12
You have to get the new. Exactly like the eval systems
42:15
tend to be pretty like you already got it kind of thing.
42:18
But one of the things we are starting to do is allow you to
42:22
report your labels back to Tecton so we can so we can feed
42:26
that to the AI. So then the AI can go, it can,
42:30
it's not going to be perfect because it may not have your
42:33
specialized evaluation system. But if it knows like, hey, these
42:37
charges were fraud and these ones weren't, then what it can
42:39
do is at least let me like, you know, guide my search for a
42:42
better feature. So then I can find features that
42:45
are like at least look really predictive from the data that I
42:48
have, right? And then I can suggest me being
42:51
AI can suggest to you say, hey, like I found these like 10
42:55
features ideas. Check them out.
42:57
They're pretty different than like the other features we have
42:59
in the system. Tell me if you think they're
43:01
cool or not. Maybe you can say, hey, like we
43:03
don't like that because we never want to use that signal as a
43:06
whatever as a feature. Or maybe you say, cool, let me
43:09
try all ten of these and then I'll I'll press like deploy to
43:12
production for some of them that.
43:13
Goes back to the expert saying, no, I've been down those
43:16
alleyways, get that out of here or I actually haven't thought
43:20
about that. Let me try it.
43:22
And how quick it feels like that would make a very quick
43:25
iteration cycle too, because it suggests a bunch of features.
43:31
Now you could potentially get on the other side of the coin where
43:33
it's like when we've got a lot of noise with all these features
43:37
that it's suggesting and how quickly can you run and see if
43:41
the feature has value or not. Yeah.
43:43
So this is up to the like your own eval system because you can
43:47
run whatever you want in your own eval system.
43:50
But you do raise a good point, like a lot of our customers, one
43:54
of the things, one of the problems we tackled in the like
43:57
stage two thing I was saying where we're like, make it work
43:59
for teams, right? Is I got these different teams
44:04
are kind of working on the same thing.
44:05
And I'm sure they're just doing a bunch of duplicate work.
44:08
And each of you know, this, this domain, all of these use cases
44:12
are pretty like data intensive, which means cost, right?
44:15
And so like, I'm worried that I'm running the same pipeline on
44:18
this team. This guy rebuilt this pipeline
44:20
and now I'm just paying double the cost.
44:22
In the sub compute that's. True.
44:24
And like we have customers where like a single pipeline can cost
44:27
10s of thousands of dollars a month, you know, depending on
44:30
your scale or whatever. CFO is cringing right now.
44:33
Yeah. And so you want, you got to be
44:34
careful about this. It's like an important thing for
44:35
them. And so we have stuff to help
44:37
them, you know, find duplicates like, hey, heads up, like these
44:40
things look like they're basically doing the same thing.
44:42
You want to like kill that kind of yeah.
44:44
And we're gonna, we're gonna have a lot more launched here.
44:47
I mean, and this now is also a good time to plug.
44:50
We're definitely hiring and engineering and we'd love there
44:52
we go some excellent people who are working on these problems.
44:56
We're working on, well, basically like if you're, if
45:00
you're a hard worker and you're a humble person and you're
45:03
curious, you know, we, we want to talk to you, right?
45:07
In the USA. In in the USA, we have a couple
45:10
of people in Latam also, but in the USA generally we've got
45:15
offices in San Francisco and New York.
45:16
So if you want to work in an in office environment, hit us up.
45:21
But yeah, like we're working on these really cool.
45:23
Like these are really cool projects, right?
45:25
And they're, they're things where like we spend time not
45:28
with random teams who don't really understand.
45:32
Like if you work on an app, on the app layer, your customer is
45:35
someone who's like, like the, the person who is not working
45:41
with technology, where they're just, you're just solving their
45:44
problem directly and they don't even appreciate what you do.
45:47
And what we do is we work with like infrastructure teams, We
45:50
work with platform teams, like, you know, a lot of the coolest
45:52
companies in San Francisco, their top AI teams, you know,
45:56
they are our customers. So we work with them everyday
45:59
and we have shared Slack channels with them.
46:01
And so we have cool ideas, we run it by them.
46:03
And so we learn a lot. It's like a really cool like
46:05
environment from that perspective.
46:06
So we have a lot of like friends and you know what all the other
46:09
different companies are doing and they're and they're ML and
46:12
AI infra. Yeah, I love it because you're
46:14
marrying the two worlds and also you're looking at it as a
46:19
production line or you're thinking this.
46:23
What we're doing here on the data sense doesn't just because
46:27
it is predictive ML that we are powering very mission critical
46:32
products of the company for, it doesn't mean that we have to
46:35
totally divert ourselves or disassociate ourselves from the
46:41
lift that you can get from generative AI.
46:43
Absolutely. We see like generative AI is
46:45
like super helpful for us and in two ways. 1 is the whole thing
46:50
of like we help our customers do a better job building their
46:53
machine learning, right? So it's like it's like Gen.
46:56
AI for Tecton kind of, right. But then, you know, people use
47:00
Tecton for a lot of Gen. AI applications too, right?
47:03
And SO11, really important kind of like like pattern that we're
47:08
seeing that a lot of people are doing, especially in the
47:10
marketing side of things is doing a hybrid of applications.
47:14
So where they go this decisioning system, we have a
47:19
bunch of models that are predictive models and they feed
47:22
into a Gen. AI application to do some
47:24
generation like some personalized text.
47:27
We have a bunch of models that are predicting like what is this
47:30
person's gender? What is this person's this
47:32
thing? And they all have to kind of
47:34
work in one coherent data system that's making an inference at
47:38
one time, right? So it's like like a hybrid.
47:40
So if you think of like predictive ML, then there's this
47:43
hybrid where it's like you use both in the same application.
47:46
And then you know, what we do in Tecton is we're delivering in
47:49
any decision that's just like context that's going into some
47:53
decision making system, some model to do something.
47:56
You can build embedding syntecton power process
47:59
unstructured data or use all of those signals to, to drive a
48:03
prompt that goes into a Gen. AI model.
48:05
And so a lot of our customers are doing that as well.
48:07
And the value of doing that is you get all of your data that is
48:10
going into any decisioning system governed through 1
48:12
central location and you get like the reuse of compute across
48:16
it. So you really have like 1
48:17
platform to integrate all of your underlying data systems.
48:20
You have one hub for that, for that data to go through before
48:24
it goes it fans out to various decisioning systems.
48:26
And that's really helpful if you have like compliance or
48:28
something like that as well.