Latest / Agentic Conversations (formally mlops.community) / GenAI Traffic: Why API Infrastructure Must Evolve... Again // Erica Hughberg // #296

Transcript

GenAI Traffic: Why API Infrastructure Must Evolve... Again // Erica Hughberg // #296 · Agentic Conversations (formally mlops.community)

0:00
So my name is Erica Huberg and I work as a community advocate at
0:05
a company called Tetrate, which means that I listen to people
0:09
out in the industry and ensure that the solutions that are
0:12
being built actually address people's real needs.
0:15
And how do I take my coffee? Unfortunately, I normally take
0:19
it cold, not by choice, but I do take it black.
0:23
So I do often end up with a cold cup of coffee on my desk because
0:27
I didn't drink it fast enough. We are back in action for
0:31
another Envelops Community podcast.
0:33
I am your host Dimitrios, and talking with Erica today, I
0:38
learned a few things about the evolution of the Internet and
0:43
how we do things on computers in general, but also her theory on
0:49
how we've been plowing towards one reality of the Internet and
0:55
because of large language models, that's all been flipped
0:59
on its head. We're not going to spoil
1:00
anything, I just want to get right into the conversation.
1:03
I thoroughly enjoyed talking with Erica.
1:05
She has the best sense of humor and I appreciate her coming on
1:09
the pod. Let's do it.
1:18
I want to jump into the tofur and potatoes on this so it
1:24
sounds horrible. We'll just go with it.
1:27
Yeah, I'm not a big fan of tofu. I'm very particular about how I
1:31
eat tofu only in a pad Thai when it's crispy.
1:34
Yeah, but there's so many ways you can make tofu.
1:38
I've made some really bad tofu in my life.
1:40
I tried to make some scrambled tofu.
1:42
That was not a good idea. Yeah, that's hard.
1:47
Yeah, I failed, so I'm not doing that again.
1:50
So anyways, let's get into the meat and potatoes of this
1:53
conversation, which is the Internet is changing a little
1:58
bit. You've been spending a lot of
1:59
time on gateways and thinking about different gateways, I
2:05
think. Yeah.
2:07
What can you break it down for us and then we'll like dive in
2:10
deeper on it. Yeah.
2:11
So I do think it's helpful to sometimes take many steps back
2:14
to how the networking of the Internet and how our
2:18
applications are communicating over like the last since 2010
2:27
really, or even before? No, before then, since the early
2:31
2000s. I got the years wrong.
2:32
Look, I'm getting old. I was like, oh, maybe it's not
2:34
that long. So the last 25 years at roughly
2:39
has been really interesting in the evolution of the Internet.
2:42
So if we go back to the early 2000s, say 2000 and one 2000 and
2:47
two 2003, what was happening during that time and how this is
2:53
relevant to Jenny, I will hopefully become obvious in a
2:56
little bit. But if you were around then I do
3:00
appreciate that everyone are older than 25 years old.
3:03
There are people who are younger than 25.
3:06
But for those of us who were around, it was a very exciting
3:09
time. We were going away from dial up
3:13
to broadband and more and more people were getting access to
3:18
the Internet and it was getting more common that companies had
3:23
websites. We saw the rise of the social
3:26
media and forums. We were on forums back then.
3:30
We were on web forums talking to people we didn't know.
3:33
Very exciting and IRC channels very cool if you know what that
3:39
is. I hope you take your
3:41
supplements, but I I. Am taking so it's rings to Drew.
3:49
Oh my God. Well, so if you do remember
3:53
these things, what was happening was that with more people
3:57
getting devices and connecting to the Internet, that also meant
4:01
that we had an greater need of handling many, many concurrent
4:07
connections to web servers. So imagine Facebook, right?
4:13
You couldn't really have a social media app with only like
4:16
a few 1000 people on it. That wouldn't be very fun.
4:19
Like when I say a few thousand, we're talking single digit
4:21
thousands, not not hundreds of thousands.
4:24
Say as we were scaling up and having this more interactive and
4:29
multi connection on the Internet, we were hitting a
4:33
problem that the traditional thread based proxies were
4:39
struggling. So what does that?
4:40
What is a thread based proxies? It would mean that you had a
4:44
single connection per thread. So imagine you went to a
4:49
restaurant and you were ordering food.
4:52
So you come in and you get the waiter for your food.
4:55
You order your tofu or whatever you like.
4:58
Maybe you find your fancy tofu dish and at that point instead
5:03
of the waiter just going around serving other tables while
5:06
you're waiting for your food to cook, the waiter would stand
5:08
there by a table and wait for your food to be ready and then
5:11
come back, go back to the kitchen and and get your food
5:14
and delivered to you when it was ready.
5:15
So that waiter would be busy the entire time your food was
5:18
cooking, but will be moved into when we, the Internet exploded.
5:24
There's a problem that is known as this as the C10K problem of
5:29
handling 10,000 concurrent connections.
5:33
And this just then continues to explode as the Internet grows,
5:36
right? This problem then this is just
5:38
isolated to 10,000 concurrent. But that problem continues to
5:41
grow as the Internet grows and the more users are on there.
5:45
But that was the beginning of that problem.
5:47
So then we got waiters that actually didn't have to stand by
5:52
the table anymore and wait for your food to be ready.
5:54
Like instead of you just sitting waiting for the food, the table
5:58
to be ready, the waiter had to stand there with you.
6:00
But then we got a better model where waiters could just put in
6:04
the order in a system and go and wait other tables.
6:09
And even if your waiter was busy, there was a runner who
6:13
could go and deliver your food when it was ready because they
6:16
know what you ordered and what table you're at because there
6:19
was a system. This was the move to event
6:22
driven proxies. So now we were able to get
6:26
orders in. So that's the event.
6:28
The order came in, the request came in and we sent it to the
6:31
kitchen after your back end target server you're getting to,
6:35
they're processing it, cooking, cooking, cooking, and then you
6:40
are sharing threads. So many connections could share
6:43
threads and different workers could pick up and deliver the
6:48
the response. You didn't have to have your
6:50
specific waiter. So that was the cool thing that
6:53
happened during that time and then we started and then what
6:57
happens after that. So they're cool.
6:59
We solved the connection problem.
7:01
We're going to under lots and lots of connections.
7:02
That was exciting, right? No one has to sit at the wait.
7:05
Well, we still have to wait, but the waiter doesn't have to wait.
7:08
And then what happens is this move to like breakdown all of
7:14
our apps. So this happens around the early
7:17
2000 and 10s. So this movement really starts
7:21
kicking in around 2015. And that is, we used to have big
7:26
monoliths. And what is a monolith really in
7:29
software engineering? I like to think of it as imagine
7:32
you have a big box and all the little components that make up
7:36
your software is in this box. So how you log in is in the box
7:42
is a little maybe imagine it like a little teddy bear you've
7:46
shoved into that box. And then you have another box
7:48
full of hair ties in the box and you have lots of little
7:52
functionalities in the box. Just throw it all in one big
7:55
box. The problem with these big boxes
7:58
was as the number of uses increased, needed to start to
8:03
scale horizontally. And this is when we come into
8:06
the world of cloud engineering and like scaling horizontally.
8:09
But imagine in this big box was one was, was many different
8:14
features from viewing images to logging in to if you think about
8:20
the social media part, like loading your feed, like get all
8:23
the posts from contacts or connections of friends and show
8:28
them. That is one feature of this big
8:30
box. But this box did so many things
8:34
and we're like, oh, if we're going to scale horizontally and
8:37
with more and more uses, this is getting more and more expensive.
8:40
So scaling this big box, so cloning this big box over and
8:46
over again started to get computationally expensive
8:49
because we needed to reserve so much memory to.
8:53
There's a bunch of waste, yeah. Yeah.
8:55
So we're like, OK, well, what if we took out the teddy bear out
8:58
the box and instead of cloning the whole box, we just cloned
9:04
the teddy bear that we put in the box, right.
9:06
So now we can have with, that's where we start breaking down the
9:13
boxes into smaller boxes. So the teddy bear now has its
9:16
own box. It doesn't have to be in the big
9:18
box with all the other junk he threw in there.
9:21
Now he's got his own box and it's a much smaller box.
9:24
So now we can scale up that much more resource efficiently.
9:27
So that's drove us to the move to change to micro services.
9:33
So we went from monolith big box with lots of stuff shoved into
9:37
it to lots of smaller boxes with just one or a few items in each
9:42
box that could scale together. So it became more, much more
9:45
resource efficient. Ah, But as we started breaking
9:49
apart these boxes, we ran into networking problem, of course,
9:53
because when we had the monoliths, it was all very
9:57
straightforward from a networking and proxy
9:59
perspective. So yeah, we'd sold the multiple
10:02
connections to the proxy. Well done, everybody.
10:04
We'd done that. But now, well, where we're going
10:07
to send the traffic to. So now we had a new problem
10:11
because as we tried to make all of our systems more resource
10:14
efficient and we were scaling up all of these services, we had
10:18
this fascinating problem that, wait a minute, the stuff's
10:25
moving around. They're on different addresses
10:27
now, and as you scale up and have more teddy bear boxes and
10:33
then the teddy bear boxes are moving around as well because
10:35
someone came up with this bright idea called Kubernetes, and you
10:39
really know what Kubernetes does.
10:41
Really what it does is moving your teddy bear box and your
10:45
other boxes around on servers, like actual computer hardware.
10:50
They're called nodes in Kubernetes.
10:52
But what it really does is playing box Tetris.
10:55
That's what Kubernetes is really good at.
10:57
It's just stuck in these boxes to maximum and most efficient
11:01
resource utilization on your notes.
11:04
So now you may be moving your teddy bear around as it feels
11:07
like because maybe it fits better over there because yeah,
11:11
Kubernetes is just about playing box Tetris with computing
11:14
resources and putting them in. So OK, so now stuff's moving
11:18
around. So the network engineers,
11:20
they're pulling their hair out. They're like, we can't keep up
11:23
with all of this moving around and changing addresses because
11:26
this is where the, this is also what becomes interesting because
11:30
we didn't now just need to handle multiple connections.
11:33
Now we have to be able to dynamically and quickly update
11:36
the proxy to where it's pointing traffic to from a logical
11:40
perspective. Oh, you want to reach the teddy
11:42
bear API? OK, these are all the places to
11:45
serve the teddy bear API and now we have to.
11:48
So this is again a big shift in how we handled proxies because
11:53
up until this point proxies like nginx was statically configured,
11:59
they were not dynamically updated as your situation
12:02
changed. So that's when we saw that that
12:05
is actually what drives the introduction of proxies like
12:08
Envoy proxy that can dynamically reload configuration.
12:14
So it doesn't need to restart at all.
12:17
So as your targets are moving around, your Teddy bass are
12:22
moving around on the different boxes, they Envoy proxy doesn't
12:27
need to restart. It just gets dynamically
12:30
updated. And now, and I say onward proxy
12:32
was really created for that. It was created in the era of
12:36
breaking things apart. So now we're taking a log or
12:39
step back. And then during this micro
12:42
services era was also cool. When we start to optimise, we
12:45
want to do loads of little requests and they're light and
12:48
they're fast and this little services like our now teddy bear
12:52
service, I imagine this like little cute pink teddy bear in
12:56
my head. By the way, this this teddy bear
13:00
service, it's it was really it had to be really fast and they
13:04
need to to be really lightweight because if you want to do also
13:08
really good horizontal scaling, we need things to start fast
13:12
response. I mean the response fast kind of
13:15
comes with it. If you can build a really
13:17
self-contained and thought through service, but so we, we
13:23
now optimized how we think about networking in this micro
13:27
services era to be fast. The process itself should
13:31
respond within single digit milliseconds, right?
13:36
It shouldn't take longer. Keep this in mind now, because
13:39
what's happened with Gen. AIA, Gen.
13:42
AI like a service, a model service is fast if it returns
13:46
the response in 100 milliseconds.
13:49
So it's so if you think that the TEDDY by APR service responded
13:52
inside of itself in the space of one or two milliseconds or five
13:58
maybe, But we're talking about anywhere from like 100 times
14:02
slower to 10 times, like between 10 to 100 times slower.
14:07
That is fastest when we're looking at LLM services, for
14:12
example, the time now we're not talking network latency, we're
14:16
just talking about the process itself responding.
14:19
So it's a lot slower at responding from first bites.
14:24
Then we have other fun stuff. We'll talk about the first bite,
14:27
time to 1st bite because LLM stuff also tends to be streaming
14:32
and we really then focused on time to 1st bite.
14:35
That was the best explain like on 5 description of the
14:39
evolution of software engineering I've ever heard.
14:44
That is amazing, from the restaurant analogy to the teddy
14:49
bear to Kubernetes just being Tetris.
14:51
That is amazing and I love each little piece of that.
14:55
And so now in this world of everything being an API and our
15:00
micro. Services.
15:02
Paradigm that we're living in. One thing that is fascinating is
15:07
not only that, OK, these models are a bit slower, but they're
15:11
gigantic, right? Yeah.
15:13
That's a little different too. I think you were mentioning that
15:16
we've been building for super small, super fast type of
15:22
workloads and AP is and now we're looking at really big and
15:29
really low type of things. Yes.
15:32
It also was really interesting is the workloads for that's an
15:36
entire topic on itself like how to manage the LLM workloads in
15:41
your computer. I think there's also this very
15:45
interesting aspect where people are looking at how small can
15:48
they be because again, we are very much gravitating to how can
15:52
we make things smaller and more scalable.
15:55
There's some really interesting ideas on how you can even maybe
15:59
run really lightweight models towards the your edge stack of
16:03
your networking. For those who may not know, in
16:06
the world of the Internet, we didn't just have to deal with
16:10
lots of many connections. We have something called content
16:13
delivery networks out in on the Internet.
16:16
So when you go on a website and there's images on that, so
16:21
imagine you just go to Facebook and on Facebook or Instagram,
16:26
there are lots of photos and every time you are loading your
16:31
phone and you are looking at photos on Instagram.
16:35
When I go and look at it on Instagram in the United States,
16:40
those photos were most likely cashed on the edge, on an edge
16:46
through a content delivery network system.
16:50
Whereas when my mother back home in Sweden, because I'm from
16:53
Sweden, when she opens Instagram and looks at photos, they don't
16:57
come all the way from some data centre in the US or in Germany.
17:02
Most likely a lot of them would be coming from a CDN from really
17:07
close to her in Sweden so that it can have that perception of
17:11
fast delivery. So there's a lot of interesting
17:14
thoughts of how can we bring at least lightweight Gen.
17:18
AI serves as closer to those edges to be closer to people so
17:23
things are faster. But I, I am not an expert on
17:27
that. I just know that a lot of people
17:28
are thinking about how small can you make these actual workloads?
17:34
How efficiently can you get them closer to people?
17:37
Even people are talking about what can you realistically run
17:40
even on a client device. Yeah, exactly.
17:43
And how you see that a little bit with what the Apple
17:49
intelligence wanted to do, right?
17:52
Yeah. All of the simple queries we're
17:55
going to do on your own device. And then if you need a bit more,
17:58
we're going to bring it to the cloud.
18:00
And. It will get done, but it's going
18:02
to be all private and in our cloud type of thing.
18:05
Yeah, and yeah, I don't have the.
18:09
I think my favorite problem is how big the requested responses
18:13
have become. The actual network traffic is
18:15
getting heavier. I guess the the the interesting
18:19
question there is like what happens now with the network
18:23
traffic? Yeah, Say, first of all, we were
18:28
really optimizing for small requests.
18:30
A lot of gateways out there, by the way, have requested response
18:34
size limits, especially if you want to be able to interrogate
18:40
the request or response body and the actual content of the
18:44
request. And why would why would you want
18:46
to do that? For security reasons.
18:48
Yes, specifically for security and in the world of large, large
18:54
language models, people are very interested in interrogating both
18:59
request and response content to either protect information from
19:05
leaking. Protect your protect information
19:12
from coming in, like malicious information coming into your
19:15
system. But that's very common as
19:16
standard web application firewall challenges.
19:19
Another interesting thing that people want to do as traffic
19:23
comes in. So imagine you're making an LLM
19:26
request the query. So you have an application
19:30
developer who's like I'm going to build cool app that people
19:33
can write cool stuff into, but the person is building the
19:36
application, they are no LLM expert.
19:40
They're like, I want to have this cool thing.
19:42
So then imagine that they're like, I just want to have a
19:46
simple like LLM API to hook into.
19:48
I don't want to care about picking a model or service
19:52
provider. That's like not my expertise.
19:55
Imagine that that developer just want to send a LLM like request
19:59
to AAPI then. Why would someone want to
20:04
interrogate the actual content of this request?
20:07
Since I've seen some really interesting things that people
20:10
are doing out there, which is like the SIM, like literally
20:13
using a lightweight LLM almost to pick a model based on the
20:19
request. So to be able to run an analysis
20:22
on what is the most appropriate model for this type of the
20:25
request, you need to have access to the body to do that decision.
20:29
Otherwise you can't, right? You need to be able to access
20:33
the body. So this obviously differs on
20:36
like how big this body is where you've start running into
20:39
problems, but you can't assume that everybody's going to be
20:43
small, right? So being having architecture and
20:47
infrastructure that allows to be able to do this for more
20:51
unpredictable sizes of the request body as and then you
20:55
have the response body being able.
20:57
That's can get really complicated, especially with
21:00
streaming how you're going to try to protect that.
21:03
But yeah, I find it very fascinating.
21:05
So that's like one problem in terms of how the network traffic
21:10
shapes up differently with it being bigger, both requests and
21:16
responses and the fact that the APIs we are dealing with are
21:19
incredibly open-ended. They are not old traditional
21:24
microservices APIs. When I say they're old like now,
21:27
that sounds sad because they're they're very near as well.
21:30
But I think we can call them old now because they were very much
21:34
about being deterministic. Having very clear.
21:38
This type of request is always going to take X amount of
21:41
milliseconds, and you wanted to be very clear on that.
21:46
The slowest 1 is going to be 50 milliseconds, for example, to.
21:52
Then you may be able to do ones that are faster, but having a
21:55
very clear idea. Whereas in the world of LLMS,
21:58
this is incredibly unpredicted. If you've got and then you go
22:01
beyond LLMS, you're going to inference models for, for
22:06
images, it gets even more mental.
22:08
We start even working with bigger amounts of data or to go
22:11
into images, then you can go into video.
22:13
Now we're dealing with even larger data.
22:16
So LLMS, even though LLMS are pushing us, imagine where we're
22:20
going with the media that isn't just text.
22:27
Yeah. And what kind of infrastructure
22:29
we will need to build out for the future that we all want.
22:34
And you hear AI influencers on the Internet talking about how,
22:41
oh, well, we're just going to not have movies be made anymore.
22:45
They'll be made for us and our individual wants and what we're
22:49
looking for off of a prompt or whatever.
22:52
If we do want a world where there's going to be a lot of
22:56
these heavier files being shared around or heavier creations from
23:03
the foundational models and that's more common, what kind of
23:08
infrastructure do we need for that?
23:10
Because if it feels like we've been going in One Direction for
23:13
the past 15 years and we've really been trying to optimize
23:17
for this fast and light, yeah. But now we're needing to take a
23:25
bit of a right turn. Yeah.
23:29
And think a bit differently around how do we hold, how do we
23:34
deal with long running connections, long ipen
23:37
connections? Also.
23:38
Because there's a difference when you look at what we did
23:41
with streaming, like when you're using Netflix and you are
23:46
streaming video, when you have signed into Netflix and you are
23:50
connecting and you are streaming.
23:54
When Netflix is streaming out there with like media to your TV
23:59
or your device, they don't need to, they don't have any need at
24:03
all to interrogate the output. So therefore, you can bypass a
24:07
lot of the things that you may be, you wouldn't have to worry
24:12
about you when you do, when you look at output from LLMS or
24:18
large visual models and things like that, you actually want to
24:24
interrogate the output before giving it to a user, which is
24:27
different from what you see with streaming Gilmore Girls on the
24:33
Internet, right? They don't need to check that
24:35
the Gilmore Girls episode is actually what it is, right?
24:39
So therefore, because at one point I thought, Erica, I said
24:44
to myself, this streaming problem and doing that at large
24:47
scale fast, that's been solved, hasn't it?
24:49
So then I was like, well, that has been solved in the scenario
24:55
where you're having 9 controlled data that you, that the data you
25:02
were sending is known and controlled.
25:03
You know exactly what it is. You are not worried that it's
25:06
going to send out something that you don't know what it is,
25:09
right? So therefore that kind of
25:11
streaming is very different from what we're looking at when we're
25:13
looking at streaming responses from Gen.
25:16
AI services because the security layers and and control of the
25:22
what we're sending becomes different.
25:24
Now I was thinking about how we move towards an Internet that is
25:31
supporting more Gen. AI use cases.
25:35
What does that look like from the networking perspective?
25:38
From the gateway perspective, I know you all are doing a ton of
25:41
work with the Envoy AI Gateway. I imagine it's not just all hey,
25:47
let's throw an awesome gateway into the mix and that solves all
25:52
our problems, right? Yeah, Yeah, it doesn't because
25:56
even the existing found it. Like if you look at what how
26:00
Envoy Proxy itself operated, we had we had to make enhancements
26:05
to Envoy Proxy itself. And then we started the Envoy
26:08
Air Gateway project to further expand on the on what proxy
26:14
capabilities and control planes. So explain a little bit what
26:17
that is in a moment. But if we take one step back and
26:20
like look at what happens in the world of gateways, I think it's
26:23
super fascinating was with the rise of Jenny I, there was a lot
26:27
of Python gateways, Python ridden gateways that came out
26:32
because it seems natural, right? Like, oh, we wrote all of these
26:36
other cool stuff in Python And a lot of people who came from the
26:40
machine learning and Gen. AI world are incredibly
26:43
comfortable with Python. And if you want to do cool stuff
26:47
like automatic model selection on an incoming request, you kind
26:52
of got to do that in Python because now you're back into the
26:55
machine learning and Gen. AI space.
26:57
So it becomes this weird thing like how do we do the smart
27:01
stuff we want to do, but also don't run into the single waiter
27:06
per table problem. So because of Python gateways,
27:12
fundamentally runs into problem and it's not about how smart the
27:17
people are writing these Python gateways, not their fault.
27:21
The problem comes back into like the foundations of the Python
27:25
language where there is only effectively there's an
27:30
interpreter because Python is an interpreted language, it is not
27:33
a compiled language. So therefore you have this
27:36
process that is interpreting Python And that is threaded.
27:41
So every coming to threads now we come into the single waiter
27:45
the table problem. And therefore even if you try
27:50
to, you can start trying to be clever and simulate the what is
27:56
referred to as event driven architecture.
28:00
Don't confuse that with Kafka. But when you look at the the
28:04
event driven architecture, a proxy is when you can have
28:07
multiple, you can have one waiter serving multiple tables
28:11
and just putting things into the system.
28:13
So it's like this little event notification within the
28:16
restaurant within the proxy. So you can simulate a lot of
28:21
that with Python, but you are going to eventually run up into
28:25
the issues and the constraints, well, we can call them issues.
28:28
The issues aren't Python itself isn't the issue.
28:31
It's just you will run into issues because of the
28:34
constraints of the Python language if you want to do large
28:38
scale of handling loads and loads of connections.
28:43
So that's what we run into problems and how do we solve
28:46
this problem? How can we like combine the
28:49
multi, you know, the restaurants where waiters can serve multiple
28:54
tables with the smart stuff that Python can do.
28:58
Then we start. That's how we end up leaning
29:00
into Envoy Proxy, which fundamentally can handle, which
29:05
is very much the the restaurants where waiters can serve many
29:08
tables, right. That is what on where proxy
29:10
really is great at and has this event driven architecture, Envoy
29:16
Proxy with Envoy Gateway, Envoy AI Gateway allows us to create
29:21
an extension mechanism where we can bring in the cool logic like
29:26
automatic model selection that's written in Python And still have
29:31
the do you know one waiter serving multiple tables Proxy.
29:38
And then being able to only when we need to go and get that
29:42
special order from the Python extension to do smart stuff only
29:47
if we need to and only if we want to.
29:50
So that is where getting the network benefits as well as
29:54
being able to bring some of the smart stuff in.
29:56
And that is what makes it really interesting on how we can bring
30:01
these worlds together. And, but we are, we are
30:06
definitely going to see I, I still believe we're going to
30:09
start seeing challenges with some of the, the time that the,
30:15
the response times like time to 1st bite and the complete
30:19
streaming. I just imagine that that's going
30:22
to continue to grow as they need and we're going to have to start
30:28
looking further into the internals of our proxies.
30:32
I I'd be surprised if we're not going to continue growing in
30:35
that space. You know what it reminds me a
30:38
lot of is the folks who build Streamlit apps.
30:44
It's like, yeah, I built this Streamlit app because I just
30:47
code in Python And I like Python.
30:50
It's you're like Streamline is probably not the best for your
30:53
front end, but it gets you somewhere.
30:56
And then when you want to take it to really productionizing it
30:59
and making it fancy and doing all that cool front end stuff,
31:03
then you can go to React or whatever.
31:05
You bust out your versal jobs or your next JS and you actually
31:11
make something that is nice for the end user or nice on the eye
31:18
and has a bit of design to it. But sometimes you just are like,
31:24
yeah, all right, cool. Well Streamlight gets me this
31:27
MVP and that's what I want. And in a way, it's almost like
31:33
what the parallel here is that there's been proxies made with
31:39
Python. They get you a certain way down
31:43
the road and you can validate ideas and you can see if it
31:46
works. And then at a certain point,
31:49
you're probably going to say to yourself, OK, now we want to
31:54
productionize this, and Python is not the best way to do that.
31:59
Yeah, like I think it's interesting because if you have
32:02
like a small user base just building something internally
32:05
for your company, you can probably be fine with your
32:08
Python gateway. Like you got a few 100 users
32:12
just inside your own company, you're probably fine but it.
32:17
But if you have even internally, if you have a lot of concurrent
32:22
connections and you start needing to deal with this, you
32:25
can't have one waiter per table that is standing there waiting.
32:29
So you got to have to figure out how you can make more use of all
32:33
of the connections. So it's sort of interesting as
32:36
we've talked about what happened at the beginning of the 2000s
32:39
with the 10,000 concurrent connection problem.
32:42
The Python gateways are effectively as we are bringing
32:46
more Jenna, Jenna AI features to the mass market, ultimately runs
32:51
into that problem. And the good news is the answer
32:54
of the concurrent connection problem has already been solved.
32:58
So then it says how can we bake these things together to get the
33:02
cool features? But at least the good news is
33:05
the concurrent connection problem has been solved.
33:10
So there's, there's almost like 2 levels that you're talking
33:12
about here, which is the trafficking and the network
33:16
trafficking. But also then understanding the
33:20
request and being able to leverage some of this cool stuff
33:25
like, oh, well, this request probably doesn't need the
33:27
biggest model. So let's route it to a smaller
33:30
model. Or maybe it's open source and
33:32
we're running it on our own. Maybe it's just the smallest
33:36
open AI model, whatever it may be.
33:39
And so the levels that you have to be playing at are different,
33:44
right? Or at least in my mind I
33:45
separate the whole networking level with the actual LLM call
33:51
and what is happening in that API.
33:54
Yeah. And then, but then you have the
33:56
smart stuff that has to happen before you send it to the LLM
33:59
service. And that's where the exciting
34:00
part like combining things like Envoy proxy with a Python
34:06
service, a filter, a Python filter to be able to do smart
34:10
stuff to make those rafting decisions.
34:13
And you don't think that that's just kicking the can down the
34:17
road? Like, you're still going to run
34:19
into problems if you're passing it off to Python at any step
34:23
along the way. Good question.
34:25
So when you are, we don't have the connection problem in the
34:28
same way at that point, because now we can have dedicated by the
34:33
Python services that can run and we can scale them independently
34:38
of the proxy. So now we are not dealing with
34:41
client to proxy or proxy to upstream connections.
34:45
Now we have the proxy that is making a call out to an another
34:50
service. So like a Python service and
34:53
that service, you know how we talked about boxes in Kubernetes
34:55
earlier, that little Python service box, we can scale it up
34:59
and scale it down based on demand.
35:02
So that thing now becomes horizontally and elastically
35:05
scalable on its own and it's not going to have any.
35:09
They won't be any noisy neighbor problems because they're not
35:11
going to block each other. And we only run into the waiter
35:16
problem when we have a limited set of waiters in our restaurant
35:22
that are associated with one request response life cycle.
35:27
Because now we just go to this little Python service when it's
35:30
done, it's out of the picture. And also, we weren't blocking
35:34
other requests because of how the Envoy architecture works.
35:38
We weren't blocking the other requests while we weren't
35:40
consulting this Python process about what to do.
35:43
I understand what what's going on then and.
35:48
Like we we call them and ask have little Python service.
35:50
What do you think about this request?
35:52
I'm going to hang up now, and when you have an answer, I'll
35:55
pick up the phone and we'll continue this journey.
35:58
Yeah. And that bottleneck only is
36:02
happening potentially at the proxy area.
36:06
It's. Not yeah, that's a connection
36:08
problem. Once it goes through the proxy,
36:11
then it's like free range and there's all kinds of stuff that
36:15
you can be dipping your fingers into it.
36:18
Says that they request life cycle inside of on.
36:20
What is the important part and how you can include external
36:23
filters so that you can it's basically rent driven.
36:27
Let's talk about like you, you basically put stuff into the
36:30
central system of on when and then knows what to do.
36:34
So it's I hope this is an oversimplified explanation by
36:39
the way. So people who are really in the
36:41
depths of onward proxy could probably be like, oh, Erica,
36:44
that is not entirely exactly in detail true.
36:48
But if you want the like 10,000 feet view, I hope that is
36:52
enough. And if you really want to dive
36:53
into it like you. Can you can.
36:55
Spend a long time learning about the internals of this, but the
37:00
good news is that it's very clever and how it handles
37:03
resources and and manages connections.
37:07
So that is really cool. But then Envoy proxy is really
37:11
hard to configure. So I'm going to go on a slight
37:13
tangent, but an important tangent and like why Envoy AI
37:16
gateway is really interesting because Envoy AI gateway does
37:22
two key things for when it comes to helping you leverage Envoy
37:26
proxy to handle traffic and and you have two every time you have
37:31
a proxy, you have a two side problem.
37:34
You don't just have, you need to configure this proxy and you
37:38
want to be able to configure it in a way.
37:40
So even if it scales up and down.
37:42
So you have like many proxies, SO1 proxy, you have a control
37:46
plane that can effectively and resource efficiently configure
37:50
this fleet of proxies. So on where AI Gateway brings
37:54
you this control plane that is extending onward gateway control
37:58
plane. And then it's like really
38:01
interesting, I really can't have time for spending on that, but
38:04
how that control plane is being very efficient in how it
38:07
configures Unword Proxy and helps you propagate
38:12
configurations across all the proxies that are run in.
38:15
And then the other part that we've added into Unword AI
38:19
Gateway is an external process that helps with specific Gen.
38:25
AI challenges like transforming requests.
38:28
So one of the things we've done is to have a unified API so that
38:35
if I'm an application developer, I don't have to learn all these
38:37
different interfaces to connect to different providers.
38:40
Smart. Because we don't have to put
38:44
that cognitive load of people who want to build cool apps, let
38:46
them build cool apps, and then we can worry about their pipes.
38:51
Yeah. And do you need one to be able
38:54
to leverage the other? I guess do you need Envoy to be
38:57
able to leverage Envoy AI Gateway?
39:00
So when you install Envoy AI gateway, it actually installs
39:04
Envoy gateway which installed which gives you Envoy proxy and
39:07
the Envoy gateway control plane and when.
39:09
So when you go through installation steps, you actually
39:12
first install Envoy gateway and it would run on a Kubernetes
39:17
cluster. I'll call this like a gateway
39:20
cluster, by the way, for reference, if you ever look at
39:23
any of my diagrams in blog posts.
39:24
But and then you've deployed that, then you there's a helm
39:29
chart and you just like install on way AI gateway and that
39:32
deploys an external process and an extension of the control
39:34
plane. So it expands on the
39:38
functionality of on way gateway and on way proxy.
39:41
So you don't really have to know that you are deploying all of
39:44
those things, but that is what happens if you deploy it.
39:49
It's all part of the Envoys CNCF project.
39:52
It's not like a set, it's part of that fat ecosystem.
39:56
So it's it's not like a separate entity per SE like the it's part
40:03
of the Envoys CNCF ecosystem. Have you noticed a lot of AI or
40:10
ML engineers starting to come into the Envoy AI gateway
40:16
project and like submit PRS? Do you look at that type of
40:20
stuff? Because I wonder how much of the
40:25
tasks or of this scope falls onto ML engineers, AI engineers,
40:32
or if it if those folks just throw it over to the Sr.
40:37
ES and the Dev OPS folks. It's a good question.
40:40
What? Where we really need the people
40:44
who have good understanding of machine learning and Gen.
40:49
AI. It really becomes, I mean look
40:52
at these intelligent Python, maybe Python based extensions.
40:57
If there are people out there who have ideas on how to do for
41:00
example, semantic routing to be able to decide that model, not
41:05
doing it in Python, please let me know.
41:07
Very interesting. This is outside of my area of
41:11
expertise, so if there's someone who can write it in Rust, please
41:16
let me know. That'll be amazing.
41:19
But the yes, we are seeing people who come in who are like,
41:22
hey, let me show you this Python extension and how it fits into
41:28
Envoy, AI Gateway and the smart stuff that is.
41:34
Yeah, definitely outside of my expertise.
41:36
I find it really fascinating and interesting.
41:39
So if people have that expertise, I would love people
41:43
to come and help build those extensions into On Way AI
41:48
Gateway and bring those features to the community.
41:51
Because fundamentally what when we look at the on way Gateway
41:55
initiative within the on way project, the On Way AI Gateway
41:58
initiative is really about what we're facing right now and how
42:03
this traffic is changing and shaping up.
42:06
It is not a single company's problem.
42:08
It is not a single users problem.
42:11
We are all running into this. So coming together and solving
42:15
these things together in open source and maintaining it
42:19
together, I think it's really exciting and seeing both vendors
42:25
and users coming into the space and collaborating.
42:29
And yeah, really sharing knowledge, as you said, like
42:32
sharing the networking knowledge with sharing the knowledge of,
42:36
well, what does what LLM functionality do we have out
42:40
there? How can we actually bring some
42:42
functionality into the gateway to make it even smarter?
42:45
And people who really understand the challenges of how they are
42:48
running other than workloads, I think is really, really
42:52
interesting combination. And I've learned so much over
42:55
the last by six 7-8 months say from collaborating with people.
43:01
So it's been really cool and exciting.
43:04
So I will say that what you are talking about here, it has been
43:09
100% validated by the community. And even the way that I, I know
43:15
it's been validated is when we do our AI in production surveys
43:20
and we try and do them every time we do like a big conference
43:22
or almost once 1/4. And we ask people what's going
43:27
on in your world and what are the biggest challenges?
43:29
What are things that you're grappling with right now?
43:32
A lot of people have written back and said some of the
43:36
hardest things are that there's this new way that we are working
43:47
with software and working with models and almost like what you
43:51
were talking about where these models are so big.
43:55
So it makes things much different to handle and it makes
44:01
everything a little bit more complex.
44:04
And then on top of that, you don't really have anywhere that
44:10
you can turn to that has definitive design patterns and
44:15
folks who have figured it out and are sharing that information
44:19
with the greater developer community.
44:23
And so I, I find it absolutely incredible that a, you're
44:29
working on it in the open. It's great that like the open
44:33
source project is trying to do that, but then B you've thought
44:38
about this idea of OK the models.
44:41
Like traffic to models, the traffic is so different.
44:46
They're so different. They bring so many other ways of
44:52
having to deal with software engineering, not just the the
44:57
fast and light way that we've been looking at it and trying to
45:00
get to. But now there's a little bit
45:02
heavier 1. And now we get these time out
45:05
problems or the model doesn't fit and there's like payload
45:10
constraints, all of these like these constraints or these
45:14
problems that people are running into because you try to bring
45:22
this new paradigm onto the old rails and you see where there's
45:28
a few cracks. Yeah, like it's I, so I in my,
45:33
my actual work, in my day-to-day work, like I, I am a community
45:39
advocate, which actually means a lot of what I do is
45:43
understanding what's happening out in the technology community.
45:47
And so I work at a company called Tetrite and we are very
45:52
invested in the Envoy project, for example, with having
45:58
engineers and people like myself being out there building in open
46:02
source. But as a community advocate, a
46:05
lot of what it is for me is like talking to people, listening to
46:08
what's really going on and the challenges they're running into
46:12
and advocating for the community when we are looking at how we
46:17
are building things going forward.
46:19
So I think what could be maybe misinterpreted being a community
46:22
advocate is I'm advocating for the solutions to the community.
46:25
No, no, no, I am advocating for the community for so that the
46:30
solutions that are being built are addressing needs in the
46:34
community and maybe sometimes even needs the community haven't
46:36
realized they've got yet. Maybe they're running into
46:39
interesting problems. Have I heard people say they are
46:41
running into challenges with their Python gateways?
46:43
Like why do you? Why does it seem like the
46:46
gateway itself is adding latency?
46:48
Well, let's talk about event driven proxies.
46:51
But this is where it's interesting.
46:54
How can you advocate and understand the challenges of the
46:57
community so that the solutions are being built meet those
47:00
needs? So yeah, I found that really
47:03
interesting for me specifically. So I find it really nice to hear
47:07
what problems people are running into.
47:08
So it's really exciting to hear like what you're seeing as well
47:11
in your community and advocating for people's real needs out in
47:17
the real world to say we don't build science projects.
47:21
And I think that is also what's so exciting.
47:23
You know, you said you had Alexa on here earlier, right?
47:25
Like collaborating with Alexa and the team in open source and,
47:31
and really bouncing ideas together and really getting to
47:36
real needs. I think that's really fun and
47:38
nice. And you know, we don't work at
47:40
the same companies, but we get to collaborate and, and drive
47:45
solutions together. Oh, it's really fun.
47:48
The thing that I'm wondering about is, do you feel like you
47:53
have to be technical to be that type of a community advocate?
48:01
That's a good question. I am very technical at the start
48:04
of my I started coding when I was 12 and I been in the gateway
48:11
space for many years. I started being in the platform
48:15
engineering and gateway space back in 20/15/16 as we started
48:23
2015 when we started breaking down those monoliths into micro
48:27
services and the need of gateways became apparent.
48:30
So I've been in that space, yeah, since 2015.
48:34
And I was in the Fintech space. And I need to be honest, this
48:40
Gen. AI and this type of traffic
48:44
patterns we're seeing now, I feel really validated because
48:48
definitely between 2015 and about at least 2023, I was in a
48:54
situation where people said those financial analysis APIs
48:57
that I was dealing with there was incredibly open-ended.
49:01
And you can be like, oh, I have this portfolio and I want to
49:05
analyse it. But there's a big difference
49:07
between analysing a portfolio with 10 U.S. equity holdings in
49:12
it versus a multi asset multinational portfolio with
49:16
10,000 holdings in it. Analysing those, even if it's
49:20
the same API endpoint you're hitting, I hope you can
49:24
understand that one of them is going to be very fast and easy
49:27
to respond to and 1 is going to take a lot of time processing
49:30
and both the input potentially and the output, especially the
49:34
output, they're talking very big outputs.
49:37
That was starting to hit the limits of what API gateways.
49:42
You know, how we talked about the, the body, the response body
49:44
limitations, We started to run into those problems because
49:48
there were so big responses. They were hitting the 10
49:51
megabyte limits of many, many gateways out there.
49:54
So I was like, for years I felt that people were telling me,
50:00
Erica, you were doing your AP is wrong.
50:02
Clearly, those financial analysis AP is, is something
50:07
wrong with your design. That's the problem, not the
50:10
gateway. And to be honest, I truly
50:13
believe that we must be doing something wrong, right?
50:16
That we have these APIs that were so unpredictable in the
50:23
time to respond and the size of the response.
50:26
But then Jenny, I and LLM APIs came around and I'm like, wait a
50:31
minute, I've seen this before an API you could put stuff in and
50:35
it'd be very different what comes out and it could be slow,
50:38
it could be big. I'm like, and now everyone seems
50:41
to be on board with like the this is a real challenge.
50:45
But actually a lot of the things we are handling dealing with
50:48
now, it would gateways in the world of LLMS is actually the
50:53
problems we are solving benefits these financial analysis APIs as
50:57
well. So these changes in how we're
51:01
looking at both the limiting usage like in LLM world, we're
51:05
using the quote token quotas. So maybe you're allowed to use
51:10
10,000 tokens in the space of whatever time frame you like.
51:16
But if you want to say a day or a week or a month and you can
51:20
then start putting these. But those are not number of
51:23
requests. You can make just a couple of
51:26
requests and hit your quota or you can make loads of small ones
51:31
and then you'll hit your quota. So that is different.
51:34
So we actually had to change how we did rate limiting like normal
51:39
rate limiting and usage limiting and Envoy proxy to be able to
51:44
allow for this more dynamic way of measuring usage.
51:48
So you don't just measure, measure usage based on number of
51:51
requests, but you can measure it on another data point.
51:55
So in this case in LLM world, it would be tokens like word
51:59
tokens. So that's fascinating to me.
52:02
And even this our observability in how we measure how fast we
52:08
are responding. So if you think about an LLM and
52:11
you're streaming a response and you're streaming tokens back,
52:14
what you're actually interested in from a performance point of
52:18
view is response tokens per second because then you know,
52:23
it's, it's, it's working like stuff's happening.
52:26
Responses are streaming through. If you then correlate that to
52:30
some of the challenges we saw in financial analysis, AP is we
52:33
didn't have word tokens, but we had, we were very concerned
52:37
about time to 1st bite. So the first bite, we started
52:40
streaming in the response. So instead of looking at the
52:44
time it takes for the entire request to process in the
52:47
financial analysis API, imagine we can monitor bytes per second
52:53
because that will tell us stuff is moving.
52:55
It's not standing still. So similar to how we are
52:58
interested in response tokens per second when we're streaming
53:01
LLM responses. If we take the parallel and look
53:04
at the financial analysis, AP is just shifting the way we're
53:08
thinking about performance and observability is also important
53:12
when we look at this infrastructure to understand the
53:14
health and of our system. Yeah, you really are.
53:18
I like that idea of, hey, the performance metrics have to
53:26
almost you, you want to try and get more creative with them.
53:30
And it reminds me of a conversation we had with Krishna
53:34
who works at Qualcomm. And he was talking about how
53:39
sometimes when you're putting AI onto edge devices, you want to
53:46
optimize for battery life. And ways that you can do that
53:50
are by streaming less tokens, or making sure that the tokens
53:54
aren't streaming at 300 tokens per second, because people can't
53:58
really read 300 tokens per second.
54:00
So why stream them that fast if people aren't going to read it
54:05
that fast? And if it can.
54:07
Mean less battery consumption. Then you stream at 20 tokens per
54:12
second or whatever it may be, whatever that happy medium is.
54:15
Yeah, absolutely. That's a really good
54:18
perspective. So yeah, like they all add up.
54:20
They were like, I think it's fascinating in this space that
54:23
how so many things are are related.
54:27
It's like I normally don't think about battery times because I
54:30
work in places where, you know, power is plugged in, it's off
54:35
the mains, right? And there's no charge up
54:38
battery. We don't have battery driven
54:39
service, right? But yeah, I do think though
54:43
they're coming back to question, do you have to be technical to
54:46
be able to be advocating for a community in the space?
54:49
Yes, I do think it would been very hard for me to not have my
54:54
technical software engineering and software engineering manager
54:58
leadership background. Interesting.
55:00
I think that that would been very hard because I'm in such a
55:04
technical space, right? It's so deeply technical and I
55:11
believe it would be very hard to not have experience in this
55:15
space because sometimes I can hear problems people are
55:19
describing like, oh, this Python gateway is somehow very
55:25
unpredictable in the latency the gateway itself adds to my
55:29
requests. Cool.
55:31
I hear that they necessarily can't express that the challenge
55:37
they're running into is that they're, they have ran out of
55:41
waiters in the restaurant. So their, their requests are
55:44
waiting outside for an available table and waiter, right.
55:48
And that is why sometimes the request takes 100 milliseconds
55:52
and sometimes the request takes 700 milliseconds.
55:57
They can explain the problem they're observing, but you have
55:59
to understand, OK, are using a Python gateway.
56:03
What are the limitations of the Python language that may be the
56:07
actual foundational cause of the problem that you are observing.
56:10
So be able to advocate for the user in that case I need to
56:15
understand. The.
56:17
The course of the pain to then advocate for solutions to be
56:25
built in the industry to address their need.
56:27
Hopefully that makes sense. I think it'd be very hard if you
56:29
don't. Often you're deep.
56:31
Often you are dealing with users observing challenges but aren't
56:35
in a position to see the opportunities as well and and
56:41
the causes of the challenges are observing.
56:44
It's almost like you're part product manager, part like
56:50
customer success or support in a way and in part community or
57:00
externally facing. I don't know how you how you
57:03
would qualify that. And so it's interesting to think
57:08
about how these deep questions, deep technical questions or
57:13
discussions are coming across your desk and you're seeing them
57:17
and then you recognize, wow, there's a pattern here.
57:21
And maybe that means that we should try to build it into the
57:26
product so that we can help these users with this pattern
57:30
because I'm, I'm keep seeing it come up.
57:33
So let's get ahead of it. Let's help create a feature or
57:36
whatever to help the users so that they don't have to suffer
57:40
from this problem anymore. Yeah.
57:43
And I think on that like being ahead of a problem because I
57:46
used to be in platform engineering in like in like
57:49
internal building API platform service platforms and such in my
57:54
old roles. And often the challenge always
58:00
is that you need to try and build things before people are
58:04
really in pain because when people are in true pain, you are
58:08
too late. So being able to see that
58:13
someone see the symptoms of something that's going to become
58:18
really painful later, early enough that you can address
58:22
them. So you have the cure when people
58:25
come around, but what has been really great is that
58:30
fortunately, unfortunately, however you want to see it, some
58:33
people had already started running into these problems.
58:37
But if you can work to solve those problems for a small set
58:41
of people early and then being able to showcase that and then
58:46
help the sort of 2nd wave of adopters not have to experience
58:52
the growing pains that the first sort of pioneers had to
58:56
experience. But yeah, generally it's like
59:00
sometimes hard to even explain to the the wider community the
59:06
purpose of doing something because they might be like this
59:09
problem you're talking about. I don't have it.
59:13
And you're like, that's great for you, That's amazing, that's
59:18
awesome. I'm happy you don't have it.
59:20
Yeah, talk to me in from it. So that's the other tricky part
59:26
of advocating for a community, because sometimes you have to
59:30
advocate for a commute for people in the community that,
59:33
but they don't feel like you need to advocate for them yet.
59:36
They're like, well, and also they, they don't need to know
59:41
what I do, you know, when I talk to people exactly.
59:46
But I do think it's an important read to see where the where the
59:50
community and the industries at, how they are.
59:55
Where are they now? How long do we think it is until
59:59
the pains will start being felt? One thing that's got to be hard
1:00:04
about your job is the shifting sands that you're building on.
1:00:08
And I say that because a friend of mine, Flores, was saying how
1:00:12
he spent in the beginning of the LLM boom.
1:00:16
He did so many hacky things to allow for greater context
1:00:24
windows on their requests. And he spent so much time on
1:00:29
that problem. And then next thing you know,
1:00:32
context windows just got really big.
1:00:34
And so he he was sitting at his desk like, damn, all that time I
1:00:39
put into this, I could have just waited three months or six
1:00:43
months and it would have happened on its own.
1:00:46
And so I think about in the world that you're living in, as
1:00:51
you're thinking through some of these problems, how do you
1:00:56
ensure, if at all, that you're working on the right problem and
1:01:00
you're not going to get into a situation where something that
1:01:06
you've worked on for the past six months now doesn't matter
1:01:11
because of another piece of the puzzle changing and totally
1:01:16
making it obsolete in a way? I think for me, when I look at
1:01:22
how the when I started looking at features to aid and enable
1:01:29
Gen. AI traffic, I got to be honest,
1:01:32
the first time someone said like AI Gateway to me, I was like
1:01:36
really like, it's just network traffic.
1:01:41
I was like really like, what are you talking about?
1:01:44
It's just network traffic. We've been dealing with network
1:01:46
traffic for a long time. Like what's up with you?
1:01:49
Like this is just a someone you know stick in an AI label on a
1:01:55
network component because. They need to raise some funds.
1:01:58
Yeah. And I was just like, this is
1:02:00
ridiculous. It's just, and I even was like,
1:02:03
I even had that moment where I looked at their problems of
1:02:06
these, like the big payloads in requests and responses and the
1:02:12
unpredictable response times and the high compute utilization.
1:02:16
I was like, yeah, I had that problem for years and I like
1:02:20
looked at. I was like, yeah, that's not a
1:02:22
Jenny I problem. I've had a problem in fintech
1:02:25
for a long time. Congratulations that you joined
1:02:27
the party. That was my initial reaction.
1:02:29
I was like, OK, fun for you, welcome to the club with these
1:02:35
problems. But it's almost that what made
1:02:38
me really excited when I saw that the problem that was being
1:02:43
run into in the Jenny I space I had experienced for over five
1:02:48
years. And I was like, well, if I've
1:02:51
experienced it for five years and everyone told me that the
1:02:56
problem I had was myself inflicted problem and then now
1:03:02
we had this problem. The amount of people who was
1:03:06
experiencing the problem had just grown.
1:03:09
That made me feel like, OK, these features, even though we
1:03:14
can drive the innovation at this point because of the explosion
1:03:18
of Gen. AI, these features don't just
1:03:21
benefit Gen. AI, they benefit beyond Gen.
1:03:24
AI. And therefore I do not currently
1:03:29
fit. This feels like fundamental
1:03:31
problems in how we handle network traffic, connections and
1:03:36
processing of request and response payloads.
1:03:39
And I so therefore in this particular space, I think we're
1:03:45
just actually late. We should have solved this a few
1:03:48
years ago. Yeah, I guess a lot of the
1:03:51
energy too, and the advancements and the money are going into the
1:03:57
model layers and application layers, but not necessarily the
1:04:04
nuts and bolts and the piping and tubing layers.
1:04:07
It's. Almost like this is a bit of an
1:04:11
afterthought when you do hit scale and when you do realize
1:04:15
that, Oh yeah, we want to make this production grade, how can
1:04:24
we do that? And you start to think about
1:04:27
that. Hopefully you don't think about
1:04:29
it once you already have the product out and it's getting
1:04:32
requests. Hopefully you're thinking about
1:04:34
it before then. Yeah, like when people come out
1:04:36
of their POC workshops. Yeah.
1:04:40
How do they get it industrialized?
1:04:43
Exactly. It's A and I do think that
1:04:46
people should start in their POC workshops, right?
1:04:49
They should be playing around, seeing what's possible.
1:04:52
Listen, I don't think people should scale their systems
1:04:56
before they need to scale them. That is like what if you have a
1:05:00
bad idea like leave it in the in the POC workshop, don't take it
1:05:05
out can stay there on a shelf. You can be like I built that
1:05:08
once that was cute and no one needed it and that's OK.
1:05:13
So it's don't scale too early, but there now at least I feel
1:05:20
that there's enough people out there that need the scale.
1:05:23
And I'm really excited about how we don't just bring these
1:05:26
features into Envoy AI Gateway, but really this notion of having
1:05:35
usage limits that isn't just number requests that features
1:05:39
available in Envoy Gateway. Now you don't need to use Envoy
1:05:44
AI Gateway to leverage that. That is now part of Envoy
1:05:48
Gateway as of the 1.3 release. So if you have a financial
1:05:53
analysis API, you can actually now you don't have to go and
1:05:56
install an Envoy AI Gateway to have more intelligent.
1:06:02
Intelligent is the wrong thing to call it.
1:06:03
Dynamic is the right thing to call it more dynamic way of
1:06:07
enforcing usage limits beyond number of requests.