Latest / Agentic Conversations (formally mlops.community) / Why Your AI Bill Will Double Before It Gets Better

Transcript

Why Your AI Bill Will Double Before It Gets Better · Agentic Conversations (formally mlops.community)

0:00
Something we were talking about a little bit earlier around Open
0:03
the Eyes, release of guaranteed capacity.
0:06
That just happened, right? I find it really interesting
0:10
because it's not advertised as a discount mechanism.
0:12
It's, it's advertised as a, you know, guaranteed capacity, you
0:16
know, and, and we're just. Saying like, if you need this,
0:19
you're always going to have it, so you're never going to be
0:22
without resources. We're not going to go down on
0:24
you. Exactly.
0:26
But from our standpoint, we haven't really seen capacity
0:28
issues. So it's almost like they're
0:30
they're solving a problem that doesn't exist yet.
0:32
That has to be very hard for you to look at and it probably makes
0:36
your job like you have a fixed price you can't go over.
0:39
I mean, there's lots of ways to address that.
0:41
At the same time you, you see the subscription model going
0:44
away and you know, usage base or a combination of that, those
0:47
things happening. And that's the reason why
0:49
because it's fixed, prices just don't work anymore.
1:04
All right. I'm here with Josh working at
1:06
Superhuman leading the fin OPS. I want to hear about your
1:10
journey into tokenomics. It's been about a three-year
1:14
journey overall, but but yeah, prior to Superhuman, and I'm at
1:19
the time when I joined them, they were Grammarly.
1:22
I was at AWS and, you know, had no exposure to AI when I was
1:26
really there outside of just managing some of the billing
1:28
aspects of it. Come to Grammar Lane, you know,
1:30
we have some hosted models that I was helping, you know,
1:33
distribute the cost and, and report on.
1:36
And then on early 2023, we started using this new thing
1:39
called Azure Open AI. So I started kind of poking
1:43
around. I'm like, I think this is going
1:44
to be kind of big. So I want to understand like if
1:47
there's a area where I can help. So I started kind of poking
1:51
around, asking questions and found all this is like a fixed
1:54
capacity that we're, you know, we want to make sure we're
1:57
utilizing enough of it. And, and, you know, also
2:00
understanding, oh, if we don't have enough is this will create
2:04
an outage for some of our products.
2:05
So we got it. So introduced some new concepts
2:08
like I've never been involved with making decisions that could
2:11
actually cause an outage on one of our products, right.
2:14
That wasn't like a thin up thing, but it kind of fell into
2:16
my scope. So, but as I started doing that,
2:18
I realized after going to some Fin OPS Foundation events that I
2:22
was one of the only ones doing that type of work back then.
2:26
So I certainly encourage anyone to, you know, even definitely in
2:31
this day and time with AI, just most practitioners, if you poke
2:34
around hard enough, you'll find a area where you can drop a lot
2:37
of impact for sure and help. But it's funny how you started
2:41
poking around and then you realized, wait a minute,
2:43
resource management is kind of seems like my job in this whole
2:47
new field. Yeah, resource management, a
2:49
very, very expensive resource, yeah.
2:51
Especially in those days, yeah, maybe talk to me about the
2:53
trends and how you saw the trends dropping and then now
2:57
reversing of the trend of the cost and.
3:00
You look at me a year or two ago, I'm like, oh, costs are
3:03
just falling off a Cliff and now it's just going to be the the
3:06
vendors fighting for market share and it's going to be raced
3:08
to the bottom. So yeah, in the first probably
3:12
year or two sold over 80% drop in cost per token.
3:17
And I think some of that was the vendors getting more mature and
3:23
the infrastructure behind the scenes and also them trying to
3:27
gain grab market share. And then over the, I'll say
3:31
starting in, you know, really last year, but over that
3:35
following year, from 2024 to 2025, the trend started to
3:38
change and, and it whether it started where it was not, you
3:42
know, maybe they were about the same to now with, you know, for
3:46
example, GBT 54IS like double the cost of five one.
3:51
I think Opus just came out today and it's like double the cost.
3:54
On top of that, you've got other skews like web search and and
3:59
reasoning and of these other aspects that are just
4:02
compounding the cost. So now we're the trends are is
4:06
not only is volume like a hockey stick right now, but costs are
4:10
rising too. No matter like what is pizza
4:12
elsewhere, rates are going up. For sure.
4:15
And how did the way that you were doing your job back when
4:20
you thought in the next 6 months, is it just trending to 0
4:25
versus now where you're kind of looking at it and you're saying
4:28
it may go up, but it definitely is not going down?
4:33
It's going to just, it's, it's going to exponentially go up.
4:38
It's that's your prediction you're calling in here.
4:41
It's only going up. It's only going volume is not
4:43
going to go down. If it's not going up then as a
4:46
business, then you know that's a bad for us, right.
4:49
So you know, and ideally we want more and more users using the
4:53
products, but we just want it to scale efficiently.
4:55
But the, you know, I would say with external LLM that's, that's
4:59
it's becoming in my opinion of larger and larger financial risk
5:03
because of the the rate trends that we're seeing here.
5:06
So volume and we know is going up.
5:08
We can't have pricing also doubling just about every time
5:11
too. And then that's where I'm
5:12
looking at exponential increases.
5:15
So fascinating. So that's making you nervous
5:17
looking at how it's been performing and recognizing that
5:21
the volumes going to continue to grow even if it stays at this
5:25
price that we're at right now. That's scary.
5:28
But there's a non 0 probability that we're looking at the prices
5:33
also increasing. And so that forces you now to
5:38
look at different options. How does their outlook change?
5:42
The bulk of of Grammarly's suggestions are based on open
5:47
source models. And that is where I really want
5:50
us to go just from a cost aspect, you know, but we're also
5:54
an AI company, so we have a special set of teams and skills
5:57
that can can make that happen. Most companies don't have that.
6:01
But I would say right now, I mean, we, we have two sides of
6:04
this AI coin. We have our production cost
6:06
which far outweighs our AI coding internal cost.
6:10
I would say right now most companies are dealing with the
6:12
AI coding side of it and that's probably what they should be
6:14
dealing with. But for us, it's like both
6:16
sides, we've got them really managed to make sure and that
6:19
that both are efficient and measuring the value in some way.
6:23
Yeah, because the product is powered by LMS, yes.
6:30
Again, mostly open source, but more and more as you want to
6:35
have very high velocity of releasing product and such.
6:38
I mean those Frontier models are great for that.
6:40
So that's where the growth is happening on the external alum
6:44
side. And you also mentioned to me
6:46
before we hit record on the hidden costs around the price
6:51
increases just with the models. Can you talk a bit more about?
6:53
That Oh yeah. So when you are looking at say,
6:56
you know, maybe using GBT 51 for whatever reason and, and five
7:01
four comes out and let's say you have data residency, U.S. data
7:04
residency requirements. So now you, you see the rates
7:07
and you start using it and you're like, you know what, we
7:09
do want to use this and we see that as two times the cost, but
7:11
we see the value here. But then you don't notice the
7:15
fact that there's a 10% data residency charge on top of that
7:19
that didn't exist with the five one model.
7:21
And that's happening with the four Opus 4, I think 46 or 47
7:25
and later. And basically Anthropic and Open
7:29
AI are both adding this premium to US based models.
7:33
You're already seeing this before this with the
7:35
hyperscalers, AWS and GCP and such.
7:38
But now it's that's another premium you're paying a lot of,
7:41
I believe a lot of companies are kind of overlooking.
7:44
Yeah, you don't realize that until you later need it.
7:47
And then you go, oh, we're getting charged for that.
7:50
Yeah. So whenever you're doing your
7:51
estimations and comparing these models, you just make sure you
7:53
look at the fine print about the this data residency piece
7:56
because at 10% adds up. Yeah.
7:59
And Speaking of estimations, I know that you've built a cool
8:02
cost calculator. You vibed it.
8:05
I vibed it. I mean it.
8:06
So I'm not an engineer. And I was telling him before
8:10
that, you know, before I started a grammar, I had really no
8:14
experience with AI at all. And which is my kind of poking
8:18
around in curiosity that kind of put me where I am today.
8:20
So I was took the initiative to say, OK, I'm going to space out
8:26
some time in my days to spend time building some agents to
8:30
help when I was practice. And I really, I knew I wanted to
8:32
build some type of calculator and didn't really know what it
8:36
was going to look like. And I built like a HTML version
8:39
and then a CLI version and it's using, We have the LLM proxy
8:43
that already has pricing in the back end of it.
8:46
So it's just grabbing that information and I did it in
8:49
about 15 minutes. And people use it.
8:53
It's just pretty new, but I've had our, some people, a part of
8:57
our evaluation team start playing around with it and they
9:00
were very happy with it. And what this is done, you know,
9:03
when you're evaluating and I'm just learning this myself too,
9:07
more about this process for the workflow at least is that they
9:11
are building these prototypes and up till this calculator was
9:15
here, they didn't really have a good way to go and estimate
9:18
cost. They would build the prototypes,
9:19
get a sense of the maybe what the request per second will be
9:22
and you know, different steps this agent's taking and the
9:25
token types and all of them and token volume.
9:27
And they would send it to me in a spreadsheet and then I'd run
9:29
the numbers and estimate it for them.
9:31
And now they can just hit the CLI and it's going to bring in
9:34
OK for this model, this type of agent would call system out and
9:37
it's safe to the calculator. And then they bring in another
9:39
model, do the same thing, save to the calculator.
9:41
And then they now they understand before they even
9:43
start doing an experiment that what the costs are going to be
9:47
and if it's even scalable. And she made the comment to me
9:50
that there have been times where they did a prototype, went to do
9:53
the experiment, was too expensive and had to start over
9:55
again. So not only is is this helping
9:58
us keep costs down, but also speeding up development velocity
10:02
because they don't have to go back to the board.
10:04
They know what the costs are going to be before they get past
10:06
the prototype typing stage. Yeah.
10:09
And that's just an example of how you recognized this is
10:13
something that's happening. Can I make it easier on the
10:17
folks who are closer to this problem?
10:20
How can we get these tooling? I don't want to be in a
10:23
spreadsheet doing estimates anymore.
10:24
That's not where my true value is.
10:26
My true value is is getting maybe building, thinking about
10:30
these tools and building them and then putting them in the
10:32
people's hands that are actually doing the work to make the tool,
10:35
the, the agents or what have you.
10:37
And then, you know, again, shift left, get that in early as
10:40
possible. I would love for the product
10:42
people to start maybe playing around with us if they have the
10:45
data to do it right. So I want to be as early in
10:47
development cycle as possible. And this really helps with that.
10:50
And honestly, this FP and a, my counterpart in FP and a
10:54
hopefully she sees this that, you know, kind of said, Hey,
10:57
could we book some kind of calculator?
10:59
And I'm like, you know, I think I can, I don't think it'd be too
11:01
bad. And and then, you know, about a
11:03
week later I had this thing. So, you know, it wasn't even
11:07
totally like me saying I need to go do this.
11:09
It was like, you know, collaboration with the
11:11
financing. Oh, I think we should we start
11:13
working toward this. And that that piqued my
11:14
curiosity and the luck I could. Do it.
11:17
And I like how the evaluation team gets to now understand this
11:21
and that's probably one of the core metrics that they're
11:23
looking at as they're evaluating if this is feasible.
11:26
Exactly. I know another piece of that is
11:30
support request to go to our channel, they can send in a
11:33
spreadsheet or what have you if they somebody wants.
11:36
And then my support bot is going to use my calculator I built to
11:38
calculate that. And so I that's one thing I
11:40
don't have to go and do anymore. OK.
11:43
So tell me more about where you're thinking of your ability
11:48
to add products or empower the engineering team, the rest of
11:53
the organization with this knowledge because they're closer
11:57
to those problems. So I know where the data is, I
12:01
know how to kind of structure like what things should properly
12:05
cost and and so that's kind of the magic I kind of have for
12:10
myself and I just need to take that.
12:12
And I think for all 5 thinnest practitioners probably have the
12:14
same knowledge, but they need to take that put it into some kind
12:17
of tooling. So it's closer to the
12:19
engineering in the development process.
12:21
And then shift their focus up in the organizational and be
12:24
focused more on strategy and and and actually just enabling the
12:30
teams to make better decisions moving forward on the products,
12:34
no matter, you know, even if it is going to be something
12:36
expensive that they have understanding all.
12:38
Yeah. Well, this is we have a
12:40
reasoning behind this because you know, product market fit or
12:43
what have you. So if everybody can move forward
12:46
from engineering up to in product to obviously the C-Suite
12:49
with making decisions on knowing what the costs are ahead of
12:51
time, we're going to be in a much better state as a business.
12:54
And if we looked back at again, going through that journey that
12:57
you've had of when you first were encountering AI on Azure
13:04
with Open AI and then versus now and the maturity that you feel
13:09
like you've been able to come into.
13:12
Can you explain some of the things that now you're like we
13:15
could not live without XYZ? Or say our telemetry streams.
13:22
Telemetry streams. From from my perspective, I
13:26
built some telemetry streams. So we have LLM proxy as home
13:29
build that that gives me the token volume by calling service.
13:33
And then a feature is another, you know, step below a calling
13:36
service. So like, for example, we have
13:38
our main AI chat and there's a bunch of connectors for that.
13:41
And that's a bunch of features, thousands of features, all those
13:44
connectors. So I take that information, bump
13:46
it into our, our, our cost reporting platform, which is a
13:50
third party and, and then allocate cost based on the open
13:53
AI bill. And I can do that for Bedrock
13:57
and, and any other vendor that we're, we're bringing into that,
14:00
that tool. And all it's doing is looking at
14:02
the token type. And then, you know, looking at
14:05
that cause versus the token volume of each service and all
14:08
these other metadata that's using it and then proportionally
14:10
allocating it by token type service and all those things.
14:14
So without that we would have, we would not have visibility to
14:17
actually what these agents. Cost.
14:19
And then are you looking at spikes and trying to figure out
14:23
what's going on? Why did we just spend a lot of
14:25
money here? Yes, I've got agents that are
14:29
starting to do that a little bit for me.
14:31
But I would say our we, we don't have a lot of spikes scales with
14:37
business hours. Just about all our production
14:39
stuff just scales with business hours.
14:41
Now we'll see a trend all of a sudden, you know, cash tokens
14:43
are going down and costs are going up.
14:45
Like we'll see things like that we need to pick up.
14:46
But it's very, very rare that we have like a big spike outside
14:50
of, you know, research and use cases or experiments that you
14:52
would expect that. But usually if it is a giant
14:54
spike, it's usually from my experiment that maybe, you know,
14:57
was a little higher than expected or, you know, research
15:00
that I knew about ahead of time. Speaking of research, you had
15:03
mentioned before we hit record about the research team and how
15:06
they're being empowered also by some of the stuff that you're
15:09
doing. Can you explain that?
15:11
Well, yeah, I'm, I'm trying to actually just today sent
15:13
something to them about sent them the calculator and said,
15:16
hey, this is going to be of use to you.
15:17
But no, they, I think what we're talking about is around
15:19
optimization where and it was touched on in the keynote today
15:22
too. And I thought it was a great,
15:23
great presentation and saying, you know, optimization in the
15:26
cloud was just, you know, optimizing resources and, you
15:29
know, buying savings plans in our eyes and things like that,
15:31
that fin OPS going to handle. Or it's just more
15:34
straightforward and it doesn't really impact the product at the
15:36
end of the day. When with LLMS, whenever you're
15:39
changing, OK, you know, cashing rates or I'm trying to reduce
15:44
output tokens and input tokens and all these things and trying
15:47
to use cheaper models that actually impacts the output and
15:49
the user experience. Drastically.
15:51
So it is so much more nuanced. So I have found much more
15:54
success with optimization when this research LED optimization
15:58
where our researchers are looking into saying, OK, I think
16:01
we could probably use maybe a mini model combined with a
16:03
standard model and not impact our latency, which latency is a
16:07
very important aspect there. You know, all our stuff needs to
16:10
be pretty much sub one second latency.
16:13
So you know, it's not going to impact latency and we're going
16:16
to still getting really good outputs and saving money.
16:19
So they, they, you know, maybe they'll come out with the
16:22
initial kind of launch of, of an agent and then from there they
16:27
look for ways to optimize it. We do the same thing with our
16:30
open source LLMS and we launched speculative decoding last year
16:33
and it saved us a bunch of money and on the on the open source
16:37
models and we're just trying to try to try to learn from those,
16:41
those teams and optimizations they did there.
16:43
And how can we apply this to these external LLMS?
16:46
And and again, it's all usually the big heavy optimizations are,
16:50
you know, really started from the research side.
16:53
And the product research is what fascinates me the most on that
16:56
because it's not like AI researchers.
16:59
It's not. It's, as you mentioned, the
17:03
product is so affected by any little changes in the AI side of
17:09
the house because you're in a very unique situation where your
17:14
product is driven by AI at the end of the day.
17:19
And so you have to make sure that that's the highest quality
17:22
product you can put out. But also, your main job is
17:26
making sure that it you're just not using and bringing a bazooka
17:30
every time you need to do the smallest little comma update or
17:33
whatever. Exactly.
17:35
And we have data behind the decisions we've made and we've
17:38
you know and I think most companies like this, but we just
17:40
like the data. You know, right now we have the
17:42
cost data and now we need to better tie that to revenue and
17:45
then other and experiments and you actually get experiment IDs
17:48
into the telemetry streams. I was talking about then seeing
17:51
what the cost of this experiment was and was it worth it or not.
17:54
So like it's, you know, while we've been in this and I've been
17:58
doing this work for three years now, we're still like getting to
18:01
that value phase is hard, especially as you acquire
18:05
companies and different products and different measurements, but.
18:07
Bring them into the fold. When you say experiments you
18:10
mean doing small little rollouts on?
18:13
Or do you mean like behind close?
18:16
We have a very mature experimentation platform where
18:18
we're actually, we'll be running live experiments on, you know,
18:21
they usually AB test to see, you know, how maybe a new new change
18:26
or new feature impacts. It could be in product
18:31
engagement type of rates. And and that's how we kind of
18:35
decide on or should this thing go full scale or do we need to
18:39
go back to the drawing board? And again, it's interesting to
18:42
me because it is product type of features, but then inside of the
18:48
core product, there's probably a lot of different experiments
18:52
that you're running with all the models or with the ways that
18:55
you're making those models useful.
18:59
It's the same product for me as the end user.
19:02
I'm still getting if it's we're talking about Grammarly.
19:04
I'm still getting suggestions on my grammar, but you behind the
19:08
scenes are running these experiments that I may or may
19:11
not know about. You may not know about and all
19:12
of a sudden you're engaging with it in a, you know, slightly more
19:17
volume and you may not even realize it, but then, you know,
19:20
our, our people are picking up on, OK, we got more, more, more
19:24
accepts here for this suggestion.
19:26
Or, or they actually, you know, the clicks on this specific
19:29
agent actually increased over time or the amount of time turns
19:33
they had with this agent increased.
19:35
Like little things like that that indicate, hey, this is
19:37
actually making a meaningful difference for our customers.
19:40
And you want to be able to, just to close the circle on this, as
19:44
you mentioned before, when you have those experiments,
19:47
recognize the cost of those experiments.
19:49
Exactly. Tie that to the cost
19:51
fluctuations that we're that I'm seeing on my side because right
19:54
now it I'll see these spikes and I do see spikes.
19:56
It's usually around experiments and I have to do a lot of
19:59
research or have you know, I'll let them go do it for me to
20:03
figure out what exactly is causing this.
20:05
And even then, the correlation is light.
20:08
It's really hard to to find a correlation.
20:10
So you know, we've got, we got a couple teams collaborating to
20:13
bring that data into our our cost reporting so we we can
20:17
actually measure the cost of those experiments.
20:19
Because I imagine the teams aren't saying like, hey, I'm
20:22
going to run this experiment now, like just in case.
20:25
Sometimes, sometimes they'll. Give you a heads up.
20:28
I've been very blessed with with a good relationship with a lot
20:31
of the the teams that do these experiments.
20:33
So the larger ones, they usually come to me and either to
20:35
estimate what this is going to cost or just kind of let me
20:39
know, hey, we're going to run this thing.
20:40
So I know about the big ones, but then they're still small
20:43
ones that are supposed to be small, but then they end up
20:45
being a little bigger than expected, right?
20:47
But overall, you know, there's really good communication
20:49
between us. So on those times where you see
20:52
the spikes or when you're getting a heads up, is that
20:55
something that now you're expecting folks to have a better
20:59
idea of because of the cost calculator that you created?
21:02
Yeah. I mean, ideally they would, they
21:04
wouldn't need me to do those estimates beforehand or at least
21:07
they'll have an idea of what the costs are going to be before
21:10
they run the experiment. And I haven't built this into it
21:13
yet, but I'd like to actually have that going into a doc, a
21:16
coded doc, you know, one of our company.
21:18
But I'd like for whatever submissions that they have in
21:21
this, this tool. And they can, I'm assuming they,
21:24
I'll probably have something where they can actually choose,
21:26
OK, this is something to submit. And then it goes to a dock where
21:29
we see all the estimations in aggregate and then can pick up
21:33
on any red flags or at least do better planning.
21:36
OK, we know we're running this thing during this date or within
21:39
the next few months. We usually don't have dates
21:41
until, you know, pretty late into the process and we can
21:44
better plan on, you know, what our finances or costs are going
21:47
to be down the line. And also compare it with the
21:49
real cost to make sure that hey is our calculations, are we
21:55
projecting, is my calculator. Broken or is, you know, request
21:59
rates are a little, you know, off.
22:01
So yeah, that's true. And what are the different
22:04
inputs that you ask folks to give you on this cost
22:08
calculator? So average request per second
22:12
more minute and then average request size.
22:14
So when I say request size, that's input tokens, output
22:18
tokens, cashed input. And these are all things that I
22:21
said they're kind of building the prototype and, and testing
22:23
out the, the outputs that's coming.
22:25
You know that they they get those token counts from as a
22:28
response body from the the open AI or Anthropic, whatever you're
22:32
using. And I wasn't clear when you were
22:34
talking about how you have the research, the product research
22:37
team, then they have this knowledge of we can do XYZ with
22:43
these models or we can cache this, or we can set up a
22:46
different architecture that will be a better product experience.
22:51
Does that then get propagated through the different teams so
22:55
that whatever next team knows oh I now have the ability to make
23:03
my experiment much cheaper? We could be better about that,
23:07
we could be better about advertising how teams are doing
23:10
this. Well, one other issue there
23:12
though is you know, when you look at our mail product, the
23:14
way they use LLMS is way different than than the way say
23:19
Grammarly uses LLMS or even our new Go product.
23:22
So that is one kind of issue. Now I think there should, could,
23:25
should be more sharing across the, the product or the business
23:29
units and, and those products because I'm sure they're still
23:32
learnings there. But again, you know, these
23:34
acquisitions happen not that long ago.
23:36
So it's, it's going to take time for those things to be sure.
23:39
And I think it's probably part of my job to make sure that
23:42
those are broad and broad kind of broadly told to the company
23:48
so that other people can't take advantage of.
23:51
But that's something I think we can improve.
23:53
Well, it does feel like what you're saying too around a lot
23:59
of times folks will try and let's use the easy example of
24:02
just caching certain results or certain tokens so that you can
24:07
spend less. But then it ends up having a
24:10
worse product experience some of the time.
24:13
And so you want to make sure that the teams aren't using that
24:18
type of architecture at the cost of like having less token cost
24:25
I. Guess Oh, exactly.
24:26
And I mean, I first and foremost is always the best product for
24:30
our customers. Like we, we are not going to
24:32
sacrifice that, especially in this very competitive area that
24:35
we're in. We're not going to sacrifice
24:37
that at all. But it's more about OK, be
24:39
dealing the best product. But what are some of the things
24:41
that that are we can still deliver the best product, but
24:44
also save money on the back end. So the team is just because
24:48
these researchers have much deeper understanding of these
24:52
LLMS, they're able to go in and tweak those certain things or
24:55
know the direction to go. And then they are in in the
24:59
position to go in and evaluate the outputs and kind of go
25:02
through that process to make sure it's all good.
25:04
There's no way a fin OPS person could could do that even if they
25:08
knew how to adjust the LLMS and make it cheaper.
25:11
Evaluating the outputs in a way a data scientist or a researcher
25:15
or a company with computational linguist as well.
25:17
They that's a whole another skill set that, you know, fin
25:21
OPS people aren't going to have. So that's what makes this a lot
25:24
harder to do. I mean, I've also tried to use
25:26
claw to optimize claw and our, our like clawed API products
25:31
and, and it gives good advice, but it's usually still just not
25:35
quite, quite there. We've already, you know, tried
25:39
it or the impact wasn't, you know, that large, but that could
25:43
be another Ave. is just use AI to optimize AI.
25:46
So tell me about how this reserving capacity kind of cost
25:53
benefit analysis goes in your head.
25:56
Do you get cheaper tokens if you reserve more capacity?
26:00
And is that something you even look at, or is it something that
26:04
has kind of gone out of fashion? So early, early on, that was
26:08
really the only reserving capacity as you're up in the eye
26:13
of the time, PT use was the only way to get low consistent
26:16
latency. And as I mentioned with with
26:18
Grammarly, you know, sub one second latency is just hugely
26:22
important for our product. So yeah, we need a low
26:25
consistent latency. So yeah, in the beginning I was
26:27
managing capacity, having to buy capacity for the peak.
26:30
And if we didn't, you know, we would scale beyond our our peak
26:33
and and have an outage. But then over time, we did see
26:37
token rates come down for that capacity, but it was a really
26:41
difficult thing to manage. One, the actual utilization of
26:46
the capacity you could get, you can get metrics to know what
26:50
your utilization was, but understanding it beforehand was
26:52
really hard because you know, everything's measured in these
26:57
units, abstracted units that Azure measures them in.
27:00
And depending on the request size or shape that the each
27:05
agent is sending, that utilization is going to be a
27:07
little different. So knowing that ahead of time
27:09
was tough. RL and proxy team actually built
27:12
a way they call it shadow traffic where it would mimic the
27:16
traffic of this thing that was going to be coming out to see
27:20
what would in load test the actual capacity and see what
27:24
the, you know, peak tokens per minute were.
27:27
And then that's how we would estimate, OK, we need to add
27:30
this much more capacity for this new thing that's about to launch
27:33
or you know, we can actually reduce capacity because things
27:35
have come down. So that was back in that day,
27:39
but it was just still very difficult to measure ahead of
27:42
time and it was a lot of this mathematical lean limbo and
27:46
exhausting. Then I want to say it was like
27:49
late 2024, early 2025, Open the Eye came out with priority
27:53
processing. So this has a late latency
27:56
SLA's, availability SLA's. It is double the cost as far as
28:00
from a rate perspective as the standard RAID.
28:04
But again, we needed a low latency.
28:06
What attracted me to it was I didn't have to manage capacity
28:09
anymore because this is completely on demand And so it
28:12
scales with our usage. Our usage scales with with
28:14
business hours. So then I did the numbers and
28:17
looked at, OK, if we had the even looked at if we had an
28:22
ideal amount of reserve PT use and then also with our proxy
28:26
sent some traffic to priority processing.
28:29
Like what you know, could that save us money in general, just
28:32
comparing PT US, you know, fully using PT us versus priority
28:35
processing. And really it came down to you
28:38
could see some savings with the minis and nanos of the world,
28:41
but you couldn't, it wasn't enough to, to need to manage
28:46
that capacity with the standard size models.
28:49
There was no savings even with using PT us.
28:51
And this is a, this is a year or so ago, it may have changed, but
28:54
I doubt it. So priority processing was an
28:57
absolute no brainer. Not on top of just making my
28:59
life easier. It also allowed it makes much
29:03
easier to allocate cost because again, it's just very difficult
29:06
to understand how much of a single service he's using of, of
29:10
this PTU. Because yeah, you could look at
29:13
token volume of raw, but tokens are not equal across agents.
29:18
And I took so it was just very complex of this and I improved
29:21
our visibility, saved us some money at that time at least and
29:25
and and it made my life a lot easier.
29:27
I can only imagine you doing this complex math like what's
29:30
the right blend of PTU or just straight token.
29:35
Pretty large people sheets out floating around.
29:38
And then you're like, wait for quality of life.
29:41
Let's just do it on this there. That should be one of the
29:44
factors too. Like you don't have to do this
29:47
crazy math anymore. You know what you're going to
29:50
get. And it also if it's the same
29:53
more or less the same price then it's a no brainer.
29:57
Exactly, exactly. And then something we were
29:59
talking about a little bit earlier around Open the Eyes,
30:02
release of guaranteed capacity. That just happened, right?
30:07
I find it really interesting because it's not advertised as a
30:10
discount mechanism. It's, it's advertised as a, you
30:13
know, guaranteed capacity, you know, and, and I'm just.
30:16
Saying like, if you need this, you're always going to have it,
30:19
so you're never going to be without resources.
30:22
We're not going to go down on you.
30:23
Exactly. But from our standpoint, we
30:25
haven't really seen capacity issues.
30:27
So it's almost like they're they're solving a problem that
30:29
doesn't exist yet. Or they're foreshadowing what's
30:33
coming. So I'm very interested to see
30:35
how this play out. It could also be first a first
30:38
step toward a savings planner RI kind of construct.
30:40
But but yeah, nonetheless, I, I see a lot of financial risk in
30:43
these external loan providers and, and prices are going to
30:46
continue to rise and or they're going to want multi year
30:50
commitments. Because with the guaranteed
30:53
capacity, you can't do that for a month, right?
30:55
Oh. No, no, they're multi year is is
30:58
what I believe that they're looking for there.
31:01
Yeah, yeah. Which I don't know 100% for
31:03
certain, but yeah, that's, that's my impression.
31:05
Yeah. That is something that now
31:08
again, you're thinking about it and you're starting to realize
31:11
these cost benefit analysis and saying, huh, if we're good
31:16
enough now we're never going to change.
31:19
But then later down the line, if it starts to get a little shaky,
31:24
now you have to recognize, is it worth it for us to get this
31:28
guaranteed capacity? Or as you mentioned earlier,
31:31
too, like they're slipping in a lot of new costs like the data
31:36
residency, but now there's this new cost on this.
31:40
And so they're almost upselling you in every way, shape and
31:43
form. Exactly.
31:45
And I think it's really hard from my perspective at least to
31:48
do any kind of multi year commitment against, you know, in
31:51
this AI world that we live in that is that's really hard to
31:55
look at all the changes in the market and say, OK, we're going
31:58
to be using, you know, said provider for three years.
32:02
So it's just it's too fast-paced.
32:04
So yeah. But if there are people that are
32:07
probably in position to, you know, be able to do that, our
32:10
proxy also gives us the ability to kind of use whomever we want
32:14
to use, which is it's nice having a Bender agnostic proxy
32:18
for that.