Latest / Agentic Conversations (formally mlops.community) / We Cut LLM Latency by 70% in Production

Transcript

We Cut LLM Latency by 70% in Production · Agentic Conversations (formally mlops.community)

0:00
Tensor RTLLM was so helpful for us to cut our latency by like up
0:04
to 70%. How can you run the least number
0:08
of GPUs needed to serve the traffic you have?
0:11
How did you deal with the cold? Start time a few things.
0:14
So first of all, we start using I'm not an AI person initially,
0:24
right? Like 6 years ago, I had no idea
0:26
what, what is all of this about? And then, you know, I was
0:30
leading engineering teams and I got my scope going bigger and
0:32
bigger. And then I was like, you know, I
0:34
need to be an AI, need to be in machine learning.
0:36
I need to get there. So I start, you know, picking up
0:40
and managing teams. I joined a company that is an AI
0:42
company doing machine learning, traditional reinforcement
0:44
learning. And I Start learning from my
0:46
team. I was like, hey, junior machine
0:49
learning engineer, how does this work?
0:52
And again, it's, it's a little bit of, you know, being humble,
0:55
but also being hungry to learn. And then from there, it's like
0:58
it turns into a passion of, of learning and growing and what
1:02
Gen. AI became like just the perfect
1:05
timing of and I really see the future as like all these senior
1:09
VPSCTOS, the classical traditional, you know, CT OS now
1:14
getting into AI. So that is shifting like if you
1:17
just did everything else except AI and now is not the time to be
1:21
competitive in the market. So I was, you know, not only
1:25
following my passion, but also, you know, adapting with the
1:28
market because I think that's it's going to be a big shift in
1:31
how we lead engineering teams. I couldn't just stay on the on
1:35
the sideline, watch it. What the AI is, is what's
1:38
happening in AII. Had to build AI, had to lead AI
1:41
and I had to learn. First of all, I had to learn AI
1:44
alongside my team. So I came up with this framework
1:48
of like learning instead of being very directive and like
1:51
being the boss and hey guys, we need to build an MCP server, We
1:54
need to do this, we need to do RAG, we need to even learn the
1:57
basics. I go and Start learning and ask
2:01
them to learn with me. Because again, one of the
2:04
problems we have seen since Gen. AI is that like you don't have
2:08
all the resources, you don't have all the AI engineers
2:11
available. Like some of these engineering
2:14
leaders like myself, we are in positions where we're asked to
2:17
do AI without the AI resources available and with no previous
2:22
team or experience building AI. So learning was #1 and then how
2:27
would you be successful at building AI and enterprise
2:30
production? So how do you take all of these
2:33
great proof of concepts and all the hype around like, oh, I can
2:36
do this, you can do that. Or look, a build a chat bot.
2:39
You can do everything in the in the platform, just an MCP and a
2:42
chat bot. And but how do you take that to
2:46
to scale? Another topic that could be very
2:48
interesting is and you might know this is building AI in
2:51
house hosting, self hosting, self managing AI, not not, you
2:55
know, leaning on all of these third party available API based
2:58
AI. Now you need to manage your own
3:00
GPU, choose a GPU, choose the architecture to choose the open
3:04
source models optimize, you know, fine tune a distill.
3:10
I mean optimize late. I mean this is what I was more
3:13
talking about recently is how you manage what I ended up
3:15
calling the AI iceberg of like AI is great and you can do all
3:19
of this, but behind it if you want to build it internally and
3:22
I don't know if you manage it that scale, all of these
3:26
invisible things that you know, performance, latency,
3:30
throughput, accuracy, quality of the responses and cost.
3:34
I mean, one of the other thing that was taken so much time for
3:37
me recently is like how a manage cost of AI so I can prove to
3:42
leadership, to the board, to the investors and to the market that
3:45
you can build AI at the, you know, a positive AOI.
3:49
And how can you retain your customers and gain more by just
3:51
building AI that doesn't break the your wallet?
3:55
So yeah, this, this kind of pyramid of cost, performance,
3:58
latency and the throughput was my focus.
4:01
And where can you move needles? You can never have all of them
4:05
optimized. So what is the, what I find out
4:10
being called like an AI Nexus? Where is that spot?
4:12
What it works really well for your use case might not work for
4:17
other use cases, but for our use case being an HR tag, you know,
4:20
focus on privacy, responsible AI performance and latency are
4:25
needed but not not the top, top priority, but you know,
4:28
efficiency and accuracy of the results and also cost.
4:31
So finding that white spot and build AI towards that.
4:34
There's a really interesting thread to pull there with cost,
4:38
because you have the cost of AI for your company, but then
4:45
there's the cost of AI using AI coding agents and yeah, for the
4:51
engineers. So there, it's almost like how
4:53
we're implementing AI with the company, whether it's internally
4:57
facing or externally facing, you're trying to optimize that
5:01
and then also prove to leadership that there's ROI and
5:04
then there's like this whole other vector of AI that's going
5:09
and being very expensive in its own right of agentic engineering
5:15
and the coding agents paradigm and.
5:18
So and the premium tokens, and I mean, we, we introduced a few
5:24
solutions and then they worked for some time.
5:27
And then Gitot Copilot as an example, like you have a bunch
5:30
of models available, but then you can you start see more and
5:33
more premium models and then some of them, their tokens would
5:36
be 10 times more expensive than regular tokens.
5:39
So at some point, yeah, within engineering, we had the, we had
5:43
the finance kind of role within engineering that was taken care
5:47
of. He's like ensuring everything
5:48
goes within the, you know, you know, allowed budget because
5:53
otherwise if you open the door to everything, it's just going
5:56
to blow up the consumptions and tokens and premium, you know
6:00
stuff. So I had to keep that in control
6:02
with, with the AWS bills and all the AI, you know, GPU's because
6:06
we are deploying models on the GPU's.
6:09
And so I had to work on scaling as an example.
6:12
So how can you run the least number of GPU's needed to serve
6:17
the traffic you have? So understanding the pattern of
6:20
traffics, like as an example, you know, we are an HR tech
6:23
solution, so people don't use it at night and weekends most of
6:26
the time. So why do I have like that many
6:29
GPUs running at that time? So we had to.
6:32
Yeah. So we had to create first a
6:35
scheduled scaling for our solutions based on the pattern
6:39
we see on the usage. And then we had to also
6:41
introduce dynamic scaling where if you have a, you know, at the
6:44
end of the year or at the end of the quarter or even at the end
6:47
of the week, sometimes people go and try to push as many, you
6:50
know, HR data like feedbacks and conversations that recognitions.
6:54
So you had to take that into account as well.
6:56
So you had to dynamically adapt and spin up GPU's fast.
7:00
If you use open the eye, you don't have to worry about any of
7:03
this. But then for me, I was like, OK,
7:05
well, we need to spin up things and they are containerized and
7:08
we need the images. But the but the when you spin up
7:11
a new container, you don't to download the model put in there.
7:14
And I was like, takes forever, yeah, I had to talk to AWS.
7:17
How can we make this faster? Should I go with Sage Maker or
7:20
solutions like that that is managed?
7:23
Or just hit bedrock. Or, or hit bedrock, which is
7:26
again, many layers. I mean, I thought if you
7:28
remember this, I had like AI came up with these layers.
7:32
I mean, they are known like model as a service, software as
7:34
a service, umm, you know, AI as a service.
7:37
And then at the bottom there was nothing as a service.
7:39
And honestly, that's that's what we're doing.
7:42
We're like we're trying to do. Everything and.
7:44
When I had the conversation with AWS Sage Maker experts from AWS,
7:48
they were like you guys doing exactly what we do for Sage
7:51
Maker behind the scenes. So yeah, you can hop on Sage
7:54
Maker, you lose control some of the configurations, but you will
7:57
have the this given for you. And because our again scale and
8:01
usage is is limited, we were able to do some of that in house
8:04
without having to rely on a on a managed service.
8:08
So yeah, dynamic scaling, scheduled scaling and can for a
8:12
proactive scaling sometimes. How did you deal with the cold
8:15
start time? If you think so, first of all we
8:19
start using faster IO operation storage from AWS which is FSX.
8:26
This is kind of new to me too. I was like, give me the fastest
8:30
way to download the file from your systems and I would I
8:33
learned about this usually. I mean, historically we used to
8:35
get this from S3. So that's one downloading
8:39
faster. Then the second thing was
8:41
instead of spinning up them the the image, the container image
8:44
and then downloading the model, we created an image with the
8:47
model embedded in it or. Ready.
8:49
You baked it in. Yeah.
8:50
So when you spin up a new Kubernetes node, it's going to
8:54
come with the image. So yeah, we saved so many, I
8:58
mean in the minutes, not just in seconds.
9:00
So we we saved minutes off the the cold start to just spin up
9:04
new new. But the GPU itself takes a
9:07
while, right? Yeah, the GPUI mean that time we
9:10
cannot do much about it, but the downloading part was the was the
9:13
triggering. Yeah.
9:14
But I'm just thinking like because there's certain
9:17
providers, GPU providers that go hard on, hey, we can spin up in
9:22
seconds type thing. I'm thinking like modal or or
9:26
others that. Was another one, yeah.
9:28
Yeah, there's another one I can't remember now.
9:32
It's not. Crock Yeah, they need to do
9:34
more. They need to do better marketing
9:36
so that there's. One that is like known to be the
9:39
fastest kind of cold start. It's like you want, you want the
9:42
new GPU, the here you go. I wish I was able to get like
9:46
really breakdown of the of the of the latency, how much time it
9:50
takes and understand, OK, this is for every step we focus on
9:53
the ones they used to take the most and definitely downloading
9:56
the models running, you know, spinning up and you know, was
9:58
taking the most. So that's where you saw the
10:00
biggest. Gain, yeah, another thing we
10:02
did, I mean, this is maybe not necessarily related to that cold
10:05
start, but yeah, optimizing using NVIDIA tools solution like
10:09
Tensor RT LLM was so helpful for us to cut our latency by like up
10:13
to 70% on the same. Yeah, Yeah, because I mean,
10:17
you're familiar with Tensor RT. So what Tensor or TLM does is
10:22
it's going to rewire your, your model in terms of like the
10:26
neural network to to be adapt to the to the architecture of the
10:30
hardware. So you define you, you tell it,
10:33
OK, I'm going to use NVIDIA, you know, whatever.
10:35
H100H100 whatever. So it knows the architecture, it
10:39
takes whatever model like LAMA three, LAMA 4, whatever and it's
10:42
kind of tweak it to just fit for that architecture, optimize it.
10:47
We saved at least 50% latency. That was like a again, that's
10:52
why I told you about the throughput, latency, cost and
10:56
performance and accuracy. Like we, we have been saving
11:00
cost and saving on latency and saving on cold starts.
11:05
And also when you use Tensor or TLLM and we use specific models
11:09
from models and GPU's, you get more batching capabilities.
11:13
Now. I mean, the first time I start,
11:14
I mean why my learning has been evolving so much is the first
11:18
time I was learning about how you deploy a model into the GPU.
11:21
I was like, OK, we use like let's say an 8 gig 8B model.
11:26
You need a 16 giga to to have it there.
11:29
OK, so why do I need to have like a 24 or why it's taken so
11:34
much kind of gig? Sorry it no, sorry if it if you
11:38
take a sick an 8B quantized to kind of four bits it takes for 4
11:45
gig of memory. So when when we were thinking to
11:48
of picking which NVIDIA model or any GPU would work best for us,
11:52
I was trying to get the smaller one where I can fit that kind of
11:55
4 gig or 8 big 8 gig. Yeah, you fit it nice and tight.
11:58
Yeah, you fit. It there and it's all good and
12:00
that's all but. That didn't work.
12:01
But then I was, you know, as as my learning was evolving, I was
12:05
like and, and, and I was even thinking of fitting 2 models in
12:09
the same GPU to try to create throughput.
12:12
So if I have two models running on the same GPUI can maybe send
12:16
twice more requests. But then what I ended up
12:18
learning is like the way the KV cache work and all of these
12:21
things. It's like you said, you get your
12:24
your model deployed and then use the rest for your KV cache.
12:29
And with the KV cache you can do batching and with tensor RTLM as
12:32
an example, you can do in flight batching.
12:34
So meaning if you when you create your batch of like say 16
12:38
requests, you know 14 are processing 2 are done.
12:42
You don't need to wait for these 14 to be done to create a new
12:44
batch. You can have two more added to
12:46
that same batch so it keeps going forever.
12:49
Again, this is where when I talked about the learning aspect
12:52
of this and like all the tools available and not be overwhelmed
12:55
and focus on optimizing 1 area at a time.
12:59
I was like, yeah, instead of trying to do two or three models
13:01
and one GPU, I'm going to do 11 model on the same GPU and use
13:08
everything else for KB cache and I'm going to have a huge
13:11
throughput. While again, with optimizing the
13:13
sensor RTLLM you would have, you will also reduce latency.
13:16
So I was like, wow, OK, I don't need to go crazy on trying to
13:20
fit as much as I can within the GPU.
13:23
Just use the memory. It's it's kind of
13:26
counterintuitive to how would you think about regular CPU and
13:29
memory you try to fit. OK, if my system will only use
13:33
this much, I will I can have, I can do more virtual systems
13:37
running on the same hardware for, for for GPU and AI.
13:40
I was like, no, no, no, no. Everything I have left, if I
13:43
can't change the number of you know how big is my batch and I
13:47
use all the remaining memory left on the video memory.
13:52
Well, we can have a lot of throughput so and with influx
13:54
going to even be better and faster.
13:57
And did you figure out any ideas around the amount of money
14:07
you're spending? Because if you get more
14:09
throughput it, it enables you to get a bigger machine.
14:13
Yeah, right. And yeah.
14:14
So increase the GPU. So obviously we went from
14:17
smaller ones to bigger ones. I don't have the exact names by
14:22
going to bigger GP us and you know, having one model,
14:25
increasing the KV cash, improving throughput, I didn't
14:28
need to run more of these machines.
14:30
So this is this is again another counterintuitive approach.
14:34
So I have my DevOps infra team coming to me and saying, OK, we
14:38
need, I mean, it's better for us to go and get reserved instances
14:41
from AWS to pay 30 percent, 40% less.
14:44
But reserved instances you need to commit to a certain number of
14:47
years. And then on in terms of the GPU,
14:50
we were trying to get the smaller GPU that gets there and
14:53
serve us the best. But then every time you pick a
14:57
GPU, there's a better GPU, right?
14:59
Faster, better, more memory, but more expensive.
15:04
And my thinking was if I can achieve a higher latency,
15:07
obviously everything will get done and leave the GPU so I have
15:11
time to get more things done. And if I increase my throughput
15:14
so the number of lanes I can get AI to be running well, I would
15:19
be able to serve more customers faster.
15:22
So I don't need that many GPUs. So what happened is I did some
15:25
math of if I'm going to go from this GPU to the next GPU, it's
15:29
going to be 30% more expensive, but I'm going to use 50% less of
15:33
the time. So it's going to be like 30%
15:36
cheaper. So we were upgrading on a
15:39
frequent basis, upgrading GPUs while saving money.
15:43
So talking about the scheduled scaling I had, I'm going to take
15:49
some random numbers. Let's say when we start the AI,
15:51
we were running again, 10 GPU's for the whole day.
15:56
And then we looked at this and we said in the morning and
15:58
night, nights and weekends, we don't have that.
16:00
So let's do 5 on these times 10 throughout the work day, like
16:04
from 8:00 in the morning to like 6:00 PM, we'll have 10 machines.
16:08
So I was just by doing that and I was doing the math how many,
16:11
you know, GPU is I'm saving what's the cost per hour and
16:14
then I was always reporting on these numbers.
16:17
This is how much I'm saving on a daily and weekly and monthly
16:20
basis. And then and then now from
16:23
instead of having 5 and 10 because they have better GPU's,
16:27
faster high throughput with the optimization using tensor TLM, I
16:32
don't need to have five in the weekend. 2 are good enough to
16:34
serve and the food throughout the day and only 10, six are
16:38
good enough. And add to that like multi
16:41
region. So we also have Europe regions
16:43
and stuff and and these kind of the same, same strategies work
16:48
but for different numbers because of the traffic and the
16:50
load you have. So over time, this is again,
16:53
this is what when reflecting on it now, I feel like I was never
16:58
able to do this. I was able to save money.
17:01
I was able to improve performance, latency and
17:05
throughput and even quality because I was using bigger
17:08
models on bigger GPU's. I was kind of optimizing all the
17:12
what I was describing in the pyramid, all the edges of the
17:16
pyramid every time with with by learning and exploring all these
17:22
techniques and technologies. And again, I don't have like an
17:25
infinite budget to hire the best people who know about all of
17:28
this and bring them. I was doing this iteratively
17:31
with my team, getting into, you know, talking to AWS, talking to
17:35
NVIDIA, understanding, you know, what tools, what, what tools we
17:38
have available to go and optimize one of each at a time.
17:43
Yeah. But it's, it's been working and
17:45
it's been magic. I don't think I've ever done
17:47
this for any sort of like cloud distributed systems before where
17:50
you can optimize all of this. There's a trade off like you
17:52
want more power or sorry, it's going to be more expensive.
17:56
You want to something else you want, you're going to trade off
17:59
something in AI. And I understand it's still like
18:01
there's a lot of trade-offs to do, but if you do them while
18:05
focus on your use case and what you need, there was a way to
18:08
optimize on many levels. Yeah, you're walking a
18:13
tightrope. Yeah.
18:15
And you recognize that if XYZ remains true, then that allows
18:21
us to get a beefier machine to have things get done faster.
18:25
And especially when we have higher demand, we're going to
18:30
really take advantage of it. And then we can scale down and
18:33
take advantage of like the lower demand.
18:36
So we don't need these machines. And because you're saving money,
18:40
that probably gives you more expendable cash to test bigger
18:45
machines. And see does this scaling law
18:48
your own little world scaling? It was again, that that is
18:51
exactly what was going on when I was talking to my team and my
18:55
team, I think we all experienced this maybe for the first time.
18:59
But my DevOps infra SRE team were like, hey, like if you go
19:05
bigger machines, you're going to pay more.
19:07
They are like 3040% more expensive.
19:10
And when I was thinking about this, I was like, yes, yeah, I
19:13
want the more expensive thing because I'm going to run it less
19:16
number of hours. My unit is not locked into how
19:20
much is the cost of the machine per hour.
19:23
It's how much I'm going to run these machines overall and we
19:26
pay per the minute or maybe even by the 2nd.
19:30
So as long as you spin up and spin, you know, spin them up and
19:33
turn them off as soon as you're done and you have this dynamic
19:36
scaling that really minimized. And you have also the scheduled
19:40
1, you can run the fewest number of hours using higher GP us to
19:48
serve more customers over time and you're saving time and
19:51
you're improving latency. All of this also the other thing
19:55
we can talk about these the models themselves.
19:57
So we have been also able to upgrade our models from to
20:00
bigger models, better ones. And I'm not even talking about
20:03
our explorations on using different models now, because
20:07
for the different use case, I have use cases where it's more
20:09
about rephrasing, polishing text, you know, I'm trying to
20:12
give you feedback, work together in HR tech.
20:15
You know, I'm going to give you feedback and there is an agent
20:17
that will help you rephrase and Polish this to be, you know,
20:20
fit, polite and nice and actionable.
20:23
I mean, actionable and insightful.
20:25
That's what we say. So that is we're going deep into
20:29
the weeds here and this is just experimentation.
20:33
But if it's based on just rephrasing your, your input and
20:38
output are small or about the same size, right?
20:44
So understanding that and understanding how prefilling and
20:46
decoding work, you can think about like maybe I can use a
20:49
smaller GPU to just do this and do it faster because it's kind
20:55
of the same input and output size.
20:58
But there are other features where you you go and generate
21:00
the whole program plan and that for that it's maybe short
21:04
instruction, but the output is big.
21:06
So teams like the prefilling phase is very quick, very small,
21:11
but it's all about decoding, it's all about generating new
21:13
tokens to just, you know, the new data.
21:15
So maybe there's a different architecture, different models,
21:19
different GPU's. If you do also load balancing
21:23
that is based on this information, you can go to the
21:26
GPU's that are that are that did the prefill phase the same.
21:31
Because of this, at the end of the day, most of the prompt or
21:33
the instructions are the same. Go and generate for me a plan
21:37
for a manager that is this XYZ. It's the main difference is in
21:41
the decoding part. So some of these GPU's might
21:44
have if you design properly might be capable of saving a lot
21:49
on the prefilling phase and go straight into like the first
21:51
token start generating immediately.
21:54
Again, these are these are next level things that I'm looking
21:58
into and there I can go into again for some use case, I can
22:01
go to smaller GPU's, smaller models and other, other use case
22:06
I can go bigger ones. So now I think the next phase is
22:09
that kind of layer of routing and proxy gateway.
22:14
Yeah, we build something we call the DLM proxy, which is like an
22:17
AI gateway, decide where to go, which GPU, but that require us
22:22
to really have load balancing. And you know, that is informed
22:27
on AI data, not just on memory, CPU and VRAM usage.
22:32
It's more about like who has more prefilling for, for this
22:35
request, I'm going to take it to that GPU because you're going
22:38
to, if you go to that GPU, just go and decode.
22:40
If you go to the other GPU, you need to prefill and then decode.
22:43
So it's getting interesting. But the good thing is like once
22:48
you build a good foundation and you really get into
22:51
understanding these, and as you said earlier, you lock a few
22:55
parameters and you focus on optimizing one of them, like
22:57
costs or latency or throughput. You can alternatively go and
23:00
improve if you want to get them all improved at the same time,
23:04
that's when you get overwhelmed. That's where you get like, Oh my
23:06
God, the sick. Yeah, there's like a new model,
23:08
a new system, new framework. Can you umm cloud AI GPU
23:13
provider that does all of this go new?
23:15
Way to quantize a new way to yeah, yeah, yeah.
23:19
And fine tune, I mean, at some point also fine tuning, we were
23:22
really actively thinking about it at the beginning to try to
23:25
adapt our use cases to, to HR tech.
23:28
But then we felt like it's, it's, it's going fine.
23:31
He like kind of foundational model are good enough to just do
23:36
it without fine tuning. So quantizing was big good to
23:39
just scale it down, but nothing more.
23:42
And then one of the other challenges though, is becoming
23:44
more and more serious is translations is, is supporting
23:49
different languages. We are in a space where our
23:51
customers are not just English speakers.
23:53
We have 30 ish language with that and we need support.
23:57
So which models do this the best?
23:59
How do you do that? How can you manage this and how
24:02
can you even evaluate accuracy and quality of these responses?
24:06
Yeah, and you need a bigger model, bigger machines for
24:10
models that can handle more of these more.
24:13
Languages, yeah. And talk to me a little bit more
24:15
about how you're championing for the ROI of AI, because all of
24:21
this is well and dandy. But then if you can't prove that
24:27
these AI generated summaries or polishing or any of that
24:31
actually provides value to the customer, then it's like, cool,
24:36
you just did a lot of engineering work to make this
24:38
incredible for an engineer. But the.
24:42
City for the NTS. Yeah, the leadership is sitting
24:44
there going like, So what? Yeah.
24:46
Again, this is part of things I get exposed to a lot.
24:50
I'm not that kind of typical practition, you know, an AI
24:54
that's just focus with the terminals and like CLI and all
24:58
of that. I do that, but also I'm very
25:00
business oriented and I'm part of like council internally at,
25:04
at the company I work for, where we put together vision and we,
25:08
we try to assess where AI will have an impact.
25:10
So before, when we started, you know, building AI, we, we came
25:14
up with this framework. I ended up calling the flywheel
25:17
framework to be able to plan for what where AI will will help in
25:22
terms of the product and then with engineering and technical
25:26
capabilities, go and build the first iterations of that.
25:30
The the third step is to run it really run it and look at the
25:33
you know what, how it's going. And then the 4th one that will
25:37
connect you back to the planning is optimizing, which I was
25:39
talking a lot about like a go and optimize, Go and improve
25:42
latency, accuracy, add support to other languages and.
25:45
Stuff. That's the end of the whole
25:48
flywheel. First you have to.
25:49
Yeah, understand if it's. Exactly.
25:51
Worth doing and. Yeah.
25:53
So in the planning phase is where we meet across different
25:56
departments. It's not just product and
25:58
engineering deciding what we do, right.
26:00
Like we work with our customer success, we work with our, you
26:03
know, legal and ethical teams. We we work with our sales.
26:07
We really try to assess where AI can help us push the business
26:12
forward. And when it comes to me and my
26:15
contribution to the conversation is like, what is the easiest AI
26:20
that we can build that can have the highest impact?
26:23
I don't want to go and build like a six months, nine months
26:26
projects and then find out that maybe, yeah, it's it was a big
26:29
investment for a small ry. I want to build the easiest
26:33
thing that will really change the experience.
26:34
So by collecting this information, work across these
26:37
and we have also customer councils.
26:40
So we really work with our champions at our, you know,
26:43
because we are B to B. So we collect these information
26:46
from our customers. And if we understand where do
26:49
they think AI will help their, their end users, their
26:52
employees? It's so funny because at the
26:55
early ages of building AI, I mean now we're talking about
26:58
ages, but the early days of building AI, we had a lot of
27:02
pushback on, on the adoption. You know, there were a lot of
27:05
concerns, lack of trust, You know, even the image of some of
27:09
these AI vendors were not great because they were big customers,
27:13
enterprise customers globally. We're concerned about using
27:16
their private data to train models.
27:18
So they didn't really trust the systems early on.
27:21
So they were. Has that changed?
27:24
A completely changed service they.
27:26
Don't care, they say train all my whatever exactly if it's a
27:29
better. A lot of customers used to say,
27:31
you know what, we don't want any data to go to any, any vendor.
27:36
We don't want the AI to be involved here because we don't
27:38
trust these vendors. So that's our first push into
27:41
building AI internally, self host, self manage.
27:45
Our first early proof of concepts where through API based
27:48
AI like open AIGPT. So we had to self host, self
27:53
manage everything. So we can go back to these
27:55
customers say, hey, your data is private, is in the boundaries of
27:58
our systems on the cloud. It's the same thing as, you
28:01
know, saving your data into a relational database.
28:04
Our AI will run within the ecosystem.
28:07
But then what it changed like a year, a year later, is that the
28:11
the same customers or most customers are asking for our AR
28:14
road map. They want to see where, where
28:17
are we adding AI, they want to even contribute and push that
28:20
road map towards what fits them best and what works for them.
28:25
At some point we had customers asking if they can bring their
28:28
own AI, like they have their own agents or they have their own AI
28:32
or they have their own enterprise accounts within these
28:36
vendors and they want to connect that.
28:38
Oh yeah. Which is very interesting.
28:40
And obviously something we're we're, I mean, I'm open to
28:44
consider. It's just, you know, how can you
28:45
make that system work where you don't control the model?
28:49
You don't control, you know, the everything about it is just, you
28:53
know, you throw the data and you wish it's going to work and you
28:56
optimize. How would you optimize for that?
28:58
You know, you know, prompt engineering and all the work you
29:00
do on a frequent basis, it's based on the idea that you know,
29:03
and control the models. If you don't anymore, that's
29:06
going to be tricky. So who is where does the
29:10
liability goes when it comes to the ownership and the issues of
29:14
the problems with it? So yeah, it shifted completely
29:17
from being very, I would say skeptical with AI and concerned
29:22
about the responsible AI and privacy to completely the
29:25
opposite, Like we want more AI and show us AI and we want AI to
29:30
be here and here and here because it's going to save time.
29:32
So all this leads back to that kind of planning phase where on
29:36
on a 2D kind of graph, you need to know what is the lowest kind
29:40
of effort techno, technically speaking, to build the AI
29:43
features or capabilities that will have the highest impact.
29:46
Start there and keep it small in terms of scope and execution.
29:50
So you don't want to spin up these huge projects and and lose
29:53
control in the middle and so. You're mapping out on that 2D
29:57
graph all the possible ways you can plug in AI.
30:01
You've heard from customers, oh, we want this feature, blah,
30:04
blah, blah. And then you're just kind of
30:08
throwing story points at how. Kind of voting, yeah, kind of
30:11
voting from, because at the end of the day, you and your team,
30:15
from an engineering perspective, an AI team will have to go and
30:17
build it, right? So you have to vote on, you
30:21
know, how much you think this is feasible.
30:24
There's a lot of research that is involved, the experimentation
30:27
that is involved. Again, this space is moving so
30:29
fast. So from an idea to turn that
30:31
idea into really an execution plan, it's going to it's going
30:34
to require you to vote and chime into, you know, evaluating these
30:40
requests. So, yeah, when you end up with a
30:42
view where you can have like a top right corner where you see
30:45
what is high impact, low effort, that's where you kind of go and,
30:49
and work and collaborate with the other teams on building.
30:53
And, and also sometimes you think about the the features of
30:56
capabilities that will enable more capabilities later in the
30:58
future. So this is where even my own
31:01
thinking about building AI has been shifting over time when I
31:04
was trying to do more vertical AI.
31:06
I'm going to do this AI capability for this domain.
31:10
Think about like goal settings. As an individual who's working
31:13
on a company, you want to build your own goals.
31:15
You know, your, your OK Rs, your whatever KP is you have to track
31:20
on a quarterly or yearly basis, but you don't want to build that
31:23
on a silo for you, just yourself.
31:25
You want to match this with your manager, your team, your
31:29
department, the OR the top company goals.
31:31
So building AI in any of this is, is for me, a vertical AI
31:35
because we're just taking goals as a domain and it kind of
31:39
empowering with, with AI. Think about doing this and other
31:43
domains like giving feedback, like doing conversations with
31:46
between managers or 360 reviews. And now you have AI running on
31:50
all of these systems. Now if you have the right
31:54
foundation and if you build AI as a platform instead of a
31:57
product, now you can build a horizontal AI which can go and
32:01
use all of these AI you build to build bigger, more complex, even
32:05
multi step AI like building programs.
32:08
You are you are someone who was promoted into a manager role.
32:12
How would you manage your team at the thirty 6090 days?
32:16
What meetings you need to set up what, what data you need to
32:19
collect about your the people who are reporting to you.
32:22
Now, if you had to do all of this without AI on your own,
32:26
it's going to take forever. It's going to take a long time.
32:28
But now with AI capabilities available in every one of these,
32:31
it's going to bring to the surface a lot of good insights
32:34
that another AI layer can really use to build the program for you
32:38
across all of these. And now go and maybe suggest,
32:42
OK, well, you need to set up goals this way for you and for
32:44
your team. I recommend like you have a
32:47
one-on-one on a biweekly and maybe you need to start
32:50
conversations with individuals. They report to you on 60 instead
32:54
of 30. And maybe you need to kind of
32:58
start to think about how would you develop skills for this team
33:03
based on what we know about your team, You have gaps and certain
33:06
things based on the goals you say.
33:07
Like again, I think of me of like managing a new team and
33:10
saying, trying to say, hey, like 6 months from now, we need to be
33:13
able to have more advanced agentic workflows and, and the
33:18
platform. But you know, AI would be
33:21
looking at all of the feedback data or all of the skills that
33:24
are in my team and figure out that I have gaps.
33:26
So it's going to ask me to either develop gaps or find
33:29
resources maybe from other teams to bring to this team so we can
33:32
achieve that goal. And this is like a deep research
33:35
report on the team type of thing.
33:38
So it's a, it's more about pulling using all of these AI
33:42
capabilities within these vertical stability AI.
33:45
So you can call deep research, but it's, it's not built that
33:49
way. It's more about, I would say,
33:51
workflow where you're pulling data from all of these systems
33:54
and then trying to summarize this and then create insights
33:58
and then go back to create actions within these domains.
34:01
And all of these verticalized ones you're trying to create in
34:06
a way that can be generalized. So you're using the same models,
34:09
it's just that the product itself is vertical.
34:13
Yeah, exactly. It's all using again, we're
34:15
still in the Mono model. Again, I think the future will
34:19
hold this capabilities of really tweaking every domain to the
34:25
right size model, the right size architecture behind, because not
34:29
every domain or feature require the same capabilities.
34:31
I talked about this earlier. Generating content like goals is
34:35
different from just polishing feedback or, you know, giving
34:39
you a draft where a conversation.
34:41
But all these verticalized pieces are then just bubbling up
34:47
the most important parts to the prompt, that is that horizontal
34:50
layer. Exactly, Yeah.
34:52
And you know what, another thing we have been also doing recently
34:55
which can lead to more tweaks and and explorations in the
35:00
infrastructure aspects or the ML OPS aspect is preprocessing
35:04
these verticals before you come and ask for a horizontal
35:07
feature. Oh nice.
35:09
Yeah. So, so if you if you can already
35:12
generate the summary of employees kind of goals or the
35:17
feedback they received over the last year or so, So when the
35:20
horizontal request come to build a plan or build the whole
35:24
program, this data is available. So you don't need to on runtime
35:29
run AI and try to summarize and do all of that.
35:31
You can have that data preprocessed where this works
35:34
really well with the ML OPS aspect.
35:36
These how can you do these batched work at A at a time
35:40
where your AI is not requested the most.
35:43
Going back to what I was talking about, the scheduled, you know,
35:47
kind of scaling, if you can figure out these times where
35:50
it's really low in usage and you have enough machines, you can
35:53
run these batch job at night or a time where it's cheap.
35:56
Yeah, it's like consolidate learnings and then boom, you're
35:59
on that job, you kick it off. Exactly.
36:01
Yeah. And you have some sort of a
36:02
queue of request, you know pre processing data.
36:05
But there's no, I mean it has a lower level of priority or
36:08
urgency. So it can be done whenever there
36:10
is a bandwidth to do. This now gives you that
36:14
framework. You hit on how you choose the
36:17
different pieces. Then you had that flywheel.
36:20
There was the next step of going and building it.
36:23
Building. Yeah, building is when you
36:25
decide what is the best model, where is the best architecture.
36:28
You know, the architect, the architecture on the on the
36:31
cloud, the cost and all of that. You play some bets, yeah, and
36:34
then you go and you optimize. That was the next.
36:35
And then you go and run it. So that's when you deploy it,
36:39
obviously you have different stages before it goes to
36:41
production, full exposure, you have it rolled out into
36:45
different stages and confirm that it's exactly what you want.
36:49
And then you go and optimize. Once it's again deployed to
36:52
production, you go and optimize and make sure, OK, well, this is
36:54
good. But there's nothing we built in
36:57
the last three years that was that we never went back in, in
37:01
optimized. So that's the beauty of this
37:03
kind of mindset of there is always opportunity again,
37:07
because in the last three years or two years, just the last
37:09
three years, so many things changed, so many new models came
37:14
up, so many new techniques to optimize the models came.
37:17
So every once in a while you go and and change and optimize.
37:20
And also internally, we had so many new requests to add to
37:23
these, again, support new languages, do it faster, do it
37:26
better, add more guardrails to safeguards, ensure that it
37:30
doesn't do this, it doesn't do that.
37:32
How would you also offline build an evaluation framework that
37:35
would go and ensure whatever changes you do to the prompts or
37:38
changing the models or the infrastructure performance and
37:42
accuracy and quality of these responses still meeting certain
37:45
level that you have set up. So optimizing, I mean if I had
37:50
to draw this flywheel like all the the first 3 would be small
37:55
parts of the circle. Optimization is at least half of
37:58
it. How do you know it's working?
38:00
How do you know that the customer is finding value in it?
38:04
Is it usage metrics? Is it?
38:06
Yeah, well, I mean it's usage metric, it's obviously another,
38:11
you know, we can also have reported data from the customer,
38:14
self reported data about their, you know, think about an MPs,
38:17
the score, right, Like their how they think of this feature is
38:21
changing the way they interact with the platform or even with
38:24
specific domains. You have also ways because when
38:29
we built all of this, we were focused on building responsible
38:31
AI before it, before anything, because we are an HR tech space,
38:35
very highly compliant, regulated, face sensitive data
38:39
all the time. Like I, I always get these
38:41
questions about like data classification.
38:43
I say everything is, is very sensitive.
38:45
So there is no, it's one class. So when you're trying to build
38:50
the responsible AI, some of these, the pillars of
38:53
responsible AI is transparency and explain ability.
38:57
So you need to be transparent about why AI did this, where
39:02
this summary come from, what are the sources of data that are
39:06
used to say that this person has this skill as an example and
39:10
then explain abilities. Sometimes when you do some of
39:13
the AI changes or generation, you explain why AI thought that
39:19
this is the best thing to do. So the think the way we think
39:24
about AI is not just a tool where you can put your brain on
39:27
the side and just use it. It's also something that will
39:31
train your your brain to do better next time to to grow as
39:36
an individual or as an employee and to develop skills as well.
39:39
So it's a very good way to learn.
39:42
So there's that. I mean, there's the reported
39:43
data. We have ways to explain to the
39:45
user how it works. We have also thumbs up, thumbs
39:48
down. Tell us if this doesn't meet
39:50
your, your needs or doesn't look good.
39:53
And then we also collect a lot of metrics and data that we have
39:56
internally to see how much the introduction of AI has changed
40:02
the way we people interact with the features and the, and the,
40:05
and the product. So understanding that is our.
40:09
Kind of promotional material is to go and tell current customers
40:13
and future customers, hey, using AI today, people are able to
40:18
like we are able to achieve 50% more goal, you know, metrics or
40:24
people are creating more, more conversations or given more
40:28
feedback because AI is helping a lot with that.
40:30
You have the author's block like you'd AI can build for you a
40:34
draft without you to have to think so much about how do you
40:37
even start when you're trying to build goals?
40:41
It's so hard for you to set two goals for the next quarter, the
40:44
next year on your own. AI can really create drafts of
40:48
goals for you that really, you know, based on the history of
40:51
all the goals you had, but also looking at your manager, your
40:54
team, your department and the vision of the company.
40:57
So again, if I had to do this on my own, it might take a few
41:00
days. Like we all been there and we
41:02
had we had, we were asked to kind of put together some
41:07
objectives and OK, Rs and then find out like, where should I
41:10
start? And you go and browse and look
41:12
at everyone else to just make some decisions.
41:14
Now AI gives you 8 gets you 80% there.
41:17
And all you need to do is like review, align, make sure you add
41:21
anything that was missing and creates this kind of hierarchy
41:25
of goals and objectives that before was really hard to
41:30
achieve. Are there any features that you
41:32
put into production and you thought they were going to be
41:35
successful? You were convinced that this is
41:38
going to add so much value to the end user's life.
41:44
We need to have it out there and for some reason or other it just
41:48
didn't work or it didn't work as you wanted.
41:50
Yeah, there's no, no direct path to success here.
41:54
You had to go through iterations and, and I think the most
41:57
successful leaders and businesses just are the ones who
42:01
had fewer, you know, scenarios like this.
42:04
So one feature I recall is when you help users rephrase feedback
42:11
they want to give to another person.
42:14
And as a user, you will go and type your feedback the way you
42:17
used to do it and use AI to just find any gaps in it, Polish it
42:20
and prove it. You might also go and give it
42:23
bullet points instead of really develop your own feedback
42:25
structure. And then AI would take that and
42:27
improve it for you. Umm, and this is early on when
42:31
you start building these kind of Polish, umm, writing assist
42:35
capabilities. The challenge with these is the
42:39
amount of things you need, the nuances and the language you use
42:43
across different languages. That's even another problem, but
42:46
the different nuances of how would you rephrase these and how
42:49
AI will take that and and change it where you know, if you don't
42:54
have the right setup for this, AI can take what you wrote and
42:57
completely write a different version that will change it
43:00
completely. Maybe we'll have the same
43:01
meaning, but completely in a different.
43:03
And then users. Way that doesn't look like you,
43:06
it's not your tone, it's not your language.
43:09
So another thing that can do again, hallucination is a big
43:12
thing. It can start going into, you
43:14
know, different directions. It can start, you know, if you,
43:19
if that message is a little bit long, it can start to lose some,
43:22
some facts and the conversation and you have to bring it back to
43:25
by by tweaking it again and again.
43:27
So I think this feature when we first initially started, got the
43:32
most feedback from the customers where it's, it's different.
43:35
It's not exactly what I what I need and I use it.
43:39
I try it and it gives me something that I'm not really
43:42
interested in using, so I drop it.
43:45
So usages may be high, by adoption is maybe low.
43:49
Yeah, value isn't there. Yeah.
43:50
So because because we never go and push anything that AI writes
43:54
on your behalf, you're still in control.
43:56
AI is going to give you suggestion that you need to
43:58
review and approve. So human in the loop is 100%
44:01
guaranteed here. So what we have been doing since
44:04
and that kind of optimizing is everything around guardrails and
44:08
safeguards and everything around adversarial testing.
44:12
Go go crazy and give it, you know, wrong feedbacks and see
44:16
how AI, because if you are, I mean, again, the whole idea here
44:19
is to help you as an individual give the best feedback possible
44:23
that is actionable and insightful.
44:25
But if you are an upset employee and you're trying to give a
44:28
feedback that is bad to someone else, AI should not empower that
44:32
and give you longer versions of that, make it even worse.
44:36
Yeah. So the guardrails what to have
44:39
had to have be set up in a way that is is like very dynamic.
44:46
It has to learn over time. We had to learn from this
44:48
feature, from the feedback to keep improving it.
44:51
So first time when you build this, you think, Oh my God, I'm
44:54
going to help so many users write great feedbacks, but then
44:58
the feedback is like, Oh well, every time I use it, it doesn't
45:01
look like a language I used or it got into talking about things
45:04
I didn't mean to say. So going back into that and
45:08
again, going going back to the basics for me every time we talk
45:11
about these is how you turn that into an AI platform, not just
45:15
tweaks to that feature or to that product.
45:18
How would do you put safeguards and guardrails as part of the
45:21
whole ecosystem where you can tune it for this feature but can
45:25
use it for other features as well and became part of the
45:27
whole ecosystem? Yeah.
45:29
So you correct it once and you don't have to go and correct it
45:33
again. Yeah, every time it comes.
45:34
Up and you have a structure to leverage this more and more in
45:38
your next product. So so that's again they're going
45:41
back to the flywheel when you go and plan maybe the next feature
45:45
after optimizing this one, you already use all the knowledge
45:47
you had and all the infrastructure you had.
45:50
So that's one of the pitfalls I see.
45:51
You know, when people build AI is you build these products in
45:55
silos and then you have to rebuild everything you have been
45:58
doing. So one of the learnings I've had
46:00
is how can we Anything we do is a platform is abstracted from
46:05
these specific use cases, but more used as an architecture
46:08
that can be spinned up for for any other thing.
46:12
So let's shift from talking about AI products that you're
46:17
creating for your company and your product to using AI
46:25
internally, primarily on the engineering team.
46:29
So agentic coding agents, agentic engineering, whatever
46:35
the term du jour is, how are you guys doing that?
46:38
How are you looking at the costs incurred there?
46:41
Because that can be chaos. I was just at an agent boot camp
46:46
a few weeks ago and folks were saying how they've created an
46:51
internal dashboard on who's using what coding tools and how
46:56
much they're spending. Not because they want to tell
46:59
someone don't spend, but they want to see like if somebody is
47:03
spending a lot of money, they should probably talk to them and
47:06
see what they're doing, how they're leveraging it so that
47:10
those people can then go and disseminate that information
47:14
across the organization and showcase like how they're a
47:17
power user. So the first thing I I
47:21
recognized is how many options are out there and how fast these
47:26
things are moving. And there's also a cultural
47:29
shift or maybe even a perception change.
47:32
Software engineers, when they first learned about Devon as an
47:34
example, the first AI engineer, we were all like, what?
47:38
No, no way. This is like a joke.
47:40
That's not going to work. I'm not saying it work, but what
47:44
I'm saying is like the perception early on was like I'm
47:47
not interested. This is not this is not for us.
47:51
They did over promise and under deliver.
47:53
Yeah, back in those days. Too, yeah, but the perception
47:55
from software engineers and technical people were like, OK,
47:58
this is not, there is no future where this will work.
48:02
And then over time again with cursor, with windsurf and with
48:06
all of these capabilities and IDE's and AI and then most
48:10
recently like more focused on coding agents and stuff like
48:13
that. Perception has been changing.
48:16
And for me, what I have been seeing is 22 curves, 1 is the
48:20
adoption growing for sure, like adoption of software engineers,
48:24
adopting AI technologies, growing to certain degrees where
48:27
today, I mean, I don't have exact data, but in internally in
48:32
my team, I think we're close to 100% if not 100%.
48:35
I don't, I don't, I don't think we have any engineer who's not
48:37
using AI. So I'm not going to assume it's
48:39
100%, but it's very high percentage.
48:42
And then the other thing is how that perception is changing.
48:45
And even in that curve of perception, there were 222 sub
48:49
lines I would say. And it depends on the seniority
48:53
level. So junior engineers or early
48:56
stage engineers have been more excited about AI.
49:00
Why? Because they can do more with
49:01
it. They can do things that used to
49:03
take years for them to learn and a lot of training and mentoring
49:07
to get there and exposure to bigger projects to do.
49:11
So now junior engineers capable of spinning up an
49:14
infrastructure, a full stack thing and, and, and look at all
49:18
of this and manage all of this without the proper training and
49:21
exposure for it. And on the other hand, the most
49:25
senior engineers, the architect level, they were very skeptical
49:28
to how AI will help them. They, they are good, they are
49:32
capable, they can do many things that others cannot do.
49:35
So thinking about the AI, they not only thought at the
49:39
beginning, AI will not change much in the way they work, but
49:43
also they were concerned about other people, especially the
49:47
less senior people, how they will use it.
49:51
And that created the gap between, again, the output of AI
49:55
versus, you know, the perception and, and how we deal with it.
49:58
So early on when we started really exploring AI, I created
50:03
an what I ended up calling the AI engineering lab.
50:06
So it's a space or it's a system where we can go all and explore
50:11
AI and with certain, you know, limits to budget, obviously.
50:17
And I mean, with the open source and alone, you have a lot of
50:20
exposure to a lot of good things.
50:22
But internally we adopted, you know, a few tools that we start
50:25
to, you know, offering seats to my team and, and go and explore.
50:30
So with these things, we, I mean with the AI engineering club, I
50:34
wanted everyone to go and explore AI without that kind of
50:38
predefined perception. I didn't want to dictate, I
50:40
didn't want to say anything. I don't want the to the senior
50:43
to influence the junior or vice versa.
50:45
I want everyone to go and explore and use it in very
50:48
limited scope to just ensure this is good.
50:51
We start with some projects that were not very top priority and
50:56
we had people create code, the news AI and go figure out these
51:00
tools and you know, bring GitHub Copilot with mixed with the
51:04
memory bank and is a very specific agents.
51:07
And then I had my senior engineers who were not like the
51:12
most active in using AI on a daily basis to kind of take a
51:16
look and review this code. And I intentionally wanted them
51:20
to give like a very honest feedback that is, that doesn't
51:25
know necessarily if this is done by AI or helped with the help of
51:28
the AI or not. And then definitely immediately
51:31
they start seeing the patterns, they start to see that the
51:34
quality is not great, the design patterns are not.
51:37
Respected. It has a smell to it it.
51:38
Has a smell and I start having feedback that hey, now that we
51:42
opened explorations to AI, yes, it's off projects, you know, all
51:46
the side projects, not something that is critical path.
51:50
But we are seeing a pattern of people using AI just to do more
51:55
without taking that time saving to review and and fix things and
52:00
remove, you know anything that is not in our best practices.
52:05
Sometimes AI would come up with its own design pattern.
52:07
While we have our own we have components components that we
52:11
use that AI is not respecting and building its own component
52:14
for UI or even for a back end. So that that is where the senior
52:18
engineers got more and more involved into the process
52:21
saying, OK, OK, definitely we have this problem.
52:24
So I want the other people to take more time to review.
52:27
So if AI is saving you 80% time or 70, use twenty, 3040% of that
52:32
time to review. So defining the the new SDLC has
52:39
been a process that we all contributed to.
52:42
It was not dictated. It was not based on a research.
52:45
It was more like a a collaboration within the
52:47
engineering team where we ended up fine tuning overtime what
52:50
should be done. And the way we ended up adopting
52:53
is like you use AI, like we all agree now AI is all over the
52:57
place, but anytime you any time saving, you do use that to
53:02
review AI. The output of your work with AI
53:06
should be exactly the same as the output of your work without
53:09
AI, meaning you need to own the code, you need to understand the
53:12
code, you need to really make sure it's adapted to our design
53:16
patterns and our components. Otherwise you're you're saving
53:20
time but also creating a lot of problems that will either blow
53:24
up now or later. And did you take certain things
53:27
from the senior engineers and immortalize them in like rules
53:33
or ways that you can make the coding agents respect the
53:39
components or the design patterns that are common
53:42
practices within the org? Yeah.
53:44
So that's where again, I, I, this AI engineering lab started
53:48
the whole conversation where, and this is where I told you
53:51
about the graphs, the lines earlier where I think it started
53:54
more with, with the mid level engineers pushing, pushing to
54:00
the adoption of AI while the most senior engineers were like,
54:04
OK, well, let's take our time. Let's be careful here.
54:06
This can create a lot of trash. Chaos in the data.
54:09
Yes and chaos. So now they have everyone's
54:12
attention. I was like OK, well tell me more
54:14
about this. And I had more time from the
54:16
most senior engineers to go and review to go get themselves
54:19
also, you know, spin up all of these tools and figure out the
54:23
best. And what we ended up adopting is
54:27
again, is leveraging a lot of memory aspects, a lot of like
54:30
instruction skills based where these agents, we go and build
54:34
something that we all agree on, we vote on.
54:36
It's like building code. Now, again, building these
54:38
skills and these instructions, it's like writing your code.
54:42
So you define what is best, you point the AI to the right kind
54:45
of design patterns, to the right structures that needs to follow.
54:50
And then over time, the output of AI has been significantly
54:53
changing, impacting the performance and velocity of
54:56
everyone in the team, but especially the mid level to, I
54:59
mean, junior to mid level. So that's, that's phase one of
55:03
AI engineering club. Phase two now is exploration
55:07
beyond what we kind of started with, because I, as I told you,
55:10
we started with junior, mid level, pushing more AI code and
55:13
then senior like, OK, well, let me, let me give you my attention
55:17
here for a second. Let's improve this.
55:18
Now everyone is using that that level we got to, but now the
55:23
push is coming from more from the senior, the most architect,
55:26
you know, kind of level people where they're going, exploring
55:29
everything under the sun, you know, cloud code and all of
55:32
these things and mostly say allies where it's changing the
55:37
way they work. It's really a whole new way of
55:42
writing code by managing all of these agents.
55:45
It's not just one or two. It's like so many agents today
55:49
saving a lot of time for the most senior engineers, which
55:52
again, at the beginning they were very skeptical to the
55:54
impact of this. And the same people that early
55:57
on said, I'm not sure I'm concerned are now like, Oh my
56:01
God, I'm like a, I'm like a God. They're running eight different
56:05
agents in parallel. Yeah, they spend, I mean,
56:07
they're using A at some point they, they are outpacing our
56:12
standards and our mechanics because again, we are, we are,
56:15
we are, we have controls in, in, in the system.
56:18
So they're spinning up their own personal machines, their own
56:21
personal subscriptions to new systems because they think it's,
56:25
it's going to take time for us to go and review security and
56:28
legal. And sure, like this is good.
56:30
Like they are just experimenting outside of the regular kind of
56:34
work because they are so excited about this.
56:37
So at the end of the day, again, this AI engineering lab started
56:41
started to just open the conversation and then kind of a
56:44
took off from there. They caught the bug.
56:46
Another thing we did is, you know, when we spin up this whole
56:50
thing, we did this survey to just have people report on AI
56:54
impact on engineering. Because I don't want to trust, I
56:57
don't want to introduce any DX tools like a like tracking, you
57:01
know, Big Brother way of like, are you using AI?
57:05
How much you're spending all of that?
57:06
I mean, I have the spending anything that is under the
57:08
enterprise accounts. We, I have this, this spending
57:10
and we are keeping that under control.
57:12
But the usage, the way you use it, I don't want to control any
57:16
of that yet. And I wanted the people to just
57:19
go on and report on the, on the data.
57:22
So their own thinking about how AI is helping them.
57:26
Do you think AI is helping you get better software engineer or
57:29
get more as a software engineer? Because there's a difference
57:32
here. If you are a mid level engineer
57:35
using AI, is that going to get you to senior earlier or it's
57:39
going to delay that? Because you don't have to think
57:41
about the problems. AI is thinking about the
57:43
problems you're just delivering. So there's a difference between
57:47
using AI to grow and be better or using AI to just do more.
57:51
Another thing I talked about earlier is like, is AI helping
57:54
you to be a better engineer on your own or a better engineer
57:58
within a team? And is it making your team
58:00
better? Is it making my whole
58:02
engineering team better? So that's these because there's
58:05
difference there and we'd, if we didn't do this in A and
58:09
distributed way where everyone was contributing and then
58:12
feedback came in and then we, we started like with so many
58:16
options. And then we, we, we got to place
58:18
where we have very few option that a lot of people agree on.
58:22
If we didn't do that, the risk could have been off creating all
58:25
of these silos of AI development where everyone is using their
58:30
own tool, everyone is spinning up something, or we could have
58:33
ended up with something that is dictated go news XYZ at IDE.
58:38
With these models, your subscription is limited to this
58:43
and that's it. And you don't give freedom to
58:45
the software engineers and they would be just locked into that
58:48
system. With the AI inside of your
58:52
product, you took a very opinionated approach and you're
58:56
optimizing to the Max and you're self hosting.
59:00
You're doing all these things that are very hard engineering
59:04
wise with the AI being used on the engineering team.
59:09
You're not really doing any of that.
59:10
Yeah. And so do you foresee a world
59:13
where eventually you're going to get to running your own Quinn or
59:17
Kimi models for the teams, you know, these coding agents that
59:21
are self hosted and you can completely rely on because of
59:26
it's cheaper and it's better in that regard?
59:30
Well, that's that conversation is happening today.
59:33
I've had people reaching out saying, hey, can we, now that we
59:36
are deep into AI development and you know, AAI coding agents, can
59:41
we spin up versions of our AI platform to support these
59:46
efforts? Yeah, because you were just
59:47
talking about how you want to create a platform.
59:50
So you've already got the primitives there, right?
59:53
Yeah. Well, well, that's the
59:56
conversation we're having as of now.
59:58
The challenge is how to adapt that platform to be a
1:00:02
development platform. So again, you need to go and
1:00:05
pick the right models. You need to go and spin up the
1:00:08
right way to support more and more agentic capabilities.
1:00:13
We have been exploring many things like MCP as an example,
1:00:17
to just spin up not only the product but also development.
1:00:20
So there is a place where these two things will overlap a little
1:00:24
bit. But that's where I need to go
1:00:26
back to the flywheel and go and plan for it, go and build it and
1:00:30
run it and then optimize. But definitely that's, that's
1:00:33
the direction. And I as of today, I have
1:00:35
conversations happening about leveraging our own platform to
1:00:38
do our own AI coding. And how are you looking at the
1:00:41
cost for the AI coding? Yeah.
1:00:45
What cost is so far under control, which is which is
1:00:49
surprising. We, we spin up these, I mean we
1:00:53
use different tools that the good thing about them is you
1:00:56
have visibility into the cost and you can set up quotas.
1:01:00
And we were for a long time in this kind of experimentation
1:01:03
stage where we use different models.
1:01:07
And then even if these models were out of quota, we go to the
1:01:09
next one and we try and we keep iterating.
1:01:12
Now we are getting a little bit locked in into what works the
1:01:14
best and this is where we are investing into enterprise
1:01:18
relationship with these vendors and solutions and just adopt
1:01:22
them at the bigger scale. But you know, for a long time I
1:01:26
had my team exposed to different options, either, you know, led
1:01:31
by by the collaboration I have with our financial team just to
1:01:35
get the number of seats I need enterprise accounts or just by
1:01:39
exploring open source as well. I think there is a lot of
1:01:42
opportunities with open source. And that's where the
1:01:44
conversation was leading to this kind of leveraging our AI,
1:01:48
because if we have the infrastructure, you can spin it
1:01:50
in a different environment, but now you have less control on the
1:01:55
the sensitivity of the data in terms of like HR tag the
1:01:59
specific models. One of the examples I have here
1:02:01
is as an example in our platform we cannot use Chinese originated
1:02:07
models as an example like when and DeepSeek, but for
1:02:10
development maybe we can. So that can create different
1:02:13
different options for us less controls because we're not
1:02:16
serving HR tech anymore, we're serving software engineering.
1:02:19
And for that different requirements, different, you
1:02:21
know, aspects that we can leverage our platform for.
1:02:25
And yeah, that's going to impact significantly cost if, if, if we
1:02:30
do follow that. Path and you can't use Chinese
1:02:32
models because it's the stance that your company took or it's a
1:02:36
compliance thing. It's, it's, it's a compli.
1:02:40
It's a, it's a customer's request, yeah.
1:02:43
We were a few people say that. Yeah.
1:02:45
So we, I mean, if we work, if you work with a like a global
1:02:49
enterprise, customers cross globe.
1:02:53
Again, this takes you back to the initial concerns we had with
1:02:56
even open AI in, in Anthropic early on because of how they,
1:03:02
you know, customers don't know how, what's the process to, to
1:03:05
train the data. So that same thing is happening
1:03:08
today with the Chinese based models where we don't know how
1:03:11
the data is being trained. We don't as customers.
1:03:14
For me personally, I mean, I know how much these keep these
1:03:18
models are capable, right? I know what they're capable of,
1:03:21
but that the missing piece and and AI sometimes is that, and I
1:03:26
talk about this a lot where it's all about, you know, the data at
1:03:30
the end of the day, the data that which which we forget to
1:03:34
mention talk about the most is the data that was used to train
1:03:37
the foundation model. Like we talked about the data
1:03:40
for fine tuning, we talked about the data for the prompts and in
1:03:43
context learning, but we missed that these models were trained
1:03:46
with a certain corpus of data. So what is that data?
1:03:49
What biases in this data? So that kind of unknown, he's a
1:03:53
no go for customers. Yeah.
1:03:55
And that's, that's a loan. He's a he's a reason we have
1:03:57
that same no go for open AI. So I can really imagine this
1:04:00
were Chinese, but I think perception will change over time
1:04:03
similar to that one and there will be more competition, you
1:04:06
know, regardless of the source. Yeah, there.
1:04:10
Well I know what is that company reflection.
1:04:14
They got paid a bunch of money by NVIDIA just to create an open
1:04:17
source lab in the US to rival the deep sea models because
1:04:23
people are asking for it. So customers are fine with the
1:04:29
same capabilities model to DeepSeek to coin if it's trained
1:04:32
in the US, if it has certain visibility into the training
1:04:35
data. So again, it's it's a more like
1:04:37
a politics and legal aspects to things, but it's not I won't say
1:04:41
it stands from the company. It's more about the space we're
1:04:44
in that is very highly regulated compliancy and you know, the
1:04:49
concerns about training and and bias in general.
1:04:52
And there is a lot of also conversations to have with
1:04:54
customers to explain how things work and how AI work and the
1:04:58
difference between using an open source model hosted self hosted
1:05:02
versus using their model through an API.
1:05:05
So sending them the data, yeah, explaining that, but the concern
1:05:08
is more about the bias that is already in the models.