Latest / Agentic Conversations (formally mlops.community) / AWS Tranium and Inferentia // Kamran Khan and Matthew McClean // #238

Transcript

AWS Tranium and Inferentia // Kamran Khan and Matthew McClean // #238 · Agentic Conversations (formally mlops.community)

0:01
My name is Matt McLean. So I lead the Annapurna customer
0:04
engineering team. So my team essentially help
0:07
customers adopt, adjust training and friendship for their
0:11
workloads. So yeah, so I work for Adams.
0:13
I've actually been working for a jobs for more than eleven years
0:16
now, and I've been working in the Annapurna and Address
0:19
training team for a couple of years.
0:21
And how I take my coffee? Well, my wife's Italian, so I
0:25
prefer to take it in Italy. And yeah, and a a cafe place
0:28
with Italian coffee tower courtroom.
0:31
Hello, my name is Kamran Khan. I lead our business development
0:34
and go to market teams around in French and Tranium and I like
0:39
all types of coffee. So generally I'll take A at home
0:44
I'll make myself a flat white and on the go I like a variety
0:48
of kind of light to medium rows and drip coffees.
0:53
Welcome back to the ML OPS Community Podcast.
0:56
I am your host Dimitrios, and we're back with another banger.
1:01
Today we're talking to Matt and Cameron about Inferencing and
1:05
Tranium, these 280 AW services that were demystified for me.
1:11
Over the next 40 minutes, I got the deep dive on what exactly
1:17
they're good for, what they're not so good for, how they work,
1:20
what they work with. If you're using Sage Maker, do
1:24
they work with that? If you're using Bedrock, how can
1:26
you utilise them? Why would I even utilise them
1:28
instead of the common GPUs and what are their strengths?
1:34
They really breakdown why they created both of these, which are
1:38
basically they said, you know what, we want a chip that is
1:42
tailor made for deep learning workloads.
1:46
And so you know how Amazon and AWS like to do it.
1:50
They just went out there and they they made it happen.
1:52
They turned it into a reality. We talked with Cameron and Matt
1:55
all about it. If you like this episode, as
1:58
always, share it with a friend. We would love to hear from you
2:02
on the Spotify feedback section. That's always great.
2:07
I get a nice little chuckle when people send in funny comments.
2:11
I read every one of them and let's get into it.
2:14
I'll see you on the other side with Matt and Camera.
2:22
All right, so Matt, I gotta call it out first.
2:25
You were in the community in 2021.
2:28
I was just looking at our conversations in Slack from
2:32
three years ago. It's awesome to finally get to
2:35
have you here on a podcast, but it's almost like we get a 2 for
2:38
1:00 today with Cameron. Two we want to talk all about
2:43
AWS is Influencia and Tranium. It's probably a good place to
2:47
start with just giving us a rundown on what it is and why AW
2:53
has invested in something like this.
2:56
Yeah, absolutely. So we've been at AWS been
2:59
investing in different types of accelerators and use cases for
3:03
customers for a very long time, right.
3:06
And so as the AI revolution has really taken hold, and I would
3:10
say like, you know, I think the, the interesting part is right,
3:14
like most consumers or most people are really picked up on
3:17
AI with the introduction of chat TPT last year, right?
3:21
Where they're like, oh man, look what I can do for me.
3:23
Look what these large language models can do for me.
3:26
But I think, you know, the US in this community have been seeing
3:29
this trend for the last 10-15 years, really since Resnet
3:33
really started showing the potential of what models can do
3:38
and how they can kind of start providing, you know, more
3:40
superhuman responses to simple applications like computer
3:44
vision and classification, but now with language, with video,
3:48
with images and so much more. So because of this, you know,
3:53
AWS has been looking at this space and like, well, how can we
3:56
offer customers more choice, higher performance, lower cost,
4:01
of course, and make it easier and more accessible to more
4:05
users and developers and customers ultimately, right?
4:08
And so they started investing in our own generation or purpose
4:14
built accelerators, our AI chips called Influencia and Tranium.
4:18
And we launched our first Infancia AI chips back in 2019.
4:23
And in 2019, it was inferential 1 focused on inference
4:26
acceleration and it was really good for CNN applications like
4:30
Yolos and and Residnets and image classifiers.
4:33
And then in 20 years time is flying right.
4:37
In 2022, we introduced Tranium and Tranium really was designed
4:44
as the successor to infrared state in some regards.
4:47
It was our second ML chip. This really brought a lot of
4:50
modernizations and really focused on LM's and generative
4:54
AI and how to accelerate and distribute large, large models
4:57
across multiple accelerators and make it very efficient and cost
5:00
efficient as well. And then last year, we
5:03
introduced Inferential 2, the successor to Inferential O.
5:07
Now we're building kind of this family of accelerators basically
5:11
to provide users choice, right? So give them opportunity to get
5:15
higher performance, lower cost, make it easier and more
5:18
accessible for more users. And we want to continue to
5:22
invest in these types of accelerators, right?
5:24
So training them two was announced at last reinvent by
5:27
Adam and it's going to be available later this year.
5:31
But it's going to come with 4X higher performance than tier in
5:34
one. And we're pretty excited about
5:36
what that's going to open up and what the possibilities that's
5:38
going to open up for a lot of users as well.
5:41
So I, I love that you're talking about the choice and bringing
5:45
more choices cause Amazon is kind of known for that,
5:49
especially at AWS, they're known for all the different choices.
5:52
And I want this conversation to really be around like, when
5:57
would we use it? Why would we use it?
5:59
And what the strong points are, the weak points and whatnot.
6:04
Of course, the first question that comes to my mind really is
6:08
about how does this compare to a GPU?
6:13
How is it different? What are things?
6:15
How does it match up right? Like and what GPUs, what should
6:19
we expect? There's like a lot of unanswered
6:22
questions that I can imagine. People are sitting at home or in
6:26
their cars listening and thinking like, OK, cool, so
6:30
train him and Influencia. It's like GPUs but not.
6:35
Yeah. So we have designed training and
6:39
inference are really to be a super specialised accelerator
6:44
for deep learning applications. So GPUs originally were there
6:50
are graphical processing unit, right.
6:52
So they were never designed for deep learning workloads.
6:55
But obviously it's a it's a great fit, but we decided to
6:59
desire something really specific to deep learning.
7:02
OI guess the key difference is, you know, comparing a GPU and
7:07
our hardware was that GPU. You have a lot of these
7:10
streaming multiprocessor, lots of thousands of cores and and
7:14
tensor cores going essentially, you know, what is a deep
7:17
learning application? Why do you need an accelerator?
7:20
It's primarily to, I would say a couple of cool things.
7:23
One, you need to accelerate matrix multiplications, right,
7:26
the linear algebra, you know, calculations so that and that
7:30
kind of forms think more than 90% of the like the operations
7:34
or the T flops, right. And any you know, this is for
7:37
both training and inference. So you need something because
7:40
our standards, you know, CPU is just not fast enough.
7:43
So GPU is good for that. And what we provide essentially
7:46
in our hardware is what we call a tensor engine.
7:49
So it's actually a systolic array that can really accelerate
7:53
these matrix multiplications. So it can do for example, a 128
7:57
by 512 in a single clock cycle. So really and it really is
8:02
designed for an accelerated those matrix multiplications.
8:06
The other part of accelerators that is needed is really high
8:10
bandwidth for memory, right? And in this respect, they're
8:13
quite similar. So we also have HPM, right?
8:15
And, and it's really designed to, you know, take the weights,
8:19
gradients and so forth, you know, that's stored and then,
8:22
you know, really quickly moved them into the cache, move it
8:27
into the accelerators to do the computation.
8:29
So that's kind of the the other key part.
8:32
So yeah. So the main difference I would
8:33
say is though the way computation is done.
8:36
OK. And that's through the Tensor
8:37
engine. I like that, and I can imagine
8:41
people are going to want to know like actual numbers and
8:45
benchmarks, and so we'll leave a link to a blog post.
8:49
But do you have any numbers off the top of your head?
8:52
Yeah, some, some numbers up. So of course, I think the best
8:56
place is gonna be our documentation.
8:59
We publish actually a range of different benchmarks and kind of
9:02
like hardware specs as well, right.
9:04
So you could learn exactly what the Tflops are.
9:07
But for training applications, we kind of like to think about
9:09
what is the effective Tflops you can achieve while training, you
9:13
know, large language models for example, right.
9:15
And so we like to kind of look at what is our effective Tflops
9:22
per dollar. So when we look at that and we
9:24
compare it to, you know traditional accelerators on the
9:28
market on AWS today, we're able to lower the cost to train
9:31
models by up to 46% using tier and one even against kind of
9:36
like the more common or you know, like latest generation of
9:40
other accelerators available on the AWS.
9:43
So it's very competitive. And at the same time on the
9:46
inference side, we see a very similar story as well, right.
9:49
So with the kind of conscious decisions we made in our
9:53
hardware really specialising on just these, you know, machine
9:58
learning workloads, rather than trying to be general purpose,
10:00
like for example, we get this question quite a bit is can I
10:03
run crypto algorithms on inference and training or can I
10:06
do like graphic simulation on inferential training?
10:10
Going back to your earlier point of like, you know, like, I think
10:13
it's like a GPU. So can I run other workloads
10:17
that require that level of horsepower?
10:19
And, and the answer is no, we, we can't actually run crypto
10:23
algorithms. We can't run, you know,
10:25
simulation workloads and not that that hardware necessarily
10:30
can't support it is like we haven't invested in the software
10:32
stack to enable them because we want to be specialised, we wanna
10:35
be focused and that gives us the ability to really hyper tune
10:40
lower cost to train, you know, to 50% lower cost to training
10:43
models. And at the same time when you're
10:45
deploying models get higher performance and reducing your
10:48
deployment cost by, you know, up to 40% as well using inferential
10:51
too. And I like this idea of,
10:54
alright, you're trying to make sure that you're looking at the
10:59
price of training or the price of inference and you're seeing
11:04
that it's going down like 46%, which is not a small number.
11:08
And I like that number when I'm especially if I'm thinking about
11:12
spending a lot of cash. If it's 46% less, that's a good
11:17
chunk of change right there. And the next obvious question I
11:22
think I have just as someone who is not super familiar with
11:26
training or influencia is do I still use CUDA?
11:31
Like what do I work with inference and training mode?
11:35
Yeah. So we don't support KUDA.
11:37
So KUDA is you know, a, a library very specific to GP's
11:41
and very specific NVIDIA GPUs. So what we have instead is our
11:46
own, we call it SDK, it's the Neuron SDK.
11:49
And So what that comprises of is essentially of kind of think of
11:52
it three blocks. So we have the framework
11:56
integration. So for example, we integrate, we
11:59
work with Pytorch, Tensorflow typically for inference and
12:03
we're very soon going to have Jack support.
12:06
So basically the the first part is to tie in.
12:08
So basically what we want to do is allow our users to stay in
12:13
the e-mail framework that they prefer, right?
12:15
Minimise the sort of code changes they need to make to use
12:18
our, our purpose built hardware. So that's kind of the first part
12:22
of the SDK. Then we have the compiler
12:23
itself. So our compiler stack is based
12:26
on XLA. XLA was originally part of the
12:30
Google Tensor Flow project, but is now, and we are one of the
12:32
founding members of the Open XLA initiative.
12:36
It's an open source initiative. So there we work with Google, we
12:40
work. With.
12:41
Others like Meta, Invidia, Apple, other companies in terms
12:45
of evolving the XLA standard. So that kind of defines the all
12:49
the operators and that kind of get lowered in the in the in the
12:54
in compilation plug and uses an intermediate representation
12:59
called stable obituary to define all those operators.
13:02
So our compiler uses hello and then it compiles essentially the
13:06
computational graphs forward backward pass optimization step
13:11
all into low level machine code, right?
13:13
And we call that the neuron executable file for in there.
13:16
So basically that runs the specific commands on the
13:19
different engines are now an accelerator and that then these
13:23
nips get run on the accelerator by a runtime.
13:25
So the runtime executes these compiled artefacts and also does
13:29
a lot of the networking, right? So when you're doing a
13:31
distributed inference or training job, then you have all
13:35
these collective communications, right?
13:37
You need to communicate between the different accelerators.
13:40
So the runtime also handles that.
13:42
So with all these kind of components, you kind of get into
13:45
end coverage of our software stacks and able to run your
13:48
Pytorch, your your code on our accelerator.
13:53
So if I'm understanding this correctly, you're really trying
13:55
to allow people to use it Torch or Tensorflow or whatever
14:00
they're used to using. But then and if you want, you
14:05
can just stop there. But if you want to go deeper,
14:08
you can through neuron. Yeah, you can.
14:11
There's a few different ways. So for example, if you know,
14:15
most users would just use, you know, standard Pytorch and then,
14:18
you know, compiled their model for Pytorch.
14:20
We use the XLA version of Pytorch, right?
14:24
That is also used by Google when running on Tpus.
14:28
So most users can just stay writing their code on Pytorch
14:31
and work fine. If they want to have more
14:33
control. Then there's a couple of
14:36
different ways that people can, you know, get better
14:38
performance. Or, for example, if there's a
14:40
certain operator that we don't support and they want to
14:43
implement that, then they can actually write their own custom
14:46
C operators. So one of the engines in our
14:48
accelerator is a signed engine. And so you can write 6 plus plus
14:53
code to execute custom operator, say Pytorch operator, and that
14:57
will run on the accelerator. In a couple of months, we're
15:02
gonna have a, a new kernel interface.
15:05
It's called Neuron kernel interface or Nikki for short.
15:08
And this will provide a if you're familiar with Triton.
15:11
So it's many customers that want to further optimise performance
15:16
and they want to do kind of what kind of tile level computations,
15:21
right? So we're, you know, reduce the
15:23
amount of kind of memory transfers between HBM and SRAM.
15:27
So if you're familiar with, for example, the Flash tension
15:30
algorithm, right, that is the core, you know nice how flesh
15:33
attention works, it's operating in ISRAM rather than going back
15:37
and forth to HPM. So you can do very implement
15:39
algorithms like Flash Attention like Mumba using Nikki that will
15:43
be launching very soon. And there's another way that
15:46
customers can get further performance have more control
15:49
over how the code is run on our accelerators.
15:53
And is it only like do they have to go together?
15:59
Are they married at the OR joined at the hip type thing?
16:02
Or if I use Tranium, then I can use whatever else I want and I
16:07
don't necessarily need to be stuck in with Influencia.
16:10
Yeah, that's actually a really important one because we get
16:13
that question all the time, right.
16:15
If you utilise or let's say you're starting with training,
16:19
you train your model and fine tune your models.
16:22
You know, starting from let's say Metas Lama three model,
16:25
which is, you know, a great choice to start and then you
16:29
fine tune it. It's working great for your
16:31
application. Are you locked in?
16:33
And the answer is no, are kind of philosophy is trying to be
16:38
part of the heterogeneous world. I think we know, you know,
16:42
working on and building with inferential and training and the
16:44
software stacks around it that users and developers are going
16:49
to be pulling and, you know, mashing together different
16:53
libraries and environments that will be best for them and maybe
16:57
not for everyone, but they'll be best for them.
17:00
And so the idea is we want to work within that world.
17:03
We know it's going to be a heterogeneous compute
17:05
environment where you're going to be utilising CPUs for certain
17:08
tasks, GPUs for certain tasks and in French and training them
17:12
for certain tasks. So when you're training a model
17:14
with training them, you could deploy that model or I should
17:17
say that the entry and exit, let's say you're fine tuning
17:20
that model is going to be starting from a checkpoint,
17:22
right? You could pick up metas,
17:24
checkpoints off Hugging Face, for example, fine tune it for
17:27
your with your datasets for your application space while
17:30
utilising training. The output is just going to be
17:33
another checkpoint that's defined by Pytorch.
17:36
And then you can then deploy that anywhere in French is a
17:39
great option, but you could pick U, you know, any GPU or embed
17:43
that model down. If it's, you know, for like in,
17:45
you know, on Prem solution or like an embedded device, you
17:49
know, there's a lot of wearables and a lot of different versions
17:53
now with like we're seeing a lot of the AI wearables, right?
17:56
Like with what humane is doing with what other solutions are
18:00
doing. And then and then vice versa as
18:03
well, right? So you could take models that
18:04
are pre trained or fine-tuned on Tranium and then deploy them to
18:09
in friendship seamlessly as well, right?
18:11
Just kind of taking those checkpoints.
18:14
Umm, so basically that's just one part of the whole workflow,
18:19
right? Like if you're trying to go and
18:22
do something with deep learning these days or AI machine
18:27
learning, whatever you want to call it, you've got all the
18:31
other pieces around it. And I instantly think about how
18:36
does this work with Sage Maker? So that's probably where we
18:40
could start. But then I want to know like,
18:42
OK, So what if I'm running Kubernetes and what if I have
18:45
Ray or what? Like how does this fit with all
18:49
of those different pieces of the puzzle?
18:51
And so with Sage Maker, that's probably the easiest one to
18:55
start with because I think a lot of people in the community and
19:00
listening are using Sage Maker. And then maybe we can go to
19:04
Bedrock and we can go to just EC2 or whatever it may be.
19:09
Yeah. I think it always starts with
19:11
the team right, responsible for, you know, either training or
19:15
deploying the models, right. And what is there for example
19:19
level of competency, what do they know today?
19:22
What is their kind of maybe company policy or standard, you
19:28
know, that they have to adopt because, you know, a lot of
19:31
companies will standardise on a certain technology platform and,
19:35
and kind of seems a little bit, you know, constrained to adopt
19:38
to that policy. And it's always I, I see it as a
19:41
spectrum of ease of use and control, right?
19:45
So taking into the bill as a team, taking the sort of the
19:49
spectrum easy of control is kind of where you kind of land in
19:51
terms of what is the best choice.
19:53
Obviously Sage Maker is an excellent, you know, the best
19:56
service and offers the full lifecycle management and it has
20:01
full support for infringing training.
20:02
So for example, there's kind of two options.
20:05
Actually on the training side, we have the kind of the standard
20:09
sort of ephemeral training jobs that made many of the listeners
20:13
are familiar with. You know, if you want to, you
20:15
know, using the Sage Maker SDK or through Studio, you can like
20:18
launch a training job, it'll spin up the instances in the
20:21
background and kind of manage infrastructure.
20:23
So that's kind of one option. We also have support for a new
20:27
service that was launched end of last year, Hyper Pod, Sage
20:30
Maker, Hyper Pod. So if customers are familiar
20:33
with kind of a slum interface and more of a fixed cluster,
20:37
right, they maybe want to have many different jobs or one long
20:40
running say training jobs. So they don't want them to be
20:42
ephemeral, they want to sort of have dedicated cluster.
20:45
Then hypothesise is a nice solution there and it handles as
20:48
well all of the kind of the failover managing essentially
20:51
the infrastructure when you have a hardware should replace the
20:54
hardware for you. O those kind of two, two options
20:58
in Sage Maker. And then on the inference side,
21:00
but you also have the endpoint support.
21:02
So you can deploy to either in French or you can even deploy to
21:06
training because training has a few more accelerators, more HBM
21:10
memory. So for large model, we'll save
21:12
that LAMA 370 be sometimes the train makes a better instance
21:18
choice for inference for doing your inference of those models.
21:21
So that's also supported in the Sage Maker endpoint service.
21:26
Oh, interesting. Yeah, that that is another piece
21:29
to it. Like Influencia basically can
21:31
take whatever you throw at it. Yes, Remember I can explain the
21:35
difference between infringing Tranium.
21:38
So essentially at the accelerator level, it's
21:41
basically the thing basically the same they have for the each
21:46
accelerator chip has two neuron cores.
21:48
So you can kind of think of a neuron core a little bit like a
21:51
GPU. So an inference, so inference
21:53
here is is essentially a, it can be deployed in more regions, it
21:58
has lower power requirements because it has less chips.
22:00
So in training we have 16 accelerators, whereas in in
22:04
friendship we have 12. Also the network connectivity
22:08
between the instances. So we're using ear networking
22:11
between the instances. So in training we have up to
22:14
1600 gigabits per second network connectivity.
22:17
Whereas an infringer we don't need, you know, typically when
22:19
you're doing inference, you don't need this high speed, high
22:22
bandwidth networking connectivity between the
22:24
instances. So it's only 100.
22:27
So yeah, so infringe share is also cheaper instance option,
22:32
but you can actually train on infringer, which is interesting.
22:35
And you can actually do inference on training.
22:37
So yeah, I know it's a little bit confusing, but they're they
22:40
hold some. Just to make it easy for
22:42
everybody. But it are there people doing a
22:46
lot of that? Like just because you can
22:48
doesn't mean you should, right? Are there people?
22:50
That are doing it. So there are so as I mentioned,
22:53
so for customers who want to deploy large alms like Llama 370
22:59
B. So when you getting into sort of
23:02
towards the 100 billion parameter models, actually
23:04
training is actually a better a better choice.
23:07
OK, OK, fascinating to think about that.
23:10
Now what about with just like an EC2 instance?
23:14
Yeah. So on EC2 there's many different
23:17
options. So we have a lot of customers
23:19
who have kind of standardised on using, for example, Kubernetes
23:23
right as their platform for doing either inference or
23:26
training. So we have full support.
23:28
We've even developed a Kubernetes plugins which will
23:32
help kind of, you know, with the resource management or for
23:35
example, your we call the neuron devices.
23:38
So for example, when you configure your Kubernetes pilot,
23:41
you can say, OK, I want to have four on your own devices.
23:44
So it'll be actually the the we have a plug in that will help
23:47
manage the, you know, reserving and allocating those resources.
23:52
And so that's a very popular option.
23:55
We have customers using parallel cluster.
23:57
So parallel cluster essentially is a for customers that like
24:01
using a slum interface. It's an open source offering
24:04
targeted kind of HPC and machine learning workloads, as I said
24:07
through a slim interface. So that is also a popular option
24:12
for training, training large models.
24:14
And then we have some good collaboration with partners.
24:17
So we have support and race. So we've worked with the any
24:20
scale team and the open source community.
24:22
So for example, you can deploy your models with reserve and
24:26
also use the Ray train when just recently we've announced support
24:30
for Ray train. So you can use their API and use
24:33
Ray as a way of of managing your training or inference workloads
24:37
and as well out of bounds. So I'm sure many people are fans
24:41
of the Metaflow open source project.
24:43
So we had a recent blog. So they have actually added
24:46
support for training and and friendship and the metaphor
24:49
platform. So that's also an another great
24:51
option customers like. Nice.
24:54
And last but not least, because there is such a rich ecosystem,
25:00
I am wondering about Bedrock and maybe you guys can just clarify
25:06
too, when you've seen people using Bedrock versus Sage Maker,
25:10
why, what the use cases are there?
25:14
I think there's many, right? And so going back to our the
25:17
beginning, you know, AWS is about offering customers choice.
25:21
And so when we think about it at AWS, it's like this.
25:25
We have this concept of like the generative AI stack, right?
25:29
And at the top you're going to have the services Amazon is
25:33
building like Transcribe, like translate things like this that
25:37
are like end to end services where the models are integrated,
25:40
the user experience is integrated.
25:42
And it's just an API or service that you use.
25:45
And, and you know, it makes it really simple for developers to
25:48
integrate, but it provides you less customizability, right?
25:53
Like you get the performance you get, you get the feature set you
25:55
get. And then overtime, it improves
25:58
the kind of next layer down that we've now added, which is like
26:02
the bedrock level, which gives you a little bit more
26:05
customizability right now, as more models have become popular
26:08
as and to consume those models is models as a service.
26:12
And that's where bedrock really sits right.
26:14
So if you want to get access to the greatest models being
26:19
produced today, right now, like from Anthropic or Mistral or you
26:24
know, AI 21 and many others, and even Amazon with our Titan
26:27
series of models, right, you can you very quickly get access to
26:32
those and other open source models and then just consume
26:35
them as an API, pay for them with a per token, you know,
26:38
tokens per dollar, right? A very simple use case there.
26:42
And now even you can even fine tune certain open source models
26:45
as well, kind of make customise them for your.
26:48
Liking as well. Are you fine tuning them with?
26:50
Training them, yeah. So we utilise bedrock heavily
26:54
utilises inference and training, but also other accelerator
26:57
types, right. So it uses, not only is it uses
27:01
Tranium, it uses inference to kind of build and serve these
27:04
models across the board. So basically all of the
27:06
resources AWS has at its disposal to engage with
27:10
customers. And I think better off makes
27:12
sense for a lot of folks. And it depends kind of where
27:15
your ML expertise is, right? So if it's like, hey, we're just
27:20
trying to build a really quick prototype or we're just at the
27:22
application level, we don't have a lot of folks that understand
27:27
how to train models because, you know, as easy as it's getting in
27:30
today's world, it still requires some experience to do it
27:33
efficiently or to even build the services around them.
27:36
So Bedrocks are great answer for that.
27:38
But if you're going down the stack a little bit, where you
27:40
want to customise the interaction of models, you
27:43
really want to own all of the data around the model and how
27:46
it's being deployed and where it's being deployed.
27:48
Certain applications where are very sensitive to data,
27:53
residency and lineage, right? Like where is the data coming
27:55
from, the train, the model, where is the servers running
27:58
everything? It can't leave our borders.
28:00
And this is probably more important for European
28:03
countries, which have really strict data privacy.
28:05
Laws love it. So for that, you know, Sage
28:08
Maker in French and Tranium or EC2 are going to be better
28:12
options. It gives you that same kind of
28:14
ease of use, but greater flexibility of where you're
28:17
deploying your models, how you're setting them up, which
28:20
libraries are they integrating with?
28:22
You know what, what other data it's Co mingling with right like
28:25
and if you're building a broader pipeline of services, you can Co
28:29
locate your accelerators into one region to improve the
28:33
overall throughput and latency. So there's a lot of benefits and
28:36
it's really just about what are the users objectives?
28:39
What is the total applicable, the end application look like
28:42
and what's the right fit for them and what's the right
28:45
choice? And actually, we have, you know,
28:48
I would say it's it's not a one, it's not an either or decision,
28:52
right. So what we see a lot or with the
28:54
customers we work with, they'll use services from bedrock,
28:58
right? They'll be using one model from
28:59
bedrock. Let's say that they're like, all
29:01
right, I want to use one of the cloud models from bedrock.
29:04
I also want to integrate with a utilising chat GPTS APIs, right?
29:09
And so they have part of their services coming from there.
29:12
And then, hey, we're going to be running all our Llama three
29:15
models at, you know, 70 billion parameter Lama three models on,
29:18
on inference and training, right?
29:20
And so it's kind of a, you know, it gives you that choice like I
29:23
can, I can get the right models for the right part of my
29:26
application and build the best experiences for users and
29:31
customers. Yeah.
29:33
And that in a way makes a tonne of sense just because you're
29:37
giving so much resilience to your application to by having
29:41
that. And so if that rule, I fully
29:44
understand Bedrock now. It's like, all right, if you
29:47
just want that API, that's Bedrock.
29:49
It's that like open AI API experience in a way where you
29:54
get a model, you call the API and you're good.
29:57
But if you want to go deeper, you want to have much more
29:59
control, that's where you're going into the Sage Maker
30:01
territory or the EC2 depending. It's like there's levels and
30:05
there's layers of how much control you need and want, and
30:09
that is which service you're going to use depending on what
30:12
you're looking. For yeah, I mean, if, if
30:14
Bedrocks kind of meets the application needs, like you're
30:17
getting the right tokens per second, the right latencies, it
30:20
has the right models that fit your application and needs.
30:24
It's, it's a great option, right, because it's super
30:26
simple. But if you need to go, you know,
30:28
kind of peel that layer back, go one layer down, you know, that's
30:32
where, you know, utilising or hosting your models or starting
30:35
even with Jumpstart, because the experience is very similar in
30:38
terms of getting a started or with utilising the Hugging Face
30:42
TGI containers. You could deploy any model
30:45
pretty easily, but you have greater control, greater
30:48
flexibility, you can kind of tune the variables that matter
30:50
the most to your, the, to the user experience that you're
30:55
trying to generate for, for your customers.
30:58
So let's just, it's, we get that question quite a bit as well,
31:02
right? Which is what's better, Bedrock
31:05
or Sage Maker or EC2? And I, I don't think that one is
31:08
necessarily better. It's just, you know, what are
31:11
the use case what what are the things that are most important
31:14
to you and your applications. Yeah.
31:16
And I do like this idea too that OK, maybe it's you're going to
31:20
have a different piece of your application using different
31:25
parts or different services depending on what your needs
31:28
are. But also I can imagine that you
31:31
get different teams using different services too because
31:36
of their needs. And so looking at it from the
31:39
point of view of like, hey, if this super simple API gets you
31:44
there, then no need to overcomplicate it with a bunch
31:47
of services, just use the API. Yeah, if customers are unsure, I
31:52
think always started the easiest to use service and then you know
31:56
then early really go down once you start, you know, finding
32:00
that based on your application or context that you need to, but
32:04
you know, it's going to have a lot more costs and a lot more
32:06
things to manage. Yeah, once you hit the limits,
32:09
then go deeper. But until you hit those limits,
32:12
don't feel like you need to start at level hard.
32:16
Well, that's, that's the funny part is like when I would say
32:19
that that kind of mindset that you just mentioned is like start
32:22
at level hard, right? Because I, I feel like a year
32:25
ago, two years ago, when we were working with customers and
32:28
users, it was always like, I have to train my own
32:30
foundational model, right? And I have to do it all from
32:33
scratch and I have to set up my data and I have to curate my
32:35
data set. And it was always starting it
32:38
hard because, you know, there was an unwillingness to use open
32:42
source to a certain degree, right?
32:43
And they're in, in kind of enterprise environments, right?
32:47
Or there was like, ohh, we have to own everything.
32:49
And I think that mindset has really changed with users,
32:52
especially in the enterprise space.
32:53
So like, no, no, I can start with open source.
32:55
I can start with the Llama model or a Mistral model or a Zephyr
32:59
or something else or a stable diffusion model.
33:02
And I can fine tune it with my own data set and do it much more
33:07
cost effectively and, and easier and with less manpower on our
33:11
side and, and make, you know, just get to market a lot
33:14
quicker. And and the I think the idea is
33:17
now is how quickly can we get it done rather than like, hey,
33:21
let's own the entire thing. Well, no.
33:23
Let's leverage as much as from the community as possible and
33:27
contribute to the community along the way.
33:30
OK. So you're talking a lot about
33:31
what you've seen out there in the wild.
33:34
I think this is a good point to transition into like actual ways
33:38
you've been seeing people using Trainum and Influencia and what
33:43
some examples are of that. One of the key partners that
33:47
will customers that we're working with is Anthropic.
33:49
So towards the end of 2023, about September 2023 we actually
33:55
had a have an agreement with them.
33:57
So they are initially deploying cloud models onto infringer and
34:02
training and these are actually going to go in bedrock as Kevin
34:05
was mentioning earlier. So we're in the process of
34:08
moving those across and Cameron also mentioned earlier training
34:12
too. So a train him two is gonna be
34:14
launched very soon towards the middle of 2024 to its end 2024.
34:19
So yeah, and propagate going to be training the next Gen models
34:23
on on the training. So, so yeah, so there are really
34:26
exciting customer to work with and yeah, to look forward to see
34:29
us seeing those models on bedroom.
34:32
And there's a, we've been working with kind of a range of
34:35
different kind of startups and enterprise customers, kind of
34:38
customers bringing AI into their traditional applications, right?
34:43
And now leveraging the alms to kind of improve the experiences
34:46
and make it more user friendly. And on the startup side
34:49
specifically, we're starting to see a lot of kind of emergence
34:52
of new complete products, right? I think one of those areas is,
34:56
you know, which is very, I, I think gaining a lot of momentum
35:00
in the market is the concept of like AI agents or autonomous
35:03
agents, right? And make them very personalised
35:07
to your workflows, to your company or to yourself
35:10
specifically, right. So one of these companies we've
35:13
been working with that just recently launched a new service
35:17
called, I think it's called my Ninja, but their company name is
35:21
called Ninja Tech. They're based here in the Bay
35:23
Area and they've been leveraging training and in friendship to
35:27
fine tune their models and to deploy them at scale on the kind
35:30
of serve all of their model needs.
35:32
And so one of the things that's interesting about them is when
35:35
they were looking out and starting and they started with,
35:38
you know, the integrations from all of the models, models as a
35:41
service, of course, as we kind of talked about.
35:44
But really quickly, they're like, you know what, we can
35:46
create highly customised, very efficient models starting from
35:52
utilising open source models, right?
35:54
And we're going to fine tune those models specifically for
35:57
our use cases, right, which include like code assistance
36:01
researchers, you know, your basic kind of like information
36:04
gathering and, and, and reporting and also scheduling
36:08
tasks, right? And outperform some of the most
36:10
expensively foundational models that are trained from scratch.
36:14
And So what training and inference, they're basically
36:16
able to kind of utilise Llama three models to serve their
36:21
customers. They're using a range of them
36:23
all the way from the small 8 billion parameter code Lama
36:27
models kind of in the middle from like you know, 30 billion
36:30
perimeter and all the way to the 70 billion parameter models and
36:33
able to reduce their inference costs and their deployment costs
36:37
by 80% by utilising should training to serve these models.
36:42
And another kind of example of this is as more customers as
36:47
well want to build more customised models as well,
36:51
right? O, we've been working with other
36:53
guys that are trying to kind of better create better tools in
36:57
this space as well, right? So, you know, fine tuning models
37:02
can be challenging at a certain point, right?
37:04
Like there's a lot of different techniques from softs to TRL to
37:07
like, how do you do implement path?
37:09
What are the right techniques to not overfit your model to make
37:13
sure it's as efficient as possible?
37:15
So we've been working with another Arthur called RCI and
37:18
they're trying to optimise this space where users have data that
37:23
needs particular sensitivities to privacy and to security.
37:29
And they can set up training environments inside of customers
37:32
domains like, you know, like in their AWS private clusters and
37:38
do end to end fine tuning of models and create really highly
37:41
specialised models at a fraction of the cost, right?
37:43
So they're utilising training them to accelerate this and
37:46
seeing up to 90% cost reduction utilising both their domain
37:52
experience, which is a combination of highly efficient
37:56
LLM fine tuning, but also concept that they've kind of, I
38:00
don't know about Pioneer, but really leaning into which is
38:03
going to be small model training, right?
38:05
So as everyone's trying to go to like larger models, 70 billion,
38:09
you know, hundreds of billions of parameter models, which can
38:12
increase the abilities of these models.
38:15
They're also thinking, well, how do I take a bunch of small
38:17
models, less than 25 billion parameter models through less
38:20
than 3 billion parameter models and create a lot of specialised
38:25
small models and merge them together into a final model,
38:28
right? So they have this model merging
38:29
technology as well. And they're utilising training
38:32
to even lower the cost of building all of these models and
38:36
deploying kind of for more wide range of use cases.
38:38
So great solutions across the board.
38:41
So like if you're looking at, you know, you can definitely
38:43
check out ninjas solutions that are live now and then RC at RC
38:48
dot AI as well. Yeah, huge shout out to RC.
38:52
I'm actually friends with their one of their founders, Brian
38:55
Benedict. Ohh, yeah, yeah.
38:56
And Brian's great. I love, I love that guy.
38:58
I love what they're doing. The model merging is
39:01
fascinating. And there's another guy in the
39:03
community, Maximilian, who is doing a lot on that front too.
39:08
And it feels like the model merging is something that is
39:15
getting very popular and it is getting to be Yeah.
39:20
The small language models theme or narrative is also bubbling U
39:27
quite a bit. I imagine you guys have been
39:31
seeing it a tonne too. Because All in all, like if you
39:34
can go smaller and you can still accomplish your goal, then it's
39:38
gonna be much better to be smaller because it's faster,
39:41
it's cheaper. Yeah, one more can you ask for?
39:43
And I think, I think that's the hidden cost that people don't
39:46
realise. They're like, well, OK, fine, I
39:47
can train my model and it's a one time expenditure.
39:50
So, you know, I'll bite that bullet.
39:52
I'll train my 70 billion parameter or 100 billion
39:55
parameter model or maybe more and it cost me X dollars.
40:00
But now if you want to serve it and keep that real time low
40:04
latency, high response engagement with your user to
40:08
actually use that model, it can be very expensive to do so right
40:12
now inferential and training and help reduce those costs, but it
40:14
is still more expensive. So if you're going to accomplish
40:17
those same, the quality of model at a fraction of the model size,
40:22
you have your deployment options open up right now.
40:24
You can use smaller instance sizes, less compute, you can
40:28
scale further without, you know, kind of like investing more in
40:32
compute cost and operating cost. So it's, I think there's a lot
40:36
of potential here as, as I think we're optimising or the
40:40
industry's optimising in two fronts, right?
40:42
Like one is how do we make, you know, models more capable and
40:46
expand their use cases? So that's going to, you know,
40:48
build super large models. But at the other end of it, how
40:51
do we make small models more efficient?
40:54
And so I think we're kind of seeing both of those trends in
40:56
RC is definitely on the, you know, their, their car across
40:59
the board. But really, I think I'm excited
41:01
to see what they're doing. Small models and model merging.
41:04
Have you guys seen agents that have been working well in
41:09
production? Yeah, absolutely.
41:11
Actually Ninja is one of those companies.
41:13
They're they're actually using a few different models that are
41:17
all running on inferential and training them to serve their
41:20
agents. It's it's great solution as
41:22
well. But I think we're going to see a
41:24
lot more agents in Amazon launched their queue products,
41:27
which is also trying to help in this space, right.
41:30
You know, personally, I think we're going to see agents kind
41:33
of infiltrate all aspects of our lives as we move forward, right.
41:37
Like I just saw that Google is replacing their Google Assistant
41:41
on all of their Android phones and tablets with Gemini, right?
41:46
So they made that announcement as well.
41:47
So I think agents, agents are are in our future across the
41:51
board, right? But I think the differentiator
41:53
will be is good high quality agents, not just let me ask you
41:57
the weather or you know, who won the basketball game last night,
42:00
but you know, like very specific and personal.
42:03
And you know, it's going to require being able to kind of
42:06
deploy and fine tune these models in, in a more of a kind
42:09
of like a reinforcement learning technique where they're
42:11
constantly learning about you, your habits and then
42:15
implementing that or updating the models as fast as possible.
42:19
Excellent, fellas. Well, this has been awesome.
42:21
I appreciate you all coming on here and talking to me about
42:25
this because I know there's a lot of confusion and just people
42:29
want to use it. Just now I was looking through
42:31
the Mobs community Slack and somebody's asking about GPU
42:35
resources in AWS and what's this and what's that?
42:40
So I guess the last question I've got for you, I imagine
42:43
there's some kind of a chart that you all have as far as
42:46
well. If you're looking for this type
42:48
of GPU, you'll probably be good with this instance of Influencia
42:53
or Tranium. Does that exist?
42:56
And if not, can we make it? I would say good starting point
42:59
is our documentation. So we have the Neuron SDK
43:03
documentation. I assume we can put a link in
43:06
the show notes and there we have a lot of information.
43:09
So for example, tutorials, getting started guides and even
43:13
we even have a table, a performance table of all the
43:16
most popular open source models, such as, you know, Lama 2, Lamma
43:20
3, Mystro stable fusion models. But in which instances it can be
43:25
applied to and what come the kind of performance you can get.
43:28
And we have different combinations, right?
43:29
Depending if people are optimising, they really need say
43:32
low latency, obviously probably won't need to use more
43:35
accelerators or maybe optimise for costs and use fewer.
43:39
So yeah, so we have many different options there.
43:42
So encourage folk to, you know, check out our documentation and
43:45
and give it a try. Awesome.
43:49
I love it guys. Thanks for doing this and it
43:53
feels like I've learned a tonne all about inference and tranium
43:58
and especially I I like the fact that it's not joined at the hip.
44:02
You get to choose like it's choose your own adventure when
44:06
it comes to this. You're able to save this 46%
44:09
cost is huge and being able to get that then also the idea of
44:16
start simple ahead and go down the stack as need be.
44:21
When it gets more complicated or you find you're hitting your
44:24
limits, then you can go from that Bedrock to the Sage Maker
44:29
to the EC2 or whatever it may be.
44:32
But it's cool to know that all along the way, if you're using
44:38
or utilising Tranium or Influencia on any of these
44:42
services, you can keep utilising it.
44:45
And it's not like, oh, this only works with Bedrock.
44:48
So it's nice to see that that's like the Amazon way, I guess
44:52
that you probably wouldn't be able to ship something if it
44:56
could only work with one of these services.
44:58
I think that's enough people have written about that, books
45:00
have been written on that subject, but that's not for
45:02
today. I'll let you guys go.
45:04
This was awesome. Thanks.
45:07
Thanks very much, it's fun.