Latest / Agentic Conversations (formally mlops.community) / FedML Nexus AI: Your Generative AI Platform at Scale // Salman Avestimehr // #230

Transcript

FedML Nexus AI: Your Generative AI Platform at Scale // Salman Avestimehr // #230 · Agentic Conversations (formally mlops.community)

0:00
Hold up, wait a minute. We gotta talk real fast because
0:03
I am so excited about the MLB's Community Conference that is
0:07
happening on June 25th in San Francisco.
0:10
It is our first in person conference ever.
0:12
Honestly, I'm shaking in my boots because it's something
0:15
that I've wanted to do for ages. We've been doing the online
0:18
version of this and hopefully I've gained enough of your trust
0:22
for you to be able to say that. I know when this guy has a
0:25
conference, it's going to be quality.
0:28
Funny enough, we are doing it. The whole theme is about AI
0:33
quality. I teamed up with my buddy MO at
0:36
Colina, who knows a thing or two about AI quality, and we are
0:41
going to have some of the most impressive speakers that you
0:45
could think of. I'm not going to list them all
0:48
here because it would probably take the next two to 5 minutes,
0:51
but just know we've got the CTO of Crews coming to give a little
0:56
keynote. We've got the CEO of you.com
1:00
come in, we've got Chip, we've got Linux, We've got the whole
1:04
crew that you would expect. And I am going to be doing all
1:09
kinds of extracurricular activities that will be fun and
1:13
maybe a little bit cringe you. You may see, hear or see me
1:18
playing the guitar. Just come.
1:20
It's going to be an awesome time.
1:21
We'd love to have you there. And that is again June 25th in
1:25
San Francisco. See you all there.
1:28
I'm Salmon Investment. I'm co-founder and CEO of
1:31
Federman. I'm also professor of EC and CS
1:35
at University of Southern California and Director of USC
1:39
Amazon Centre on Trustworthy. I love coffee.
1:44
I take it pour over American style.
1:48
Welcome back, MLB's community. We are here with another
1:52
podcast. I'm your host, Dimitrios, and
1:55
today we're talking to Solomon all about the Fed ML platform.
1:58
I really appreciated how he broke down the maturity levels
2:02
that he's been seeing out in the wild as he goes and he's been
2:06
creating Fed ML. They have a specific gene AI
2:08
platform these days and he's been talking to customers and
2:14
asking them what their specific pain points are.
2:18
And they go out there in the wild, they see what the pain
2:20
points are. And he also sees and was able to
2:24
convey to me in this conversation how customers can
2:28
get stuck in different places. And what almost like this
2:31
maturity level is when you encounter hurdles, whether
2:35
that's on the deployment hurdles, it's the evaluation
2:39
hurdles, the fine tuning hurdles.
2:41
And he also showed a bit of a circular motion of, yeah, you
2:47
deploy, then you collect data and you evaluate that data and
2:51
you go back and you retrain and then you deploy.
2:54
And it's like, huh, all this whole AI thing sounds vaguely
2:59
familiar. I feel like I've seen that
3:02
before. I just can't put my finger on
3:05
where. Ohh yeah.
3:07
Now I remember, this is what we've been doing in MLS for
3:11
ages. So Sullivan was great.
3:15
He broke down their journey and why and what they've been doing
3:19
with the Fed ML crap platform and who it's for, who he's seen
3:24
using it and what the inspiration or what the motives
3:28
behind people wanting to use a platform like Fed and they'll
3:30
have been hope You enjoy this as always.
3:33
Leave us a review, give us some stars, give us some love on
3:38
social and share it with just one friend.
3:41
If you think it will be valuable for them, look forward to seeing
3:45
you on the other side, Solomon. Right now you're based in San
3:57
Francisco, I'm guessing. In Palo Alto, yeah.
4:01
Alright, and you're living the dream running Fed ML.
4:04
I know that you have this whole generative AI platform that we
4:09
want to get into, but I think we should just start with like who
4:14
have you been talking to? What are some challenges that
4:17
you've been seeing out there that gave you the inspiration
4:20
for what you are doing? That's right.
4:23
Especially if you look at it like the generative or you know
4:27
there has been rapidly growing and every day there are new
4:31
challenges, right. And the way we view it is there
4:34
is a segment of the market that they want to build their own
4:38
models, right. So there is like a challenge of
4:41
ownership. Ownership I think is very broad
4:44
in the sense that maybe many developers want to build
4:47
applications off of maybe other models like let's say open Air
4:52
or GT4 etcetera. But there is a large group of
4:55
developers who want to have ownership.
4:58
Enterprises also want to have ownership.
5:01
For example, your data, your I. So the first challenge we are
5:05
seeing is how can we help people who want to build, deploy their
5:11
own models, their own applications in this complex
5:15
landscape, right. There is a very complex general
5:18
software stack that you need to know on how to build, how to
5:23
deploy your models, how to create applications.
5:27
And one part of you know the challenges that we are seeing is
5:29
again ownership and how to be your own opener.
5:35
Yeah, the 2nd we are hearing is scalability.
5:40
OK, so scalability would mean you know as maybe your
5:44
application becomes popular or you want to maybe serve many
5:47
users, then you need to have capability to scale your
5:50
application. Maybe at the scale that ChatGPT
5:53
is scaling right? There is a lot of infrastructure
5:56
behind ChatGPT. I mean the developer only sees
6:00
the API, but there is a lot of capability.
6:02
Azure cloud behind it, Azure oil behind it, etc.
6:06
So I would say ownership and scalability are the two key
6:12
worries that we are seeing again across individual developers as
6:15
smaller startups, enterprises. Yeah.
6:20
With that ownership, I think one thing that I've been hearing
6:23
over and over is control and how people want control not only of
6:28
the models but just of what is going on in their system.
6:33
And you get to a point where I think that it's it's kind of in
6:37
the journey of, alright, we'll start with something really fast
6:41
and see if it has value with Open AI.
6:44
And then ideally we're going to transition off of it if you can
6:49
use Open AI at the beginning because there's a lot of
6:51
companies that can't just go and use that Open AI API.
6:57
I'm sure you've talked to quite a few of them and they have to
7:02
figure out, alright, how can we iterate quickly, right?
7:06
How can we make sure that we're not losing out Because we can't
7:09
hit that API and we can't just figure out like product
7:13
iterations over and over and empower anyone on the product
7:18
team, or on any team for that matter, that knows how to work
7:22
with API's to create something with AI.
7:26
Exactly. And that control even simply,
7:28
let's say you have your model or maybe the API or you are calling
7:32
having observability, monitoring, How are people using
7:36
that, what are the interactions, when does the model maybe give
7:40
the wrong answers, what is the feedback, how to bring that for
7:44
your next iteration, etcetera. Definitely.
7:47
Ownership is very broad. Ownership control.
7:50
Privacy is also connected to that, like as you want to have
7:53
ownership right, you want to have privacy.
7:56
Again, another concern that you know, many companies deal with.
8:01
I mean, as you are using these services, how much are you
8:04
leaking and how much are you exposing all of them?
8:08
I would say ownership is very broad.
8:11
How are you dealing with that problem of data leakage and
8:15
privacy? There are different ways.
8:17
So one part of it is like on premise deployment, right?
8:24
So for example, let's say you are an enterprise, You have a
8:27
lot of proprietary data that you want to use for building your
8:31
models. And later when you maybe you
8:33
want to offer the service to your own and you know users,
8:37
your own employees, etc for them and we have a you know a great
8:42
solution that offers fully on premise is like you are
8:47
building, you are deploying everything between your own VPN.
8:52
So that's great because nothing is leaving your trusted
8:56
environment. So that's where you can think
8:59
about it, what enterprises are looking full control, full
9:03
ownership, but they need to have the capability in house being a
9:09
platform we can offer that. So that's one way.
9:12
The other one is, you know, let's say you are a developer,
9:15
right? You want to pin the solution,
9:17
right? That's where you can have your
9:18
own dedicated endpoints and deploy the models on your own
9:23
dedicated server. For example, if you are using an
9:27
API that you are getting right, these calls will go to other
9:30
servers, maybe other models. Something that many startups
9:34
will come to us is because they get dedicated deployment of
9:38
their models or their agents, right?
9:41
That way it's like belongs to them right?
9:45
The ownership so you can trust then the other layer of it
9:48
actually goes higher. It's kind of as you build that
9:51
essentially I mean your models will expose the training data.
9:55
So there is no way around that. It's like, you know the train
9:57
model has been used, right? If the model you interact with,
10:01
it can expose the training data. I would call that a safety layer
10:05
that we are building. So it's kind of understanding
10:08
before sanitising the response or maybe filtering the
10:11
questions. So that becomes a like a layer
10:15
that after building and deploying the model you want to
10:19
make sure you analyse and figure out what questions are
10:23
appropriate to respond. What questions for example maybe
10:27
are asking sensitive data about your company and you should
10:30
philtre them, but I think the safety layer becoming very
10:33
important layer. So essentially you know there
10:36
are different layers at the time of building the model, when you
10:39
are deploying the model and after production, how to bring
10:42
in privacy. Yeah, it does feel like if you
10:47
wanted to do this yourself, what would these different pieces
10:50
entail, right, The building of the model, what have you
10:54
generally seen people using and doing in that regard?
10:57
And then when you need to deploy it, what are some other things,
11:00
common design patterns for that? And then once it's out there in
11:04
the wild, everything that is also a potential for people to
11:09
do if they want to do it on their own.
11:10
Because it it's almost like most people have it too easy when
11:15
they can just hit the Open AI API and you get a little bit
11:20
spoiled if you can do that or if you can just like talk with Chad
11:23
CBT. But then if you want to
11:25
productionized something in house like you're saying, you're
11:29
probably going to have to hire an engineer And whatever that
11:33
engineer salary is these days I hear AI engineers or ML
11:38
engineers aren't cheap and so whatever that salary is plus
11:42
whatever the cost of doing the business on top of it is going
11:47
to be. So have you seen people that are
11:51
trying to do this in house and then they come and they say
11:55
alright, we are pivoting to an easier solution?
12:02
Exactly. Actually we have seen people who
12:04
want to build it in house. The problem of that is exactly
12:08
the layers that you mentioned. Maybe you mentioned the
12:09
engineers, but I would even add more to it.
12:12
Typically to build it, you start with the maybe ML scientist.
12:16
ML scientist knows the mother, knows the algorithms, etc.
12:20
Then you need to have Amelia engineer, right?
12:23
Then you have to have the inferiority because to building
12:26
in house you need to have a very strong Infratil.
12:29
Typically many companies maybe you are good at the mill.
12:32
Science and engineering even if you can get the engineering
12:35
infra is another layer. You never want to, you know have
12:39
that and I think that's the role of let's say Father Mel as a
12:43
platform company. The goal is you don't need that,
12:47
right, That that's exactly the exciting part.
12:50
We help them along the journey to deploy their own
12:55
applications, maybe create their own models, right?
12:58
That way you don't need to hire the infra team.
13:00
Why would you need it? I mean the federal platform is
13:03
going to take care of all those challenges at the in front here.
13:08
What does the ML engineer would do with toy two?
13:10
You know maybe simplify what the mill scientists what to deploy
13:14
so makes it simpler. So the angle that we have is
13:19
don't give up on your dream of ownership and control to go with
13:24
a simple approach, right building off of the models that
13:28
you don't control, the models that can change, it's still do
13:31
that what we help you along the way.
13:33
So that's how I respond to that. Now talk to me about the
13:37
platform itself, because as you probably are very familiar with
13:42
it, there's a tonne of LM tooling options out there.
13:48
Once you've deployed that model, it's not just that you're using
13:51
the model and then you're done. If you have some kind of a rag
13:55
solution, you've got a lot of glue that goes in between that.
13:59
Are you also adding glue or are you saying we've got your model
14:04
endpoint? We also can give you a bit of
14:06
guardrails as you were talking about and then you can bring
14:09
your vector database or you can bring your whatever
14:13
orchestration framework that you like.
14:16
Yeah, I I would say actually for us is of course we offer the
14:20
glues, but it's much more beyond that.
14:21
It's like the depth of what you can do, right?
14:25
So essentially as you mentioned, there are maybe many two links
14:27
that you can quickly get an API from a model, combine it with a
14:31
rack system, create an agent. What I would say, they don't
14:35
give you a depth that you need for a commercial enterprise
14:39
solution like a large scale. Let's say you want to have a
14:43
like a very large, largely deployed, usable application
14:48
that is used at million active users.
14:51
So what are those? For example?
14:52
Let's say in the journey you start with a model.
14:55
The first step of it is that you need to deploy and serve the
14:58
model. That one is much more beyond
15:02
than just getting the API that can serve the model right.
15:05
API is just the upfront that you would see in the back of it.
15:09
If you want to have the, you know, like your own way of
15:13
serving the model now that you need to maybe create dedicated
15:16
endpoint, let it autoscan freely as the demand goes up, maybe
15:21
create more endpoints and more replicate across more GPUs.
15:25
And as the demand goes down less.
15:27
That's the way that you can manage the cost.
15:29
That's the way that you can control the quality.
15:33
As you are serving the model, you need to have a lot of
15:36
observability. For example, how people are
15:38
using that, What is the LA they are getting?
15:42
What is the query per second. Even more, what are the
15:45
prediction locks? For example, what are the has
15:48
been the questions? What has been the answers so you
15:51
can use them later in fine tuning.
15:54
So what I would say is that for example, even in the deployment
15:57
part, there is a lot that we can offer people to build a scalable
16:02
endpoint that is production ready.
16:04
Like we let people productionized not just having a
16:07
simple like a demo. Yeah.
16:10
So that's likely on the deployment.
16:12
Then we try to let people go along the journey.
16:15
Like typically you deploy the model, you have a scalable
16:18
endpoint. The next phase is that as you
16:21
collect more data and people are interacting, you always want to
16:24
make your model better. Yeah, So that's where we help
16:28
people fine tune the models, right?
16:31
We help people. For example, we offer zero code
16:33
fine tuning right? That way it's like we have
16:36
already. Let's say you are a maybe
16:38
software engineer. You are a data scientist.
16:40
Maybe you cannot write the code for fine tuning that way.
16:44
You can just change some hyperparameters and start point
16:47
tuning the model from the data that you collect in the
16:51
deployment. Then you want to do more
16:54
advanced, for example maybe a distributed training of it.
16:57
We offer mill OPS for the training, but the other angle of
17:01
it, as you mentioned, is that maybe you want to start doing,
17:03
maybe creating an agent with it, so on and so forth.
17:06
We also support, but essentially it would say it's kind of a for
17:10
each of those as you want to go deeper and again the vision is
17:14
that you want to build an application at a scale of
17:16
ChatGPT and what opening is doing for your own application.
17:21
We help you do it without having a huge infra team or no infra
17:25
team at all. So let's say that I do have one
17:29
or two overworked rares and they're saying alright cool, I
17:34
want to use Federal, but we've got our GKE instances or we've
17:40
got a WSEKS or EKC instances and we want to know how that plugs
17:48
in to Fed ML. Is it just something that you
17:51
can put on top of it? How does how does it interact
17:54
with the infrastructure that I already have?
17:57
Actually that's a great point. So another key component of it,
18:00
as I mentioned that you're part of, we spend a lot of time, the
18:05
last part we say your general oil platform at a scale, the
18:09
scale would mean female is a platform that you can deploy it
18:13
multicloud, for example, let's say Azure or is a platform
18:17
running on Azure cloud or Vertex AI platform for Google Cloud.
18:22
Federal is a general platform that you can run it on any cloud
18:27
and that's how we let people escape.
18:29
Not on your question. Let's say you are a maybe a
18:32
startup company. You are using many instances.
18:34
Maybe you are getting credit and using AW right.
18:38
What you can do is add your own instances or your own you know
18:43
clothes inside your own compute that you can use in Federman.
18:48
But the beginning advantage that we offer is Federal already
18:52
offers multi clot through partnering with many GPU
18:56
providers, all right. So that way you don't need to
19:00
shop around and find the GPUs that you would need now.
19:04
So for example, let's say you are a, you know, maybe a
19:06
developer, there are many choices that you have for
19:09
offering GPUs, right? There are many GPU providers and
19:13
the prices are changing, the capabilities are changing.
19:17
Federal gives you the offer that you don't need to shop around
19:20
right is a multi cloud, multi provider.
19:23
We always find the best resources for you, so it becomes
19:27
both offering you platform and the infra.
19:31
And the difference is that this infra is multi cloud multi
19:34
provide, but you can also add your own infra to it is very
19:39
easy. It's like you know, you add your
19:40
instances quickly and use federman to deploy launch point
19:44
tool on your own instances. And so talk to me about the
19:47
evaluation piece because I know that is a hot topic when it
19:50
comes to how people are doing it, what they're doing, what
19:54
kind of results they're seeing. Have you noticed any trends on
19:57
best practises when it comes to evaluating the models and the
20:03
products that people are putting out there?
20:06
Opportunities. Maybe one part of evaluation is
20:09
just how good your model is. For that one there are, you
20:13
know, various benchmarks, I would say for general purpose
20:16
and norms that are very well known.
20:18
You know, benchmarks that you can use to assess your model.
20:22
It becomes more challenging as you are building vertical
20:25
elements. So for example, let's say you
20:27
want to have a very good L&M that is focusing on maybe
20:31
healthcare questions. So there is less of benchmark, I
20:35
mean there are some benchmark, but I think is less known for
20:38
different verticals, how good your model is.
20:41
OK. So that's where I would say one
20:43
part that's maybe the first year there has been a lot of effort
20:47
on benchmarking for general purpose and then apps vertically
20:53
focused NMS. I think there is going to be many more benchmarks
20:56
coming up to help people evaluate the models right and
21:00
figuring out how well are they doing.
21:02
Or maybe you want to choose between different models.
21:04
Which one are you going to choose?
21:06
For example, for coding like helping you writing codes, there
21:10
are already good benchmarks, right?
21:13
So that's where I would say an example of a vertical LLM, that
21:16
there are good benchmarks, you can evaluate it and do that.
21:19
But many others we have lack of benchmark.
21:23
The other part of it is like other measures.
21:25
For example, you have a model, how much is, you know,
21:28
hallucinating? How much is the credibility
21:32
score of it? And that's the other side that
21:35
we are seeing, for example. There are multiple layers for
21:37
that. It's like you want to assess on
21:39
the simulation, you want to assess the correctness, you want
21:42
to assess the privacy leakage in the model.
21:45
I think these are converging into separate layers that maybe
21:49
I think soon you can just click and choose which one of them you
21:52
want to enable, which one of them you don't want to have.
21:55
But there is also much deeper question behind it.
21:58
Deeper question is for example, you can think about let's say an
22:03
LLM comes up with a response. How would I assess the
22:08
correctness of response? Especially when the questions
22:12
are open-ended, right? So that's a very interesting
22:17
problem, very challenging problem.
22:19
The reason is, you know, again, assessing correct correctness.
22:23
Typically in machine learning there has been a lot of work
22:26
when it's like a binary hypothesis testing or multiclass
22:29
hypothesis testing. You know what the classes are
22:32
and you want to figure out is the class being detected
22:35
correctly. That's why there is a lot of
22:38
good assessment and evaluation, say for computer vision models
22:42
where the classes are clear. Either the object is a cat or a
22:45
dog, or you know something. Lines are very different.
22:49
These are dynamic responses. They are open-ended, no clear
22:53
classes, becomes a very interesting research problem by
22:58
itself, which is like, you know, how would I assess the
23:01
correctness or evaluate it. For that, Actually we do a lot
23:05
of research in our group. For example, we have a recent
23:08
paper on meaning aware response scoring.
23:13
It's kind of What is the meaning of this sentence and how to put
23:17
weight on different tokens, right?
23:20
So for example, let's say you ask me, do you ask an LM, right?
23:24
What is the capital of France? All right.
23:28
And then he would say the capital of France is is a
23:31
beautiful city called Paris. In this sentence, the key
23:35
component is Paris. Not all tokens have the same
23:39
weight. That's the only token that
23:42
affects whether the response is correct or not, right?
23:46
So it's kind of how would I assess?
23:47
By paying attention to the meaning, the correctness of the
23:50
sentence, and figuring out the important tokens, less important
23:53
tokens, etc. So you can imagine I mean those
23:57
questions you know on there. When you go to the practical
23:59
side, people are looking for simpler, quicker things to do.
24:03
As you go towards the research, it becomes very deep questions
24:07
that are still we don't know, you know how to address them.
24:10
But the research community is also moving very fast in the
24:13
domain and I feel you know, there is gonna be a lot of
24:15
innovation coming for evaluation by itself to the community,
24:20
which can help, you know, even building better models as we
24:23
evaluate them better, helps to figure out what is the new
24:28
training data to be used to make the performance better because
24:33
that's the bottleneck that we are facing now.
24:37
So now this research paper and being able to wait tokens
24:40
differently, are you then able to pipe that back into Fed Mill
24:46
and the platform and fine tune with that meaning awareness?
24:52
Definitely exactly As I mentioned, it's time to think
24:54
about it like when you have for example prediction logs of your
24:58
model. What it would mean is that you
25:00
are monitoring how people are asking questions and how the LLM
25:05
is responding. With such a metric, first of all
25:08
you can figure out what are the pairs of questions and answers
25:12
that the LM is performing poorly.
25:14
Your model is not performing well.
25:17
Now you have two choices. Either you can use maybe hue one
25:21
in the loop to come up with a better response right?
25:24
And use that as a next phase for preparing the data that can
25:29
remove the weaknesses of your model or create different
25:33
responses. And that's exactly the loop
25:35
there that we are talking about. It's like as you deploy the
25:38
model, you can observe and monitor how people are
25:41
interacting. How would I use those
25:43
interactions to constantly make it better?
25:48
Yeah, it reminds me of the DevOps Figure 8.
25:51
It's just got a little bit of a extra Figure 8.
25:55
And I know back in the day in the ML OPS community, someone
25:59
put together like an ML OPS Figure 8 with an extra figure on
26:03
it and now it it almost feels like that.
26:06
Or there's you deploy, you evaluate, you get the
26:09
information from the evaluation, which then helps you retrain the
26:13
model and so on and so forth. Then you deploy the new model
26:17
and and then you're able to do it.
26:19
You know, like I recently posted something on LinkedIn about how
26:27
the deployment strategies that people have or the ways that
26:31
people can deploy their models are sometimes a little bit funny
26:38
and wonky and especially maybe not as much these days.
26:41
But you can get people doing. You catch people doing something
26:46
little quick and dirty and it's probably not best practise, but
26:49
whatever it works, are there it within like Feta Mill?
26:53
Do you have ways to be able to test if a model?
26:58
If you're rolling out a new model, can you like Canary
27:00
deploy that or can you do ways so that it is like AB testing it
27:06
and then you're able to see, did we with our latest fine tuning
27:11
just totally disrupt this model or is it still good and do we
27:17
have confidence, like to deploy in confidence?
27:21
Definitely meaning that one component of it is exactly as
27:25
you mentioned as you are creating different endpoints
27:28
maybe. Let's say you have model A,
27:29
Model B. For each of those you create
27:31
endpoints. The first Hawkeye part is as a
27:34
developer is like you want to quickly check it.
27:38
OK, So what we offer is, for example, when you create your
27:40
endpoint, there is a playground already built for your endpoint.
27:45
So there you can quickly interact with it.
27:47
That's what I would call a quick check as a developer before
27:52
going and calling the endpoint in the application, right, right
27:56
there, right, ask the questions, make sure it is working fine,
28:00
maybe there are some issues, take the endpoint down, et
28:03
cetera. So that's like the first is
28:05
like, think about it, a playground embedded in your
28:09
endpoint creation. OK, so that's like the first
28:12
thing developer typically wants before going to the application.
28:16
You know, use the. The second part of it is to end
28:20
points rocking the question to both of them.
28:24
Alright so for example, as you are routing to both generating
28:28
the response and having your AB test, we call that the API
28:32
Gateway. Meaning that for your same
28:35
endpoint in the application, you can decide how to route the
28:39
question and maybe sometimes route the question to both
28:42
endpoints and generate the response.
28:46
This creates the data for your AB testing.
28:49
So that's what I would call is like the another functionality
28:51
having the gateway that you have for your endpoints and third it
28:56
becomes more of a analysing. That's where we give you the
29:00
prediction logs, looking at the responses and then you can make
29:03
your decision or even you can let your users.
29:06
For example we have a customer what they wanted in the
29:09
application having this gateway that automatically or maybe
29:13
randomly changes between the two and then they have a thumbs up
29:17
and thumbs down from their users.
29:20
So that way they are randomly collecting which is a great data
29:25
for the you know for the AB test with the human feedback which is
29:30
toned down, they are happier or they are not happy as they are
29:34
changing the model. Yeah, we had one bot in the MLB
29:40
Slack community and it had the thumbs up and thumbs down, and
29:43
then it also had the little mushroom symbol if it ever
29:46
hallucinated so that it could give feedback right away.
29:49
Like, yeah, it's not only that this wasn't helpful, but I think
29:52
it was totally made-up. And then it was, it was another
29:56
signal to to say to the the team, like, yeah, here's a
30:00
instance where the data is a little bit tainted or the output
30:06
is not what you would expect. And so you were mentioning
30:10
different customers like who have you been seeing using this
30:14
platform, Like who's this for? What have they been using it
30:17
for? Is it mainly just chat bots or
30:20
what else are they building? Yeah.
30:24
So let let me explain a little bit.
30:25
So first of all, as a platform, right, we try to attract many
30:29
groups. For example, this can be
30:31
individual developers, it can be startups who are about to build
30:35
their applications deployed and larger scale enterprises.
30:40
As you can imagine, each of them have different demands and the
30:43
reason they call, right? So for example, if you are a
30:46
star doll who are building your application, maybe to give you
30:49
an example of applications, let's say you are binding the
30:52
best general models for 3D modelling, all right.
30:57
Or maybe you are building the best general model that is doing
31:01
vertical LLM for example is going to be a great healthcare
31:05
summarizer because that's the solution you wanna offer.
31:09
Maybe as a SAS or ANONYM solution to others.
31:12
Typically as startups, you know the challenges that they would
31:16
be facing is scaling their solution, building the
31:20
infrastructure like similar to again chat, GT offering the
31:24
capability on their own model. What it would mean is they come
31:29
because they want to have the capability of a platform that
31:32
lets them deploy their models, scale their models, gives them
31:36
the safety layer for their models, right.
31:39
Maybe these are not the capabilities that originally
31:42
they have because the original capabilities that they are great
31:45
at building a proprietary model, but they are not as strong in a
31:51
scaling, deploying the solution and making it ready for
31:54
production, adding the safety measure to it or monitoring the
32:00
traffic, autoscaling it right again cost becomes a major
32:04
barrier. If they allocate many GPUs to
32:07
their model, they will be bankrupt.
32:10
They are sort of right. They cannot afford like ChatGPT
32:13
spending. What is it like a 500K a day on
32:17
serving the model. So they come, we help them with
32:20
their, you know the the, the way that we let people do across
32:25
multiple cloud multiple providers to lower the cost rise
32:30
for thermal cost for example is 4X5X cheaper than what you would
32:34
get on the computer from let's say major cloud providers,
32:38
right. So they can lower the cost.
32:40
They don't need Infra team. They can scale their solution
32:43
and deploy their solution getting it ready for production.
32:48
We see a lot of excitement there.
32:49
You know we talked to many as startups, they come, they deploy
32:53
their solutions with us for example.
32:55
And again you have a chat bot which is doing a healthcare
32:58
summarization. That's great.
33:00
We help you be the back end and the platform that you need for
33:04
deploying and scaling your solution.
33:07
Then as I mentioned, then gradually they start fine tuning
33:11
that, making the model better. We let them, OK, you can do all
33:15
of that, we need them. So that's how they stay and they
33:17
do more things inside. Then the other side of it is
33:22
like larger enterprises as I mentioned, larger enterprises
33:26
ownership and everything on premise becomes a critical
33:31
component. Now let's say you are a company
33:34
a lot of engineers want to use, Maybe a coding Co pilot, right?
33:40
But you don't want this data is so sensitive that you don't.
33:42
Typically they put it in a VPN0 connection to the outside,
33:47
right? So it's not about you know he's
33:49
like fully on premise platform for building the solution,
33:53
offering it to the employees etc.
33:56
We help a lot with that. That's the scenario that they
33:58
need on Prem capability to build it and then maybe offer it later
34:03
to their customers and then individual developers.
34:06
Typically they come because they need to have access to their
34:09
state-of-the-art models. For example, Loma Three was
34:12
released a few days ago, currently on Federal modern
34:16
hobbies available as a developer.
34:18
You can go there, access the API, use Loma Three, maybe
34:23
quickly launch an application with that, create an agent with
34:26
the, but they start doing more things with that.
34:29
So it's kind of people come for models, people come for the
34:32
capabilities and we also offer you know other features, for
34:36
example, let's say individual developers, maybe they want to
34:40
do other capabilities, for example having observability and
34:45
maybe the experiment tracking for their larger scale
34:48
distributed training, right. You want it meant scientist you
34:51
like, you need that capability. You can use that in Female, or
34:55
maybe you know you're working on Federated Learning.
34:58
Female is the only ML OPS platform that you can run
35:02
Federated Learning inside the mill OS without needing to give
35:07
you the full platform capability.
35:09
We'll talk to me a little bit more about that because I know
35:12
there is some history with Federated Learning in the
35:15
platform and you have an open source product, right, that is
35:19
in the Federated Learning space. So what's the evolution bit?
35:26
That's right. So actually you know the company
35:28
fed them well now we are two years old.
35:30
So currently is much broader services focusing on generative
35:34
alloy platform. But originally we started with
35:37
offering Federated Learning and there is an open source library
35:41
devoted to that. But she's actually currently
35:44
ranked one on offering Federated Learning.
35:47
For the audience who don't know about it, Federated Learning is
35:50
about decentralised machine learning.
35:53
Is like how would I train my model on distributed data
35:58
without connecting the data so it becomes a privacy preserving
36:03
method. For example, you want to train
36:04
an LLM on the data of the users across a smartphone, but you
36:09
don't want to collect that data and invade the privacy.
36:12
But still you want to offer an Lam that can tune to their
36:15
questions. For example, let's say Google
36:18
currently uses Federated Learning for the next word
36:21
prediction that you are using when you are typing on on your
36:26
phone. That's how they learn and create
36:28
a model that can predict your next word without violating your
36:33
privacy. So it's a very important
36:34
scenario. Typically it enhances I would
36:38
say bending models by accessing private data in a privacy
36:43
preserving manner. I would say maybe the reason and
36:47
actually we offer generalised services right now with the
36:50
foundation models, since they are trained on massive amount of
36:53
data, they are already doing great.
36:56
Like the bottleneck I would say in the first year hasn't been
36:59
more data, has been more of compute and the cloud offerings
37:05
to let you build the application.
37:07
That's why in Federman or focus has been daily over the past
37:11
year is like making sure that people can build their general
37:14
solutions with the cloud offering.
37:16
I would say as the time progresses, maybe a year from
37:19
now, my prediction would be that data becomes much more of an
37:23
important bottleneck because that's the way that you can
37:27
build much better applications, right?
37:30
They say you are building your vertical and your mode is going
37:33
to be the data you use to build that.
37:36
And I think it's becoming more important and people maybe you
37:39
know would be more interested to use Federated Learning as a way
37:43
to access more data for training and making the general
37:47
applications better. For now, we see a lot of
37:51
attraction means that there are many companies maybe because of
37:55
the privacy they're using Federated learning, they come to
37:58
us. So that's one scenario.
38:00
The other scenario is as I mentioned, many developers,
38:03
maybe they are experimenting, they want to have research,
38:06
figuring out how much this way of training a model can be
38:11
useful. You know for their scenario they
38:15
can come and try that on Federman and deploy that.
38:18
Yeah, it does feel like there's like steps to it as far as
38:22
maturity goes. And so you laid them out nicely
38:28
there. When it comes to OK, the first
38:30
step or the first barrier that you have to overcome is just
38:34
getting something out working. Then you start to get the
38:38
evaluation. That's like Step 2.
38:40
Then you start to say alright I want a more verticalized model,
38:44
I need more data than just my evaluation data.
38:47
To fine tune this. How can I start to create a
38:52
little bit more pointed solution and is that a good
38:57
representation of of what it is like is did I miss any of these
39:02
different challenges or steps that you've been seeing on this
39:05
maturity journey? No, actually I loved your
39:08
Demetrius. It's kind of first is deploying
39:11
the model right, putting it in production, offering your
39:15
solution in a way that is ready for production.
39:18
People come to us, we help. Then you evaluate, monitor,
39:23
figuring out how is it doing so that you can evaluate it and
39:27
then build the next version. Typically that version you fine
39:30
tune it, maybe you centrally improve that, etc.
39:33
I would say in the future exactly is going to be maybe
39:36
Federated across many of your users by using their private
39:40
data. So that can be maybe a future.
39:42
But for now, we see a lot of the demand is maybe on the first
39:45
three steps, putting it in production and deploying the
39:48
model, observing it, figuring out you know how it has been
39:53
used, fine tuning that and putting it back in the product.
39:57
Yeah, I do like this vision of. It needs to go Federated, but
40:02
maybe it's not right now that it needs to be there.
40:06
Right now, there's bigger blockers than what Federated can
40:11
offer. Correct, especially for the
40:14
general, I would say that's what I mentioned maybe next year is a
40:18
become a great sport de Janeiro. But for many other models like
40:22
as I mentioned like a small model for a next word prediction
40:25
or maybe a model for different applications, there is demand
40:29
for it, right. So that's what we are seeing,
40:32
but generally correct is less. I've think that you are open to
40:39
talking about it, but you've been brewing up some stuff
40:41
secretly at Fed ML. Like what are you working on
40:45
that will be your next big announcement or development?
40:49
That's right. Actually you know something
40:52
special about our companies. We have an extremely fast pace,
40:56
right. Even imagine in these two short
40:58
years we have been going from Federated learning now full
41:02
general solution etcetera. We are moving very fast.
41:05
Something we are very excited about is soon we are releasing
41:09
our own foundation model. OK.
41:12
So meaning that our vision is platform providing platform to
41:17
developers, enterprises, companies together with the
41:21
foundation model of course. You know in federal modern hall
41:25
we host a variety of open source models.
41:27
But the goal of our own foundation model that we have
41:30
been training from scratch right is the vision is aligned with
41:34
the way that we want to bring ownership to the people.
41:38
So what do I mean by that? So first of all our vision is
41:41
that this foundation model we are focusing on a small Elms as
41:45
opposed to 70 billion or maybe 100 billion parameter models or
41:50
focus is to having the best small models, let's say the best
41:54
2 billion, three billion, 70 billion parameter model.
41:58
And the the biggest announcement we are gonna have soon is going
42:02
to be alongside of revealing that model that is going to be
42:06
ranked you know in the top in that sense of the performance in
42:10
the smaller scale and Elms. Now why that, what is the vision
42:14
for it. So you see if our vision was
42:17
that we discussed is the ownership right, you have to
42:19
work with some foundation models that you can afford to maybe
42:25
dedicated deployment of it in a low cost manner on reasonable
42:29
GPUs etcetera, right. So that's where a smaller scale
42:33
model is great for ownership. All right.
42:36
So let's say again, you are a startup, you're a company who
42:39
are building your own solution. If you can get a great
42:43
performance with a small model, right that is very well aligned
42:48
with all your idea of ownership because the maintenance of it,
42:51
the deployment of the, the training of it, improvement of
42:54
it all makes sense to you as opposed to the biggest model,
42:59
right. So it's kind of we are seeing
43:02
again for ownership you need the right size model, typically
43:05
smaller of course the performance should be high, but
43:08
that's where we want to empower people to build more own more.
43:13
The other side of 80s, I would say again right now a lot of
43:17
foundation model building or the applications that are built on
43:21
top of that is cloud driven, right.
43:25
I would say in a year or so there is going to be a lot of
43:27
demand that you are also seeing a bit brewing this year on on
43:32
device, generally solutions fully on device.
43:36
For example, let's say you have a fully on device AI agent on
43:39
your phone that is doing and helping you with many tasks.
43:43
So in those scenarios, again you cannot go with a large model and
43:47
also aligns very well with our vision of Federated Learning
43:50
which is pushing and bringing e-mail development to the edge,
43:55
right. So we are integrated edge cloud
43:57
platform or foundation model lets you easily go from cloud to
44:03
the edge and for that you need such a foundation model that is
44:08
great for the cloud, great for the itch.
44:11
OK, so that's our vision, meaning that this foundation
44:13
model optimised offered on Federman leading people to only
44:17
letting people to, you know, making low cost as well as
44:21
letting people to bring it on device.
44:24
And with that we are also revealing a new concept, you
44:27
know federation of models. So federation of models for
44:32
example, you have heard about mixture of experts.
44:35
Mixture of experts is a way to combine moderns as a way to
44:38
create for example, maybe you know mixed run right, you
44:42
combine smaller 70 billion parameter models to create a
44:45
very strong model by combining several of them.
44:48
Model federation is also our angle on how to combine a
44:52
smaller scale models maybe for a cloud solution that is competing
44:57
and becoming a very strong collective model that competes
45:02
with the large model. So these are the two, you know,
45:05
recent innovations that we have been brewing in the past couple
45:09
of months and we'll be announcing soon.
45:12
I think everybody would love to go back to or.
45:16
Everybody understands, at least conceptually, the benefits of
45:21
smaller models. And so if you can prove that for
45:23
your specific use case that a much smaller model can do the
45:27
trick, it's a no brainer to be using that smaller model, just
45:33
because it's so much easier on so many different dimensions,
45:36
right? Yeah.
45:37
And I I do like this idea that you're talking about of pushing
45:42
LMS to the edge. I think I recently just saw a
45:45
demo about someone doing this with Roma Three that was
45:50
quantized or distilled. I can't remember exactly it and
45:54
I'm pretty sure it was the the small Llama Three.
45:57
Like how are we going to be interacting with these LM's on
46:01
our phones? Because that's my big question
46:04
is if it's just the chat bot, I don't really see the utility in
46:08
that quite yet and there's a lot of moments that I would rather
46:13
use my as an that way to navigate my phone.
46:17
So I don't know if you've seen anything that's that's
46:19
interesting in that sphere. Yeah, I would say, I think the
46:23
biggest capability would be having the agent on the phone.
46:28
So what is this agent? So this is the agent that
46:31
interacts with many API on your phone to do a multi step task
46:36
for you. So for example, let's say this
46:39
agent right can interact with many API's.
46:42
For example, maybe your websites, your various accounts
46:46
you have and you would say maybe planning something for you.
46:50
For example, find a meeting time for me that works for me and
46:56
Dimitris and send an e-mail on the response.
46:59
Multiple steps the agent writes simplifies you.
47:03
OK, so essentially I would say the role of it would be that
47:07
typically if you want you can do many things on your phone.
47:09
They require a lot of steps and interacting with many different
47:13
apps. This can streamline the process
47:17
for you do many tasks and automating it for you, right?
47:22
So that way you can maybe again you can make it simple.
47:26
It's like you know, you ask the agent right?
47:29
This is the topic I want to have a summary of all the maybe ML
47:34
OPS companies that are dealing with, you know, building and
47:38
deploying LMS. For those you know, list the names, provide
47:43
the one page, one paragraph summary and create that on a
47:46
note and e-mail it to me. Very good.
47:49
So it goes what the website does, the search, gets the
47:52
summaries, put it on a note that summarises that create the
47:57
e-mail and send it, that's great.
47:59
Like it's your own assistant. So meaning that the idea of if
48:03
we believe that AI assistants are gonna be everybody's gonna
48:07
have their own AI assistant, that they make it better on the
48:12
phone is the right place because phone is always with you.
48:14
How do you interact with your AI system?
48:18
So that's what I see as the angle.
48:20
Ohh, OK yeah, I hope that day comes about.
48:26
I wonder how far we are from that because of the demos that
48:31
I've seen with agents these days.
48:33
It feels like it's still a little bit of a ways off.
48:37
And I don't know if it's a few months off, a year off or 10
48:41
years off. You know, for us to get to an
48:44
ability where agents can be reliable and I can trust them to
48:48
have my credit card and buy me a plane ticket and a hotel.
48:52
And it's not going to like buy me a plane ticket to Bali when I
48:55
want to go to Barcelona or something.
48:57
No, no, you're right. I mean, there is going to be
48:58
verifications in the middle, meaning that depends on the
49:01
sensitivity of the task. Some of them you can trust, some
49:04
of them you can, but I think quicker ones are going to be
49:08
many content creations on the phone.
49:10
So that one you don't want to do it on the cloud because of the
49:14
cost and moving that is very well aligned.
49:17
So those, I would say immediately coming content
49:20
creation on the phone, right is very important.
49:23
For example, let's say advertisers want to use that to
49:26
create contents, right? So for example, let's say
49:28
personalised advertising, when the content is on the fly
49:31
generated for you or maybe you want to create a various content
49:36
like modifying your image etc. Not all of them need to be a
49:40
cloud service. It can happen on your phone, for
49:43
example, you take a photo, that photo you want to interact and
49:47
say maybe you know, improve the image, put it next to the other
49:50
ones, etc. Such applications, consumer
49:53
applications are currently available.
49:56
You do a lot of that on the cloud, but imagine somebody
49:59
offers that application completely on the phone.
50:01
Lower cost even, you know, can be much cheaper.
50:05
Because if I'm offering that on the clothes, you have to
50:08
compensate my cloth cost. If it is on the phone, you just
50:11
need to compensate my innovation.
50:13
You're using your own compute, so I would say those are
50:18
quicker. Immediate, right?
50:20
Some of those generations instead of on the cloud pushing
50:23
it under up costs will come lower.
50:27
Cloud costs would be removed and people maybe feel also safer
50:31
because your day, your image for example, is not leaving your
50:34
phone. Still, you can create exciting
50:37
images with it, right? Privacy is going to be amplified
50:41
that we call immediate. The other ones, you're right.
50:45
I mean, these agents need to be evolved, people need to train
50:48
them becoming embedded in your life, etc.
50:50
That becomes more of a future. Maybe next year if we have
50:54
another podcast together, we will reflect on those.
50:58
Yeah, I hope it's next year. That would be nice because I'm
51:01
ready to stop working as much and let the agents work for me.
51:05
As I wonder though, yeah, that's I think the biggest thing that
51:09
is on my mind because there is a huge movement for agents and
51:14
everybody saying, you know, agents are coming, everybody
51:17
that plays with them, they really are trying hard to make
51:20
them work and be reliable. And it is very obvious to me,
51:25
this vision of if we have agents, what better place to
51:29
have them than on our. Exactly.
51:32
Right. Like, and why not have that all
51:34
be Federated? Because the phone and like
51:38
Federated Learning go hand in hand.
51:40
Exactly. It's kind of like when I think
51:43
about Federated Learning, I think about my phone.
51:45
Definitely. I think that's the right way to
51:47
think. Yeah, So, well, I love this.
51:50
I appreciate you talking to me, Salman and giving me the whole
51:54
rundown on what Fed ML is doing, the Gen AI platform that you've
51:59
created, the loop of how things work within the platform.
52:04
And so if anybody wants to check it out, I encourage them to
52:09
reach out to you on LinkedIn. We'll leave a link to your
52:12
LinkedIn in the description or also just go to Fed ML at
52:15
website and check that out, it's in the description.
52:18
So wonderful pleasure talking to you, Dimitris.