Latest / Agentic Conversations (formally mlops.community) / From Virtualization to AI Integration // Lamia Youseff // # 175

Transcript

From Virtualization to AI Integration // Lamia Youseff // # 175 · Agentic Conversations (formally mlops.community)

0:00
My name is Lumia Yusuf and I'm executive director at Jazz
0:03
Computing and visiting lecturer at Stanford Graduate School of
0:08
Business, and I take my coffee with milk, espresso with milk in
0:12
the morning. Welcome back to the ML OPS
0:16
Community Podcast. I am your host Dimitrios, and
0:19
today we are talking with none other than Lamia.
0:22
I'm flying solo. Got no co-host today, but I
0:26
don't really think I needed one because Lamia was such an
0:30
incredible guest and how could you not be?
0:33
Let's just give a bit of context around what she's been up to.
0:37
She's been in AI and ML for the last 25 years.
0:41
We dive all into that. First, she was in academia.
0:45
Then said, you know what, I'm gonna go and make a change in
0:48
the world and implement some of this stuff.
0:50
Actually, I think she was recruited.
0:52
She was heavily recruited because why wouldn't you be?
0:55
Again, if you're that good, you gotta get out there and make a
0:59
change in industry. She went and worked for every
1:02
single thing that you can imagine.
1:04
Google, Microsoft, Apple and Facebook.
1:07
That's basically all of them right there.
1:09
Boom. And now she is doing some really
1:14
cool stuff around helping startups in generative AI.
1:18
And so if you are one of those people that is starting
1:22
something in the generative AI space or just the LLM space, you
1:26
should reach out to her. Because she has got some
1:28
incredible experience and advice.
1:31
And my favourite thing from our conversation was when she talked
1:36
about how she was on the Google Cloud team in.
1:41
The early twenty 10s and how her and a handful of other people
1:46
created Google Cloud. That for me is like history in
1:50
the making, Its recent history, but it's still history and I
1:53
found it fascinating. So we dive deep into that.
1:56
We then. Go into what she is doing at
2:01
jazz computing and what the mission is and how she sees
2:06
parallels now with the AI and ML movement. 2.
2:12
The cloud computing movement in 2011.
2:17
And we finish U with what I thought, took me back a little
2:20
bit, but she was talking about how Kubernetes got its start,
2:24
you know in 2015 or 2014. I can't remember the exact date
2:28
and I should know that again the history part of me just goes
2:31
right out the window. But she talked about how there
2:34
is no real unifying layer that you get in the ML world.
2:40
And so we pontificate on why that may be and how that can
2:44
change. I mean, we don't have any
2:46
Googles going out there. Throwing their whole weight
2:49
behind a certain standard like Kubernetes, will it ever happen?
2:54
I don't know. That's a big question to be
2:56
solved by the community. So enjoy this one.
3:01
And if you can do one thing, it would mean the world to me if
3:04
you just tell one friend, one friend about this podcast so
3:09
that we can continue to have it grow.
3:11
Let's get into the conversation with I'm really excited about
3:21
this conversation because I feel like you have had just about all
3:28
the experience, whether it has been in academia or working for
3:33
gigantic companies, the ones that we've all heard of and
3:37
maybe love or hate. And then also the experience
3:41
working with startups and smaller companies that are maybe
3:43
more resource constrained and so.
3:46
I I look forward to tackling all of those different areas and
3:51
pieces of your life, but it's probably worth starting.
3:56
With how you got into the field and specifically you started
4:01
down the AI path 25 years ago? Before it was cool.
4:07
Way before it was cool, I would say.
4:10
What was that like? Ohh.
4:13
It's a lot of really interesting stories.
4:16
So I grew up in Egypt, in Cairo, and back in the days I remember
4:21
my mum, she was working on her dissertation and in order for
4:24
her to be able to write down that dissertation, she have to
4:26
handwrite it and then take it to the typewriter.
4:29
To the local typewriter, which is a shop in another
4:33
neighbourhood. You have to travel an hour and a
4:35
half in Cairo traffic in order to be able to sit there beside
4:39
that typewriter, the person who writes the handwritten notes
4:42
down into basically a. Typewrite.
4:45
Oh wow. So she didn't even write it.
4:47
She had somebody else that had to write it and knew how to type
4:51
and it was an hour in traffic hour and half each week.
4:55
And my family at that time decided that it's time to get a
4:58
computer and it was about $10,000 for Windows 3.
5:02
Leven we got it was huge. Who put it in their living room
5:06
and this corner it was just like amazing, very shiny and
5:10
exciting. And we were allowed, my brother
5:14
and I, we were allowed to play on this computer for only one
5:17
hour a day and very careful and, you know, like basically, you
5:20
know, keep it clean. And but then we heard about
5:23
computer viruses through the local newspaper that was
5:25
delivered in our home. And we're like, oh, no, we
5:28
should be very careful. Of course, you know, like it
5:30
wasn't connected to the Internet, the only way that a
5:32
virus could get in later on or through, you know, like a floppy
5:35
disc or something. But but we were very careful.
5:38
You were very alarmed. So every time we open the
5:40
computer we check all of the files as much as we can,
5:43
especially the ones on the desktop, until we saw a file
5:47
that's called Chapter 7 and another file a couple of days
5:50
later called chapter five with double click on it.
5:53
We don't understand the word in it that was the Word document
5:56
and we agree that this is likely the viruses or the virus that
6:00
they were talking about. So we start deleting it.
6:03
Ohh no the first file. We delete the second file until
6:07
few days later when we hear our mum on the phone talking to the
6:10
computer specialist saying that the computer is deleting her
6:14
dissertation. Right.
6:18
Looking at how a moment, you know, like what's going on.
6:23
So this is how we got into computers.
6:25
And I went to study at the computer science department in
6:29
the American University in Cairo.
6:31
And there was this young professor that came back, just
6:34
came back from the US who did his work on fuzzy networks or
6:38
artificial neural networks. And after taking the
6:41
introductory AI class that was talking about expert networks
6:44
and expert systems and all of, you know, like the more common
6:49
AI concepts, I got me student in that class, artificial neural
6:54
networks, be able to learn about the fascinating field of AI and
7:00
deep learning. And I remember very vividly my
7:04
mom's coming back from a doctor visit to one of those days when
7:07
I was taking the class and she shared that the doctor suspects
7:10
that she had breast cancer and she had to go through biopsy and
7:14
so on. And at the same time in the
7:16
classroom we're taking a our moment was to build a neural
7:21
network, a basically a very basic predictor to be able to
7:26
predict benign versus malignant breast cancer.
7:29
And in my mind it was wow, if we can do that using computer
7:33
science and using very basic tools in my mind of being able
7:38
to train a network and being able to actually possibly save
7:43
lives by predicting and by being able to detect breast cancer
7:47
early on, I can save my mum's life.
7:49
I can save many woman's life. Luckily and fortunately, my mom
7:55
skins or tumours turned out to be benign and she's healthy.
7:58
She's doing good. So it was really good.
8:00
I got introduced to the world of AI, machine learning and and
8:04
back then when I was writing this homework assignment was
8:08
written in C The connection between the different neurons
8:11
were were C pointers and in order to parallelize it I had to
8:15
borrow my dad's computer while he was sleeping.
8:18
My brother's computer wife moving the code on floppy discs
8:22
in order to run it. I mean it was not connected over
8:24
the Internet. All of them.
8:25
We had only one online, one dialogue line, so only one
8:28
computer can be connected at a time and being able to just run
8:32
and explore the amazing space of AI and machine learning.
8:36
And of course, when I have a pointer C pointer that didn't
8:39
point to the right neuron, it's a memory dump and the computer
8:42
just restarts it. So once we're extremely painful
8:45
at that time and got very excited about this, became a
8:51
research assistant at the time in this specific area and
8:53
decided that I want to go to the US, come here to the US in order
8:57
to be able to continue the study of this fascinating field at the
9:01
graduate level. However, what I found when I
9:03
applied is got I got the advice that AI is a dead field.
9:09
Don't waste your life something. Do something else usually.
9:11
Wait a minute. Am I confused?
9:13
Alright, so I was like asking as as as a young researcher, what's
9:18
the problem? Why not?
9:20
And they lack lack of computational resources like OK
9:22
let me take this problem on and then you know I will come back
9:25
to machine learning later. So that's how it led to me
9:28
working on cloud computing and scientific code and scientific
9:32
basically computing in the early days getting me to high
9:35
performance computing, getting me to virtualization on cluster
9:38
computing which turned out to be one of the early days of cloud
9:41
and cloud computing. So writing papers in this field,
9:44
trying to share their knowledge and trying to share the
9:47
landscape of how things are developing back in 2000 and
9:50
seven 2008 at that time. Wow.
9:53
So you saw from the get go. A practical application of AI,
10:00
and you saw that it could actually be used to.
10:04
Help save lives, and I'm sure you've kept an eye on all of the
10:09
advancements that are happening within the health fields in AI
10:13
and the ability that it has now. And I imagine you would have
10:20
never have thought back in those days its capabilities and what
10:24
it's able to do, but it. It's almost like that inspired
10:29
you to take the plunge and then go deeper down the rabbit hole
10:33
and you found different bottlenecks along the way, and
10:38
one of these bottlenecks. Was this compute resources?
10:42
And so that got you into the cloud and the cloud was a very
10:47
new thing at that time, right? Like what was that?
10:51
Like because it's almost. It's funny it what in 2007 the
10:57
cloud was in actual thing where they're already the big players
11:03
offering the cloud like was a WS one of the did they have AWS?
11:07
Yeah, I don't know the timeline on that.
11:09
Actually. I think it was 2000 and six
11:12
2007, maybe four 2006 when we were invited.
11:16
Some students were invited to write some code and run it on an
11:21
Amazon system. They didn't tell us what's the
11:23
system, it just remote and we're becoming beta tester for that
11:27
system and we're just seeing you know like how it's going to
11:29
work. So I believe with the time I
11:32
just, I don't even remember what was the application, but I was
11:34
kind of you know like trying to find about the system through
11:38
the combat, trying to see you know like where things are.
11:40
And it was clear to me it was a virtual machine running on a
11:44
remote computer. And then it turned out later on
11:46
to be EC2 and we're just testing EC2.
11:48
They didn't tell us what we were testing just like basically
11:51
running some basic codes. But then through the next year
11:54
we just you know like and became more and more interested in this
11:57
capability. So as you see Santa Barbara at
12:00
the time where working on cluster computing and I was
12:02
working with virtualization. So I'd intersection of both.
12:05
We build a system that's called Eucalyptus back in the days and
12:09
Eucalyptus basically is a product cloud.
12:11
And I mean we're jumping around, we're very excited because it
12:14
basically like the media as soon as it got out in the tech crunch
12:18
world it was like this is the Amazon EC2 equivalent, this is
12:22
you know in the private cloud, this is what's going to change
12:24
the world. So it was the very exciting
12:26
times. What all we did was to get the
12:28
APIs from AC2 and from Amazon AWS and put it on top of our
12:32
cluster that have virtual machine running behind it.
12:36
Later on it was it was integrated into Red Hat.
12:39
The team decided to build a company around it.
12:42
Unfortunately I wasn't able to join them because of my role.
12:45
I got the opportunity to go to MIT and to work on and further
12:48
research. At that time, my my aspiration
12:51
was to go for an academic role and it was very hard to say no
12:54
to MIT. So it's definitely you know like
12:58
they they tried to take it as fast and as far as they could.
13:02
Having said that, it was really early days of everything that
13:05
was happening in cloud and everything that you know like
13:07
the ideas forming and and things coming together.
13:10
What changed in you that made you?
13:14
Not go for academia or a lifelong academia pursuit.
13:20
It was difficult to make that decision, but it was Google
13:23
Cloud. Really.
13:25
Well, yeah. 2011 Google came and they told me at the time I was
13:32
in Boston at MIT and they said we have this exciting project
13:36
that's going on in Seattle. We want you to move next weekend
13:41
in order to be able to join this project and I thought asking So
13:46
what project it is? Why do we need mean?
13:49
What's going on? Can you tell me?
13:50
And no, it's all confidential. You will know when you come.
13:54
Was excited and intrigued, and so I decided to go.
13:58
And my thinking at the time was I'm going to go there for six
14:01
months, build some research connections, learn something new
14:05
and come back to my position at MIT and might basically continue
14:09
my academic rule. The confidential project turned
14:12
out to be Google Cloud. I was one of the founding
14:14
engineers over there and it was amazing.
14:17
It was so much fun just seeing the very early stages and very
14:22
early basically like building technical capabilities that's
14:26
going to take over the world, if I may say, but also be used by
14:30
millions and millions of users out of the door and building
14:35
really production system that's very different than research
14:38
system. So in our research system,
14:40
usually what we do is that we push the design space by
14:45
introducing new ideas, writing and building a prototype,
14:48
especially in system research and publishing it with the hope
14:51
that's going to be adopted and is going to push the frontier
14:54
one step at a time. Of course, you have the ability
14:56
to be able to to dream big and to design new things and to
14:59
think about new systems in industry.
15:02
It's not about pushing as much with the design.
15:04
It's more about finding and building systems that's going to
15:07
scale, that's going to solve problems, that's going to have
15:10
product market fit and that's going to take us to the next
15:13
level, which is something that I really enjoyed doing at Google
15:17
in the very early days. And Google is such an amazing
15:19
company, especially back in the days when we were 12 engineers
15:23
working on Google Cloud Engine. Just 12 it was you know like
15:28
they put us in a corner hidden under the Seattle office that
15:31
happened to be under a bridge and they told us you know like
15:33
if you if you go and talk to anyone don't tell them that
15:36
you're working on the cloud. So we're very secretive and
15:39
amazingly, I mean basically one of the fun stories I remember
15:43
from that day actually that was our our head of engineering at
15:47
that time went on and build Copernicus later on.
15:51
Craig McCluskey was the head of product for the cloud engine.
15:55
So Attract also was one of the founding member of Eucalyptus
15:58
later on and went on to have a lot of really good influence on
16:01
the whole open source and virtualization world and
16:04
container world. But one of the funny stories I
16:07
remember is the team of 12 were trying to find a meeting room in
16:11
that in that Seattle office, and we couldn't find the meeting
16:15
room. That's enough for 12 people.
16:16
All of the meeting room was small and our biggest concern
16:19
was when the team grow to 20 people, what are we going to do
16:24
with meeting rooms? It's going to be proper looking
16:26
today, you know, like I don't know.
16:27
They are 10,020 thousand, I don't know how many engineering
16:31
in the organisation. I hope they have enough meeting
16:34
rooms. That's gonna be enough for you,
16:35
for everybody. In those days, there was no
16:39
Kubernetes, right? You mentioned how Joby was part
16:43
of the team. And.
16:45
Was there Docker? Like, what were you building
16:49
there? There was no Docker.
16:50
So like, paint the picture of what was around and what you
16:54
were building back then and how it looked.
16:58
There was virtual machines. I don't think we had containers
17:02
back then. How is this the era of VMware
17:06
blowing up and being seared? Not even.
17:10
I think the VMware came in the very early stages of of
17:14
virtualization and because of the performance overhead back
17:18
then, so it wasn't as much adopted as a virtualization
17:23
technology within infrastructure.
17:25
I might be wrong here, but this is my understanding and
17:28
therefore the cloud provider. There was then the part of
17:31
virtualization layer that was adopted by AWS in the early days
17:36
as insurers became a company in the US.
17:38
It was initially a research project that came out from
17:40
University of Cambridge. And then basically building a
17:43
very small part of virtualized layer that allowed a lot of
17:46
interconnectivity between the user space and the system space
17:50
underneath it, which is allowed it to be able to improve the
17:53
performance and be able to provide a lot of the competition
17:56
and resources without adding overhead or virtualization.
17:59
And actually this is some of the work that I've done at UC Santa
18:02
Barbara studying that performance implication of this
18:05
kind of art of virtualized systems.
18:07
I don't remember exactly what was the stat like at Google.
18:10
It has been at least 10 years at this point.
18:12
So. But what I remember is that is
18:15
it was using all internal tooling system including Borg as
18:19
one of the systems and trying to come up and alley internal
18:24
technology from encryption. So how do we encrypt the disc
18:28
from starting the VM, how do we store the VM, how do we decrease
18:32
the time? And this was actually one of the
18:34
features that I worked on. How do we decrease the startup
18:37
time for the VM such as that when the user comes and
18:39
basically you know like create a new VM then boom, they have a VM
18:43
started right away. We started from something like 3
18:46
minutes. We got it down into 1012 seconds
18:51
if I'm not mistaken. I'm sure now it's much shorter
18:54
than that. But just this is one of the
18:56
challenges that we had in the early days.
18:57
I mean users are not going to sit there for three minutes
18:59
waiting for the VM to that was you know like or encryption.
19:04
How do you encrypt the discs such that you can create some
19:07
isolation between the different VMS that are running on the
19:09
scene on the same physical hardware?
19:12
So that was another challenge, very basic challenges, but this
19:16
was our world back then, which is building the small components
19:20
one at a time and bringing them together in order to be able to
19:22
scale to the next level. And as you see now with the
19:28
cloud and what is almost like taking for granted, how do you
19:33
view the evolution from then until now And is it because I
19:37
just think about have we kept things that maybe didn't need to
19:44
be kept or are there was there pieces in that original?
19:49
Architecture or your original designs that have stood the test
19:52
of time? Or has everything kind of been
19:55
replaced? Since then?
19:58
It has evolved, so it's definitely an exercise of
20:01
evolution. I remember writing this paper on
20:04
2008 on the ontology of the cloud and basically breaking
20:09
down the different component it was with in collaboration with
20:12
IBM Watson. At that time my mentor Dilma De
20:14
Silva was one of the early engineers and early folks who
20:18
have built key fully two one of the early operating systems that
20:21
are also had a lot work. Cloud had a lot of, you know,
20:24
like aspirations and inspiration from rooting in.
20:29
But in this paper I identified basically different layers.
20:33
One of them is Infrastructure as a service, another one was
20:36
Storage as a service. I believe the cert one was
20:39
Communication as a service, what I call CAS that turns out to be
20:42
network, cloud, networking layer and then basically the
20:46
application layer. There was no virtualization
20:48
layer over there. There was no orchestration over
20:51
there between the different layers.
20:53
So I mean basically Copernicus did not exist back then.
20:56
There was the closest that we remember from those days was
21:00
basically you know like virtualized some way of kind of
21:02
you know keeping track between the different VMS because
21:05
remember back in, I mean we're talking 2000 and eight 2009, The
21:09
biggest cloud deployment we could imagine for a private
21:14
client would be 5 VMS, 10 VMS, 20 VMS maybe so that.
21:19
Once you basically how far we could go with our thinking until
21:24
quickly during especially my time at Google, I got to work
21:26
with customers. So I moved from engineering to
21:28
product management because I really wanted to know how
21:31
customers are using the technology that we have built
21:34
and got to work with some of the biggest customers that were
21:37
migrating back then to Google Cloud.
21:41
And very quickly got to realise that actually the use cases are
21:44
very different than what we thought about in the very early
21:47
days. So I remember one of them was
21:50
believe December 2011 or 2012 when there was a big outage for
21:57
AWS and customers very quickly realised that specifically one
22:02
of the streaming services, one of the biggest streaming
22:04
services that build on that cloud from the get go.
22:07
They quickly realised that actually in order to be able to
22:10
provide their end user with significant availability and
22:15
always be available during Christmas times or peak times on
22:18
their usage, they have to deploy on different clouds.
22:21
By the time they were looking to migrate between AWS and Google
22:26
Cloud later on I went to Microsoft and they realised that
22:29
they are also migrating to Microsoft.
22:31
So they were building this in between the different clouds and
22:35
basically that's the multi cloud use case and they have
22:39
deployment. So one of the projects that I
22:40
worked on was building a very fat vibe between the storage
22:44
system at Iowa and the storage system at Google Cloud in order
22:47
to be able to enable them to move, I believe 10 terabytes of
22:51
data every night. At that time, 10 terabytes was
22:54
wow, can we do that? Yeah, So it was.
22:58
It was just you know like very early days on putting together
23:01
the different components and bringing it back to the end user
23:05
in order to be able to build the virtual use case, the high
23:09
availability characteristics that we want in a cloud as well
23:11
as the coordination and orchestration between different
23:14
cloud systems that happen to be working and deployed on a multi
23:19
cloud environment. And so you mentioned something
23:22
before we hit record about how you see so many parallels with
23:27
the new AI movement and the old or the early days of the cloud
23:32
movement. And one that I was just thinking
23:35
about as you were talking about this is the amount of time that
23:39
people are willing to wait for this novel thing.
23:42
People aren't really willing to wait 3 minutes to spin up their
23:46
virtual machine, but they will wait 10 to 15 seconds and it's
23:50
kind of the same. With their answer, if you prompt
23:54
the AI these days, they're willing to wait a good 10 to 15
23:58
seconds, but after that it gets a little old.
24:01
Or maybe you're not willing to do that consecutively.
24:05
What are some other parallels that you are seeing now verse
24:11
that absolutely whenever I watched the whole interaction
24:16
that's happening today between startups and cloud Titans and
24:20
the VC world and the whole ecosystem, it just it's deja vu
24:25
2000 and 2013 early days of cloud.
24:28
And first of all that the competition between the
24:31
different cloud providers of was going to be able to basically
24:35
win the market when in my mind it's going to be a multi market.
24:38
And that's what we saw a multi you know like a multiplayer and
24:42
that's what we saw with cloud. So basically today we see each
24:46
of the different cloud providers trying to influence their users
24:49
by saying you know we have the best AI capabilities, we have
24:52
the best use cases. It's going to be very similar in
24:55
my mind in terms of users having to use possibly several of them
24:59
at the same time in order to be able to bring the application,
25:02
the application to the use case. So I'm waiting to see an
25:06
orchestrator similar to Copernicus that's also going to
25:08
going to work with. The AI will include at the
25:11
infrastructure level, so that's one thing that I'm keeping an
25:13
eye for. Can we double click on that?
25:15
Because that's something fascinating.
25:17
I was talking to a friend, Raul about the other day on how
25:20
there's not really a unifying layer right now for these AI
25:24
workloads and. It feels like you could build
25:29
some kind of abstraction on top of Kubernetes to get.
25:34
To kind of abstract away all of these headaches and these
25:38
thousand paper cuts that you get by trying to do ML workloads or
25:43
AI workloads with Kubernetes, and then you have some kind of
25:46
an operating system there. But what is like, what is your
25:51
view on that? Because I love this idea of
25:54
yeah, there's Kubernetes which has been super successful in
25:59
DevOps, but is there another? Kubernetes that we build on top,
26:04
that's specific for ML OPS, possibly.
26:08
If you bring together the ideas around machine learning,
26:12
platform alization, whether it's normal flu or the end to end
26:16
infrastructure and components along with the cloud and how you
26:20
diverse the different capabilities that's being
26:23
basically provided by different cloud providers in different
26:27
areas or in different in AI specific capabilities.
26:30
In this case, along with the ability to be able to scale,
26:34
have high availability, low latency services in between all
26:37
of these you'll find that we need something that's unifying
26:41
all of these and that can seemingly coordinate between
26:45
them. I've seen some use cases in
26:47
machine learning workloads where you would take your workload and
26:51
train it on GPUs, but then when you're ready to run it for
26:55
inference to put it in production, you would move back
26:58
to Sage Maker for example. Or you would use the certain LLN
27:03
in a let's say Microsoft Azure, but then the fine tuning would
27:08
be done somewhere else. Yeah.
27:11
Hey everyone. My name is Aparna, founder of
27:13
Verizon, and the best way to stay up to date with ML OPS is
27:16
by subscribing to this podcast, just because of the capabilities
27:22
that's being available here or there, that's not unified across
27:25
the different players. So there is a need for one layer
27:29
that's going to connect between all of these.
27:31
Copernicus was successful at the infrastructure level, but if we
27:35
take it 1 abstraction level U, then there's another layer
27:39
that's still being developed today by somebody.
27:42
I'm keeping an eye, you know, if you hear of any, please do let
27:46
me know. Yeah, where we can build this
27:48
unified infrastructure and machine learning platform that
27:53
not only hides all of the details of which cloud you're
27:56
running into what capabilities that you're using, rather than
28:00
its only provides higher capability with the engaged
28:05
business use case in mind, which is what Cooper needs to have
28:08
done in this case when it comes to infrastructure and to cloud
28:12
infrastructure. Well, I I love the idea of how
28:16
Kubernetes has effectively enabled a whole ecosystem of
28:21
tools that can be built and billion dollar companies that
28:26
are made because of what Kubernetes enables and it feels
28:30
like. Ever since the beginning of the
28:33
Milos community, one thing that we didn't have was that unifying
28:39
open source project that all the rest of the tools can be built
28:42
on. And there's no CNCF, There's no
28:45
we. Maybe that's a little bit on me
28:48
because I'm lazy and I didn't end up creating a big foundation
28:52
that we could have all of these different tooling companies be
28:55
part of, but it always felt to me like there was a whole or.
29:02
We are missing. Some open source project that
29:06
will help at least. Get.
29:10
What One of the creators of Cube Flow when we talked to him.
29:13
David Aronchick. You may know him from
29:17
underground. So.
29:20
He said that. We need to create contracts with
29:26
a small C That was his thing. Like, not his hard requirements,
29:31
but something where it is clear and you do have a way to plug
29:39
into an ecosystem. The only thing is, I'm sure
29:43
you've seen that joke or that meme where it's like, oh.
29:46
We there's all these 12 standardisation frameworks and
29:52
nobody's using any of them. We need to create one that
29:56
everyone's going to use. And then the next day it's like
29:59
now there's 13 standardisation frameworks that nobody is using.
30:02
And so it felt. Hard to meet, to create
30:09
something that would get that community involvement or that
30:14
backing around and that everyone would adopt and that is just
30:18
like. Quite the endeavour.
30:21
And so I think you do need the power of.
30:26
Really big companies behind it that give their stamp of
30:30
approval and say we're going to support this and we're going to
30:33
use this and then. You have a shot, but I am.
30:38
I am excited about if that ever will happen.
30:42
And I'm also a little bit weary. I think there is a need for it.
30:45
I just wonder. If it's going to be possible,
30:49
right? Because what you're asking for
30:51
is coordination from so many. Big companies that have their
30:56
way of doing things, it is possible, it's going to take
31:00
time. But once all of the incentives
31:04
online and especially when it comes to pushing forward the
31:09
current status of the machine learning world and AI world,
31:12
things are going to change. So I mean we have several open
31:15
conversations today when we talk about AI and the current status
31:19
of the AI world. One of them is around
31:21
regulations, another one is around hallucinations and LM,
31:24
Third one around safety and security, forced about privacy
31:28
but also about standardisation. And when you look at the
31:32
statistics, especially in the enterprise world, 90% of
31:36
businesses around the world have made some investment in AI, have
31:40
tried to find how they can use and leverage AI in their
31:43
systems. But that statistics highlight
31:46
that 15% of them is able to take their AI to production and make
31:51
use of AI in their systems and infrastructure.
31:53
So that just highlights the complexity of the world that we
31:57
are working with, but also the gap that exists today between
32:02
the capabilities that we have developed as an AI community at
32:05
the machine learning community across the different fields.
32:08
Not just in natural language processing or not necessarily in
32:11
LLM, MMS or not necessarily through the current generative
32:14
AI bull. Rather than in all types of deep
32:18
learning. Specifically, in what's going to
32:21
bring the powerful use cases to the enterprise and move forward
32:25
the state-of-the-art in order to be able to actually save lives,
32:29
cure diseases, be able to provide the economic
32:32
acceleration the eye can provide for our economy and our systems
32:36
around the world. I guess when you look at.
32:40
The parallels again between the cloud and the AI, nascent AI
32:46
movement that is happening right now.
32:48
Kubernetes didn't come out right until 2016 or 17 if I remember
32:52
correctly. It was maybe 2015, I can't
32:56
remember. But all that's to say that the
32:59
cloud had been around for a few years and so potentially the
33:04
Kubernetes for. The ML and AI movement is still
33:10
a year or two out because things need to settle a bit more before
33:14
you can have that unifying layer be brought to the surface.
33:18
I would say it started somewhere around the thinking.
33:21
At least. The idea started shipping around
33:23
2013 fourteen. I remember very well an e-mail
33:29
that Craig Mcluckie sent through the product management internal
33:32
list. I mean there was also like few
33:33
people on this list back then of, you know some of the early
33:37
ideas at that time, independently I was writing a
33:40
patent on visualisations of virtual machines of how we can
33:43
build another layer of the virtualization.
33:45
So I'm sure this patent work that I was doing with him and
33:49
the button player was awarded to Google with my name on it.
33:53
So I mean I believe Google have used it as far as I know to be
33:56
able to initialise cooperates later on.
33:59
So it was basically ideas were forming very early and there was
34:03
some exchange in 2013, two 1014. So in this sense I think the
34:08
ideas are already out there today for Copernicus for machine
34:12
learning or the machine learning abstraction layer that's going
34:15
to provide the orchestration similar capability to auto have
34:18
seen whose coordinates back in the days at the provide
34:21
abstraction level for for cloud and for the multi cloud.
34:24
Nonetheless, it's not going to be successful until we can need
34:28
the use case, and the use case is still early on.
34:30
So that was how fast AI is moving nowadays.
34:34
It might be, you know, like next year.
34:36
It might be next week, nobody knows.
34:38
But it's definitely an exciting time.
34:40
Well, one thing that there is no shortage of in the AI and really
34:45
data space are. Orchestration tools, right?
34:50
I can name probably 5 or 10 just off the top of my head.
34:55
But what we're talking about isn't exactly an orchestration
35:00
tool, right? It's a little bit deeper than
35:02
that. It's more of.
35:04
A way for you to slug your system and your tool and your
35:09
environment into. A layer that is.
35:16
Compatible. With hopefully.
35:20
A lot more layers, so you can have your node and you you plug
35:25
into the greater good, whatever that may be, and you know that
35:31
everything's going to work because everything is
35:36
standardised to that greater good.
35:38
Exactly exactly true. I mean think about it today, in
35:41
order to feature engineering you need to use one system and then
35:45
plug it into another system in order to do the training.
35:47
And then depending on which training environment that you're
35:50
using, then you need to you know, like integrate it with
35:53
different kind of environments. But also then you need to go
35:56
into the inference which supposedly could be on the same
36:00
cloud or on a different cloud and it comes with different
36:03
trade-offs. But then you have to move to
36:04
another cloud and when you move to another cloud then you will
36:07
need different monitoring tools. Then you will need different
36:10
production system characteristics in order to be
36:13
able to deploy your machine learning operation capabilities
36:17
in order to make sure that the system is operational.
36:20
But then once system that's operational becomes the model,
36:24
the greatest switches you know like given in any model because
36:26
of the changes in data distribution.
36:28
At least that would be one reason.
36:30
But basically you go back and you would be right start the
36:33
journey all again using different tools.
36:35
So there is no unifying component between all of these.
36:39
And while it's much better than developing machine learning
36:43
models in C and point, but it's also the case that the more
36:48
simple and streamlined the end to end process is going to be,
36:51
the easier it is to be able to integrate it into other use
36:55
cases to bring it to more production systems and to more
37:00
user facing applications. And that's all going to advance
37:05
the way that we are all doing and building and scaling machine
37:08
learning for the next set of applications that's going to
37:11
come out. Yeah.
37:13
And one other point that I think about when you talk about that
37:17
headache of developing or basically getting something into
37:21
production is how if a tool that you have chosen to use then gets
37:28
deprecated and then you have to go and figure out how to rip out
37:32
that and then get a new tool that actually works better or
37:37
companies? Move away from supporting open
37:40
source projects and what happens when you bet the farm on using
37:45
that or there's a breaking change and a new update of the
37:48
open source project. So figuring out all of that is
37:52
added complexity and it would be like I understand the dream, the
37:57
utopia of having this. I do wonder who and how it's
38:02
going to be. Created right?
38:04
Because it would be super useful to have.
38:07
I just think that you would have to have somebody that can.
38:11
Get everyone to play nicely with each other.
38:14
True. I mean, let alone of course,
38:16
experimentation, which is another, you know, like, yeah,
38:18
works in terms of how you manage it and how you scale it and
38:22
agree with you. I mean, it's an opportunity if
38:25
you would like to see it this way.
38:27
Yeah, yeah, totally. You have to have a incredible
38:30
diplomat take on the challenge. That's what I think through it.
38:36
It actually is in this white space between strategy,
38:42
technology and mark it where you find the real opportunities and
38:47
where you find the real exciting and you know like challenges if
38:51
you'd like to say. So this is more of a strategic
38:55
angle of what's going to happen in the next three years in AI
38:59
machine learning and how can we bring AI machine learning
39:01
capabilities and tooling and models to wider audience and
39:06
specifically to enterprises who are going to be able to
39:09
integrate it because otherwise they're going to be disrupted,
39:11
they're going to lose their businesses.
39:13
But also how do we move forward with the whole field.
39:16
So as I started my story about the typewriter, I mean the
39:19
typewriter job went away. Yeah.
39:22
And now, you know, like we have computer, I mean you never hear
39:24
of somebody type writing their dissertation anymore.
39:28
I'm glad, especially if they have kids.
39:31
Of course, it's also the case of evolution and the case for for
39:36
moving forward of how can we move the whole field forward by
39:39
providing better standardisation, by providing
39:42
better strategy, by bringing together the big players to
39:44
address the really difficult questions that is challenging
39:48
for the entire field today. So there is one other piece too
39:52
that I think is a symptom of the complexity that you laid out and
39:57
the pain points. And that is how I've been seeing
40:00
it. Quite a few people.
40:03
Especially in the that are leading product, they say ohh,
40:08
that whole mess that you talked about of trying to get, we'll
40:13
say quote unquote classical machine learning model into
40:16
production. We can bypass that whole thing
40:19
if we just hit the Open AI API and we can just add that into
40:26
our product and we don't have to think about all this data
40:30
pipeline or data cleaning and all of the the hard parts of
40:36
getting machine learning into production.
40:39
And so it feels like they almost bypass that in a way and you you
40:44
see that being incorporated into products and their offerings.
40:50
But I also am very aware that A not everyone can do that because
40:55
of regulation, or they have some constraint around being able to
41:01
send data out or just bring some API into their feature set or
41:07
into their product. And B what I feel like is people
41:13
iterate really quickly they test out their product features with
41:17
that but then you get to a point where you want more control and
41:22
you want to be able to better manage and know have more
41:28
reliability with that feature set.
41:31
And So what you're getting from Open AI, when they update their
41:36
model and all your prompts go sour or they are down for 24
41:41
hours and or they have their rate limit and you have to
41:44
figure out some hacky way to go and create a million different
41:48
AI Open AI keys. And so I do notice that when you
41:55
get to a certain level of maturity after you've proven out
41:58
that new feature with AI and you've basically skipped the
42:03
line, you then realise, OK now let's bring it in house.
42:07
And when you bring it in house then you eventually have to go
42:10
through all those challenges like we talked about.
42:12
So I do, I do feel like, but at some point, or I should say on
42:19
the other side, some people don't ever hit that level where
42:23
they need to bring it in house. Correct.
42:28
Absolutely. First of all I should say how
42:30
grateful I am for Open AI and Shaggy BBC last November when
42:35
they put forward the Shaggy between like interface and and
42:39
change the conversation or around AI and she is a
42:42
conversation. I mean basically we came from oh
42:44
this is science fiction. This is only, you know, like the
42:46
movies that we see in the movie theatre to a conversation and
42:49
every dinner table of you know, like everyone's like thinking
42:54
about what change BC can do for us and what APIs and and you
42:58
know, like. Open the eyes doing and and you
43:00
know like this is the future, this is where you know things
43:03
are going. So in this sense we have taken
43:05
the conversation like open the eye, releasing SADC, taking the
43:10
conversation and help the entire team, the entire field to be
43:13
able to move it to the next level, to be able to have more
43:16
intelligent conversations around what AI can do for different use
43:19
cases. I would say, you know, like that
43:21
was the point that we, we were able at the field to go from
43:25
summers and winters beforehand to all it's real, it's coming.
43:29
And finally, we can have real conversation about how do we
43:34
basically build an ecosystem and address all of the different
43:37
questions that are open about around.
43:41
The eye. Having said that, I agree with
43:43
you. It's all about the use case.
43:44
At the end of the day, it's all about the product development.
43:47
It's all about the end user. And it's all about having a good
43:52
understanding of the trade-offs that are available to us today
43:56
by using APIs versus building our orchestra versus the actual.
44:01
Implications slash capabilities that we need for specific use
44:05
cases. So when we talk about health
44:07
data, for example, and privacy implications and HIPAA
44:10
compliance requirements for health data and how we can
44:15
integrate this and some of the LLM capabilities into healthcare
44:19
environments, then it's a different conversation than
44:22
we're talking about question and answering services or
44:25
questioning a question or summarization even capabilities
44:28
for public data or for Wikipedia pages.
44:30
So basically having good. Understanding between this and
44:33
that as well as between the different product phases and
44:36
product development I use cases is going to help us also being
44:39
able to address and help the rest of the basically deployment
44:43
of machine learning across different verticals and
44:46
different use cases. One of the things that comes to
44:49
mind right away is specifically for startups for example, who
44:54
want to build products and iterate very quickly in order to
44:57
be able to bring them to market. Their challenge?
45:00
Their trade-offs are very different than when.
45:02
Talking about zombies or Fortune 500 who are building products,
45:06
So when it will come to startups, it's very critical for
45:09
time to market. It's very critical for them to
45:11
iterate quickly on the product at low cost, as low cost as
45:14
possible. So when we tell them about
45:16
building an LLM from scratch or even, you know, accusing Lama,
45:20
of course have have changed the landscape quite a bit since they
45:23
released a couple of days ago. But basically, the focus should
45:28
be time to market. Their focus should be to get to
45:30
the user. Iterate as fast as you can be
45:32
able to. They did, they use case and get
45:34
to product market fit. But once you are at product
45:36
market fit then it's a question of scalability, it's a question
45:39
of of other consideration including privacy and user
45:43
privacy. That's also the case with
45:45
Fortune 500 from the get go because they have also some
45:48
compliance and privacy requirements that might not
45:50
necessarily be addressed with a API access kind of an event.
45:56
So it's use case specific that the goal here is to provide as
46:00
much information and educate which is something that we're
46:03
doing at. Just computing in order to be
46:05
able to bring this forward and in order to be able to enable.
46:09
The the investors to enable the the the product managers to
46:13
enable the executives at at large organisations, but also
46:17
startup founders to be successful in building and
46:20
addressing specific use cases for AI, which is what we all
46:23
want is to advance that state of the field.
46:26
So tell me more about jazz computing because I wanted to
46:29
touch on this a little bit. Absolutely.
46:31
So we're very early stage at this point.
46:33
We are a new kind of firm that we're building for the creating
46:39
the connective tissue between investors, Fortune 500 and SMBs
46:44
and then between startups. What we have found is that in
46:47
order to move the whole field forward, we need to exchange
46:51
information and provide good understanding, good foundations
46:55
for the three players, the three sets of players in the market,
47:00
in order to be able. To make them play together,
47:03
improvise together. I know you're a big music
47:06
person. We have playing the guitar.
47:08
We'll give you that. Idea Would you write song?
47:10
So how do we bring all of this different instruments together
47:13
in order to be able to move the economy forward with the help of
47:18
AI? Let me give you a specific
47:19
example. Fortune 500 and SMB companies,
47:23
they know that AI is going to disrupt some of the use cases,
47:26
but they are lacking the knowledge of what use cases in
47:29
their systems that needs to be addressed.
47:31
How can they integrate? And what's the organisational
47:35
technology and strategic capability that they need to
47:39
acquire in order to be able to address all of that.
47:42
So we have them by understanding some of this upcoming disruption
47:46
and how can they protect themselves and being able to
47:49
evolve as a company in order to be able to get to the next
47:52
level. Posters on the other side, they
47:56
also want to invest in the eye, they want to increase their
47:59
basically enable goodbye companies in order to be able to
48:02
see where they can help with good investments, good returns
48:08
for their lips. So we help them also with
48:10
forming and evolving their investments thesis but also
48:14
through some educational tools, through some advisory services
48:17
and being able to evaluate and due diligence.
48:19
Some of the AI companies that's coming their way, some of them
48:22
are built on solid foundations, some of them does not have the
48:25
same. That functions as as others, so
48:27
basically levelling the playing field to a great extent.
48:32
And then the third player in this case is the startups, which
48:34
are developing very, very sophisticated technology, but
48:37
they might not necessarily have all of the capabilities around
48:41
them. In order to be able to take the
48:43
product to market, specifically marketing specifically, what
48:47
investors investors look for in specifically around what do we
48:51
need to raise an impulse structure, what does cost
48:54
structure mean for us, especially from a technology
48:55
point of view? Being a technical person myself
48:59
and this was an eye opener for me when I went to Stanford
49:01
Graduate School. Of business.
49:03
So this is another area where we kind of you know like help them
49:07
and connect them with the investors in order to be able to
49:09
build this ecosystem and push the full use case forward.
49:13
But also connect them with large corporations that are going to
49:16
be able to help them put their technology that they have
49:18
developed into use cases and be able to distribute and be able
49:22
to build their distribution channels.
49:24
So this is the white space where we want to play in and we.
49:27
Today are playing in very early stage.
49:31
We're still defining what's our product market fit.
49:33
So we're experimenting and we're learning a lot, which is
49:35
something that I love. We have a newsletter, we have a
49:39
podcast, we have a weekly announcements on our LinkedIn
49:44
and on our website we have several market research papers
49:47
that's already published. We talk, our team goes and gives
49:51
talk around the world in order to be able to increase this
49:53
understanding of the trade-offs. Similar to what we have said,
49:57
you know, we just discussed on Open AI versus building our own
50:01
LMS and be able to advance the whole conversation to the next
50:04
level. And this is our goal.
50:05
This is our mission. All of those links to the great
50:09
stuff you're putting out and all of that and awesome content in
50:12
the description in case anyone wants to check it out.
50:15
Otherwise it is jazz computing. I love the name.
50:18
It's hard to forget that one. As a huge jazz fan myself and
50:24
you kind of picked it out. You probably saw the guitar in
50:26
the background or maybe you've seen me making a fool of myself
50:29
on the Internet before. It's music and.
50:32
Music is amazing at how it kind of brings together, especially
50:35
for jazz being very different than other types of music.
50:39
They're usually they don't rehearse jazz improvisations
50:43
beforehand, rather than they just discuss few at 5 minute
50:46
things. Yeah, but then as soon as the
50:48
band comes together, every one of them plays a different
50:51
instrument. It all comes together, they
50:53
improvise and then it's a beautiful piece.
50:55
Not always, but most of the time so.
50:58
Similar to how the ecosystem can work together, and I think we'll
51:03
work together in order to be able to improvise the next
51:06
advancement in AI and machine learning and the acceleration of
51:10
the whole economy using this new and get rid of some of the
51:13
typewriter stories. So, OK.
51:17
Well, thank you for coming on here.
51:18
And I got through about two questions out of the 10 that I
51:22
wanted to ask you. So I think we may have to have a
51:25
Part 2 on this. Demetrius, it was such a
51:28
pleasure. Meeting with you today.
51:30
I loved our conversation. Hey, I'm Vishnu.
51:36
I'm a data scientist at first hand and I definitely think that
51:39
you should subscribe to the ML OPS Coffee Sessions podcast.
51:43
It's the best podcast out there to stay on top of what ML OPS
51:46
actually. Is to talk to the true thought
51:49
leaders in the. Space and by the way, Demetrius
51:51
is absolutely hilarious. What a weird guy.
51:54
You should definitely subscribe to the podcast.