Latest / Agentic Conversations (formally mlops.community) / Physical AI: Teaching Machines to Understand the Real World

Transcript

Physical AI: Teaching Machines to Understand the Real World · Agentic Conversations (formally mlops.community)

0:00
So you kind of flip the problem on its head.
0:02
Instead of constantly asking the questions, the model and giving
0:05
back an answer, you basically set up this lens and then you
0:08
basically stream your real time physical AI data through the
0:13
lens. And now you get this very rich
0:14
structured output that you can then build into a big a bigger
0:19
system, which we typically call a physical agent.
0:27
You're in the US, right? I am, yeah.
0:29
Palo Alto, yeah. But I would take you for more of
0:32
a Celsius kind of guy you're. Right.
0:35
You're right. You're right.
0:36
I grew up in Northern Ireland, so that's where my slightly
0:39
interesting accent is coming from.
0:41
Yeah. It is one that I couldn't place
0:43
right off the bat. Yeah, I've been, I've been here
0:47
since 2011 is when I moved to the US and to go to do my
0:52
postdoc. So yeah, I've been, I've been in
0:53
the US for a while. So yeah, it's a bit of at this
0:56
point. And you've been doing cool stuff
0:58
with physical AI. And I know that you were at
1:02
Google, you did a lot of incredible things there, have
1:07
since spun out a company, you're leading the charge with physical
1:13
AI. But I recognize that my first
1:16
thing when I heard the word physical AI, I thought instantly
1:20
about robots. I imagine you get that a lot.
1:25
Can we just clarify terms? What are you talking about when
1:28
you say that? Sure, sure.
1:31
This is a great place to start. So I mean, robots are definitely
1:34
physical AI. I mean, they're a key part of
1:35
it. But when others I think in this
1:39
field talk about physically AI, they may really just think about
1:42
robots and maybe autonomous driving.
1:45
And then they maybe stop there, right?
1:48
At archetype, we are thinking much bigger than that.
1:52
And we're really thinking of any use case in the physical world,
1:55
in the real world where you have some form of sensor and you need
1:58
to make sense of it, right. So just think for a moment how
2:01
broad that is. I mean, of course, that includes
2:03
robots. It includes self driving.
2:05
But you know, think of like a large factory, big machines in
2:09
those factories, they have hundreds of different types of
2:12
sensors in them typically to measure the the pressure
2:15
pressure and the voltage and the, you know, gear rotations
2:19
per second and all of these type of things.
2:21
And they're great indicators of what's going to happen with that
2:23
machine, right? Think of all the cameras you
2:26
might have around that factory to kind of help monitor the CFT
2:29
of workers and you know, how efficient the logistics are
2:32
getting shipped out and everything else, right.
2:34
So think of all the kind of the electrical grid that's physical
2:38
AI, right, VRAR, you know, like all anything where you have a
2:42
sensory and you're trying to basically do projection onto the
2:44
world and do projection digitally on top of that, right.
2:47
So it's, it's very broad. It's, it's a really big area.
2:51
And really we have a, you know, an archetype we're trying to
2:55
build this horizontal platform that understands the physical
2:57
world that customers can use to build on top of for their
3:00
vertical. And the, the breadth of
3:03
customers we have that come to us is, is kind of wild.
3:07
Everything from, you know, autonomous driving to, you know,
3:10
people in factories to, you know, folks that want to put
3:13
sensors, you know, 2 kilometers underground in a big drill,
3:17
drill head or something. And each of these users is very
3:22
different. The requirements are very
3:23
different. But really the thing that unites
3:26
them all is they all have some sensor in the world and they
3:30
struggle to make sense of it with, you know, traditional
3:32
machine learning techniques or traditional signal processing
3:35
techniques. And they, they want, they're
3:38
basically missing that intelligence layer to map
3:40
sensors into reasoning and then to be able to take some action
3:43
on top of that, right. So for us, physical AI is really
3:46
that intelligence layer that's able to kind of perceive, reason
3:49
and act on the physical world in any form.
3:51
So it's very broad. And you're not talking about a
3:56
model that will be multimodal by default, that can understand
4:04
things as like a higher level of abstraction.
4:07
You throw everything at it, whether it is camera data or
4:11
it's sensor data that comes tabular form.
4:14
It's more like specialized models in that case for the
4:20
specialized use case. Well, our, our goal is to build
4:24
1 big foundation model that customers can plug lots of
4:27
different sensors into. And then they can use natural
4:29
language to, to love that, focus it on their use case.
4:33
Yeah, when it comes to actually deploying the model, they may
4:36
need to take that really big model and then break off a slice
4:39
of it and I'll, you know, compress it down for their use
4:41
case to do, you know, for example, to deploy it literally
4:44
inside a drill that goes many kilometers underground.
4:48
I'm not sure if you need to know every single possible fact in
4:51
the world to put it underground, right you there's probably.
4:54
Every Bob Dylan song. Yes, every Bob Dylan song,
4:56
right? Or, you know, you know, all,
4:58
all, all of the presidents of the US.
5:00
Like there's a lot of knowledge there that is required initially
5:04
for the foundation model to understand what's happening.
5:07
But then once you know the use case, you can break off a subset
5:11
of that model, compress it very efficiently.
5:13
And then that's the thing you actually deploy in the real
5:15
world, right? But kind of the, the BS model or
5:18
like the mothership model that the customers start with is a
5:21
much, much bigger model that you can plug in many, many different
5:24
types of sensors and then you can apply natural language to
5:27
focus what that model is paying attention to.
5:31
So, you know, typical use cases could be you plug in, you know,
5:35
four different cameras, for example, that are overlooking
5:38
the factory and then all of the time series sensors from a big
5:41
large machine inside the factory.
5:43
And you focus on like what is the health of that machine right
5:47
now, right. So imagine if a, a technician
5:50
comes up to it and like starts to take it apart to do
5:52
maintenance on it versus the gearbox in the machine that
5:55
starts slowly getting out of out of sync without taking in all of
5:59
those different sensors. Like the model has no idea why
6:02
the gearbox rotation is changing, right?
6:04
It could be because it's getting out of sync or it could be
6:06
because actually no, there's a person about to repair this
6:09
right now and they're taking off all of the, you know, outer
6:11
brackets and that's causing the machine to start to vibrate
6:14
more, for instance, right. So being able to plug in all of
6:17
these different modalities and use natural language to steer
6:20
the model for real time safety alerts or for been able to
6:25
predict when a machine might need to be repaired or been able
6:27
to actually help that technician as they're walking through the
6:30
repair manual. The model can read the manual,
6:33
it can watch what the user is doing in the world.
6:35
It can look at the sensors that come out of the machine.
6:38
It can fuse all that together. It can and it can help the
6:40
technician actually move through the steps to repair the machine
6:44
safely and in the right order, for instance.
6:45
All right, so we one big model that you can essentially use
6:48
natural language to control and then focus on the task in the
6:52
real world. Add some wild stuff.
6:55
Talk to me more about building that model, because that's where
6:59
the fun part comes in, I imagine.
7:02
Yeah, yeah. So when, when, when?
7:07
So let me let me go back in and give some context here.
7:09
So I mean, I've been working on on on sensors since the, you
7:12
know, started my career. Even before I started my PHDI
7:15
was doing a lot of stuff with real time sensors, trying to
7:18
understand them initially with the signal processing,
7:21
processing techniques and heuristics and everything else.
7:23
And I was I've been working on machine learning since about
7:25
2006. So, you know, long before it was
7:28
cool for deep learning was even a thing.
7:31
So I've been able to go through all these trends and you know,
7:33
it's been been a wild ride. It's been it's been a lot of
7:36
fun. And before I started Archetype,
7:40
I was at Google for almost 10 years, worried about a lot of my
7:43
Co founders. And you know, we've all been
7:45
working on some form of sensing in the physical world for
7:48
different parts of our career, right?
7:49
Either or on the ML stack for myself, or more of the signal
7:52
processing stack for, for, for Jamie, my Co founder or chief
7:55
scientist. And Leo, who's our designer has
7:58
been looking at it more from the interaction standpoint and so on
8:00
and so forth, right. So when we left Google in 2023,
8:07
we, we saw what was happening in the foundation model space and
8:10
we were able to kind of extrapolate of like, this is
8:13
going to go well beyond just text, right?
8:15
This is going to go well beyond vision.
8:17
This is going to go potentially to every single sensor there is
8:20
in the world. And like this, this is the
8:21
missing link that we've all been looking for in our careers to
8:24
like actually understand, you know, what's happening in the
8:27
world, right? So when we left Google, Chucky
8:31
BT had kind of just exploded. Text models were a very big
8:34
thing and we knew that we wanted to build a foundation model
8:39
fund, you know, for sensing for the physical world that could
8:42
take in any form of sensor data, right?
8:45
But to actually build that model, the architecture of it is
8:48
extremely difficult because the techniques you use to build a
8:53
large language model or a vision language model are very
8:56
different than the techniques that you would need to build.
8:59
You know, something that where you can plug in LIDAR, for
9:01
example, right? Or models where you can take
9:03
hundreds of different time series sensors across all of
9:06
these. You know, like a like a large
9:07
wind farm, for example, right, where every turbine might have
9:10
hundreds of sensors in it. And you may, you maybe want to
9:12
plug in all of those sensors across the entire wind farm.
9:16
So now you can see the kind of micro and macro patterns between
9:20
each each of the turbines and like the whole holistic field,
9:25
right? Like just the techniques to
9:27
build this, you know, didn't exist, right?
9:28
So we had to really go back to first principles and look at
9:31
like, how do we build a model where you can plug in any form
9:35
of sensor? And in most cases, it won't have
9:40
nice Internet size data sets with very nice human labels
9:45
annotated for them for, you know, 10 or 20 years of, you
9:48
know, things you can go on strip.
9:50
And it's like the big technical challenge there of how to do
9:53
that. So there's no common crawl.
9:55
Yeah, Exactly. Exactly.
9:58
So the we're able to leverage things like common Crawl, you
10:03
know, to start the model, but you know, that's, that's the
10:05
start. That's the tip of the iceberg,
10:06
right? There's just so much more that
10:08
has to happen there. So we had to basically go back
10:09
to basics and look at how do we take in all of these different
10:12
types of sensors, all of this data that's actually not on the
10:16
Internet and look at fundamental techniques that we can leverage
10:19
that and then use that to build very large, you know, techniques
10:24
that do leverage the same kind of recipe that you would see in
10:27
an LLM, right. So there's there's a big self
10:30
supervised pre training phase, there's post training, there's
10:33
RL fine tuning and so forth. But the strategies of how you
10:37
build those data sets of how you do the alignment of how you
10:39
synchronize different sensors, it's a it's a very different
10:42
approach to actually make this. So, you know, if you squint at
10:46
it, it looks very like an LLM, but do you actually go and build
10:49
it and build the data sets and think about how to, how to
10:51
construct this? It really requires us to go back
10:54
to, to basics, right? But our, our, our fundamental
10:58
kind of thesis is similar to what's happened in language,
11:02
right? Where if you look at the
11:04
original GPT, you know, one paper, there's a very
11:08
interesting phenomenon was picked up there where simply by
11:12
learning to predict the next token over over these text
11:17
sequences. And there they were using I
11:19
think early it was a script a lot of data from, you know,
11:22
Amazon reviews, right. So they were able to actually
11:26
have the model predict on its own the sentiment of the review,
11:30
right? Whether it was a positive
11:31
sentiment or a negative sentiment, right?
11:33
Simply by predicting the next token in the sequence, right?
11:36
Which is, which is kind of wild. If you step back and think about
11:39
it, that by being able to predict the next token, you
11:42
know, in this arbitrary sequence, you actually start to
11:45
like pull out these very rich sentiment models that other
11:49
people in the NLPS NLP space were struggling to build, right?
11:53
And suddenly this model like out of nowhere is able to do it
11:56
really well. So we have a similar, a similar
12:00
thesis on sensor data, which is simply by observing the physical
12:04
world, the, these really large, powerful models can, can
12:08
fundamentally start to learn the, the, the like the intrinsic
12:12
principles that govern the word world through, through these
12:15
sensors and then be able to model and, and leverage that
12:19
knowledge in the same way that an LLM can, you know, read the
12:23
Internet and then suddenly it understands, you know, human
12:26
history and it can, you know, generate beautiful essays and
12:28
all of these type of things. We could do the same thing with
12:31
sensor data. So that's kind of, you know, the
12:33
fundamental thesis is the same. How you go about actually
12:36
building it is, you know, it requires a little bit of a
12:39
little bit of innovation. There's so many avenues that I
12:44
want to go down, but the one that I think will probably be
12:47
the most valuable right now for myself is the idea of the world
12:53
models and how that plays into what you're doing.
12:57
Is there any crossover? Again, I'm trying to like
13:00
clarify the terms because I see things pop up on social media
13:04
for a blip and I kind of am following it.
13:08
I understand what it is a little bit, but I don't necessarily do
13:12
a deep dive. And then I'm graced with someone
13:15
like your presence on this podcast, and I get to go like,
13:17
OK, what's going on here? Yeah, Yeah.
13:20
OK. This is this is a great thing to
13:21
deep dive on. So there's definitely some
13:23
overlap, but I think how we look at world models is very
13:26
different than most of the community, right.
13:28
So, and that's fundamentally that a lot of folks who are
13:32
working on world models, they're, they're very vision
13:37
based, right? There's an intrinsic bias there
13:40
to take in a bunch of vision and spatially aligned data and use
13:47
that to build a almost like a digital twin, a projection of
13:52
the physical world, which can then actually be, it doesn't
13:55
have to be the, the digital twin.
13:57
It can be now a completely simulated world, but it's
14:00
extremely anchored on you can think of it as as as 4D
14:07
dimensional data where you have the 3D dimensions of the world
14:10
plus time, right? So you can basically move around
14:13
the, you know, these virtual worlds that you can generate by
14:15
observing, you know, huge amounts of video data, for
14:18
example, right? And while while that's
14:21
definitely, you know, a world model, I would not argue against
14:24
it. The problem is it's extremely
14:26
biased on vision, right? So if you want to take another
14:31
sensor modality and capture things in the world like CO2 for
14:35
example, I mean, how do you project that into this kind of
14:38
very vision dominant system, right?
14:43
How do you how do you take a temperature sensor and project
14:46
that into into this type of system, right?
14:48
And then kind of leverage that and bootstrap that, right.
14:50
So the way we think about world models is that we're trying to
14:59
allow the core of our of our foundation models called Newton.
15:04
So we're trying to allow Newton to learn this world model itself
15:08
from the sensor data, but actually to learn the
15:11
representation of that model directly from the sensor data
15:15
and not have this kind of human bias of having a very structured
15:19
kind of, you know, 3D, you know, 3D form that the very much
15:24
biases the model towards, umm, vision based systems, which is,
15:28
you know, very, very useful. But to actually capture the,
15:32
the, the subtle details that are happening in the real world, you
15:34
need to well beyond vision, right?
15:35
There's a lot of things in the world that like you can't see,
15:38
you can't touch, you can't, you can't sense, right?
15:40
And that a camera cannot pick up, but you know, you need, you
15:44
know, and the electromagnetic spectrum, you know, all of the
15:46
chemical kind of processes that might be happening and all the
15:50
things that are happening in the electrical grid.
15:51
I mean, you can't see that with a camera, right?
15:53
So even even take a teapot right on the stove, unless there's
15:57
steam coming out of it, you can't really tell what the
15:59
temperature of the teapot is, right?
16:01
You know, he there, there. There's like fundamental things
16:03
that are that are missing. And sure, if you go into the
16:05
other spectrums like infrared, for instance, then you might be
16:09
able to start to get these things.
16:10
But the fundamental point here is we believe like these world
16:14
models need to go beyond vision. I need to bring in a lots of
16:18
other modalities to be able to fundamentally understand that,
16:21
you know, the subtleties of what's happening in the world.
16:24
And then to be able to reason about the world or to be able to
16:27
act on top of it or be able to predict what might happen in the
16:30
future, for example, which is, you know, what a lot of world
16:33
model models are being used for, you need to be able to capture
16:36
all of these modalities. It's kind of so there's
16:39
definitely an overlap of how others in the field look at
16:41
world models. But I think we we're trying to
16:43
go beyond these kind of very vision based models that
16:46
essentially are being used to build, you know, fancy computer
16:49
games at this point. It's it's cool, but a lot of our
16:52
customers are coming to us. They're not trying to build a
16:54
computer game rendering of their factory.
16:57
They're trying to optimize the throughput of the factory, for
16:59
example, right. Or they're not trying to,
17:02
they're not trying to build, you know, a nice digital twin of the
17:07
of the windmill farm. They want to actually understand
17:09
why, you know, machine #62 has dropped 10% in efficiency today,
17:14
for example, right? Things like this, yeah.
17:17
Is there certain data that you throw at these models that you
17:22
should be waiting differently? Well, I guess one of the one of
17:27
the the biggest ones is the multimodal nature of just
17:35
looking at the same phenomenon in the in the physical world,
17:38
but through different modalities.
17:40
I, I guess that's one of the, you know, one of the things we
17:42
are betting on in terms of we're trying to build a model which is
17:48
natively multimodal in multimodal out right.
17:51
So I can talk a little bit about that, but and multimodal in here
17:55
means you can either take multiple instances of the same
17:59
sensor. So it could be multiple cameras,
18:00
for example, that are overlapping or non overlapping,
18:03
you know, in a factory or in an autonomous car.
18:06
Or it could be hundreds of different time series sensors
18:08
and you know, one large big industrial machine, right.
18:11
So it could either be multi instance or it could be
18:13
multimodal, right? So you maybe have a camera and a
18:16
pressure sensor, or you have Lidar and a radar and the, you
18:21
know, the, the gearbox RPMS, for example.
18:24
And each of those sensors, even the very simple ones, they
18:27
capture details that other types of sensors just fundamentally
18:31
can't, right? They're they're, they're,
18:32
they're, they're orthogonal to each other.
18:34
So by combining in these different modalities into one
18:38
model and to, to be able to do deep fusion on those sensor
18:42
modalities, it allows the model to understand these fundamental
18:48
relationships that you really can't solve with just one
18:52
sensor, right? And there's, there's a lot of,
18:55
there's a lot of problems in the world where you need more than
18:57
one sensor modality to be able to solve it.
19:00
The teapot example is like a very simple, a very simple case
19:03
of that, right? The thing I mentioned before of
19:06
the large machine and you start to see these anomalies in the
19:10
signal, well, is that because there is actually a malfunction
19:13
in the machine or it's actually, no, someone is, the technician
19:17
is actually just working on the machine or the machine next to
19:19
it, and that's kind of an unknown anomaly.
19:21
It's like it's explained, you don't need to stop the factory
19:24
today to go and fix it. It's like, yeah, yeah.
19:26
Well, that's schedule maintenance.
19:28
It's kind of the normal thing, right?
19:29
That's Scott. Yeah, it's Scott again.
19:33
Yeah, yeah, yeah. Now.
19:36
If you extrapolate this out, it feels like you can have infinite
19:43
amount of data sources. Where do you stop?
19:47
Well, this is it. This is one of the big, you
19:50
know, challenges of working on the physical world, right?
19:52
There's a very long till required to make the system
19:56
work. Some, you know, folks that are
19:57
working on autonomous driving know this right, That you have
20:01
to, you know, after you've driven a few.
20:04
A few 100 miles, like the system, you know already it
20:07
starts to like gain some intelligence, but you have to
20:10
drive billions of miles to be able to capture like all the
20:13
possible things that happen because you know, frankly in the
20:16
physical world, it's, it's on 24/7, right?
20:18
And never stops. And most of the time not things,
20:25
things are not that interesting, right?
20:27
Most intersections, if you have a a bunch of sensors there, most
20:31
of the time there are not accidents, thankfully, right.
20:34
But like you have to look at maybe, you know, six months,
20:37
seven months, 8 months of data before you actually see, you
20:40
know, that rare event happening right by by nature, it's a rare
20:43
event or an anomaly. So going back to your question,
20:47
how much data do you need? Well, you need a small amount of
20:50
high quality data that captures all of these, you know,
20:53
different parts of the distribution.
20:55
But unfortunately to have like sample those, you know,
20:58
important moments, you do need, you need to look up most of the
21:01
distribution and therefore you need to look at a lot of data.
21:04
So a big part of what we're we're building an archetype.
21:07
There's a lot of the ability to take in these huge data streams
21:12
that are essentially continuous streams from the physical world
21:15
and be able to very quickly triage those, be able to catalog
21:20
them to be able to understand, you know, what parts of that
21:24
data is useful versus, you know, what data you need to store in
21:27
cold storage versus what you can just throw away because you know
21:30
it it has very little value in it, for example.
21:32
So this is one of the big challenges of working when in
21:34
the physical world that just like you need to, it's very hard
21:39
to capture those little moments that really help push the
21:42
performance of the model forward.
21:44
Or from an eval perspective, it takes a lot of sampling in the
21:48
world to capture those hard moments that really show where
21:50
your model is breaking so that you know them what to do next to
21:53
go and improve it, right? Yeah, by definition they're
21:56
anomalies. And that's literally.
21:59
You know what an anomaly mean by definition?
22:00
Well, you know a rare event is rare, so you have to.
22:04
Maybe you have to two years, three years of data before you
22:07
see that event, right. So these are some of the
22:10
challenges. And you know, again, going back
22:12
to that first principles thing we talked about like this is not
22:15
a new thing with, you know, foundation models and everything
22:18
else like this. This is this is a problem with
22:20
the physical world. I've had this problem in my
22:22
whole career. I've been trying to solve this
22:25
problem many, many times. And unfortunately to solve it,
22:28
you know, in the, in the previous kind of deep learning
22:31
waves where everything was focused on mostly supervised
22:34
methods as opposed to, you know, self supervised and everything
22:37
else, you would have to go and get, you know, a very high
22:41
quality labeled samples for for every instance, right?
22:45
You'd have to go and get millions of of, of, of, of
22:48
labeled pairs to be able to build these, you know, models
22:50
you could ship when we were building radar models back at
22:53
Google, like there is no imagenet for radar.
22:58
We have to go and build those data sets ourselves.
23:01
And we we'd have to go and, you know, do these massive data
23:05
collection campaigns and, you know, collect millions of
23:08
examples of people doing motions and gestures and picking up
23:11
mobile phones and, you know, holding them in weird ways and
23:14
just to capture like all these nuances.
23:16
And then give it to the intern, yeah.
23:19
Exactly. I'm like, we'd have these huge
23:21
teams. It was, it was most of most of
23:23
the effort of those big production prop products.
23:26
Honestly, or, and this is the thing, I think I've heard
23:29
Carpathy say this as well when he went to Tesla, which is, you
23:32
know, when you're a researcher in academia, you do all the fun
23:35
things and you're doing all the research and you're building the
23:37
papers and you're working on the algorithms and you know, you're
23:39
drinking coffee and sitting and thinking deeply.
23:42
And in most cases, you're lucky enough to be able to kind of
23:45
download some reference data set and work on that and then
23:47
publish a paper on it. Right.
23:49
When you go to industry, it inverts and most of your time is
23:54
built on building the protocols, like what's the data collection
23:58
plan? You know, how are you going to
24:00
work with legal to kind of get this out?
24:02
You know, what are the product requirements of the features and
24:05
the actual research part of our part of the system?
24:08
It's much smaller, you know, in terms of like what you're
24:11
actually spending your time on. It's still equally is fun, but
24:14
you know, it's a very different kind of set of problems you have
24:16
to solve that, like really build production models, yeah.
24:19
Well, I appreciate that your mind went to the idea of the
24:24
anomalies and the massive amount of data that you need in order
24:27
to accomplish what you're doing in the in the form of the size
24:33
of data. What I was thinking is the
24:36
diversity of data sources and data streams.
24:40
So as you were mentioning, you've got Lidar, you've got
24:43
temperature, you've got whatever it's there's time series data,
24:47
there is words, so and then you have cameras.
24:51
If you again extrapolate that out, when is enough, enough?
24:58
Because you could say, well, maybe we need to get some kind
25:02
of infrared sensor in there. Maybe we need to test the
25:05
humidity of the air. Maybe we how can you know that
25:09
you have enough data points or can you ever?
25:13
Are you, should you just be trying to get as many as
25:15
possible? Yeah.
25:17
This is a great question and you know, it's always, you know,
25:21
the, the, the, The Dirty secret of machine learning that no one
25:24
really wants to talk about, particularly when they look at
25:26
log log plots, is that you get these diminishing returns,
25:30
right, in terms of how much data you put in versus, you know, how
25:33
much improvement you're, you're going to get out of the system.
25:36
So I always like to think about your question.
25:39
There's kind of two axes to think about it, right?
25:40
One is more from a from a product driven axes and one is
25:44
more from kind of an evaluation performance, you know,
25:47
reliability, trust perspective, right.
25:49
So on the product perspective, it's well, I mean, what sensors
25:53
are customers coming to us with that they want to plug into this
25:56
model, right? Because, you know, it's it's all
25:58
cool. Us, you know, being, you know,
26:01
very, very academic about it and being like, we want to support
26:04
infinite sensors. But like, at what point do we
26:06
have a customer come to us and say like, Oh yeah, yeah, like
26:09
great, you support all my sensors.
26:10
Cool. You know, so we definitely have
26:13
not got there yet. And I think it will take us
26:15
many, many, many more years before we get there.
26:18
But there is this, you know, classic kind of perdo
26:20
distribution where most of the customers are coming with a
26:23
small subset of common sensors, right?
26:26
Most of our customers come to us, they have some form of, of
26:29
camera fleet and they have some form of time series sensor
26:34
fleet, right? And in most cases they, they
26:38
have both and they want to have a model where they can plug both
26:40
of these sensors into, right. But the classic example I've
26:43
already given, you know, you have a big factory, you have a
26:46
bunch of machines there. They have hundreds of time
26:49
series sensors in the machine measuring pressure and RPMS and,
26:53
and you know, oil temperature and electrical consumption and
26:58
so forth. And they want to take the
27:03
systems measuring those large industrial machines and they
27:07
want to then fuse that with me. All the cameras that are around
27:09
the building that are actually looking at what, you know what,
27:12
how are those machines being operated by humans or how is
27:15
that being connected to the kind of the machine next to it, for
27:18
example, or you know, what is all the logistics of things
27:21
moving around the factory and so forth, right.
27:23
So one aspect of this is how rich and diverse are those
27:29
sensors that customers are bringing to us, right for this.
27:32
This is where, you know, again, going back to that first
27:34
principles concept, this is where we're trying to build
27:36
fundamental techniques where if a customer does come to us with
27:39
a completely new type of time series sensor or camera, ideally
27:44
we're building general enough encoders that they can plug that
27:47
into our model without us having to retrain it from scratch,
27:49
right? Because the structure of the
27:52
data is close enough to everything else we've trended
27:55
on. And the the fundamental
27:58
processes that those sensors are measuring are similar enough to
28:02
what we've seen before. So the model already understands
28:05
some of the key components. And maybe you can either use it
28:07
out-of-the-box or you can, you can tune it with a little bit of
28:10
data to kind of recalibrate the, the model to understand your
28:14
special new type of, you know, if Bosch comes out with a brand
28:17
new temperature sensor tomorrow, right?
28:18
Which they should be able to plug it into our model and it
28:20
should just work, right. But then there are some
28:23
customers that come to us with very, very rich unique sensors
28:29
like in the RF to me. And for instance, where the,
28:32
where the, where the literal structure of the data itself
28:34
looks nothing like what we, what we've given the model before.
28:38
And that's where we are trying to build general recipes where
28:44
we can take a large amount of unlabeled data for that sensor
28:47
and be able to use it to build these very powerful encoders
28:51
that can then be plugged into Newton.
28:53
And then you can kind of retrain Newton to pull in that new mode,
28:56
that new modality type, right? So you don't have to train them
28:59
all from scratch, but you will need to build a new custom
29:03
encoder for, you know, this very, you know, exotic type of
29:06
RF sensor, for example. But then if they come with
29:09
another type of RF sensor, you should be able to plug that now
29:12
into the same encoder and and reuse it and and bootstrap from
29:14
there. So part of this is coming up
29:16
with those general recipes and then building the right software
29:21
abstractions and platform tools around it so that our customers
29:24
can actually do that themselves. They can come with their own
29:26
data. They can run our platform on
29:27
their infrastructure, so the data never leaves their
29:30
networks, etcetera, etcetera. And they can use that to build
29:34
their own custom encoder without having to know any of the kind
29:37
of deep science that's going on under the hood to kind of build
29:39
those things. So, yeah, so that's on the
29:43
that's on the kind of sensor side to quickly talk about the
29:47
the other aspect, the other axis, which is when is enough
29:51
data enough, It's more on the evaluation axis, right.
29:56
So there, this is where I've, you know, learned a bunch of
30:01
hard lessons in building production, production systems
30:05
where it's very different from how academia typically thinks
30:09
about building production models, right.
30:12
So, you know, if you open any machine learning textbook, you
30:15
know, chapter 1, it'll be take your data, everything is IID,
30:19
everything can be, you know, you can take a random split 80% of
30:22
it, use it for training, 20% user for testing, call grit.
30:25
You know, you do that and you get these great numbers, you
30:28
you, you know, publish your paper, ship them all, etcetera.
30:30
Etcetera. Following it were so easy.
30:32
Yeah, everything, if everything was that that easy.
30:34
So when when building production models, this is a really bad
30:41
plan fundamentally, right? The main reason being that the
30:48
distribution that you typically capture in that data set is not
30:53
fully representative of the product and the real world,
30:57
right? And I've made a few mistakes in
31:00
my early career where I would go out and I would have teams
31:04
collect the training data first for a product and you'd be
31:07
randomly splitting and, you know, keeping back a subset of
31:10
this that you would be testing on.
31:12
And then later on you'd realize like the, you know, we're
31:15
getting like 99% performance, you know, on our held out test
31:19
set, you know, in our eval. But like when we try the
31:22
product, it's it's kind of crop, it's like it's the best, it's
31:25
80%, maybe it's 60%, right? Oh, man, I laugh because I think
31:29
everybody's been there at least once, Yeah.
31:32
And you know, it took me a long time to work out that the way to
31:36
build a test set is not to build the training set and make the
31:39
model work and then start to build the test set.
31:42
It's like from day zero, you build the test set.
31:45
And ideally you build the test set using a different protocol
31:49
and a different team, maybe even in a different part of the, the
31:51
country. So you really start to see the
31:53
differences between your training set and your test set
31:56
and the way to build the test set.
31:57
And this gets to your question of how much data is enough is
32:01
that before you build the test set, you need to sit down and
32:04
you, you basically, I like this framework.
32:06
It's called bucketed analysis, right?
32:08
Which is instead of just looking at some proxy number, let's take
32:12
the most obvious one, like accuracy, for example, right.
32:16
Having accuracy over your test set is, you know, useful for
32:19
exact presentations and, you know, simple graphs and papers
32:22
and things. But there's a lot of nuances in
32:25
there in terms of like if you have a model which is 92%
32:28
accurate and you have another model which is 92.5% accurate,
32:32
Well, is the second model actually better than the first
32:34
model? I mean, you would, you maybe
32:36
think it is, it's five, you know, .5% better.
32:39
But in production it may be fundamentally worse, right?
32:43
And you don't know because you aggregate everything into a
32:45
single single number. So a really good framework that
32:50
I use a lot is this bucketed analysis technique.
32:52
Or essentially before you start any data collection, you sit
32:56
down and you break apart your problem and you try to
32:58
understand like what are the key variables in the system you're
33:00
building, right, that are going to impact performance, right?
33:03
And some of those are human variables, you know, height,
33:06
width, gender, skin tone, you know, if it's a, if it's a wrist
33:11
based device versus something you're going to put on your eyes
33:14
versus something you're, you know, you're holding your hand,
33:16
There's going to be a bunch of things that are going to impact
33:19
the performance of that system, right?
33:21
Then you have to think about, you know, other aggressors, you
33:24
have to think about other, you know, other key variables in the
33:27
system. And typically you come up with
33:29
maybe 678 critical variables for your system.
33:33
And now you can do the mathematics of, well, that's,
33:36
that's not a 7 or 8 dimensional system.
33:39
And you can basically say for each of those key variables,
33:43
there is a bucket, right, that captures we need to find enough
33:49
participants that are Sam foot tall, that have darker skin,
33:54
that have blue and green knives and that, you know, really like,
33:58
you know, cucumber sandwiches or something, right?
34:01
Right. So now you've got this bucket,
34:03
which represents that slice of the distribution.
34:05
And then you say, OK, well, what is our confidence level on that
34:08
bucket? And then you can actually say,
34:10
well, you know, if we want to be 95% confident, for example, in
34:14
that performance of that bucket, that tells us how many samples
34:17
we need to get of, you know, 7 food people that like cucumber
34:20
something. Oh, I see.
34:22
Yeah. OK.
34:23
I know you can measure accuracy and F1 and you know all your
34:26
typical metrics on that specific bucket.
34:29
But more importantly, as you're, because again, you should do
34:34
this at like day -1 before you even start the project, as you
34:38
start the project, you can already start to build out your
34:40
test set. And initially you're going to
34:42
start and all those buckets are going to be empty, right?
34:44
So now as you get data, you can basically see where, which parts
34:47
of the distribution you're actually filling up.
34:49
You'll never fill that full distribution, right?
34:51
It's, it's, it's with six or seven variables, you're already
34:54
at millions of, of, of, of buckets.
34:57
And for each bucket, you may need 10s, hundreds, thousands of
35:00
users to be more statistically certain, right?
35:04
So like that's already like everyone's data collection
35:07
budget is already blown up just just by this, right?
35:09
So you have to use very sophisticated sampling
35:11
techniques about which buckets you pick and etcetera, etcetera.
35:14
But essentially what you can do is as you are building out this
35:18
test set, you can look at the performance of those buckets.
35:21
And now you can actually say for a specific bucket, is the
35:24
performance good enough, right? And once it's good enough,
35:28
ideally in your training data, you tell your training team,
35:32
stop collecting data for that type of thing.
35:34
We're already good enough in that area.
35:36
Go over here. You basically have a torch now
35:38
that you can kind of shine in these kind of this analysis
35:41
framework and it will help guide you on what data you should
35:44
collect, right? So this is a very powerful
35:46
framework to help like structure projects and like build
35:50
production AI models because it allows you to start to see,
35:54
well, first of all, where where are your gaps in testing, right?
35:58
Because when you actually do that analysis, you'll be like
36:01
shocked on like production systems when you do this
36:03
bucketed slicing, how hard little of the distribution
36:06
you've captured, you know, in most cases you realize, oh wow,
36:10
we are 95% accurate because most of our data samples land in this
36:15
bucket. But actually if we, if we
36:17
normalize by per bucket performance and we basically
36:19
cop, you know, once we hit a certain threshold of, of samples
36:23
in that bucket, we can basically cop the performance and not, not
36:25
continue to like improve our performance in that bucket.
36:28
And then you normalize that by all the other buckets you
36:30
rallies, you're not at 95%, you're at 60%, right, Because
36:34
you've also. Been old, you see everywhere.
36:37
It's such a more clear picture of what you're looking at as far
36:42
as performance. And as you mentioned in the
36:44
beginning, it's really nuanced when you're looking at
36:48
performance or accuracy as just one number as opposed to
36:51
breaking it up into these buckets.
36:53
Yeah, yeah. And then of course you can
36:54
aggregate all those together to get one nice metric that you can
36:57
track over time. So you can like for a bigger
36:58
team, you know, you can't go into that level of detail.
37:01
And you can actually like, is your, is your performance
37:03
improving over time? Is your coverage on those test
37:06
buckets improving over time? And normally what you see is
37:09
this kind of this kind of sawtooth wave like trend where
37:15
the performance increases and then you basically expand to a
37:18
whole new set of variables that either you you you couldn't
37:23
collect in the early days, or maybe you don't even know about
37:25
when you started the campaign. And you discovered like, oh,
37:28
actually this, you know, like whether the person exercise or
37:31
exercise or not before they use the product actually impacts the
37:34
performance of the product. OK, that's another variable we
37:36
have to add. So you're going to see
37:37
performance will increase and then you'll hit this next and
37:40
performance will drop and then it will increase again and then
37:43
it will drop. And you know, over over 612, you
37:45
know, 18 months, you'll build the actual performance you need
37:49
to really launch these things. So to go back, this is a long
37:52
way of saying to go back of how much data is enough.
37:55
I think it's either when the the customers tell you, you have
37:58
covered all the sensors or it's when you've actually looked like
38:01
you have these very rich, you know, eval slices and you can
38:04
actually say for for the product we're trying to ship, this is
38:08
the performance we need ahead. And now we're really confident
38:11
like for 95% of all these buckets that we do have the true
38:15
performance numbers on, on, you know, where the system is going,
38:17
right? You know, and typically you
38:19
never, you never reach that because you always add in more
38:22
complexity. There's always in a new featured
38:24
feature you want to add, or you know, there's always something
38:26
that will add one more dimension that you know keeps growing the
38:29
system, right? Exactly.
38:31
It's goes back to that whole thing of you're doing a ton of
38:37
R&D, I'm sure as you're also creating a product that folks
38:42
are are using on a daily basis. Yeah, I don't, you know,
38:46
honestly that's the type of team or company or, you know, that's,
38:50
that's that's where I really loved.
38:51
I love to be on those edges of, you know, going from zero to 1
38:55
and then really taking that and building a product around it.
38:57
And then, you know, building like a rich set of features and
39:00
things that people love on top, on top of that core R&D.
39:03
But yeah, it really starts from like a core like piece of
39:05
technology and and that requires, you know, normally
39:09
some significant innovation to work out how you're going to
39:13
solve that. And this is where I should
39:15
mention that, you know, one of the one of our tricks of, of how
39:18
how we do that with my my Co founders is so I mentioned, you
39:22
know, Jamie, who's who's, you know, my long term collaborator
39:26
and our chief scientist, and we're kind of both on the on the
39:29
on the technical side of the team, right.
39:31
But then we have, we have Leo who's our, our chief designer.
39:35
So he's coming in and really looking at things from an
39:37
interaction standpoint. And then we have Brandon who's
39:41
our COO and really he's like, he's just like the best operator
39:45
I've ever met. And he's just amazing at like
39:47
smashing through walls and just getting things done and, you
39:51
know, being able to like, really understand where a product needs
39:55
to be built and like just been building teams and executing
39:58
and, you know, changing laws when they need to be changed to
40:01
allow things to get chipped and all this stuff.
40:03
Right. And then we have Ivan, who's
40:04
our, our CEO, who is our CEO, who is, you know, kind of the
40:11
glue of he's able to look at all these things and kind of bring
40:13
them together. And like our, our big secret is
40:16
while we're building fundamental technology and we're always
40:19
building it from the ground up, from a technology standpoint,
40:22
from the very start, we bring in design and interaction and we
40:25
really think about how people will use the technology.
40:29
And then we use that both to kind of guide where the
40:31
technology is going, but also to pert some strong constraints on
40:37
the technology itself to allow us to actually ship it, right?
40:40
Because if you don't put those constraints on it, you're just
40:43
trying to, you're trying to boil the ocean and solve every
40:45
problem at once. And it's, it's, it's, it's too
40:48
much, right? You just, you can't, you know,
40:49
you, there's always this like volley of death that R&D goes to
40:54
die in, right? Or like you try to take an R&D
40:57
project and you typically pass it to a production team and they
41:00
will take it and ship it. And, you know, that doesn't go
41:03
well for like very new technology.
41:05
You really have to, you know, make sure that that's
41:07
successful. And one of the tricks of making
41:08
this happen is making sure that, you know, we use interaction as
41:11
a way to kind of put some strong constraints on this and really
41:14
think about how people will use the technology and how to kind
41:18
of guide them and perk, you know, make it easier for the
41:20
model itself to solve these problems, etcetera.
41:22
Well, that's a great segue into what we were talking about
41:25
before we hit record on the construction sites and the use
41:29
case around that. Can you connect those dots of
41:34
the interaction, how you ship something so that you had those
41:37
constraints with this use case of you've now got sensors and
41:42
all of this multimodal data for a construction site.
41:46
Yeah, that's a great, that's a great.
41:47
That's a great segue. So construction is a, is, is a
41:53
really good example of why doing things in the physical world is
41:56
hard, right? So we worked with the, the
42:01
Japanese construction, you know, this Monmouth company, they're
42:06
fundamental. They're, they're, they're
42:08
Kajima. If you've ever been to Japan,
42:11
they're the company that built the island that the airport sits
42:15
on top of. They didn't, they didn't build
42:17
the airport. They built the island that the
42:19
airport is, you know, is resting on top of in Japan.
42:22
It's phenomenal. So they build, you know, bridges
42:25
and tunnels and islands and you know, the buildings and
42:28
everything, right? And they, we work with them on a
42:33
five year project and we come in at the end where they, we were
42:37
basically moving a river, right, as as one does on a normal day,
42:41
right. And to avoid flooding.
42:45
And they essentially had four or five years worth of data of all
42:50
of these cameras placed around different parts of the river as
42:54
the construction teams put these large excavators on barges,
42:57
floated them out in the river. And basically we're dredging the
43:00
river to, to, to basically widen and deepen the river to, to, to,
43:03
you know, avoid flooding, right? And their big goal was to
43:07
understand from that data, how to estimate the, the throughput
43:13
of each of the, the dredging teams, right?
43:15
Like, you know, the team had a schedule, they were supposed to
43:17
dredge 6 hours that day. Did they actually do six hours
43:20
or did they 2 or did they do 0 because of maintenance issues or
43:24
weather or etcetera, etcetera. And this is a great example of
43:27
why doing AI in the real world is hard because like a
43:30
construction site is literally different every day, right?
43:33
The thing that was safe yesterday is no longer longer
43:36
safe today, right? Because they've they've moved
43:38
the bridge a little bit further out, you know, over the over the
43:40
river or you know, that system is now, you know, that wall is
43:44
now 3 meters higher than it was before.
43:46
So if you fell off it yesterday, you'd be OK.
43:48
If you fall off it today, you're going to break your leg, right?
43:50
Like these type of things. So you have weather, you have,
43:54
you have very large sites that are 25 square kilometers and you
43:58
know, the, the, the, the excavator is now just barely
44:02
visible, you know, in the, in the furthest realms of, of the
44:05
camera. And you have to get the model to
44:06
be able to understand this. So a few things we did to really
44:11
use interaction to solve this. One of the big concepts we have
44:14
in our model is called a lens. And you can think of this as the
44:18
ability to, instead of like a chat bot where you ask a
44:21
question, it gives you back an answer.
44:22
You ask a question, it gives you back an answer.
44:24
You can basically set up a lens and connect a real time data
44:28
stream to it. And then the lens will
44:30
constantly analyze the data with whatever focus you've perked on
44:34
that lens, right? So take this simple example.
44:37
You want to work out if your package has arrived at your
44:39
home. You don't want to keep having to
44:41
ask your doorbell like, did my package arrive?
44:43
Did my package arrive? Did my package arrive, right?
44:45
You want to just say like, notify me when my package
44:47
arrives today. So you set up a lens that will
44:49
continuously run and then when something happens, it can
44:52
actually output an answer, right?
44:54
So we use this interaction technique for construction where
44:57
instead of constantly having to ask the model like what's the
45:00
efficiency of the work team today, you set up this lens
45:04
before you before you get the data.
45:07
And now you actually run that lens on either prerecorded
45:09
videos or live data streams. And now the model as it's
45:13
running will actually basically output this rich structured
45:18
response of you know, is the work team resting, you know, is
45:23
the barge on the river or is it actually dredging or is it being
45:25
moved, you know, etcetera, etcetera.
45:28
So you kind of flip the problem on its head.
45:30
Instead of constantly asking the questions, the model and a
45:32
giving back an answer, you basically set up this lens and
45:35
then you basically stream your real time physical AI data
45:40
through the lens. And now you get this very rich
45:42
structured output that you can then build into a big a bigger
45:46
system, which we typically call a physical agent.
45:49
And that's the system you can now actually deploy where you
45:52
basically deploy these physical agents in the real world that
45:54
will in real time measure the productivity of the construction
45:58
teams, for example, right? And it will give you back
46:01
notifications and responses, but you don't have to keep asking a
46:03
questions of like you know, is the team operating well today,
46:07
for example. Is a way for you to tell the
46:11
model what you care about. Yes, beforehand exactly.
46:14
And then it's it's interactive, right?
46:16
So it's real time. So you're going to adjust the
46:17
focus in real time. You can ask your questions if
46:20
you want to and it can give you back the answers.
46:22
But it will, it'll essentially you give the, you give the, the,
46:25
the model some explicit goal of like what you're interested in.
46:30
And that the model now I can on, on its own, autonomously analyze
46:34
that and then work with you to actually like give you back a
46:37
report or like send you a notification.
46:39
Or you can ask it a follow up question and it can then give
46:41
you the answer. But it doesn't have to do it
46:43
from scratch, right? And then of course you can have
46:45
dozens of lenses running on the same sensor stream to look at
46:48
different aspects of the data. That's pretty wild.
46:51
So I can get a daily report of how my construction site is
46:54
going. Exactly, or, or, or so you can
46:58
get a daily report or you can have if a worker happens to go
47:03
up in one of the rigs and now is in it and it it'll immediately
47:06
notify you on your pager and say like, hey, you know, you know,
47:11
Claire has now climbed up on like a dangerous, like pay more
47:15
attention to her to make sure that she's safe, for example,
47:17
right. Yeah.
47:18
Or you can say like here's here's the work at this section
47:22
of the river two months later. Here's not the productivity of
47:26
this section of the river. Why is 1 section 20% lower
47:30
productivity? Like help me analyze this and
47:32
understand this big. Scott Scott's over there, which
47:36
is a little side gig. It's over there and it happens
47:39
to be winter, which means there's a lot more rainfall,
47:42
which means the river is much higher, which means there's a
47:44
lot more dredge kind of, you know, dirt in it, etcetera,
47:46
etcetera. But ultimately it's Scott.
47:48
We should move, Scott. And so it will be able to
47:52
extrapolate that kind of information of, hey, there's all
47:55
this stuff that's happening and this we have a confidence level.
48:01
I'm sure that we think it's because of these things.
48:05
Exactly. So I'm, I'm joking about
48:08
weather, but actually that's a, that's a, that's a real thing on
48:10
that project where we fused in from these multimodal camera
48:14
rigs where they're all capturing different aspects of, of the
48:17
work. We fused in the local weather
48:20
station reports. So we got rainfall, we get
48:22
temperature, all those things we fused in the, the, the, the,
48:26
the, the hydrometer sensors that are on the river that actually
48:30
tell you the height of the river, right?
48:32
And things like water flow and things like this.
48:35
We fuse these all into Newton. And now you can actually ask
48:38
questions like, you know, show me the top, you know, temp top
48:43
10% of, of, of, of, of teams that are working, you know, this
48:47
month, show me the bottom 10%. Like what's, what's the
48:50
difference between these three teams?
48:51
You know, I can actually go and tell you, oh, I, because of the
48:54
weather, there was a storm 3 days before.
48:57
And it takes several days for that storm to kind of impact
49:00
downstream, right? And because of that, even though
49:03
the team was able to go and work that day, it was a bit wet, but
49:06
it wasn't that bad because of the storm 2 days before.
49:09
It's actually not decreased the productivity of the, of the, of
49:12
the system, right. So it's a great example again
49:14
of, of that multimodal data where you're bringing in the
49:16
weather data plus the cameras. And now you can combine it
49:19
together and look at more than just 5 minutes of data.
49:22
You can work look at many days of data.
49:23
And because of that, you can start to see these bigger
49:25
patterns that even the engineers on the site weren't able to pick
49:28
up. Yeah.
49:29
So what do they do with these insights?
49:32
Well, mostly. Mostly they're using this for
49:35
doubt to help them, to help them the way they structure their
49:41
teams. A lot of the work is being done
49:42
by subcontractors, right? So a lot of this is from a
49:45
compliance standpoint to make sure that the team they're
49:49
subcontracting and they're, they're getting the work.
49:51
You know, for example, a six hour shift to go and dredge,
49:54
like are they actually going and doing 6 hours of dredging?
49:56
And if they're not, is it because of the weather?
49:58
Or is it because the drill bit broke that day and they had to
50:01
spend 2 hours repairing it? Or is it because, you know, Sam
50:05
didn't turn up the work that day and the, the team wasn't, you
50:07
know, it didn't have the barge driver and therefore it couldn't
50:09
do do the work? And then based on that, how do
50:11
you then go and start to optimize the site so that, oh,
50:14
next time we see a storm coming, let's actually just have the
50:17
team go and work on this section of the site instead, because
50:20
they're going to be a lot more, they're going to be a lot more
50:22
productive for for instance. And then we'll go back to that
50:25
area of the river where we know four days after the storm it's
50:29
OK to dredge there, for instance, right?
50:32
Wow. Dude this is so fascinating I
50:36
think we'll end it there. Unless is there anything that I
50:40
didn't ask you that you wanted to talk about?
50:45
Well, I think that it's been a fun conversation.
50:48
I'd love to talk about this stuff.
50:49
I guess the final thing I can end on is we have a huge amount
50:53
of customers coming in. You can probably tell that the
50:56
team is growing quite a bit. So what we are hiring if folks,
50:59
you know, want to come and work in physical AI, you know, hit me
51:02
up on LinkedIn or maybe we can, we can attach our careers page
51:05
to, to the podcast and folks, you know, because, yeah, there's
51:09
a lot of work to be done. And, you know, a lot of very
51:12
cool things happening now in the physical world.
51:14
And I think in the next few years, this area, this, this,
51:16
this space is going to is going to explode.
51:18
It's going to be huge. So, yeah, you know, come and
51:21
come and join us. Come and work on very cool
51:23
technology in the real world. Is it hiring in the Bay Area?
51:28
Hiring across it's in the Bay on in Europe.
51:31
So yeah, we're, we're, we're, we are hiring A-Team across.
51:34
We're we're just distributed company.
51:35
So yeah, we, we have folks, we have folks all around the world.
51:38
The two major axes is, you know, in the US and and and and and in
51:43
Europe. But yes.
51:44
Engineering sales. Yes, everything.
51:47
Engineering, sales research. Yeah.
51:50
Excellent.