Latest / Agentic Conversations (formally mlops.community) / The Missing Data Stack for Physical AI

Transcript

The Missing Data Stack for Physical AI · Agentic Conversations (formally mlops.community)

0:00
And it's not, it's not time. It's like, oh, this is annoying
0:03
to wait for. It's like the world continues to
0:05
do things, you know, independent on how fast or slow you are.
0:09
Like if you're a slow person, that doesn't change the speed of
0:13
the rest of the world, right? And that's the same with the
0:15
robot. OK, so I'm Nico W, the CEO of
0:23
Rerun, and for my bulk use of coffee, it's pretty it's filter
0:29
coffee with milk, but I'm a enjoyer of of many kinds of
0:33
coffee. I think we should kick this off
0:37
with an overview of your opinions on physical AI versus
0:45
robotics, because I've heard about robotics and I've also
0:48
heard from people who are doing robotics that the big let down
0:52
is there's not a lot of AI inside of robotics.
0:55
Most robotics companies that you hear about.
0:58
Sure. So I think physically I as a
1:02
term, I at least saw it popularized by Jensen quite
1:06
recently. So therefore it's obviously
1:09
subject to a lot of the problems of hype, right?
1:13
But I was really happy when that happened because we've been
1:15
looking for a term that that kind of encompasses products
1:21
that use AI or, you know, AI in the broader stance.
1:26
Maybe, you know, the classic I just algorithms that, that do
1:29
intelligent stuff to like really deep models and that kind of
1:34
stuff. And the tenant basically applies
1:36
intelligence to the physical world, either to just analyse it
1:40
or to do something in it. So I, I include like intelligent
1:45
sort of maybe somewhat autonomous robotics and that,
1:48
but also spatial computing and sort of, I don't know, we
1:52
thought of something like long tail physical intelligence,
1:56
like, I don't know, like security applications, They're
2:00
just a lot of different, I don't know, sports analytics.
2:03
So just a very large amount of stuff you might want to do with
2:07
intelligence software that's somehow interacting with the
2:09
world. So I put personally or we put
2:13
with all of those things into the bucket of physical AI.
2:16
Like video games too, when you're grabbing all those
2:20
sensors. And yeah, I think well for our
2:23
what we run is doing, I think it's very relevant.
2:26
Maybe I don't put video games into physical AI even though
2:29
it's so close that, but you could talk about it.
2:31
I think there's also the other Jason, things are like like
2:35
generative media you could totally put in there.
2:37
There's so many similar kind of patterns to the software and the
2:44
data and how you build these products that some aspects are
2:47
like physical sort of generative media, particularly if they're
2:52
like video and they're trying to be more like realistic or so on.
2:55
But that that's close enough that you could in some some
2:59
cases talk about that too. But I think most people don't
3:01
mean that when they say physical AI.
3:04
So I think I think there's a span of people meaning like the
3:08
stuff that I said, which is like the broad sounds to some people
3:10
just meaning it's robotics with a cool name.
3:13
I think you're both of those happen.
3:14
Yeah. It it basically, if I'm grasping
3:18
it correctly, it's AI that's out in the world, in the physical
3:24
space, almost in a way that we can touch it.
3:26
It's tangible. Yeah, like interacting with the
3:29
the real world, don't I think. Not on the it's.
3:33
Easy to It's easy for us like tech people to forget, but in
3:38
most of the world's GDP takes place in the physical world.
3:41
It's historically software has really not participated in that
3:48
other than maybe administering stuff.
3:49
So I mean, you have software for maybe managing a doctor's
3:53
appointment, but it's not the, it's not the robot sort of like
3:57
doctor, right? You have maybe software for
4:01
keeping track of the building like the schedule of like
4:05
construction or something, but. You know we're sending the
4:07
invoices. Exactly.
4:08
But you don't have you're, you don't have software tech that is
4:12
just doing all the construction. So I think that that's what we
4:15
have in front of us that's really happening right now and
4:17
becoming possible. And so in that sense, in the
4:21
broad sense, I think physical AI set up to transform large, huge,
4:26
huge parts of the economy. So I at least believe that it it
4:30
has the potential and looks like it's going to also do that to be
4:36
one of the biggest changes to the world economy sort of in
4:39
history. Now, I happen to agree with you,
4:43
but I also want to raise a point, which is we've been
4:49
hearing that same thing from those IoT folks for the past
4:52
like 2 decades and I still have not seen IoT totally transform
4:58
the way we live, right? Maybe there's cool stuff that
5:02
you have with smart homes if you're really into that.
5:04
Or I noticed that certain parking garages have sensors on
5:09
if there are free parking spots. Yeah, none of those.
5:12
I would buck you into life Transformational.
5:16
Got it yeah. I, I'm not going to defend the
5:19
IoT hype. I never understood it
5:20
personally, right. But certainly with anything new,
5:24
right, that hasn't happened yet. So it hasn't happened yet.
5:27
So it could not right. So it comes down to belief, I
5:30
guess. I, I, I come down to thinking, I
5:34
know the thing I maybe never understood that IoT was like, it
5:37
sounded like a lot of very people who like tech talking
5:41
about like, what is it? It's like, yeah, you're
5:42
connecting everything. It's like that's not in itself
5:45
solving a problem maybe used to solve 1, but it's not actually
5:48
describing something that you could do.
5:50
But I think performing work in the real world or, or sort of
5:56
automatically understanding what's going on in the real
5:58
world, those are like that's work.
6:01
Like that's about that. Like that's how you unlock value
6:04
for for real people. I think that's pretty
6:06
unambiguous. So in that sense that is pretty
6:09
different. But certainly it's up to if the
6:11
technology works and if they can be, you know, brought to market
6:15
in an effective way. But but I'm I think it's pretty.
6:18
These categories are very different in those ways.
6:21
Why now? Why do you think that we are on
6:24
the precipice of things changing?
6:30
I think it's, I mean, it's mainly I think the it's the AI
6:34
part of physical AI and it that's not said that all the
6:39
great solutions will be dependent, like where AI would
6:42
be the most important thing. But I think the the real world
6:46
is super complex and, and this just sort of unending
6:52
complexity. There's a long tail of just
6:55
things that can go wrong. The world where the Super messy,
6:59
it's very hard to build super general products that's are very
7:04
large markets with when the software is not intelligent
7:11
enough in that like fussy way like not able to handle
7:14
ambiguous sort of fussy situations where things change.
7:22
So because yeah, with classic tech, like for the physical
7:24
world, the way to make a effective product is to
7:28
constrain the use a lot so that you can make right, like a
7:32
person can write an algorithm that handles each situation.
7:36
And this would be like what we see these days with the coffee
7:39
making robots. I think that's the.
7:41
I just constrain it maybe, maybe it's a coffee making robot or or
7:45
something like 1 cell in the manufacturing line.
7:49
It's like super repeatable, but you just constrain it a really,
7:51
really lot. Then if that thing is valuable
7:54
enough, you can put all the effort into just making
7:57
something for that. But a lot of the physical world
8:03
things that are out there are much more messy than that.
8:06
So yeah, basically you need something that's more flexible
8:12
and more able to handle ambiguity.
8:13
And that's really the, the kind of that's what the technology of
8:18
of sort of modern ML and AI is, is, is about enabling.
8:23
So I think that's that's one and and what that enables then is if
8:28
you can address a larger market, you can invest more into the
8:31
hardware. The hardware is also important
8:33
that it's not only I but you, you need to be able to invest in
8:37
in the hardware as well to make it good, to make it.
8:42
Yeah, I guess high quality, but also low, low, low cost, right.
8:46
That comes from scale hardware is a scale game.
8:49
So and when you have scale, then you can generate sort of you get
8:56
lots of side benefits. You think of like what happened
8:58
in mobile phones, right? That generated a huge amount of
9:00
like a ecosystem that produces components, right?
9:04
But then you can use to create reasonably priced other like
9:08
sort of the more niche products. So the mobile phone ecosystem
9:14
drove the ability to make drones, right?
9:18
Then you can make it good cheap drone because of that.
9:20
I'm not dropping the word but but because, because of that
9:25
ecosystem basically. Almost like you have this
9:29
innovation and there's those secondary and tertiary.
9:34
So I think that's that's really the big thing that a couple of
9:37
things need to come together. First, you need to have a
9:42
technology that can handle this messiness, which enables you to
9:45
build hardware products that serve much bigger markets, that
9:50
enables you to invest heavily into those products, which gets
9:54
you the ability to get that scale and that kind of gets you
9:57
that like scale flywheel going and particularly for AI and so
10:01
on, you actually need that flywheel as well for data
10:03
collection. But so for a really good AI, you
10:06
need lots of hardware to collect data to improve the models that
10:13
then allows you to deploy again, I'm going to get better data
10:17
because they're not doing more advanced things, right?
10:18
So you need that flywheel going and you also need the scale
10:21
flywheel that that is what leads to good hardware like effective
10:26
hardware products that at a good price.
10:28
So to get that ball rolling, you also need hype.
10:32
That's actually a really important component that you
10:34
need to be able to believe in the future and invest deeply
10:36
into it. I think the ChatGPT like LLM
10:40
side of AI has provided that. So so that started that out and
10:46
and that I think generated a lot of interest.
10:48
And then there's been some like within the field of robotics,
10:52
there's been some some big sort of breakthroughs, I guess in
10:57
methods like scalable robotics, learning methods, which really
11:01
has been a dream for a while. That's my understanding at
11:05
least, but but not the reality. So it's scale where just mean
11:09
scalable in the same way that you tend to talk about the AI,
11:11
right? We mean you can throw like more
11:15
data and more compute that it then it gets better.
11:17
Yeah. And like we take that for
11:19
granted now that the lambs, but that was has not been the case
11:21
in robotics for forever, right. But a couple of years ago this I
11:26
was aware of the the first line of papers that show this
11:30
properties for this RT1RT2 RTX kind of line of papers.
11:34
I don't know if you know them. Yeah, there may have been
11:38
something else that really start everything off, but that's from
11:40
my perspective what I what I saw that also kicked off like, OK,
11:45
we're seeing scalable methods here too now.
11:47
So that from my perspective, what like the combination of
11:51
like having seen that with alarms and then seeing that now
11:54
also in robotics method is is what started to really, really
11:58
get the ball rolling. And now like the the hype is
12:01
very, very real in in the space and it's a huge, huge amount of
12:05
investment. And I I think that that is for
12:07
particular when hardware is involved that's actually
12:09
necessary. So that make a long winded I'm
12:13
sorry, I think to to that question, but but I think that's
12:16
those are the reason for why why it's now.
12:19
Can you break down the life cycle of how physical AI is
12:26
trained? Like what models are we using?
12:28
What are the ways that we're collecting data?
12:30
Is it all through cameras? Is it through other sensors and
12:35
how the platforms look? What do you need to enable if
12:38
you want to be putting these models out into the world?
12:43
Because I think it has a lot of extra complexities since you are
12:49
deploying to the edge in a way, but I don't know how much of
12:54
edge deployments. Can you then also offload
12:57
certain tasks to the cloud? Like what does that whole thing
13:01
look like? I feel like I am not clear and
13:04
it's of course case dependent. Yeah.
13:07
Maybe we can just take one specific case and talk through
13:11
that. Sure.
13:13
Oh yeah, it's super case, but they're so complex and you can I
13:16
guess imagine any to those, like if you can imagine some set up
13:20
someone's doing that, but maybe super high level.
13:24
I like to think of like the two major systems that you need to
13:27
think about this like the online systems and offline systems.
13:32
So with online systems, I mean just the things that are running
13:37
let's I'm going to say robot now, but it could be some non
13:40
robot thing. But the thing that's running
13:42
like when the robot is is doing stuff in the world is running on
13:46
the robot. It doesn't matter.
13:47
It could be running like some of the maybe hit an API that runs a
13:50
model and back so that I include that.
13:53
But like mentally you can think of like what's running on the
13:55
robot that's understanding the world, planning, making
13:59
decisions like picking stuff up, like acting or whatever.
14:03
So that's the online systems. And then you have your offline
14:08
systems where you're like, basically you're using maybe
14:13
running stuff from, you know, your laptop or workbench or, or
14:17
you know, on some, some data center somewhere.
14:21
And those that's gonna be about like observability, like wait,
14:26
what's happening right now with my like fleet of robots?
14:28
Maybe it's gonna be where you maybe prototype new algorithms
14:32
and new ideas we run like analytics to understand
14:36
performance or just dig into things.
14:39
Or they're just trying to understand your data that you're
14:41
collecting and where you curate, collect and curate and transform
14:48
like data through data, data pipelines into data that's ready
14:53
for training and then train and and deploy and sort of all those
14:56
those things. So I put that in the bucket of
14:58
offline systems. And how many?
15:00
Sorry, just in this fictional scenario, how many models
15:06
typically would be running on device or online?
15:11
Yeah. So that's a hard, hard question
15:14
to answer, but I think we can maybe think about, maybe take a
15:17
little bit the historical perspective.
15:19
So if we start thinking about, so running on device, we're
15:22
talking about the online systems.
15:24
So classically everything that was running online was like,
15:27
yeah, no machine learning or maybe some, you know, maybe you
15:30
learn some classifier that that's something or whatnot.
15:32
But it's like mostly handwritten stuff acting with 3D planning
15:37
algorithms and so on. But it's all like whatever C++
15:40
algorithms written by a, you know, robotics engineer or
15:44
something optimizing the state of of, you know, doing SLAM
15:49
where where it's still robot, all that kind of stuff.
15:53
And so that's like how things were done before.
15:56
And then, you know, deep learning happened and might
15:58
start switching out small modules like, oh, we actually
16:00
our computer vision sort of works a little bit.
16:02
So maybe, yeah, we just detect objects.
16:05
But then it's just running at some frequency.
16:08
Run 1, you know, object detector, right?
16:10
And everything else is like handwritten still.
16:13
It's just a little bit detector runs on whatever every 5th frame
16:18
or every this, you know, camera frame, right?
16:20
And we're detecting things, but then the rest of the all the
16:23
pipelines are kind of treating that that's like nothing special
16:27
about that, like it's just some data, right?
16:29
Algorithms to fuse that over time and reason about what to do
16:32
when and so on. So that's maybe the next step.
16:35
But I put that into the let's see, when was Alex net 2012?
16:40
Was it 14? I can't, I remember, but like
16:42
that yeah, 12. So even that like 2018 kind of
16:46
era like stop them. I just said it with a ton of
16:49
confidence. I'm going to Fact Check that
16:50
right now. I was like, no, No 12, yeah.
16:54
It's definitely. 12, Yeah. No, I haven't.
16:56
It feels, it feels right to me, but it's that range and and then
17:01
so then, you know, you start that works and you add some more
17:04
models into there and and you start, but it's still just like
17:07
like modular. You have a mod 1 model.
17:10
We have many models. Maybe they do like single things
17:13
like maybe you have another model that's like look at some
17:17
other input signals or images and they output like an estimate
17:21
of the motion, something like that.
17:23
So you just have like these small mod modules.
17:25
It's more like it's a library. But then you can think of it as
17:29
as like it's just a function, the best stuff.
17:33
So that's that's been sort of there's been a trend of just
17:36
more and more of those things, right?
17:39
And that has a lot of problems because in fact, you can't treat
17:42
them as black boxes because there's a lot of uncertainty.
17:45
And like, you know, ML models, don't you know, they even like
17:49
really high performing ones, they only work well when they're
17:52
sort of on the rough, like operating on roughly the same
17:55
kind of datas they were trained on.
17:57
And yeah, the only way to like, that's hard, right?
18:00
That's a hard problem. How do you know when you're
18:01
outside of that data and so on. And they get a lot bottle a lot
18:04
of these, stitch them together with like handwritten algorithms
18:08
and it gets a mess. And it's pretty hard to build
18:10
complicated systems. I, I think this is how people
18:13
try to build like self driving cars with this, this approach.
18:19
Well, it didn't really work right.
18:22
So, so I guess the trend from there is to just be, I mean,
18:26
the, the idea of deep learning is, you know, do things
18:29
untended. And that's been happening more
18:31
and more basically. So over time you just take, OK,
18:33
now we have like 4 modules like, OK, can we swap them all out to
18:36
have it be like 1 neural net that does more things end to
18:39
end. So that's, that's I think
18:41
generally been the trend. So that can go quite extreme.
18:47
I think in, in like some of them like very end to end focus, like
18:52
humanoid projects. You could have like 2 neural
18:54
Nets or maybe one they called one, but it's really like 2 And
18:58
there you might have one lower level one that's faster and
19:02
smaller that's focused on like fast low level, like whole body
19:06
control. So it's really taking like I
19:08
knew like signals and maybe pressure and some other like
19:12
sensors like that. And just it has some target of,
19:16
you know, the pose or where it should be, like how the body
19:19
pose should be. And, and it's like basically
19:21
doing that control that you might previously have done with,
19:25
with more classic, like optimization based methods.
19:29
And then you have some larger neural net that's like maybe no
19:33
skills like that, a higher level and like go reach for this thing
19:37
or or so on. And that thing can be slower.
19:39
So that's something like maybe even have a third level above
19:43
that takes like text input and plans and, and stuff like that.
19:46
That's if you like very, very AI first.
19:48
But yeah, they could swap out these a lot with with
19:50
handwritten systems and and so on.
19:52
But yeah, it's I haven't I haven't seen a lot of like
19:57
single neural Nets things that does everything I think that
20:01
marketed, but I don't know if that happens in practice.
20:05
You bring up a great point is that in these systems,
20:11
especially the online ones, you are so constrained by different
20:17
things because you're out in the world and whether that's you
20:21
have to be hyper focused on battery or you have to be
20:25
focusing on speed. Nobody wants to have a robot
20:28
that you tell to do something and then 20 minutes later it
20:31
comes back and it's like, actually I can't do that.
20:33
I went through and I planned it out and no, I researched the
20:38
topic and I can't get to it, right.
20:40
So what are some other constraints or things that you
20:44
need to be cognizant of when you're doing stuff in that
20:48
realm? I think the most important
20:52
difference that is like, yeah, the most, yeah, the most
20:55
important is time. And and it's not, it's not time.
20:58
It's like, oh, this is annoying to wait for.
21:00
It's like the world continues to do things independent on how
21:04
fast or slow you are. Like if you're a slow person,
21:08
that doesn't change the speed of the rest of the world, right?
21:10
And that's the same with the robot.
21:11
If it it's doing something and it's like, oh, let me grab this
21:15
thing and then that thing is moved and it didn't matter if
21:18
you collected it, you know, it was going to, it was going to do
21:21
it, right? It's not there anymore right
21:22
there and so on. And that's very different from
21:25
even your ChatGPT style interaction, right?
21:29
You would love it to be fast because that feels better, but
21:32
it's still this like single stream of, you know, you take
21:35
the inputs and then, you know, process them all and they, you
21:37
know, give you some output, right?
21:39
There's not really a concept of our world evolving around that.
21:43
So time is just so because yeah, that that changes everything
21:48
really. You, you need to be much, much
21:50
more sophisticated about how you think about that in everything,
21:53
how you understand, you know what your software just did,
21:57
right? You need to keep track of, of
21:59
how everything evolves over time.
22:01
And you maybe have multiple notions of time, like the
22:05
compute time, the real world time that like the clock, like
22:08
whatever's happening in the real world.
22:10
Then maybe have an algorithm that takes a certain amount of
22:12
like CPU time and you, or like how many, a certain amount of
22:16
iterations. And maybe you want to keep track
22:18
of like, oh, what time was this, sample that?
22:21
And then when am I making this decision?
22:23
But decision is a little bit later in time that you made it
22:25
because you have to compute stuff, but it's relating to old
22:29
information. So just dealing with time is the
22:31
really, really big thing that you get into.
22:35
Sounds really messy on the back end too when you're trying to
22:40
create systems and you need to look at all the different ways
22:45
of time being interpreted. Yeah, it it, it gets messy.
22:49
So you need to build. I mean, it increases the
22:51
complexity of like the data tools that you you need, right?
22:56
It's pretty different than like I train one image classifier.
23:00
You can stick shockingly simple, right A problem in comparison.
23:05
And these are robotics models or whatever other things.
23:07
They're operating on sequences of times.
23:10
Even that is more complex. But then they have some internal
23:13
like notion of like steps or something.
23:15
And then that in the real world is like, oh, and sort of
23:19
overlaid on the real time of the real system they're operating
23:22
with. So that that is time is the
23:24
really, really big thing, I'd say.
23:26
Yeah, I know. Then there's obviously whatever
23:28
restore constraints and battery and things like that that are
23:31
really difficult, but similar and to to other things like you
23:35
have constraints and maybe they're, they're, they're more
23:38
difficult in the, you know, on the edge, but it's still the
23:41
same idea. Yeah.
23:43
Well, yeah, talk to me about the data side of this because that
23:46
feels like again, it would be very hard to deal with all of
23:50
the all of this different data that you're getting in different
23:54
formats. And specifically all of the
23:59
video data has got to be very heavy.
24:03
And then how your training models with the video data, you
24:06
might have some like time sensors or just time data and so
24:13
more tabular style. Exactly.
24:16
I think so we had this idea about the online and offline
24:19
systems and so on the robot, right, on the online systems,
24:24
what you'll do is you're trying to record what happened.
24:28
Yeah. And so the real world we had
24:30
this like things happen at different rates.
24:32
Maybe you have a, you know, image, your video is happening
24:35
every like a 30 FPS, but maybe you have like motion sensors,
24:40
they're going at like 1000 Hertz, so very different rates.
24:43
Sometimes these things are like kind of district, like a robot
24:45
can be like a distributed system to even have different clocks
24:48
and stuff, all this data changing at like different
24:51
rates. And you're also recording what
24:53
happened. So you can't, you don't really
24:56
know the exact shape of the data set beforehand because you're
24:59
recording what happened, right? So these these things like the
25:02
data that you're recording there is super messy.
25:05
It's like kind of like basically logs, right?
25:08
But it's logs of multimodal data streams, so lots of different
25:11
types. It could be like 3D information,
25:13
this different struct, like this data is often structured and
25:16
like deeply nested structures and so on.
25:20
And you have maybe audio and video and 3D sensors, motion of
25:25
different kinds, internal like metrics.
25:28
So really, really messy and like really complex and difficult to
25:32
handle from a data perspective effectively, because you have
25:35
this problem of like combining really fast, small signals with
25:41
large, heavy, like, you know, big tensor, some images, some
25:45
point clouds and stuff like that, that maybe are slower and
25:47
storing that together is actually pretty hard.
25:49
So classic robotics or in general you'll, you'll have like
25:53
on the system, you'll you tend to have store data to this like
25:57
very specialized file formats that are sort of very right
26:01
optimized are interesting. They're just good at like
26:03
recording exactly what happened and to do minimal operations to
26:06
just get them onto disk really fast.
26:09
That's that's like the without like disrupting anything that's
26:11
running on board. And then so that's the step one.
26:14
Then you want to get the data off the robot, they upload it
26:18
and that's a, you know, depending on the volumes, maybe
26:20
you upload all of it or you have some selective like only upload
26:23
when something happened or that kind of thing.
26:26
But somehow you got to get it to the more centralized place where
26:29
you can you can use it. And that's where you're throwing
26:32
it into like an S3 bucket or is it still?
26:34
I would say like, like just to make it super simple, right?
26:37
The absolute most simple like thing would just be, yeah, you
26:41
periodically write these logs to file and then you have a little
26:46
job that uploads them to S3 to some S3 bucket and then you have
26:50
them there. So that's that would be start
26:52
one Part 1. Wait, so that's simple.
26:55
What is advanced? Well, advance just of getting it
27:00
off, it's like, OK, we're going to be run like collecting so
27:04
much data that it doesn't even make sense to upload.
27:08
Like if you think about a self driving car, you they will
27:12
collect the data when they dock back somewhere and then you'll
27:16
just swap out the like this, the SSDs, right, then put in some
27:20
new SSDs and you may never upload it.
27:24
Or if you do, you need to send like trucks of SSDs to AWS,
27:28
right? Or you have your own local.
27:30
So it there you could, you know, make choices, right?
27:33
You only upload it when it's needed.
27:36
You have some kind of like storage architecture where you
27:41
keep everything like on a, you know, local data centre that you
27:47
have like, right that where the data is collected and you just
27:49
have it there until you upload the metadata and you go fetch it
27:53
when it's you can get really, really complex at large scale,
27:58
But let's keep it simple. You just, yeah, write these
28:00
files and upload them. Let's just, let's assume that's
28:02
possible, right? So, but even, sorry, actually
28:07
even before that, you have another problem.
28:08
So you want to be able to look at the current state of a robot.
28:11
So you want to like visualization is super, super
28:13
important just to like basically you want to see what all the, if
28:18
you want to, you're working on a robot, then you want to be able
28:20
to live visualize like all these streams of data.
28:23
Like you want to see a 3D, if it's work, you know, if it has a
28:26
3D understanding of the world, you want to see that like 3D
28:29
map. And you want to see the, the
28:31
robot, you know, walking around in in that map and see what it
28:34
sees and see the internal state of different algorithms and what
28:39
are all the camera feeds? And you want to be able to
28:41
scroll back and forth in time, right?
28:43
To so wait, like something goes wrong when I scroll back, wait,
28:46
what happens? Right.
28:47
So, so that's, that's something that you, you need when you
28:52
build these kind of systems, just live visualization.
28:54
And then you want to look at those files that you recorded
28:57
after the fact, right then and just analyse those.
28:59
And that's just like a per session kind of observability.
29:03
So it's a super, super core, core aspect.
29:06
OK, so that that that's important.
29:10
And so even before going to offline systems like this set of
29:15
like just like recording what happened to some write optimized
29:18
file and then having some visualize visualizer to like
29:21
either look at the files or look at, you know, live.
29:27
Like you cannot build these products without that those
29:29
things. And even in classic robotics,
29:31
like that's how you would you need those things.
29:33
And in classic robotics, there's a sort of Ross like robot
29:37
operating system be the most sort of commonly used setup that
29:41
gives you this like data recording and some like
29:44
visualization capabilities. And there are like slightly more
29:47
modern visualizers that are built for that that scenario,
29:52
but they're really like, and so they're great.
29:54
They work well for that. And that's like Orvis Webvis,
29:58
Xvis, Foxglove, they're a bunch of tools in that vein.
30:01
They're kind of like robotics log visualizers, really
30:04
important. And that was like designed for
30:08
this world of like pre ML world where where the that was the
30:12
main complexity of your product, like what run, you know, on the
30:15
robot, right? So, but you need that.
30:18
But then you know what happens, then you can think about what,
30:21
what happens in the off sort of what you, you now want to train
30:25
improve your models, right? So you've uploaded this data and
30:29
the current state of the world, at least is that you the need to
30:35
make that data usable by sort of the, the kind of systems that
30:41
you you have to do ML OPS, right?
30:44
So ML, you know, data, data pipelines and and and so on.
30:47
So before training, you won't have it in there or TF records
30:52
or, you know, whatever HDF 5 files that are, you know, just
30:56
optimized, ready to train. And those all those things tend
31:00
to be like very structured and do not know they're not like
31:04
good at storing this sort of messy log style data on top of
31:09
that. So, so and you want to also run
31:12
analytics like, you know, run some statistical job, compute
31:16
metrics, all that kind of stuff. So basically all the offline
31:20
data tooling that's out there, if it's like data breaks or
31:23
Datadog or whatever things they don't, these tools do not
31:28
understand like this kind of physical AI robotic style data.
31:31
They do not know, the storage systems do not know how to read
31:35
into these like log structured, like messy file formats.
31:39
They, they want everything to be like a table, you know, with
31:42
columns and and so on that that's, you know, they don't
31:45
know how to handle like unaligned, like huge unaligned
31:48
data. And they can't, there's no built
31:51
in visualization, which is like crucial for debugging, right?
31:55
So, so then you end up building like teams in the building this
31:59
very complex data pipelines to try and transform and clean the
32:03
data and do like data set creation.
32:06
And because those like offline systems don't understand like
32:09
really the source structure of the data, these these things get
32:11
super complex. Sounds miserable.
32:14
So that's yeah, super miserable. And then it's super complex and
32:17
like really brittle. And then you don't have the
32:20
ability because you don't have any built in visualization.
32:23
So so you don't really have the ability to debug back.
32:27
If if you know your last like right before training, you just
32:30
suddenly like all the data is showing up upside down.
32:33
Like where did that happen? Like you don't even have built
32:37
in visualization. So maybe like I've talked to
32:40
like self driving companies who, who started using rerun and like
32:44
found bugs like, oh, we were training on something and you
32:47
know, the some the orientation of something was flipped for two
32:51
years during training, I was giving bad performance and like,
32:53
because of some like no one saw it.
32:55
We didn't have a good. It was too hard to like debug
32:58
the data pipe, like the state, you know, of like after each
33:01
step in the data pipeline that it, it was just too hard to do.
33:04
So, so that that kind of stuff, yeah, that stuff gets really
33:09
complicated. So you just end up like for
33:10
these robotics companies in a tough spot, right?
33:12
They end up with two stacks. Like classically, you have your
33:16
like online data system stuff that was built for like
33:20
classical robotics, but that's understand the, the kind of
33:23
data. And you have your offline
33:25
systems that are built for like large scale learning and stuff
33:27
like that. But they don't understand
33:29
physically data at least don't talk to each other.
33:32
And yeah, it's just a it's a mess.
33:34
That's that's the kind of base state of of the world.
33:37
You created some visualizations, right?
33:40
And you decided to or tools to help with the visualization and
33:44
so that the physical AI can understand the world and you can
33:48
see where and how they're understanding the world.
33:52
And you decided to open source it.
33:54
Can we talk a bit about everything that you've been open
33:58
sourcing until now and the inspiration behind that?
34:04
Sure. They're just to like to frame it
34:06
first like So what we run as the companies doing, we're basically
34:09
trying to to solve the problem I just talked about.
34:13
So like we want to build like a, a new, a unified kind of data
34:18
stack that handles like both this like online and offline
34:21
scenarios for physical AI so that you get like a consistent
34:25
like ECT, use experience with kind of built in virtualization
34:28
and like much more efficient and ECT use like querying and things
34:31
like that. Because we have to kind of the
34:35
data stack understands both of these kinds of like types of
34:38
data. OK, so we started out 2 1/2,
34:43
what is 3 years ago roughly, and spent the most of the 1st 2 1/2
34:47
years, I'd say on an open source project.
34:51
It's called, you know, rerun. Like I like the company and, and
34:55
that project is focused on like logging and visualizing
35:00
multimodal data that changes over time.
35:03
So a broader application than just like just, but broader than
35:08
robotics. So we start up actually focused
35:11
on more like computer vision outside of robotics.
35:13
And I've kind of expanded into to be much broader.
35:18
And so that is a project that has you have like SDKS in
35:24
Python, Rust and C that allows you to, you can think of it like
35:29
it's like you'd log text, something logometric, something
35:32
like that. But you can log sort of
35:37
anything, you know, like a tensor or 3D point cloud build
35:41
up before like 3D scene of things happening or, you know,
35:44
normal metrics and video and have everything connected like
35:48
cameras moving around and you like hover an image and it will
35:51
like highlight where that ray shoots out in 3D.
35:54
So those kinds of things and allows you also to scroll back
35:58
and forth in time. I just got to say, this is kind
36:00
of some Star Wars shit right here.
36:02
This is what you know when they plug into the droids and stuff.
36:05
That's what I imagine they're seeing on their little computer.
36:08
I hope so. Yeah, I hope, I hope so.
36:10
It's pretty cool. Like if I can, you know, say so
36:12
myself, it's it's pretty cool application or framework.
36:17
So that's that. Yeah.
36:18
We've been building out of source project and I mean it's
36:21
pretty extreme thing. So we basically like said, OK,
36:24
none of the old things work. We rebuilt sort of this whole
36:27
stack like like this data logging visualization stack from
36:32
scratch in Rust where I took a lot of inspiration from like how
36:36
modern gaming engines are built. So the data model is built
36:38
around like a Entity component system.
36:40
Race. It's basically more like
36:43
composable data model. And so we had this goal that we
36:46
wanted to unify. If we talked about this online
36:49
and offline system, we wanted to unify the open source project to
36:51
unify the visualization side of that.
36:54
She should be able to use the same visualization framework for
36:59
like your like dirty little Python script where you kind of
37:02
like where you might use like Matlotlab or something.
37:04
You want to just, I have a little algorithm and I want to
37:06
just pop in some data, blah, you know, just dump it in and it
37:10
should just show it. And, and they've been able to
37:12
analyse it to go back and forth some time and that kind of thing
37:15
to like your centralized like visualization dashboard.
37:18
So I don't know if you've seen this like maybe marketing video
37:20
from Waymo or something where they show all the like lidars
37:24
and map like things updating on a map and all that kind of
37:28
stuff. So teams, you know, used to
37:30
rerun to build those centralized things.
37:32
And also actually recently where our last release allow you to
37:36
build like data annotation apps. So like you can have
37:39
interactivity, click on things, it'll like respond back with
37:42
like the data that you clicked on.
37:43
So you can build like data annotators with it.
37:46
And that's good for these anomalies that sometimes you'll
37:49
hit or the edge cases. You need to, I mean you work on,
37:53
yeah, you tend to annotate data like label data.
37:56
There are many different. It could be whatever, right?
37:59
You always need to be doing that.
38:00
Yeah, that's. True.
38:00
There's a lot of that, right? There could be, oh, here's
38:02
something weird happened, but it could also be, yeah, I'm just
38:05
gonna, this is how we annotate data, you know?
38:08
And it's, yeah, it's basically like wherever you want to look
38:11
at your data, which you should want to do a lot.
38:14
You want to have a consistent view.
38:15
You want that to be like you ideally like that to look the
38:18
same like wherever it is, like if it's in production or a
38:22
little script or, or and people use it to visualize maybe the
38:27
their evaluation runs during training, like like training
38:30
pipelines, there's a lot of different things.
38:33
So we wanted it, we knew that it was the goal was to unify all of
38:36
that to be able to do it in the same framework and that required
38:39
like extreme flexibility and performance and so on.
38:42
So that that's been the goal there.
38:44
That's a never ending job, but I think we've come quite far,
38:47
pretty good adoption. And I think there both like
38:50
spatial computing and robotics and like from you know, 2%
38:55
startups and I think now like Meta and Apple and Unitary and
39:02
Hugging Face and forgetting companies, but like they usually
39:06
run an open source projects at least, yeah.
39:10
Damn. So it's, it's used, you know,
39:13
from smallest to the largest. And so that's, that's been
39:16
really cool to see. And I think that it's about we
39:18
really focused on like extreme ease of use and flexibility when
39:22
you want to do like whatever you as a researcher need to do the
39:27
performance. So that's, that's been that that
39:31
project. And yeah, that's open source.
39:33
It's always going to be open source.
39:35
Yeah. And it's almost like you went
39:37
with the visualization aspect of the open source side rerun.
39:43
And then when you thought about building an actual product, how
39:46
did you think, all right, we're just going to complete the cycle
39:50
and incorporate rerun into a greater platform?
39:55
We think that they kind of needed to reinvent the whole
39:58
data stack, right? And so the open source project
40:02
forced us to do a couple of things.
40:04
So one of them is to really develop like a really good data
40:08
model, because you need a data model that's like expressive.
40:12
But also fit for purpose enough, so that's easy enough to use,
40:15
but also and composable and flexible and extendable and all
40:17
that stuff. And that's really difficult.
40:19
And you need to have something like that, that can also be
40:22
performant together with the right query engine.
40:25
So, so those, those two parts are kind of, we've been forced
40:30
to, to build, so query engine basically to build a visualiser
40:34
like this that's fast and flexible and allows you to like
40:36
scroll back in time. And I talked about this like
40:39
unsynchronised streams of data, you need to have a basically
40:41
build a small query engine or like a small in memory database
40:45
to make that that work well. So we had to develop that as
40:48
well. So those the core pieces.
40:53
So do that that for the Korean just really focus on time
40:55
alignment and and those kinds of like robotics.
40:57
I was going to say that's probably like one of the biggest
41:01
boons that you can give someone is just making sure that all
41:05
these disparate sources of data can line up.
41:08
So if some kind of a event that I want to look into, I say what
41:15
happened there with everything? How do I get a 360 view of
41:19
what's going on there as opposed to, OK, I see that something
41:23
happened with this sensor, did anything happen with the other
41:25
sensor? And now I got to go sift through
41:27
the data and try and figure out where in time that is with that
41:31
certain data source. Yeah, no, it's it's both hard
41:36
and really important. And so, so that's that we kind
41:40
of forced to work on those technical challenges for the
41:42
open source. And so for a commercial product
41:47
that we're working on right now, it's it's kind of you can think
41:50
of it as a database that has that has sort of visualization
41:56
built in. So it's it's a storage and
41:59
indexing engine. It's like a queer that that
42:02
basically. So this this thing is built to
42:06
for the constraints that that we have here.
42:08
So they have like source data in like varied forums of sort of
42:12
the right optimized like robotic style file formats that you need
42:15
to you need to have a plug in system so you can support many
42:18
different. So you need to be able to handle
42:20
this like recording like unstructured style, like data
42:24
sets. So like maybe say you know like
42:25
100,000 recordings of what happened on robot and you also
42:28
need to understand like normal tabular data, right?
42:31
So you need to understand both of them.
42:33
So like a storage index engine that can make working without
42:36
like fast and unified. Yeah, I guess a data model that
42:41
this this like gives a way to consistently for interface with
42:45
this data. So you want to that you want to
42:48
maintain. So you want to have like you
42:50
want to visualize any data that you have storage.
42:52
So but you also want to have a query engine that can operate on
42:56
top of it. So you need this consistent data
43:00
model to do that. And then yeah, the next step
43:03
about that is the the query engine.
43:06
And there what you really want is to be able to have this query
43:10
engine sort of understand this data, the physical AI data
43:13
model. So what that can mean is, is
43:17
like one simple thing is maybe OK, you have a data pipeline,
43:19
you have your raw data and then you run some transformations on
43:22
it and produce some nice more structured, easy to work with
43:25
table. You'd like to not lose all the
43:28
semantic information like what does like the first column mean?
43:33
This is a column of 3D point clouds.
43:36
And you want to know that like we're a column of like, you
43:39
know, 3D positions that are part of a point cloud that change
43:42
your time like that. Maybe one column, another might
43:44
be a video, another might be, you know, some sensor reading or
43:49
something. You want to like keep track of
43:51
what everything means. And if you do, then you can sort
43:55
of visualize sort of something and debug like a table that's
43:59
like 5 steps into your data pipeline.
44:00
So that's one of the things you want to have your query engine
44:03
like maintain that data model. And the other is you want to be
44:05
able to do sort of robotics, sort of oriented operations in
44:10
the query engine. So imagine like writing a SQL
44:12
expression and one of the parts of it is like, oh, like doing
44:16
time alignment. And you might want to do things
44:21
like do 3D transforms in the SQL expression because you want to
44:26
have all your data come out in a different in the reference frame
44:30
of, you know, your robots gripper, something like that.
44:34
So you want this, I mean, maybe abstract if you haven't worked
44:36
with this kind of data, but the ability to kind of push this,
44:39
these kinds of operations into query engine can make make
44:43
you're just working with the data like a really lot like a
44:46
lot easier. And so that's one and the next
44:50
part and then kind of a data catalog that can understand this
44:53
kind of data set. So it's really it's like the
44:55
full data stack. So that's that's kind of what
44:58
our commercial product is that we're we're building towards.
45:02
You talked about these janky pipelines before.
45:05
Does this eliminate the need to create pipelines?
45:10
Or are you still seeing folks create pipelines just with a
45:15
much more high quality data? I I think I think so First off,
45:22
this is like still in development.
45:24
We have a couple like, you know, paying design partners working,
45:27
working with what what we have now, but it's still pretty
45:30
early. So this is the first to preface
45:31
it. Yeah, You know, to not I don't
45:35
want to like say that we have something that we don't yet.
45:38
Yeah, but I think our our goal is for you not to have to have
45:43
like any steps in your pipeline. Wow, that I think that in all
45:48
cases that's not achievable, but, but you should be able to
45:51
just record and then build up a series of queries and like train
45:57
off a lot directly like that. I want, I want that to be
46:00
possible, right. That's not going to be most
46:03
efficient or the best way to do things because you'll want to
46:08
yeah, that that's just not, you want to save the intermediate
46:12
results and you want to be able to inspect them and do quality
46:14
control and and and like not redo all the computation like
46:18
during training. That that doesn't really make
46:19
sense efficiency wise or kind of structure wise to to do that.
46:23
But I, I want that, I want that to be possible such that when
46:28
you want to iterate really fast, you just have very, very few
46:30
sort of materialized intermediate steps.
46:34
And then as you know what you want to do, you kind of say, OK,
46:36
these like parts of the pipeline should be like stored as you can
46:42
flexibly choose that. So in reality, I think any
46:45
company is going to have multiple steps in their
46:46
pipelines, but hopefully they're a lot easier to manage, a lot
46:50
more efficient to to run and build and and a lot less complex
46:54
than than they have to be now. So that that's kind of the goal.
46:57
Is there anything you want to talk about that we haven't hit
47:00
on? Yeah, there are things that if
47:02
you think about the rerun, So the thing that's mostly widely
47:04
deployed is our open source project.
47:06
And that's there are massive, you know, MAG 7 style like
47:11
companies that have switched over such that at this point all
47:15
the computers and for instance, they do use this rerun to like
47:20
they use to debug it. And that goes from the little
47:25
systems that the the researchers do to how they debug their data
47:28
to kind of really like all the way through.
47:31
So like very wide deployment like that and it's hard to like
47:35
give a specific like example there, but that just reduces
47:38
friction at every point, right? Yeah, and and kind of increases
47:43
productivity. Like, you know, how how do you
47:44
even like put the value and like looking at your data, it's the
47:47
sort of the core, core thing that oils the wheels for for
47:51
everything you do. So that on a broader sense, it's
47:54
that, but I mean, maybe specifically there, there are
47:57
things like that, like very well funded, like self driving
48:02
companies finding long, you know, multi year bugs in their
48:07
data pipelines that were leading to bad like model performance.
48:11
But after adopting like using rerun to the bug their data
48:14
pipelines, that's that's another kind of example of that nature.
48:17
Meta. How are they using it with the
48:19
Rayburn glasses? So what's public from from Meta
48:24
is the Aria glasses. That's the new ones that are
48:26
coming out. No, this is like the research
48:28
glasses. So this is, they're pure like
48:33
data capture devices that are open research and yeah, they're
48:39
basically pair of glasses with a lot of sensors on them.
48:43
And so we run their rerun is, is, is the kind of official
48:48
visualizer for the data sets to come from there.
48:50
So like there's like the ego for the data sets.
48:53
For instance, they record a bunch of data in the home and
48:56
things like that. It's also built into the Aria
48:59
like dev toolkit that's the main visualizer there, that project
49:03
that's used for special computer things and also quite more
49:07
commonly used in robotics to to like collect like, you know,
49:10
robotic style data, but you know, collected by human and and
49:14
tried to like retarget that to robotics applications.
49:16
So that's so there. It's just like that.
49:19
You know, what actually I, I would love to ask you about is
49:22
you mentioned, you mentioned before there's different schools
49:25
of thought or ways that folks are trying to implement
49:30
successful robotics or physical AI.
49:34
One is as many sensors as possible, the other is the least
49:39
amount of sensors as possible. Are there other vectors that
49:43
folks play on that you've seen interesting or surprising for
49:49
you? Interesting or surprising?
49:52
I don't know. I, I think the, the kind of
49:55
major vectors I think about are differently, like how
50:00
deterministic is something like, how OK are you with like just
50:03
saying, ah, you know, we learnt to train the model and that
50:06
performs seems to perform well versus like, no, I need to have
50:09
like mathematical guarantees about some behaviours.
50:14
That'll be one other one that's like modularity.
50:17
So the extreme being like, oh, we have one neural net that does
50:20
everything. We don't have any code.
50:22
It's just a neural, it's just a neural net, but that's extremely
50:27
you think about another being like, you know, it's very
50:28
important to have modules and to test all the modules separately.
50:31
And and so this the extent that which you go after and value
50:36
modularity versus sort of performances.
50:39
I think another really, really big, big one.
50:42
And then in general, like some, some teams don't even believe in
50:45
training at all. They think or like they
50:47
shouldn't, you shouldn't use much machine learning at all.
50:50
That's certainly a bunch of folks like that.
50:52
They're like, it's, you know, unreliable doesn't work.
50:54
You just your perception should maybe they use it for like
50:59
detection and things like that, but like nothing sort of more
51:04
complex. So you use SLAM to build, you
51:06
know, 3D maps of the world and you write classic planners that
51:10
decide how to move and they feel like all these technology are
51:14
proven and you should use that and for certain application
51:17
that's that's totally the right way to go.
51:20
But then some people are a bit more purist about it, but but
51:23
that stuff is certainly still around them and probably the
51:28
right approach in a lot of more like structured environments you
51:31
can can make that work really well, like a, you know, a
51:34
warehouse or something. Yeah.
51:35
Yeah, exactly. Like if you know what
51:38
environment you're playing in then there's probably a lot
51:41
stronger case to make it the least stochastic.
51:48
Or what's the word stochastic as possible?
51:51
It's nice to know what's going on, right?
51:53
And we can make things fast and cheaper and and so on.
51:56
But it's, there's a lot of benefits.
51:58
Of course, it comes with a lot of like taking training
52:01
seriously comes with a lot of costs.
52:03
It it really increase increases the complexity of your offline
52:07
systems And the more end to end you go, the more like there's
52:12
kind of this like you can have simplify your online systems,
52:14
but just going fully end to end. But then you get like really
52:17
complex offline systems like instead.
52:21
So yeah, there's, it's engineering, right?
52:23
There's trade-offs. It's not the, it's not magic.
52:25
Those are fascinating trade-offs.
52:28
That's really cool to think about.