Latest / Agentic Conversations (formally mlops.community) / Beyond AGI, Can AI Help Save the Planet? // Patrick Beukema // #225

Transcript

Beyond AGI, Can AI Help Save the Planet? // Patrick Beukema // #225 · Agentic Conversations (formally mlops.community)

0:00
Hold up, wait a minute. We gotta talk real fast because
0:03
I am so excited about the MLB's community conference that is
0:07
happening on June 25th in San Francisco.
0:10
It is our first in person conference ever.
0:12
Honestly, I'm shaking in my boots because it's something
0:15
that I've wanted to do for ages. We've been doing the online
0:18
version of this. Hopefully I've gained enough of
0:21
your trust for you to be able to say that.
0:23
I know when this guy has a conference, it's going to be
0:27
quality. Funny enough, we are doing it.
0:30
The whole theme is about a I quality.
0:34
I teamed up with my buddy Moe at Kalina who knows a thing or two
0:38
about AI quality and we are going to have some of the most
0:43
impressive speakers that you could think of.
0:46
I'm not going to list them all here because it would probably
0:49
take the next two to 5 minutes, but just know we've got the CTO
0:54
of Cruise coming to give a little keynote.
0:57
We've got the CEO of you.com come in, we've got Chip, we've
1:02
got Linux, we've got the whole crew that you would expect.
1:05
And I am going to be doing all kinds of extracurricular
1:10
activities that will be fun and maybe a little bit cringe.
1:15
You, you may see, hear or see me playing the guitar.
1:19
Just come. It's going to be an awesome
1:20
time. We'd love to have you there.
1:22
And that is again, June 25th in San Francisco.
1:26
See you all there. So Patrick Bugema and I'm the
1:33
head of the Environmental I team at AI2, within Applied Science
1:37
at AI2. And I like lattes from
1:41
Lighthouse Coffee Roasters in the Fremont neighbourhood of
1:45
Seattle. And they are, I should say,
1:50
incredibly good at latte art. And if you know, you know.
1:55
But check it out if you're in in Seattle.
1:58
Lighthouse Coffee. Welcome back to the MLS
2:01
Community Podcast. I'm your host, Dimitrios.
2:04
We are rocking and rolling today with Patrick and what a
2:08
conversation this was. It was cool.
2:13
We got to talk about environmental AI.
2:16
I didn't know much about this topic, I didn't understand much
2:19
about this topic, but Patrick was able to bring the
2:22
engineering mindset and his feet on the ground to explain to me
2:28
how they are doing it at AI2 and he was so open.
2:32
It was so cool to see. It was also very nice to see
2:36
that he is super pragmatic about how he goes about this.
2:40
Some of my favourite stories that are probably going to stay
2:43
with me for a lot longer than the next week, week and 1/2
2:48
that, you know, my attention span is quite like a goldfish
2:51
sometimes. But these stories about how
2:55
they're using satellite imagery to detect when people are
3:01
fishing in the wrong place and the cost of getting a prediction
3:06
wrong is so fascinating. And what they do to ensure that
3:11
they do not get predictions wrong so that they do not have
3:15
to waste 10s of thousands of dollars to get the Coast Guard
3:21
to travel miles and miles off of the coast to see if there is a
3:25
boat actually fishing in where it should not be fishing.
3:28
Because if you're predicting that there is a boat out there
3:31
doing some illegal fishing and then you tell the Coast Guard
3:34
and the Coast Guard goes and it's actually not a boat.
3:36
It's just as he put it, someone building a wind farm or these
3:42
windmills, then you've got a little bit of a problem on your
3:45
hand because you ended up wasting a lot of resources and
3:49
taxpayers money. That was just one little piece
3:52
of this whole conversation. I really hope you enjoy it.
3:55
And as always, if you do leave a like, subscribe all that fun
4:00
stuff, I would love to hear in the comments what you thought of
4:03
this conversation and. Let's.
4:06
Get right into it with gotcha. I know where exactly where I
4:15
want to start, which is about five years ago.
4:18
I was very proud of myself because I took this course.
4:23
It was like in one of those MIT free learning courses and it was
4:26
called Learning How to Learn and it feels like you did that same
4:34
thing except on steroids. Can you breakdown what you're
4:38
learning how to learn journey was?
4:40
I studied logic like early on in in in in undergrad, which I
4:46
really enjoyed because it was this like universal language for
4:50
making sense of the world. And I remember one of my
4:54
professors at the time, he told me that the great thing about
4:57
logic is that no matter what language, you know, even if you
5:03
don't understand it, if you can basically encapsulate whatever's
5:07
being said in logic, and you should be able to like, you
5:10
know, Boolean logic, predicate logic.
5:12
Yeah, you can make sense of it. And like, you could even
5:17
understand an alien, is what he said.
5:19
And, and that really stuck with me because it's this, you know,
5:23
it's effectively like if you learn logic, like basic logic,
5:27
I'm talking about, you know, basic predicate logic, you can
5:30
make sense of a lot of the world.
5:33
And that eventually led me to a masters at Carnegie Mellon, also
5:37
in logic, like formal symbolic logic, very far from deep
5:41
learning at this point. But during my masters, I started
5:45
studying neural networks and I did my thesis on our current
5:49
neural networks. This was a long time ago before
5:53
Pytorch and Tensorflow. I wrote those Nets in Java, if
5:57
you can believe that you had to work very hard, many thousands
6:01
of line to do something that's probably, you know, you could do
6:04
an in a in a few lines at this point.
6:07
And so I got really into, into neural networks and, and
6:11
especially learning and how neural networks learn, how we
6:15
can build neural networks that, that learn and, and, and then
6:17
how we as humans learn. And so I, I, I ended up doing a
6:25
pH D in computational neuroscience and, and, and I
6:28
focused on plasticity and, you know, neural plasticity and how
6:32
neural populations patterns change over time.
6:36
And, you know, to your point, how we can like accelerate that
6:41
process. And, you know, we don't know a
6:43
tonne about how the brain works. It's kind of a new field, but we
6:49
know a little bit about motor learning.
6:50
And so I, you know, I think it's a fascinating subject and it's,
6:55
it's definitely anchored kind of how I approach artificial
7:00
intelligence and deep learning even today.
7:04
OK, so you went deep down the rabbit hole with that, then you
7:07
got out of school. Yeah, said Hey, I like this.
7:12
The deep learning thing and it seems like it's getting easier
7:15
you're now doing a lot of stuff you're heading up the
7:20
environmental AI team right you're the technical lead of
7:23
that and within the applied sciences organisation.
7:27
I would love to dive into what you're doing now, like what your
7:32
main focus is because I think they're is a lot to be inspired
7:36
from by talking to you about what you get to spend your day
7:41
in day out doing. Yeah, it's, it's very much a
7:46
privilege. And I consider myself extremely
7:50
fortunate to have this role. And it's also a responsibility.
7:57
And the fact that we are where we are and we, you know, we're
8:01
at a nonprofit and, but we're doing environmental artificial
8:06
intelligence and it's, it's, it's really for the motto of AI2
8:10
is, is AI for good, AI for the common good?
8:13
And we think about that a lot. So like the artificial
8:15
intelligence that we're building needs to benefit humanity,
8:19
right? It needs to like uplift
8:21
humanity. That is our central focus.
8:24
As opposed to say, you know, creating the best models that
8:27
can make the most money, right? That's, that's not really the
8:30
goal. It's more like, how do we use
8:32
artificial intelligence today to improve people's lives,
8:37
basically so. Can you break down for us?
8:40
Like what is environmental AI? It's a great question.
8:43
It's also a young field, I guess even younger than neurosciences.
8:49
There are many different, I should say there that there are
8:53
many different challenge challenges within the, you know,
8:57
within environmental AI in general.
9:00
We are primarily focused on conservation and sustainability.
9:05
So there are, there are four teams within the applied science
9:12
organisation within AI, two. So there's a climate modelling
9:16
team, they're focused on, you know, very large scale
9:18
simulations of how the planet is changing overtime from a climate
9:23
and atmospheric perspective. There's a wildlife team called
9:27
Earth Ranger, and they are primarily focused on how do we
9:32
first monitor wildlife? And you know, unfortunately
9:36
there's a lot of, there's a lot of poaching, right?
9:41
And, and so how do we kind of monitor wildlife to, to protect
9:47
wildlife for future generations? And then there's a wildfire team
9:52
or a team focused on sort of resource man from wild from a
9:56
wildfire perspective and, and a fire management perspective.
9:59
So I don't know if you spent any time on the West Coast in the
10:02
last couple of years, but it's been pretty rough.
10:05
And, and I live in Seattle and we've gotten fairly major, you
10:10
know, smoke clouds that just sit in Seattle for like a month at a
10:15
time. And that never used to happen,
10:16
right? That's kind of a new thing.
10:18
And it's, it's pretty devastating.
10:21
And obviously it, you know, it's, you know, people are
10:26
suffering, many people are suffering as a result.
10:29
And then there's a, a marine, a maritime intelligence platform
10:35
called Skylight and they're focused on oceanic health.
10:39
So the, the health basically healthy oceans, basically how do
10:42
we create and preserve healthy oceans?
10:44
So those are the four areas of focus within AI.
10:48
Two, there's, of course, you know, many more areas to focus
10:52
on within environmental AI more broadly, but those are the, the
10:55
primary areas. And so we, we, we use artificial
10:59
intelligence like in, in each of these areas, we're, we're using
11:01
artificial intelligence to advance the mission.
11:04
And so we use a variety of techniques.
11:06
That was exactly where I was going to go with this.
11:08
Like what are some of the models?
11:11
It seems like some of these prediction models are more of
11:15
the quote unquote, traditional ML.
11:17
Maybe instead of a churn prediction, you're going with a
11:20
weather prediction, that type of thing.
11:23
And then I imagine there's probably from knowing you for
11:28
about the last half hour that we've been chatting, there's
11:32
probably some really advanced stuff happening too.
11:33
I would imagine you're doing some deep learning fun.
11:36
So can you break down all the different AI flavours you're
11:40
using? Yeah, good question.
11:43
So it's 100% deep learning. That's the only real technology
11:47
that we're we're building and or that we find to be most relevant
11:54
state-of-the-art. We have the the luxury and
11:58
privilege and fortune to work with extremely talented research
12:03
scientists at AI two. So there's a computer vision
12:06
team who we work really closely with called Prior.
12:09
There's also natural language teams and NLP teams.
12:12
Aristo is one of the teams that we work closely with.
12:16
We're a very collaborative team and I should just emphasise that
12:19
because although we do create neural Nets for bespoke use
12:24
cases, it's a team effort. The technologies that are
12:29
necessary for us are so computer vision is, is critical and, and
12:35
for geospatial artificial intelligence where we're, you
12:37
know, we collect a lot of data from satellites, NASA and
12:41
European Space Agency, and that's a lot of imagery And so
12:45
computer vision, but computer vision running in real time at
12:48
global scale against satellite imagery.
12:50
And these are all like deep learning based state-of-the-art
12:54
models developing developed in house.
12:56
So everything we do has been developed in house, that's
12:59
bespoke system. Occasionally we need to invent
13:02
or create. Probably shouldn't use the word
13:05
invent, but create new bespoke neural net architectures for
13:09
specific tasks that haven't been well studied.
13:14
You can imagine that in environmental AI, there may be a
13:17
lot of problems that haven't gotten as much attention as say
13:21
LM's have, for example. And so the research, the, the
13:25
technologies aren't quite as advanced as as they maybe need
13:30
to be. And so occasionally we need to
13:32
develop, you know, new technologies, new deep Nets.
13:35
And like I said, everything is 100% neural network based.
13:39
Basically going back to the neuroscience kind of background
13:43
that I have that that's really heavily anchored kind of how I
13:48
approached neural net or, you know, technology development in
13:51
general. Like basically what I attempt to
13:54
do, or what I found is most useful when developing new
13:59
neural Nets for a given use case is to figure out how the best
14:02
humans in the world do it right. That the, the, the real experts
14:07
like, identify who the experts are, study how they do it, and
14:11
then emulate how they're doing whatever that task is in the
14:15
form of the neural net like, and it might be a collection of
14:18
neural Nets or, you know, many different neural Nets.
14:21
Or you might have a computer vision net and you know, more
14:24
like language. Nad and then some bespoke thing.
14:26
But try to figure out how the best humans do and then, because
14:30
that will give you a pretty good architecture for how you should,
14:33
you know, be building your your your software.
14:37
Are you ever inspired by nature? And I just asked this because I
14:43
think I've seen things about how ohh, this drone was much more
14:48
effective because one of the inventors watched a Firefly, for
14:52
example, or something like that. Or you specifically looking at
14:56
humans and saying I'm going to try and emulate how this works
15:01
like whoever it is the best? Or am I totally missing the
15:05
point where you're saying I'm just looking at the best
15:07
researchers and how they're creating their scientific
15:10
method? So our goal is to solve
15:13
problems, not necessarily like create theory or write research
15:18
papers like. Our goal is to ship software,
15:22
right? And so when I say like we create
15:25
these neural Nets, it's, it's because we're trying to solve a
15:28
problem. Like we see a problem, we want
15:31
to give them the best possible solution or we want to provide
15:34
the best possible solution, the best model, right?
15:38
And so the, the best model that generally what I found in in my
15:43
career is that the best models are usually neural Nets that
15:47
emulate what, what the best humans do.
15:50
And, and in many cases we might not know exactly how, you know,
15:54
expert humans do a task, but that's generally a good strategy
15:58
for building a, a strong foundational architecture for
16:03
solving a given problem. And you know, that's how
16:08
computer vision originally developed right there there are
16:11
loose at this point. And, and since we've kind of
16:14
moved from CNN's into more like Transformers Envision, broadly
16:18
speaking, but originally it's kind of like the idea was let's
16:22
emulate how the human brain, how visual cortex understands and
16:26
processes information that might give us a reasonable, you know,
16:32
starting architecture for how a computer might do it.
16:36
Obviously, we've strayed quite far from neurobiology at this
16:40
point when we think about like how GT4L's work today and the
16:45
modern day, but that was the original inspiration, right?
16:49
The the sort of neurobiological information processing and, and
16:54
you know, how do, how do we emulate that in neural net form?
16:58
And so that's what I'm talking about.
16:59
And, and, and when I say like we're solving a problem, like we
17:03
don't necessarily need to create one single, you know, end to end
17:07
neural net that is like, you know, the, the one ring or
17:10
whatever. We can have a bunch of neural
17:12
Nets that are put together in very similar ways to how a human
17:18
solves a task, right? Because we, we're not just NLP
17:23
machines or not just CV machines, right?
17:26
Like we do or not vision machines or, or language
17:29
machines, right? We do a lot.
17:30
And so I that's where I'm drawing inspiration from O.
17:35
While some of us were contemplating the colours of the
17:37
rainbow, Patrick was over here shipping.
17:39
That is what you're telling me it's so you're you're solving
17:43
real problems, shipping real products and making sure that
17:46
that gets out to the world for actual value.
17:50
So I. Appreciate that.
17:51
Yeah, yeah, we need to ship. We need to ship faster, right?
17:54
We need there, there's an urgency to the environmental
17:57
threats that we face, right? And so we need to move faster.
18:02
And so I think a lot about how can we speed U?
18:05
How do we pick up the pace, right?
18:08
We're a small team, right? And.
18:10
Speaking along those lines, it feels like there's quite a bit
18:13
of research that you're doing, but then there's a lot of
18:16
engineering that you're also doing and so.
18:19
What? Is in your eyes the percentage
18:23
of one and the other that you're doing?
18:25
And what is your ideal percentage?
18:27
Maybe it's just 100% engineering shipping constantly because
18:31
you've found the solutions Or is it like, yeah, we still want to
18:35
always be dedicating X percent to R&D?
18:39
I've found that R&D is best when it's fundamentally and
18:46
engineering task. Even if you're just fine tuning
18:51
an existing model, right like an off the shelf architecture, if
18:54
you're just fine tuning that, you still have to do some R&D.
18:58
And what I've found works best is when you, you make the
19:05
research like that, R&D just another engineering discipline,
19:10
right? So it's Cici D you take all the
19:13
best practises that have been developed for decades from the
19:16
software engineering community and you just adopt and adapt
19:21
those to the MLD that sorry to the machine learning lifecycle,
19:25
to the machine learning like research workflows.
19:28
And it works great, right? You, you know, it really is at
19:35
least what we do that this applied work is, in my opinion,
19:38
it's, it's, it's, it's engineering, right?
19:41
It's, I mean, we, we certainly have to do research or we work
19:44
with research scientists who are pushing the absolute limit like
19:48
the bleeding edge of computer vision, you know, and they're
19:52
literally inventing like new ways of doing computer vision
19:54
for geospatial artificial intelligence, for example.
19:58
But what, what we are doing is I would say it's much more
20:02
engineering oriented. And I find that research.
20:08
Can and should be, you know, scaffolded or basically anchored
20:13
by the best practises from engineering.
20:19
So the, the, the our there is no in in on our teams there
20:24
anyways, on on on my team, I should say there's no
20:28
productization step, right? Like it's, it's already been
20:33
productionized. The when you start doing
20:35
research and when you start doing experimentation, it's
20:38
already, it's CI, CD, there's integrated testing.
20:42
The, you know, you generally want to get your models into a
20:50
real world scenario as fast as possible, right into the hands
20:54
of users as fast as possible so that you can start building out
20:58
a feedback loop and, and really start iterating on the bottom
21:01
line, whatever is going to actually move the needle.
21:04
And so we, we do everything from from the very beginning were
21:12
like from the very beginning of the the R&D process where we're
21:16
treating it like production basically, if, if that makes
21:19
sense. Not only makes sense, but it is
21:23
fascinating to hear because then you don't get slowed down by
21:28
this idea of, OK, we've got something here now what do we do
21:31
with it? You're starting with the end in
21:34
mind. And how do you even know if it's
21:37
good, right? Like, how many papers translate?
21:42
You know how many research papers actually translate?
21:44
How many are even reproducible? Yeah, right.
21:46
I mean, in neuroscience, replication and reproducibility
21:49
rates are abysmal, right? They're they're, they're
21:53
horrible, right? And and that's also true for
21:56
machine learning, where you might expect the the rates to be
22:02
pretty good, right? I remember Joel Pinos keynote
22:05
from NURBS in 2014, I think it was Montreal.
22:11
You know, she looked at the literally of all the submissions
22:15
and I think it was RL reinforcement learning to, to
22:18
the conference to Europe conference, right?
22:20
One of the premier ML conferences in the world.
22:22
And she just her team, you know, attempted to replicate or
22:27
reproduce those those experiments, right?
22:30
And like they looked at like 60 papers, the results were very
22:33
bad, wasn't right. And this is NURBS and RL, that's
22:38
the state of academia and research, right?
22:43
So if you want something to translate, if you want something
22:47
to actually work in the real world, right, and and that's
22:52
what that's what we're doing. That's not the entirety of of
22:55
research, nor should it be like obviously there needs to be
22:58
pure, pure research, like along lines of bore, but what we're
23:02
doing is more likely the long. So like it's use inspired
23:05
research, like much more. If you've heard of past errors
23:08
quadrant. So it's, it's, it's yes, there's
23:11
a quest for fundamental understanding, but in the
23:13
service of actual humans, right? We're so I think if that's your
23:19
goal, then it's important to have an extremely tight feedback
23:23
loop, right with your users and get that feedback loop as fast
23:28
as possible, right? Get the models you're building
23:32
into the hands of users. Whoever's going to use that
23:36
technology and benefit from it so that they can help you and
23:40
tell you like make them part of the process.
23:42
And this is especially, you know, important in the context
23:47
of conservation and sustainability where there's
23:50
obviously an extreme unequal distribution of wealth and
23:55
resources in the world. And you know, we need to
24:02
position and put the stakeholders of these models as
24:10
they need to be central to, to this conversation to the
24:16
development of the technology as opposed to like an afterthought
24:20
or like, I don't know if you've heard the term colonial science
24:24
or like helicopter science, where basically a bunch of like,
24:28
you know, we come in from our ivory tower and we drop these
24:31
incredible models and wherever, right?
24:33
And then we go away and we've saved the world, right?
24:37
Like that, That's, that's not how it works.
24:39
We. Think we did?
24:40
Yeah, right, we think. We did and, and maybe we, we
24:43
said we did in the paper, but it turns out that, you know, the F1
24:48
scores were 95% in the paper and, you know, 1% once they
24:52
actually were. Tested.
24:53
On real world, out of sample, out of distribution data.
24:57
Well, not only that, but I I think it's fascinating too,
25:00
where you're like, OK, we're not gonna go and hide away for X
25:05
amount of time working on our stuff.
25:08
And then until we think it's perfect to show to the world,
25:11
we're going to just as soon as possible get it out there
25:15
because part of our process is getting people's hands on it and
25:20
letting them inform us if this is a valuable.
25:24
I think that's crucial into what you're saying, if it's valuable,
25:27
if it's reproducible, like if it's actually working.
25:30
And a lot of times we've heard on this podcast, it doesn't need
25:33
to be necessarily like down the vein of the research, but how
25:38
many times people have built things, they've gone and they've
25:42
tried to put it out, and it turns out that they didn't
25:47
understand the briefing correctly.
25:49
They didn't actually know what the business needed.
25:52
And so they built something that was completely useless.
25:56
And. If you the more time that you're
25:59
spending building, the more time you've wasted when you get it
26:04
out and you realise, ohh, this is actually not as valuable as
26:07
we thought it was going to be. I.
26:08
Think the best AI is generally speaking, A-Team effort, right?
26:13
We're not only do you have the stakeholders of that technology
26:16
as part of the development process, right, the recipients,
26:19
but it's, it's, it must be a team effort.
26:23
Like truly great AI cannot be built in a vacuum, right?
26:28
You need, you need product and engineering and research and the
26:31
ML folks and the users and feedback and speed.
26:36
Alright, real quick, let's talk. For a minute about our sponsors
26:39
of this episode, Making It All Happen, Lattice Flow AI.
26:44
Are you grappling with stagnant model performance?
26:46
Gartner reveals a staggering statistic that 85% of models
26:50
never make it into production. Why?
26:52
Well, reasons can include poor data quality, labelling issues,
26:56
overfitting, underfitting, and more.
26:59
But the real challenge lies in uncovering blind spots that lurk
27:02
around until models hit production.
27:05
Even with an impressive aggregate performance of 90%,
27:08
models can plateau. Sadly, many companies optimise
27:13
for prioritising model performance for perfect
27:17
scenarios while leaving safety as an afterthought.
27:20
Introducing Lattice Flow AI, the pioneer in delivering robust and
27:25
reliable AI models at scale. They are here to help you
27:29
mitigate these risks head on during the AI development stage,
27:33
preventing any unwanted surprises in the real world.
27:36
Their platform empowers your data scientists and ML engineers
27:39
to systematically pinpoint and rectify data and model errors,
27:44
enhancing predictive performance at scale.
27:48
With Lattice Flow AI, you can accelerate time to production
27:50
with a reliable and trustworthy models at scale.
27:54
Don't let your model stall. Visit Lattice Flow Dot AI and
27:59
book a call with the folks over there right now.
28:02
Let them know you heard about it from the Mops Community Podcast.
28:05
Let's get back into the show. It's, it's a team effort, right?
28:10
And it's very deeply collaborative and you need all
28:14
the pieces to come together, you know, and and everyone to be
28:20
really invested in kind of shipping the highest quality
28:27
artificial intelligence that's possible, right?
28:29
That that's state-of-the-art in, in order to actually be, I
28:35
think, delivering what users need.
28:38
And, and again, you know, coming back to the urgency of some of
28:43
our major challenges in the environment related to
28:47
conservation or sustainability or climate change, you know, you
28:50
name it, we need to move fast, right?
28:53
We, we really need to pick up the pace.
28:55
And so, you know, if you want to develop and iterate at breakneck
29:00
speed, then your engineering must be razor sharp, right?
29:06
And and. So we want to iterate really
29:09
quickly, but we want to iterate on the thing that matters,
29:13
right? Let's ensure that whatever those
29:16
innovations are, are ultimately of service to the community,
29:22
right? And and being open, you know,
29:26
being transparent. So we open source all our stuff
29:29
of course, and including the data and the training and the
29:32
weights and the model, etc, etc. And that helps, right?
29:36
That goes a long way because people can criticise it, people
29:39
can improve upon it, people can find bugs, right?
29:44
And, and so, and, and people, the users of your technology can
29:50
look at exactly what you did and perhaps offer feedback, right?
29:55
They don't need to be machine learning engineer, AI engineer
29:58
or whatever to, to like look at what you did right and, and,
30:02
and, you know, teach you or help you do a better job.
30:08
Yeah, right. So I, I think we need to be
30:10
quite humble honestly about, you know, what the technology is
30:14
that, that we're building and perhaps steer the conversation a
30:19
little bit away from say AGI or how close we are to AGI and more
30:25
towards like, what is, how is the technology actually
30:30
benefiting humanity today? And how do we ensure that it's,
30:35
you know, there's an equitable distribution of these, you know,
30:40
of this incredible innovation? Now you're speaking my language.
30:45
There is one thing that I I think is fascinating with what
30:48
you all are doing and that is in general when we talk to people
30:52
on this podcast, because it is so industry focused, right?
30:57
And mostly everybody that we've spoken to before are working at
31:01
companies. It's very clear when you're
31:04
working on a company, well, usually it's very clear that the
31:09
metric you're looking at is how much money are we saving?
31:12
How much money did we make for you?
31:15
I feel like there's a whole different question that is being
31:18
asked. So I'm fascinated to know what
31:22
is the metrics that you're looking at that is, is it how
31:26
long people are using these models?
31:28
Is it like lives saved? How do you even calculate that?
31:31
I mean, I I wanna go deep down into what are metrics that you
31:35
keep a close eye on? One of the advantages of say
31:39
being on a startup and my wife is a chief AI scientist at a, at
31:43
a, a mere learning in a tech startup.
31:45
So we, we talk about kind of this a lot and, and sort of how
31:48
startups work versus nonprofits versus academia, etc.
31:52
And I will say that it's, it's not easy, right?
31:56
And but it's, but it's really critical also.
31:59
And so there's a couple, I would say, simple strategies that we
32:06
can use. Um, like ultimately, you know,
32:11
we want to know if our actions are causally improving the
32:16
bottom line, whether it's, you know, conservation and
32:18
sustainability, it's, it's not so trivial to measure those
32:21
things. So there are, I would say, not
32:26
necessarily proxies, but good indicators that you're on the
32:28
right track, right. For example, if you're if other
32:32
people are funding your work, right, if, if, if you're getting
32:36
a lot of interest in, in your technology, right, if people are
32:41
using it, it, it, you know, those are all really good signs.
32:45
And of course, like external money is just in the sense that
32:52
being at a nonprofit, like we don't charge people for our, the
32:56
technology, right? We, we give everything away for
32:59
free. And there are ways in it to
33:04
compare, you know, our models against, you know, others in the
33:08
industry. And so, so, so we anchor on that
33:11
quite a bit and we try to be open and, and really transparent
33:15
about our models. Like I said, we, we, we open
33:18
source our, our work, right. And so, and one reason for that
33:21
is that we want to be very clear that like this is this is where
33:25
we're at. We are not perfect.
33:27
We need to do a better job. Here are the areas in which we
33:30
are not doing a great job, right Is is are there other people who
33:34
have solved, you know, some of the challenges that we have yet
33:37
to solve? So there are, umm, there are a
33:42
lot of advantages to to to being open, which you might not have.
33:47
For example, if you're at a startup where you can directly
33:50
tie sort of, you know, the money to the to the model outputs.
33:58
I see that. So it's it's almost like you by
34:01
being open, get all different kinds of metrics that you can
34:05
track one being how excited is the greater community about what
34:11
we just put out. Another being, hey, did this
34:15
create more funding? Are people seeing this as a
34:18
trigger? I imagine there's gotta be like
34:20
a little bit of a a tightrope because you don't want to just
34:23
do it for funding, right. But if that is something that
34:27
happens off of the back of what you've created, then you're
34:30
like, oh OK, that's another signal that we can look at and
34:33
you can tally up all these different signals, which is not
34:37
necessarily like how much money did we save the company or how
34:40
much money did we make the company.
34:42
It's more like a holistic view. It's.
34:45
Not simple, I should say, like and, and we are constantly
34:49
evolving our or I am constantly involving sort of my perspective
34:53
on this because I've, I've worked on many different teams
34:56
and, and this is, this is the first nonprofit that I've worked
34:59
at. And really at the end of the
35:02
day, we're just trying to ship high quality, the best, best in
35:06
class, you know, artificial intelligence.
35:09
And we do have external signals because there are, although
35:12
there are not conventional leaderboards in the sense of
35:16
like, like there are for, for LM's right now.
35:20
We do hear from from other companies, other organisations
35:23
that are computer vision in some cases is better at lower poor
35:29
quality resolution than their models are at much higher
35:33
resolution, right. So that's a good indicator.
35:36
So for example, in in satellite, and we do a lot of work with
35:38
satellite imagery. And so we have computer vision
35:40
models that are taking a bunch of satellite imagery from public
35:43
datasets. So NASA and European Space
35:45
Agency data and these are not super high res images.
35:48
These are like 1020 metres a pixel.
35:51
And we have these computer vision models, they're literally
35:53
running constantly and constant, you know, doing object
35:57
detection, doing, doing different tasks and what we've
36:01
heard from conference and stuff because we, we do our best right
36:05
to, to get our, to speak and, and, and share with the
36:08
community what we're doing. We, we've heard from competitor
36:11
or not, I shouldn't say they're competitors, but private
36:14
companies who are building these same models, right, with far
36:18
better data that, you know, we're doing or they think we're
36:24
doing a better job, right? So that's a good signal.
36:26
It's not a perfect signal and you know, it's, it's effectively
36:30
anecdotal, right? But but it is a signal.
36:33
So there's a lot of signals like that.
36:34
The other thing I'd say is that I love this work and I was just
36:38
thinking about what when I first joined, I, I remember they, they
36:46
told me one of the problems that we were gonna work on is we were
36:49
gonna with this low resolution data, we're going to infer that
36:54
some object, some, some vessel say is, is in a marine protected
36:59
area or not where it's supposed to be, right?
37:01
But we have this like really low resolution data, right?
37:03
Because we get the entire planet at low resolution.
37:08
But if you want high resolution, like James Bond sort of level,
37:12
right resolution, then you need to send a satellite, right?
37:16
Literally ask a satellite or commercial satellite to that
37:20
location and take a picture, right?
37:22
Like take a high resolution picture O we do that as well and
37:25
we have a budget to do so. But just imagine if you're
37:30
building a computer vision model and shipping it and it's in
37:33
production and it says there's a ship out here and then you're
37:37
going to based on this this evidence, you're going to
37:42
literally send a satellite and that's going to be your method
37:46
of validation for that. Let's controller model.
37:50
Yeah, it's you need to be confident, right?
37:54
And so that's why I was kind of speaking about how critical it
37:59
is to get into production as quickly as possible, right?
38:03
Get into the real world as quickly as possible.
38:05
You don't want to be wrong. Well, you don't want to be wrong
38:08
that often, right? If you're having to task a
38:10
satellite to a specific location, which is not cheap.
38:15
So I kind of have a picture of how this works in my head, but I
38:19
would love for you to explain the blank parts to me.
38:23
You have data streams, which are satellite imagery.
38:26
You're getting those from different sources, disparate
38:28
sources, which are all public datasets, right?
38:30
They're low resolution. And you have something like
38:35
you're trying to make sure that some fisherman or some boat is
38:40
not where it shouldn't be. Maybe it's fishing where they
38:43
shouldn't be fishing, or maybe it is just because of the path
38:47
that you're supposed to be taking and you have this
38:51
computer vision model that is saying yes or no.
38:55
It's kind of like identifying objects and boats and then you
39:03
can say, well this one, we have X percent confidence that
39:08
there's a boat in this area, so we're gonna send a satellite and
39:12
get a high resolution photo. Now you get that ground truth,
39:18
it then goes back. I imagine you have some way to
39:22
reincorporate that data and then potentially retrain the model.
39:26
How does that look like? What happens then once you've
39:29
got that and you say, OK, this was a hit or this was a miss, We
39:33
want to update the model to make sure that we don't send out
39:37
satellites randomly to random places when they don't need to
39:40
be or we can get a higher confidence score that this is
39:45
actually like what it's saying. It is how are you basically
39:49
continuously training that model?
39:52
What is? What does that pipeline look
39:54
like? Yeah, I'm a big believer in
39:58
continuous iteration and continual improvement and and,
40:02
you know. Pretty much no matter where you
40:06
start, right, on the sort of performance axis of a machine
40:09
learning model, if you're continuously improving through
40:12
whatever means, eventually you will be doing a pretty good job,
40:16
right? So I think folks in at least
40:19
what I've seen is that a lot of teams, a lot of ML researchers
40:23
and and engineers sort of underestimate that, that
40:26
feedback loop, a lot of times it's extremely expensive, right?
40:31
So clearly GBT 4 or whatever is not being retrained with every
40:35
thumbs down in the app. I I don't know how I think what
40:40
are we talking about like 100 million ish, right.
40:42
Some some some something along those lines.
40:44
Yeah, we're not retraining it on the fly.
40:48
He also have not as large of models as GBT 4, but we are
40:54
running some fairly large computer vision models which
40:57
cannot be retrained as part of C ICD.
41:00
It just wouldn't be, it wouldn't make sense, right?
41:02
It would be far too costly. And and yet we want to have a we
41:07
want to be able to respond. We want to not make the same
41:09
mistake twice because our users tend to be experts in this
41:13
space, right? They're, they're generally
41:14
speaking maritime intelligence experts in this specific context
41:17
of, of skylight. And so if they say this, this
41:21
was wrong, as you said, we, we want to respond to that.
41:26
What we, what I found very useful to avoid having to
41:31
retrain the big expensive, you know, deep net like feature
41:35
extraction or whatever you're using.
41:36
That's kind of the base computer vision model.
41:38
It's just put it, put a little lightweight teaching model, a
41:41
little lightweight teaching neural net on top of that model.
41:45
Very small, right? Like just a simple CNN say in
41:49
the context of so our a lot of our base feature extractors
41:52
we're using swinge Transformers now which kind of the field has
41:56
moved to Transformers and it makes sense.
41:59
But anyways, the, the, the, the you can put a very lightweight
42:05
CNN on top of these big models that's trained via supervised
42:09
learning from just literally like correct, incorrect.
42:13
And we have in our, in our platform, in the skylight
42:16
platform, there's just a simple thumbs up, thumbs down, right?
42:20
And so that is enough signal to train a lightweight CNN that can
42:25
be put on top of your big model, right?
42:29
That is that, that simple model can easily be trained as part of
42:35
CICDI mean it's super cheap and we have these things running and
42:39
I've open sourced or we've opened sourced one of them.
42:43
It's online, you can check it out.
42:44
But it's, it's literally just a big net and then a simple CNN on
42:48
top of it. And that part is continuously
42:50
retrained. And it's, we're only talking
42:52
about like 100,000 parameters or something, right?
42:55
Like it's super tiny, but it's enough of a teaching signal to
43:00
refine the outputs, right? So that we don't make the same
43:04
mistake again. Or, or maybe there's a, an oil
43:08
platform or a lot of cases like there's new infrastructure being
43:12
built on the open ocean and the high seas, like wind turbines or
43:16
whatever. And the model just literally
43:17
hasn't seen in construction wind turbines as part of its training
43:22
data, right? It's just, it just didn't happen
43:24
to catch that right when the satellite was, was passing
43:27
overhead. So, and we're seeing a lot more
43:28
of this, right, because wind turbines are, are, are coming up
43:31
more and more. And so we can just put these
43:34
little, these lightweight models on top of the, the, the big
43:37
models, retrain those as part of CI CD and then and make sure we
43:41
have regression testing and integration testing so that
43:43
we're not actually getting worse over time.
43:45
And so that's what we've done in the case of the computer vision
43:48
models. It's not applicable to all these
43:50
cases, but it is. I, I found it extremely cost
43:55
effective and fairly straightforward, right?
43:58
These are simple Nats like, and they're so light that they
44:02
literally run on CPU like a GitHub runner, right?
44:05
You don't even need a GPU. I mean, it's, it's, it's cheap.
44:10
Do you only have one of these models, or are there a whole
44:15
suite of them testing for different things in general?
44:18
We have many different models so and not just for like object
44:23
detection and satellite imagery, but we have many different kinds
44:26
of data sources and satellite imagery is a big, is a big use
44:30
case. So I've, I've talked a fair
44:31
amount about it, but the, you know, we have tonnes of
44:36
different data sources and not just public data, but commercial
44:39
data to, you know, we get whatever data is necessary to
44:45
help our users, right? And our users, by the way, in
44:48
the case of Skylight are, are under resourced coastal state
44:51
governments. And so they, they rely on this
44:54
technology to protect their oceans worldwide.
44:59
And, and as I said, they're maritime.
45:02
They're, they're experts, generally speaking and what
45:04
they're looking at. And so the, the, the, the other
45:09
very useful feature that I've found is to, to help improve the
45:13
models and continuously improve is to just literally show the
45:18
user what the model was seeing, right?
45:22
Just just a simple crop, for example, it seems very basic,
45:25
but like, and it's harder to do for Transformers and for NLP,
45:29
but just literally show, just be transparent, like here's what
45:32
the model saw. Do you think it's correct?
45:36
And if not, just give us a thumbs down and we'll try not to
45:40
make the same mistake again. So it's very simple, right?
45:43
It's very basic, but you know, allowing users to be part of the
45:49
improvement process, I think is really critical and it goes a
45:54
long way. And so are you saying that
45:57
you've got that one gigantic model and then these smaller
46:03
lightweight models and you have various lightweight models that
46:07
sit on top of it? So whatever this gigantic model
46:10
output something, you've got different edge cases that these
46:15
smaller models are looking for. Or like in the case of, OK,
46:19
there's a wind turbine that's being built, this smaller model
46:22
knows that and a few other things which it's been briefed
46:26
on by the experts. Do you have like various
46:30
different models that are catching that output, or is it
46:33
one that is good enough because there's only a few ways that it
46:37
deviates? I see where you're saying and,
46:40
and no, we have many different models, many different big base
46:43
models that that are doing these and, and in general, you know,
46:47
I, I understand and I'm sensitive to the fact that
46:50
folks, especially in the research and academic community,
46:54
like having the one model, like the, the one big model that can
46:59
do everything right. And I totally get that.
47:02
And it makes sense. Like after all, we have one
47:04
brain, right? But it's is from from what I've
47:10
experienced and in our for our use cases, it's it's makes much
47:14
more sense to develop bespoke models for each data source or
47:19
specific use case. So each satellite for example,
47:22
or each. Ohh, I see.
47:24
Yeah, each SO. We, we have computer vision
47:26
models for many different satellites, right?
47:28
And they're, the data sources can be completely different.
47:31
So you might be thinking of optical data where it's just
47:34
like RGB or whatever. But also there's, there's tonnes
47:38
of different satellites. I mean, there are like NASA and
47:40
the European Space Agency and then all of the commercial
47:43
satellites. There are so many different ways
47:45
of taking pictures and getting data from the Earth, from the
47:49
atmosphere only God yeah, you know, so, so there's tonnes of
47:53
different types of data. And and I'll just give one
47:56
simple example. So one of the models that we
47:58
have that's open source, it's called Veers vessel detection.
48:01
So it's it's night time, it's night, Night Lights.
48:05
Basically, you can turns out that you can pretty easily
48:10
identify lights from public data on the high seas in real time.
48:16
They're they're literally this data exists.
48:18
So I didn't know that before I joined this team, but turns out
48:21
that that's the case. And so it's it's literally
48:23
sensitive. The sensor on a board of beer
48:25
satellite. It was specifically designed to
48:28
detect for, for atmospheric use cases for like clouds, basically
48:35
to look at clouds. That's why it was launched by
48:37
NASA and Noah. It was for clouds and weather
48:41
prediction and, and those kinds of things.
48:43
But um, and moonlit clouds specifically, it turns out that
48:47
you can repurpose it pretty effectively, actually very
48:50
effectively for lights. And I mean, you can see, it's
48:55
crazy. It's so exquisitely sensitive.
48:58
You can see single like street lights on highways.
49:03
It's unbelievably sensitive, but it's low resolution.
49:08
So it's because we're covering the entire planet.
49:11
So in general, there's a tradeoff between, you know,
49:13
spatial coverage and latency and resolution.
49:16
So it, it gets, we get the whole planet multiple times a day and
49:20
night actually from multiple satellites.
49:22
And Noah just launched another one actually.
49:24
So there are now 3 satellites. These are constantly circling
49:28
the globe. They're polar orbiting.
49:30
They're just taking pictures all the time of lights everywhere,
49:33
not just the ocean, but land too.
49:35
And so Google has like Earth at night.
49:37
And that's, that's where this image comes from.
49:40
And, and Apple has it too. Actually, I recently noticed on
49:42
my phone that they added it to their Apple Maps or whatever you
49:46
can see like the planet at night.
49:47
This is the these satellites that I'm talking about are the
49:50
data that produces these these pretty pictures just just for
49:54
context. And so these are night, Night
49:58
Lights basically so very different from RGB.
50:02
And I found that or we found that they really benefited from
50:08
custom like bespoke architectures as opposed to
50:13
using some kind of like conventional grayscale or RGB
50:19
computer vision model well. So I understand now much better.
50:24
It's all these different types of data sources that you have.
50:29
You're creating models that are hyper custom for that data
50:35
source because they're going to perform way better than some
50:39
gigantic model that understands all the data sources and, and so
50:44
that, that makes total sense now.
50:46
And I I really like that. I mean, we certainly could
50:50
approach this from like a foundational model perspective.
50:53
And, and we work with, as I mentioned, we work with prior,
50:55
which is the computer vision research team at, at AI2 and,
50:59
and they build extremely powerful or you know, very
51:02
performance state-of-the-art foundational models for
51:05
geospatial CV for geospatial computer vision.
51:09
And, and we use those as backbones in some of the models,
51:13
but in general, we build every, you know, new satellite or use
51:17
case. If, if the foundational model
51:21
works best for that use case, great, right?
51:23
But R bar our metric is the performance of the model, not
51:29
whether we're using a single model for every single
51:31
satellite, right? So if we can get even a percent
51:35
better performance out of a custom model or building, you
51:39
know, we will take that, right? We, we, we really strive for at
51:45
or exceeding human level performance on these tasks.
51:47
And it's really important in the context of, you know, what I was
51:52
talking about with, with maritime intelligence, because
51:54
if you're an under resource coastal state government with a
51:57
pretty tiny budget for say, conservation and, you know,
52:02
these types of problems and, and were saying that there's a
52:06
vessel somewhere out at sea which is going to cost 10s of
52:09
thousands of dollars in fuel to even get to, right.
52:13
We better be right. Yeah, we better be right.
52:17
And so we really value, you know, precision and really we
52:26
wait precision more heavily than recall, at least for these,
52:29
these specific use cases. And and we need to be damn near
52:36
perfect, right? To to be effective to be used
52:42
and and so if if if we need a custom model for some use case
52:47
then we will build it. Right, or we will, you know.
52:50
As I said, there's a team, team effort like we.
52:53
We will collaborate together and build the model that's
52:55
appropriate for that use case. This, I feel, is only gonna be
53:00
part one of what I would love to have is an infinite part series
53:06
because it feels like we just like hit the tip of the iceberg,
53:11
especially because we had a full conversation before we hit
53:14
record. And we didn't even get into any
53:17
of that stuff that we talked about before we had record,
53:19
which I wanted to talk about. So I'm going to have to have you
53:22
back on here. I really appreciate you,
53:24
Patrick, for coming on here, for doing what you're doing and
53:28
helping out humanity in this way, helping out the environment
53:32
and good old Mother Earth. And I want to just say thanks.
53:38
Thank you so much I I'm very grateful for this opportunity
53:41
and and I appreciate this conversation and and yeah we
53:45
only have one one planet right so let's let's treat it let's
53:50
treat it well take. Care of it all, right.