Latest / Satellite image deep learning / Chained Models for High-Res Aerial Solar Fault Detection

Transcript

Chained Models for High-Res Aerial Solar Fault Detection · Satellite image deep learning

• 0:05
  SPEAKER_02: Hi, everyone.
• 0:05
  SPEAKER_02: Hi, Jonathan.
• 0:06
  SPEAKER_02: Welcome to the podcast.
• 0:07
  SPEAKER_00: Hey, nice to meet you.
• 0:09
  SPEAKER_02: Brilliant.
• 0:09
  So we're here to talk about ZiteView and the solution you
• 0:12
  developed around
• 0:13
  SPEAKER_02: monitoring solar installations.
• 0:14
  SPEAKER_02: Why don't you kick us off with a bit of background to that?
• 0:17
  SPEAKER_00: Cool, yeah.
• 0:18
  SPEAKER_00: So I'm Jonathan Lewowski.
• 0:19
  SPEAKER_00: I lead the AI department at ZiteView.
• 0:22
  I've been at ZiteView for about four years and both Connor and
• 0:27
  Isaac have worked
• 0:28
  SPEAKER_00: with me at ZiteView for several years,
• 0:30
  SPEAKER_00: a part of that.
• 0:32
  SPEAKER_00: And I'll hand it over to Isaac to give his background.
• 0:38
  SPEAKER_01: Yeah, so I'm Isaac Worley.
• 0:40
  SPEAKER_01: I actually don't work at ZypeView anymore,
• 0:42
  but I was able to help publish this paper and get it across the
• 0:46
  finish line.
• 0:48
  SPEAKER_01: But yeah,
• 0:48
  I was at ZypeView for about four years and pretty much built
• 0:53
  along with Connor and
• 0:54
  SPEAKER_01: John,
• 0:55
  SPEAKER_01: the majority of all the ML models,
• 0:59
  SPEAKER_01: one of which you see here,
• 1:00
  SPEAKER_01: which is actually a pipeline of large ML models.
• 1:03
  SPEAKER_01: But I have a PhD in electrical engineering from UTSA,
• 1:06
  SPEAKER_01: and both John and Connor are also from UTSA as well,
• 1:10
  SPEAKER_01: and PhDs.
• 1:11
  SPEAKER_02: Brilliant.
• 1:13
  SPEAKER_02: Connor?
• 1:15
  SPEAKER_03: Yeah, hi, I'm Connor Wallace.
• 1:17
  SPEAKER_03: I'm a senior AI engineer at Zeitview.
• 1:20
  SPEAKER_03: I have been there for about four years with Jonathan and Isaac.
• 1:25
  Yeah,
• 1:25
  focusing in,
• 1:27
  yeah,
• 1:28
  aerial inspections,
• 1:29
  remote sensing domains,
• 1:31
  things like that.
• 1:32
  And yeah,
• 1:33
  like Isaac said,
• 1:34
  I'm finishing my PhD at UTSA,
• 1:37
  also in electrical engineering.
• 1:38
  SPEAKER_03: But yeah, excited to talk about some of the work we've done.
• 1:42
  SPEAKER_02: Brilliant.
• 1:43
  SPEAKER_02: Thank you for the intros, everyone.
• 1:45
  SPEAKER_02: Jonathan,
• 1:45
  why don't you just tell us a bit more detail about what you've
• 1:47
  been up to at
• 1:47
  SPEAKER_02: Zeitview then?
• 1:49
  SPEAKER_00: Cool, yeah.
• 1:49
  SPEAKER_00: I'll start with a brief introduction of Zeitview,
• 1:51
  and then I'll kind of give a high-level overview of this model
• 1:55
  that we built.
• 1:56
  So here at Zeitview,
• 1:58
  we inspect high-value renewable energy infrastructure.
• 2:02
  SPEAKER_00: This spans from solar to wind to utilities to commercial properties,
• 2:08
  SPEAKER_00: along with utilities such as transmission lines.
• 2:12
  SPEAKER_00: And at a high level,
• 2:12
  SPEAKER_00: what we do,
• 2:13
  we have various methods of collecting data for all these
• 2:16
  different verticals.
• 2:17
  SPEAKER_00: It could be drone, aircraft.
• 2:19
  SPEAKER_00: It could be human taken with cameras.
• 2:22
  SPEAKER_00: It could be ground robots.
• 2:24
  SPEAKER_00: We have several ways to capture this data.
• 2:26
  SPEAKER_00: And we own that piece as well.
• 2:29
  SPEAKER_00: And once that data is collected,
• 2:31
  we then will have to process that data to try to find out
• 2:34
  what's wrong with all
• 2:36
  SPEAKER_00: those different infrastructures,
• 2:37
  SPEAKER_00: pointing out what anomaly types there are,
• 2:40
  SPEAKER_00: having severities,
• 2:42
  SPEAKER_00: recommendations for maintenance,
• 2:43
  those types of things so that our customers know what exactly
• 2:48
  in their portfolio
• 2:50
  SPEAKER_00: has issues and be able to best prioritize their issues across their whole
• 2:53
  SPEAKER_00: portfolio.
• 2:55
  SPEAKER_00: And typically, these asset owners will own
• 2:58
  SPEAKER_00: infrastructure,
• 2:59
  SPEAKER_00: not just in one of those verticals,
• 3:01
  SPEAKER_00: they'll own both solar and wind and properties.
• 3:05
  So they all want to be able to have one consolidated
• 3:07
  maintenance plan that they can
• 3:09
  SPEAKER_00: basically budget for across all their assets,
• 3:12
  SPEAKER_00: not just individual assets in one vertical.
• 3:16
  SPEAKER_00: With that being said,
• 3:18
  One of the very first things we did a few years ago is what we
• 3:21
  called NAS,
• 3:22
  SPEAKER_00: which is North American Solar Scan,
• 3:24
  where we scanned every solar site that was greater than one
• 3:27
  megawatt in North
• 3:29
  SPEAKER_00: America.
• 3:31
  SPEAKER_00: And with that,
• 3:32
  SPEAKER_00: we used aircraft where we went and flew every single site,
• 3:36
  SPEAKER_00: collected all RGB and IR images of all those sites,
• 3:41
  SPEAKER_00: stitched them into orthos,
• 3:43
  SPEAKER_00: With those orthos,
• 3:44
  we are able to then train AI models to detect solar panels and
• 3:48
  anomalies on these
• 3:49
  SPEAKER_00: panels so that we could scale analysis to find all the various issues.
• 3:54
  And there's several different issues,
• 3:55
  which I'll let Connor and Isaac kind of go into.
• 3:57
  SPEAKER_00: And once we have all those issues identified,
• 4:00
  we can have our human review team actually review all the
• 4:04
  critical anomalies,
• 4:05
  SPEAKER_00: confirm that they're correct.
• 4:07
  And then after that,
• 4:08
  we're able to assign health scores to all of the sites in North
• 4:12
  America.
• 4:13
  SPEAKER_00: And this provided two benefits.
• 4:15
  SPEAKER_00: One is if you are an asset owner,
• 4:17
  SPEAKER_00: you can quickly get a,
• 4:19
  SPEAKER_00: hey,
• 4:19
  I'm going to pay a cheap amount of money and I can get the
• 4:21
  health score for all my
• 4:22
  SPEAKER_00: sites because we've already flown those.
• 4:25
  SPEAKER_00: And then if there is sites that have major issues,
• 4:28
  then we can pay additional money to go and figure out exactly
• 4:32
  what's wrong with
• 4:33
  SPEAKER_00: each of those sites and get detailed reports.
• 4:37
  This also would serve as a marketing tool or sales tool for our
• 4:40
  sales team where
• 4:41
  SPEAKER_00: they can go and target
• 4:43
  SPEAKER_00: key customers that have sites with issues and be like,
• 4:46
  SPEAKER_00: hey,
• 4:47
  SPEAKER_00: did you know that these issues existed on your site?
• 4:49
  If not,
• 4:50
  why don't you pay us to be able to give you more details into
• 4:53
  those sites?
• 4:54
  SPEAKER_00: So at a high level, that's what we did.
• 4:58
  But I'll pass it over to Connor just to go a little bit deeper
• 5:02
  into each of those sections.
• 5:03
  SPEAKER_03: Thank you.
• 5:05
  SPEAKER_03: Yeah,
• 5:06
  SPEAKER_03: so like Jonathan alluded to,
• 5:08
  the main objective was to be able to localize and map out
• 5:14
  anomalies in a solar site
• 5:16
  SPEAKER_03: given infrared thermal imagery.
• 5:19
  SPEAKER_03: And so this is at like,
• 5:23
  really large scale,
• 5:24
  10 square kilometer sites to the largest sites in North
• 5:28
  America.
• 5:30
  SPEAKER_03: So this pipeline had to be scalable.
• 5:32
  SPEAKER_03: It also had to be accurate and robust.
• 5:34
  We wanted to maximize true positives and minimize false
• 5:38
  positives as much as possible.
• 5:41
  SPEAKER_03: And so to do that, we kind of have a multi-stage pipeline.
• 5:47
  SPEAKER_03: The kind of core algorithm behind it all is based off of
• 5:51
  SPEAKER_03: like a YOLO object detector,
• 5:53
  that its main objective is to be able to detect these anomalies
• 5:57
  in the infrared
• 5:58
  SPEAKER_03: imagery.
• 6:00
  SPEAKER_03: But that's not enough, we found.
• 6:01
  We found that the main objective is to be able to minimize
• 6:08
  false positives by kind
• 6:09
  SPEAKER_03: of leveraging some of the thermal features we see in the image.
• 6:13
  SPEAKER_03: So we have a couple of other models in that pipeline.
• 6:17
  We also have,
• 6:18
  and I can go ahead and pull up kind of a schematic of what that
• 6:22
  looks like.
• 6:27
  SPEAKER_02: Quick one.
• 6:28
  SPEAKER_02: You mentioned that you're using object detection.
• 6:30
  Was it the obvious choice or did you try a few different
• 6:32
  approaches before settling
• 6:33
  SPEAKER_02: on using object detection?
• 6:36
  SPEAKER_03: We always try a few different approaches when we first get started.
• 6:40
  SPEAKER_03: Um, uh,
• 6:44
  a few of the standard types,
• 6:46
  so classification,
• 6:47
  object detection,
• 6:49
  segmentation.
• 6:50
  SPEAKER_03: With this, you'll see in the imagery,
• 6:54
  SPEAKER_03: I'm going to share my screen real quick.
• 6:57
  SPEAKER_03: So these are the actual anomalies we were trying to detect.
• 7:00
  SPEAKER_03: So we have diode failures,
• 7:03
  SPEAKER_03: hot cells,
• 7:04
  which are these kind of localized areas of the solar panels
• 7:08
  that have gone out and
• 7:10
  SPEAKER_03: are increasing in temperature,
• 7:13
  SPEAKER_03: multiple hot cells.
• 7:15
  SPEAKER_03: An entire panel could be offline.
• 7:18
  String outages where we say an entire kind of grouping of
• 7:21
  panels has gone offline
• 7:23
  SPEAKER_03: and is therefore increasing its temperature and then misaligned panels.
• 7:28
  SPEAKER_03: And so for some of these, it just became clear that it was
• 7:34
  The most scalable solution was to just treat this as a standard
• 7:38
  object detection problem.
• 7:43
  SPEAKER_03: That's at least what we went with.
• 7:45
  But yeah,
• 7:46
  these are the main anomaly types that we were trying to detect
• 7:49
  in the imagery.
• 7:50
  SPEAKER_03: Yeah, one thing that we'll touch on, I'll probably let Isaac touch on.
• 7:56
  You can see in these images,
• 7:57
  even just in these few samples,
• 8:00
  the variation
• 8:02
  SPEAKER_03: in kind of thermal variance in the image.
• 8:07
  SPEAKER_03: Just in these is pretty drastic.
• 8:11
  SPEAKER_03: And we see this problem is kind of...
• 8:16
  SPEAKER_03: explodes once you start looking at larger and larger scale size,
• 8:19
  SPEAKER_03: because as the temperature ranges across the site,
• 8:21
  SPEAKER_03: it gets really,
• 8:22
  SPEAKER_03: really difficult to really make those features pop.
• 8:27
  SPEAKER_03: So these are the defect types we were trying to detect.
• 8:30
  SPEAKER_03: Let me go ahead and
• 8:36
  SPEAKER_00: And one thing I also want to know is to answer your question about,
• 8:39
  SPEAKER_00: did we try different model types?
• 8:40
  SPEAKER_00: So at ZiteView,
• 8:41
  SPEAKER_00: we've built out,
• 8:41
  SPEAKER_00: we call our ML generalized repo,
• 8:44
  SPEAKER_00: where we have the very most common types of models for classifiers,
• 8:48
  SPEAKER_00: option detection,
• 8:49
  SPEAKER_00: segmentation.
• 8:51
  And we have basically abstracted that out in a way that we can
• 8:53
  basically point our
• 8:55
  SPEAKER_00: models using config files to our data in AWS and just say,
• 8:59
  SPEAKER_00: try this model,
• 9:00
  SPEAKER_00: try this model,
• 9:01
  SPEAKER_00: try this model.
• 9:02
  SPEAKER_00: So-
• 9:02
  SPEAKER_00: Once we have our data set up,
• 9:04
  it's really easy for us to rapidly try all these different
• 9:08
  models and then
• 9:09
  SPEAKER_00: basically figure out which one works the best.
• 9:12
  SPEAKER_02: Right.
• 9:12
  SPEAKER_02: And was the evaluation criteria put on metrics?
• 9:15
  SPEAKER_02: Are we looking at kind of the cost of running the model or other factors?
• 9:19
  SPEAKER_00: For this fact, for this, we were, this is a really rapid project.
• 9:22
  We went from like zero to full blown inspection of all these
• 9:27
  sites in just a few months.
• 9:29
  SPEAKER_00: So this was just whatever got us across the finish line.
• 9:32
  SPEAKER_00: We did not optimize for price in this first round.
• 9:35
  However,
• 9:35
  if we do another round of this,
• 9:38
  then we would start optimizing for price.
• 9:41
  SPEAKER_02: Sorry.
• 9:43
  SPEAKER_01: Sorry.
• 9:43
  SPEAKER_01: Most of the models aren't, like, that large.
• 9:47
  SPEAKER_01: Like, for the most part, we use, like, unets and, you know, low V5s.
• 9:50
  And it's also something to mention,
• 9:53
  like,
• 9:53
  most of the jobs are batch processes.
• 9:55
  SPEAKER_01: So,
• 9:55
  SPEAKER_01: like,
• 9:57
  SPEAKER_01: when images get uploaded,
• 9:59
  SPEAKER_01: the ortho gets stitched together,
• 10:01
  SPEAKER_01: there's not this,
• 10:02
  SPEAKER_01: like,
• 10:02
  SPEAKER_01: super quick turnaround time that needs to happen.
• 10:06
  um so it's okay if like a batch job takes like an hour to run
• 10:09
  across the whole the
• 10:10
  whole ortho um as long as we are able you know if we reduce the
• 10:14
  batch size use use
• 10:15
  a you know less costly gpu instance um that actually is better
• 10:19
  for us because we
• 10:20
  don't have an actual need to run it that quickly so for this
• 10:24
  workflow speed and
• 10:25
  cost is not such a big issue so i therefore conclude that
• 10:30
  accuracy and maybe ease
• 10:31
  SPEAKER_02: of use for the main drivers
• 10:35
  SPEAKER_01: Exactly.
• 10:35
  SPEAKER_02: Yeah.
• 10:37
  SPEAKER_02: One other little question.
• 10:37
  SPEAKER_02: You mentioned that you could try the different model types,
• 10:40
  SPEAKER_02: but each of them has different requirements on annotation format,
• 10:43
  SPEAKER_02: for instance.
• 10:44
  SPEAKER_02: How did you manage that?
• 10:47
  SPEAKER_00: Yes,
• 10:47
  SPEAKER_00: we had a team do the initial labeling,
• 10:50
  SPEAKER_00: and we started with the most complex type of labeling.
• 10:54
  SPEAKER_00: And then from there,
• 10:55
  you can always work yourself backwards to the less complicated
• 10:59
  if you have the
• 11:00
  SPEAKER_00: pixel level from the start.
• 11:03
  SPEAKER_02: Right.
• 11:03
  SPEAKER_02: So you started with pixel level annotations.
• 11:05
  SPEAKER_02: And from that, you can derive the other formats.
• 11:10
  SPEAKER_00: OK, very interesting.
• 11:10
  SPEAKER_00: We try to do that as much as we can, depending on cost.
• 11:13
  SPEAKER_02: Right.
• 11:14
  SPEAKER_02: And did you annotate all of the data you captured across North America?
• 11:18
  SPEAKER_02: Or did you somehow sample it and choose a subset to annotate?
• 11:22
  SPEAKER_00: When we started this, we had a small subset.
• 11:26
  SPEAKER_00: and we train the model.
• 11:28
  SPEAKER_00: So then as we were collecting more data throughout North America,
• 11:31
  we are able to update that because you'll see there's always a
• 11:34
  human in the loop in
• 11:35
  SPEAKER_00: this process.
• 11:36
  SPEAKER_00: So in the end, we get annotations.
• 11:39
  SPEAKER_00: So now we do have
• 11:42
  SPEAKER_00: all the annotations for every single site in North America.
• 11:46
  SPEAKER_00: But when we first started, we started with a small subset of sites.
• 11:49
  SPEAKER_00: Performance got to some metric.
• 11:52
  And then as we collected more data,
• 11:53
  we got more variants,
• 11:54
  more information.
• 11:56
  SPEAKER_00: Then our models got better and better.
• 11:58
  SPEAKER_00: And then eventually they got to the point where it is today.
• 12:01
  SPEAKER_02: Right.
• 12:01
  SPEAKER_02: Okay.
• 12:02
  SPEAKER_02: Brilliant.
• 12:04
  SPEAKER_01: Yeah, we initially started with a subset over just California.
• 12:08
  And then we started expanding over the entire CONUS and the
• 12:13
  rest of North America.
• 12:14
  And we very quickly noticed our training set over California is
• 12:20
  drastically
• 12:20
  different than North America because of the temperature of
• 12:25
  different ground
• 12:25
  SPEAKER_01: materials.
• 12:27
  SPEAKER_01: you know,
• 12:27
  SPEAKER_01: depending on time of day when you fly,
• 12:29
  SPEAKER_01: the different weather,
• 12:30
  SPEAKER_01: like how cold is it outside?
• 12:32
  You'll actually see that like one of the biggest issues we had
• 12:37
  was like the
• 12:37
  SPEAKER_01: pre-processing where you're trying to make sure that,
• 12:40
  SPEAKER_01: you know,
• 12:42
  SPEAKER_01: Are we doing patch level normalization?
• 12:43
  Are we doing,
• 12:44
  you know,
• 12:45
  taking the statistics of the full ortho at that point?
• 12:48
  SPEAKER_01: The normalization,
• 12:50
  which is kind of just prevalent in like all geospatial kind of
• 12:53
  computer vision,
• 12:55
  is can I normalize this correctly to like work across different
• 12:58
  geographic areas so
• 13:00
  SPEAKER_01: that it generalizes?
• 13:01
  SPEAKER_01: That was actually our biggest issue because you really want,
• 13:05
  SPEAKER_01: for the computer vision models,
• 13:06
  SPEAKER_01: you really want to make like the hotspots pop.
• 13:09
  SPEAKER_01: And getting that,
• 13:10
  SPEAKER_01: like,
• 13:10
  SPEAKER_01: temperature range and normalization is really tough,
• 13:14
  SPEAKER_01: actually,
• 13:14
  SPEAKER_01: to make it work.
• 13:16
  SPEAKER_01: Because it may work for,
• 13:17
  SPEAKER_01: like,
• 13:17
  SPEAKER_01: 80% of sites,
• 13:18
  SPEAKER_01: and then you actually visually look at,
• 13:20
  SPEAKER_01: like,
• 13:21
  SPEAKER_01: what you normalized,
• 13:22
  SPEAKER_01: like,
• 13:22
  SPEAKER_01: the 20 other percent of sites,
• 13:24
  SPEAKER_01: and it just looks terrible.
• 13:25
  SPEAKER_01: Like, it doesn't actually make anything pop.
• 13:27
  um and so then you kind of have to figure out like a sort of
• 13:31
  like histogram
• 13:32
  equalization step where you're trying to make all these um have
• 13:36
  the same range as
• 13:38
  this other set of data um and once once you get that it's
• 13:41
  pretty easy um but i will
• 13:44
  note that like the reason why you might look at um you know our
• 13:47
  pipeline diagram
• 13:48
  and see like well why don't you just use like an instant
• 13:50
  segmentation model
• 13:52
  SPEAKER_01: Just do it end to end.
• 13:53
  SPEAKER_01: And we actually find that that actually does significantly worse.
• 13:57
  So we found that it's significantly easier to use separate
• 14:02
  models and then use
• 14:04
  SPEAKER_01: other models,
• 14:04
  SPEAKER_01: like a segmentation model,
• 14:05
  SPEAKER_01: to actually filter false positives.
• 14:08
  SPEAKER_01: Because not only do hotspots show up on the panels,
• 14:11
  SPEAKER_01: but if there are other buildings that have metal roofs and whatnot,
• 14:14
  SPEAKER_01: which is pretty common,
• 14:15
  those will have many hotspots and get picked up very easily by
• 14:19
  an object detector.
• 14:21
  being able to filter only to the areas that have the panels is
• 14:27
  actually super useful.
• 14:28
  SPEAKER_01: And we actually find that it's useful also for normalization.
• 14:31
  So if I have this anomaly on this other building that has a
• 14:34
  really high temperature
• 14:36
  SPEAKER_01: spot on it,
• 14:37
  that's actually messing with my image normalization statistics
• 14:43
  quite greatly.
• 14:44
  SPEAKER_01: So it's actually easier if you filter all the area off
• 14:48
  SPEAKER_01: and only use the statistics of,
• 14:50
  SPEAKER_01: or at least try to get only the statistics of the panels,
• 14:53
  SPEAKER_01: the panel temperatures to then re-normalize that.
• 14:56
  SPEAKER_02: Right.
• 14:57
  SPEAKER_02: So just to clarify,
• 14:58
  SPEAKER_02: you're masking anything that you've not already determined is a panel,
• 15:01
  SPEAKER_02: and you've determined it is a panel using an accurate model,
• 15:03
  SPEAKER_02: okay?
• 15:04
  SPEAKER_01: Exactly.
• 15:05
  And one real-world example of this problem that we encountered
• 15:08
  is when we did the
• 15:09
  SPEAKER_00: California scan only,
• 15:11
  SPEAKER_00: we flew mainly during the cooler season,
• 15:13
  SPEAKER_00: so it wasn't super hot.
• 15:14
  SPEAKER_00: And then when we flew Texas,
• 15:16
  SPEAKER_00: it was in the middle of the summer,
• 15:18
  and there was like rocks on the ground that would get
• 15:21
  significantly hotter than
• 15:23
  SPEAKER_00: even the hot spots on the panels,
• 15:25
  SPEAKER_00: and that would just throw off our normalizations.
• 15:28
  SPEAKER_00: So those are the type of things that we encountered that...
• 15:31
  When we started small in California and then we scaled out to
• 15:34
  be nationwide,
• 15:36
  we had to deal with a lot of those variations that we just
• 15:38
  couldn't predict ahead
• 15:39
  SPEAKER_00: of time.
• 15:40
  SPEAKER_02: Yeah.
• 15:41
  SPEAKER_02: And I think for me,
• 15:41
  the interesting bit is that you mentioned is it was kind of
• 15:44
  easier to manage
• 15:44
  multiple models coupled together as you generalized rather than
• 15:48
  just having one
• 15:49
  SPEAKER_02: model that you tried to make generalize.
• 15:52
  SPEAKER_02: Can you sort of share some intuition around why that might be?
• 15:58
  SPEAKER_01: Yeah, I think it's... Actually, I'll let Connor answer that.
• 16:01
  SPEAKER_01: I've been talking too much.
• 16:03
  I think we were probably both going to say the same thing,
• 16:06
  which is,
• 16:06
  I mean...
• 16:10
  I think that's a common thing that you see throughout all of
• 16:14
  machine learning is if
• 16:15
  you are successfully able to decompose problem into more
• 16:19
  manageable sub problems,
• 16:22
  SPEAKER_03: you're always kind of better off.
• 16:24
  SPEAKER_03: You could always build out a bigger model and hope that it
• 16:28
  SPEAKER_03: you know,
• 16:29
  learns the correct features or learns to ignore like the
• 16:31
  hotspots that Jonathan was
• 16:33
  SPEAKER_03: mentioning.
• 16:34
  SPEAKER_03: But we find that it's actually,
• 16:36
  SPEAKER_03: it's actually,
• 16:37
  SPEAKER_03: it's more intuitive from a modeling standpoint that like,
• 16:40
  let's say if you are trying to train an object detector to
• 16:43
  detect these hotspot
• 16:45
  SPEAKER_03: anomalies,
• 16:47
  SPEAKER_03: you want to just go ahead and
• 16:49
  remove any kind of biases so it can really hone in on the
• 16:53
  actual problem at hand
• 16:55
  SPEAKER_03: instead of having to learn that it needs to ignore,
• 16:57
  SPEAKER_03: you know,
• 16:57
  SPEAKER_03: kind of random hotspots in the image.
• 17:00
  SPEAKER_03: So that's kind of the intuition.
• 17:03
  SPEAKER_03: And that's why we have like this multi-stage approach.
• 17:06
  And we do that in kind of subsequent stages that build upon
• 17:10
  each other to kind of
• 17:12
  SPEAKER_03: build out the final end product.
• 17:15
  SPEAKER_03: I don't know if Isaac wants to kind of build on top of that.
• 17:19
  SPEAKER_01: No, that pretty much summarizes everything.
• 17:22
  You're trying to make the problem easier for each model in the
• 17:25
  pipeline by not
• 17:27
  SPEAKER_01: making it have to do everything altogether.
• 17:31
  And I would say this is true for pretty much all AI,
• 17:33
  even like generative AI and coding agents.
• 17:36
  SPEAKER_00: If you work with coding agents,
• 17:38
  SPEAKER_00: you know,
• 17:38
  SPEAKER_00: if you just say,
• 17:39
  SPEAKER_00: solve this really giant problem,
• 17:41
  SPEAKER_00: the coding agent's going to struggle.
• 17:43
  SPEAKER_00: You really have to be smart in how you break down problems.
• 17:46
  And I think that's really important when it comes to real world
• 17:49
  use cases,
• 17:50
  because people typically are going to want to know how does it
• 17:55
  get from point A to
• 17:55
  SPEAKER_00: point B and not just like,
• 17:57
  SPEAKER_00: oh,
• 17:57
  SPEAKER_00: it gets there.
• 17:58
  SPEAKER_00: So I think there is value in being able to explain.
• 18:01
  SPEAKER_00: how it's actually making those predictions.
• 18:04
  SPEAKER_02: Yeah.
• 18:04
  SPEAKER_02: And once you have this,
• 18:05
  SPEAKER_02: this pipeline,
• 18:06
  SPEAKER_02: presumably some parts have a much bigger impact than others.
• 18:09
  Like an early pre-processing will affect the performance and
• 18:12
  all downstream tasks.
• 18:13
  So does that kind of guide where you also spend your energies
• 18:16
  in terms of doing R and D work?
• 18:21
  SPEAKER_01: Can you read?
• 18:22
  SPEAKER_01: Absolutely.
• 18:22
  SPEAKER_01: Yeah.
• 18:23
  SPEAKER_01: So,
• 18:23
  SPEAKER_01: so basically it lets us kind of figure out like based on metrics,
• 18:27
  SPEAKER_01: like what,
• 18:28
  what model is actually having issues with the generalization in
• 18:31
  particular.
• 18:32
  SPEAKER_01: And it lets us kind of pinpoint and focus on those and,
• 18:36
  SPEAKER_01: like spending our time because obviously like you would think,
• 18:39
  SPEAKER_01: oh,
• 18:39
  I have to retrain this model and then retrain this model and
• 18:42
  then retrain this
• 18:42
  SPEAKER_01: model.
• 18:43
  And then I'm having all these updates and it becomes like too
• 18:46
  overwhelming than
• 18:46
  SPEAKER_01: just having a single model.
• 18:48
  SPEAKER_01: But we actually find that,
• 18:50
  SPEAKER_01: you know,
• 18:50
  it tends to be that like one model is underperforming and we
• 18:54
  can just create more
• 18:55
  SPEAKER_01: label data for that model in particular.
• 18:58
  SPEAKER_02: Right, okay.
• 19:00
  So you're continually evaluating all these different models and
• 19:03
  identifying where
• 19:04
  SPEAKER_02: you need to sort of bump them up,
• 19:06
  SPEAKER_02: basically.
• 19:07
  SPEAKER_02: Exactly.
• 19:08
  SPEAKER_00: And then another side benefit of doing this way,
• 19:10
  SPEAKER_00: from my experience,
• 19:11
  SPEAKER_00: almost every product I've developed,
• 19:15
  SPEAKER_00: breaking it into smaller manageable models,
• 19:17
  there ends up being other downstream things that would want to
• 19:21
  use these sub models
• 19:23
  SPEAKER_00: that maybe we don't necessarily think about,
• 19:24
  SPEAKER_00: like panel detection.
• 19:27
  SPEAKER_00: the UI might want to show where all the panels are.
• 19:30
  If we have done a straight end-to-end model and then the
• 19:34
  product team was like,
• 19:34
  SPEAKER_00: hey,
• 19:35
  we want to be able to draw all the panels on the UI so that we
• 19:38
  can do panel counts
• 19:39
  SPEAKER_00: and things like that,
• 19:41
  SPEAKER_00: then we have to build that model anyway.
• 19:42
  So I find that breaking it up into these manageable models then
• 19:47
  enables other
• 19:48
  SPEAKER_00: downstream tasks to use those models as well.
• 19:51
  SPEAKER_02: Yeah.
• 19:53
  SPEAKER_03: Another way to kind of think about it intuitively is with this problem in
• 19:59
  SPEAKER_03: particular,
• 19:59
  SPEAKER_03: we have this kind of hierarchy of
• 20:03
  SPEAKER_03: of kind of fine-grained localization that we want to achieve.
• 20:06
  SPEAKER_03: And so we start from,
• 20:07
  SPEAKER_03: you know,
• 20:07
  SPEAKER_03: if you just said,
• 20:08
  SPEAKER_03: give me the entire ortho and tell me where all the,
• 20:10
  SPEAKER_03: you know,
• 20:11
  SPEAKER_03: little hotspots are in this giant ortho,
• 20:14
  SPEAKER_03: that's going to be really,
• 20:15
  SPEAKER_03: really difficult for a model to learn that really fine-grained detail.
• 20:20
  SPEAKER_03: And so we basically like have this multi-stage pipeline where we
• 20:24
  like we basically zoom in like closer and closer and closer as
• 20:28
  the stage goes on.
• 20:29
  So we start with kind of a higher level semantic segmentation
• 20:32
  model that is like a
• 20:34
  SPEAKER_03: more global model that is able to kind of reduce the feature space.
• 20:39
  Then we have this panel detection model that just identifies
• 20:42
  the panels themselves,
• 20:43
  SPEAKER_03: the boundary of the panels that is like a little bit more fine grained.
• 20:47
  SPEAKER_03: And then finally is the actual like
• 20:49
  SPEAKER_03: root of the problem, which is detecting the anomalies.
• 20:54
  And so basically our architecture is kind of like a cascade of
• 21:02
  features in the
• 21:04
  SPEAKER_03: image going from global all the way down to pixel level.
• 21:08
  SPEAKER_02: Got it.
• 21:09
  And did you find that you needed models for every part of the
• 21:13
  pipeline or were
• 21:14
  SPEAKER_02: heuristics satisfactory in some areas?
• 21:20
  SPEAKER_00: Go ahead, John.
• 21:21
  SPEAKER_00: Yeah, I was going to say, so there was two use cases in this model.
• 21:27
  SPEAKER_00: One,
• 21:27
  SPEAKER_00: I think,
• 21:28
  SPEAKER_00: I don't know if Connor,
• 21:29
  SPEAKER_00: you have slides for this,
• 21:30
  SPEAKER_00: but one we call our operational efficiency,
• 21:32
  SPEAKER_00: which is looking for those things he had mentioned,
• 21:34
  SPEAKER_00: hotspots,
• 21:36
  SPEAKER_00: diode failures and panel off lines.
• 21:39
  SPEAKER_00: And those can actually be mapped to how much power you're losing.
• 21:42
  SPEAKER_00: So a hotspot, you'd be losing about 15% power or so per hotspot.
• 21:48
  SPEAKER_00: A diode failure, about 33%.
• 21:50
  SPEAKER_00: And then a panel offline, you're losing 100% power for that panel.
• 21:54
  So you can actually map that to how much money are you losing
• 21:58
  because of these anomalies.
• 22:00
  SPEAKER_00: But on the flip side, we also have our temperature safety rating.
• 22:04
  And that is basically what's the likelihood of a fire happening
• 22:08
  because of this
• 22:10
  SPEAKER_00: temperature difference.
• 22:11
  So if there's a delta or a difference in the temperature or a
• 22:14
  hotspot gets too hot,
• 22:16
  SPEAKER_00: it can actually cause a fire,
• 22:18
  SPEAKER_00: which is a safety hazard.
• 22:20
  SPEAKER_00: So using traditional,
• 22:22
  SPEAKER_00: the AI models that we were showing above,
• 22:24
  SPEAKER_00: that's how we were detecting hotspots,
• 22:26
  SPEAKER_00: diode failures,
• 22:27
  SPEAKER_00: and panel offline.
• 22:30
  SPEAKER_00: But simply by having panel outlines,
• 22:34
  SPEAKER_00: we could calculate using heuristics what is the actual delta,
• 22:37
  SPEAKER_00: which is the max temperature minus the median temperature.
• 22:39
  And if that delta is above some value,
• 22:43
  it's immediately a hazard for a fire.
• 22:46
  SPEAKER_00: And then if it's on a rooftop site versus ground mount site,
• 22:49
  SPEAKER_00: your tolerance for those deltas are changing.
• 22:52
  So those are like more statistical approaches versus AI
• 22:56
  approaches that still rely
• 22:57
  SPEAKER_00: on the underlying assumption that we could detect where these things are.
• 23:01
  SPEAKER_02: Right.
• 23:03
  SPEAKER_02: So you need the ML model to identify and localize,
• 23:06
  but then you can just go back to kind of standard calculations
• 23:10
  and alerts generated
• 23:12
  SPEAKER_02: off those.
• 23:14
  SPEAKER_01: Correct.
• 23:14
  SPEAKER_01: Yeah.
• 23:14
  SPEAKER_01: And if you talk to the actual like,
• 23:17
  SPEAKER_01: you know,
• 23:17
  SPEAKER_01: before we had joined the company,
• 23:18
  SPEAKER_01: Zeitview,
• 23:20
  SPEAKER_01: which used to be called Drumbase,
• 23:22
  SPEAKER_01: had a lot of like solar analysts already at the company.
• 23:24
  And if you talk to them and we kind of like dived in deep to
• 23:29
  kind of understand
• 23:29
  SPEAKER_01: what their like analysis pipeline is,
• 23:31
  SPEAKER_01: like what do they do when they open up an ortho?
• 23:35
  SPEAKER_01: And look for this.
• 23:36
  SPEAKER_01: They actually will like use this.
• 23:38
  SPEAKER_01: They used to use schematics of the actual site.
• 23:41
  SPEAKER_01: And then obviously it would be like misaligned to the ortho.
• 23:45
  So they would have to readjust it to like overly on top of the
• 23:48
  actual panels in the ortho.
• 23:50
  SPEAKER_01: And then they would use that to look at temperatures for each panel.
• 23:53
  SPEAKER_01: Yeah.
• 23:54
  And so we're basically just replicating that process with this
• 23:58
  curiosity-based
• 23:59
  approach where we use ML to actually get the aligned panel
• 24:03
  outlines and then we can
• 24:05
  run some generic curiosity algorithms across each panel to look
• 24:10
  for these safety
• 24:11
  SPEAKER_01: issues.
• 24:12
  SPEAKER_02: Got it.
• 24:14
  And I just want to briefly talk about the role of the human in
• 24:17
  the loop going forward.
• 24:18
  SPEAKER_02: You have this new workflow,
• 24:19
  SPEAKER_02: as you've mentioned,
• 24:20
  SPEAKER_02: some previous manual steps are now not required.
• 24:23
  What is the role of people in maintaining and building on these
• 24:26
  systems in the future?
• 24:30
  SPEAKER_00: Yeah, so this infrastructure is very critical.
• 24:33
  SPEAKER_00: So it's very costly and very expensive to run.
• 24:37
  So our customers are very keen on making sure that their
• 24:41
  performance is well.
• 24:43
  So there's always going to be that human in the loop where they
• 24:46
  expect super high
• 24:47
  SPEAKER_00: precision and recall on these anomaly types.
• 24:50
  SPEAKER_00: So really, the humans are not going anywhere.
• 24:53
  SPEAKER_00: Our goal is to make the humans more efficient so that as we go from
• 24:58
  you know,
• 24:59
  a hundred customers to a thousand customers to 10,000
• 25:01
  customers.
• 25:03
  SPEAKER_00: We don't have to scale analysis linearly.
• 25:05
  SPEAKER_00: We can hopefully scale it in like much less than linearly.
• 25:09
  And then also the quality piece is if you're looking at these
• 25:13
  orthos all day and
• 25:15
  SPEAKER_00: there's hotspots that scattered around,
• 25:17
  SPEAKER_00: you're looking at panels,
• 25:18
  SPEAKER_00: diode failures,
• 25:19
  you're having to adjust normalization values to see if you can
• 25:23
  get hotspots to pop
• 25:24
  SPEAKER_00: up and diode failures to pop up.
• 25:26
  SPEAKER_00: the humans are going to make mistakes.
• 25:28
  SPEAKER_00: Like,
• 25:28
  SPEAKER_00: I don't care who you are and how much time you spend on this,
• 25:31
  SPEAKER_00: you're going to make mistakes.
• 25:32
  SPEAKER_00: It's a very hard problem for humans to do.
• 25:35
  So our number one goal was just to increase the quality of our
• 25:39
  analysis by using this AI model.
• 25:43
  SPEAKER_02: Right.
• 25:44
  That'd be a nice place to sort of maybe talk about how people
• 25:47
  evaluate your solution.
• 25:49
  SPEAKER_02: Is there like a standardized test set?
• 25:50
  SPEAKER_02: And you can say, yes, we achieve better than average on this test set.
• 25:54
  How do people compare your product to,
• 25:56
  you know,
• 25:57
  a fully human based approach?
• 26:01
  I don't think we have truly done a comparison of fully human
• 26:05
  versus this.
• 26:06
  SPEAKER_00: Like I said,
• 26:07
  SPEAKER_00: this was like a rapid like,
• 26:08
  SPEAKER_00: hey,
• 26:09
  we should do this North America solar scan and we went from
• 26:11
  zero to what we have
• 26:13
  SPEAKER_00: today in a matter of months.
• 26:15
  So it was one of those rapid things that we just knew like
• 26:18
  simply having humans
• 26:19
  alone and trying to scale out to all the sites in North America
• 26:23
  was just not going
• 26:24
  SPEAKER_00: to happen without AI.
• 26:25
  SPEAKER_00: It would have been, we would have had to scale it to
• 26:29
  SPEAKER_00: probably 500 plus people without this.
• 26:32
  SPEAKER_00: And we knew that simply wasn't an option.
• 26:34
  So we were able to limit how much we had to scale while still
• 26:39
  maintaining quality.
• 26:41
  And we did do some benchmarks just to kind of prove the
• 26:45
  usefulness of this.
• 26:47
  SPEAKER_01: So generally what would happen is we would fly a site,
• 26:50
  SPEAKER_01: we'd run our pipeline over it,
• 26:52
  SPEAKER_01: and then we would have a human in the loop
• 26:54
  you know do it before like with and without these extra
• 26:58
  predictions um and just
• 27:00
  time how long it takes and it actually significantly increased
• 27:03
  the uh like
• 27:04
  turnaround time for the final like human reviewed version
• 27:09
  versus if they did it
• 27:10
  from scratch is is significantly different um but yeah other
• 27:15
  than that there's not
• 27:15
  SPEAKER_01: really like
• 27:16
  SPEAKER_01: There's not really open source benchmarks or anything,
• 27:19
  SPEAKER_01: mostly because high-res drone imagery is generally proprietary.
• 27:23
  SPEAKER_01: You're not going to be able to just find this online anywhere.
• 27:27
  SPEAKER_01: So we definitely had to create our own benchmarks.
• 27:31
  SPEAKER_01: And because we already have human analysts,
• 27:32
  SPEAKER_01: it helps us to just measure how much we are speeding up their workflow.
• 27:39
  SPEAKER_02: Brilliant.
• 27:40
  SPEAKER_02: So obviously anybody can sort of say,
• 27:42
  SPEAKER_02: okay,
• 27:43
  a human is now three times faster and kind of evaluate that in
• 27:46
  terms of their own
• 27:46
  SPEAKER_02: sort of business cases.
• 27:48
  SPEAKER_02: Brilliant.
• 27:49
  SPEAKER_02: You're presenting this work.
• 27:50
  SPEAKER_02: Is this for people in the industry?
• 27:53
  SPEAKER_02: Is it academic?
• 27:54
  SPEAKER_02: Where are you presenting it?
• 27:56
  SPEAKER_01: Yeah,
• 27:56
  so it was already presented at CVPR at the Perception Beyond
• 28:00
  the Visual Spectrum
• 28:02
  SPEAKER_01: Workshop,
• 28:02
  SPEAKER_01: the sister workshops at Earth Vision,
• 28:06
  SPEAKER_01: which is focused a lot on like multispectral and thermal computer vision.
• 28:12
  SPEAKER_01: And so, yeah, that was already presented.
• 28:14
  SPEAKER_01: These slides were presented by Connor at that workshop.
• 28:18
  SPEAKER_02: Brilliant.
• 28:20
  SPEAKER_02: I'm just curious, does this approach, is it kind of unusual?
• 28:24
  SPEAKER_02: Do you see other people taking this approach in different domains?
• 28:28
  SPEAKER_02: How does it stand out?
• 28:30
  SPEAKER_01: I think a lot of other companies don't really publish what they're doing,
• 28:35
  which was actually a huge push for us at ZyteView is to try to
• 28:40
  publish our work
• 28:42
  SPEAKER_01: that we're doing,
• 28:43
  SPEAKER_01: mostly because it's...
• 28:45
  SPEAKER_01: it's not really about the models, right?
• 28:47
  SPEAKER_01: Like it's, it's about gathering the, the necessary data to use.
• 28:52
  SPEAKER_01: So really if someone else was going to,
• 28:54
  SPEAKER_01: if some other company was going to like scoop our IP,
• 28:56
  SPEAKER_01: it would,
• 28:56
  they would have to basically replicate like this long process
• 29:00
  of scanning all of
• 29:01
  SPEAKER_01: the large solar sites in North America.
• 29:03
  SPEAKER_01: Like if you notice a lot of our models are nothing crazy.
• 29:06
  SPEAKER_01: We didn't,
• 29:07
  SPEAKER_01: We can try various architectures and stuff like that,
• 29:11
  SPEAKER_01: but realistically,
• 29:12
  it comes down to the data and how it generalizes across
• 29:17
  different geographic
• 29:18
  SPEAKER_01: regions.
• 29:19
  SPEAKER_00: And one thing I'll say,
• 29:20
  SPEAKER_00: too,
• 29:21
  SPEAKER_00: is that here at Zeitview,
• 29:22
  we're really firm believers in the traditional AI models that
• 29:27
  have been used now
• 29:28
  SPEAKER_00: for several years.
• 29:29
  SPEAKER_00: When you compare them to the SOTA models, the performance gains you get
• 29:35
  aren't really that significant where we find that it's putting
• 29:39
  more of our time and
• 29:40
  effort into the actual data itself and creating these simple
• 29:45
  models that can be
• 29:46
  SPEAKER_00: pieced together to solve real problems.
• 29:48
  SPEAKER_00: We get way more benefit from that.
• 29:52
  SPEAKER_00: we're going to chase some SOTA model and try to do it end to end and,
• 29:56
  SPEAKER_00: you know,
• 29:56
  SPEAKER_00: try to create the most complicated model possible for academia purposes.
• 30:01
  SPEAKER_00: But from industry purposes, we find we have this generalized repo.
• 30:05
  SPEAKER_00: We can train these models super quickly once we have the data.
• 30:09
  SPEAKER_00: So if we spend our time collecting the data,
• 30:10
  SPEAKER_00: getting it labeled correctly,
• 30:12
  we can feed it into our generalized models really quickly and
• 30:15
  piece it together.
• 30:16
  And we're able to build this whole system and get it production
• 30:18
  ready in a matter
• 30:19
  SPEAKER_00: of a few months.
• 30:22
  SPEAKER_01: Sorry, I didn't mean to talk over you.
• 30:25
  SPEAKER_01: I just want to say,
• 30:27
  SPEAKER_01: so like,
• 30:28
  SPEAKER_01: yeah,
• 30:28
  SPEAKER_01: just to kind of put on to that point,
• 30:31
  SPEAKER_01: like a lot of the models that we are using under the hood,
• 30:35
  SPEAKER_01: you know,
• 30:35
  SPEAKER_01: use libraries like segmentation models,
• 30:37
  SPEAKER_01: PyTorch,
• 30:38
  SPEAKER_01: we use Torch image models,
• 30:39
  SPEAKER_01: or Tim,
• 30:40
  SPEAKER_01: shout out to those,
• 30:42
  SPEAKER_01: and kind of standardizing our,
• 30:45
  SPEAKER_01: like,
• 30:45
  SPEAKER_01: models around the
• 30:46
  those libraries allows us to use like new pre-trained weights
• 30:50
  that come out in the
• 30:51
  SPEAKER_01: future rather than,
• 30:52
  SPEAKER_01: you know,
• 30:53
  get cloning some random like researchers repo and using their
• 30:57
  one-off model.
• 30:59
  It really becomes hard to update if we want to use some new
• 31:02
  pre-trained weights or
• 31:03
  SPEAKER_01: some new model.
• 31:04
  SPEAKER_01: We're always going to be chasing,
• 31:05
  SPEAKER_01: we're always going to be chasing the way the models like come out.
• 31:08
  SPEAKER_01: We're always, we would always be chasing like the next best model.
• 31:11
  SPEAKER_01: And that's not a great use of time.
• 31:13
  SPEAKER_03: Yeah,
• 31:14
  SPEAKER_03: our approach is that when we're starting a project,
• 31:19
  SPEAKER_03: we want to make sure that it's already production quality,
• 31:23
  SPEAKER_03: like pretty much from the get-go.
• 31:25
  SPEAKER_03: We don't want to treat projects like research projects.
• 31:30
  SPEAKER_03: We treat all of them as products pretty much like from the start.
• 31:35
  And so that puts a heavy emphasis on making sure we have a
• 31:39
  really robust kind of
• 31:40
  SPEAKER_03: centralized,
• 31:41
  SPEAKER_03: generalized training architecture.
• 31:44
  SPEAKER_03: Yeah,
• 31:45
  SPEAKER_03: but it does make it really convenient,
• 31:47
  SPEAKER_03: like Isaac said,
• 31:48
  to have kind of to be based on these libraries that already are
• 31:53
  pumping in kind of
• 31:54
  SPEAKER_03: new state of the art models for us to play with.
• 31:58
  SPEAKER_00: And we'll say an asterisk to that.
• 32:00
  That doesn't mean we don't go and try some of these state of
• 32:02
  the art stuff.
• 32:04
  SPEAKER_00: So for example,
• 32:05
  we've tested out some several VLMs and we found that Florence 2
• 32:08
  worked well for a
• 32:09
  SPEAKER_00: lot of our use cases.
• 32:10
  SPEAKER_00: So we spent the time then to really take Florence 2's code
• 32:15
  SPEAKER_00: generalize it.
• 32:16
  SPEAKER_00: And by that,
• 32:17
  I mean like wrapping it in our standard config files that we
• 32:20
  typically do for all
• 32:20
  SPEAKER_00: our model training,
• 32:22
  making sure that we create our production ready inference code
• 32:25
  around it and
• 32:27
  SPEAKER_00: integrate it into our generalized model repo.
• 32:29
  So then now we have a new model that's in our toolkit that we
• 32:32
  can easily train,
• 32:34
  easily evaluate and easily deploy in a production environment
• 32:37
  that is available to
• 32:38
  SPEAKER_00: all our ML scientists now.
• 32:40
  SPEAKER_00: So we kind of have this like,
• 32:41
  SPEAKER_00: if there is something soda we want to try,
• 32:43
  SPEAKER_00: we kind of do a quick proof of concept on some ideas.
• 32:46
  SPEAKER_00: And if it seems promising,
• 32:47
  then we spend the time to make it production ready and fit into
• 32:51
  our generic
• 32:51
  SPEAKER_00: training repo.
• 32:54
  SPEAKER_02: Brilliant.
• 32:54
  SPEAKER_02: And that, that pipeline is ready to go for the next task.
• 32:57
  SPEAKER_02: Is it what wind farms, something else?
• 32:59
  SPEAKER_00: Yep,
• 33:00
  so if we had a new task tomorrow that we thought Florence 2 is
• 33:03
  the best option,
• 33:04
  SPEAKER_00: we can now just point it at the data and say Florence2.train.
• 33:08
  SPEAKER_00: And if it works,
• 33:10
  then we can deploy it and it's quickly from training to
• 33:13
  evaluation to deployment in
• 33:14
  SPEAKER_00: a matter of weeks rather than if we had some random GitHub repo,
• 33:19
  SPEAKER_00: then we would have to like convert that to inference code.
• 33:23
  And then you go through that whole bug smashing process of
• 33:26
  trying to get inference
• 33:27
  SPEAKER_00: code working properly at scale.
• 33:29
  SPEAKER_00: So we really are pretty strict about,
• 33:32
  SPEAKER_00: like,
• 33:33
  we have to move it into the ML generalized repo before we are
• 33:37
  committing to make it
• 33:37
  SPEAKER_00: production ready.
• 33:39
  SPEAKER_02: Brilliant.
• 33:40
  SPEAKER_02: Well, congratulations to the team on the development.
• 33:43
  SPEAKER_02: And I think it's a really exciting looking solution.
• 33:46
  If people want to follow along the updates from SiteView,
• 33:49
  where's the best place?
• 33:51
  SPEAKER_00: LinkedIn is probably the best place.
• 33:53
  SPEAKER_00: So ZiteView posts quite a bit on LinkedIn.
• 33:55
  SPEAKER_00: And then I think all three of us are also active on LinkedIn.
• 33:58
  SPEAKER_00: So typically anything ZiteView related and AI related, we post about.
• 34:03
  SPEAKER_02: Brilliant.
• 34:04
  SPEAKER_02: Okay.
• 34:04
  SPEAKER_02: LinkedIn is the place.
• 34:05
  SPEAKER_02: Thanks again for joining me and see you next time.
• 34:08
  SPEAKER_00: Thanks, Robin.
• 34:09
  SPEAKER_00: Thank you.