Latest / Elon Musk Podcast / Tesla AI Day 2022 Optimus Robot reveal

Transcript

Tesla AI Day 2022 Optimus Robot reveal · Elon Musk Podcast

• 0:00
  The Black Friday savings at the Home Depot, you'll find top
• 0:02
  brand kitchen appliances with Innovative features that can do
• 0:05
  more to your holidays. Can be more ovens with built-in
• 0:08
  air, fryers for baking, the perfect cookies, dishwashers
• 0:11
  with smart Tech to clean everything from bakeware to
• 0:13
  festive mugs and high-capacity refrigerators to keep leftovers.
• 0:17
  Fresh chopped Black Friday savings and get up to 30% off,
• 0:21
  plus instantly save up, to 750 on select GE kitchen packages.
• 0:24
  At the Home, Depot a Dewar's, get more done offer.
• 0:27
  Valid m s through November 30th was only store online for
• 0:29
  details. If the Venture X card from
• 0:32
  Capital One gives you more of what you love.
• 0:34
  Like premium travel benefits and access to Taylor Swift tickets.
• 0:38
  I do love her earn five. Times miles on flights and 10
• 0:42
  times. Miles on hotels through Capital
• 0:44
  One travel. Enjoy your stay in sweet 13, 12
• 0:46
  13. That's Taylor's.
• 0:48
  Lucky number plus, get access to Taylor Swift.
• 0:50
  The air is tour presented by Capital One.
• 0:53
  Maybe I'll see you there. The Venture X card from Capital
• 0:56
  One, what's in your wallet terms of pliancy capitalone.com for
• 0:59
  details? The kissing run, Rashad JCPenney
• 1:05
  for thousands of deals solo. No coupons needed this weekend,
• 1:08
  save up to 50% on kitchen electrics from Brands like,
• 1:11
  Keurig and Cuisinart, say yes, please to Diamonds and
• 1:14
  gemstones. Now, 1999 each and bundle up the
• 1:16
  famine coats starting at 1499. We got your holiday.
• 1:24
  A valid on select items. 1118 311 20 excluded from coupons.
• 1:27
  Exclusions apply see storage AC p-- dot Comfort details.
• 1:34
  When you need Auto Parts, O Reilly Auto.com is just a few
• 1:37
  clicks away. We offer convenient options for
• 1:39
  you to get your parts quickly order online and pick up for
• 1:42
  free. At your local, O'Reilly Auto
• 1:44
  Parts store. Will even bring it out curbside
• 1:47
  or you can have your parts delivered right to your door
• 1:49
  with free shipping on most orders over. $35, visit O Reilly
• 1:52
  Auto.com The Robert can actually do a lot more than we just
• 2:04
  showed you. We just didn't want it to fall
• 2:06
  on its face, so will will show you some videos.
• 2:10
  Now of the robot, doing a bunch of other things.
• 2:15
  Yeah, which are less risky. Yeah, we're so close that screen
• 2:19
  guys. Yeah.
• 2:28
  Yeah, we wanted to show a little bit more what we've done over
• 2:30
  the past few months with a pad and just walking around and
• 2:33
  dancing on stage, just humble beginnings but you can see the
• 2:40
  autopilot neural networks running as is just retrained for
• 2:43
  the bud directly on that. On that new platform that's been
• 2:47
  watering can. Yeah, when you see a rendered
• 2:49
  view that's that's the robot. What's the best?
• 2:52
  The world of robots. He's so it's very clearly
• 2:55
  identifying objects that like, this is the object it should
• 2:58
  Pick up picking it up. Yeah.
• 3:05
  We use the same process as we did for the pie dough to connect
• 3:08
  data in train your on networks, that we didn't Deploy on your
• 3:11
  robot. That's an example.
• 3:13
  That illustrates the upper body, a little bit more, something
• 3:19
  that we like try to nail down in a few months over the next few
• 3:22
  months, I would say to Perfection.
• 3:26
  This is really an actual station in the Fremont Factory as well,
• 3:29
  but it's working at. Yep.
• 3:31
  So That's not the only thing we
• 3:43
  have to show today, right? Yeah, absolutely.
• 3:44
  So that was what you saw was what we call Bumble.
• 3:49
  See that's our sort of rough development robot using semi
• 3:54
  off-the-shelf actuators but we actually have gone a step
• 3:59
  further than that already. The teams that incredible job
• 4:03
  and we actually have an optimist bought with fully Tesla designed
• 4:07
  and built actuators battery pack control.
• 4:11
  System everything it wasn't quite ready to walk but I think
• 4:16
  it will work in a few weeks but we wanted to show you the robot.
• 4:21
  Something that's actually fairly close to.
• 4:24
  What will go into production and and show you all the things it
• 4:28
  can do. So let's bring it up.
• 4:32
  Do it. So hear you sing Optimus with?
• 5:24
  These are the with the three degrees of freedom that we
• 5:28
  expect to have in Optimus production unit 1, which is the
• 5:32
  ability to move. All the fingers independently
• 5:34
  move, the to have the thumb, have two degrees of freedom.
• 5:39
  So it has opposable thumbs and both left and right hand.
• 5:42
  So it's able to operate tools and do useful things.
• 5:46
  Our goal is to make a useful humanoid robot as quickly as
• 5:50
  possible. And we've also designed it using
• 5:54
  the same discipline that we use in designing the car, which is
• 5:58
  to say to design it for manufacturing, such that as
• 6:02
  possible to make The robot at in high volume at low cost with
• 6:07
  high reliability. So that's incredibly important.
• 6:10
  I mean, you've all seen very impressive, humanoid robot
• 6:14
  demonstrations, and that's great.
• 6:16
  But what are they missing? They're missing a brain that
• 6:20
  they don't have the intelligence to navigate the World by
• 6:24
  themselves. And they're also very expensive
• 6:27
  and made in low volume. Whereas this, this is often
• 6:32
  missed is society. And Extremely capable robot but
• 6:35
  made in very high volume. Probably ultimately, millions of
• 6:38
  units and it is expected to cost much less than a car.
• 6:44
  I just bring. So I would say probably less
• 6:47
  than 20 thousand dollars would be my guess.
• 6:56
  The potential for optimistic is I think appreciated by very few
• 7:01
  people. As usual has the demos and
• 7:08
  coming in hot. So yeah, the teams put put in
• 7:17
  and the team has put in an incredible amount of work.
• 7:20
  It's working days. You know, some days we see
• 7:23
  running the 3M oil that to get to the demonstration today.
• 7:28
  Super proud of what they've done is they've really done a great
• 7:31
  job. I just like, give a hand to the
• 7:32
  whole option was team. So you know that now there's
• 7:45
  still a lot of work to be done to refine Optimist and improve
• 7:51
  it obviously because there's just optimism version 1 and
• 7:55
  that's really why we're holding this event which is to convince
• 8:00
  some of the most talented people in the world like your guys to
• 8:04
  join Tesla and help make it a reality and bring it to fruition
• 8:09
  at. Such that it can help millions
• 8:13
  of people and the potential likes it is really boggles the
• 8:18
  mind because if say, like what is an economy and economy is
• 8:25
  sort of productive, entities times, the productivity capita
• 8:30
  times output productivity per capita at the point at which
• 8:33
  there is not a limitation on capita.
• 8:36
  The it's not clear what an economy.
• 8:38
  Even means that point it Me becomes quasi infinite.
• 8:43
  So What, you know, taken to fruition in the hopefully benign
• 8:49
  scenario the, this means a future of abundance, a future
• 8:57
  where there is no poverty where people, you can have whatever
• 9:03
  you want in terms of products and services.
• 9:09
  It really is a fundamental transformation of civilization
• 9:14
  as we know it. Obviously we want to make sure
• 9:18
  that transformation is a positive one and safe and but
• 9:24
  that's also why I think Tesla as an entity during this being a
• 9:29
  single class of stock publicly traded owned by the public is
• 9:34
  very important and should not be overlooked.
• 9:37
  I think this is essential because then if the public
• 9:41
  doesn't like, what Tesla's doing the public can buy shares and
• 9:44
  Tesla and vote. Differently.
• 9:47
  This is a big deal like it's very important that I can't just
• 9:52
  do what I want. You know, sometimes people think
• 9:55
  that that's not true. So You know, that it's very
• 10:02
  point that the corporate entity that has that makes this happen
• 10:07
  is something that the public can properly influence and so I
• 10:13
  think the Tesla structure is is ideal for that.
• 10:20
  Like I said that self-driving cars will certainly have a
• 10:25
  tremendous impact on the world. I think they will improve the
• 10:30
  productivity of Transport by at least a half order of magnitude,
• 10:35
  perhaps an order of magnitude perhaps more optimists.
• 10:40
  I think has Maybe a to order of magnitude potential Improvement
• 10:49
  in economic output. Like, it's not fair, it's not
• 10:54
  fair. What the limit we actually even
• 10:56
  is so, But we need to do is in the right way.
• 11:03
  We need to do it carefully and safely and ensure that the
• 11:07
  outcome is one, that is beneficial to civilization and
• 11:12
  one that Humanity wants can't this is also extremely important
• 11:17
  obviously. So And I hope you will consider
• 11:24
  joining Tesla to achieve those goals.
• 11:31
  It has a we really care about doing the right thing here or
• 11:34
  Spire to do the right thing and and really not pay the road to
• 11:39
  hell with good intentions. And I think the road is the road
• 11:41
  to hell is mostly paved with bad intentions, but every now and
• 11:43
  again, there's a good intention in there.
• 11:46
  So we wanted to do the right thing.
• 11:48
  So you know, consider joining us and helping make it happen with
• 11:52
  that. Let's, let's move on to the next
• 11:54
  phase. All right.
• 12:03
  So you seen a couple of robots today.
• 12:05
  Let's do a quick timeline recap. So last year we unveiled, the
• 12:09
  Tesla bought concept but a concept doesn't get us very far.
• 12:12
  We knew we needed a real development and integration
• 12:15
  platform to get real life learnings as quickly as
• 12:17
  possible. So that robot that came out and
• 12:20
  did the little routine for you guys.
• 12:21
  We had that within six months built working on software
• 12:25
  integration Hardware AIDS over the months since then.
• 12:29
  But in parallel, we've also been designing the Next Generation.
• 12:32
  This one over here. So this guy is rooted in the
• 12:37
  foundation of sort of the vehicle design process.
• 12:40
  You know, we're leveraging. All of those learnings that we
• 12:42
  already have. Obviously there's a lot that's
• 12:45
  changed since last year but there's a few things that are
• 12:47
  still the same. You'll notice we still have this
• 12:49
  really detailed focus on the true human form.
• 12:52
  We think that matters for a few reasons but it's fun we spend a
• 12:55
  lot of time thinking about how amazing the human body is.
• 12:59
  We have this incredible range of motion typically really amazing
• 13:03
  strength. Fun exercises.
• 13:05
  If you put your finger tip on the chair in front of you,
• 13:08
  you'll notice that there's a huge range of motion that you
• 13:12
  have in your shoulder and your elbow, for example, without
• 13:14
  moving your fingertip, you can move those joints all over the
• 13:17
  place. But the robot, you know, it's
• 13:20
  been a function is to do real useful work and it maybe doesn't
• 13:24
  necessarily need all of those degrees of freedom right away.
• 13:27
  So we've stripped it down to a minimum sort of 28 fundamental
• 13:30
  degrees of freedom. And then, of course, our hands
• 13:32
  in addition to that, Humans are also pretty efficient at some
• 13:37
  things and not so efficient in other times.
• 13:39
  So for example, we can eat a small amount of food to sustain
• 13:42
  ourselves for several hours, that's great, but when we're
• 13:46
  just kind of sitting around no offense, but we're kind of
• 13:49
  inefficient, we're just sort of burning energy.
• 13:52
  So on the robot platform where we're going to do is we're going
• 13:54
  to minimize that. Idle power consumption, drop it
• 13:56
  as low as possible. And that way we can just flip a
• 13:59
  switch and immediately the robot turns into something that does
• 14:02
  useful work. So let's talk about this latest
• 14:06
  generation in some detail, shall we?
• 14:09
  So on the screen here you'll see in Orange are actuators which
• 14:12
  we'll get to in a little bit. And in blue our electrical
• 14:15
  system. So now that we have our sort of
• 14:18
  human based research and we have our first development platform,
• 14:22
  we have both research and execution to draw from for this
• 14:25
  design. Again, we're using that vehicle
• 14:27
  design foundations. So we're taking it from concept,
• 14:30
  through design and Analysis and then build and validation.
• 14:36
  Along the way. We're going to optimize for
• 14:37
  things like cost and efficiency because those are critical
• 14:40
  metrics to take this product to scale eventually.
• 14:44
  How are we going to do that? Well, we're going to reduce our
• 14:46
  part count and our power consumption of every element
• 14:49
  possible. We're going to do things like
• 14:51
  reduce the sensing in the wiring at our extremities.
• 14:54
  You can imagine a lot of mass in your hands and feet is going to
• 14:57
  be quite difficult and power consumption of to move around.
• 15:01
  And we're going to centralize both our power distribution and
• 15:04
  our compute to the physical center of the platform.
• 15:08
  So in the middle of our tour, so actually it is the tour.
• 15:11
  So we have our battery pack. This is sized at 2.3 kilowatt
• 15:15
  hours which is perfect for about a full day's worth of work.
• 15:19
  What's really unique about this battery pack is it has all of
• 15:21
  the battery Electronics integrated into a single PC be
• 15:25
  within the pack. So that means everything from
• 15:27
  sensing to fusing charge management and power
• 15:32
  distribution is all on one, all in one place.
• 15:36
  We're also leveraging both our Vehicle Products and our Energy
• 15:40
  Products to roll all of those key features into this battery.
• 15:44
  So that's streamlined, manufacturing, really efficient
• 15:48
  and simple cooling, methods battery management and also
• 15:51
  safety. And of course we can leverage
• 15:54
  Tesla's, existing infrastructure and supply chain to make it.
• 15:59
  So going on to sort of our brain, it's not in the head but
• 16:02
  it's pretty close also in our tour so we have our Central
• 16:06
  Computer. So as you know, Tesla already
• 16:08
  ships, full self-driving, computers in every vehicle, we
• 16:11
  produce. We want to leverage both the
• 16:14
  autopilot hardware and the software for the humanoid
• 16:17
  platform, but because it's different in requirements and in
• 16:20
  form factor we're going to change a few things first.
• 16:23
  So we still are going to, it's going to do everything that a
• 16:26
  human brain does processing Vision data.
• 16:29
  You making split-second decisions based on multiple
• 16:32
  sensory inputs and also Communications.
• 16:35
  So to support Communications, it's equipped with wireless
• 16:38
  connectivity as well as audio support and then it also has
• 16:42
  Hardware level security features, which are important to
• 16:44
  protect both the robot and the people around the robot.
• 16:49
  So now that we have our sort of core, we're going to need some
• 16:53
  limbs on this guy and we'd love to show you a little bit about
• 16:56
  our actuators and are fully functional hands as well.
• 16:59
  But the first before we do that, I'd like to introduce Malcolm.
• 17:02
  Who's going to speak a little bit about our structural
• 17:04
  foundation for the robot. Thank you.
• 17:17
  Tessa have the capabilities finalized, highly complex
• 17:19
  systems. Don't get much more complex than
• 17:22
  a crash. You can see here a simulated
• 17:25
  crash and bottle three superimposed on top of the
• 17:27
  actual physical crash, it's actually incredible how accurate
• 17:31
  it is. Just to give you an idea of the
• 17:33
  complexity of this model. It includes every nut bolt and
• 17:36
  washer every spot Weld and it has 35 million degrees of
• 17:40
  freedom, quite amazing. And it's true to say that if we
• 17:44
  didn't have models like this, we wouldn't be able to make the
• 17:46
  safest Dazzle the world. So can we utilize our
• 17:50
  capabilities are methods from the automotive side to influence
• 17:54
  a robot? Well, we can make a bottle.
• 17:59
  And since we had crashed software use the same software
• 18:01
  here, we can make it fall down the purpose of.
• 18:04
  This is to make sure that if it falls down, ideally, it doesn't.
• 18:07
  But it's superficial damage. We don't want it to.
• 18:11
  For example, break its gearbox, that is arms.
• 18:13
  That's equivalent of a dislocated shoulder is a robot
• 18:17
  difficult and expensive to fix. So, we wanted to dust yourself
• 18:20
  off. Get on with the job.
• 18:21
  It's been different. We've also take the same bottle
• 18:27
  and we can drive the actuators, using the inputs from a
• 18:30
  previously solve bottle. Bring it to life.
• 18:34
  So, this is producing the Motions for the tasks.
• 18:37
  We want the robot to do, these tasks are picking up boxes.
• 18:40
  Turning squatting, walking upstairs, whatever the set of
• 18:44
  tasks are, we can place them bottle.
• 18:46
  This is showing just simple walking, we can create the
• 18:49
  stresses it all our components that helps us optimize the
• 18:52
  components These are not dancing robots.
• 18:58
  These are actually the modal Behavior, the first five modes
• 19:00
  of the robot. Typically when people make
• 19:03
  robots they make sure the first mode is up around the top single
• 19:07
  figures of towards 10 Hertz, who is a do this is to make the
• 19:11
  controls of walking easier for difficult to walk.
• 19:14
  If you can't guarantee where your footer, it wobbled around
• 19:17
  That's okay to make one robot. We want to make thousands, maybe
• 19:20
  Millions. We haven't got the luxury of
• 19:23
  making from carbon fibre and titanium.
• 19:25
  We want to make them on plastic, things are not quite as stiff.
• 19:29
  So we can't have these high targets of call them, dumb
• 19:32
  targets. We've got to make them work at
• 19:35
  lower targets. So is that is that going to
• 19:37
  work? Well, if you think about it,
• 19:39
  sorry about this but we just bags of soggy jelly and Bones
• 19:43
  thrown in. We're not high frequency.
• 19:46
  If I started my leg I don't vibrate at 10 Hertz.
• 19:50
  We people operate a lot of frequency so we know the robot
• 19:53
  actually. Can It just fits controls
• 19:55
  harder. So we take the information from
• 19:57
  this, the modal data and the stiffness of feeder into the
• 20:00
  control system that allows it to walk.
• 20:05
  Just changing tack slyly looking at the knee we could take some
• 20:09
  inspiration from biology and we can look to see what the
• 20:12
  mechanical advantages of the knee.
• 20:14
  Is it turns out? It actually represent quite
• 20:16
  similar for but a leak and that's quite nonlinear.
• 20:20
  That's not surprising really because if you think when you
• 20:22
  bend your leg down the talk on your knee is much more when it's
• 20:26
  bent, that is when it's straight.
• 20:28
  So you'd expect a nonlinear function and in fact that the
• 20:32
  biology is nonlinear this matched it quite accurately.
• 20:37
  So that's a reputation for by link is obviously don't
• 20:39
  physically for by link to the said the characteristics are
• 20:42
  similar but me Benny down that's not very scientific.
• 20:46
  Let's be a bit more scientific. We played all the tasks through
• 20:50
  the through this graph for this is showing picking things up
• 20:53
  walking squatting. The Tesla said we did on the
• 20:56
  stress and that's the talk at seen at the knee against the
• 21:02
  knee, bend on the horizontal axis, this is showing the
• 21:05
  requirement for the need to do all these tasks.
• 21:07
  And then put a curve through it. Surfing over the top of the
• 21:10
  piece and that's say this is what's required to make the
• 21:14
  robot. Do these tasks.
• 21:19
  So if we look at the four bar link, that's actually the green
• 21:22
  curve, and it's saying that the non-linearity of the 4 bar leak,
• 21:26
  is actually linearized. The characteristic of the force,
• 21:29
  what that really says, is that slowed, the force?
• 21:31
  That's the biggest, the actuator have the lowest possible Force,
• 21:34
  which is the most efficient we want to burn ended up.
• 21:36
  Slowly, what's the blue curve for the blue curve is actually,
• 21:40
  if we didn't have a four bar link, we just had an arm
• 21:42
  sticking out of my leg here, with an actuator on it.
• 21:46
  A simple tube our leak. That's the best you could do.
• 21:49
  Simple to bar link and it shows that that would create a much
• 21:52
  more force in the actuator we should not be efficient.
• 21:57
  So what they look like in practice?
• 21:59
  Well, It's just see, but it's very tightly packaged.
• 22:03
  In the knee you'll see a good chance found on the second.
• 22:05
  You'll see the full bar link there is operating on the
• 22:08
  actuator. This is determined the force and
• 22:11
  the displacement on the actuator and that pass you over to
• 22:14
  constitute us to tell you a lot more detail about how these
• 22:16
  actuators and they didn't design optimized.
• 22:19
  Thank you. Hey There, Michael.
• 22:28
  So I am, I would like to talk to you about the design process and
• 22:34
  actuator portfolio in our robot. So there are many similarities
• 22:39
  between a car and a robot. When it comes to powertrain
• 22:42
  design, they may be. The most important thing that
• 22:45
  matters here is energy, mass and cost.
• 22:48
  We are carrying over most of our designing experience from the
• 22:52
  car to the robot. So in the particular case, you
• 22:58
  see a car with to drive units and the drive units are used in
• 23:03
  order to accelerate the car zero to 60 MPH time or drive a City
• 23:09
  Drive side. While the robot that has 28
• 23:13
  actuators, it's not obvious. What are the tasks actuator
• 23:18
  level? So we have tasks that are higher
• 23:22
  level like walking or climbing stairs or Carrying a heavy
• 23:26
  object, which need to be translated into joint in to join
• 23:31
  specs. Therefore, we use our model,
• 23:34
  that generates the torque speed trajectories for our joints,
• 23:39
  which subsequently is going to be fed in our optimization model
• 23:44
  to run through the optimization process.
• 23:49
  This is one of the scenarios that the robot is capable of
• 23:52
  doing which is turning and walking.
• 23:55
  So when we have this torque speed trajectory, We Lay it over
• 23:59
  inefficiency map of an actuator. And we are able along the
• 24:04
  trajectory to generate the power consumption and the energy
• 24:09
  cumulative energy for the task versus time.
• 24:13
  So this allows us to define the system cost for the particular
• 24:17
  actuator and put a simple Point into the cloud.
• 24:20
  Then we do this for hundreds of thousands of actuators by
• 24:24
  solving. In our cluster and the red line,
• 24:26
  denotes, the part of round, which is the preferred area
• 24:30
  where we will look for optimal. So the X denotes the preferred
• 24:35
  actuator design, we have picked for this particular joint.
• 24:38
  So now we need to do this for every joint.
• 24:40
  We have 28 joints to optimize and we parse our Cloud.
• 24:45
  We parse our Cloud again for every joint spec.
• 24:49
  And the Red X is this time the note, the bespoke actuator
• 24:53
  designs for every joint. The problem here is that we have
• 24:57
  too many unique actuator designs and even if we take advantage of
• 25:01
  the Symmetry still, there are too many in order to make
• 25:04
  something must manufacturable. We need to be able to reduce the
• 25:08
  amount of unique actuator designs.
• 25:10
  Therefore we run something called commonality study with we
• 25:14
  parse our Cloud. Again looking this time for
• 25:18
  actuators that simultaneously Meet The Joint performance
• 25:21
  requirements for more than one joint the same time.
• 25:24
  So the resulting portfolio is six actuators and they show in a
• 25:28
  color map, the middle figure and the actuators can be also viewed
• 25:34
  in this slide. We have three rotary and three
• 25:38
  linear actuators all of which have a great output force or
• 25:43
  Torque per Mass. The rotary actuator in
• 25:48
  particular, has a mechanical clutch, integrated on the lock
• 25:51
  high-speed site, angular contact, ball bearing and on the
• 25:54
  high-speed side and on the low speed side across roller bearing
• 25:59
  into the gear. Train is a strain wave gear and
• 26:04
  there are three integrated sensors here and bespoke
• 26:09
  permanent magnet machine. The linear actuator.
• 26:19
  I'm sorry. The linear actuator has
• 26:22
  planetary rollers and an inverted planetary Screw As a
• 26:26
  gear train, which allows efficiency and compaction and
• 26:30
  durability. So in order to demonstrate the
• 26:34
  force capability of our linear actuators, we have set up an
• 26:39
  experiment in order to test it under its limits.
• 26:46
  And I will let you enjoy the video.
• 26:55
  So, our actuator is able to lift.
• 27:05
  Science, proves quality sleep, is vital to your mental
• 27:08
  emotional and physical health. The Sleep Number 360, smart bed,
• 27:12
  senses, your movements and automatically, adjust to help.
• 27:14
  Keep you both effortlessly comfortable, and it's
• 27:17
  temperature balancing. So you stay cool.
• 27:19
  So you're your best for yourself and those you care about most
• 27:22
  life-changing. Sleep only from Sleep Number.
• 27:24
  Don't miss our weekend, special save, 50% on the Sleep.
• 27:27
  Number 360 Limited Edition, smart bed plus special financing
• 27:29
  ends, Monday to learn more. Go to sleep number.com, special
• 27:32
  financing subject to credit approval.
• 27:33
  Minimum monthly payments quiet see store for Has the Black
• 27:36
  Friday savings at the Home Depot.
• 27:37
  You'll find top brand kitchen appliances with Innovative
• 27:40
  features that can do more to your holidays.
• 27:42
  Can be more ovens with built-in air, fryers for baking, the
• 27:44
  perfect cookies, dishwashers with smart Tech to clean
• 27:47
  everything from bakeware to festive mugs and high-capacity
• 27:51
  refrigerators to keep leftovers. Fresh chopped Black Friday
• 27:55
  savings and get up to 30% off, plus instantly save up, to 750
• 27:58
  on select GE kitchen packages, at the Home, Depot a Dewar's,
• 28:01
  get more done off of Alabama, second through November 30th us
• 28:04
  only store online. Details.
• 28:07
  A halftone. 9-foot concert grand, piano. and, This is a
• 28:19
  requirement, it's not something nice to have because our muscles
• 28:24
  can do the same when they are direct driven.
• 28:26
  When there are directly driven or quadricep muscles can do the
• 28:30
  same thing. It's just that they need is an
• 28:33
  up. Hearing linkage system that
• 28:35
  converts the force into velocity at the end effector of our heels
• 28:39
  for purposes of giving to the human body agility.
• 28:44
  So this is one of the main things that are amazing about
• 28:46
  the human body and I'm concluding my part at this point
• 28:50
  and I would like to welcome my colleague.
• 28:51
  Mike was going to talk to you about hand design.
• 28:54
  Thank you very much. Say something penis.
• 29:01
  We just saw how powerful a human and a humanoid actuator can be.
• 29:06
  However, humans are also incredibly dextrous Human hand
• 29:12
  has the ability to move at 300 degrees per second.
• 29:15
  There's tens of thousands of tactile sensors.
• 29:19
  It has the ability to grasp and manipulate almost every object
• 29:22
  in our daily lives. For our robotic hand design, we
• 29:26
  were inspired by biology. We have five fingers and
• 29:29
  opposable thumb. Our fingers are driven by
• 29:32
  metallic tendons that are both flexible and strong. because the
• 29:36
  ability to complete wide aperture power grasps while also
• 29:39
  being optimized for precision gripping of small, thin and
• 29:43
  delicate objects, So why a human like robotic hand?
• 29:48
  Well, the main reason is that our factories in the world
• 29:50
  around, us is designed to be ergonomic.
• 29:53
  So what that means is that it ensures that objects in our
• 29:55
  Factory are graspable. But it also ensures that new
• 29:58
  objects that we may have never seen before can be grasped by
• 30:01
  the human hand and by our robotic hand as well, the
• 30:05
  converse there is pretty interesting because it's saying
• 30:08
  that these objects are designed to our hand instead of having to
• 30:10
  make changes to our hand, to accompany a new object.
• 30:15
  Some basic stats about our hand is that a six actuators and 11
• 30:18
  degrees of freedom? It has an in hand controller,
• 30:21
  which drives the fingers and receive sensory feedback sensor,
• 30:25
  feedback is really important to learn a little bit more about
• 30:27
  the objects that were grasping and also for proprioception and
• 30:31
  that's the ability for us to recognize where hand is in
• 30:33
  space. One of the important aspects of
• 30:37
  our hand is that it's adaptive this adaptability is involved.
• 30:40
  Essentially as complex mechanisms that allow the hand
• 30:42
  to adapt, the objects being grasped.
• 30:46
  Another important part is that we have a non back drivable
• 30:48
  finger drive this clutching mechanism allows us to hold and
• 30:51
  transport objects without having to turn on the hand Motors.
• 30:55
  You just heard how we went about going, we went about designing
• 30:58
  the test about Hardware. Now I'll hand it off to Milan
• 31:01
  and our autonomy team to bring this robot to life.
• 31:12
  All right, so all those cool things we've shown earlier in
• 31:16
  the video we're posted possible just in a matter of a few
• 31:19
  months, thanks to the amazing work that we've done a little
• 31:22
  pilot over the past few years most of those components ported
• 31:26
  quite easily over to the but environment.
• 31:28
  If you think about it we're just moving from a robot on Wheels to
• 31:31
  our beloved on legs. So some of them components are
• 31:34
  pretty similar and some other require more heavy lifting The
• 31:39
  for example, our computer vision you are networks or poorly
• 31:43
  directly from autopilot to the Bots situation.
• 31:47
  It's exactly the same occupancy Network that will talk into a
• 31:50
  little bit more details later with the autopilot team, that is
• 31:53
  now running on the. But here in this video, the only
• 31:56
  thing that change really is the training data that we had to
• 31:58
  recollect. We also trying to find ways to
• 32:03
  improve those occupancy networks.
• 32:06
  Using work made on your Radiance fields to get really great
• 32:10
  volumetric, rendering of the butts environments, for example
• 32:13
  here. So Machinery, that the but might
• 32:15
  have to interact with. Another interesting problem to
• 32:21
  think about is in indoor environments, mostly with that
• 32:25
  sense of GPS signal. How do you get about to navigate
• 32:28
  to its destination? Say, for instance, to find its
• 32:31
  nearest charging station. So, we've been training, module
• 32:35
  networks to identify high frequency features, key points,
• 32:39
  within the buds, camera streams and track them across frame over
• 32:43
  time as the but navigates through its environment.
• 32:46
  And we're using those points to get a better estimate of the
• 32:50
  butts pose and trajectory within its environment as it's working.
• 32:57
  We also did quite some work on the simulation side.
• 32:59
  And this is literally the autopilot simulator to which
• 33:02
  we've integrated the robot Locomotion code.
• 33:05
  And this is a video of the motion control code running in
• 33:09
  your palate, similar simulator showing the evolution of the
• 33:12
  robots, work overtime. So, as you can see, we started
• 33:15
  quite slowly in April and start accelerating as you unlock more
• 33:18
  joints and deeper more Advanced Techniques, like arms balancing
• 33:22
  over the past few months. So Locomotion is specifically
• 33:26
  one component that's very different as we're moving from
• 33:29
  the car to the but environment and so I think it warrants a
• 33:32
  little bit more depth and I'd like my colleagues to start
• 33:35
  talking about this now Thank you Milan.
• 33:46
  Hi everyone. I'm Felix.
• 33:47
  I'm a robotics engineer on the project and I'm going to talk
• 33:50
  about walking walking seems easy, right?
• 33:54
  People do it every day, you don't even have to think about
• 33:57
  it but there are some aspects of walking which are challenging
• 34:01
  from an engineering perspective, for example, physical self
• 34:05
  awareness. That means having a good
• 34:08
  representation of yourself. What is the length of your
• 34:10
  limbs? What is the mass of your limbs?
• 34:13
  What is the size of your feet? That matters also having an
• 34:17
  energy-efficient gauge, you can imagine just different styles of
• 34:20
  walking and all of them are equally efficient.
• 34:25
  Most important, keep balance, don't fall and of course also,
• 34:29
  coordinate the motion of all of your limbs together.
• 34:33
  So now humans do all of this naturally but as Engineers or
• 34:37
  what is this? We have to think about these
• 34:39
  problems and if I'm going to show you how we address them in
• 34:42
  our Locomotion planning and control stack, so we start with
• 34:46
  Locomotion planning and our representation of the bond.
• 34:49
  That means a model of the robots kinematics, Dynamics, and a
• 34:52
  contact properties, and using our model and the desired path
• 34:57
  for the Box. Our Locomotion planner generates
• 35:00
  reference trajectory is for the entire system.
• 35:04
  This means feasible trajectories with respect to the assumptions
• 35:08
  of our model. The planet currently Works in
• 35:11
  three stages. It's starts planning footsteps
• 35:14
  and ends with the entire motion photosystem and let's dive a
• 35:18
  little bit deeper in how this works.
• 35:20
  So in this video we see footsteps being planned over a
• 35:23
  planning Horizon following the desired path and we start from
• 35:28
  this and add. Then for trajectories that
• 35:31
  connect these footsteps using too often heel strike just as
• 35:34
  the humans just as humans do and this gives us a larger stride
• 35:39
  and less You've been for high efficiency of the system.
• 35:43
  The last stage is then finding a center of mass trajectory, which
• 35:46
  gives us a feat dynamically feasible, motion of the entire
• 35:49
  system to keep balance. As we all know, plans are good,
• 35:54
  but we also have to realize them in reality.
• 35:57
  Let's say, how see how we can do this.
• 36:08
  Thank you, Felix. Hello everyone.
• 36:10
  My name is Anand. I'm going to talk to you about
• 36:12
  controls. So, let's take the motion plan
• 36:16
  that Felix just talked about and put it in the real world on a
• 36:20
  real robot. Let's see what happens.
• 36:25
  It takes a couple steps and falls down.
• 36:28
  Well, that's a little disappointing, but we are
• 36:31
  missing a few key pieces here, which will make it work.
• 36:36
  Now, as Felix mentioned, the motion planner is using an
• 36:40
  idealized version of itself and a version of reality around it.
• 36:45
  This is not exactly correct it. Also expresses its intention
• 36:50
  through trajectories and wrenches, wrenches, our forces
• 36:54
  and torques that it wants to exert on the World to locomote.
• 37:00
  Reality is way more complex than any similar model.
• 37:04
  Also, the robot is not simplified.
• 37:06
  It's got vibrations and modes, compliance sensor noise, and on,
• 37:11
  and on, and on. So what does that do to the real
• 37:15
  world when you put the body in the real world?
• 37:18
  Well, the unexpected forces caused on model Dynamics, which
• 37:22
  essentially the planet doesn't know about and that causes
• 37:24
  destabilization, especially For A system that is dynamically
• 37:29
  stable, like bipedal locomotion. So what can we do about it?
• 37:33
  Well, we measure reality, we use sensors and our understanding of
• 37:38
  the world to do state estimation and status to me here, you can
• 37:42
  see the attitude and pelvis pose, which is essentially the
• 37:45
  vestibular system in a human along with the center of mass
• 37:49
  trajectory being tracked when the robots walking in the office
• 37:52
  environment. Now we have all the pieces we
• 37:56
  need in order to close the loop. So we use our better bot model.
• 38:01
  We use the understanding of reality that we have gained
• 38:04
  through State estimation and we compare what we want, which is
• 38:08
  what we expect the reality expect, the reality is doing to
• 38:11
  us in order to add corrections to the behavior of the robot.
• 38:18
  Sure the robot certainly doesn't appreciate being poked but it
• 38:22
  doesn't admit Bill job of staying upright.
• 38:26
  The final Point here is a robot that walks is not enough.
• 38:31
  We need it to use its hands and arms to be useful.
• 38:35
  Let's talk about manipulation. Hi everyone.
• 38:49
  My name is Eric, robotics engineer on Shabbat and I want
• 38:53
  to talk about how we made the robot manipulator things in the
• 38:56
  real world. We wanted to manipulate objects
• 38:59
  while looking as natural as possible and also get there
• 39:03
  quickly. So, what we've done is, we've
• 39:06
  broken this process down into two steps.
• 39:08
  First is generating, a library of natural motion references or
• 39:12
  we could call them demonstrations and then we've
• 39:14
  adapted these motion references online, to the current
• 39:17
  real-world. Operation.
• 39:20
  So let's say we have a human demonstration of picking up an
• 39:23
  object. We can get a motion capture of
• 39:25
  that demonstration which is visualized right here as a bunch
• 39:29
  of key frames representing the locations, the hands, the elbows
• 39:32
  at or so we can map that to the robot using inverse kinematics.
• 39:37
  And if we collect a lot of leaves, now, we have a library
• 39:40
  that we can work with. But a single demonstration is
• 39:45
  not generalizable to the variation in the real world.
• 39:48
  For instance, this would only work for a box in a very
• 39:51
  particular local location. So we've also done is run these
• 39:56
  reference trajectories through a trajectory optimization program,
• 40:00
  which solves for where the hand should be, how the robot should
• 40:03
  balance during when it meets the adapt the motion to the real
• 40:08
  world. So for instance, if the box is
• 40:11
  In this location. Then our Optimizer will create
• 40:15
  this trajectory instead. Next Lawns going to talk about.
• 40:22
  What's next for The Optimist. Tesla lat.
• 40:24
  Thanks Thanks. Right.
• 40:33
  So hopefully by now you guys got a good idea of what we've been
• 40:35
  up to over the past few months. We start having something that's
• 40:39
  usable but it's far from being useful to still a long and
• 40:43
  exciting road ahead of us. I think the first thing within
• 40:46
  the next few weeks is to get Optimist at least at par with
• 40:50
  one both. See the other but prototype you
• 40:52
  saw earlier and probably Beyond. We also going to start focusing
• 40:57
  on the real, use case at one of our factories and really going
• 41:01
  to try to try to Nail this down and I run out all the elements
• 41:05
  needed to deploy. This product in the real world,
• 41:08
  I was mentioning earlier, you know, indoor navigation,
• 41:11
  graceful for management or even servicing.
• 41:14
  All components needed to scale this product up but I don't know
• 41:19
  about you. But after seeing what we shown
• 41:22
  tonight I'm pretty sure we can get this done within the next
• 41:24
  few months or years and make this project a reality and
• 41:28
  change the entire economy. So I would like to thank the
• 41:31
  entire Optimus team for their hard work over the past few
• 41:35
  months. I think it's pretty amazing.
• 41:36
  All of this was done in barely six or eight months.
• 41:39
  Thank you very much. The kids run, Richard JC Penney
• 41:51
  for thousands of deals solo. No coupons needed this weekend,
• 41:55
  save up to 50% on kitchen electrics from Brands, like,
• 41:57
  Keurig and Cuisinart, say yes, please to Diamonds and
• 42:00
  gemstones. Now, 1999 each and bundle up the
• 42:03
  famine coats starting at 1499. We got your holiday excluded
• 42:13
  from coupons. Exclusions.
• 42:15
  Apply see storage acb.com Tails the Venture X card from Capital.
• 42:19
  One gives you more of what you love like premium travel
• 42:22
  benefits and access to Taylor Swift tickets.
• 42:25
  I do love her or in five times miles on flights and 10 times.
• 42:29
  Miles on hotels through Capital One travel.
• 42:31
  Enjoy your stay in sweet 13, 12 13.
• 42:34
  That's Taylor's. Lucky number plus, get access to
• 42:37
  Taylor Swift. The era's tour presented by
• 42:39
  Capital One. Maybe I'll see you there.
• 42:41
  The Venture X card from Capital One.
• 42:43
  What's in your wallet terms applying see capitalone.com for
• 42:46
  Details. Hey, everyone.
• 42:58
  I am Ashok. I lead the autopilot team
• 43:00
  alongside Milan. God is coming so hard to top
• 43:04
  that Optimus section will try nonetheless.
• 43:09
  Anyway. Every Tesla that has been built
• 43:12
  over the last several years, you think has the hardware to make
• 43:16
  the car drive itself? We have been working on the
• 43:19
  software to add higher and higher levels of autonomy.
• 43:24
  This time around last year, we are roughly 2,000 cars driving.
• 43:28
  Our fsdb DOT software. Since then, we have
• 43:32
  significantly improved the soft press robustness and capability
• 43:35
  that we have. Now shifted to 160,000 customers
• 43:38
  as of today, This is not come for free.
• 43:49
  It came from the sweat and blood of the engineering team for the
• 43:52
  last one year. For example, we train 75,000
• 43:57
  neural network models. Just last one year, that's
• 44:00
  roughly model, every eight minutes that's coming out of the
• 44:04
  team and then we evaluate them on a large clusters and then we
• 44:07
  ship 280. One of those models that
• 44:10
  actually improve the performance of the car.
• 44:12
  And this pace of innovation is happening throughout the stack.
• 44:16
  The the planning software, the infrastructure of the tools even
• 44:19
  hiring everything is progressing to the next level.
• 44:25
  The fsdb, the software is quite capable of driving the car.
• 44:30
  You should be able to navigate from parking lot parking lot
• 44:32
  aling CDC driving stopping for traffic lights and stop signs
• 44:37
  negotiating with objects at intersections making turns and
• 44:40
  so on. All of this comes from the
• 44:46
  camera streams that go through neural networks that run on the
• 44:49
  car itself. It's not coming back to the
• 44:50
  server or anything. It runs on the car and produces
• 44:53
  all the outputs to form the world model around the car and
• 44:56
  the planning software drives the car based on that.
• 45:01
  Today, we'll go into a lot of the components that make up the
• 45:03
  system. The occupancy Network acts as
• 45:07
  the base geometry layer of the system.
• 45:10
  This is a multi-camera video. You don't Network That, from the
• 45:15
  images predicts, the full physical occupancy of the world
• 45:19
  around the robot. So anything that's physically
• 45:22
  present trees walls, buildings course balls, what have you it
• 45:27
  predicts, which physically present it?
• 45:28
  Predicts them along, with their future motion?
• 45:34
  On top of this base level of geometry, we have more semantic
• 45:38
  layers in order to navigate the roadways, we need the lens of
• 45:41
  course. But then the roadways have lots
• 45:44
  of different lanes and they connect in all kinds of ways.
• 45:47
  So it's actually really difficult problem for typical
• 45:50
  computer vision techniques to predict the set of planes and
• 45:52
  the connectivities. So we reach all the way into
• 45:55
  language Technologies. And then pull the
• 45:57
  state-of-the-art from other domains are not just computer
• 46:00
  vision to make this task possible.
• 46:04
  For vehicles, we need that full kinematic state to control for
• 46:07
  them. All of this directly comes from
• 46:11
  neural networks. Video streams raw video streams.
• 46:14
  Come into the Network's. Go through a lot of processing
• 46:17
  and then outputs the full kinematic state that positions
• 46:20
  velocities acceleration. Jerk all of that directly comes
• 46:24
  out a networks with minimal post-processing.
• 46:26
  That's really fascinating to me. Because how is this even
• 46:29
  possible? What world do we live in that?
• 46:31
  This magic is possible that this network's predicts fourth,
• 46:34
  derivatives of these positions, when people thought we couldn't
• 46:37
  even detect these objects, My opinion is that is not come for
• 46:42
  free, it recommend tons of data. So we had to bit sophisticated
• 46:47
  Auto leveling systems that Shone through raw sensor data, run on
• 46:51
  ton of offline compute on the server's.
• 46:53
  It can take a few hours run expensive, neural networks.
• 46:57
  This is the information into labels that frame.
• 47:00
  Our in-car neural networks On top of this.
• 47:04
  We also use our simulation system to synthetically create
• 47:07
  images and since it's a simulation, it's trivially have
• 47:10
  all the labels. All of this goes through a
• 47:15
  well-oiled data engine pipeline where we first trained, a
• 47:19
  baseline model with some data, ship it to the car, see what the
• 47:23
  failures are. And once we know the failures,
• 47:26
  we mine the fleet for the cases where it fails provide the
• 47:29
  correct labels and add the data to the training.
• 47:32
  Set this process, systematically fixes the issues.
• 47:36
  And we do this for every task that runs in the car.
• 47:40
  Yeah, and to train these new, massive new our networks.
• 47:42
  This year, we expanded our training infrastructure by
• 47:45
  roughly 40 to 50% so that it starts at about 14,000 gpus
• 47:50
  today across multiple training clusters in the United States.
• 47:55
  We also worked on Rai competitor, which now supports
• 47:58
  new operations needed by those neural networks and map them to
• 48:01
  the best of our underlying Hardware resources.
• 48:05
  And our inference engine today is capable of Distributing the
• 48:09
  execution of a single neuron Network.
• 48:11
  Across two independent system on chips.
• 48:14
  Essentially, two independent, computers interconnected within
• 48:17
  the simple, self-driving computer, and to make this
• 48:20
  possible, we have to keep a tight control on the end-to-end
• 48:23
  latency of this new system. So we deployed more advanced
• 48:26
  scheduling code across the 40 50 platform.
• 48:31
  All of this neural networks running in the car to get that
• 48:34
  produced the vector space, which is again, the model of the world
• 48:36
  around the robot of the car, then the planning system
• 48:39
  operates on top of this coming up with trajectories that avoid
• 48:42
  collisions or smooth, make progress towards the
• 48:45
  destination. Using a combination of model
• 48:47
  based optimization plus neural network that helps optimize it
• 48:51
  to be really fast. Today we are very excited to
• 48:56
  present progress on all of these areas.
• 48:59
  We have the engineering leads standing by to come in and
• 49:01
  explain these various blocks and these power not just the car,
• 49:05
  but the same components also run on The Optimist robot that Milan
• 49:08
  showed earlier. With that, I welcome people to
• 49:11
  start talking about the planning section.
• 49:20
  Hi, all I a real giant. Let's use this intersection
• 49:25
  ciliary to dive straight into how we do the planning and
• 49:28
  decision-making in autopilot. So, we are approaching this
• 49:32
  intersection from a side street and we have to yield to all the
• 49:35
  crossing vehicles. Right.
• 49:37
  Miss, as we are about to enter the intersection.
• 49:40
  The Pedestrian, on the other side of the intersection decides
• 49:43
  to cross the road without a crosswalk.
• 49:45
  Now, we need to yield to this pedestrian, yield to the
• 49:48
  vehicles from the right? And also understand the relation
• 49:51
  between The Pedestrian and the vehicle on the other side of the
• 49:54
  intersection. So, a lot of these entry object
• 49:58
  dependencies that we need to resolve in a quick glance.
• 50:03
  And humans are really good at this.
• 50:05
  We look at a scene, understand all the possible interactions
• 50:09
  evaluate the most promising ones, and generally end up.
• 50:12
  Choosing a reasonable one. so, let's look at a few of these
• 50:16
  interactions that autopilot system evaluated, We could have
• 50:20
  gone in front of this pedestrian with a very aggressive launch
• 50:23
  General lateral profile. Now obviously we are being a
• 50:25
  jerk to The Pedestrian and you would spook The Pedestrian and
• 50:28
  is cute but We could have moved forward slowly short for a gap
• 50:33
  between The Pedestrian, or L, the vehicle from the right.
• 50:36
  Again, we are being a jerk to the vehicle coming from the
• 50:38
  right, but you should not outright reject.
• 50:41
  This interaction in case this is only safe interaction available.
• 50:46
  Lastly, the interaction we ended up choosing stay slow initially,
• 50:51
  find reasonable Gap and then finish them anywhere.
• 50:53
  After all the agents past. Evaluation of all of these
• 50:58
  interactions is not trivial, especially when you care about
• 51:02
  modeling the higher order, derivatives for other agents.
• 51:06
  For example, what is The Logical jerk required by the vehicle
• 51:09
  coming from the right? When you assert in front of it,
• 51:13
  Relying purely on collision. Checks with modular predictions
• 51:16
  will only get you so far because you will miss out on a lot of
• 51:19
  valid interactions. This basically boils down to
• 51:22
  solving the multi-agent, joint trajectory planning problem over
• 51:26
  the trajectories of ego and all the other agents.
• 51:30
  Now how much ever you optimize? There's going to be a limit to
• 51:32
  how fast you can run this optimization problem, it will be
• 51:35
  close to close to order of 10 milliseconds even after a lot of
• 51:38
  incremental consummation Now, for a typical crowded
• 51:43
  unprotected, left say you have more than 20 objects, each
• 51:47
  object, having multiple different future modes, the
• 51:50
  number of relevant interaction combinations will blow up.
• 51:56
  Will the planner needs to make a decision, every 50 milliseconds.
• 51:59
  So how do we solve this in real time?
• 52:03
  we rely on a framework, what we call as interaction search,
• 52:05
  which is basically a paralyzed research over a bunch of
• 52:08
  maneuver trajectories The state space here corresponds to the
• 52:13
  kinematic state of ego, the kinematic state of other agents,
• 52:16
  the nominal future multiple multi model predictions and all
• 52:20
  the static entities in the scene.
• 52:24
  The action space is is where things get interesting.
• 52:28
  The user set of maneuver trajectory candidates to Branch
• 52:31
  over a bunch of interaction decisions and also incremental
• 52:35
  goals for a longer Horizon maneuver.
• 52:38
  Let's Walk Through This research very quickly, to get a sense of
• 52:41
  how it works. We start with a set of vision
• 52:44
  measurements namely Lanes occupancy, moving objects.
• 52:48
  These get represented as posture actions as well as latent
• 52:51
  features. We use this to create a set of
• 52:54
  gold candidates Lanes. Again, from the latest at work
• 52:58
  or unstructured regions which correspond to a probability
• 53:00
  mask, the right from Human demonstration.
• 53:05
  Once we have a bunch of these gold candidates, we create see
• 53:07
  trajectories using a combination of classical optimization
• 53:10
  approaches as well as our Network planner again, trained
• 53:13
  on data from the customer Fleet. Now, once we get a bunch of
• 53:17
  these three trajectories we use them to start, branching on the
• 53:21
  interactions. We find the most critical
• 53:24
  interaction in our case. This would be the interaction
• 53:27
  with respect to The Pedestrian, whether we are certain front of
• 53:30
  it or into it. Obviously, the option on the
• 53:32
  left is a high penalty option. It likely won't get prioritized.
• 53:37
  So, we Branch further onto the option on the right, and that's
• 53:40
  where we bring in more and more complex interactions building.
• 53:43
  This optimization problem incrementally, with more and
• 53:45
  more. Ants.
• 53:47
  And that research keeps flowing, dancing on more interactions,
• 53:50
  branching and more goals. Now, lot of tricks, here lie, in
• 53:54
  evaluation of each each of this node of the tree search.
• 53:59
  Inside each node initially, we started with creating
• 54:02
  trajectories using classical optimization approaches where
• 54:05
  the constraints, like I described would be added
• 54:07
  incrementally. And this would take close to 125
• 54:10
  milliseconds per action. Now even though this is fairly
• 54:14
  good. Number when you want to evaluate
• 54:16
  more than 100 percent fractions, this does not scale.
• 54:20
  So we ended up building light weight variable networks that
• 54:23
  you can run in the loop of the planner.
• 54:26
  These networks are trained on human demonstrations from the
• 54:28
  fleet as well as offline solvers with relaxed time limits.
• 54:34
  But this we were able to bring the rundown runtime down to
• 54:37
  close to 100 microseconds per action.
• 54:46
  Now, doing this alone is not enough because you still have
• 54:49
  this massive research that you need to go through and you need
• 54:52
  to efficiently through the search space.
• 54:55
  So you need to do a new scoring on each of these.
• 54:58
  Trajectories few of these are fairly standard.
• 55:00
  You do a bunch of collision checks, you do a bunch of
• 55:02
  comfort analysis, what is the jerk and accept required for a
• 55:05
  given maneuver. The customer Fleet data plays an
• 55:09
  important role here again. We run two sets of again, light
• 55:13
  weight variable networks both really augmenting each other.
• 55:17
  One of them train from interventions from the fsdb, the
• 55:19
  fleet which gives a score on How likely is a given manure to
• 55:23
  result in interventions over the next few seconds and second
• 55:27
  which is purely on human demonstrations.
• 55:29
  Human driven data, giving a score on how close is your given
• 55:32
  selected action to a human driven trajectory.
• 55:36
  This coating helps us to the search space t, branching
• 55:40
  further, all the interactions and focus the compute on the
• 55:42
  most promising outcomes. The cool part about this
• 55:48
  architecture is that it allows us to create a cool blend
• 55:51
  between our data driven approaches.
• 55:54
  There you don't have to rely on a lot of hand engineered costs
• 55:57
  but also grounded in reality with physics-based checks.
• 56:02
  Now a lot of what I described was with respect to the agents,
• 56:05
  we could observe in the scene but the same framework extends
• 56:08
  to objects behind occlusions We use the video feed from eight
• 56:14
  cameras to generate the 3D occupancy of the world.
• 56:18
  The blue mask here, corresponds to the visibility region, we
• 56:21
  call it. It basically gets blocked at the
• 56:25
  first occlusion. You see in the sea, we consume
• 56:27
  this visibility mask to generate what we call as ghost objects
• 56:30
  which you can see on the top left.
• 56:33
  Now, if you model this Pawn regionals and the state
• 56:35
  transitions of this ghost object correctly, If you tune your
• 56:40
  control response as a function of their existence likelihood,
• 56:43
  you can extract some really nice human like behaviors.
• 56:47
  Now, I'll pass it on to fill to describe more on how we generate
• 56:50
  is occupants in it works. Thank you.
• 56:59
  Hey guys, my name is Phil. I will share the details of the
• 57:02
  occupancy Network would be able to over the past year.
• 57:06
  This network is our solution to model, the physical world in 3D
• 57:09
  around our cars. And it is currently not shown in
• 57:13
  our customer facing visualization.
• 57:15
  And what you'll see here is the road Network output from our
• 57:19
  internal Dev tool, The octopus is that work takes video streams
• 57:25
  of all our other cameras input, produces a single unified,
• 57:30
  volumetric occupancy in Vector, space directly for every 3D
• 57:35
  location around all the car, it predicts the probability of that
• 57:39
  location being occupied or not, since it has video context, it
• 57:44
  is capable of predicting obstacles that are occluded
• 57:47
  instantaneously. For each location.
• 57:51
  It also produces a state of semantics such as curb car
• 57:56
  possession and road debris. As color-coded here.
• 58:04
  Occupancy flow is also predict for motion since the model is a
• 58:08
  generalized Network. It does not tell static and
• 58:11
  dynamic objects explicitly. It is able to produce and model
• 58:16
  the random motion such as a swerving trainer here.
• 58:21
  This network is currently running or Tesla's with FSD
• 58:25
  computers and it is incredibly efficient runs about every 10
• 58:29
  milliseconds with our new linear accelerator.
• 58:33
  So, how does this work? Let's take a look at the
• 58:35
  architecture. First, we Rectify each camera
• 58:39
  images with the camera calibration, and the images were
• 58:42
  shown here were given to the network is actually not a
• 58:45
  typical a bit RGB image. As you can see from the first
• 58:49
  image on top. We're giving the 12 Pedro photo
• 58:52
  count, imagery to the network. Since it has four bits, more
• 58:56
  information, it has 16 times better than a branch as well as
• 59:01
  reduced latency since we don't have to run ISP in the loop
• 59:04
  anymore. We use a set of red lights and
• 59:07
  by of FPS, as a backbone to extract images of space features
• 59:13
  next, we construct a set of 3D position query along with the
• 59:17
  empty space features as keys and values fit into a attention.
• 59:21
  Module, the output of the attention module is high
• 59:25
  dimensional space. Your features This special
• 59:28
  features are online temporarily using vehicle odometry to derive
• 59:34
  motion. Not this spatial temporal
• 59:39
  features goes through a set of deconvolution to produce the
• 59:42
  final occupancy, and occupancy, flow output.
• 59:45
  They're formed as fixed size voxel grid, which might not be
• 59:48
  precise enough for presenting on control in order to get a higher
• 59:53
  resolution. We also produced per voxel
• 59:55
  feature Maps, which will feed into MLP, with 3D spatial Point
• 1:00:00
  queries to get position and semantics at any arbitrary
• 1:00:04
  location. After knowing the model better,
• 1:00:09
  let's take a look at another example here.
• 1:00:11
  We have an articulated bus parked on right side row
• 1:00:14
  highlighted as an l shaped box up here.
• 1:00:17
  As we approach the bus start to move the blue, the front of the
• 1:00:21
  cart turns blue first indicating, the model predicts.
• 1:00:25
  The front of us has a nonzero occupancy flow.
• 1:00:30
  And as the bus keeps moving the entire bus turns blue and you
• 1:00:34
  can also see that the network predicts, the precise curvature
• 1:00:38
  of the bus. Well, this is a very complicated
• 1:00:42
  problem for traditional object detection Network.
• 1:00:45
  As you have, you see, whether I'm going to use one Cube or
• 1:00:48
  perhaps ask you to feed it the curvature, but for occupancy
• 1:00:51
  Network, since all we care about is the occupancy in the visible
• 1:00:55
  space and we will be able to model the curvature precisely.
• 1:01:01
  Besides the voxel grid, the occupancy Network, also produces
• 1:01:05
  a driver surface. The driver surface has both 3D
• 1:01:08
  geometry and semantics. They are very useful for
• 1:01:11
  control, especially on Healy and the curvy roads.
• 1:01:15
  The surface and the box of gray are not predicted independently.
• 1:01:19
  Instead the voxel grid actually aligns with the surface
• 1:01:23
  implicitly. Here, we are at a here Quest
• 1:01:28
  where you can see the 3D geometry of the surface bring
• 1:01:32
  predicted nicely. Printer can usually this
• 1:01:35
  information to decide the, perhaps we need to slow down low
• 1:01:38
  for the Hillcrest. And as you can also see the
• 1:01:41
  voxel, great alliance with the surface consistently, Besides
• 1:01:47
  the Box source and the surface were so very excited about the
• 1:01:50
  recent breakthrough, Enduro, Radiance field, or loaf.
• 1:01:55
  We're looking into both incorporate some of a lie
• 1:01:58
  smeller features into accessing Network training as well as
• 1:02:01
  using our Network output as the input state for Nerf.
• 1:02:07
  As a matter of fact, Ashok is very excited about this.
• 1:02:09
  This has been his personal weekend project for a while.
• 1:02:17
  This nurse because I think Academia is building out of
• 1:02:21
  these Foundation models for language using like tons of
• 1:02:24
  large data sets for language. Anything for vision, nerves are
• 1:02:28
  going to provide the foundation models for computer vision
• 1:02:31
  because they are grounded in geometry and geometry gives us a
• 1:02:35
  nice way to supervise this networks and freezes of the
• 1:02:37
  requirement Define. And on top when you need Auto
• 1:02:44
  Parts, O Reilly Auto.com is a few clicks away.
• 1:02:47
  We offer convenient options for you to get your parts quickly
• 1:02:50
  order online and pick up for free.
• 1:02:52
  At your local, O'Reilly Auto Parts store.
• 1:02:54
  Will even bring it out curbside or you can have your parts
• 1:02:57
  delivered right to your door with free shipping on most
• 1:02:59
  orders over. $35 visit O Reilly Auto.com and the supervision is
• 1:03:11
  essentially free because you just have to differential be
• 1:03:13
  rendered these images. So I think in the future this
• 1:03:17
  all coincidence work idea of our images come in and then the
• 1:03:20
  network produces consistent, volumetric representation of the
• 1:03:25
  scene that can then be differentiable rendered into any
• 1:03:28
  image. That was observed.
• 1:03:30
  I personally think is the future of computer vision and we do, we
• 1:03:34
  do some initial work on it right now.
• 1:03:36
  But I think in the future both Tesla and in the Academia we
• 1:03:40
  will see that these combination of One-Shot prediction of Full
• 1:03:45
  automatic occupancy will be the, that's my personal bet.
• 1:03:50
  Censorship. So here's an example, Ernie
• 1:03:53
  result of a 3D Reconstruction from our Fleet data.
• 1:03:57
  Instead of focusing on getting perfect RGB reprojection in
• 1:04:01
  image space, our primary goal here is to accurately represent
• 1:04:05
  the wording study space for driving and we want to do this
• 1:04:08
  for all our Fleet data over the world in all weather and
• 1:04:11
  lighting conditions. And obviously this is a very
• 1:04:14
  challenging problem and we're looking for you guys to help.
• 1:04:18
  Finally, the occupancy network is trained with large out
• 1:04:21
  labeled data set without any human in the loop.
• 1:04:25
  And with that, I'll pass to him to talk about what it takes to
• 1:04:28
  train this network. Thanks though.
• 1:04:36
  All right, everyone. Let's talk about some training
• 1:04:40
  infrastructure. So we've seen a couple of videos
• 1:04:43
  know, four or five, I think and Cara Moore and worry more about
• 1:04:48
  a lot more Clips on that. So we've been looking at the
• 1:04:52
  occupancy networks, just from Phil, just fills videos.
• 1:04:56
  It takes one point four billion frames the train that Network,
• 1:05:00
  what? You just saw.
• 1:05:01
  And if you have 100,000 gpus, it would take one hour.
• 1:05:05
  But if you have one GPU it, Would take 100,000 hours.
• 1:05:10
  So that is not a Humane time period that you can wait for
• 1:05:13
  your training job to run, right? We want to ship faster than
• 1:05:15
  that. So that means you're going to
• 1:05:17
  need to go parallel. So you need more compute for
• 1:05:20
  that. That means you're going to need
• 1:05:22
  a supercomputer. So this is why we've built
• 1:05:25
  in-house. Three supercomputers comprising
• 1:05:27
  of 14,000 gpus, where we use 10,000 gpus for training and run
• 1:05:32
  4000 gpus for auto leveling. All these videos are stored in
• 1:05:38
  30 Theta. B of A distributed managed video
• 1:05:41
  cash. You shouldn't think of our data
• 1:05:44
  sets as fixed. Let's say, as you think of your
• 1:05:47
  image net or something, you know with like a million frames, you
• 1:05:50
  should think of it as a very fluid thing.
• 1:05:52
  So we've got a half a million of these videos flowing in and out
• 1:05:56
  of this cluster, these clusters every single day.
• 1:06:00
  And we track 400,000 of these kind of python video
• 1:06:04
  instantiations every And so that is that's a lot of calls.
• 1:06:09
  We are going to need a capture that in order to govern the
• 1:06:11
  retention policies of this distributed video cash.
• 1:06:15
  So, underlying, all of this is a huge amount of infra all of
• 1:06:18
  which we build and manage in house.
• 1:06:22
  So you cannot just by, you know, 14,000 views, and then a 30
• 1:06:26
  petabytes of Flash nvme and just put it together.
• 1:06:29
  And let's go train. It actually takes a lot of work
• 1:06:32
  and I'm going to go into a little bit of that.
• 1:06:35
  What you actually typically want to do is you want to take your
• 1:06:37
  accelerator so that could be the GPU or Dojo which we'll talk
• 1:06:41
  about later and because that's the most expensive component,
• 1:06:47
  that's where you want to put your bottleneck and so that
• 1:06:49
  means that every single part of your system is going to need to
• 1:06:53
  outperform this accelerator. And so that is really
• 1:06:56
  complicated. That means that your storage is
• 1:06:59
  going to need to have the size and the bandwidth to deliver all
• 1:07:02
  the data down into the nodes. These nodes need to The right
• 1:07:05
  amount of CPU and memory capabilities to feed into and
• 1:07:09
  your machine learning framework. This machine learning framework,
• 1:07:12
  then is to hand it off to your GPU and then you can start
• 1:07:15
  training, but then you need to do so across hundreds or
• 1:07:18
  thousands of GPU in a reliable way in lockstep, and in a way
• 1:07:23
  that's also fast. So you're also going to need
• 1:07:25
  interconnect, extremely complicated, we'll talk more of
• 1:07:28
  a dojo and second So first, I want to take you to some
• 1:07:33
  optimizations that we've done on our cluster so we're getting in
• 1:07:37
  a lot of videos and video is very much unlike let's say
• 1:07:41
  training on images or text, which I think is very well
• 1:07:44
  established. Video is quite literally a
• 1:07:46
  dimension, more complicated. And so that's why we needed to
• 1:07:51
  go end to end from the storage layer, down to the accelerating,
• 1:07:55
  optimize every single piece of that because we train on the
• 1:07:58
  photon count videos that come directly from our Fleet, we
• 1:08:02
  trained on those directly. We do not post processors those
• 1:08:05
  at all the way it's just done. Is we seek exactly to the
• 1:08:09
  frames? We select for our batch?
• 1:08:11
  We load those in including the frames that they depend on.
• 1:08:14
  So these are your iPhones or a key frames.
• 1:08:16
  We package those up movement, a shared memory, move them into a
• 1:08:19
  double bar from the GPU and then use Hardware.
• 1:08:22
  Decoder, that's only accelerated to actually decode the video.
• 1:08:27
  So, we do that on the GPU natively and it's all in a very
• 1:08:30
  nice python. Torch extension doing so unlock
• 1:08:34
  more than 30% training speed, increase for the occupancy,
• 1:08:37
  networks and Freda. Basically whole CPU to do any
• 1:08:41
  other thing. You cannot just do training with
• 1:08:46
  just videos. Of course, you need some kind of
• 1:08:47
  a ground truth and that is actually interesting problem as
• 1:08:51
  well. The objective for storing your
• 1:08:53
  ground. Truth is that you want to make
• 1:08:56
  sure you get to your ground truth that you need in the
• 1:08:58
  minimal amount of file system, operations and load in the
• 1:09:01
  minimal size of what you need in order to optimize for aggregate
• 1:09:05
  cross cluster, throughput. Because you should see a
• 1:09:08
  computer cluster as one big device, which has internally
• 1:09:12
  fixed constraints. And Households.
• 1:09:14
  So for this we rolled out a format that is native to us.
• 1:09:19
  That's called small. We use this for our ground
• 1:09:21
  truth, our feature cash, and any inference outputs.
• 1:09:24
  So a lot of dancers that are in there and so just the cartoon
• 1:09:27
  here, let's say, these are your, is your table that you want to
• 1:09:30
  store. Then that's how that would look
• 1:09:32
  out if you rolled out on disk. So, what you do is, you take
• 1:09:35
  anything you'd want to index on. So, for example, video
• 1:09:37
  timestamps, you put those all in the header.
• 1:09:40
  So that in your initial header, eat, you know, exactly where to
• 1:09:43
  go on this. Then, if you have any tensors,
• 1:09:46
  you're going to try to transpose the dimensions to put a
• 1:09:49
  different dimension last as he contiguous Dimension.
• 1:09:52
  And then also try different types of compression.
• 1:09:55
  Then you check out which one was most optimal and then store that
• 1:09:58
  one, this is actually a huge step.
• 1:10:00
  If you do feature caching unintelligible output from the
• 1:10:03
  machine Learning Network, rotate around the dimensions, a little
• 1:10:06
  bit, you can get up to 20 percent increase in efficiency
• 1:10:09
  of storage. Then when you store that we also
• 1:10:15
  ordered columns by size so that all your small columns and small
• 1:10:18
  values are together. So that when you see for a
• 1:10:21
  single value, you're likely to overlap with a read on more
• 1:10:24
  values, which we'll use later so that you don't need to do
• 1:10:27
  another file system, operation. So, I could go on and on, I just
• 1:10:32
  went on on touch, on two projects that we have
• 1:10:35
  internally, but this is actually part of a huge, continuous
• 1:10:38
  effort to optimize the compute that we have in-house.
• 1:10:42
  So accumulating in aggregating through all these optimizations,
• 1:10:45
  we now trainer occupancy networks, tries as fast just
• 1:10:48
  because it's twice as efficient. And now, if we add in bunch more
• 1:10:52
  compute and go parallel, we can now train this hours instead of
• 1:10:55
  days. And with that, I'd like to hand
• 1:10:58
  it off to the biggest user of compute John.
• 1:11:08
  Hi everybody. My name is John Emmons.
• 1:11:10
  I lead the autopilot, Vision team.
• 1:11:13
  I'm going to cover two topics with you today.
• 1:11:15
  The first is how we predict lanes, and the second is how we
• 1:11:18
  predict the future behavior of other agents on the road.
• 1:11:22
  In the early days of autopilot, we modeled the lane detection
• 1:11:25
  problem is an image space instance, segmentation task.
• 1:11:29
  Our network was super simple though.
• 1:11:31
  In fact, it was only capable of producing Lanes from a of a few
• 1:11:34
  different kinds of geometries specifically, it would segment
• 1:11:38
  the ego Lane, it could segment adjacent lanes, and then it has
• 1:11:41
  some special casing for forks and merges this simplistic
• 1:11:44
  modeling of the problem worked for highly structured roads.
• 1:11:46
  Like highways, But today we're trying to build a system that's
• 1:11:51
  capable of much more complex Maneuvers.
• 1:11:53
  Specifically we want to make left and right turns at
• 1:11:55
  intersections where the road topology can be quite a bit more
• 1:11:57
  complex and diverse. When we try to apply this
• 1:12:00
  simplistic modeling of the problem here, it just totally
• 1:12:03
  breaks down. Taking a step back for a moment.
• 1:12:07
  What we're trying to do here is to predict the sparse set of
• 1:12:09
  lame instances and their connectivity.
• 1:12:12
  And what we want to do is to have a neural network that
• 1:12:14
  basically predicts, this graph where the nodes are the lane
• 1:12:17
  segments in the edges and code the connectivity between these
• 1:12:20
  Lanes. So we have is our lane
• 1:12:24
  detection, neural network. It's made up of three
• 1:12:27
  components. In the first component, we have
• 1:12:30
  a set of convolutional layers attention layers, in other,
• 1:12:32
  neural network, layers that encode the video streams from
• 1:12:35
  our 8 cameras on the vehicle, in produce a rich visual
• 1:12:38
  representation. We then enhance this digital
• 1:12:42
  representation with a course roadmap level Road level map
• 1:12:47
  data, which we encode with a set of additional neural network
• 1:12:50
  layers that we call the lane guidance module.
• 1:12:53
  This map is not an HD map but it provides a lot of useful hints
• 1:12:56
  about the topology of lanes. Instead of intersections the
• 1:12:58
  lane counts and various roads in a set of other attributes that
• 1:13:01
  help us. The first two components here,
• 1:13:06
  produced a dense tensor, that sort of encodes the world.
• 1:13:09
  But what we really want to do is to convert this dense tensor
• 1:13:12
  into a smart set of lanes in their connectivities.
• 1:13:15
  We approach this problem like an image captioning task.
• 1:13:19
  Where the input is this dense tensor in the output text is
• 1:13:22
  predicted into a special language that we developed at
• 1:13:24
  Tesla for encoding Lanes in their connectivities.
• 1:13:27
  In this language of lanes, the words and tokens are the lane
• 1:13:30
  positions in 3D space in the ordering of the tokens
• 1:13:34
  introverted modifiers in the tokens encode, the connective
• 1:13:36
  relationships between these Lanes By modeling.
• 1:13:39
  The task is language problem. We can capitalize on recent
• 1:13:42
  autoregressive, architectures and techniques, from the
• 1:13:45
  language Community for handling the multiple daily of the
• 1:13:47
  problem. We're not just solving the
• 1:13:49
  computer vision problem at autopilot, we're also applying
• 1:13:52
  the state of the art and language modeling, the machine
• 1:13:53
  learning more generally I'm not going to dive into a little bit
• 1:13:57
  more detail, just language component.
• 1:14:01
  What I have depicted on the screen here is the satellite
• 1:14:03
  image which sort of represents the local area.
• 1:14:05
  Around the vehicle, the set of nodes and edges is what we refer
• 1:14:09
  to as the landgraf and it's ultimately what we want to come
• 1:14:11
  out of this neural network. We start with a blank slate.
• 1:14:17
  We're going to make our first prediction here at this Green
• 1:14:19
  Dot. This green dots position is
• 1:14:22
  encoded as an index into a coarse grid which discretize has
• 1:14:25
  the 3D World. Now, we don't predict this index
• 1:14:28
  directly because it would be too computationally expensive to do.
• 1:14:30
  So there's just too many grid points and printing a
• 1:14:33
  categorical distribution over. This has both implications that
• 1:14:36
  training time and test time. So instead what we do is we just
• 1:14:39
  recharge the world coarsely first.
• 1:14:41
  We predict a heat map over the possible locations and then we
• 1:14:44
  latch in the most probable location condition on this.
• 1:14:48
  We then refine the prediction and get the precise point.
• 1:14:53
  Now we know where the position of this token is, we don't know.
• 1:14:55
  Its type. In this case though, it's a
• 1:14:57
  beginning of a new Lane. So we refer to it as a start
• 1:15:00
  token. And because it's a star token,
• 1:15:03
  there's no additional attributes in our language.
• 1:15:07
  We then take the predictions from this first forward pass and
• 1:15:09
  we encode them using a learn traditional and wedding which
• 1:15:12
  produces a set of tensors that we combine together.
• 1:15:16
  Which is actually the first word in our language of lanes.
• 1:15:18
  We had this to the first position our sentence here.
• 1:15:22
  We then continue this process by preaching.
• 1:15:24
  The next link point in a similar fashion.
• 1:15:28
  Now this Lane point is not the beginning of a new Lane.
• 1:15:30
  It's actually a continuation of the previous Lane so it's a
• 1:15:34
  continuation token type. Now it's not enough just to know
• 1:15:38
  that this Lane is connected to the previously prepared to play
• 1:15:40
  in. We want to encode.
• 1:15:41
  It's precise geometry which we do by regressing, a set of
• 1:15:44
  spline coefficients. We then take this Lane, we
• 1:15:49
  encode it again and added as the next word in the sentence.
• 1:15:53
  We continue protecting these continuation, Lanes, until we
• 1:15:55
  get to the end of the prediction grid.
• 1:15:58
  We then move on to a different Lane segment, so you can see
• 1:16:01
  that signed up there. Now it's not top allegedly
• 1:16:03
  connected to that pink point. It's actually forking off of
• 1:16:06
  them that blue. Sorry that Greenpoint there.
• 1:16:10
  So it's got a fork type and Fork tokens actually point back to
• 1:16:15
  previous tokens from which their Fork originates.
• 1:16:19
  So you can see here, the fourth Point predictor is actually the
• 1:16:21
  index zero. So it's actually referencing
• 1:16:23
  back to tokens that has already created like you would in
• 1:16:25
  language. We continue this process over
• 1:16:29
  and over again. And so we've enumerated all of
• 1:16:30
  the tokens in the lane graph and in the network predicts, the end
• 1:16:34
  of sentence token. Yeah, I just wanted to note
• 1:16:37
  that. The reason we do this is not
• 1:16:40
  just because we want to build something complicated.
• 1:16:42
  It's almost feels like a turing-complete machine here.
• 1:16:44
  With neural networks though is that we tried simple approaches,
• 1:16:47
  for example, trying to just segment, the lanes along the
• 1:16:50
  road or something like that. But then the problem is, when
• 1:16:53
  this uncertainty say, you cannot see the road clearly and there
• 1:16:56
  could be two lanes or three lanes, and you can tell a simple
• 1:16:59
  segmentation based approach would just draw.
• 1:17:01
  Both of them is kind of a 2.5 Lane situation and the post
• 1:17:05
  processing, algorithm would collide History fail when the
• 1:17:07
  predictions are such again, the problems don't end there.
• 1:17:11
  I mean, you need to predict these connective conditions like
• 1:17:13
  these connective Lanes inside of intersections, which it's just
• 1:17:16
  not possible with the approach that ashok's mentioning, which
• 1:17:18
  is why we had to upgrade to this sort of event like or laps like
• 1:17:20
  this segmentation would just go Haywire but even if you try very
• 1:17:23
  hard to you know put them on separate layers it's just really
• 1:17:26
  hard problem. But languages offers a really
• 1:17:28
  nice framework for more getting a sample from the posterior, as
• 1:17:33
  opposed to, you know, trying to do all of this in post
• 1:17:35
  processing. I didn't exactly stop for this
• 1:17:38
  autopilot, right? John.
• 1:17:39
  This can be used for Optimus. Again, you know, I guess they
• 1:17:42
  wouldn't be called Lanes, but you can imagine, you know, sort
• 1:17:44
  of in this, you know, stage here that you might have sort of
• 1:17:47
  paths, that sort of, you know, encode the possible places that
• 1:17:50
  people could walk. Yeah.
• 1:17:52
  It's basically, if you're in a factory or a home setting, you
• 1:17:56
  can just ask the robot. Okay.
• 1:17:57
  Let's me, please route to the kitchen or please route to some
• 1:18:01
  location, The Factory, and then we predict the set of Pathways
• 1:18:04
  that would go through the aisles, like, the robot and say,
• 1:18:06
  okay, this is how you get to the kitchen.
• 1:18:08
  It just really gives us a nice framework to model these
• 1:18:10
  different paths. Let's simplify the navigation,
• 1:18:13
  problem for the downstream planner.
• 1:18:18
  All right, so ultimately what we get from this Lane detection
• 1:18:21
  network is a set of lanes in their connectivities which comes
• 1:18:24
  directly from the network. There's no additional step here
• 1:18:26
  for specifying these, you know, dense predictions into into
• 1:18:30
  dispersed ones. This is just the direct
• 1:18:31
  unfiltered output of the network.
• 1:18:36
  Okay, so I talked a little bit about Lanes, I'm going to
• 1:18:38
  briefly touch on how we model and predict the future paths in
• 1:18:42
  other semantics on objects. So I'm going to go really
• 1:18:45
  quickly through two examples. The video on the right here,
• 1:18:48
  we've got a car. That's actually running a red
• 1:18:50
  light and turning in front of us.
• 1:18:52
  What we do to handle situations. Like this is we predict a set of
• 1:18:55
  short time Horizon future trajectories on all objects.
• 1:18:58
  We can use these to anticipate the dangerous situation here and
• 1:19:02
  apply, whatever braking and steering actions required to
• 1:19:04
  avoid a collision. In the video on the right,
• 1:19:07
  there's two vehicles in front of us.
• 1:19:09
  The one of the left lane is parked apparently, it's being
• 1:19:12
  loaded unloaded, I don't know why the driver decided to park
• 1:19:14
  there, but the important thing is that our neural network
• 1:19:17
  predicted it, that it was stopped, which is the red color
• 1:19:20
  there. The vehicle in the other lane
• 1:19:22
  is, you notice also the stationary, but that one's
• 1:19:24
  obviously just waiting for that red light to turn green.
• 1:19:26
  So even though both objects are stationary and have zero
• 1:19:28
  velocity, it's the semantics that is really important here so
• 1:19:32
  that we don't get stuck behind that, awkwardly parked car.
• 1:19:37
  Predicting, all of these agent attributes, present, some
• 1:19:39
  practical problems, when trying to build a real-time system, we
• 1:19:42
  need to maximize the framerate of her optic section stack.
• 1:19:45
  So that autopilot can quickly react to the changing
• 1:19:47
  environment. Every millisecond, really
• 1:19:49
  matters here to minimize the inference latency.
• 1:19:52
  Our neural network is split into two phases.
• 1:19:55
  In the first phase we identified locations in 3D space where
• 1:19:59
  agents exist in the second stage.
• 1:20:01
  We then pull out tensors at those three locations.
• 1:20:04
  A pendant with additional data that's on the vehicle.
• 1:20:07
  And then we do the rest of the processing.
• 1:20:10
  This first vacation step allows the neural number to focus
• 1:20:12
  compute on the areas. That matter most, which gives us
• 1:20:15
  Superior performance for a fraction of the latency cost.
• 1:20:19
  So putting it all together, the autopilot business that creates
• 1:20:22
  more than just the geometry and kinematics the world.
• 1:20:24
  It also predicts a rich set of semantics which enable safe in
• 1:20:27
  human like driving. I'm not going to hand things off
• 1:20:30
  to streak. Will tell us how we run.
• 1:20:31
  All these cool neural networks on our FSD computer.
• 1:20:33
  Thank you. Hi everyone, I'm free today I'm
• 1:20:45
  going to get glimpse of what it takes to run this efficient
• 1:20:47
  networks in the car and how do we optimize for the inference
• 1:20:50
  latency? Today, I'm going to focus just
• 1:20:53
  on FS Elaine's Network that John just talked about.
• 1:20:59
  So when you started this track, we wanted to know if we can run
• 1:21:03
  this FS Elaine stadtwerke natively on the trip engine,
• 1:21:06
  which is our in-house neural network accelerator that we
• 1:21:09
  built in the officially computer.
• 1:21:11
  When we build this Hardware, we kept it simple and we made sure
• 1:21:16
  it can do one thing. Ridiculously fast dense dot
• 1:21:19
  products, but this architecture is auto regressive, and I
• 1:21:24
  traitor where it crunches through multiple attention,
• 1:21:27
  attention blocks in the Inner Loop.
• 1:21:29
  Producing sparse points directly at every step.
• 1:21:32
  So it's the challenge here was, how can we do this sparse Point,
• 1:21:36
  prediction, and sparse computation on a dense dot
• 1:21:38
  product engine. Let's see how we did this on the
• 1:21:41
  trip. So the network predicts the heat
• 1:21:46
  map of most probable spatial locations of the point.
• 1:21:50
  Now, we do a ARG Max and a one heart operation, which gives the
• 1:21:55
  one hard encoding of the index of the spatial location.
• 1:21:59
  Now we need to select the embedding associated with this
• 1:22:02
  index from an embedding table that is learned during training
• 1:22:07
  to do this on trip. We actually built a lookup table
• 1:22:10
  in s RAM and we engineer the dimensions of this embedding
• 1:22:14
  such that we could achieve all of this thing with just matrix
• 1:22:18
  multiplication. Not just that, we also wanted to
• 1:22:23
  store this embedding into a token cash so that we don't re
• 1:22:26
  compute this for every iteration rather, we use it for future
• 1:22:29
  Point prediction. Again, we pull some tricks here
• 1:22:32
  where we did all these operations just on the dot
• 1:22:35
  product engine. It's actually cool that our team
• 1:22:39
  found creative ways to map all these operations on the trip
• 1:22:42
  engine in ways that were not even imagined when this Hardware
• 1:22:46
  was designed. But that's not the only thing we
• 1:22:50
  had to do to make this work. We actually implemented a whole
• 1:22:54
  lot of operations and features to make this model compilable to
• 1:22:59
  improve the in Tate accuracy as well as to optimize performance.
• 1:23:03
  All of these things helped us run the 75 million parameter
• 1:23:07
  model just under 10 milliseconds of latency consuming just 8
• 1:23:11
  watts of power. But this is not the only
• 1:23:16
  architecture running in the car. There are so many other
• 1:23:18
  architectures modules and networks in running the car to
• 1:23:23
  give a sense of scale. There are about a billion
• 1:23:26
  parameters of all the networks combined producing around
• 1:23:29
  thousand your network single signals.
• 1:23:32
  So we need to make sure we optimize them jointly and such
• 1:23:37
  that we maximize the computer glaciation throughput and
• 1:23:40
  minimize the latency. So we built a compiler, just for
• 1:23:46
  neural networks that shares the structure to traditional
• 1:23:49
  compilers. As you can see, it takes the
• 1:23:52
  massive graph of urine Nets with 150 K nodes until scientific a
• 1:23:57
  connection takes this thing, partitions them.
• 1:24:00
  Into independent sub graphs and come Compares.
• 1:24:03
  Each of the sub graphs natively for the inference devices.
• 1:24:07
  Then we have a neural network Linker which shares the
• 1:24:10
  structure to traditional Linker. When we perform this link time
• 1:24:14
  optimization there, we solve an offline optimization problem for
• 1:24:20
  with computer memory and memory bandwidth constraints, so that
• 1:24:24
  it comes with an optimized schedule that gets executed in
• 1:24:26
  the car. On the run time.
• 1:24:29
  We designed a hybrid scheduling system, which basically does
• 1:24:33
  heterogeneous scheduling on one SOC and distributed scheduling
• 1:24:37
  across both the soc's to run these networks in a model
• 1:24:40
  parallel fashion, to get 100 tops of computerization.
• 1:24:45
  We need to optimize across all the layers of software, right?
• 1:24:49
  From turning the network, architecture the compiler, all
• 1:24:52
  the way to implementing a low latency high bandwidth RDMA link
• 1:24:56
  across both the soc's, And in fact going even deeper to
• 1:25:00
  understanding and optimizing the cache coherent and noncoherent
• 1:25:04
  data Paths of the accelerator in the soc.
• 1:25:07
  This is a lot of organization at every level in order to make
• 1:25:10
  sure we get the highest frame rate and as every millisecond
• 1:25:14
  counts here, And this is, this is just the, this is the
• 1:25:21
  visualization of the neural networks running in the car.
• 1:25:24
  This is a digital brain. Essentially, as you can see,
• 1:25:27
  these operators are nothing but just the matrix multiplication,
• 1:25:30
  convolution to name a few real operations running in the car.
• 1:25:35
  To train and train this network with the billion parameters, you
• 1:25:38
  need a lot of label data. So again is going to talk about
• 1:25:42
  how do we achieve this with the auto leveling pipeline Thank
• 1:25:54
  you, sure. Hi everyone.
• 1:25:56
  I'm Yang yanzhao and I'm leaving a symmetric fission and
• 1:25:59
  autopilot. So yeah, let's talk about Auto
• 1:26:04
  labeling. So we have several kinds of all
• 1:26:08
  the labeling Frameworks to support various types of
• 1:26:11
  networks. But today I'd like to focus on
• 1:26:14
  the awesome Lanes not here. So to successfully trained and
• 1:26:19
  generalize this network to everywhere.
• 1:26:22
  We think we went tens of millions of troops from probably
• 1:26:26
  1 million, 1 million intersection or even more.
• 1:26:30
  So How to do that. So it is certainly achievable to
• 1:26:37
  Source sufficient amount of trips because we already have as
• 1:26:40
  team explained earlier, we already have like, five hundred
• 1:26:43
  thousand trips per day, cash rate, however, converting all
• 1:26:48
  those data into a training form is a very challenging technical
• 1:26:51
  problem. To solve this challenge, we've
• 1:26:56
  tried various ways of manual and auto labeling.
• 1:26:59
  So from the First Column to the second from the second to the
• 1:27:03
  third, each Advance provided as nearly hundred X Improvement in
• 1:27:07
  throughput, but still, we won an even better or labeling machine.
• 1:27:12
  That can provide providers providers, good quality,
• 1:27:16
  diversity, and scalability. To meet all these requirements.
• 1:27:23
  Despite the huge amount of engineering effort required
• 1:27:26
  here, we've developed a new auto labeling machine, powered by
• 1:27:30
  multitude of reconstruction. So these can replace five
• 1:27:35
  million hours of manual labeling with just 12 hours on cluster
• 1:27:39
  for labeling 10,000 troops. So how we solved, there are
• 1:27:44
  three big steps. The first step is high Precision
• 1:27:47
  trajectory and structure recovery by multi-camera fissure
• 1:27:50
  inertia odometry. So here all the features
• 1:27:54
  including ground, surface are inferred from videos by neural
• 1:27:56
  networks, then tracked and reconstructed in the vector
• 1:28:00
  space. So the typical drift weight of
• 1:28:04
  this trajectory in Coral is like 1.3 centimeters per meter and
• 1:28:08
  0.45 million per meter, which is pretty decent considering its
• 1:28:13
  compact compared requirement. Then the recovery surface and
• 1:28:17
  roll details are also use as a strong guidance for the later.
• 1:28:20
  Manual, verification stop. This is also enabled in every
• 1:28:25
  FSD. Beagle.
• 1:28:26
  So we get pre-process trajectories and structures.
• 1:28:29
  Along with the trip data The second step is 42 reconstruction
• 1:28:37
  which is the peak and core piece of this machine.
• 1:28:40
  So the video shows how the previously shown trip is
• 1:28:43
  reconstructed and aligned with other trips.
• 1:28:46
  Basically other trees from different people, not the same
• 1:28:49
  vehicle. So this is done by multiple
• 1:28:51
  internships. Like coarse alignment pairwise
• 1:28:54
  matching join optimization. Then furthers surface
• 1:28:57
  refinement, in the end. The human analyst comes in and
• 1:29:02
  finalizes the label. So each heavy stuffs are already
• 1:29:06
  fully paralyzed on the cluster. So the entire process usually
• 1:29:11
  takes just a couple of hours The last tab is actually older
• 1:29:17
  labeling, the new trips. So, here we use the same multi
• 1:29:22
  trip, alignment engine, but only between pre-built reconstruction
• 1:29:26
  and each new trip. So, it's much much simpler than
• 1:29:30
  fully reconstructing all the clips or together.
• 1:29:33
  That's why it only takes 30 minutes per trip to order label,
• 1:29:37
  instead of manual several hours of manual labeling, And this is
• 1:29:43
  also the key of scalability of this machine.
• 1:29:49
  This machine easily scales, as long as we have available
• 1:29:53
  compute and trip data. So about 50 trips were newly
• 1:29:58
  Auto labeled from this scene, and some of them are shown here.
• 1:30:01
  So 53, from different vehicles. So this is how we capture and
• 1:30:08
  transform the space time, slices of the world into the network
• 1:30:11
  supervision. Yeah.
• 1:30:13
  One thing I'd like to note is that again just talked about how
• 1:30:16
  we oughta label our lanes but we have Auto laborers for almost
• 1:30:20
  every task that we do including our planner and many of these
• 1:30:23
  are fully automatic no humans involved, for example, for
• 1:30:25
  objects or the kinematics their shapes, their Futures,
• 1:30:29
  everything just comes from Auto labeling and the same is true
• 1:30:32
  for occupancy to. And we have really just build a
• 1:30:34
  machine or Round this. Yeah, so if you can go back, one
• 1:30:37
  slide. One more it says parallelized on
• 1:30:42
  cluster so so that sounds pretty straightforward but it really
• 1:30:47
  wasn't. Maybe it's fun to share our
• 1:30:49
  something like this comes about. So a while ago we didn't have
• 1:30:53
  any auto labeling at all and then someone makes a script, it
• 1:30:57
  starts to work, it starts working better, until you reach
• 1:31:00
  a volume. That's pretty high, and we
• 1:31:01
  clearly need a solution. And so there were two other
• 1:31:05
  engineers in our team who are like, you know, that's an
• 1:31:07
  interesting, you know, What we needed to do, was build a whole
• 1:31:10
  graph of essentially python functions that we need to run
• 1:31:14
  one after the other first, you pull the clip.
• 1:31:16
  They do some cleaning, then you do some Network inference, and
• 1:31:19
  another Network influence until you finally get this.
• 1:31:22
  But so you need to do that as a large-scale.
• 1:31:24
  So I tell them, we probably need to shoot for, you know, 100,000
• 1:31:28
  Clips per day or like hundred thousand items that seems good.
• 1:31:32
  And so, the engineer said, well, we can do a bit of postgres and
• 1:31:37
  bit of elbow grease. We can do.
• 1:31:39
  It? Meanwhile, we are bit later and
• 1:31:41
  we're doing 20 million of these functions every single day.
• 1:31:46
  Again, we pull in around half a million clips and on those we
• 1:31:49
  run it done a functions each of these in a streaming fashion and
• 1:31:52
  so that's kind of the back end. Infrared is also needed to not
• 1:31:55
  just run training, but also auto-leveling.
• 1:31:57
  Yeah, it really is like a factory that produces labels and
• 1:32:01
  it's like production lines. Yield quality inventory, like
• 1:32:04
  all of the same Concepts applied to this label.
• 1:32:07
  Factory that applies for you know, the factory for cars.
• 1:32:10
  That's right. Okay.
• 1:32:14
  Thanks Tim Allen Show. So, yeah, so concluding, this
• 1:32:18
  section, I'd like to share a few more challenging and interesting
• 1:32:21
  examples for Network, for sure. And even for humans, probably.
• 1:32:26
  So, from the top, there's like example, for like lack of Lights
• 1:32:30
  case or foggy night, or roundabout and occlusions pipe.
• 1:32:34
  Have your questions by parked cars and even rainy night with
• 1:32:37
  the raindrops on camera lenses, these are challenging.
• 1:32:41
  But once their original scenes are Fully reconstructed by other
• 1:32:44
  Clips. They all of them can be Auto
• 1:32:46
  labeled so that our cars can drive even better through these
• 1:32:50
  challenging scenarios. So, now let me pass the mic to
• 1:32:54
  David to learn more about how cities creating the new world on
• 1:32:56
  top of these labels. Thank you.
• 1:33:05
  Thank you again. My name is David and I'm going
• 1:33:07
  to talk about simulation. So simulation plays a critical
• 1:33:11
  role in providing data that is difficult to source and or hard
• 1:33:15
  to label. However 3D scenes are
• 1:33:18
  notoriously slow to produce. Take for example, the simulated
• 1:33:22
  seen playing behind me. A complex intersection from
• 1:33:26
  Market Street in San Francisco. It would take two weeks for
• 1:33:30
  artists to complete. And for us, that is painfully
• 1:33:33
  slow. However, I'm going to talk about
• 1:33:36
  using the agins automated ground truth labels along with some
• 1:33:39
  brand-new. Schooling that allows us to
• 1:33:41
  procedurally. Generate this seen in many like
• 1:33:43
  it and just five minutes. That's an amazing 1,000 times
• 1:33:47
  faster than before. So let's dive in to I was seeing
• 1:33:51
  like this is created. We start by piping the automated
• 1:33:55
  ground truth labels into our simulated World creator.
• 1:33:58
  Tooling inside the software, Houdini starting with Road
• 1:34:02
  boundary labels, we can generate a solid Road mesh and reads.
• 1:34:05
  Apologize it with the landgraf labels, this helps inform
• 1:34:09
  important Road details, like Crossroads slope and detailed
• 1:34:12
  material blending. Next, we can use the line data
• 1:34:16
  and sweep geometry across its surface and project it to the
• 1:34:19
  road creating Lane paint decals. Next using media and edges we
• 1:34:27
  can spawned Island geometry and populate it with randomize
• 1:34:30
  foliage. This drastically changes the
• 1:34:32
  visibility of the scene. Now the outside world can be
• 1:34:35
  generated through a series of randomized.
• 1:34:38
  Heuristics modular building generators, create visual
• 1:34:41
  obstructions while randomly placed objects like hydrants can
• 1:34:44
  change the color of the curbs, while trees can treat drop
• 1:34:47
  leaves below it, obscuring lines, or edges.
• 1:34:51
  Next, we can bring in map data to inform positions of things
• 1:34:55
  like traffic traffic lights or stop signs.
• 1:34:57
  We can trace along. It's normal to collect important
• 1:35:00
  information like number of lanes and even get accurate street
• 1:35:03
  names on the signs themselves. Next using land graph.
• 1:35:07
  We can determine Lane connectivity and spawned
• 1:35:10
  directional Road markings on the road and their accompanying road
• 1:35:13
  signs. And finally, with landgraf
• 1:35:16
  itself we can determine Lane adjacency and other useful
• 1:35:20
  metrics the spawn randomize traffic permutations inside our
• 1:35:23
  simulator. And again this is all automatic
• 1:35:26
  no artists in Loop and happens within minutes.
• 1:35:29
  And now the sets us up to do some pretty cool things.
• 1:35:34
  Since everything is based on data and heuristics, we can
• 1:35:36
  start to fuzz parameters to create visual variations of the
• 1:35:40
  single ground truth. It can be as subtle as object
• 1:35:43
  placement and random material swapping to more drastic
• 1:35:46
  changes. Like entirely new biomes are
• 1:35:48
  locations of environment like Urban Suburban or rural this
• 1:35:53
  allows us to create infinite, targeted permutations for
• 1:35:56
  specific ground truce that we need more ground Truth for and
• 1:36:01
  all this happens within a click of a button.
• 1:36:05
  We could even take this one step further by altering our ground
• 1:36:09
  truth itself, say John wants his Network to pay more attention.
• 1:36:12
  The directional Road markings to better detect an upcoming
• 1:36:15
  Capital left turn lane. We can start to procedurally
• 1:36:19
  alter our landgraf inside the simulator to help folks, to
• 1:36:22
  create entirely new flows through this intersection to
• 1:36:25
  help Focus. The Network's attention to the
• 1:36:28
  road markings to create more accurate predictions.
• 1:36:29
  And this is a great example of how this tooling allows us to
• 1:36:34
  create new data that could never be.
• 1:36:35
  Elected from The Real World. The true power of this tool is
• 1:36:41
  in its architecture and how we could run all tests in parallel
• 1:36:44
  to infinitely scale, So you saw the tile Creator tool and action
• 1:36:49
  converting the ground truth labels into their counterparts.
• 1:36:53
  Next, we can use our tile extractor tool to divide this
• 1:36:56
  data into geohash tiles about 150 meters square in size.
• 1:37:01
  We then save out that data into separate geometry and instance
• 1:37:05
  files, this gives us a clean source of data, that's easy to
• 1:37:08
  load and allows us to be rendering engine agnostic for
• 1:37:11
  the future. Then using a tile loader tool.
• 1:37:16
  We can summon any number of those cash tiles using a Geo
• 1:37:19
  hash ID. Currently we're doing about
• 1:37:22
  these five by five tiles or 3x3 usually centered around Fleet
• 1:37:25
  hotspots or interesting language off locations in the tile
• 1:37:29
  loader. Also converts these tile sets
• 1:37:32
  into you assets, for consumption, by the Unreal
• 1:37:35
  Engine gives you a finished project product from what you
• 1:37:38
  saw in the first slide. And this really sets us up for
• 1:37:42
  size and scale. As you can see, on the map
• 1:37:45
  behind us, we can easily generate most of San Francisco
• 1:37:49
  city streets. And this didn't take years or
• 1:37:52
  even months of work or rather two weeks by one person, and we
• 1:37:56
  can continue to manage and grow all this data using RPG Network
• 1:38:00
  inside of the tooling, this allows us to throw compute at it
• 1:38:04
  and regenerate all these tilesets overnight this ensures
• 1:38:08
  all environments are of consistent quality and features
• 1:38:12
  which is super important for training since new ontology Xin
• 1:38:15
  signals are constantly released And now if to come full circle
• 1:38:23
  because we generated all these tilesets from ground truth data,
• 1:38:26
  they contain all the weird intricacies from The Real World.
• 1:38:29
  We can combine that with the procedural Visual and traffic
• 1:38:32
  variety to create Limitless, targeted data for the network to
• 1:38:36
  learn from That concludes the same section.
• 1:38:39
  I'll pass it to Kate to talk about how we can use all this
• 1:38:42
  data to improve autopilot. Thank you.
• 1:38:54
  Thanks David. Hi everyone.
• 1:38:56
  My name is Kate Park and I'm here to talk about the data
• 1:38:59
  engine which is the process by which we improve our neural
• 1:39:02
  networks via data. We're going to show you how we
• 1:39:06
  deterministically solve interventions via data, and walk
• 1:39:09
  you through the life of this particular clip.
• 1:39:12
  In this scenario, autopilot is approaching a turn and in
• 1:39:16
  correctly, predicts, that Crossing vehicle as stopped for
• 1:39:19
  traffic and thus, the vehicle that we would slow down.
• 1:39:22
  For in reality, there's nobody in the car, it's just awkwardly
• 1:39:26
  parked. We built this, tooling to
• 1:39:28
  identify the mispredictions correct, the label and
• 1:39:32
  categorize this clip into an evaluation.
• 1:39:34
  Set this particular clip happens to be one of 126 that we've
• 1:39:39
  diagnosed as challenging parked cars at turns because of this
• 1:39:43
  infra, we can curate this evaluation set without any
• 1:39:47
  engineering resources custom to this particular challenge case.
• 1:39:52
  To actually solve that challenge case, requires mining thousands
• 1:39:56
  of examples like it and it's something Tesla can trivially,
• 1:39:59
  do we simply use our data sourcing infra request data, and
• 1:40:04
  use the tooling shown previously to correct.
• 1:40:06
  The labels by surgically targeting the mispredictions of
• 1:40:10
  the current model. We're only adding the most
• 1:40:13
  valuable examples to our training set.
• 1:40:16
  We surgically fix 13,900 clips and because those were examples
• 1:40:22
  where the current model struggles, we don't even need to
• 1:40:25
  change the model architecture, a simple weight update with this.
• 1:40:28
  New valuable data is enough to solve the challenge case.
• 1:40:32
  So you see, we no longer predict that Crossing vehicle as stopped
• 1:40:35
  as shown in Orange, but parked as shown in red, In Academia, we
• 1:40:41
  often see that people keep data constant but a Tesla, it's very
• 1:40:45
  much the opposite we see time and time and again, that data is
• 1:40:49
  one of the best if not the most deterministic lever to solving
• 1:40:52
  these interventions. We just showed you the data
• 1:40:56
  engine Loop for one challenge case.
• 1:40:58
  Namely, these parked cars at turns, but there are many
• 1:41:01
  challenge cases, even for one signal of vehicle Movement.
• 1:41:05
  We apply this data engine Loop to every single challenge case
• 1:41:08
  we've diagnosed, whether it's buses, curvy Road, stopped
• 1:41:11
  Vehicles, parking, lots, and we don't just add data once we do
• 1:41:16
  this again, and again to perfect.
• 1:41:17
  The semantics, in fact, this year, we updated our vehicle
• 1:41:21
  movement signal 5 times and with Weight update trained on the new
• 1:41:26
  data. We push our vehicle movement,
• 1:41:28
  accuracy up and up. This data engine framework
• 1:41:33
  applies to all our signals, whether they're 3D multicam
• 1:41:36
  video, whether the data is human labeled, Auto labeled or
• 1:41:40
  simulated, whether it's an offline model or an online model
• 1:41:43
  model and Tesla's able to do this at scale because of the
• 1:41:47
  fleet Advantage, the infra that are Eng team has built, and the
• 1:41:51
  labeling resources that feed our networks to train on all this
• 1:41:55
  data. We need a massive amount of
• 1:41:57
  compute, so I'll hand it off to Pete and Ganesh to talk about,
• 1:42:00
  Out, the dojo supercomputing platform.
• 1:42:03
  Thank you. A petty.
• 1:42:13
  Thanks everybody. Thanks for hanging in there.
• 1:42:14
  We're almost there. My name is Pete Benton.
• 1:42:18
  I run the custom silicon and low voltage teams at Tesla.
• 1:42:23
  And my name is Ganesh blanket, I had on the dodgy program.
• 1:42:32
  Thank you. I'm frequently asked.
• 1:42:35
  Why is the car company building a supercomputer for training and
• 1:42:39
  this question fundamentally misunderstands the nature of
• 1:42:44
  Tesla at its heart. Tesla is a hard core technology
• 1:42:48
  company. All across the company people
• 1:42:51
  are working hard in science and engineering to advance the
• 1:42:55
  fundamental understanding and methods that we have available
• 1:42:59
  to build cars, Energy, Solutions, robots, and anything
• 1:43:04
  else. So can we can do to improve The
• 1:43:06
  Human Condition around the world.
• 1:43:08
  It's a super exciting thing to be a part of and it's a
• 1:43:11
  privilege to run a very small piece of it in the semiconductor
• 1:43:15
  group. Tonight, we're going to talk a
• 1:43:16
  little bit about dojo and give you an update on what we've been
• 1:43:20
  able to do over the last year. But before we do that, I wanted
• 1:43:23
  to give a little bit of background on the initial design
• 1:43:26
  that we started a few years ago. When we got started the goal was
• 1:43:29
  to provide a substantial Improvement to the training
• 1:43:33
  latency for our autopilot team. Some of the largest neural
• 1:43:36
  networks, they train today run for over a month which inhibits
• 1:43:40
  their ability to rapidly explore Alternatives and evaluate them.
• 1:43:45
  So, you know, a 30 x speed up would be really nice if we could
• 1:43:49
  provide it at a cost competitive and energy competitive way to do
• 1:43:53
  that. We wanted to build a chip with a
• 1:43:56
  lot of arithmetic arithmetic. Units.
• 1:43:59
  That we could utilize that a very high efficiency.
• 1:44:01
  And we spent a lot of time studying whether we could do
• 1:44:04
  that using DRM various packaging ideas, all of which failed and
• 1:44:10
  in the end, even though it felt like an unnatural act, we
• 1:44:12
  decided to reject dram as the primary storage medium for this
• 1:44:16
  system. And instead focus on S Ram
• 1:44:19
  embedded. In the chip that's ramp
• 1:44:21
  provides. Unfortunately, a modest amount
• 1:44:24
  of capacity, but extremely high bandwidth and very low latency
• 1:44:27
  and that enables us to achieve High utilization with your
• 1:44:30
  arithmetic units. Those choices.
• 1:44:35
  That particular choice led to a whole bunch of other choices.
• 1:44:38
  For example, if you want to have a virtual memory, you need page
• 1:44:41
  tables. They take up a lot of space.
• 1:44:43
  We didn't have space, so no virtual memory.
• 1:44:46
  We also don't have interrupts. The accelerator is a bare-bones.
• 1:44:51
  Rob piece of Hardware that's presented to a compiler.
• 1:44:54
  And the compiler is responsible for scheduling everything that
• 1:44:57
  happens in a deterministic way. So there's no need or even with
• 1:45:00
  desire for interrupts in the system.
• 1:45:03
  We also chose to pursue Zu model, parallelism as a training
• 1:45:08
  methodology, which is not the typical situation most most
• 1:45:13
  machines today. Use data parallelism, which
• 1:45:15
  consumes additional memory capacity, which we obviously
• 1:45:18
  don't have. So all of those choices.
• 1:45:21
  Let us to build a machine. That is pretty radically
• 1:45:25
  different from what's available today.
• 1:45:28
  We also had a whole bunch of other goals, and one of the most
• 1:45:31
  important ones was no limits. So we wanted to build a compute
• 1:45:34
  fabric that would scale on in an unbounded way, for the most
• 1:45:37
  part. I mean, obviously, there's
• 1:45:39
  physical limits now and then, but, you know, pretty much.
• 1:45:43
  If your model was too big for the computer, you just had to go
• 1:45:45
  buy a bigger computer. That's what we were looking for
• 1:45:49
  today. The way package machines are
• 1:45:51
  package, there's a pretty fixed ratio of, for example, GPU CPUs
• 1:45:55
  and D Ram capacity and Work capacity and we really wanted to
• 1:45:59
  disaggregate all that. So that, as models involved, we
• 1:46:03
  could bury the ratios of those various elements and make the
• 1:46:07
  system more flexible to meet the needs of the autopilot team.
• 1:46:12
  Yeah. And it's so true.
• 1:46:13
  Be like No Limits philosophy was our guiding star all the way,
• 1:46:18
  all of our choices were centered around that.
• 1:46:22
  And, and to the point that we didn't want traditional data
• 1:46:25
  center infrastructure to limit, With our capacity to execute
• 1:46:30
  these programs at speed. So that's why we, that's why
• 1:46:35
  we're sorry about that. That's why we integrated
• 1:46:40
  vertically. Our data center entire data
• 1:46:44
  center by doing a vertical. Integration of the data center,
• 1:46:48
  we could extract new levels of efficiency.
• 1:46:51
  We could optimize power delivery Cooling and as well as system
• 1:46:56
  management across. Ross, the whole data center
• 1:46:59
  stack rather than doing Box by box and integrating that those
• 1:47:04
  boxes into Data Centers. And to do this, we also wanted
• 1:47:11
  to integrate early to figure out limits of scale for our software
• 1:47:16
  workloads. So we integrated Dojo
• 1:47:18
  environment into our autopilot software very early and we
• 1:47:22
  learned a lot of lessons. And today, Bill Chang will go
• 1:47:27
  over our hardware update as well as some of the challenges that
• 1:47:31
  we faced along the way and regime Korean will give you a
• 1:47:36
  glimpse. Of our compiler technology, as
• 1:47:39
  well as go over some of our cool results.
• 1:47:49
  Thanks Pete. Thanks, Ganesh.
• 1:47:52
  I'll start tonight with a high level vision of our system.
• 1:47:56
  That will that will help set the stage for the challenges and the
• 1:48:00
  problems were solving. And then also, how software will
• 1:48:04
  then leverage this for performance?
• 1:48:07
  Now, our vision for Dojo is to build a single unified
• 1:48:10
  accelerate, a very large, one software would see a seamless
• 1:48:15
  compute plane with globally, addressable, very fast memory
• 1:48:19
  and all connected together with uniform, high bandwidth and low
• 1:48:23
  latency. Now, to realize this we need to
• 1:48:29
  use density to achieve performance.
• 1:48:32
  Now, we leverage technology to get this density in order to
• 1:48:35
  break levels of hierarchy, all the way from the chip to the
• 1:48:39
  scale-out systems. Now silicon technology has has
• 1:48:44
  used, this has done this for decades.
• 1:48:46
  Chips, had followed Moore's law to for density and integration
• 1:48:51
  to get per performance scaling Now, a key step in realizing
• 1:48:56
  that Vision was our training tile.
• 1:48:59
  Not only can we integrate 25 dies at extremely high
• 1:49:03
  bandwidth, but we can scale that to any number of additional
• 1:49:07
  tiles by just connecting them together.
• 1:49:11
  Now last year, we showcased, our first functional training tile
• 1:49:16
  and at that time we already had workloads running on it.
• 1:49:21
  And since then, the team here has been working hard and
• 1:49:25
  diligently to deploy this at scale.
• 1:49:29
  Now we've made amazing progress and had a lot of Milestones
• 1:49:32
  along the way. And of course we've had a lot of
• 1:49:35
  unexpected challenges, but this is where our fail fast
• 1:49:39
  philosophy. Has allowed us to push our
• 1:49:41
  boundaries Now pushing density for performance presents, all
• 1:49:48
  new challenges, one area is power delivery.
• 1:49:53
  Here, we need to deliver the power to our compute Dy and this
• 1:49:57
  directly impacts our top-line compute performance.
• 1:50:01
  But we need to do this at unprecedented density.
• 1:50:04
  We need to be able to match. Our die pitch with a power
• 1:50:08
  density of almost 1 amp per mm Square.
• 1:50:12
  And because of the extreme integration, this needs to be a
• 1:50:15
  multi-tiered vertical power solution.
• 1:50:19
  And because there's a complex heterogeneous material stack up,
• 1:50:22
  we have to carefully manage the material transition, especially
• 1:50:26
  CTE. Now, why does the coefficient of
• 1:50:31
  thermal expansion matter in this case?
• 1:50:34
  CTE is a fundamental material property and if it's not
• 1:50:38
  carefully managed that Stack Up would literally rip itself
• 1:50:42
  apart. So, we started this effort by
• 1:50:47
  working with vendors to deliver to develop this power solution,
• 1:50:52
  but we realize that we actually had to develop this in house.
• 1:50:56
  Now, to balance schedule and risk, we built quick iterations
• 1:51:01
  to support, both our system bring up in software development
• 1:51:05
  and also to find the optimal design and stack up that would
• 1:51:08
  meet our final production goals. And in the end, we were able to
• 1:51:12
  reduce CTE over 50% and meet our performance by 3x over our
• 1:51:19
  initial version. Now, needless to say, finding
• 1:51:23
  this optimal material stack up while maximizing performance at
• 1:51:27
  density is extremely difficult. Now we did have unexpected
• 1:51:34
  challenges along the way, here is an example where we push the
• 1:51:38
  boundaries of integration that led to component failures.
• 1:51:43
  This started, when we scaled up to larger and longer workloads,
• 1:51:47
  and then intermittent intermittently a single site on
• 1:51:50
  a tile would fail. Now, they started out as
• 1:51:54
  recoverable failures, but as we pushed some much higher and
• 1:51:57
  higher power, these would become permanent values.
• 1:52:03
  Now, to understand this failure, you have to understand why and
• 1:52:07
  how we build our power modules solving density at every level
• 1:52:13
  is that is is the Cornerstone of actually achieving our system
• 1:52:16
  performance now, because our XY plane is used for high bandwidth
• 1:52:21
  communication, everything else must be stacked vertically.
• 1:52:26
  This means all other components other than our dye must be
• 1:52:30
  integrated into our power modules.
• 1:52:32
  Now that includes our clock and our power supplies and also our
• 1:52:36
  system controllers, Now, in this case, the Pharaohs were due to
• 1:52:41
  losing clock output from our oscillators.
• 1:52:45
  And after an extensive debug, we found that the root cause was
• 1:52:49
  due to vibrations on the module from piezoelectric effects.
• 1:52:53
  Are nearby capacitors. Now, singing caps are not a new
• 1:52:59
  phenomenon and in fact, very common in power design, but
• 1:53:03
  normally clock chips are placed in a very quiet area of the
• 1:53:06
  board and often not affected by power circuits.
• 1:53:10
  But because we needed to achieve this level of integration, these
• 1:53:14
  oscillators need to be placed in very close proximity.
• 1:53:18
  Now, due to our switching frequency and then the vibration
• 1:53:21
  resonance created, it caused out of plane vibration on our mems
• 1:53:26
  oscillator that caused it to crack.
• 1:53:31
  Now, the solution to this problem is a multi-prong
• 1:53:33
  approach. We can reduce the vibration by
• 1:53:36
  using soft terminal caps We can update our mems part with a
• 1:53:43
  lower Q factor for the out of plane Direction.
• 1:53:48
  And we can also update our switching frequency frequency to
• 1:53:52
  push the resonance further away from these sensitive bands.
• 1:53:58
  Now, addition to the density at the system level, we've been
• 1:54:03
  making a lot of progress at the infrastructure level.
• 1:54:07
  We knew that we had to read examine every aspect of the data
• 1:54:11
  center infrastructure in order to support our unprecedented
• 1:54:15
  power and cooling density. We brought in a fully custom
• 1:54:19
  design CD you to support dojos dense, cooling requirements.
• 1:54:24
  And the amazing part is we're able to do this at a fraction of
• 1:54:26
  the cost versus buying off the shelf and modifying it And since
• 1:54:32
  our Dojo cabinet, integrates enough, power and cooling to
• 1:54:35
  match an entire row of standard, it racks, we need to carefully
• 1:54:40
  design, our cabinet in an infrastructure together.
• 1:54:44
  And we've already gone through several iterations of this
• 1:54:47
  cabinet to optimize this. And earlier this year, we
• 1:54:51
  started load testing our power and cooling infrastructure.
• 1:54:54
  And we were able to push it over to megawatts before we tripped
• 1:54:57
  our substation, and got a call from the city.
• 1:55:04
  Now, last year we introduced only a couple of components of
• 1:55:07
  our system, the custom D1 die and the training tile, but we
• 1:55:11
  tease the exit pod. As our end goal will walk
• 1:55:15
  through the remaining parts of our system that are required to
• 1:55:18
  build out this exit pod. Now, the system tray is a key
• 1:55:23
  part of realizing our vision of a single accelerator, it enables
• 1:55:28
  us to seamless seamlessly connect tiles together.
• 1:55:31
  Not only within the cabinet, but between cabinets, we can connect
• 1:55:36
  these Tiles at very tight spacing across the entire
• 1:55:40
  accelerator. And this is how we achieve our
• 1:55:42
  uniform communication. This is a laminated busbar that
• 1:55:47
  allows us to integrate very high power mechanical, and thermal
• 1:55:51
  support and an extremely dense integration.
• 1:55:54
  It's 75. Mm in height and and support six
• 1:55:58
  Tiles. At 135 kg.
• 1:56:01
  This is the equivalent of three to four fully loaded, high
• 1:56:05
  performance racks. Next, we need to feed data to
• 1:56:11
  the training tiles. This is where we've developed
• 1:56:14
  the dojo interface processor. It provides our system with high
• 1:56:19
  bandwidth D Ram to Stage our training data and it provides
• 1:56:23
  full memory bandwidth to our training tiles.
• 1:56:26
  Using TTP, our custom protocol that we can use to communicate
• 1:56:30
  across our entire accelerator It also has high speed, ethernet.
• 1:56:35
  That helps us extend this custom protocol over standard ethernet
• 1:56:39
  and we provide native Hardware support for this with little to
• 1:56:43
  no software overhead. And lastly, we can connect
• 1:56:47
  connect to it through a standard Gen4 pcie interface. now we pair
• 1:56:54
  20 of these cards portray, and that gives us 640 gigabytes of
• 1:56:59
  high-bandwidth DRM And this provides our disaggregated
• 1:57:03
  memory layer for our training tiles.
• 1:57:06
  These cards are our high bandwidth in just path.
• 1:57:09
  Both through pcie and ethernet. They also provide a high rate x
• 1:57:14
  z connectivity path. That allows shortcuts across our
• 1:57:18
  large Dojo accelerator. Now we actually integrate the
• 1:57:24
  host directly underneath our system tray.
• 1:57:27
  These hosts provide our in just processing and connect to our
• 1:57:31
  interface processors through pcie.
• 1:57:34
  These host can provide Hardware video decoder support for
• 1:57:39
  video-based training. And our user applications land
• 1:57:43
  on these host that we so we can provide them with a standard x86
• 1:57:48
  Linux environment. Now, we can put two of these
• 1:57:54
  assemblies into one cabinet and pair it with redundant.
• 1:57:58
  Power supplies that do direct conversion of three-phase 480.
• 1:58:02
  Volt AC power 252 volt, DC power Now, by focusing on density, at
• 1:58:12
  every level, we can realize the vision of a single accelerator.
• 1:58:18
  Now, starting with the uniform nodes, on our custom D1 die, we
• 1:58:22
  can connect them together and are fully integrated training
• 1:58:25
  tile. And then finally seamlessly
• 1:58:29
  connecting them across cabinet boundaries to form our Dojo
• 1:58:33
  accelerator. And all together, we can house
• 1:58:37
  to full accelerators in our exit pod for a combined.
• 1:58:41
  One exaflop of ml compute. Now, all can all together, this
• 1:58:45
  amount of technology and integration has only ever been
• 1:58:49
  done a couple of times in the history of compute.
• 1:58:53
  Next, we'll see how software can leverage this to accelerate
• 1:58:56
  their performance. thanks Bill, my name is Rajiv and I'm going
• 1:59:08
  to talk some numbers SAR software stack begins with the
• 1:59:12
  pie torch extension that speaks to our commitment to one
• 1:59:15
  standard python Which models out of the box.
• 1:59:19
  We're going to talk more about our jit compiler and be in just
• 1:59:22
  pipeline that feeds the hardware with data.
• 1:59:25
  Abstractly performances tops times, utilization times
• 1:59:29
  accelerator occupancy. We've seen how the hardware
• 1:59:32
  provides Peak Performance, is the job of the compiler to
• 1:59:35
  extract utilization from the hardware while code is running
• 1:59:38
  on it. And it's the job of the interest
• 1:59:41
  pipeline to make sure that data can be fed at Ruppert high
• 1:59:44
  enough for the hardware to not ever starve.
• 1:59:48
  So let's talk about why communication bound models are
• 1:59:50
  difficult to scale. But before that, let's look at
• 1:59:53
  why resident 50 like models are easier to skill.
• 1:59:56
  You start off with a single accelerator, one of the forward
• 1:59:58
  and backward, pass has followed by the optimizer.
• 2:00:02
  Then to scale this up, you run multiple copies of this on
• 2:00:05
  multiple accelerators and while the gratings produced by the
• 2:00:08
  backward pass, do need to be reduced.
• 2:00:09
  And this introduces, some communication, this can be done
• 2:00:12
  Pipeline with the backward pass. This set of scales fairly well,
• 2:00:19
  almost linearly. For models with much larger
• 2:00:24
  activations, we run into a problem as soon as we want to
• 2:00:27
  run the forward. Pass the batch size that fits in
• 2:00:30
  a single accelerator is often smaller than the batch Norm
• 2:00:32
  surface. So, to get around this
• 2:00:34
  researchers, typically one the setup on multiple accelerators,
• 2:00:38
  in sync, bash, normal mode, this introduces latency bound,
• 2:00:41
  communication to the critical path of the forward pass, and we
• 2:00:44
  already have a communication bottleneck And while there are
• 2:00:48
  ways to get around this, they usually involve tedious.
• 2:00:50
  Manual work, best suited for a compiler and ultimately there's
• 2:00:55
  no skirting around. The fact that if your state does
• 2:00:57
  not fit in a single accelerator, you can be communication about
• 2:01:03
  and even with significant efforts from our ml Engineers,
• 2:01:06
  we see such models don't scale linearly the dojo system was
• 2:01:11
  built to make such models work at how utilization the high
• 2:01:15
  density and equation is was built to not only accelerate the
• 2:01:18
  compute bound portions of a model but also the latency bound
• 2:01:22
  portions, like a batch Norm or the bandwidth bound portions,
• 2:01:26
  like a gradient, all reduced or a parameter all gather.
• 2:01:31
  A slice of the dojo mesh can be carved out to run any model.
• 2:01:35
  The only thing you just need to do is to make the slice large
• 2:01:38
  enough to fit a bathroom surface for their particular model.
• 2:01:43
  After that, the partition presents itself as one large
• 2:01:46
  accelerator, being the users from having to worry about the
• 2:01:49
  internal details of execution. And as the job of the compiler
• 2:01:54
  to maintain this abstraction fine-grain synchronization
• 2:01:58
  Primitives in uniform. Low latency makes it easy to
• 2:02:01
  accelerate, all forms of parallelism across integration
• 2:02:04
  boundaries, tensors are usually stored, Chardon as RAM and
• 2:02:07
  replicated. Just in time for layers
• 2:02:10
  execution, we depend on the high Dojo being with to hide this
• 2:02:14
  replication time. Pensive replication and other
• 2:02:17
  day of transfers are overlapped with compute and the compiler
• 2:02:20
  can also recompute layers. One is profitable to do so, We
• 2:02:25
  expect most models to work out of the box is an example.
• 2:02:29
  We took the recently released stable diffusion model and got
• 2:02:32
  it running on dojo and minutes out of the box.
• 2:02:35
  The compiler was able to map it in a model.
• 2:02:37
  Pelham Manor on 25, Dojo dies. Here's some pictures of a cyber
• 2:02:42
  truck on Mars generated by stable, diffusion running on
• 2:02:45
  dojo looks Looks like it. Still has some ways to go before
• 2:02:56
  matching the Tesla Design Studio team.
• 2:03:00
  So we've talked about how communication bottlenecks can
• 2:03:02
  hamper scalability. Perhaps an acid test of a
• 2:03:05
  compiler, and the underlying Hardware is executing across
• 2:03:09
  diet Bachelor. Like mentioned before, this can
• 2:03:12
  be a Serial bottleneck. The communication phase of a
• 2:03:15
  bachelor begins with nodes Computing.
• 2:03:16
  Their local mean and standard deviations then coordinating to
• 2:03:20
  reduce these values, then broadcasting these values back
• 2:03:23
  and then they resume their work in parallel.
• 2:03:27
  So what would an ideal bathroom look like on 25 Dojo dies?
• 2:03:31
  Let's say the previous less activations are already split
• 2:03:34
  across the ice. We would expect the 350 nodes on
• 2:03:39
  each die to coordinate and produce die.
• 2:03:41
  Local mean and standard deviation values.
• 2:03:44
  Ideally, these would get rid further videos with the final
• 2:03:46
  value, ending somewhere in towards the middle of the tile.
• 2:03:51
  We would then hope to see a broadcast of this value.
• 2:03:53
  Radiating from the center. Let's see how the compiler,
• 2:03:57
  actually executes. A real bathroom operation across
• 2:04:00
  25. Does the communication trees
• 2:04:02
  were extracted from the compiler and the timing is from a real
• 2:04:06
  Hardware one. We're about to see eight
• 2:04:08
  thousand seven fifty nodes, on 25 dies, coordinating to reduce
• 2:04:12
  and then broadcast the bass drum mean and standard deviation
• 2:04:15
  values. Die, local reduction followed by
• 2:04:20
  global reduction towards the middle of the tie.
• 2:04:23
  Then the reduced value broadcast radiating from the middle
• 2:04:27
  accelerated by the Harbor's broadcast facility.
• 2:04:33
  This operation takes only five microseconds on 25 Dojo dice.
• 2:04:38
  The same operation takes 150 microseconds on 24 gpus.
• 2:04:42
  This is an orders of magnitude improvement over gpus and while
• 2:04:47
  we talked about and already saw operation in the context of a
• 2:04:49
  batch Norm, it's important to reiterate that the same
• 2:04:52
  advantages apply to all other communication Primitives and
• 2:04:56
  these Primitives are essential for large-scale training.
• 2:05:01
  So how about full model performance?
• 2:05:03
  So while we think that resonate, 50 is not a good representation
• 2:05:06
  of real world, Tesla workloads. It is a standard Benchmark so
• 2:05:10
  let's start there. We are already able to match
• 2:05:13
  the, a 100 die for dog. However, perhaps a hint of dojos
• 2:05:17
  capabilities is that were able to hit this number with just a
• 2:05:20
  batch of 8, / die, but Dojo was really built.
• 2:05:23
  A tackle larger complex models. So, when we set out to tackle
• 2:05:28
  real-world workloads, we looked at the usage patterns of her
• 2:05:31
  current GPU cluster. And to model, stood up the auto
• 2:05:34
  labeling networks across of offline models, that are used to
• 2:05:37
  generate ground truth, and the occupancy networks that you
• 2:05:40
  heard about. The are living that perks are
• 2:05:43
  large models that have higher asthmatic intensity.
• 2:05:46
  All the occupancy networks can be in just bound, we chose these
• 2:05:50
  models because together, they account for a large chunk of our
• 2:05:53
  current GPU cluster usage and they would challenge the system
• 2:05:56
  in different ways. So how do we do on these two
• 2:06:01
  networks? The results were about to see
• 2:06:03
  where measured on Multi die systems for both the GPU and
• 2:06:07
  Dojo but normalized important numbers.
• 2:06:10
  On our Auto labeling Network were already able to surpass the
• 2:06:14
  performance of an a 100 with our current Hardware running, on our
• 2:06:17
  older generation, prm's on our production Hardware, with our
• 2:06:21
  nerve, your arms. That translates to doubling the
• 2:06:23
  throughput of an a 100. And our model showed that with
• 2:06:27
  some key compiler optimizations we could get two more than 3x.
• 2:06:31
  The performance have been a 100, we see even bigger leaps on the
• 2:06:35
  occupancy Network. Almost 3x with our production
• 2:06:39
  Hardware, with room for more. So, what does that mean for
• 2:06:52
  Tesla with a current level of compiler performance, we could
• 2:06:55
  replace the ml computer of one, two, three, four, five and six
• 2:07:01
  GPU boxes with just a single Dojo tile.
• 2:07:13
  And this Dojo tile cost less than one of these GPU boxes.
• 2:07:20
  It is what it really means is that networks that took more
• 2:07:25
  than a month to train. Now take a less than a week.
• 2:07:31
  Last one, we measure things, it did not turn out so well at the
• 2:07:35
  pythons level we did not see our expected performance out of the
• 2:07:38
  gate and this timeline chart shows, our problem, the teeny
• 2:07:42
  tiny little green bars. That's the compiled code running
• 2:07:45
  on the accelerator. The row is mostly white space
• 2:07:48
  where the hardware is just waiting for data.
• 2:07:54
  With our dense. I'm old compute Dojo host
• 2:07:56
  effectively have 10x more ammo compute than the GPU host.
• 2:08:00
  The data loader is running on this one.
• 2:08:01
  Host simply couldn't keep up with all that ml hard work.
• 2:08:06
  So, to solve our data loader scalability issues, we knew we
• 2:08:09
  had to get over the limit of this single host.
• 2:08:12
  The Tesla transport protocol moves data seamlessly across,
• 2:08:15
  host tiles and ingest processors.
• 2:08:18
  So we extended the Tesla transport protocol to work over
• 2:08:21
  ethernet. We didn't build the dojo network
• 2:08:23
  interface card, that zdenek to leverage.
• 2:08:26
  TTP, over ethernet. This allows any host with a
• 2:08:29
  diner card, to be able to dma to.
• 2:08:31
  And from other TTP on points, So we started with the dojo mesh,
• 2:08:37
  then we added a tier of data loading host equipped with the
• 2:08:40
  Dina card. We connected These Hoes to the
• 2:08:45
  mesh via an ethernet switch every host in this data.
• 2:08:48
  Loading tier is capable of reaching all ttpm points in the
• 2:08:51
  dojo mesh via hardware-accelerated dma.
• 2:08:57
  After these optimizations went in, our occupancy run from 4% to
• 2:09:02
  97 percent. So the data loading sections
• 2:09:05
  have reduced They'll the data loading sections have reduced
• 2:09:13
  drastically. And the ml Hardware is kept
• 2:09:15
  busy. We actually expect this number
• 2:09:17
  to go to 100% pretty soon. After these changes went in, we
• 2:09:21
  saw the full expected speed up from the Peter Schuler and we
• 2:09:24
  were back in business. So we started with Hardware
• 2:09:29
  design that breaks through traditional integration
• 2:09:31
  boundaries in service of our vision of a single giant
• 2:09:34
  accelerator. We've seen how the compiler and
• 2:09:36
  in just layers build on top of that Hardware.
• 2:09:40
  So, after proving our performance on these complex,
• 2:09:42
  real world networks, we knew what our first large-scale
• 2:09:45
  deployment would Target our high rithmetic intensity
• 2:09:48
  auto-leveling Networks. Today, that occupies 4000 gpus
• 2:09:53
  over 72 GPU Rex with our dense computer in a high-performance.
• 2:09:58
  We expect to provide the same throughput with just for Dojo
• 2:10:02
  cabinets. And these for Dojo cabinets will
• 2:10:13
  be part of our first exit, but that we plan to build by quarter
• 2:10:16
  one of 2023 This will more than double Tesla's out of labeling
• 2:10:21
  capacity. The first exit part is part of a
• 2:10:30
  total of seven exit parts that we plan to build in Palo Alto
• 2:10:34
  right here across the wall. And we have a display cabinet
• 2:10:40
  from one of these extra parts for everyone to look at.
• 2:10:45
  Six tiles. Densely, packed on a tray. 54,
• 2:10:49
  petaflops of compute 640, GB of high, bandwidth memory, with
• 2:10:53
  power, and host defeated A lot of confusion.
• 2:11:02
  And we're building out new versions of all our cluster
• 2:11:05
  components, and constantly improving our software to hit
• 2:11:08
  new limits of skill. We believe that we can get
• 2:11:11
  another 10x improvement with our next Generation hard work.
• 2:11:16
  And to realize our ambitious goals, we need the best software
• 2:11:19
  and Hardware Engineers. So please come talk to us or
• 2:11:22
  visit Tesla.com a. I thank you.
• 2:11:39
  Let me know. All right, so we hopefully that
• 2:11:48
  was enough detail and now we can move to questions and guys like
• 2:11:58
  I think I got the team came out kind of came out on stage and we
• 2:12:04
  really wanted to show the depth and breadth of Tesla and
• 2:12:10
  artificial intelligence. The computer hardware robotics
• 2:12:14
  actuators and and try to really shift the perception of the
• 2:12:19
  company away from, you know, a lot of people think we're like,
• 2:12:24
  just a car company or we make cool cars, whatever, but they
• 2:12:28
  don't have most people have no idea that Tesla is arguably the
• 2:12:33
  leader in real-world, AI hardware and software and that
• 2:12:38
  we're building. What is arguably the first some
• 2:12:44
  of the most radical computer architecture since the cray-1
• 2:12:49
  supercomputer? And I think if you're interested
• 2:12:51
  in developing some some of the most advanced technology in the
• 2:12:55
  world that's going to really affect the world in a positive
• 2:12:58
  way tells us the place to be. So yeah, let's fire away with
• 2:13:03
  some questions. I think there's a mic at the
• 2:13:08
  front and a mic at the back. Or just throw Mike's of people
• 2:13:18
  jump off of the mic. Yeah.
• 2:13:21
  Hi, thank you very much. I was impressed here.
• 2:13:26
  Yeah, I was impressed very much by Optimist, but I wonder why
• 2:13:31
  they don't driven the hunt. Why did we choose attend?
• 2:13:34
  Do driven approach for the country because tendons are not
• 2:13:37
  very durable and why spring-loaded?
• 2:13:44
  This is pretty cool. Awesome.
• 2:13:46
  Yes, that's a great question. You know when it comes to any
• 2:13:49
  type of actuation scheme, there's trade-offs between, you
• 2:13:52
  know, whether or not it's a tender and system or some type
• 2:13:54
  of linkage based system. My close to your mouth.
• 2:13:57
  Let me closer. Jeremy, cool.
• 2:14:01
  So yeah, the main reason why we went for a tendon based system
• 2:14:05
  is that, you know, first we actually investigated some
• 2:14:07
  synthetic tendons but we found that metallic boating cables.
• 2:14:11
  Are, you know, a lot stronger. One of the advantages of these
• 2:14:15
  cables is that it's very good for part reduction.
• 2:14:19
  We do want to make a lot of these hands.
• 2:14:20
  So having a bunch of parts, a bunch of small linkages ends up
• 2:14:24
  being, you know, a problem when you're making a lot of
• 2:14:26
  something, one of the big reasons that, you know, Tendons
• 2:14:31
  are better than linkages in a sense is that you can be anti
• 2:14:34
  backlash. So anti backlash essentially,
• 2:14:37
  you know, allows you to not have any gaps or you know stuttering
• 2:14:41
  Motion in your fingers spring-loaded.
• 2:14:44
  Mainly what spring-loaded allows us to do is allows us to have
• 2:14:49
  active opening. So instead of having to have two
• 2:14:52
  actuators to drive the fingers closed and then open, we have
• 2:14:56
  the ability to, you know, have the tendon drive them close and
• 2:14:59
  then the springs passively. And this is something that's
• 2:15:02
  seen in our hands as well, right?
• 2:15:04
  We have the ability to actively flex and then we also have the
• 2:15:07
  ability to extend. Yeah, our goal with Optimus is
• 2:15:12
  to have a robot that is maximally useful as quickly as
• 2:15:16
  possible. So there's a lot of ways to
• 2:15:18
  solve the various problems of a humanoid robot and we're
• 2:15:23
  probably not barking up the right Tree on all the Technical
• 2:15:27
  Solutions. And I should say that we're
• 2:15:29
  we're open to evolving the Solutions that you see here,
• 2:15:32
  over time, we're not, they're not locked in stone but we do we
• 2:15:36
  have to pick something in and we want to pick it.
• 2:15:39
  Pick something that's going to allow us to produce the robot as
• 2:15:43
  quickly as possible. And have it likes it be useful
• 2:15:46
  as quickly as possible. We're trying to follow the goal
• 2:15:49
  of fastest path, to a useful robot that can be made at
• 2:15:53
  volume. And we're going to test the
• 2:15:55
  robot internally at Tesla in our Factory and to see he liked how
• 2:16:01
  useful is it? Because you have to have a
• 2:16:04
  you're going to close the loop on reality to confirm that the
• 2:16:06
  robot is, in fact, useful and yeah.
• 2:16:12
  So we're going to use it to build things and we're confident
• 2:16:17
  we can do that with the hand that we have currently designed.
• 2:16:19
  But there's I'm sure they'll be had version to version 3 and we
• 2:16:22
  may change the architecture of quite significantly over time.
• 2:16:30
  Hi. You're The Optimist.
• 2:16:33
  Robot is really impressive. That you did a great job bipedal
• 2:16:37
  robots are really difficult. But what I notice might be
• 2:16:41
  missing from your plan is to acknowledge the utility of the
• 2:16:47
  human spirit. And I'm wondering if Optimus
• 2:16:50
  will ever get a personality and be able to laugh at our jokes
• 2:16:53
  while they've well folds are closed. yeah, absolutely, I
• 2:16:58
  think we want to have Really fun versions of optimists.
• 2:17:04
  And so that Optimus can both do be your utilitarian and do
• 2:17:09
  tasks, but can also be kind of like a friend and a buddy and,
• 2:17:14
  and hang out with you. And I'm sure people will think
• 2:17:18
  of all sorts of creative uses for this robot and you know, the
• 2:17:25
  thing once you have the core intelligence and actuators
• 2:17:29
  figure it out, then you can Actually, you know put all sorts
• 2:17:34
  of costumes, I guess, unless on the robot, I mean you can make
• 2:17:39
  the Robert look, you can scan the robot in many different ways
• 2:17:47
  and I'm sure people will find very interesting ways to to,
• 2:17:52
  yeah, versions of optimist. Thanks for the great
• 2:17:59
  presentation. I wanted to know if there was an
• 2:18:01
  equivalent to interventions in Optimist, it seems like labeling
• 2:18:06
  through moments where humans disagree with what's going on is
• 2:18:08
  important. And in a humanoid robot that
• 2:18:12
  might be also desirable source of information.
• 2:18:19
  That's what I say. Yeah, I think we will have ways
• 2:18:26
  to remote operate. The robot and intervene.
• 2:18:28
  When it does something bad, especially when we are training
• 2:18:31
  the robot and bringing it up and hopefully we, you know, design
• 2:18:35
  it in a way that we can stop the robot from, if it's going to hit
• 2:18:38
  something, we can just like, hold it, and then we'll stop it.
• 2:18:40
  Won't like, you know, crush your hand or something and those are
• 2:18:43
  all intervention data. Yeah.
• 2:18:46
  And we can do not form of simulation systems to where we
• 2:18:48
  can check for collisions and supervise that there's a bad
• 2:18:51
  actions. Yeah, so after Miss we want
• 2:18:55
  overtime to for to be, you know, an Android kind of Android that
• 2:18:59
  you see in in in Sci-Fi movies like Star Trek the Next
• 2:19:03
  Generation like data, but obviously we could program the
• 2:19:06
  robot to be less robot like and more friendly and and you know,
• 2:19:11
  can obviously learn to emulate humans and feel very natural.
• 2:19:15
  All. So as AI in general improves, we
• 2:19:19
  can add that to the robot and it should be obviously able to do
• 2:19:26
  simple instructions or even into it what it is that you want.
• 2:19:31
  So you could give it a high level instruction and then it
• 2:19:34
  can break that down into a series of actions and and take
• 2:19:37
  those actions. Hi yeah, it's exciting to think
• 2:19:45
  that with The Optimist you will think that you can achieve
• 2:19:49
  orders of magnitude of improvement in economic output,
• 2:19:54
  that's really exciting. And when Tesla started the
• 2:19:57
  mission was to accelerate the Advent of renewable energy or
• 2:20:01
  sustainable transport. So with The Optimist, do you
• 2:20:05
  still see that mission being thus mission statement of Tesla?
• 2:20:09
  Or is it going to be updated with you know me In to
• 2:20:13
  accelerate the Advent of no infinite, abundance or it,
• 2:20:18
  Limitless Limitless, economy. Yeah, it mean it is not strictly
• 2:20:23
  speaking. Optimus is not strictly
• 2:20:25
  speaking. Directly in line with
• 2:20:31
  accelerating sustainable energy it you know to agree that it is
• 2:20:36
  more efficient at getting things done than a person.
• 2:20:39
  It is I guess help with if you know, sustainable energy.
• 2:20:43
  But I think the mission effectively does is somewhat
• 2:20:46
  broaden with the Advent of Optimus to, you know, I don't
• 2:20:51
  know making the future awesome. So you know I think you look at
• 2:20:55
  Optimist and I know about you but I I'm excited.
• 2:20:58
  To see what Optimus will become. And, you know, this is like, you
• 2:21:04
  know, if you could I mean you can tell like any given
• 2:21:07
  technology. Are you do you want to see what
• 2:21:11
  it's like in a year? Two years three years, four
• 2:21:14
  years, five years, ten. I'd say for sure.
• 2:21:17
  You definitely want to see what's happening with Optimus
• 2:21:20
  whereas you know a bunch of other Technologies or you know
• 2:21:23
  sort of plateaued but name names here.
• 2:21:28
  But you know so I think after this is gonna be incredible in
• 2:21:39
  like 5 years, 10 years, like mind-blowing.
• 2:21:41
  And I'm really interested to see that happen.
• 2:21:43
  I hope you are too. Thank you.
• 2:21:48
  I have a quick question here. Justin.
• 2:21:51
  And I was wondering, like, are you planning to extend like
• 2:21:55
  conversational capabilities for the robot and my second
• 2:21:59
  follow-up question to that is, what's like the end goal?
• 2:22:03
  What's the end goal with optimist?
• 2:22:07
  Yeah, Optimus were definitely of conversational capabilities.
• 2:22:10
  So You're be able to talk to it and have a conversation and it
• 2:22:16
  would feel quite natural. So from a technical standpoint
• 2:22:21
  I'm I do I think it's a keep evolving and I'm not sure where
• 2:22:29
  it ends up, but someplace interesting, for sure.
• 2:22:34
  You know, we always have to be careful about the, you know,
• 2:22:36
  don't go down the Terminator path.
• 2:22:39
  That's a I thought for my maybe we should start off with a video
• 2:22:43
  of like the Terminator. Starting off with the, you know,
• 2:22:46
  skull crushing, but that might be enough.
• 2:22:48
  You want to get too seriously? So, you know, we we do want
• 2:22:53
  Optimus to be safe. So we are designing in
• 2:22:57
  safeguards where you can locally, stop the robot and You
• 2:23:05
  know, with like basically a localized control ROM that you
• 2:23:08
  can't update over the Internet, which I think that's quite
• 2:23:11
  important essential frankly. So like a localized, stop
• 2:23:19
  button, remote, remote, control, something like that.
• 2:23:24
  That cannot be changed. But it's definitely going to be
• 2:23:32
  interesting. It won't be boring.
• 2:23:40
  Okay, yeah, I see you today. You have a very attractive
• 2:23:43
  product with told you and it's applications.
• 2:23:46
  So I'm wondering what's the future for Tokyo platform.
• 2:23:48
  We would like to provide to like a infrastructure infrastructure
• 2:23:52
  as a service, like a wa sou be like a sealed.
• 2:23:55
  Achieve like the Nvidia. So, basically what the future
• 2:23:58
  because of that, I say you use 70 meters, which is the
• 2:24:01
  development cost like a easily over 10 million u.s. dollars.
• 2:24:05
  How do you make the penis? Is like a penis wise?
• 2:24:09
  Yeah, I mean Dojo is a very big computer and actually will be
• 2:24:16
  use a lot of power and needs a lot of cooling so I think it's
• 2:24:19
  probably going to make more sense to have Dojo operate in
• 2:24:22
  like Amazon web services manner then to try to sell it to
• 2:24:26
  someone else. So the the most that would be
• 2:24:30
  the most efficient way to operate Dojo is just have it be
• 2:24:34
  a service that you can use that's available online.
• 2:24:39
  And that where you can train your models way faster and for
• 2:24:43
  less money. And as the world transitions to
• 2:24:48
  software 2.0, and that's on the bingo card.
• 2:24:55
  And someone I know, it has to know how to drink five, two
• 2:24:57
  killers. So, let's see.
• 2:25:03
  Software 2.0 will use a lot of neural, net training.
• 2:25:11
  So it kind of makes sense that over time.
• 2:25:16
  As there's more more neural, net stuff, people will want to use
• 2:25:21
  and the fastest lowest cost neural network training system.
• 2:25:25
  So I think is a lot of opportunity in that direction.
• 2:25:32
  Hi. My name is Alicia honey on.
• 2:25:35
  Thank you for this event is very inspirational.
• 2:25:39
  My question is I'm wondering what is your vision for Humanity
• 2:25:46
  robots? That understand our emotions and
• 2:25:51
  art and can contribute to our creativity?
• 2:25:58
  Well, I think there's this you're already seeing robots
• 2:26:01
  that At least. Are able to generate very
• 2:26:05
  interesting art with like, like Dolly and Delhi to and think
• 2:26:12
  we'll start seeing AI that can actually generate even movies
• 2:26:17
  that have a that have coherence like interesting movies and tell
• 2:26:20
  jokes. So it's quite remarkable how
• 2:26:23
  fast AI is advancing. At many companies besides Tesla.
• 2:26:32
  we're headed for a very interesting future and yeah, so
• 2:26:38
  You guys want to come out on that?
• 2:26:39
  Yeah, I guess The Optimist reward can come up with physical
• 2:26:43
  art, not just digital art, you can, you know, you can ask for
• 2:26:47
  some dance moves in text or voice and then you can produce
• 2:26:50
  those in the future. So, it's not affect physical
• 2:26:52
  heart, not just digital art. Oh yeah.
• 2:26:56
  Yeah. Computers can absolutely make
• 2:26:58
  physical art. Yeah.
• 2:26:59
  Arm sent an advance shortly. Soccer or whatever.
• 2:27:03
  It needs to get more agile but or time for sure.
• 2:27:09
  Thanks so much for the presentation for the Tesla
• 2:27:12
  autopilot. Slides I noticed that the models
• 2:27:15
  that you are using or heavily motivated by language models and
• 2:27:19
  I was wondering what the history of that was and how much of an
• 2:27:22
  improvement it gave. I thought that that was a really
• 2:27:24
  interesting. Curious choice to use language
• 2:27:26
  models for the lane transitioning.
• 2:27:29
  So there are sort of two aspects for why we transition to
• 2:27:31
  language modeling. So do a talk talk loud and
• 2:27:34
  close. Okay, okay, got it.
• 2:27:39
  Yeah, so the language models help us in two ways.
• 2:27:42
  The first way is that it lets us predict lanes that we couldn't
• 2:27:44
  have. Otherwise as I shook mentioned
• 2:27:46
  earlier basically when we predicted lanes and sort of a
• 2:27:49
  dense 3D fashion, you can only model certain kinds of lanes but
• 2:27:53
  we want to get those criss-crossing connections
• 2:27:55
  inside of intersections. It's just not possible to do
• 2:27:57
  that without making it a graph prediction.
• 2:27:59
  If you try to do this with dense segmentation is just doesn't
• 2:28:01
  work. Also the link prediction is a
• 2:28:05
  multi-modal problem, sometimes you just don't have sufficient
• 2:28:08
  visual information. To know precisely how things
• 2:28:10
  look, on the other side of the intersection.
• 2:28:12
  So you need a method that can generalize and produce coherent
• 2:28:16
  predictions. You don't want to be predicting
• 2:28:18
  two lanes in three lanes. At the same time, you want to
• 2:28:20
  commit to one in a generative model like these language models
• 2:28:23
  provides that All right. Oh hi.
• 2:28:30
  My name is Giovanni. Yeah.
• 2:28:33
  Thanks for the presentation. That's really nice.
• 2:28:37
  I have a question for FSD team. So for the neural networks, how
• 2:28:44
  do you test a hottie to unit test software units?
• 2:28:47
  That sounded like? Do you have like a punch or I
• 2:28:51
  don't know, maybe thousands or? Yes, cases.
• 2:28:56
  Where The neural network that after you train it, you have to
• 2:29:00
  pass it before you release it to as a product or it.
• 2:29:04
  Yeah, what's your software unit testing strategies for this?
• 2:29:08
  Yeah, glad you asked that's like a series of tests that we have
• 2:29:11
  defined starting from, you know, unit test for the software
• 2:29:14
  itself. But then for the neural network
• 2:29:15
  models, we have VIP sets defined where, you know, you can Define
• 2:29:20
  if you just have a large test set.
• 2:29:22
  That's not enough. What we find.
• 2:29:23
  We need like sophisticated VIP sets for different.
• 2:29:27
  Aw, and then we Q8 them and grow them over the time of the
• 2:29:30
  product. So, over the years, we have
• 2:29:33
  hundreds of thousands of examples where we have been
• 2:29:36
  failing in the past that we have curated.
• 2:29:38
  And so we for any new model we test against the enter history
• 2:29:42
  of these barriers and then keep adding to this test set on top
• 2:29:46
  of this. We have Shadow modes where we
• 2:29:48
  ship these models in silent to the car and we get data back on
• 2:29:51
  where they're failing or succeeding and there's extensive
• 2:29:55
  QA program. It's very hard to ship a
• 2:29:58
  regression. There's like nine levels of
• 2:30:00
  filters before it hits customers, but then we have
• 2:30:03
  really good infra to make this all efficient.
• 2:30:07
  I'm one of the curators, so I QA the car.
• 2:30:10
  Yeah, like, create a stir. Yeah, so I'm constantly in the
• 2:30:15
  car just being queuing, like, whatever.
• 2:30:17
  The latest alpha build is that doesn't totally crash finds a
• 2:30:22
  lot of bugs. Hi, great event.
• 2:30:27
  I have a question about foundational models for
• 2:30:31
  autonomous driving. We have all seen that big models
• 2:30:35
  that really can when your skill up with data and model
• 2:30:38
  parameter, right? From GT3 to Palm it, can
• 2:30:42
  actually. Now, do reasoning, do you see
• 2:30:44
  that is essential Skilling up foundational models with data
• 2:30:49
  and size? And then, at least you can get a
• 2:30:52
  teacher model, right? That potentially can All the
• 2:30:56
  problems and then you this tale to a student model is that how
• 2:31:00
  you see foundational models? Relevant 407.
• 2:31:04
  That's quite similar to our Auto labeling model so we don't just
• 2:31:07
  have models that run in the car. We train models that are
• 2:31:10
  entirely offline. There are extremely large that
• 2:31:13
  can't run in real time on the car.
• 2:31:15
  So we just run those offline on the server's.
• 2:31:17
  Producing. Really good labels that can then
• 2:31:21
  train the online networks. So that's one form distillation
• 2:31:24
  of these. Teacher student models, kind of
• 2:31:28
  the foundation models. We are building some really,
• 2:31:30
  really large data sets that you know, are multiple petabytes.
• 2:31:34
  And we are seeing that some of these tasks work really well
• 2:31:37
  when we have this large data sets like a Mattox.
• 2:31:39
  Like I mentioned we do in all the kinematics out of all the
• 2:31:42
  objects and up to the fourth derivative and people thought we
• 2:31:46
  couldn't do detection, IT cameras, detection depth,
• 2:31:48
  velocity acceleration. And imagine how precise this
• 2:31:52
  have to be for the signal higher order, derivatives to be
• 2:31:54
  accurate. And this all comes from these
• 2:31:57
  kind of large data sets and large models. so we're seeing
• 2:32:00
  the equivalent of foundation models in our own way for
• 2:32:03
  geometry and kinematics, and things like those You want to
• 2:32:08
  add anything job? Yeah, I'll keep it brief
• 2:32:11
  basically. Whenever we train on a larger
• 2:32:13
  data set, we see Bigfoot. Okay.
• 2:32:17
  Basically, whenever we train on a larger dataset, we see big
• 2:32:19
  improvements in our model performance, and basically,
• 2:32:22
  whenever we initialize our networks with, you know, some
• 2:32:24
  pre-training step from some other auxiliary task, we
• 2:32:27
  basically see improvements the self supervised door, supervised
• 2:32:30
  with large data sets, both helped a lot.
• 2:32:35
  Hi. So at the beginning Ilan said
• 2:32:38
  that Tesla was potentially interested in building
• 2:32:40
  artificial general intelligence systems given the potentially
• 2:32:44
  transformative impact of technology like that.
• 2:32:46
  It seems prudent to invest in technical AGI, safety expertise
• 2:32:51
  specifically. I know Tesla does a lot of
• 2:32:54
  technical narrow AI Safety Research I was curious if Tessa
• 2:32:58
  was intending to try to build expertise in technical
• 2:33:02
  artificial, general intelligence safety specific Lee.
• 2:33:06
  Well, if I mean, if we're starts looking like we're getting be
• 2:33:10
  making a significant contribution to artificial
• 2:33:13
  general intelligence and then we'll for sure.
• 2:33:15
  Invest in in safety, I'm a big believer in AI safety.
• 2:33:19
  I think there should be an AI sort of regulatory Authority at
• 2:33:24
  at the government level. Just as there is a regulatory
• 2:33:27
  Authority for anything that affects Public Safety.
• 2:33:30
  So we have regulatory Authority for aircraft and cars and sort
• 2:33:35
  of Food and Drugs. Yes.
• 2:33:36
  And because they affect Public Safety and AI also affects
• 2:33:40
  Public Safety. So I think and this is not
• 2:33:43
  really something that government I think understands yet, but I
• 2:33:46
  think I think there should be a referee that is ensuring or
• 2:33:50
  doing trying to ensure Public Safety for AGI.
• 2:33:57
  Anything of like, well what are the elements that are necessary
• 2:34:00
  to create a GI like the accessible data set is extremely
• 2:34:07
  important and if you've got a large number of cars and
• 2:34:12
  humanoid, robots processing, petabytes of video data and
• 2:34:20
  audio data from The Real World. Just like humans that that's
• 2:34:25
  that might be the biggest data set.
• 2:34:26
  Probably. The biggest data set because in
• 2:34:30
  addition to that you can obviously incrementally scan the
• 2:34:33
  internet. But what the internet can't
• 2:34:36
  quite do is have millions or hundreds of millions of cameras
• 2:34:40
  in the real world. And with like said with audio
• 2:34:44
  and and other senses as well. So I think we're probably will
• 2:34:49
  have the most amount of data and probably the most amount of
• 2:34:55
  training power. Therefore probably we will make
• 2:35:00
  a contribution to a GI. Hey, I noticed the semi was back
• 2:35:09
  there, but we haven't talked about it too much.
• 2:35:11
  I was just wondering for the semi truck.
• 2:35:13
  What are the changes? Your thinking, about?
• 2:35:15
  From a sensing perspective. Imagine there's very different
• 2:35:18
  requirements, obviously than just a car if and if you don't
• 2:35:22
  think that's true. Why is that true?
• 2:35:24
  No, I think basically you can drive a car.
• 2:35:28
  I mean think about what drives any vehicle.
• 2:35:30
  It's a biological neural net with with eyes were Essentially.
• 2:35:36
  So, if and really a, what is your primary sensors, are two
• 2:35:44
  cameras, on a slow gimbal, a very slow gimbal.
• 2:35:48
  That's, that's your head. So if, you know of biological
• 2:35:53
  neural net with with two cameras on a slow, gimbal can drive a
• 2:35:56
  semi truck then. If you've got like eight cameras
• 2:36:00
  with continuous 360-degree Vision, operating at a higher
• 2:36:04
  frame rate and much higher reaction, Run rate than I think
• 2:36:06
  it is obvious that you should be able to drive a semi or any of
• 2:36:09
  any vehicle much better than human.
• 2:36:14
  Hi, my name is Akshay. Thank you for the even assuming,
• 2:36:19
  you know, Optimus would be used for different use cases and
• 2:36:22
  would evolve at different piece. For these use cases, would it be
• 2:36:27
  possible to sort of develop and deploy different software and
• 2:36:31
  Hardware components independently and deploy them?
• 2:36:35
  You know in the in optimist so that the overall you know
• 2:36:40
  feature development is faster for optimism.
• 2:36:48
  Okay. All right.
• 2:36:49
  What we did not comprehend. Unfortunately, our neural net to
• 2:36:53
  not comprehend the question. So next question, Hi.
• 2:37:02
  I want to switch the gear to the autopilot.
• 2:37:04
  So when you guys plan to roll out the FST beta, two countries
• 2:37:09
  other than the US and Canada. And also my next question is
• 2:37:13
  what's the biggest bottleneck or the technology Co barrier using
• 2:37:16
  in the current or the power of the stack and how you envision
• 2:37:19
  to solve that to make the autopilot is considerably better
• 2:37:23
  than human in terms of web performance Matrix X safety
• 2:37:26
  assurance and the human component is I think you Also,
• 2:37:30
  mention 4V fsdb wherever you are, guys going to combine the
• 2:37:34
  highway and the city is a single stack and some architectural big
• 2:37:38
  Improvement, can you maybe explain a bit about on that?
• 2:37:41
  Thank you. Well, that's a whole bunch of
• 2:37:43
  questions, will we? We're hopeful to be able to, I
• 2:37:49
  think from a technical standpoint FSD beta should be,
• 2:37:53
  it should be possible to roll on SF is T beta worldwide by the
• 2:37:57
  end of this year. But we, you know, for a lot of
• 2:38:02
  countries we need regulatory approval.
• 2:38:05
  And so we are somewhat gated by the regulatory approval in other
• 2:38:08
  countries. But you know, but I think from a
• 2:38:14
  technical standpoint, it will be ready to go to a worldwide beta
• 2:38:19
  by the end of this year and there's quite a big Improvement
• 2:38:22
  that we're expecting to release next month.
• 2:38:25
  That will always be especially good at assessing the velocity
• 2:38:31
  of fast-moving, Crush traffic and a bunch of other things.
• 2:38:34
  So anyway, elaborate Yeah guess so there used to be a lot of
• 2:38:42
  differences between production autopilot and the pole,
• 2:38:45
  self-driving beta. But those differences have been
• 2:38:47
  getting smaller and smaller over time I think just a few months
• 2:38:50
  ago we now use the same vision. Only object in stack in both FSD
• 2:38:55
  and in the production autopilot on all vehicles there's still a
• 2:38:59
  few differences. The primary one being the way
• 2:39:01
  that we predict Lanes right now. So we upgraded the modeling of
• 2:39:04
  Lane so that I could handle these more complex.
• 2:39:06
  Geometries like I mentioned in the talk, In production on a pie
• 2:39:09
  that we still use a simpler lane model but we're extending our
• 2:39:13
  current Deputy beta models to work in all sort of Highway
• 2:39:17
  scenarios as well. Yeah.
• 2:39:19
  And the the version of FST better that I drive actually
• 2:39:22
  does have the integrated stack. So this uses the FST, stack both
• 2:39:27
  the city streets and Highway and works quite well for me.
• 2:39:32
  We need to validate it in all kinds of weather like heavy
• 2:39:35
  rains snow dust and just make sure it's working as better than
• 2:39:42
  the production stack in across a wide range of environments, but
• 2:39:48
  it was pretty close. ooh that I mean I think it's, I don't know,
• 2:39:52
  maybe Definitely be before the end of the year and maybe
• 2:39:57
  November. Yeah.
• 2:39:59
  In our personal drives, the FST stack on Highway drives, already
• 2:40:02
  way. Better than the production stack
• 2:40:03
  we have and we do expect to also include the parking lot stack as
• 2:40:08
  a part of the FSC stack before the end of this year.
• 2:40:11
  So that will basically bring us to you, sit in the car in the
• 2:40:15
  parking lot and drive till the end of the parking lot and The
• 2:40:17
  Parking Spot before the end of this year.
• 2:40:19
  And in terms of like the fundamental, the fundamental
• 2:40:23
  metric to optimize against is how many miles per hour in
• 2:40:27
  between in necessary intervention.
• 2:40:29
  So just massively improving the, how many miles the car can drive
• 2:40:36
  on in full autonomy before and intervention is required.
• 2:40:39
  That is safety critical. So, yeah, that's that's the
• 2:40:46
  fundamental metric that we're measuring every week and we're
• 2:40:50
  making radical improvements on that.
• 2:40:55
  Hi. Thank you.
• 2:40:57
  Thank you so much for the presentation, very inspiring.
• 2:41:00
  My name is Daisy. I actually have a non technical
• 2:41:03
  question for you. I'm curious, if you are back to
• 2:41:06
  your twenties, what are some of the things you wish you knew
• 2:41:10
  back then? What are some advice?
• 2:41:12
  You would give to your younger self?
• 2:41:25
  Well, I'm trying to figure out something useful.
• 2:41:28
  To say. Yeah.
• 2:41:32
  Yeah. Joint Tesla will do one thing.
• 2:41:38
  Yeah, I think just going to try to expose yourself to as many
• 2:41:43
  smart people as possible. And I read a lot of books.
• 2:41:52
  You know, I do that. Did do that though.
• 2:41:55
  So I think there's some Merit to just also like not being like
• 2:42:03
  necessarily too intense and and like enjoying the moment a bit
• 2:42:09
  more. I would say two twenty or twenty
• 2:42:11
  something me just a you know stop and smell.
• 2:42:15
  The roses occasionally would probably be a good idea, you
• 2:42:20
  know? It's like when we were
• 2:42:22
  developing the the Falcon 1 rocket And on the kwajalein
• 2:42:28
  atoll and we have this beautiful little island that we're
• 2:42:31
  developing the rocket on and not.
• 2:42:33
  Once did that during that entire time that I even have a drink on
• 2:42:36
  the beach? I'm like well I should have had
• 2:42:38
  a drink on the beach that would have been fine.
• 2:42:44
  Thank you very much. I think you have excited all of
• 2:42:47
  the robotics people with with Optimus.
• 2:42:50
  This feels very much like 10 years ago in driving but as
• 2:42:55
  driving has proved to be harder than it actually look 10 years
• 2:42:58
  ago. What do we know now that we
• 2:43:00
  didn't ten years ago? That would make for example, AGI
• 2:43:03
  on a humanoid come faster Well, I mean it it seems to me that
• 2:43:09
  HEI is advancing very quickly hardly a week goes by without
• 2:43:15
  some significant announcement. and yeah, I mean This point.
• 2:43:23
  Like AI seems to be able to win at almost any rule, based game.
• 2:43:29
  It's able to create extremely impressive art.
• 2:43:36
  Engage in conversations that are very sophisticated you know
• 2:43:43
  write essays and these these just keep improving and there's
• 2:43:50
  so much more so many more talented people working on AI.
• 2:43:55
  And the hardware is getting better.
• 2:43:56
  I think it's a, a eyes on a super like a strong exponential
• 2:44:01
  curve of improvements, independent of what we do at
• 2:44:05
  Tesla and obviously will benefit somewhat from that exponential
• 2:44:10
  curve of improvement. With a, i Excessive just also
• 2:44:16
  happens to be very good at actuators that Motors Motors
• 2:44:19
  gearboxes controllers, Power Electronics, batteries sensors.
• 2:44:25
  And you know, really like I say that, you know, the biggest
• 2:44:29
  difference between the robot on four wheels and the robot with
• 2:44:33
  arms and legs is is getting the actuators, right?
• 2:44:37
  Actually, it's an actuators and sensors problem.
• 2:44:41
  And obviously you know how you control those actuators and
• 2:44:44
  sensors but it's a yeah, actuators and sensors and how
• 2:44:49
  you control the actuators? It's I don't.
• 2:44:52
  We have to have like the ingredients necessary to create
• 2:44:54
  a compelling robot and we're doing it.
• 2:44:56
  So Hi Lon you are actually bringing the humanity to the
• 2:45:05
  next level literally Tesla and you are bringing the humanity,
• 2:45:08
  the next level. So you said Optimus Prime
• 2:45:12
  Optimus will be used in next Tesla Factory.
• 2:45:15
  My question is, will a new Tesla Factory will be fully run by
• 2:45:20
  Optimus program? and and when can general public
• 2:45:26
  order humanoid, Yeah, I think it'll, it'll we're going to
• 2:45:30
  start Optimist with very simple tests in the factory.
• 2:45:35
  You know, like maybe just like loading apart like you saw in
• 2:45:37
  the video loading apart. You know, carrying a pot from
• 2:45:42
  one place to another or loading apart into a one of our more
• 2:45:46
  conventional robot cells to, you know, that that world's body
• 2:45:52
  together. So we'll start you know just
• 2:45:55
  trying to how do we make it useful at all and then and then
• 2:45:58
  gradually expand the number of situations where it's useful.
• 2:46:03
  And I think that that whatever situation is where Optimus is
• 2:46:07
  useful will grow exponentially. Like really, really fast in
• 2:46:14
  terms of when people can order one.
• 2:46:17
  I don't know. I think it's not that far away.
• 2:46:20
  Well, I think you mean, when can people receive one?
• 2:46:25
  So, I don't know. I'm like, I'd say probably
• 2:46:29
  within three years, not more than five years went within
• 2:46:33
  three to five years. You could probably receive an
• 2:46:35
  optimist, I feel the best way to make the progress for a gi's to
• 2:46:44
  involve as many smart people across the water as possible,
• 2:46:47
  and given the size and resource of Tesla compared to robot
• 2:46:51
  companies and given the state of human research at the moment,
• 2:46:55
  wouldn't make sense for the kind of Tesla to sort of Open Source
• 2:46:59
  from of the simulation? How do our parts I think Tesla
• 2:47:03
  can still be the dominant platformer, where it can be
• 2:47:05
  something like an Android OS or like iOS stuff for the entire
• 2:47:10
  human research with Would that be something that rather than
• 2:47:13
  keeping The Optimist to just Tesla researchers or the factory
• 2:47:17
  itself can open it and let the whole world exploding, my
• 2:47:20
  research? I think we have to be careful
• 2:47:29
  about Optimus being potentially used in ways that are bad
• 2:47:34
  because that is one of the possible things to do.
• 2:47:37
  So, I think we're, you know, we're provide optimists where
• 2:47:45
  you can provide instructions to Optimist but where those
• 2:47:48
  instructions are, you know, governed by some laws of
• 2:47:52
  robotics that you cannot overcome.
• 2:47:58
  So, you know, not doing harm to others and Without I think
• 2:48:05
  probably have quite a few safety related things with Optimus.
• 2:48:09
  Yeah. So I wrote it will just take
• 2:48:12
  maybe a few more questions and then and then and then thank you
• 2:48:14
  all for coming. Questions one deep and one broad
• 2:48:21
  on the Deep for Optimus. What's the current and what's
• 2:48:24
  the ideal controller bandwidth? And then in the broader question
• 2:48:29
  there's this big advertisement for the depth and breadth of the
• 2:48:32
  company. What is it uniquely about Tesla
• 2:48:36
  that enables that? Anyone want to tackle the
• 2:48:40
  bandwidth question? So the technical bandwidth of
• 2:48:45
  the close to your mouth, okay? For the bandwidth question, you
• 2:48:49
  have to understand or figure out, what is the task that you
• 2:48:52
  wanted to do and what is the free if you took a frequency
• 2:48:56
  transform of that task? What is it that you want your
• 2:48:58
  limbs to do? And that's why you get your
• 2:49:00
  bandwidth from. It's not a number that you can
• 2:49:02
  specifically, just say, you need to understand your use case.
• 2:49:04
  And that's from, that's where the bandwidth comes from.
• 2:49:08
  What are the broad question? I don't.
• 2:49:23
  I researched on the band. The question I think we're
• 2:49:25
  probably will just end up increasing the bandwidth or
• 2:49:28
  rear, you know, which translates to the effect of dexterity and
• 2:49:34
  reaction time of the, of the robot, like, you get it safe
• 2:49:38
  safe. It's not one hurts and it's
• 2:49:42
  maybe you don't need to go all the way to 100 Hertz, but I do
• 2:49:46
  maybe 10, 20, 50, I don't know it.
• 2:49:48
  But over time, I think the bandwidth will increase quite a
• 2:49:52
  bit. Or translated to dexterity and
• 2:49:55
  latency. You'd want to minimize that over
• 2:49:59
  time. Yeah.
• 2:50:02
  Minimize latency maximize dexterity in terms of breadth
• 2:50:07
  and depth. I guess we're we've got, we're
• 2:50:11
  pretty big company at this point.
• 2:50:13
  So we've got a lot of different areas of expertise that we
• 2:50:15
  necessarily had to develop in order to make autonomous or in
• 2:50:18
  order to make electric cars and then order to make autonomous
• 2:50:21
  electric cars. So we've just Tesla is like a
• 2:50:26
  whole series of startups basically and so far, they've
• 2:50:32
  almost all been quite successful.
• 2:50:35
  So we must be doing something right.
• 2:50:37
  And I, you know, I consider one of my core responsibilities
• 2:50:42
  writing company is to have an environment where great
• 2:50:46
  Engineers can flourish and I think in a lot of companies on a
• 2:50:51
  maybe most companies, if somebody's a really talented
• 2:50:55
  driven engineer, the they're unable to actually their talents
• 2:51:01
  are suppressed affect a lot of companies and it's, you know,
• 2:51:06
  and some of the companies that the engineering Talent is
• 2:51:09
  suppressed in a way that is, maybe not obviously bad but
• 2:51:13
  where it's just so comfortable and you paid so much money and
• 2:51:17
  you, but your The output you actually have to produce is so
• 2:51:21
  low. That is like a Honey Trap, you
• 2:51:23
  know, so like there's a few Honey Trap.
• 2:51:26
  It's places in Silicon Valley, where they don't like
• 2:51:28
  necessarily don't seem like bad places for engineers but he have
• 2:51:32
  to say like a good engineer went in and what did they get out and
• 2:51:37
  the output of that? Engineering Talent is seems very
• 2:51:41
  low even though there seem to be enjoying themselves that's why I
• 2:51:47
  call it the the few Honey Trap companies in Silicon Valley,
• 2:51:50
  Tesla is not a Honey Trap. We're demanding and it's like
• 2:51:54
  going to get a lot of shit done and it's going to be really
• 2:51:57
  cool. And it's not going to be easy,
• 2:52:02
  but if you are a super talented engineer, your talents will be
• 2:52:10
  used. I think to, a greater degree
• 2:52:14
  than anywhere else. You know, SpaceX also that way.
• 2:52:19
  So Highland of I have two questions, so both to the
• 2:52:26
  autopilot team. So the thing is like I have been
• 2:52:28
  following your progress for the past few years.
• 2:52:30
  So today, you have made changes on like the lane detection, like
• 2:52:34
  you said that like previous, you're doing instance, somatic
• 2:52:36
  segmentation. Now you guys are built transfer
• 2:52:38
  models for like building the lanes.
• 2:52:41
  So what are another some other common challenges which you guys
• 2:52:44
  are facing right now, like which you are solving in future as a
• 2:52:47
  curious Engineers. So that like we as a researcher
• 2:52:50
  can work on those, Start working on those.
• 2:52:52
  And the second question is like, I'm really curious about the
• 2:52:54
  data engine. Like you guys have like stole a
• 2:52:58
  case, like why the car is stopped.
• 2:53:00
  So how are you finding cases, which is very much similar to
• 2:53:03
  that from the data, which you have like.
• 2:53:05
  So little bit more on the data engine would be great.
• 2:53:07
  So that's it for our star answer.
• 2:53:11
  The first question using occupancy that work as an
• 2:53:13
  example. So what you saw in the
• 2:53:17
  presentation did not exist a year ago.
• 2:53:19
  So we only spent one year About time, we're shipping, wouldn't
• 2:53:22
  Health occupancy Network and Q. Have a One Foundation model,
• 2:53:27
  actually to represent the entire physical world, around
• 2:53:31
  everywhere. And you always a condition is
• 2:53:34
  actually really, really challenging.
• 2:53:35
  So, only over the year ago, we're kind of like driving a 2d
• 2:53:39
  were if there's a war and it uses curb what kind of represent
• 2:53:43
  with the same static Edge, which is obviously, you know, not not
• 2:53:47
  ideal, right? There's a big difference between
• 2:53:49
  a curb and War when you drive you Make different choices,
• 2:53:52
  right? So after we realized that, what
• 2:53:54
  we go to 3D will have you basically rethink the entire
• 2:53:57
  problem and think about how we address that.
• 2:54:00
  So this will be like one example of a challenges, we have a web
• 2:54:04
  of conquer in the past year, Yeah, to answer the question
• 2:54:10
  about how we actually source examples of this tricky stopped
• 2:54:13
  cars. There's a few ways to go about
• 2:54:15
  this but two examples are one we can trigger for disagreements
• 2:54:19
  within our signals. So let's say that parked bit
• 2:54:22
  flickers between parked and driving will trigger that back.
• 2:54:26
  And the second is, we can leverage more of the Shadow mode
• 2:54:28
  logic. So if the customer ignores the
• 2:54:30
  car, but we think we should stop for it will get that data back
• 2:54:34
  to. So these are just different like
• 2:54:36
  various trigger logic that allows us to get those data.
• 2:54:39
  And pains back. Hi, thank you for the amazing
• 2:54:46
  presentation. Thanks so much.
• 2:54:48
  So there are a lot of companies that are focusing on the AGI
• 2:54:52
  problem, and one of the reasons why such a hard problem is
• 2:54:55
  because the problem itself is so hard to Define, several
• 2:54:58
  companies have several different definitions, they focus on
• 2:55:01
  different things. So, what is Tesla house, tester,
• 2:55:03
  defining, the AGI problem. And what are you focusing on
• 2:55:06
  specifically? Well, we're not actually
• 2:55:11
  specifically focused on a GI, I'm simply saying that AGI is so
• 2:55:15
  is seems likely to be an emergent property of what we're
• 2:55:19
  doing because we're creating all these autonomous cars, and
• 2:55:24
  autonomous humanoids that are actually with a truly gigantic
• 2:55:32
  data stream. That's coming in and being
• 2:55:34
  processed. It's by far the most amount of
• 2:55:38
  real-world data. And they do you can't get by
• 2:55:41
  just searching the internet because you have to be out there
• 2:55:43
  in the world and interacting with people and interacting with
• 2:55:46
  the roads. And, and just, you know, Earth
• 2:55:50
  is big place and reality is messy and complicated.
• 2:55:53
  So, so, I think it's sort of like, like you to just, it just
• 2:55:58
  seems likely to be an emergent property of if you've got, you
• 2:56:02
  know, tens or hundreds of millions of autonomous vehicles
• 2:56:04
  and, and maybe even a comparable number of humanoids, maybe more
• 2:56:08
  than that on the human heart. Run.
• 2:56:10
  Well, that's just the most amount of data and if that that
• 2:56:14
  videos being processed, it just seems likely that you know, the
• 2:56:19
  cars will will definitely get way better than human drivers
• 2:56:23
  and the humanoid robots will become increasingly
• 2:56:29
  indistinguishable from humans perhaps. and so then, like I
• 2:56:35
  said, you have a Emergent property of AGI.
• 2:56:45
  I believe you know, humans collectively are sort of a super
• 2:56:49
  intelligence as well especially as we improved the data rate
• 2:56:53
  between humans. Anything like that seem to be
• 2:56:55
  way back in the early days of the internet was like the
• 2:56:58
  internet was like Humanity, acquiring a nervous system where
• 2:57:03
  now all of a sudden any one element of humanity could know
• 2:57:07
  all of the knowledge of humans by connecting to the internet,
• 2:57:11
  almost all the knowledge on suddenly, here's part of it,
• 2:57:13
  whereas Ously, we would exchange information by osmosis it by you
• 2:57:18
  know, by would have liked in order to transfer data.
• 2:57:21
  So you would have to write a letter somewhere, we have to
• 2:57:23
  carry the letter by person to another person and then a whole
• 2:57:27
  bunch of things in between and then it was like yeah I mean
• 2:57:34
  insanely slow when you think about it and even if you were in
• 2:57:38
  the Library of Congress you still didn't have access to all
• 2:57:40
  the world's information and you certainly couldn't search it.
• 2:57:44
  And obviously very few people are in the Library of Congress.
• 2:57:48
  So I mean, one of the Great I sort of equality elements like
• 2:57:57
  the internet is has been the most, the biggest equalizer in
• 2:58:00
  history in terms of access to information and knowledge.
• 2:58:06
  And any student of History I think would agree with this
• 2:58:09
  because, you know, you go back a thousand years.
• 2:58:11
  There were very few books like like and books would be
• 2:58:14
  incredibly expensive but only a few people knew how to read and
• 2:58:17
  only if an even smaller number of people even had a book now.
• 2:58:22
  Now look at it like you can Access any book instantly?
• 2:58:25
  You can learn anything for basically, for free is pretty
• 2:58:29
  incredible. So, you know, I was asked
• 2:58:36
  recently, what period of History would?
• 2:58:40
  I prefer to be at the most? And my answer was right now.
• 2:58:46
  This is the most interesting time in history and I read a lot
• 2:58:49
  of history. So let's let's do our best to
• 2:58:53
  keep that going. Yeah.
• 2:58:57
  And to go back to one of the earlier questions.
• 2:58:58
  I would ask her like you can the the thing that's happened over
• 2:59:02
  time with respect to Tesla autopilot is that we've just the
• 2:59:09
  The neural Nets have gotten have gradually absorbed more, and
• 2:59:13
  more software. And in the limit, of course, you
• 2:59:16
  could simply take the videos as seen by the car and compare
• 2:59:22
  those to the the steering inputs from the steering wheel and
• 2:59:25
  pedals which very simple inputs. And it in principle, you could
• 2:59:30
  train with nothing in between because that's what humans are
• 2:59:34
  doing with the biological neural, net, you could train
• 2:59:37
  based on video and And the, what trains the video is the moving
• 2:59:43
  of the steering wheel and the pedals with no other software in
• 2:59:46
  between, in between, we're not there yet, but it's gradually
• 2:59:50
  going in that direction. All right, but we will last
• 2:59:56
  question. I think we got a crush on the
• 3:00:01
  front here. They're right there.
• 3:00:04
  I will do two questions. Fine.
• 3:00:08
  Thanks for such a great presentation.
• 3:00:10
  Well, the old question last with FSD being used by so many people
• 3:00:16
  do you think, what's the come, how do you evaluate the
• 3:00:18
  company's risk tolerance in terms of performance statistics
• 3:00:21
  and do you think there needs to be more transparency or
• 3:00:23
  regulation from third parties as to how what's good enough and
• 3:00:28
  defining, like Thresholds for performance across some many
• 3:00:33
  miles. Sure.
• 3:00:35
  We'll the, you know, the number one design requirement at Tesla
• 3:00:43
  is safety so that it goes across the board.
• 3:00:47
  So in terms of the mechanical, safety of the car, we have the
• 3:00:51
  lowest probability of injury of any cars ever tested by the
• 3:00:53
  government for just passive mechanical safety essentially,
• 3:00:58
  crash structure and airbags or whatnot.
• 3:01:03
  We have the best Best highest rating for active safety, as
• 3:01:07
  well. And is going to get to the point
• 3:01:12
  where your active safety is. So ridiculously good, it's like
• 3:01:16
  just absolutely better than human.
• 3:01:20
  And then with respect to autopilot, we do publish this.
• 3:01:24
  Broadly speaking the statistics on miles driven or with cars
• 3:01:30
  that have no autonomy or Tesla cars with no autonomy with Kind
• 3:01:34
  of Hardware one Hardware to Hardware 3 and then the ones
• 3:01:39
  that are in FST beta. And we see steady improvements
• 3:01:43
  all along the way. And, you know, sometimes there's
• 3:01:46
  this dichotomy of, you know, should you wait until the car is
• 3:01:52
  like on Earth, three times safer than a person before deploying
• 3:01:56
  any technology? But I think that's that's
• 3:01:58
  actually morally wrong at the point of which he believed that
• 3:02:02
  that adding autonomy. Deuces, injury and death.
• 3:02:09
  I think you have a moral obligation to deploy it, even
• 3:02:12
  though you're going to get sued and blamed by a lot of people,
• 3:02:16
  because the people whose lives you saved, don't know that their
• 3:02:19
  lives are saved. And the people, the people who
• 3:02:23
  do occasionally die or get injured, they definitely know or
• 3:02:26
  their state does that it was, you know, whatever.
• 3:02:29
  There's a problem with with, with autopilot.
• 3:02:32
  That's why you have to look at the The numbers, in terms of
• 3:02:35
  total miles driven, how many accidents occurred?
• 3:02:38
  How many accidents were serious? How many fatalities and we've
• 3:02:41
  got oh well over three million cars in the road.
• 3:02:43
  So this it's that's a lot of miles driven every day.
• 3:02:47
  It's not going to be perfect but what what matters it is that is
• 3:02:50
  that it is very clearly safer than not deploying it.
• 3:02:56
  Yeah. So last question, Thanks for the
• 3:03:06
  last question here. Okay.
• 3:03:14
  Hi. So I do not work on Hardware, so
• 3:03:19
  maybe the hardware dream and you guys can enlighten me, why is it
• 3:03:24
  required that there be symmetry in the design of optimist?
• 3:03:29
  Because humans, we have handedness, right?
• 3:03:33
  We are, we use some set of muscles more than others or
• 3:03:37
  time. There's wear and tear, right?
• 3:03:39
  So maybe you'll start. You see some joint failures or
• 3:03:42
  some actuator failures. More over time, I understand
• 3:03:47
  that this is extremely pre-staged.
• 3:03:49
  Also, we, as humans have based so much fantasy and fiction over
• 3:03:56
  super human capabilities, like all of us.
• 3:03:58
  Don't want to walk right over there.
• 3:04:00
  We want to extend our arms and like we have all these, you
• 3:04:04
  know, a lot of fantasy, Fantastical designs so
• 3:04:08
  considering everything else that is going on.
• 3:04:11
  Terms of batteries and intensity of compute.
• 3:04:15
  Maybe you can leverage all those aspects into coming up with
• 3:04:20
  something. Well, I don't know more
• 3:04:23
  interesting in terms of your the robot that you're building and
• 3:04:27
  I'm hoping you're able to explore those directions.
• 3:04:31
  Yeah. I mean I think it would be cool
• 3:04:32
  to have like, you know, make Inspector Gadget real.
• 3:04:35
  That would be pretty sweet. So yeah, I mean, right now we're
• 3:04:40
  just want to make Make basic humanoid what work well and our
• 3:04:44
  goal is fastest path to a useful humanoid robot.
• 3:04:48
  I think this is this will ground Us in reality literally and
• 3:04:53
  ensure that we are doing something useful.
• 3:04:57
  Like one of the hardest things to do is to be useful to
• 3:05:02
  actually and then and then to have high utility under the
• 3:05:05
  curve of like, how many people did you help taught?
• 3:05:08
  You know and how much help did you Provide to each person on
• 3:05:14
  average. And then how many people did you
• 3:05:15
  help the total utility like trying to actually ship useful
• 3:05:20
  product? That people like to a large
• 3:05:23
  number of people is so insanely hard.
• 3:05:26
  I it's boggles the mind, you know, it's like I said, like
• 3:05:29
  man, there's a hell of a difference between a company
• 3:05:31
  that has shipped product and one has not sure product.
• 3:05:34
  It's a game. This is night and day.
• 3:05:36
  And then even once you ship product, can you make the cost?
• 3:05:39
  The value of the output? Worth more than the cost of the
• 3:05:42
  input, which is again, insanely difficult, especially with
• 3:05:45
  Hardware, so. But I think over time, I think
• 3:05:49
  we cool to do creative things and have like eight arms and
• 3:05:53
  whatever and have different versions and maybe, you know,
• 3:05:58
  they'll be some Hardware like companies that are able to add
• 3:06:03
  things to an optimist like maybe we, you know, add a power port,
• 3:06:08
  or something like that or attached.
• 3:06:10
  Then you can add You know, add attachments to your Optimist,
• 3:06:13
  like you can add them to your phone, it could be.
• 3:06:16
  So a lot of cool things that could be done over time and it
• 3:06:18
  could be, maybe an ecosystem of small companies that or big
• 3:06:21
  companies that make add-ons for Optimus.
• 3:06:25
  So with that, I'd like to thank the team for their hard work.
• 3:06:31
  You guys are awesome and And I think and I thank you all for
• 3:06:40
  coming and for everyone online. Thanks for tuning in and I think
• 3:06:45
  this will be one of those great videos where you can.
• 3:06:47
  Like if you you can fast-forward to the bits that you find most
• 3:06:50
  interesting. But we try to give you a
• 3:06:53
  tremendous amount of detail literally.
• 3:06:56
  So you can look at the video at your leisure and you can focus
• 3:07:00
  on the parts that you find interesting and skip the other
• 3:07:02
  parts. So thank you all.
• 3:07:05
  And we'll do this to try to do this every year and we might do
• 3:07:08
  a part, a monthly podcast even so, but I think it'd be, you
• 3:07:12
  know, great to sort of bring you along for the ride and and like
• 3:07:17
  show you what cool things are happening.
• 3:07:20
  And yeah, thank you. All right, thanks.
• 3:07:33
  That gets run, Rashad JCPenney for thousands of deals solo.
• 3:07:36
  No coupons needed this weekend, save up to 50% on kitchen
• 3:07:40
  electrics from Brands like, Keurig and Cuisinart, say yes,
• 3:07:43
  please to Diamonds and gemstones.
• 3:07:44
  Now, 1999 each and bundle up the famine coats starting at 1499.
• 3:07:49
  We got your holiday. A valid on select items. 1118 3,
• 3:07:56
  1120, excluded from coupons exclusions apply C Storage
• 3:07:59
  acb.com for details