Latest / In the Loop AI / We Stepped Inside a Memory from 2005 (Apple SHARP Gaussian Splats)

Transcript

We Stepped Inside a Memory from 2005 (Apple SHARP Gaussian Splats) · In the Loop AI

0:00
Josh: Ejaz, I've been obsessed with these things called splats for like six months.
0:03
Josh: And just last week, we had a new breakthrough in this revolutionary weird world of 3D imagery.
0:09
Josh: So I want to start with this image that we're looking at on screen right now,
0:11
Josh: because this is, it's pretty unbelievable.
0:14
Josh: And for the people who are just listening, what it is, is there's a dude with
0:18
Josh: the normal backyard and he has a Vision Pro and he has a photo that he took
0:22
Josh: in the past of the same backyard in a different season.
0:25
Josh: And with this new Splat technology, he's able to overlay a previous picture
0:29
Josh: onto a current place and then actually walk through it as if he's able to relive
0:34
Josh: the memory for the first time using these Apple Vision Pro headsets.
0:37
Josh: So it's this unbelievable technology that Apple released just last week that
0:41
Josh: allows you to relive the past in the present in a way that is totally immersive,
0:47
Josh: totally submersive using these things like the Vision Pro or any sort of virtual reality headset.
0:52
Josh: The new technology has some pretty unbelievable examples. So that's what we're
0:55
Josh: going to cover in this episode is the weird world of 3D splats and how you're
0:59
Josh: actually able to turn any photo you've ever taken, any video you've ever taken
1:03
Josh: into something you could actually step into and relive again like it's the first time.
1:07
Ejaaz: Yeah, it's pretty crazy. It looks like he's taken a picture of his garden from
1:11
Ejaaz: December 10th, 2017, when it
1:14
Ejaaz: was snowing and he's kind of transposed it onto his garden in real time.
1:18
Ejaaz: I think the thing that shocked me the most from this, Josh, I did a double
1:20
Ejaaz: take was they have a doctor who prop in
1:23
Ejaaz: here it's like a telephone box who has
1:26
Ejaaz: this in their gun that this is pretty insane um but i
1:29
Ejaaz: saw a more recent example actually of this technology
1:32
Ejaaz: this week um in a slightly uh
1:35
Ejaaz: uh crazy application so uh
1:38
Ejaaz: you might have heard that uh some some epstein files got released uh and some
1:42
Ejaaz: documented footage pictures videography got released and someone decided to
1:47
Ejaaz: splat the entire thing so what you're looking at here isn't a real uh video
1:51
Ejaaz: but rather a series of images which have been splattered to form this kind of
1:55
Ejaaz: like 3D immersive experience.
1:58
Ejaaz: And he was able to generate this in a couple of minutes, which is insane.
2:02
Ejaaz: And to kind of prove to you that this is like a real thing, what you're seeing
2:06
Ejaaz: on my screen right now is a Google Drive of basically all the leaked images from these files.
2:13
Ejaaz: And you can literally click on any of these.
2:17
Ejaaz: Let's go with blue guest room two. and if you press w to zoom in you can now
2:23
Ejaaz: literally zoom in and peek around the entire thing like let's see how close i can go first can
2:28
Josh: You look at what's on the mic oh.
2:29
Ejaaz: My god wait what's under the bed okay we don't know we don't know but like i
2:35
Ejaaz: can't read the title you can really get into it oh i'm under the bed now
2:38
Josh: Oh you're under.
2:39
Ejaaz: Officially under the bed anyway this stuff is is just insane josh and uh this
2:45
Ejaaz: is due to apple shop's model right
2:48
Josh: Yes, this is the coolest thing. So Apple Sharp just released this model last
2:53
Josh: week, which coincided with the release of these files, which created this funny
2:56
Josh: convergence of two technology, or I guess a technology and breaking news at once.
3:01
Josh: But I do want to talk about what splatting actually is because we're saying
3:04
Josh: this funny word a lot. No one actually has a clue what it means.
3:06
Josh: It's basically a way to make a 3D scene by storing it as a cloud of tiny.
3:11
Josh: Soft little blobs instead of building traditional 3D models.
3:13
Josh: So in the past, each as we've all played video games before,
3:16
Josh: it requires a large machine to run them because it's a lot of textures and polygons.
3:21
Josh: And there's just a lot of detail required to render fidelity with what these
3:26
Josh: blobs do, what these splats do is each blob.
3:29
Josh: It's kind of like a puff of spray paint and it's floating in 3D space.
3:32
Josh: And then every puff has a position, size, shape, color and transparency.
3:36
Josh: So to render an image from any viewpoint the computer
3:39
Josh: projects all those puffs onto your screen and then blends them together
3:42
Josh: like layering this transparent paint so it's much more efficient
3:45
Josh: than the previous way of doing this which is
3:48
Josh: lots of crazy rendering lots of compute required
3:52
Josh: and if you were to create something like we're showing on screen now which
3:55
Josh: is an example of 3d skulls that are sitting with dynamic
3:58
Josh: lighting and it looks really real it's something you'd see in a video game you
4:01
Josh: would normally have to render that overnight it would take forever to
4:04
Josh: do but what this new splatting technology does and what apple's ai
4:07
Josh: model does is it allows you to take this high fidelity
4:10
Josh: rendering and turn it into a splat by using
4:13
Josh: these blobs and that way you could render it as a single asset
4:16
Josh: on something as light as your smartphone so it turns these really detailed complicated
4:21
Josh: compute 3d images into something very simple so simple in fact that you can
4:26
Josh: actually do it yourself you can make images yourself you can do this for your
4:30
Josh: own content and you can do it for free almost instantly. It's really cool.
4:34
Josh: And this is thanks to Apple's new Sharp model that they released last week,
4:37
Josh: which is open source, which allows people to go around and play with it themselves,
4:40
Josh: which I think is a really cool new paradigm for this technology.
4:43
Ejaaz: So the way I'm thinking about this, Josh, and correct me if I'm wrong,
4:47
Ejaaz: is if I take a 2D image, right, it's 2D and it's composed of a bunch of pixels, right?
4:54
Ejaaz: If I use this splat technology on that image, it turns every one of those pixels
4:58
Ejaaz: into a kind of like 3D object, which is why like these skulls,
5:03
Ejaaz: for example, isn't a video of like the skulls from all different orientations.
5:07
Ejaaz: It's like a couple of pictures of these pile of skulls or even maybe an AI generated
5:12
Ejaaz: image or whatever and it's
5:15
Ejaaz: splattered into these like 3d blobs and i can now kind of like maneuver around
5:19
Ejaaz: it and look around is that kind of roughly on the right track
5:22
Josh: Yeah and it's funny you'll notice with like the epstein examples
5:25
Josh: there actually is a lot of fidelity in the way that it's rendered so if you
5:28
Josh: look at a video game in the past that you've played normally you'll look at
5:32
Josh: the details and it's very fuzzy it doesn't look very real because the computer's
5:35
Josh: trying to save rendering of compute for the things that you're actually looking
5:38
Josh: at that matter but these scenes that use this new technology They look so photorealistic
5:43
Josh: because you're essentially repainting the world from these millions of soft dots.
5:47
Josh: And it's fast because rendering is mostly throw blobs onto a screen efficiency
5:50
Josh: rather than heavy 3D geometry or this like slow ray tracing that you're seeing on a computer.
5:55
Ejaaz: Got it. And so the number one question that pops into my head then would be,
6:00
Ejaaz: well, what's the cost difference for like doing this in the traditional way
6:04
Ejaaz: versus like in the current splatting way?
6:06
Ejaaz: And how long does this take? We got this tweet by Brad Lynch who tried out this this Apple software.
6:12
Ejaaz: And he said he generated what we're looking at on the screen right now,
6:15
Ejaaz: which is, I think it's an image of him sitting by like an open ocean and like,
6:19
Ejaaz: you know, he's got his Apple Vision Pro on and he's like kind of peering around
6:23
Ejaaz: and he's in his living room. He's moving the images side by side.
6:26
Ejaaz: And it made it, it took him 10 seconds to generate this on his MacBook Pro.
6:31
Ejaaz: And I'm assuming because he used Apple Sharp, which you just said was open source,
6:34
Ejaaz: it's the cost of downloading the software and just running that on your computer.
6:38
Ejaaz: Is that, Do I have that roughly correct?
6:39
Josh: It's totally free. It's totally open source. And it is totally available for users of any computer.
6:45
Josh: I mean, you could render it on a laptop. You could render it on a phone.
6:49
Josh: You could do this instantly, anytime.
6:51
Ejaaz: Hold your horses, Josh. If you're saying that, that means we should do this ourself, right?
6:56
Josh: Perhaps we should do a demo for the audience.
6:58
Ejaaz: Let's not take his word for it. Let's do this ourselves.
7:00
Josh: Let's do it ourselves. We're going to do it live here.
7:03
Ejaaz: Yes, we have the Apple Shop kind of software here. We're going to upload an image.
7:08
Ejaaz: Let me see if I can find a convenient image of Josh oh what do you know
7:12
Ejaaz: what do you know i have it here i'm curious how long this is gonna take look
7:15
Ejaaz: at us good looking gentlemen on the screen we look good in 2d i would love to
7:19
Ejaaz: know how good we look in 3d and
7:21
Josh: That's the fun thing too is if you have like loved ones or your children or
7:25
Josh: people who are younger you could really freeze these moments in time special memories special trips.
7:29
Ejaaz: You've gone we're done josh by the way sorry oh my god before i even finish
7:32
Ejaaz: my explanation we're done okay here
7:34
Josh: We are let's see this walk me through.
7:35
Ejaaz: The picture here oh it's like a watercolor effect you can see it materializing
7:39
Ejaaz: oh it's rendering oh this is weird this is weird i'm zooming in oh wait i need
7:46
Ejaaz: to oh my god wait can we oh my god i can see you from like above yeah
7:51
Josh: Wait try to go closer and get.
7:52
Ejaaz: In deeper wait hang on let me get in deeper let me get into this face get
7:55
Josh: Up in our faces.
7:56
Ejaaz: Wait how do i how do i like zoom oh my god oh oh so we can like see us from
8:01
Ejaaz: the side too that's wait that's pretty crazy david's face is a little warped
8:06
Ejaaz: david's the guy the less handsome guy uh in the center um our arms are looking
8:11
Ejaaz: pretty good we're going to the gym oh we could look at us from above as well that's pretty get in
8:15
Josh: There scroll into the image i'm.
8:16
Ejaaz: Trying i'm trying to sorry i don't i don't mean to zoom into
8:19
Ejaaz: our crotches here but uh this is the this is the furthest i can go in which
8:23
Ejaaz: i'm kind of upset about hang on maybe if i maximize oh wow okay that that does
8:26
Ejaaz: look better oh my god well what i'm impressed with here josh is number one how
8:31
Ejaaz: quick that took to take it's funny on the screen it said uh it gave us a countdown from 60 seconds.
8:37
Ejaaz: And it ended up producing the entire rendering that we're looking at right now
8:41
Ejaaz: with 47 seconds to go. So it took 15 seconds to make.
8:44
Ejaaz: And I'm touching my laptop right now. It is not warm at all.
8:48
Ejaaz: So I'm assuming it didn't cost anything energy-wise as well.
8:52
Ejaaz: Also, the fidelity of this, Josh, is actually really, really good.
8:56
Ejaaz: It's way better quality.
8:57
Ejaaz: I mean, I think I look a bit kind of out of it. I look like I've had a few drinks.
9:02
Ejaaz: Maybe I did on the night, actually. But it looks really good.
9:06
Josh: Yeah, it's impressive how quickly it's able to render this and how low cost
9:10
Josh: it is and how lightweight it is. I mean, you could just run this in a browser very simply.
9:13
Josh: It's not a very compute intensive thing. And it's really cool.
9:16
Josh: So this is an example of a photo.
9:19
Josh: There are three kind of tiers to the splatting. There's the photo first.
9:22
Josh: And then second is this in between before we get to videos, which is this example
9:26
Josh: that you're seeing with Casey Neistat's studio.
9:28
Josh: Now, a lot of people who watch YouTube, they know Casey Neistat.
9:30
Josh: They love Casey Neistat.
9:31
Josh: And this is the most iconic place in the world of Casey.
9:35
Josh: What the meta team has done is actually go
9:38
Josh: through and create a giant splat of the studio so that
9:41
Josh: anybody with a vr headset can actually put the headset on and walk
9:44
Josh: around it now what we're looking at on screen it looks like it's an
9:47
Josh: actual video of the studio but the reality is is that it's one
9:50
Josh: large splat and it is a full fidelity
9:53
Josh: splat so if you put on goggles and you walk through
9:56
Josh: the space you can actually go and read the bindings
9:59
Josh: on the books you can like walk up to all
10:02
Josh: the shelves and peer at all the little things all the little trinkets that
10:05
Josh: are on them it is a full and total high fidelity
10:08
Josh: scan of the studio but in a very lightweight way if you used to try to do this
10:13
Josh: you would need a supercomputer to render this and you need a supercomputer to
10:16
Josh: run it you couldn't even do this on goggles that would be like disconnected
10:20
Josh: from a computer now thanks to this new technology you can scan real places into
10:26
Josh: the cyberspace and it's kind of acting as that,
10:28
Josh: almost like a preservation technique, where if there's a place that you love,
10:32
Josh: if there's a place you want to remember, you can actually scan it and then relive in that forever.
10:35
Josh: You can capture this place in its full fidelity, exactly how it is today.
10:39
Josh: And I thought this was a really cool example.
10:41
Ejaaz: I mean, what I find super cool about this is like in the traditional way,
10:46
Ejaaz: you would have to take a million pictures and stitch them all together,
10:49
Ejaaz: which would have taken you hours and hours and efforts and probably a bunch
10:53
Ejaaz: of people to get involved to help you produce.
10:55
Ejaaz: Also, I like that it's to scale as well, Josh.
10:58
Ejaaz: Like a lot of these simulation kind of videos or games that I've tried with
11:02
Ejaaz: Apple Vision Pro and stuff like that just seems kind of unrealistic.
11:05
Ejaaz: Obviously, like maybe you're playing a fantasy game or something.
11:08
Ejaaz: This is like to scale. It's like you're walking through, you're not going to bump into anything.
11:12
Ejaaz: I just think it's awesome. But then the natural question that pops into my mind
11:15
Ejaaz: is, well, can you do this with video?
11:18
Ejaaz: And I think, you know, we had our answer a few months ago earlier this year
11:21
Ejaaz: when this viral tweet went about of this guy who's kind of like directly speaking to a camera.
11:28
Ejaaz: But you can see in this video that someone's navigating around him.
11:31
Ejaaz: And this is just like, you know, a 2D video taken head on of this man sitting down in his chair.
11:37
Ejaaz: And he is able to navigate around him in every single different type of direction.
11:42
Ejaaz: You can peer at him at a kind of like angle perpendicular to him.
11:46
Ejaaz: You can see kind of like the way that his jaw looks like. And this is all generated
11:50
Ejaaz: through splats. None of that is real.
11:53
Ejaaz: None of that was actually filmed with a camera to the side of him.
11:56
Ejaaz: This is all generated via splats. Super cool.
11:59
Josh: It's fun to think of...
12:02
Josh: When you capture things to think of your camera as a paintbrush or
12:05
Josh: maybe even a can of spray paint like we were talking about earlier
12:08
Josh: where if you can just capture the smallest amount
12:11
Josh: of detail of a specific part of an image you can then
12:14
Josh: render it all fully in a 3d way like
12:17
Josh: we're seeing another image here where you can zoom in on the video you can pan
12:20
Josh: left and right and that's because it was kind of scanned like it was this can
12:24
Josh: of spray paint you you want to kind of spray paint things and then you could
12:27
Josh: relive them and capture them and i think it's such a cool new paradigm where
12:30
Josh: they're driving through the city streets or they're watching someone dancing
12:32
Josh: or whatever these examples are.
12:34
Josh: If there's something in your life that you want to capture, you can just do it and then relive it.
12:39
Josh: And this is particularly interesting if you're a user of iPhone,
12:42
Josh: because Apple's really the company who has been leading the charge of this.
12:45
Josh: And if you use an iPhone, you're aware of the camera sensor,
12:49
Josh: right? How they're kind of lined up.
12:50
Josh: And when you shoot a video using these top two, you actually give real 3D spatial depth to it.
12:56
Josh: And that's also because there's a lidar scanner on the bottom of the camera as
12:59
Josh: well so apple has all the tools here built in to
13:02
Josh: create the highest fidelity splats possible and now they're rolling out the
13:07
Josh: software to enable that to happen even more so on these handheld devices that
13:10
Josh: we all use it's like i guess the last example is it's kind of like if you take
13:14
Josh: a black and white photo you could add color to it this is taking a two-dimensional
13:18
Josh: photo and adding a third dimension to it and that's a really cool unlock my.
13:21
Ejaaz: Mind naturally goes on to um like applications like what can i use this technology for.
13:27
Ejaaz: And I think through a bunch of the examples that we've shown so far,
13:30
Ejaaz: it's kind of cinematic and maybe even veered towards gaming as well.
13:36
Ejaaz: Hollywood is the instant industry that I think of that I'm like is going to
13:39
Ejaaz: get completely run over by this, right?
13:42
Ejaaz: I know for a fact that they spend months, in some cases years,
13:46
Ejaaz: to render a single visual splat that we've been looking at throughout this entire episode.
13:54
Ejaaz: And so I think that this is going to cut costs down by like tens of millions of dollars.
13:58
Ejaaz: And it is going to cut time down and even jobs as well. I know they have teams
14:02
Ejaaz: of different people with different skill sets to stitch all the images together,
14:05
Ejaaz: to get the right grading, lighting, to like post-processing of a bunch of these
14:09
Ejaaz: images and then kind of make these visuals.
14:11
Ejaaz: There's no way that this doesn't get disrupted by it. It also got me thinking
14:14
Ejaaz: of one of my favorite shows on Apple.
14:18
Ejaaz: Finally, we're talking about Apple and now they have this. Like one of their
14:21
Ejaaz: hit shows is Severance, right?
14:22
Ejaaz: And I remember season two, they have this like crazy scene where like we've
14:26
Ejaaz: got the camera panning around him in all different kinds of ways.
14:28
Ejaaz: And Ben Stiller had an interview on this where he said each episode costs roughly $20 million to make.
14:34
Ejaaz: And this particular scene alone costs $10 million.
14:39
Ejaaz: Now, if he had something like a splat technology, right, he could generate this
14:44
Ejaaz: in a couple of minutes or even under a minute, like we showed ourselves earlier on. And for a fraction.
14:50
Ejaaz: Of the cost it just it just it's a no-brainer for me josh
14:53
Josh: Yeah i don't want to say that hollywood is
14:56
Josh: under attack but they are definitely in need
14:59
Josh: of rapid innovation quick because this is
15:02
Josh: a second front that there is being disruption on we
15:05
Josh: had we talked about google's vo3 and all the video generation
15:08
Josh: models how well they understand the world how good
15:10
Josh: they're able to generate video now this merges that
15:13
Josh: gap where you can actually take the real world
15:16
Josh: but you could capture it much more efficiently than you ever have before and
15:19
Josh: much more fully which creates a lot more dynamic optionality
15:22
Josh: for these shots so if you can't create it
15:25
Josh: in the real world using a splat well then you can create it in the digital world
15:28
Josh: using ai and what i understand is that people
15:31
Josh: in hollywood they're already starting to use stuff like this where they are
15:34
Josh: capturing things once instead of 10 times and they're using ai they're using
15:39
Josh: splats to just kind of massage the scene to get exactly what they want if it
15:44
Josh: wasn't perfect on that first try and it just saves a huge amount of money But
15:47
Josh: there are more use cases for this.
15:49
Ejaaz: Yeah, so you put this one in the dock, Josh, of a Swiss glacier collapse in 4D.
15:56
Ejaaz: Is this, like, can Splats be used as, like, a prevention model for these kinds of things?
16:00
Josh: Yeah, so earlier this year, there was a big landslide in the Swiss Alps,
16:03
Josh: and it took out an entire village, and it was very dangerous.
16:06
Josh: It created a very uncertain situation.
16:08
Josh: Times because it's hard to access that area and people didn't know what was affected.
16:12
Josh: So a helicopter came through with a big camera array and it just swept the whole
16:16
Josh: valley. And you could see on the video the before and after.
16:18
Josh: And they captured this incredibly high fidelity splat up the valley.
16:23
Josh: They could then diagnose immediately what areas were most in danger,
16:26
Josh: what areas needed the most help, how much danger there actually was.
16:29
Josh: And they were able to observe all of the things at any time without needing
16:34
Josh: a specific video feed of a specific area. So let's say you were looking at a
16:37
Josh: specific location on the mountain.
16:39
Josh: Well, you could just pause the splat and you could zoom in on that one area,
16:42
Josh: even if you didn't capture it with a video camera.
16:44
Josh: So there's a lot of utility for these outside of just entertainment.
16:47
Josh: There's also safety and other things like this.
16:50
Josh: I just thought it was a fun example of a real world use case that actually happened earlier this year.
16:54
Ejaaz: Love it. It's been a longtime dream
16:57
Ejaaz: of mine to go to Japan, and I've been fortunate enough to go a few times.
17:00
Ejaaz: And I'm going again next year. And I kind of thought about like,
17:04
Ejaaz: how do I share this experience with different people?
17:07
Ejaaz: And I spotted this one, Josh, where it's a tweet that goes, we've made it possible
17:11
Ejaaz: to walk through the hot spring town of Yamagata, Ginzan's Onsen with an avatar.
17:16
Ejaaz: And this is like a real life rendering. And it looks like a game because,
17:20
Ejaaz: you know, it's been generated one of using a splat machine or splat model.
17:24
Ejaaz: It is to scale. These are real life shop fronts and stores and homes and streams.
17:31
Ejaaz: But obviously, it's like a simulated game environment. And it got me thinking
17:34
Ejaaz: like, this looks like something out of GTA, Josh, right?
17:37
Ejaaz: And I'm thinking like, this would change the way that you create simulated realities.
17:41
Ejaaz: Like imagine the Sims game, but using real life worlds, and it can be generated
17:47
Ejaaz: in real time to reflect different kinds of people, personalities and shops.
17:51
Ejaaz: Like imagine if New york city was updated every
17:54
Ejaaz: single second or day or every hour uh to
17:57
Ejaaz: reflect accurate goings-ons in that city i just think this changes gaming forever
18:01
Ejaaz: right because one of the things that i loved about gaming is that uh it had
18:05
Ejaaz: kind of like a preset story but then when you got to the end of the story i
18:07
Ejaaz: was like damn i can't i now have to wait like a year or two years until the
18:11
Ejaaz: second one comes out gta what is it five or six which one have we been waiting on for
18:16
Josh: Oh we're waiting on six.
18:17
Ejaaz: Six right so we've been waiting
18:18
Josh: For like 12.
18:19
Ejaaz: Years 12 years over a decade for this game
18:22
Ejaaz: now you can get the second game or the third game or the fourth
18:25
Ejaaz: game or the fifth game immediately if we had these kind of
18:27
Ejaaz: generated simulated realities but
18:31
Ejaaz: it kind of like i played this out in my head josh um
18:34
Ejaaz: the end game for these splats surely has
18:37
Ejaaz: to be world models right world models um is supposedly going to be a big trend
18:43
Ejaaz: next year in ai models where you create these simulated realities or environments
18:48
Ejaaz: of the real world that we live in today and you stick in an AI agent or an AI
18:52
Ejaaz: model to kind of generate synthetic data.
18:55
Ejaaz: So it lives out its life and it kind of figures out how humans perceive things.
18:59
Ejaaz: Aren't splats just world models?
19:01
Josh: They're not actually. I think splats, you can think of a splat kind of like
19:05
Josh: what a neural link is to the human brain and AI.
19:08
Josh: A splat is kind of like to the physical world and the virtual world.
19:13
Josh: It's the bridge that combines the two together.
19:15
Josh: So what we just saw in that last example is,
19:19
Josh: you're able to walk through Japan and capture it with the camera and then merge
19:23
Josh: that real world data into cyberspace.
19:26
Josh: And if it was a world builder, it would just kind of create these virtual worlds.
19:30
Josh: So what I see is kind of the way this goes would ideally be a combination of
19:35
Josh: the two where splats are unbelievably efficient and are easy to capture the real world with.
19:40
Josh: And then you could take an AI model, a world model, and you
19:43
Josh: could apply extra fidelity on top of it depending on how much compute
19:46
Josh: you have so you could think of the splat as
19:49
Josh: a way of scanning the real world into the digital world and then
19:52
Josh: the ai world models are a way to increase the fidelity
19:55
Josh: using neural nets to predict what should be there
19:58
Josh: to fill in all the blank spaces and to make it feel
20:01
Josh: like more of a a real world plus experience
20:04
Josh: so if you were walking through your childhood house and you were scanning it you
20:07
Josh: can take a low or a high res but not totally high res version with a splat and
20:12
Josh: then use ai to enhance it so then you can actually capture this place that's
20:15
Josh: special to you and relive it forever using this cool new technology so this
20:18
Josh: very much feels like a bridge into this future hybrid between the real world
20:22
Josh: and the digital space okay.
20:25
Ejaaz: That makes sense so if the mission is to help ai understand humans in all forms
20:32
Ejaaz: of the way that they sense things the way that they perceive things sight um
20:36
Ejaaz: audio visual stuff um instead of like relying on them to kind of generate it
20:41
Ejaaz: from a bunch of data that we feed them,
20:43
Ejaaz: we can kind of take our reality and surroundings, compress it into this splat
20:47
Ejaaz: model, and then feed that into an AI model, a world model,
20:51
Ejaaz: a simulated reality that they're kind of operating in.
20:53
Ejaaz: And they'll have a more accurate depiction of how humans perceive the world,
20:57
Ejaaz: which will then accelerate us to whatever the hell AGI is going to end up being.
21:01
Josh: Yeah, it's like if you played with like Nano Banana Pro, for example,
21:04
Josh: and you added an image that was old, and maybe it was a black and white image
21:07
Josh: that was very low quality it can add color and it can make it feel more high
21:11
Josh: quality that's kind of what this does before for more virtual spaces.
21:15
Ejaaz: Well, what I like about that is we're just going to end up with an abundance
21:19
Ejaaz: of data and data has been lacking.
21:22
Ejaaz: I think at this point, every single model has been trained on the same corpus
21:26
Ejaaz: of data and we need to start tapping into private buckets of data to add kind
21:31
Ejaaz: of value or intelligence to an AI model.
21:33
Ejaaz: This kind of bypasses that entirely
21:35
Ejaaz: and creates this kind of synthetic but really accurate environment.
21:39
Ejaaz: That's super cool. And then like in terms of like the end game here,
21:42
Ejaaz: Josh, like do you think Apple is going to forge the path here?
21:44
Ejaaz: Has Apple somehow dug themselves out of the grave or rather dug up themselves
21:49
Ejaaz: out of the grave that they've kind of left themselves? They haven't been involved in AI or anything.
21:53
Josh: No, they haven't been. And this is not by any means a real attempt at AI.
21:58
Josh: This is kind of a separate thing.
22:00
Josh: This is in regards to their vision platform. This is kind of like what the future of compute is.
22:05
Josh: Everyone's building a pair of glasses. We have Meadow. We have Google now.
22:08
Josh: We have, I'm sure Microsoft is working on HoloLens.
22:12
Josh: Apple is the Vision Pro. everyone is working on this new spatial compute platform.
22:17
Josh: Apple is definitely furthest along this path and granted the Vision Pro gets
22:20
Josh: a lot of hate because it's very expensive it doesn't have a lot of use cases
22:23
Josh: but what you're seeing here is an early prototype for what the future of this
22:26
Josh: compute will look like when applied to actual consumer products.
22:30
Josh: So if you scroll down to one more beneath this there's a really fun example
22:33
Josh: where we take the splats that were mentioned that we mentioned in the video
22:38
Josh: and you can actually put them and pin them on your wall in your apartment.
22:41
Josh: So as you walk through the real world, you're able to pin these photo frames
22:46
Josh: and it has the splat built in so you could walk and actually look into the photo
22:49
Josh: and relive that experience.
22:51
Josh: There's some other examples that they have where they've pinned widgets on the
22:53
Josh: wall and as you walk into your apartment, these widgets that are like a calendar
22:57
Josh: will just be present on your wall.
22:59
Josh: And again, it's this merging of the digital and physical worlds and it looks
23:03
Josh: real. It looks like it's embedded inside the wall. It's kind of embossed in.
23:07
Josh: But what this leads to is this merging. It's this...
23:12
Josh: Combination of digital and physical through these augmented glasses
23:15
Josh: that we're going to get and splats are a really important part
23:18
Josh: of that so when apple open sourced their model earlier this week that
23:21
Josh: was a really big deal because it allows other developers to also lean into
23:24
Josh: this and you could see even in this example you could scan in people's faces and you
23:27
Josh: could speak to them in real time as if they were sitting right in front of you so it's
23:29
Josh: this this fun entry into the metaverse and this is almost what i wish i saw
23:33
Josh: meta was doing because meta being their new name meta they should be leaning
23:38
Josh: into building some sort of a metaverse which is the combination of these two
23:41
Josh: worlds and it seems like apple's actually the furthest ahead and this is kind
23:44
Josh: of what it'll look like when it's implemented across consumer products as we go it.
23:48
Ejaaz: Looks like something out of star trek or star wars um like you know you got
23:52
Ejaaz: this holographic simulations of people speaking to you i think um
23:57
Ejaaz: A major trend that's helped us get to what we're looking at today in front of
24:01
Ejaaz: us and make all of this feasible is just massively reduced costs of things.
24:05
Ejaaz: Like we've just spoken about like the cost of producing a splat or like a Hollywood
24:10
Ejaaz: traditional version of this would have cost tens of millions of dollars.
24:13
Ejaaz: And now it takes a couple of seconds and download an open source software.
24:16
Ejaaz: That is just massive. I think the next biggest trend
24:19
Ejaaz: is going to be the form factor specifically like i
24:22
Ejaaz: can't help but sorry hate on how
24:25
Ejaaz: big and uh bulky these uh where's
24:28
Ejaaz: the where is it um big and bulky the uh apple vision pros look on on people's
24:34
Ejaaz: heads i'm like that just looks so dorky it also kind of reminds me of um google
24:39
Ejaaz: glass which is obviously a completely different product but looked also really
24:42
Ejaaz: dorky and crazy for people to wear uh it seems like the form factor is going to be glasses, Josh.
24:47
Ejaaz: Meta's making them. It was leaked this week that Apple is also potentially working
24:53
Ejaaz: on a glasses version. That's not going to be Apple Vision Pro.
24:56
Ejaaz: It's going to be much more slimmer, sleeker, thinner.
24:59
Ejaaz: And then you have Google that's releasing Google Glass 2 2.0 next year.
25:04
Ejaaz: And then Amazon apparently is even releasing one as well. So it seems like glasses
25:08
Ejaaz: are going to be the form factor.
25:09
Ejaaz: I think it's now cheap enough to produce at a much larger scale so that anyone
25:14
Ejaaz: and everyone can use them.
25:15
Ejaaz: But also I think like the components of making these glasses,
25:18
Ejaaz: like the transistor and stuff are also cheap enough to run this technology.
25:21
Ejaaz: So we're kind of at a culmination of all these trends coming together where
25:25
Ejaaz: it's going to make the spatial reality that you've just described happen in
25:29
Ejaaz: real life, which is super cool.
25:30
Josh: We're getting close. It's like apples are big, bulky, and expensive.
25:33
Josh: They're $3,500. They weigh a couple of pounds. They work unbelievably well. That's what you want.
25:38
Josh: Meta's glasses google's glasses they
25:41
Josh: suck they're cheap they fit on your head but
25:44
Josh: they're a terrible experience so as we converge to the
25:47
Josh: middle of whatever that is as we reach apple's quality
25:50
Josh: with meta's form factor that's when you're going
25:52
Josh: to start to see this stuff everywhere because it will be cheaply accessible
25:56
Josh: and a really phenomenal experience and like you
25:59
Josh: said these costs are coming down quickly it is only a matter of
26:01
Josh: time until we get that perfect middle ground and this
26:04
Josh: hybrid product exists where we do start to get these experiences
26:07
Josh: available to us in our day-to-day lives in a package that is reasonable to walk
26:12
Josh: around in on a day-to-day basis so that's one of the things i'm most excited
26:15
Josh: about is this new frontier of hardware that is paired and supercharged by ai
26:19
Josh: and all these other cool pieces of software like splats that we're seeing um
26:23
Josh: unveiled pretty much every week now so.
26:25
Ejaaz: The metaverse is basically becoming a reality
26:28
Ejaaz: and i i'm so glad that we've moved on
26:31
Ejaaz: from it being a fad to it being a reality
26:34
Ejaaz: maybe we were just kind of like five years
26:37
Ejaaz: too early with all the nft stuff from the crypto sector that
26:40
Ejaaz: we saw uh way back when um but this is this is
26:43
Ejaaz: awesome and i'm excited to kind of see this scale to real life applications
26:46
Ejaaz: josh like it's all and well seeing like these demos of things but i can't wait
26:49
Ejaaz: to see the first uh splat movie so that i can kind of like see things from different
26:53
Ejaaz: orientations get people's different perspectives i can't wait for it to hit
26:57
Ejaaz: gaming and fast forward gta 6 7 and 8 so we don't going to have to wait another
27:01
Ejaaz: decade for these releases.
27:04
Ejaaz: And I'm excited about the costs and the form factors that are going to come
27:06
Ejaaz: with this, like being able to wear glasses.
27:08
Ejaaz: I'm curious, right? Because I was super skeptical when AirPods became a thing.
27:12
Ejaaz: And then I'm like, oh, I wear them all the time now. And it's just kind of like embedded in culture.
27:16
Ejaaz: It's going to change the way humans kind of interact with each other,
27:19
Ejaaz: which is going to be super, super cool.
27:21
Ejaaz: But that is it for the rest of this episode.
27:24
Ejaaz: It has been quite a week and quite the year.
27:28
Ejaaz: This is our Christmas day episode we hope you feel cheerful you know Josh and
27:33
Ejaaz: I came in our best Christmas attire I came with uh
27:36
Ejaaz: Let's call it the coal for being bad, I guess, the coal color.
27:41
Ejaaz: And Josh, you're repping red.
27:42
Ejaaz: That's awesome. If you are somehow so passionate about Limitless and you're
27:47
Ejaaz: listening to this, I just want to say thank you. That's frigging awesome.
27:50
Ejaaz: There have been thousands and thousands of you that have joined our community,
27:54
Ejaaz: that have subscribed to us, that listened to us week in, week out.
27:58
Ejaaz: And it means the world to us. It means the world to me, especially.
28:02
Ejaaz: And it's just awesome to have you guys here. We know about like 80% of you aren't subscribed.
28:08
Ejaaz: In the spirit of Christmas, we would love if you tap that subscription button,
28:13
Ejaaz: if you tap the notification button, or if you're listening to this on Spotify
28:16
Ejaaz: and you don't even want to see our faces, give us a rating. It would mean the world to us.
28:20
Josh: Also, this is one fun fact is we just crossed 100 episodes for Christmas.
28:24
Josh: So in terms of Christmas gifts, one, we have 100.
28:27
Josh: Two, thank you. You guys listening is the Christmas gift. And three,
28:30
Josh: we will continue to post all throughout the holiday season as a gift to you
28:33
Josh: for supporting us. All year round.
28:35
Josh: So thank you for the support. As always, you know, this is like the best part
28:39
Josh: of it is just being able to just like see the success of it,
28:42
Josh: see people listening, sharing it with your friends who would also enjoy it.
28:45
Josh: Ejaz ran into someone the other day who randomly was talking about the show
28:49
Josh: but didn't recognize Ejaz's face because he had never watched the videos,
28:52
Josh: only recognized the voice, which I thought was so funny.
28:54
Josh: So it's a really nice thing to see the message spreading.
28:58
Josh: So thank you for that. I guess happy holidays to all who are listening.
29:01
Josh: We are going to keep the episodes.