Latest / The Practical AI Digest / Diffusion Models: AI Image Generation Through Noise

Transcript

Diffusion Models: AI Image Generation Through Noise · The Practical AI Digest

• 0:00
  Welcome back to the Deep Dive. So we're shifting gears a little
• 0:03
  bit today. Yeah, a new format.
• 0:05
  A new format. Usually we look at, you know,
• 0:08
  the results of technology, how it changes society, the market
• 0:11
  impact. Right, the big picture stuff.
• 0:12
  But today we are taking a strictly engineering
• 0:16
  perspective. We're launching what we're
• 0:18
  calling the Practical AI Digest. I love this.
• 0:22
  So we're going to take one specific architecture, just one
• 0:25
  strip away all the marketing hype and explain exactly how it
• 0:30
  works so you can actually understand the mechanism under
• 0:32
  the hood. Exactly.
• 0:33
  And the topic for today is something that I think a lot of
• 0:37
  people feel they understand on a surface level, but the actual
• 0:40
  mechanics are quite surprising. Yeah, we are looking at the
• 0:43
  engine behind the massive, massive explosion in AI art
• 0:47
  we've seen over the last couple of years.
• 0:49
  We're talking about latent diffusion models, LDMS. This is
• 0:52
  the thing that powers Stable Diffusion is basically the
• 0:55
  reason you can now generate these incredible high fidelity
• 0:58
  images on a home computer. Instead of needing a giant
• 1:01
  server farm. And that is the key hook here.
• 1:04
  I mean everyone has seen the output you know mid journey
• 1:07
  deli, stable diffusion, you type a cyberpunk city in the style of
• 1:12
  Van Gogh and boom 5 seconds later you have a masterpiece.
• 1:16
  But very few people ask the next question, which is what is the
• 1:20
  engine driving this? Yeah.
• 1:22
  And specifically, how did we get from models that needed, like
• 1:25
  you said, a super powder to something that can run on a
• 1:29
  gaming laptop? And that's the core conflict.
• 1:31
  You know the problem we need to address before this paper?
• 1:34
  The high resolution image synthesis with late diffusion
• 1:36
  models paper. This whole field was incredibly
• 1:39
  expensive. It was a witch man's game,
• 1:41
  totally. Let's put a number on expensive
• 1:43
  because I think people just assume AI lives in the cloud and
• 1:46
  it's, you know, basically free. Oh, it's so far from free.
• 1:50
  Yeah. I mean, if you look at the
• 1:51
  source material, they highlight that training one of these
• 1:53
  powerful pixel based diffusion models.
• 1:55
  Yeah, it took hundreds of GPU days.
• 1:58
  Hundreds. Yeah, they cite examples from
• 2:00
  like 150 up to 1000 V 100 days. OK, let's just clarify that for
• 2:05
  you listening. AV 100 is a top tier enterprise
• 2:08
  grade graphics card. So 1000 V 100 days means if you
• 2:12
  had one of these very expensive cards, it would take you almost
• 2:16
  three years to train the model once, just once.
• 2:19
  Exactly. And that's just the training
• 2:20
  part, the inference. So actually making an image
• 2:23
  after you type a prompt, that was also painfully slow.
• 2:26
  Why? Because the neural network had
• 2:28
  to sequentially evaluate a massive, massive amount of data.
• 2:32
  So the mission of this deep dive is to unpack the architecture
• 2:34
  that solved this bottleneck. We're not just making images.
• 2:38
  We are, I guess, making them efficiently.
• 2:41
  We're going to break down the latent part of latent diffusion
• 2:44
  models. It's what democratize high
• 2:46
  resolution image generation and the whole trick.
• 2:49
  The core idea was separating the compression phase from the
• 2:52
  generative phase. OK, so let's get into that
• 2:55
  Section 1, The intuition and the problem with pixels.
• 2:57
  So before we can understand why latent diffusion is so special,
• 3:00
  we need to understand what standard diffusion even is.
• 3:03
  Right, the baseline. If I'm a developer looking at
• 3:06
  this for the first time, what's the basic mechanism?
• 3:08
  The basic idea is something called iterative denoising.
• 3:11
  It's. You can think of it as a model
• 3:13
  that's designed to learn a data distribution.
• 3:15
  Bye. By gradually removing noise.
• 3:18
  It's a two way St. the forward. Process and the reverse process.
• 3:22
  Right, so the forward process is fixed.
• 3:24
  It's easy. You take a clean image, let's
• 3:27
  say a photo of the cat, and you just gradually add Gaussian
• 3:31
  noise to it. A little bit at a time.
• 3:32
  Step by step, so you know, step one, it's a little grainy.
• 3:36
  Step 50 it's very grainy. By step 1000 is just Ure static,
• 3:40
  the image is completely gone. OK, that seems simle enough.
• 3:43
  Ruining a iPhoto is easy. It is the magic is the reverse
• 3:46
  process. This is where you train a neural
• 3:48
  network to look at that Ure static and predict the noise
• 3:52
  that was added at the previous step.
• 3:54
  Ah, so it's learning to go backwards.
• 3:56
  Exactly. Because if you can predict the
• 3:57
  noise, you can subtract it. Yeah.
• 3:58
  And if you do that over and over, step by step backwards
• 4:01
  from total randomness, you eventually reveal a clear image.
• 4:05
  So the model is kind of like a professional art restorer.
• 4:08
  It's looking at this messy noisy signal and saying, OK, I know
• 4:12
  what part of this is just static and what part is actual
• 4:15
  structure. And it just wipes the static
• 4:16
  away. That's a perfect way to put it.
• 4:18
  And ideally you end up with a beautiful high fidelity image.
• 4:21
  You do, but here's the problem, and it's a huge problem with
• 4:26
  standard models like Dally 2 or the original guided diffusion
• 4:30
  models. They do all of this in pixel
• 4:33
  space. Meaning it's literally
• 4:35
  calculating every single pixel. Every single pic.
• 4:38
  Think about it, you have a 10/24 by 1024 image.
• 4:41
  That's over 1,000,000 pixels and each.
• 4:42
  One has three color channels. Red, green and blue.
• 4:45
  So it's more like 3,000,000 data points, and the model has to
• 4:49
  process that entire massive grid for every single step of the
• 4:53
  denoise. Every step which can be hundreds
• 4:55
  or thousands of steps. OK, I see it now.
• 4:57
  And this is where we hit what the paper calls the pixel space
• 5:00
  bottleneck, right? And this right here is probably
• 5:03
  the most critical conceptual take away from this whole thing.
• 5:06
  Digital images contain a massive amount of information, right?
• 5:09
  But not all of that information is actually important.
• 5:13
  Semantically important. What do you mean by semantically
• 5:15
  important? OK, well imagine a photo of a
• 5:18
  brick wall. The semantic information is this
• 5:21
  is a brick wall. It's red.
• 5:22
  The mortar is Gray. The concept.
• 5:24
  The concept exactly, but the actual digital image also
• 5:28
  contains all this high frequency detail.
• 5:31
  High frequency meaning like the microscopic grain on the brick.
• 5:35
  Yes, imperceptible noise. Tiny, tiny shifts in the
• 5:39
  gradient of a shadow on one single brick.
• 5:43
  The exact texture of the mortar if you zoomed in 1000 times.
• 5:46
  All these little details take up a huge amount of data to encode.
• 5:50
  But do they matter to me as the person looking at the picture?
• 5:53
  Not really. I mean, if I showed you a
• 5:55
  picture of that wall and then I showed you another one where I
• 5:57
  just slightly blurred the texture or shifted the noise
• 5:59
  pattern, you still just say that's a brick wall.
• 6:02
  Right, the wallness of it hasn't changed.
• 6:04
  Exactly. But the standard diffusion
• 6:06
  models, they don't know that they.
• 6:07
  Can't tell the difference between the concept and the tiny
• 6:09
  details. No, because they're what are
• 6:11
  called likelihood based models. Mathematically, they're
• 6:14
  obligated to model everything, so they spend a completely
• 6:19
  disproportionate amount of their compute power modeling these
• 6:23
  imperceptible high frequency details.
• 6:25
  So it's like trying to write a book summary by, I don't know,
• 6:29
  analyzing the microscopic fibers of the paper on every single
• 6:32
  page. Instead of just reading the
• 6:33
  plot. Instead of just reading the plot
• 6:35
  points, yeah. That is the perfect analogy.
• 6:37
  You are burning these incredibly expensive GPU cycles just to
• 6:42
  figure figure out the texture of the paper, but all you really
• 6:45
  want is the story. That's the difference between
• 6:47
  perceptual compression and semantic compression.
• 6:49
  So the goal of the latent diffusion model is to stop
• 6:53
  looking at the paper fibers. Yes, the goal is to find a
• 6:55
  representation of the image, a different space that is
• 6:59
  computationally way way cheaper, but is perceptually equivalent
• 7:03
  to the original. We want to find a space where we
• 7:05
  can do the heavy lifting of generation without being bogged
• 7:07
  down by millions of useless pixels.
• 7:10
  Precisely. OK, this brings us to Section 2,
• 7:13
  the system architecture. The paper splits this whole
• 7:16
  system into two very distinct phases, and phase one you said
• 7:20
  is strictly about compression. Right, first we need to get out
• 7:24
  of pixel space and to do that they introduce a component
• 7:27
  called an auto encoder. OK, for the engineers listening,
• 7:30
  let's breakdown the specs of this auto encoder.
• 7:32
  So an auto encoder has two parts.
• 7:35
  They have an encoder, which we can call egoder, which we'll
• 7:37
  call encoder and encoder. The encoder takes your high
• 7:39
  dimensional image 6 LR. That's your massive grid of RGB
• 7:42
  pixels, and it crunches it down to something called a latent
• 7:46
  representation. We call that D getter.
• 7:48
  So $6 is the big original image and Zeidol is the tiny
• 7:52
  compressed version, correct? And Zelda lives in what we call
• 7:55
  latent space. It's a much, much lower
• 7:58
  dimensional space. The paper talks about a specific
• 8:01
  parameter for this, a down sampling factor, which they call
• 8:03
  Philidol. And what is that control?
• 8:05
  It just controls the ratio of compression.
• 8:07
  So if Philidol is equal to 4, the latent representations has a
• 8:11
  height and width that's 1/4 of the original image.
• 8:13
  And if $5 is 8, it's 18. So we're shrinking the map.
• 8:18
  But I mean, that brings me back to my earlier concern.
• 8:20
  When you shrink an image, you lose quality.
• 8:23
  If I take a big photo in Photoshop and I shrink it down
• 8:25
  to 10% and then try to blow it back up, it looks awful.
• 8:29
  It's all pixelated and blurry. How does the LDM avoid that?
• 8:34
  This is such a critical design choice in the paper.
• 8:37
  They don't just use a standard loss function like an L1 or L2
• 8:40
  loss. Mean squared error things like.
• 8:42
  That right, those standard loss functions tend to produce really
• 8:46
  blurry results because the model is trying to average out all the
• 8:50
  uncertainty. Right, the mean of all possible
• 8:52
  sharp edges ends up being just a blurry gradient.
• 8:55
  Exactly. So to keep things sharp, they do
• 8:57
  something clever. They use a perceptual loss
• 9:00
  combined with a patch based adversarial objective.
• 9:03
  Adversary. OK, that sounds like we're in
• 9:05
  Jan territory. It's borrowed directly from Jans
• 9:07
  Generative Adversarial Networks. They essentially have a second
• 9:11
  network, a discriminator, that looks at the reconstructed image
• 9:14
  that comes out of the decoder. And it critiques it.
• 9:16
  It critiques it. It basically asks does this
• 9:19
  little patch of the image look like a real texture?
• 9:22
  Is this sharp or is this blurry mush?
• 9:25
  So it forces the auto encoder to be honest, it can't just get
• 9:28
  away with a blurry approximation.
• 9:30
  No, it has to maintain what they call local realism.
• 9:33
  Local realism I like. That, and that's what ensures
• 9:36
  that even though Siller is highly compressed, when you run
• 9:40
  it back through the decoder dollar, the output image looks
• 9:43
  crisp and detailed. Now, there's a technical nuance
• 9:46
  here about regularization. The paper mentions two ways to
• 9:49
  keep this latent space well behaved.
• 9:52
  Why do we care if the latent space is well behaved?
• 9:54
  Well, imagine if you just mapped a bunch of images to a latent
• 9:57
  space and the points were just scattered completely randomly.
• 10:00
  So a cat is here, and a millimeter away is a toaster.
• 10:04
  In in between it's just garbage static.
• 10:06
  That would be a high variance chaotic space.
• 10:08
  You can't navigate that. You can't, and you certainly
• 10:11
  can't train a diffusion model on it.
• 10:14
  Diffusion models need a smooth continuous space that they can
• 10:17
  walk through, so regularization is about forcing the latent
• 10:21
  vectors to follow a standard predictable distribution.
• 10:25
  And what are the two options they talk about?
• 10:27
  Option A is called KL regularization.
• 10:30
  This adds A slight penalty that kind of nudges the latent
• 10:33
  distribution towards a standard normal distribution is very
• 10:36
  similar to how variational auto encoders or VA ES work.
• 10:40
  OK. And option B.
• 10:41
  Option B is VQ regularization, which stands for Vector
• 10:45
  quantization. This uses a kind of code book of
• 10:48
  learned vectors. It quantizes the latent space
• 10:51
  into discrete separate steps. This is more like what you see
• 10:54
  in VQGAN. So which one did they end up
• 10:56
  going with? They experimented with both, but
• 10:58
  the kale regularization version is what became the standard for
• 11:01
  Stable Diffusion. It just offers a really nice
• 11:04
  balance of quality and stability.
• 11:05
  OK, so let's just recap phase one.
• 11:07
  We take an image, we use an encoder to crunch it down into a
• 11:10
  tiny efficient latent vector 7. We then use a decoder to prove
• 11:14
  that we can turn Zeiger's back into a sharp looking image, but
• 11:16
  we haven't generated anything new yet.
• 11:18
  Not a thing. We're just compressing and
• 11:20
  decompressing. It's just building the stage.
• 11:21
  Exactly. Phase 2 is where the performance
• 11:24
  actually happens. This is latent diffusion.
• 11:26
  This is the core shift. We are no longer diffusing
• 11:29
  pixels, we are diffusing the latent vectors.
• 11:32
  Right. We take that compressed
• 11:33
  representation 70 hours, we add noise to that, and then we train
• 11:37
  a model to remove the noise from Ziller.
• 11:40
  So we are essentially hallucinating the compressed
• 11:43
  concept of an image rather than the image itself.
• 11:46
  That's a great way to put it. And because ZZS is so, so much
• 11:49
  smaller than the original Image 60's, the computational cost
• 11:53
  just drops off a Cliff. OK, let's talk about the
• 11:56
  backbone. What's the neural network that's
• 11:57
  actually doing this work? Is it a transformer?
• 12:00
  It feels like everything is a transformer these days.
• 12:02
  Right, that's the default assumption.
• 12:04
  Daily one used a transformer. It did, but no.
• 12:07
  And this is a really specific and I think very smart
• 12:10
  engineering decision. They use a time conditional U
• 12:13
  net AU. Net.
• 12:14
  Why AU? Net.
• 12:15
  So the paper argues that Transformers are naturally
• 12:18
  sequence based. They treat data like a stream of
• 12:21
  words in a sentence. One after another.
• 12:24
  But images, even latent images, they aren't sequences, they're
• 12:28
  2D spatial grids. They have structure.
• 12:30
  The top left corner is related to the top right corner in a way
• 12:33
  that the first word of a sentence isn't necessarily
• 12:36
  related to the last. Exactly.
• 12:38
  A unit is built from convolutional layers, and
• 12:41
  convolutions are designed specifically to process spatial
• 12:44
  grids. They have what's called an
• 12:45
  inductive bias towards images. They naturally understand things
• 12:50
  like locality and, you know, translation and variance, so.
• 12:53
  By using a unit, they're working with the structure of the data
• 12:56
  and not trying to force it into a different shape, and I assume
• 12:58
  this is faster. Much faster.
• 13:00
  Remember our down sampling factor $5?
• 13:03
  If $5 is 8, the latent space has 64 times fewer data points than
• 13:08
  the original pixel space, so the UNET is processing a tiny, tiny
• 13:12
  fraction of the data that a standard diffusion model would
• 13:14
  have to. And that explains how we can run
• 13:16
  this on a single consumer GPU. That's the whole ball game.
• 13:20
  This design allows for training on limited resources and,
• 13:24
  crucially, very efficient inference.
• 13:27
  Let's look at the math for just a second.
• 13:29
  We don't need to get bogged down, but for the engineers
• 13:31
  trying to implement this, what's the objective function?
• 13:34
  What is the model actually trying to minimize?
• 13:36
  It's a simplified version of the standard diffusion loss.
• 13:39
  The formula is the LLDM equals the expectation of the squared
• 13:44
  difference between epsilon and epsilon Theta.
• 13:47
  OK, let's translate that into plain English for sure, yeah.
• 13:50
  You have the actual noise that's epsilon which you just sampled
• 13:53
  from a normal distribution. OK, And you have the predicted
• 13:56
  noise that's epsilon Theta, which comes from your neural
• 13:59
  network, EU net, right? EU net looks at the noisy latent
• 14:02
  vector Z dollar at a specific time step T dollars, and it
• 14:07
  makes a prediction. The objective is simply to make
• 14:10
  the predicted noise as close as possible to the actual noise.
• 14:13
  So the model is just playing a guessing game.
• 14:15
  It's saying I see this messy static.
• 14:17
  I think the noise you added looks like this specific
• 14:19
  pattern. And we grade it on how close
• 14:21
  it's guess was. If it guesses the noise
• 14:24
  correctly, we can abstract it and take one step closer to a
• 14:27
  clean image. That's it.
• 14:29
  So now we have a system that can generate random images from
• 14:32
  noise, and it can do it really efficiently.
• 14:35
  But we don't want random images. No, we want specific images.
• 14:38
  We want a cat eating pizza. Right, and this brings us to
• 14:41
  Section 4, conditioning. This is the mechanism that
• 14:45
  enables text to image generation.
• 14:48
  How do we tell the model what to generate?
• 14:51
  I mean EU net speaks this spatial image language, but our
• 14:55
  prompts are in language sequence language.
• 14:57
  We need a bridge between the two.
• 14:59
  We do and the solution they implemented is brilliant.
• 15:01
  It's called cross attention. This is a mechanism that's
• 15:04
  borrowed from the transformer world, isn't?
• 15:06
  It it is, yeah. They effectively graphed these
• 15:09
  cross attention layers onto the unit backbone, but they make the
• 15:12
  really smart architectural move here.
• 15:15
  They separate the text processing from the image
• 15:17
  generation. They introduce what they call a
• 15:19
  domain specific encoder, which they label as taffeta.
• 15:24
  Taffeta. It's a totally separate model.
• 15:26
  It could be bird, it could be a colleague model that takes your
• 15:29
  input, say your text prompt right, and IT projects it into
• 15:32
  an intermediate representation. So the text goes into its own
• 15:36
  dedicated processor first, before it ever touches the image
• 15:39
  generator. Yes, and only then is that
• 15:42
  process text representation fed into the unit through the
• 15:46
  attention mechanism. If you think of the attention
• 15:49
  formula, attention to QKV. Query key and value.
• 15:54
  Right, let's map those terms to this system.
• 15:56
  OK, who's the query? The query comes from the unit.
• 15:58
  It's coming from the intermediate layers of the image
• 16:01
  in progress. You can think of it as the image
• 16:02
  asking questions. And the keys and values the.
• 16:05
  Keys and values come from our conditioning input, so the text
• 16:08
  prompt after it's been processed by that tough head encoder.
• 16:11
  So visually, can you walk me through what's happening here?
• 16:14
  OK, imagine the unit is trying to denoise a little patch of the
• 16:18
  latent image. It generates a query that is
• 16:21
  effectively asking what is supposed to be in this top left
• 16:24
  corner right now. OK, the crossattention layer
• 16:27
  then takes that query and compares it to the keys from the
• 16:30
  text prompt. If the prompt is a red cat on a
• 16:33
  blue table, the keys for red and cat might light up for that
• 16:37
  area. They become.
• 16:37
  Relevant. They become relevant and then
• 16:39
  the values provide the actual information.
• 16:42
  This area should have red furry texture.
• 16:45
  That's a fantastic explanation that So the image generation
• 16:47
  process is literally querying the text prompt for guidance at
• 16:51
  every single step of the denoising.
• 16:52
  And this design is so incredibly flexible because the unit only
• 16:56
  interacts with the conditioning through this clean cross
• 16:58
  attention interface. The conditioning input can be
• 17:02
  anything. Doesn't have to be text.
• 17:04
  Nope. As long as you have a domain
• 17:05
  specific encoder atop of data that can turn your input into
• 17:08
  the right shape for the key and value pairs, you can.
• 17:11
  Plug anything in. You can condition on anything.
• 17:13
  You can use semantic maps, so like a color-coded layout.
• 17:16
  You can condition on class labels or even other images.
• 17:20
  And this explains why the whole Stable Diffusion ecosystem is so
• 17:24
  modular. People are constantly swapping
• 17:26
  out these conditioning encoders to build things like control net
• 17:29
  or IP adapter. Without having to retrain the
• 17:31
  core unit from scratch. It's a plug and play
• 17:33
  architecture for control signals.
• 17:35
  Exactly. OK, so we've covered the auto
• 17:38
  encoder, the latent diffusion process itself, and the
• 17:41
  conditioning. Now let's get practical Section
• 17:44
  5 trade-offs in tuning. If I'm an engineer trying to
• 17:48
  implement this, or even just a power user trying to optimize my
• 17:51
  workflow, what are the levers I need to know about?
• 17:54
  The paper spends a lot of time on that down sampling factor,
• 17:57
  $5. Yes, this is the sweet spot
• 17:59
  analysis. They tested a whole range of
• 18:01
  factors and five five of 124816 all the way up to 32.
• 18:07
  And just to remind everyone, $5.00 is how much we compress
• 18:09
  the image, right? So let's look at the extremes.
• 18:12
  What happens at five $302.00? That's a massive amount of
• 18:16
  compression. It is at feet three, 302 hours.
• 18:20
  You're squashing a 256 by 256 image all the way down to an 8
• 18:25
  by 8 grid. It's tiny.
• 18:27
  And the benefit is speed I assume.
• 18:29
  The benefit is the diffusion model is lightning fast.
• 18:31
  It has almost no data to process.
• 18:34
  But the downside is the auto encoder just fails.
• 18:37
  It abstracts away way too much information.
• 18:39
  And that becomes too simple to be useful.
• 18:41
  Exactly. The perceptual quality just
• 18:43
  stagnates. You lose all the texture, you
• 18:45
  lose text legibility, the reconstruction looks.
• 18:48
  I don't know, sketchy because the latent space just doesn't
• 18:51
  have the capacity to store the fine details anymore.
• 18:53
  OK. And what about the other
• 18:54
  extreme? And four of one or two?
• 18:57
  Well, 4111 is just pixel space. You're right back to the
• 19:00
  original problem. So it's slow high memory usage.
• 19:02
  You're not gaining any of the efficiency benefits that make
• 19:05
  the LDM architecture so great. So what's the verdict?
• 19:07
  Where's the Goldilocks zone? So the paper concludes that the
• 19:10
  audible balance is somewhere between in five of four and in
• 19:12
  five of eight. This is where they achieved the
• 19:15
  best FID scores. And FI.
• 19:17
  First yet inception distance. It's basically the standard
• 19:20
  metric for measuring how real generated images look compared
• 19:24
  to a data set of real images. Lower is better.
• 19:27
  OK. And the models they call LDM 4
• 19:30
  and LDM 8 down sampling by 4:00 and 8:00, they achieved
• 19:33
  state-of-the-art results on day sets like Celebi HQ and
• 19:37
  Imagenet. And they were faster.
• 19:38
  They significantly outperformed the pixel based models in sample
• 19:42
  throughput, which is speed, while also getting lower FID
• 19:45
  scores which is better quality. It is so rare in engineering to
• 19:47
  get faster and better at the same time.
• 19:50
  Usually you have to pick one. It's very rare, and that's
• 19:53
  really why this paper was such a breakthrough.
• 19:55
  But there's another practical feature that I think is even
• 19:57
  cooler. It's called convolutional
• 19:58
  sampling. And what does that?
• 20:00
  Enable. It enables resolution solution
• 20:01
  independence. Most transformer based models
• 20:04
  are totally stuck generating images at the exact resolution
• 20:07
  they were trained on. Right.
• 20:08
  If you trained on 256 by 256, you get 256 by 256.
• 20:13
  That's it. But LDMS are different.
• 20:15
  Because the model is fully convolutional.
• 20:17
  It learns features rather than fixed positions.
• 20:20
  It learns what a tree looks like or what sky looks like.
• 20:23
  Which means you can take a model that was trained on 256 by 266
• 20:27
  images and just apply it to a 10/24 by the 424 canvas.
• 20:31
  Yeah, it doesn't just break. It doesn't break, it just
• 20:33
  generates more stuff. If you ask for a landscape,
• 20:36
  it'll just generate A wider landscape with more trees and
• 20:39
  more mountains. The paper shows these amazing
• 20:41
  examples of generating huge, consistent panoramas that are
• 20:46
  way larger than any of the training images.
• 20:48
  That is. Huge for creative applications.
• 20:50
  It implies the model understands textures and relationships on a
• 20:53
  local level, so it can just tile them indefinitely.
• 20:56
  It also enables these really powerful editing workflows like
• 20:59
  in painting. So filling in a missing part of
• 21:01
  an image. Exactly.
• 21:02
  You can mask out a part of an image and just ask the model to
• 21:05
  fill it in. The LDM handles this completely
• 21:07
  naturally. It just treats the unmasked part
• 21:10
  as known latent codes and then diffuses the mask part to match
• 21:13
  the context. How does it stack up up against
• 21:16
  specialized tools built just for that?
• 21:19
  So the paper compares it against specialized in painting
• 21:22
  architectures like a model called Llama, and the LDM
• 21:25
  achieves competitive performance.
• 21:27
  Which is pretty remarkable for a general purpose.
• 21:30
  Model. It's amazing.
• 21:31
  It wasn't built just for in painting, but it happens to be
• 21:33
  great at repair work too. And what about super resolution
• 21:37
  upscaling? Same story, you can use LDMS to
• 21:40
  upscale low res images and because they're likelihood based
• 21:45
  they avoid something called mode collapse.
• 21:47
  That's a classic jam problem right?
• 21:49
  Where the model just learns to output the same generic good
• 21:52
  looking face over and over. Again, exactly.
• 21:55
  LDM's preserve diversity, and unlike the big autoaggressive
• 21:59
  models, they don't need billions and billions of parameters to do
• 22:02
  it. They're just efficient, diverse,
• 22:04
  and sharp. So if we step back and look at
• 22:06
  the big picture here, the LDM architecture really is a kind of
• 22:09
  best of all worlds approach. I think so.
• 22:12
  They took the perceptual quality of Jans through that auto
• 22:15
  encoders discriminator, they took the stability and diversity
• 22:18
  of diffusion models, they took the control of Transformers
• 22:21
  using cross attention, and they combined all of it into a system
• 22:25
  that's efficient enough to run on consumer hardware.
• 22:27
  That is the summary. They identified that the pixel
• 22:30
  was the wrong unit of currency for generation.
• 22:34
  The latent vector is the right unit.
• 22:37
  Let's wrap up with the aha moment.
• 22:40
  For me, the genius of latent diffusion is that separation of
• 22:44
  concerns. You don't need to paint every
• 22:46
  single pixel from scratch. You can dream in this compressed
• 22:50
  abstract space, the latent space, and then you have a
• 22:54
  separate dedicated translator, the decoder, to turn that dream
• 22:57
  into a high fidelity reality. I like to think of it like a
• 23:00
  chef versus a chemist. The diffusion model is the chef,
• 23:04
  It's the artist. It decides what ingredients go
• 23:07
  together to make a delicious meal.
• 23:09
  The auto encoder is the chemist. It worries about the precise
• 23:11
  molecular structure of the food. But the chef doesn't need to
• 23:15
  know chemistry to cook, right? They just need to know what
• 23:17
  flavors work together. And by separating those roles.
• 23:20
  The chef can work much much faster.
• 23:22
  And the impact of the separation has been huge.
• 23:24
  It completely reduced the barrier to entry from tech giant
• 23:27
  supercomputer down to single GPU.
• 23:30
  It effectively democratized high resolution image synthesis.
• 23:35
  Yeah, and that's where we're seeing this incredible explosion
• 23:38
  of creativity and tools in the open source community right now.
• 23:42
  So here's a final provocative thought for you to Mull over.
• 23:45
  We've just spent this whole time talking about how compressing
• 23:47
  images into latent space let's us generate visuals efficiently,
• 23:51
  right? We're already seeing similar
• 23:53
  logic being applied to audio generating music by diffusing
• 23:56
  spectrograms in a latent space. Yeah.
• 24:00
  What happens when we start applying this latent diffusion
• 24:03
  logic to video day or to entire 3D environments?
• 24:07
  We are already standing on that precipice.
• 24:09
  I mean, if we can dream up a static snapshot on a home
• 24:12
  computer today, are we moving toward a future where we can
• 24:15
  generate entire realities? You know, movies, video games,
• 24:18
  immersive worlds, all of it all optimized to run on the hardware
• 24:21
  you already on? We're moving from generating
• 24:23
  snapshots to generating realities.
• 24:25
  And the engine under the hood is exactly what we talked about
• 24:27
  today. It's going to be a wild.
• 24:29
  It certainly is. I hope this deep dive gave you a
• 24:32
  solid mental model of how the magic trick actually works.
• 24:36
  Thanks for listening and we'll see you in the next deep dive.