Latest / The Practical AI Digest / Synthetic Data: Artificial Data for Real Insights

Transcript

Synthetic Data: Artificial Data for Real Insights · The Practical AI Digest

• 0:00
  Welcome back to the Deep Dive. Today we are wading into waters
• 0:03
  that feel, honestly, a little dangerous.
• 0:06
  Dangerous. That's a strong way to start.
• 0:07
  Well, think about it. We're talking about high stakes
• 0:09
  medicine. We're talking about diagnosing
• 0:11
  rare legal diseases, situations where accuracy is literally life
• 0:17
  or death. And the premise of the research
• 0:20
  we're looking at today is that the best way to save those lives
• 0:22
  is to use, well, fake data. But that way it does sound a bit
• 0:27
  like malpractice, doesn't it? Right.
• 0:29
  I mean, if I'm a patient and I hear my doctor is training their
• 0:32
  diagnostic AI on patients that you know don't even exist, I'm
• 0:35
  getting a second opinion immediately.
• 0:38
  And you'd. Be right to be skeptical, but
• 0:41
  this is the big butt for today. What if I told you that relying
• 0:44
  only on real patient data is actually what's making the AI
• 0:47
  fail? How so?
• 0:49
  That the real data is, in a way, mathematically broken when it
• 0:53
  comes to the sickest people. That is the paradox we are
• 0:55
  unpacking today. This is our Practical AI Digest
• 0:58
  edition. So for all of you out there who
• 1:00
  are building these systems, this one's for you.
• 1:02
  We are looking at a stack of research, primarily a new
• 1:06
  technical paper from Scientific Reports that was validated on
• 1:11
  COVID-19, kidney disease and Dengo datasets.
• 1:14
  A pretty serious lineup of diseases.
• 1:16
  Exactly. And we're also pulling in some
• 1:18
  industry specs from Blue Gen. AI and an explainer from IBM
• 1:22
  Technology. But we aren't just talking
• 1:25
  theory here. We are breaking down a specific
• 1:27
  buildable framework. That's the key.
• 1:30
  We're looking at a hybrid pipeline.
• 1:31
  We're going to talk about combining classical statistical
• 1:35
  tricks with some some really heavy hitting deep learning.
• 1:38
  Specifically. Specifically Deep CT, Jan and
• 1:41
  something called residual networks or resnets.
• 1:43
  And just to really set the stage for you builders listening, by
• 1:46
  the end of this, we want you to walk away with the recipe.
• 1:49
  We're going to cover the architecture, the specific hyper
• 1:51
  parameters. I'm talking learning rates,
• 1:53
  batch sizes, the works, and how to validate it so you don't end
• 1:57
  up, you know, just hallucinating patient data.
• 1:59
  It's basically a master class in handling the imbalance data set
• 2:02
  problem. So let's start there.
• 2:04
  The intuition You mentioned that real data is broken or biased.
• 2:09
  I think most people assume data is neutral.
• 2:11
  It's just numbers in a spreadsheet.
• 2:14
  How can a spreadsheet have a bias against sick people?
• 2:17
  It's not a bias in the human sense.
• 2:19
  It comes down to the math of optimization.
• 2:21
  We call it majority class bias. OK, think about a typical
• 2:25
  hospital database. Let's say you pull records for,
• 2:28
  I don't know, 1000 patients, OK. 1000 rows in my CSE file.
• 2:32
  I've got it open. Now, how many of those thousand
• 2:34
  people actually have a rare, specific form of kidney disease?
• 2:38
  Very. Few I'd imagine.
• 2:39
  Maybe 50, maybe even fewer. Right.
• 2:41
  The vast majority are there for check UPS or broken arms or the
• 2:44
  flu. They don't have this specific
• 2:46
  critical condition. Exactly.
• 2:48
  So you have a 95 to 5 split, 950 healthy people, 50 sick people.
• 2:53
  Now you feed this to a standard machine learning classifier, say
• 2:57
  a random forest or a basic neural net.
• 3:00
  The models only goal is to maximize its accuracy.
• 3:03
  It just wants to get the highest score possible.
• 3:05
  That's it. And that's where the trap is.
• 3:06
  The model quickly realizes, wait a minute, if I just predict
• 3:11
  healthy for every single person, I'm right 95% of the time.
• 3:16
  It's the lazy student strategy. It crams for the easiest part of
• 3:19
  the test and aces it, but it doesn't learn anything.
• 3:22
  Precisely. In a computer science class, 95%
• 3:25
  accuracy is amazing. You get an A.
• 3:28
  But in an oncology ward, that model is a catastrophe.
• 3:31
  It's worse than useless. It has 0% sensitivity.
• 3:35
  It missed every single cancer case.
• 3:38
  It maximized its score by completely ignoring the minority
• 3:42
  class. So the data itself isn't
• 3:45
  prejudiced in a way we'd recognize, but the distribution
• 3:48
  of the data forces the algorithm to ignore the edge cases.
• 3:51
  And in medicine, the edge cases are usually the ones that need
• 3:54
  saving. This is why we need synthetic
• 3:55
  data. We have to artificially inflate
• 3:58
  that minority class so the model can't just brush it aside
• 4:00
  anymore. The IBM source had a really good
• 4:02
  analogy for this. I thought they used football.
• 4:04
  The Premier League one, right? Yeah, that's a great way to
• 4:07
  think about. It walk us through it.
• 4:08
  OK, so if you trained an AI on the last 20 or 30 years of
• 4:12
  English football and and asked it to predict the next winner,
• 4:15
  what would it say? Manchester City, Manchester
• 4:18
  United, Chelsea. One of the big rich clubs,
• 4:21
  right? It creates a simple rule in its
• 4:23
  head. Rich big teams win.
• 4:25
  It's seen that pattern over and over.
• 4:28
  It's the majority class. And then 2016 happens.
• 4:31
  And Leicester City wins the whole thing.
• 4:32
  The ultimate outlier, a total statistical anomaly.
• 4:36
  Exactly. If your model is only looking at
• 4:38
  the majority history, it says that event is impossible, but it
• 4:42
  happened. Synthetic data lets us simulate
• 4:45
  thousands of Leicester City seasons.
• 4:47
  It lets us create these what if scenarios that force the model
• 4:51
  to learn the characteristics of the underdog.
• 4:53
  Or, in our case, the characteristics of the rare
• 4:55
  disease. That's the perfect parallel.
• 4:57
  We aren't just copying data, we are creating new examples that
• 5:00
  follow the same underlying rules of physics and biology, but they
• 5:03
  fill in the gaps where a real data is thin.
• 5:06
  So it's crucial to understand we aren't just typing in random
• 5:09
  numbers. Oh, absolutely not.
• 5:11
  That would be useless. We're learning the statistical
• 5:13
  distribution, the hidden shape of the data, and then we're
• 5:16
  sampling from that shape to create new plausible points.
• 5:20
  Now before we get to the how, I want to clarify the use case
• 5:23
  because the blue Gen. AI material mentions 2 main
• 5:26
  reasons for using synthetic data.
• 5:28
  One is privacy, which is massive.
• 5:31
  Huge. Yeah.
• 5:32
  GDPRAPA, right? If the patient is completely
• 5:36
  fake, you can't get sued for leaking their data.
• 5:38
  You can share it with researchers freely.
• 5:40
  True, that's what we call the replacement use case.
• 5:43
  But for this deep dive, for what this paper is about, we are
• 5:46
  focused on augmentation. OK, what's the difference?
• 5:49
  We aren't throwing away the real data.
• 5:50
  We are supplementing it. We are taking those 50 real sick
• 5:53
  patients and generating say 450 new synthetic sick patients.
• 5:58
  So you turn the 50 into 500. And suddenly the lazy student
• 6:03
  can't get an A by ignoring them. They have to actually study.
• 6:06
  OK, let's get into the engineering.
• 6:07
  This hasn't been a straight line to success.
• 6:09
  There's been an evolution here, a couple of tiers of technology.
• 6:14
  A pretty clear one actually. It's really been a journey from
• 6:16
  simple geometry to complex deep learning.
• 6:21
  We can call tier one classical resampling.
• 6:23
  This is stuff you could run on a laptop back in 2010.
• 6:26
  Pretty much the big name here, the one everyone learns, is
• 6:29
  SOBT. SMOTE Synthetic minority
• 6:33
  oversampling technique. It's a staple in data science
• 6:36
  interviews. It is, and it's intuitively
• 6:39
  very, very simple. Imagine your data points are
• 6:41
  plotted on a 2D graph. You have your big cluster of
• 6:44
  healthy patients and then you have this little cluster of 50
• 6:47
  sick patients floating off in a corner.
• 6:50
  OK, I can picture it. SMODI picks one of those sick
• 6:53
  patients at random, then it finds its K nearest neighbors,
• 6:56
  so the other sick patients that are closest to it on the grass.
• 6:59
  And then it just connects the dots.
• 7:00
  Literally it draws a straight line between the patient it
• 7:03
  picked and one of its neighbors, and then it just picks a random
• 7:06
  spot on that line and says new patient here.
• 7:08
  So it's just linear interpolation.
• 7:10
  It's all it is. So if patient A has high blood
• 7:13
  pressure and patient B who's nearby has slightly higher blood
• 7:17
  pressure, smell may just assumes A valid patient C exists
• 7:21
  somewhere in the middle. Precisely.
• 7:22
  The math is just X nu plus Lambda X neighbor a she.
• 7:27
  It's simple, and for some problems it works surprisingly
• 7:30
  well, but there's a massive flaw.
• 7:32
  Biology isn't a straight. Line lingo biology is messy.
• 7:36
  It's nonlinear. Complex diseases don't always
• 7:39
  follow a straight, predictable path between two patients.
• 7:42
  If you just draw lines, you might be creating patients that
• 7:44
  are biologically impossible. Or just not very representative
• 7:47
  of the true complexity, right? Plus, COT is kind of dumb in how
• 7:51
  it samples. It treats every minority point
• 7:54
  the same. It doesn't care if a data point
• 7:55
  is easy to classify or really confusing and sitting right on
• 7:58
  the fence. And that's where the upgrade
• 8:00
  came in, right? Ada sign.
• 8:01
  ADA sign hole adaptive synthetic sampling.
• 8:04
  This is like SMOTE scene with a bit of a brain.
• 8:06
  It looks at the data set and asks where's the model
• 8:08
  struggling the most. It hunts for the hard to learn
• 8:11
  examples. Yes.
• 8:13
  It looks at each minority point, each sick patient, and counts
• 8:17
  how many of its neighbors are from the majority class, the
• 8:20
  healthy patients. So if a sick patient is totally
• 8:23
  surrounded by healthy patients, it's.
• 8:25
  In what we call a hostile environment, it's an ambiguous
• 8:28
  case. Adis YN sees that and assigns a
• 8:32
  higher weight to that point. It says we need more troops here
• 8:36
  on the frontline. So it generates more synthetic
• 8:38
  data specifically along that messy decision boundary to try
• 8:42
  and clear up the confusion. Exactly.
• 8:44
  It's much smarter, but it's still geometric.
• 8:46
  It's still just drawing lines. And that brings us to the limit
• 8:48
  of Tier 1. It does when you have high
• 8:51
  dimensional data. We're talking hundreds of
• 8:52
  columns with gene expressions, complex symptom interactions,
• 8:56
  lab results. Simple geometry just breaks down
• 8:58
  completely. You can't draw a line in 500
• 9:00
  dimensional space and expect it to mean anything.
• 9:02
  You need something that can learn the manifold.
• 9:04
  The curved, twisted shape of the data's probability distribution.
• 9:08
  You need deep. Learning and enter the Jans
• 9:11
  generative adversarial networks. Tier 2, the famous cat and mouse
• 9:16
  game. Or maybe artist and critic is a
• 9:19
  better way to put it. Break it down.
• 9:20
  You have two neural networks fighting each other.
• 9:23
  The generator is the artist. It tries to create a fake
• 9:26
  sample, like a fake patient record.
• 9:28
  The discriminator is the critic. It looks at the fake record
• 9:31
  alongside a real 1 and has to guess which is which.
• 9:34
  And if the discriminator spots the fake.
• 9:36
  The generator gets punished, gets a bad signal and has to go
• 9:40
  back and try harder next time it learns from its mistake.
• 9:43
  And this goes on and on for. Thousands of rounds until
• 9:46
  eventually the generator is so good at creating fakes that the
• 9:49
  discriminator is no better than guessing its accuracy is 50%.
• 9:52
  And at that point you have a master forger, you have high
• 9:55
  fidelity synthetic data. That's the theory, but here's
• 10:00
  the big snag. Standard jams are terrible at
• 10:01
  sreadsheets. Why?
• 10:03
  I mean, they can generate photorealistic faces of people
• 10:06
  who don't exist. Why is a spreadsheet of patient
• 10:09
  data harder than a human face? It's a great question.
• 10:12
  It's because of the data types. Pixels in an image are all more
• 10:15
  or less the same kind of thing. They're continuous values from
• 10:18
  zero to 255. They flow into each other
• 10:21
  smoothly. But a spreadsheet, it's a total
• 10:23
  mix. You have a column for blood
• 10:25
  pressure, which is a continuous number.
• 10:27
  From right next to it is gender, which is a discrete category.
• 10:30
  Then blood type another category.
• 10:33
  This mix of discrete and continuous data completely
• 10:36
  confuses standard Jans. And the distribution of those
• 10:39
  numbers is usually really weird, right?
• 10:41
  Like blood pressure isn't just one simple bell curve, it might
• 10:44
  have a couple of different peaks for different patient groups.
• 10:46
  Exactly. It's multimodal, yeah, and
• 10:49
  standard Jans suffer from something called mode collapse.
• 10:52
  Mode collapse. They find one type of easy to
• 10:54
  fake patient one mode in the data and they just keep
• 10:58
  generating that same kind of patient over and over again.
• 11:00
  They ignore all the variety and especially the rare cases.
• 11:03
  Which defeats the entire purpose of what we're trying to do.
• 11:06
  We need the rare stuff. That is precisely why the
• 11:09
  Scientific Reports paper didn't just use a vanilla Jan, they
• 11:12
  used something called Deep CT Jan.
• 11:15
  OK, let's breakdown that acronym.
• 11:16
  CT Jan. Right, so the T is for tabular
• 11:20
  because it's designed for spreadsheets.
• 11:22
  The C is the real magic here. It stands for conditional.
• 11:26
  Conditional. And that lets us explicitly tell
• 11:29
  the generator what we want. We can give it a condition, we
• 11:31
  can say I want you to generate a sample, but only for the
• 11:34
  minority class. So instead of just hoping it
• 11:36
  stumbles upon making a sick patient, you can actually force
• 11:39
  its hand. You condition the input, you say
• 11:42
  give me a plausible breakdown of a patient with dengue fever, and
• 11:46
  this forces the model to learn the specific complex
• 11:50
  distribution of that class, which is what prevents mode
• 11:53
  collapse. So that's the engine, the deep
• 11:55
  C2 Gan, but the paper we're discussing today proposes a
• 11:58
  hybrid framework. They didn't just turn on the Jan
• 12:01
  walkway, they built a whole pipeline around it.
• 12:03
  This is the real how to section for the engineers listening.
• 12:06
  If you're building this, you need to follow these three steps
• 12:09
  they laid out. OK, Step 1, preprocessing.
• 12:11
  This is standard data hygiene. You can't train a model on empty
• 12:15
  cells for any missing numbers that used mean imputation.
• 12:20
  For missing categories they used mode imputation.
• 12:23
  So fill in the blanks with the average or the most common
• 12:26
  value. Nothing fancy, but essential.
• 12:29
  Absolutely essential garbage in garbage.
• 12:31
  Out Step 2 is interesting. They call it initial balancing.
• 12:35
  This is a really clever move. Before they even touched the big
• 12:38
  expensive deep learning model, they used the classical methods
• 12:42
  we just talked about, Seals, matter, SMOTE to create a
• 12:45
  roughly balanced data set. Wait, so they use the old tech
• 12:48
  to prep the data for the new tech?
• 12:50
  Why do that? Think of it as priming the
• 12:52
  canvas before you paint. If you feed a drastically
• 12:56
  imbalanced data set like 95 to 5 into again, even a conditional
• 13:00
  Jan might struggle to converge at the beginning.
• 13:02
  It's a hard problem. OK by balancing it roughly with
• 13:05
  eight is 1 first maybe to like a 6040 split?
• 13:08
  They gave the deep CT Yan a much easier starting point, a head
• 13:11
  start. That makes sense.
• 13:12
  And then Step 3, advanced augmentation.
• 13:14
  Right. They feed this primed, roughly
• 13:17
  balanced data into the deep CT Jan but they modified the
• 13:21
  architecture. This is the core innovation.
• 13:24
  I think they added a resnet backbone.
• 13:26
  This is the most technical part of the paper and probably the
• 13:28
  most important. Resnet stands for residual
• 13:31
  networks. We usually hear about this in
• 13:33
  the context of image recognition.
• 13:34
  Resnet 50, Resnet 101. So why are we jamming an image
• 13:38
  recognition architecture into a generator for tabular data?
• 13:43
  It seems like a weird. Fit It's all about solving a
• 13:46
  classic deep learning problem called the vanishing gradient.
• 13:48
  OK, Eli, five time. Give me the simple version.
• 13:51
  What is the vanishing gradient? Imagine you're playing a game of
• 13:54
  telephone, but the line of people is a mile long.
• 13:57
  That's a deep neural network with many layers.
• 13:59
  OK, you whisper a message at the start.
• 14:01
  That's your input data. By the time it gets to the
• 14:04
  person at the very end, the output layer, the message is so
• 14:07
  distorted and quiet and basically gibberish.
• 14:10
  The person at the end has no idea what the original signal
• 14:12
  was. The signal just dies out as it
• 14:14
  passes through all the layers, correct?
• 14:17
  And in a neural network, that signal is the gradient.
• 14:19
  It's the error signal that tells the network how to update its
• 14:22
  weights, how to learn. If that signal dies, the model
• 14:25
  can't learn. It stops updating.
• 14:27
  It gets stuck. And resnet fixes this with skip
• 14:31
  connections. The shortcut.
• 14:33
  That's the perfect way to think of it.
• 14:35
  Instead of passing the message only to the very next person in
• 14:38
  line, you also get to shout it to the person two or three spots
• 14:41
  down the line. So the original information gets
• 14:43
  to jump ahead. Mathematically, instead of a
• 14:45
  layer trying to learn the full complex function FX, it only has
• 14:50
  to learn the residual or the difference FX plus X.
• 14:53
  That plus X is the skip connection.
• 14:55
  It carries the original unmodified input forward.
• 14:58
  So it never forgets where it started, it always has the
• 15:01
  original context. Precisely.
• 15:02
  And for tabular data this is absolutely be crucial because
• 15:05
  the correlations can be very subtle and exist at different
• 15:08
  levels. You have simple patterns like
• 15:10
  older people have higher risk, but also very complex high level
• 15:13
  patterns like this specific enzyme level combined with this
• 15:16
  blood type and this genetic marker equals danger.
• 15:19
  And the skip connections help the model capture both.
• 15:22
  Exactly. It allows the generator to
• 15:23
  capture both the simple low level features and the complex
• 15:27
  high level dependencies without getting lost in the noise of a
• 15:30
  very deep network. It just stabilizes the whole
• 15:33
  training process. OK, I want to get granular on
• 15:35
  the hyper parameters here. The paper actually lists the
• 15:38
  settings they used. So if I'm a data scientist
• 15:41
  sitting at my GPU right now trying to reproduce this for a
• 15:44
  project, what am I typing into my code?
• 15:46
  Get your notebooks out. Yeah, for the core deep CT Jan
• 15:49
  training, they kept the learning rate very, very low .0002.
• 15:54
  That is tiny. The default in most frameworks
• 15:56
  is what .0001? Why go so slow?
• 16:00
  Stability. GNS are notoriously unstable.
• 16:03
  It's a delicate tug of war. If one side, the generator or
• 16:06
  the discriminator, learns too fast, it overpowers the other
• 16:09
  and the whole system falls apart.
• 16:11
  The rope snaps. The model just oscillates and
• 16:13
  never finds a good solution. Right.
• 16:14
  So you have to creep up on the solution.
• 16:16
  Slow and steady adversarial training.
• 16:18
  Makes sense. What about batch size?
• 16:20
  128 That's a pretty classic Goldilocks number.
• 16:22
  It's small enough that it fits in memory and provides a bit of
• 16:25
  stochastic noise, which helps with generalization, but it's
• 16:28
  also large enough to give you a stable, reliable estimate of the
• 16:31
  gradient. And the embedding dimension for
• 16:34
  a categorical features. We use between 128 and 256 and
• 16:37
  they said it depends on the complexity of the data set.
• 16:40
  This is for converting those text based categories like city
• 16:44
  or hospital ID into dense numerical vectors that the math
• 16:48
  can actually work with. Got it.
• 16:49
  Now for the Resonant backbone specifically, what did they use
• 16:53
  there? They used a Resonant 18
• 16:54
  architecture. That's one of the lighter,
• 16:56
  faster versions of Resonant. But the real secret sauce I
• 16:59
  think was the regularization they applied.
• 17:02
  They use dropout at a rate of .3.
• 17:03
  Which means they are randomly turning off 30% of the neurons
• 17:07
  during each training step. Yes, exactly.
• 17:11
  And they also use weight decay at 1 E 5.
• 17:14
  Both of these are absolutely crucial because remember, our
• 17:17
  starting data set of sick patients is tiny, maybe 50 or 60
• 17:21
  people. Deep learning models are
• 17:23
  effectively massive memory machines.
• 17:25
  If you let them, they will just memorize those 50 patients
• 17:28
  perfectly. That's overfitting.
• 17:30
  Extreme overfitting. The model becomes a parrot, not
• 17:33
  a thinker. Dropout forces it to be robust.
• 17:37
  You can't rely on any single neuron, because that neuron
• 17:40
  might be switched off at any time.
• 17:42
  It forces the model to learn the underlying concept of the
• 17:45
  disease, not just the specific patient records in the training
• 17:48
  set. OK.
• 17:49
  So we've built the generator, we've primed it with ATIS YN,
• 17:51
  we've stabilized it with Resnet and we've tuned it with these
• 17:54
  very specific slow hyper parameters.
• 17:57
  We generate our new synthetic data.
• 17:58
  Now we have a nice big balanced data set.
• 18:01
  We still need a classifier to actually diagnose the patients.
• 18:04
  Did they just use something simple like random forest?
• 18:06
  They compared a few. They looked at Random Forest,
• 18:09
  XG, Boost, KNN, but the hands down winner, the one they
• 18:12
  focused on was Tabnet. Tabnet, another deep learning
• 18:15
  model. Why use a deep learning
• 18:17
  sledgehammer when a screwdriver like Random Forest usually works
• 18:20
  just fine for tabular data? Because Tabnet is a deep
• 18:23
  learning model that's specifically designed to beat
• 18:26
  decision trees at their own game, it has an architecture
• 18:30
  that really tries to mimic the way a decision tree, or even a
• 18:33
  human doctor thinks. It uses a mechanism called
• 18:37
  sequential attention. Sequential attention.
• 18:40
  What does that mean in this context?
• 18:43
  Think about how you diagnose a car problem.
• 18:45
  You don't check everything all at once.
• 18:46
  You pop the hood, You check the battery first.
• 18:49
  Is it dead? No.
• 18:51
  OK, now check the starter. Is that making a clicking sound?
• 18:53
  It's a step by step process of elimination.
• 18:56
  Exactly. Tabinet does this.
• 18:58
  It uses a learnable mask at each step to focus on a specific
• 19:01
  subset of features. For the first step, it might
• 19:03
  only look at age and blood pressure.
• 19:05
  Then based on that, for the second step, it decides to look
• 19:08
  at hemoglobin and leukocytes. So it learns not just what's
• 19:11
  important, but in what order to look at things.
• 19:14
  And it learns what to ignore this.
• 19:15
  The other key part uses a very specific activation function
• 19:19
  called antmax. Antmax.
• 19:20
  I've heard of softmax. Softmax is what's typically
• 19:23
  used. It gives a probability score to
• 19:25
  everything, but nothing is ever truly 0A feature might be
• 19:29
  .00001% important at Max, allows for true sparsity.
• 19:35
  It can actually say you know what, these 10 columns in the
• 19:38
  spreadsheet. They're completely irrelevant
• 19:40
  for this decision probability. Zero.
• 19:42
  It effectively deletes the noise from the calculation.
• 19:44
  Which is huge. It makes the model much more
• 19:46
  efficient and critically, more interpretable.
• 19:49
  It tells you exactly which features it chose to ignore.
• 19:52
  So that's the full stack, a two element baiting feeding into a
• 19:55
  deep CTGN with a resnet backbone which generates data to train a
• 19:59
  tab net classifier. That's the recipe.
• 20:01
  Now the big question, does it actually work?
• 20:04
  Or is this just a cool engineering science project that
• 20:06
  looks good on paper? The validation results are
• 20:08
  honestly startling, but first we have to define how they tested
• 20:12
  it. This is a hell I will die on.
• 20:13
  You have to use TSTR. Trade on synthetic, test on
• 20:16
  real. It is the golden rule of
• 20:18
  synthetic data validation. You train your model only on the
• 20:22
  augmented or fully synthetic data, but you must must must
• 20:26
  test it on a holdout set of real flesh and blood patients that
• 20:30
  the model has never seen before. Because if you test on your own
• 20:33
  synthetic data, you're just grading your own homework.
• 20:35
  Exactly. It's a meaningless self
• 20:37
  congratulatory loop, but if a model that was trained entirely
• 20:41
  on fake data can accurately diagnose a real patient it has
• 20:46
  never met. And you know, the synthetic data
• 20:48
  successfully captured the fundamental underlying rules of
• 20:51
  the disease. That's the proof.
• 20:53
  So what were the scores? What did they find?
• 20:55
  OK. For the chronic kidney disease
• 20:56
  data set using the full hybrid framework, they achieved 99.4%
• 21:01
  accuracy. Wow, that is that's basically
• 21:03
  perfect. What about the others?
• 21:04
  For COVID-19 they hit 99.2% and for Ding Fever 99.5%.
• 21:10
  That's incredible. How did the more traditional
• 21:12
  models compare on the same tasks?
• 21:13
  They were good, don't get me wrong, but Tabnet consistently
• 21:16
  beat Random Forest and XG boost across the board on these
• 21:19
  augmented data sets. But the most interesting failure
• 21:22
  to me was KNN. Key nearest neighbors, the one
• 21:25
  that works based on distance. Right, it's a very simple
• 21:27
  algorithm and in some of their experiments KNN actually
• 21:30
  performed worse when they added the synthetic data.
• 21:33
  How is that possible? How does more data make a model
• 21:36
  worse? Because KNN relies purely on
• 21:39
  geometric distance and synthetic data, even very high quality
• 21:43
  data like this introduces a tiny, tiny bit of noise.
• 21:47
  It's not perfect, it fuzzes the decision boundaries just
• 21:50
  slightly. And KNN is super sensitive to
• 21:53
  that. Very sensitive.
• 21:54
  It got confused by these new fake neighbors that weren't
• 21:57
  placed perfectly in the geometric space.
• 21:59
  It started making mistakes it wouldn't have made before.
• 22:01
  That is a huge practical take away for anyone listening.
• 22:04
  If you are going to implement this kind of synthetic strategy,
• 22:07
  you might need to upgrade your classifier too.
• 22:10
  You can't just plug this into a legacy KNN system and expect
• 22:13
  magic to happen. You really can't.
• 22:15
  You need a more robust classifier like tab net that can
• 22:18
  use an attention mechanism to filter out that slight noise and
• 22:21
  find the real signal. There was one other metric in
• 22:23
  the paper I wanted to ask about, the similarity score.
• 22:27
  They said they achieved between 84 and 87% right?
• 22:31
  Which sounds like AB Plus why wouldn't we want 100%
• 22:34
  similarity? That's a great question because
• 22:37
  it's counterintuitive. If your synthetic data is 100%
• 22:40
  similar to your real data, you've just cloned it.
• 22:42
  You've perfectly memorized the training set.
• 22:45
  Which is a privacy violation and it's terrible for
• 22:48
  generalization. Exactly.
• 22:49
  You haven't learned anything new.
• 22:50
  You want the synthetic data to be close enough to capture the
• 22:54
  statistical properties the physics of the system, but
• 22:57
  different enough to be novel and to cover new ground. 87% is
• 23:01
  actually a very healthy score. Means you're in the sweet spot.
• 23:04
  So the model is incredibly accurate, but in medicine,
• 23:08
  accuracy isn't enough. I can't just walk into a
• 23:10
  patient's room and say, well, the computer says you're sick,
• 23:12
  so bye. I mean to know why?
• 23:14
  The black box problem. Yes.
• 23:17
  This brings us to the fifth section of the paper,
• 23:19
  Interpretability. To open up that black box, they
• 23:22
  used SHAP values. SHAP Shapley additive
• 23:27
  explanations. It comes from game theory,
• 23:28
  doesn't it? It does, yeah.
• 23:30
  It basically treats the model's prediction like a cooperative
• 23:33
  game where every feature, every symptom, every lab result is a
• 23:38
  player on a team. OK?
• 23:40
  And it calculates how much each individual player contributed to
• 23:43
  the final score, to the final diagnosis.
• 23:46
  It tells you who the MVP was for that prediction.
• 23:48
  So it gives us a clinical reality check.
• 23:51
  The ultimate reality check If your model is 99% accurate, but
• 23:55
  it tells you the most important feature for diagnosing cancer is
• 23:57
  the patient ID number, you know something is fundamentally
• 24:01
  broken. You know it's garbage.
• 24:02
  So let's look at what the model actually learned for the
• 24:05
  COVID-19 data set. What was the top predictor
• 24:07
  according to SHAP? The top feature was leukocytes.
• 24:10
  White blood cells. Exactly.
• 24:12
  A high white blood cell count indicates an infection.
• 24:15
  It's one of the most basic tenets of medicine.
• 24:17
  So that's biology one O 1 the model learned the right.
• 24:19
  Thing it validates that the synthetic data preserved the
• 24:22
  true underlying biological signal for kidney disease.
• 24:26
  The top 2 features it flagged were hemoglobin and hematocrit.
• 24:30
  OK, so low hemoglobin means anemia, and kidneys produce a
• 24:34
  hormone erythropoietin that tells the body to make red blood
• 24:38
  cells. So if your kidneys fail, you
• 24:40
  can't make that hormone and you become anemic.
• 24:42
  See, the model figured out the entire renal anemia connection
• 24:46
  just by looking at the synthetic data.
• 24:48
  It learned real medicine. But the dengue data set, that
• 24:52
  one had a really interesting curveball.
• 24:54
  Oh. What was it?
• 24:55
  The top predictor wasn't a blood test.
• 24:57
  It wasn't a symptom. It was the column labeled year
• 25:01
  the. Year like the calendar year the
• 25:03
  patient was seen 2018-2019. And at first you might think,
• 25:07
  OK, that's a data leak, that's a bias, that's a mistake.
• 25:10
  But then you should think about what dengue fever is.
• 25:12
  It's an outbreak disease. It's carried by mosquitoes.
• 25:14
  It's epidemiological, it comes in waves.
• 25:16
  It's seasonal encyclical. The model learned that if a
• 25:19
  patient walks into the clinic during 2019, which is a big
• 25:22
  outbreak year, the data set their prior probability of
• 25:25
  having dengue is way, way higher than if they walked in during
• 25:28
  2018. That is fascinating.
• 25:30
  It didn't just learn Physiology from the synthetic data, it
• 25:34
  learned epidemiology. It learned about public health
• 25:37
  trends. And it also found that the
• 25:39
  remarks column, which was this unstructured text notes for the
• 25:42
  doctors, was highly predictive. Which Tabnet is well suited to
• 25:46
  handle because of the embedding layers we talked about.
• 25:48
  It shows that the deep CTGN was able to encode these really
• 25:52
  complex nonlinear relationships between calendar year, the free
• 25:58
  form text notes and the final diagnosis.
• 26:00
  It's incredible, really. So we have a working system,
• 26:03
  it's accurate, it's interpretable, but I want to
• 26:06
  bring this back down to earth. This is the practical AI digest.
• 26:09
  There is no free lunch in engineering.
• 26:12
  What is the cost? What's the trade off for this
• 26:14
  fancy hybrid framework? The cost is time and compute and
• 26:17
  the paper is very transparent about this which is great to
• 26:20
  see. They timed it.
• 26:21
  Training a standard Smoke plus a Tabnet model took them less than
• 26:24
  10 minutes. So, quick coffee break and
• 26:25
  you're done, right? But training the full hybrid
• 26:28
  model, the ADCNN pre balancing, the deep CDD GAN with a resnet
• 26:31
  backbone and then the Tabnet classifier, that took 42
• 26:34
  minutes. So that's more than a four fold
• 26:36
  increase in training time. It is now.
• 26:39
  In the grand scheme of things, 42 minutes isn't long if you're,
• 26:42
  say, training ChatGPT from scratch.
• 26:45
  But in a clinical production environment where you might be
• 26:48
  retraining these models every night on new data, that adds up.
• 26:52
  So you are paying in GPU hours for this extra couple of
• 26:56
  percentage points of accuracy. You are, and you have to ask if
• 26:59
  it's worth it. For life threatening diseases
• 27:02
  like these, I would argue absolutely yes, that extra 1 or
• 27:05
  2% in accuracy translates to real people not getting
• 27:09
  misdiagnosed. But if you're using this to
• 27:11
  predict, say, customer churn, maybe the simpler, faster SOD is
• 27:15
  good enough. The Blue Gen.
• 27:16
  AI source had some great implementation advice here too.
• 27:19
  They talked about the point of diminishing returns.
• 27:21
  It's. So crucial for the builders out
• 27:23
  there. Don't just turn on the generator
• 27:25
  and create a billion new rows of data.
• 27:27
  It doesn't work that way. Blue Gen. suggests starting by
• 27:30
  generating 2 to five times your original minority data set size.
• 27:34
  I have 1000 real rows of the rare disease I should aim to
• 27:37
  generate, maybe 2000 to 5000 synthetic ones.
• 27:40
  Exactly. The study showed that if you go
• 27:43
  much beyond that, you aren't really adding new information
• 27:45
  anymore. You're just adding noise and
• 27:48
  increasing your training time for no benefit.
• 27:50
  The accuracy starts to plateau. You have to find that sweet
• 27:53
  spot. And what about their hybrid
• 27:55
  workflow idea? This is just best practice for
• 27:57
  safety and reliability. Use the synthetic data for
• 28:00
  training. Use it to cover all those weird
• 28:03
  edge cases, the Leicester City moments we talked about, but
• 28:05
  never let synthetic data leak into your final validation set.
• 28:09
  Your final exam must be on real world problems.
• 28:12
  Always, and you have to constantly monitor for
• 28:15
  hallucinations. If your similarity score starts
• 28:18
  to drop or your TFTR performance tanks, it might mean your
• 28:22
  generator has drifted and is inventing relationships that
• 28:25
  don't exist in reality. You need those automated checks
• 28:28
  before you deploy anything to a real clinic.
• 28:30
  So to summarize the entire stack we've uncovered today, we have a
• 28:33
  fundamental problem. Imbalance causes lazy AI that
• 28:37
  misses the most critical cases. We have a solution, the hybrid
• 28:41
  framework. We prep the data with a classic
• 28:43
  tool like Adis Wyan. We generate new data with a
• 28:47
  powerful conditional deep CT jam that's been stabilized by Resnet
• 28:51
  skip connections, and then we classify with a smart, attentive
• 28:54
  model like Tabnet. And we validate everything with
• 28:57
  TSTR. It's a robust, modern and
• 29:01
  frankly very effective pipeline. As we wrap up, I want to leave
• 29:05
  our listeners with the thought that came from the conclusion of
• 29:07
  the paper. They mentioned real time
• 29:09
  streams. This felt like a genuine glimpse
• 29:10
  since the near future. This is where it gets really
• 29:12
  exciting. Beyond just a single static data
• 29:14
  set. And imagine a synthetic data
• 29:16
  generator that isn't a script you run once a year.
• 29:19
  Imagine it's a live running service.
• 29:21
  So it's connected directly to the hospital's intake feed.
• 29:24
  Exactly. Every single night it
• 29:26
  automatically ingests the day's new patient cases.
• 29:29
  And if a new strain of a virus starts to appear, like we all
• 29:32
  saw happen with the COVID variance, or if the common
• 29:34
  symptoms of Denguing start to shift slightly because of a
• 29:37
  climate change factor. The generator would see that new
• 29:40
  pattern emerging in the real time data.
• 29:42
  And it would immediately generate thousands of new
• 29:45
  simulated cases of this new pattern.
• 29:48
  It would then use that data to retrain the main diagnostic
• 29:51
  model all overnight automatically.
• 29:54
  So by the time the doctors clock in for the morning shift, the AI
• 29:57
  has already practiced on thousands of cases places of the
• 29:59
  new strain before the human doctors have even seen their
• 30:03
  second real patient of the day. Precisely.
• 30:06
  It turns the AI from a historian, something that only
• 30:09
  looks at the past, into a scout. It can adapt to a changing
• 30:13
  disease faster than a human run public health system can.
• 30:16
  That is slightly terrifying, but also incredibly hopeful.
• 30:20
  It's the future of adaptive real time diagnostics is where this
• 30:24
  is all heading. So here is your call to action
• 30:26
  for the week. If you're an engineer sitting on
• 30:28
  an imbalance data set, and I know you are because basically
• 30:31
  all real world data sets are, try implementing Deepctgen.
• 30:35
  Don't just settle for the old SEM OT standby.
• 30:38
  But please don't forget the resnet skip connections or your
• 30:41
  gradients will vanish into thin air and you'll spend a week
• 30:43
  debugging. It and always always check your
• 30:45
  feature importance if patient ID is your top predictor.
• 30:49
  Delete your model, delete your features and start over from
• 30:52
  scratch. Wise words.
• 30:54
  Thanks for listening to this deep dive.
• 30:56
  We'll catch you on the next one. Goodbye and Abby Building.