Latest / The Practical AI Digest / Graph Neural Networks: Learning from Connections, Not Just Data

Transcript

Graph Neural Networks: Learning from Connections, Not Just Data · The Practical AI Digest

• 0:00
  Welcome to the Deep Dive. We're the show that tries to cut
• 0:02
  through the hype and get to the core of complex research, making
• 0:06
  it clear and hopefully usable. Yeah, making it actionable.
• 0:10
  Exactly. And today we're diving into
• 0:12
  something really fascinating in AI.
• 0:15
  Think about this for a second. So much of the world isn't neat
• 0:18
  rows and columns. Oh, definitely not.
• 0:20
  Think social networks, supply chains, even biological systems.
• 0:25
  Right. Or how chemicals bond together.
• 0:27
  Or traffic flowing through a city.
• 0:29
  It's all interconnected. It forms these intricate webs,
• 0:32
  these graph structures. Precisely.
• 0:35
  And that kind of data, this graph structure data, it's
• 0:38
  historically been tricky for standard machine learning.
• 0:41
  Yeah, models that love grids, like for images, or sequences
• 0:44
  like for text. They kind of stumble when things
• 0:47
  get messy and interconnected like this, don't they?
• 0:49
  You really do. It's like trying to understand a
• 0:52
  city by only looking at isolated building blueprints.
• 0:56
  You miss the roads, the utilities, the people moving
• 0:58
  between them. You miss the connections.
• 0:59
  That's a great way to put it. And that brings us right to our
• 1:02
  topic graph neural networks, or GNS.
• 1:06
  These are really a different breed of AI, aren't they?
• 1:08
  Absolutely. They're a branch of AI and
• 1:10
  machine learning specifically designed to understand and,
• 1:15
  crucially, make predictions on this kind of interconnected
• 1:18
  data. They finally let us make sense
• 1:20
  of these complex networks. So our mission today for this
• 1:23
  deep dive is to really unpack GNNS.
• 1:25
  We want to get into not just what they are, but why they're
• 1:28
  so powerful. Yeah, and how they actually work
• 1:30
  under the hood. We'll look at some real world
• 1:32
  examples too, things you might already be using without
• 1:34
  realizing it. And the goal here is clarity.
• 1:37
  We want you to walk away with a solid conceptual grasp, maybe
• 1:40
  even enough detail that if you're an engineer, you can
• 1:43
  start thinking about how to apply these ideas.
• 1:45
  Sounds good, where should we start?
• 1:46
  Let's start right at the beginning.
• 1:48
  We say graph neural network. What does the neural network
• 1:51
  part actually do with the graph? How do they come together?
• 1:54
  OK, so AGNN at its heart is a neural network, but it's built
• 1:58
  specifically for data structured as a graph.
• 2:01
  Remember, graph is just entities.
• 2:02
  We call them nodes connected by relationships.
• 2:05
  We call those edges. Like people and friendships on
• 2:08
  Facebook or airports and flight paths.
• 2:10
  Exactly. People are nodes.
• 2:12
  Friendships are edges. Airports are nodes, flight
• 2:15
  routes are edges. The neural network part comes in
• 2:18
  because GNNS learn representations of these nodes
• 2:22
  and edges. They learn how to process these
• 2:24
  complex, often non Euclidean structures.
• 2:27
  Non Euclidean, meaning they don't fit neatly onto a flat
• 2:30
  plane or a straight line like a grid or sequence does.
• 2:33
  Precisely, they handle the irregular complex shapes of real
• 2:36
  world networks. OK, so the graph gives us the
• 2:39
  structure. How does the GNN actually learn
• 2:41
  from all those connections? What's the mechanism?
• 2:44
  This is the really cool part. The core idea is usually called
• 2:47
  message passing. Imagine each node in the network
• 2:50
  as having some initial information, some features.
• 2:52
  Like a user's profile info or an Adam's type.
• 2:55
  Right then, in each step of the GNN, nodes essentially send
• 3:00
  messages derived from their current features across the
• 3:03
  edges to their direct neighbors. OK, so they're talking to their
• 3:06
  neighbors. Exactly.
• 3:07
  And then each node receives all the messages from its neighbors
• 3:10
  and it aggregates them, combines them in some way, often with its
• 3:14
  own existing information. Aggregates like summing them up
• 3:17
  or averaging? Yeah, it could be summing,
• 3:19
  averaging, maybe something more complex.
• 3:21
  We'll get into different ways later.
• 3:22
  But the key is this iterative process.
• 3:25
  Send messages, receive messages, update your own state based on
• 3:28
  those messages. And this happens over multiple
• 3:30
  steps or layers. Typically, yes.
• 3:33
  Multiple layers allow information to propagate further
• 3:36
  out across the graph. A node doesn't just learn from
• 3:39
  its immediate neighbors, but eventually from its neighbors,
• 3:41
  neighbors, and so on. So it builds up a richer picture
• 3:44
  of its place in the wider network.
• 3:46
  Exactly. It learns A representation that
• 3:48
  captures not just its own features, but also the structure
• 3:52
  of its local neighborhood and potentially even global
• 3:55
  properties of the entire graph. It's context aware.
• 3:58
  That makes intuitive sense. Why is this message passing
• 4:01
  approach so important? Why not just use the node
• 4:04
  features alone? Because in so many real world
• 4:07
  problems, the connections are the crucial information.
• 4:11
  Traditional models that ignore the graph topology, the specific
• 4:15
  pattern of connections, miss out on a huge source of signal.
• 4:19
  Like trying to predict if someone will click an ad without
• 4:21
  knowing what their friends like. Perfect example, GN NS
• 4:24
  explicitly leverage that topology.
• 4:27
  They use the connections to inform the learning process and
• 4:30
  this makes them incredibly effective for specific tasks.
• 4:32
  Such as? Well, things like node
• 4:35
  classification. You predict a property for each
• 4:37
  node. Is this bank transaction
• 4:39
  fraudulent? Is this protein involved in a
• 4:41
  specific disease? OK, classifying individual items
• 4:44
  within the network. Right.
• 4:46
  Or link prediction. Can we predict if a new
• 4:48
  connection is likely to form? Will these two users become
• 4:51
  friends? Will this drug interact with
• 4:53
  this target protein? Predicting future relationships
• 4:55
  makes sense for recommendations too.
• 4:57
  Definitely, and also graph classification.
• 5:00
  Here you're classifying the entire graph.
• 5:02
  Is this molecule likely to be toxic?
• 5:04
  Does this social network exhibit community structure?
• 5:08
  Yeah, typical of, say, political polarization.
• 5:11
  Wow, OK, so node link and graph level predictions, that's quite
• 5:15
  versatile. Can we ground this in some real
• 5:17
  world examples you mentioned? We might be interacting with
• 5:19
  them already. Absolutely.
• 5:21
  Let's talk about drug discovery first.
• 5:22
  This is a huge one. Think about a molecule.
• 5:24
  It's a natural graph, isn't it? How so?
• 5:27
  Well, the atoms are the nodes, the chemical bonds between them
• 5:30
  are the edges. I see.
• 5:32
  And you could add features right?
• 5:33
  Like the type of atom carbon, oxygen as node features.
• 5:36
  Exactly, Atom type charge, things like that and even edge
• 5:39
  features like is it a single bond or double bond?
• 5:42
  Okay, so you have this molecular graph.
• 5:44
  What can a GNN do with it? It can learn to predict the
• 5:48
  molecule's properties directly from its structure.
• 5:51
  Will it dissolve in water? Will it bind to a certain
• 5:53
  receptor? Will it kill bacteria?
• 5:56
  And that leads to things like finding new drugs.
• 5:59
  Precisely. The Halocen story is probably
• 6:01
  the most famous example. Here, researchers use GN NS to
• 6:04
  screen a massive library of existing chemical compounds.
• 6:08
  OK. They train the GNN to predict
• 6:10
  whether a molecule would inhibit the growth of bacteria like E
• 6:13
  coli, and the model flagged a compound called Halocen.
• 6:17
  Which wasn't known as an antibiotic before.
• 6:19
  No. It had been investigated for
• 6:21
  diabetes I think, but it's antibiotic potential was
• 6:24
  completely overlooked by traditional methods.
• 6:26
  The GNN saw something in its structure and it turned out to
• 6:29
  be a potent broad spectrum antibiotic.
• 6:31
  Wow, so the GNN literally helped discover a new antibiotic
• 6:34
  candidate just by looking at the graph structure of molecules.
• 6:38
  It's a prime example of how GNNS can accelerate scientific
• 6:42
  discovery. It's really transforming parts
• 6:44
  of chem, informatics and drug development.
• 6:46
  That's genuinely game changing. What's another area you
• 6:48
  mentioned? Something we use daily.
• 6:50
  Yeah, navigation. Think about Google Maps or
• 6:52
  similar apps. How do they estimate your
• 6:54
  arrival time? Your ETA.
• 6:57
  I guess they look at traffic speeds, distance.
• 6:59
  Right, but how do they combine all that information across a
• 7:02
  huge complex Rd. network that's constantly changing?
• 7:05
  OK, I see where you're going. The road network is a graph.
• 7:08
  Exactly. Intersections are nodes, roads
• 7:10
  are edges, and you have features on those edges.
• 7:13
  Rd. length, speed limit, current traffic speeds, maybe historical
• 7:17
  traffic patterns. So the GNN processes this
• 7:19
  constantly evolving graph of traffic.
• 7:21
  Precisely. Google actually deployed GN NS
• 7:24
  to significantly improve their ETA predictions.
• 7:28
  They found that GN NS could model the complex dependencies
• 7:31
  how congestion on one Rd. effects surrounding areas much
• 7:34
  better than previous methods. And did it make a difference?
• 7:37
  A huge difference. They reported reductions in
• 7:40
  negative user outcomes, basically wildly inaccurate ET
• 7:43
  as by over 40% in cities like Sydney and Taichung after
• 7:47
  deploying GN NS. 40% fewer bad ET as I definitely noticed that.
• 7:51
  That's impressive. It really is.
• 7:52
  Shows how GNNS can handle dynamic large scale graph data
• 7:56
  effectively. And I bet recommendation systems
• 7:58
  are another big one, like suggesting products or content.
• 8:01
  You got it. Recommendations are a natural
• 8:04
  fit. You have users, you have items,
• 8:07
  products, movies, pins, whatever.
• 8:08
  And you have interactions between them.
• 8:10
  Purchases, ratings, clicks. That's a graph.
• 8:13
  A bipartite graph, often users connected to items.
• 8:17
  Yes, or even more complex graphs involving user user connections
• 8:20
  too. GN NS can learn embeddings for
• 8:23
  users and items based on this graph structure, leading to
• 8:26
  better recommendations. Any specific examples?
• 8:29
  Pinterest's Pin Sage models a well known one.
• 8:32
  They built a GNN to operate on their massive graph.
• 8:35
  Billions of nodes, billions of edges connecting users, pins and
• 8:39
  boards. Billions.
• 8:40
  How do they even compute on a graph that big?
• 8:43
  That's where clever engineering comes in.
• 8:44
  Pin Sage uses techniques based on something called Graph Sage.
• 8:48
  The core idea is instead of trying to aggregate messages
• 8:51
  from all of a node's neighbors, which could be thousands or
• 8:53
  millions on Pinterest, you intelligently sample a smaller
• 8:56
  subset of neighbors. So you get an approximation of
• 9:00
  the neighborhood information without the massive
• 9:01
  computational cost. Exactly.
• 9:04
  It allows GNNS to scale to web size graphs.
• 9:07
  Penn Sage uses this to learn embeddings that capture visual
• 9:11
  and semantic similarity, powering their recommendations.
• 9:14
  OK, these examples, drug discovery maps, recommendations
• 9:17
  that they really drive home the why.
• 9:19
  Now let's dig deeper into the how.
• 9:21
  What's actually going on inside a GNN layer?
• 9:24
  You mentioned message passing, but how does that translate
• 9:27
  mathematically? Right, let's get a bit more
• 9:29
  technical. A useful analogy is to think
• 9:31
  about convolutional neural networks, CNNS, which are so
• 9:35
  dominant in computer vision. OK, CN NS use kernels or filters
• 9:39
  that slide across an image. Exactly.
• 9:41
  An image is basically a grid, a very regular graph.
• 9:44
  ACNN kernel is a small matrix of learnable weights.
• 9:47
  It slides across the image looking at a small patch of
• 9:50
  pixels, a local neighborhood at a time, and computes a feature
• 9:54
  based on that patch. It explains the fact that the
• 9:57
  same pattern might appear anywhere.
• 9:59
  So it captures local patterns and because the grid is regular,
• 10:02
  the same kernel works everywhere.
• 10:03
  Precisely. The challenge for GN NS is to
• 10:06
  generalize that idea. Create an operator that acts on
• 10:09
  a node's local neighborhood. But for graphs that are
• 10:12
  irregular, node neighborhoods can vary wildly in size and
• 10:16
  structure. So we can't just slide a fixed
• 10:18
  size kernel around. Nope, We need something more
• 10:21
  flexible. So if I'm an engineer designing
• 10:23
  one of these layers, what properties am I aiming for?
• 10:26
  What makes a good GNN layer? Good question.
• 10:29
  You are a few key things. One, efficiency computation and
• 10:34
  memory usage should ideally scale reasonably, maybe linearly
• 10:37
  or near linearly with the number of nodes and edges.
• 10:40
  Makes sense, you don't want it to blow up on large graphs.
• 10:42
  Two parameter sharing. Like ACNN kernel, you want a
• 10:46
  fixed number of learnable parameters regardless of the
• 10:49
  graph size. The same layer should work on a
• 10:51
  small graph or a massive 1. OK.
• 10:53
  So it learns general principles of neighborhood interaction. 3
• 10:56
  Locality. The operation should primarily
• 10:59
  focus on nodes. Immediate neighborhood, say it's
• 11:01
  direct connections, one hop neighbors.
• 11:03
  Deeper layers can then capture wider context.
• 11:06
  Similar to how CNN's build up complexity layer by layer. 4
• 11:11
  Ability to handle varying neighborhood sizes and maybe
• 11:14
  assigned different importance to different neighbors.
• 11:16
  Not all connections are equal, right?
• 11:18
  Some friends influence you more than others.
• 11:20
  And five, inductive capability. This is crucial.
• 11:24
  The model should be able to generalize to graphs it hasn't
• 11:27
  seen during training. It should learn structural
• 11:29
  patterns, not just memorize the training graph.
• 11:31
  That's a demanding list. How do we actually achieve this
• 11:33
  mathematically? What's the core operation?
• 11:36
  The fundamental idea boils down to combining 2 things, the nodes
• 11:40
  features and the graph structure.
• 11:42
  The connections. Let's represent all the node
• 11:44
  features as a matrix, say H Each row is a node, each column is a
• 11:48
  feature dimension, and let's represent the graph structure
• 11:51
  using the adjacency matrix A. This is a matrix where aij is 1
• 11:57
  if node I is connected to node J and 0 otherwise.
• 12:00
  Standard graph representation. The simplest GNN like operation
• 12:03
  might look something like this. You multiply the adjacency
• 12:06
  matrix A by the feature matrix H.
• 12:08
  What does AH do? Let's for each node I it sums up
• 12:12
  the feature vectors H of all nodes J that are connected to I.
• 12:15
  Exactly. It aggregates the features from
• 12:17
  the neighbors. Then you typically multiply this
• 12:20
  result by a learnable weight matrix W to transform the
• 12:23
  features and often apply a non linear activation function like
• 12:27
  real U, So something like real UAHW.
• 12:30
  OK, that seems like a basic message passing step neighbors
• 12:34
  features summed and transformed. But wait, you mentioned earlier
• 12:38
  that a only defines connections to other nodes.
• 12:41
  Does AH include the nodes own features?
• 12:43
  Excellent point. Usually the standard adjacency
• 12:46
  matrix A has zeros on the diagonal, meaning nodes aren't
• 12:49
  connected to themselves. If you just use AH, a node only
• 12:53
  gets information from its neighbors, potentially losing
• 12:55
  its own original information after the transformation.
• 12:57
  Which seems like a bad idea. You want the node to remember
• 13:00
  itself. Definitely so.
• 13:02
  A very common and important practical trick is to add self
• 13:05
  loops to the graph before doing the aggregation.
• 13:08
  Mathematically this is super simple.
• 13:09
  You just add the identity matrix I which has ones on the diagonal
• 13:12
  and zeros elsewhere, to the adjacency matrix A.
• 13:15
  So instead of a you use a plus I.
• 13:18
  Precisely. Now when you compute A+ IH, each
• 13:22
  node sums the features of its neighbors plus its own features.
• 13:26
  It's a simple tweak, but crucial for stable learning in most GN
• 13:30
  NS. Got it.
• 13:31
  Add self loops using a plus I. So this basic real UA plus IHW
• 13:37
  operation forms the foundation. Are there different variations
• 13:41
  on this theme? Different ways to do the
• 13:42
  aggregation. Absolutely.
• 13:44
  That basic operation gives rise to the first type of GNN layer
• 13:47
  we can discuss, which we could call some pulling.
• 13:49
  OK some pulling because it literally just sums major
• 13:52
  features plus self. Exactly.
• 13:54
  He knew Ray LUA plus IHW Straightforward, but it has a
• 14:00
  potential practical problem. What's that?
• 14:02
  Imagine a node with thousands of neighbors, like a celebrity on a
• 14:05
  social network. When you sum up all their
• 14:07
  transformed features, the resulting vector, and he's new
• 14:10
  for that node can become huge in magnitude.
• 14:13
  The scale can explode just because of the number of
• 14:15
  connections. Right.
• 14:16
  This can make the gradients during training very large or
• 14:18
  unstable, making it hard for the model to learn effectively.
• 14:21
  So some pulling is simple but potentially unstable for graphs
• 14:25
  with highly varied node degrees. How do we fix that?
• 14:29
  The most common fix leads to our second type mean pulling.
• 14:32
  Instead of just summing, we average.
• 14:34
  OK, how do you implement averaging using matrix
• 14:37
  operations? You normalize the A plus I
• 14:40
  matrix. For each node, you divide the
• 14:42
  contribution from its neighbors and a self by its degree, the
• 14:45
  number of neighbors plus one. For the self loop,
• 14:48
  mathematically, you multiply A + I by the inverse of the degree
• 14:51
  matrix. DD is a diagonal matrix with no
• 14:55
  degrees on the diagonal, so the operation becomes something like
• 14:58
  real you D1A plus IHW. So D1 handles the division by
• 15:04
  the degree for each node. That's right, it effectively
• 15:07
  computes the mean of the features from the neighborhood,
• 15:09
  including the node itself. This prevents the scale
• 15:11
  explosion issue because you're averaging, not summing.
• 15:14
  And this is generally more stable.
• 15:15
  Much more stable. Mean pulling is a really solid,
• 15:18
  simple baseline. It works well especially for
• 15:20
  inductive problems where you need to generalize to new unseen
• 15:23
  graphs. OK.
• 15:24
  Some pulling then mean pulling for stability.
• 15:27
  You mentioned a really popular one earlier, GCM.
• 15:30
  Yes, the Graph Convolutional Network GCN layer introduced by
• 15:33
  Kipps and Welling. This is probably the most cited
• 15:36
  and widely used GNN layer currently.
• 15:38
  What's its specific innovation? Is it different from mean
• 15:41
  pulling? It uses a slightly different,
• 15:43
  more sophisticated normalization of the adjacency matrix.
• 15:46
  Instead of just D1A plus I, it uses what's called symmetric
• 15:50
  normalization, D12A plus ID 12. D2 the minus half power.
• 15:56
  What does that achieve? Intuitively, it bounces.
• 15:58
  The influence mean pulling D1A normalizes based on the receiver
• 16:02
  nodes degree. Symmetric normalization
• 16:04
  considers both the senders and receivers degrees.
• 16:07
  It tends to work very well empirically, often slightly
• 16:10
  outperforming mean pulling. Why do you think it's so
• 16:12
  popular? It hits a sweet spot.
• 16:14
  It's simple, computationally efficient, scalable and perform
• 16:17
  strongly on many benchmark tasks.
• 16:20
  It became a go to layer for many researchers and practitioners.
• 16:22
  OK, GCN uses that symmetric normalization.
• 16:26
  What if the edges themselves have important features like the
• 16:29
  type of chemical bond or the strength of a connection?
• 16:33
  Some MEAN and GCN seem focused on node features and
• 16:37
  connectivity. Great point that brings to a
• 16:39
  more general framework message passing neural networks and
• 16:43
  PNNS. OK.
• 16:44
  Sounds more fundamental. It is, and PNNS define the
• 16:46
  process more explicitly. They typically have 3
• 16:49
  components. 1A Message function. This computes a message for each
• 16:53
  edge, potentially considering the features of the sending
• 16:55
  node, the receiving node, and any features of the edge itself.
• 16:59
  2 An aggregation function. This takes all the incoming
• 17:03
  messages for a node from its neighbors and aggregates them.
• 17:06
  For example some mean Max three an update function or read out.
• 17:11
  This combines the aggregated message with the nodes current
• 17:13
  feature vector to produce the nodes new feature vector for the
• 17:16
  next layer. So it explicitly models message
• 17:18
  creation based on sender, receiver and edge features.
• 17:21
  That sounds much more expressive.
• 17:22
  It is very expressive. If you have rich edge
• 17:24
  information, MPN ends are often the way to go.
• 17:27
  However, there's a potential downside.
• 17:29
  Computing and storing potentially complex message
• 17:31
  vectors for every single edge can become very memory
• 17:34
  intensive, especially on large dense graphs.
• 17:38
  They can also sometimes be prone to overfitting if you don't have
• 17:40
  enough data to learn the complex message functions well.
• 17:43
  So powerful, but potentially resource hungry and data hungry.
• 17:47
  Best for maybe smaller graphs with really informative edges.
• 17:50
  That's often where they shine, yes, like in molecular graphs
• 17:53
  where bond types are critical. OK, we've covered summing,
• 17:57
  averaging, symmetric normalization, and explicit
• 18:00
  message passing. What about the idea of learning
• 18:03
  the importance of neighbors? GCN and mean pulling?
• 18:06
  Treat all neighbors after normalization somewhat equally,
• 18:10
  right? That's a key limitation they
• 18:12
  have, and that's exactly what Graph Attention networks Gats
• 18:15
  were designed to address. Attention like in Transformers.
• 18:18
  Very much inspired by the success of attention in sequence
• 18:20
  models. Yes.
• 18:22
  The core idea in GAP is instead of using a fixed normalization
• 18:25
  like mean or symmetric, let the model learn how important each
• 18:28
  neighbor is for updating a given node.
• 18:30
  Just learn that. For each edge connecting node J
• 18:33
  to node ijat computes an attention coefficient, let's
• 18:36
  call it Alfie. This coefficient represents the
• 18:39
  importance of node J's features to node I.
• 18:42
  And how is Alfie calculated? It's typically calculated using
• 18:45
  a small neural network and attention mechanism that takes
• 18:48
  the features of node I and node J as input.
• 18:52
  This mechanism is learned during training.
• 18:54
  So the network learns a function to decide attention scores.
• 18:57
  Exactly. Then these raw attention scores
• 19:00
  for all of no dies neighbors are usually normalized using a soft
• 19:04
  Max function so they sum to one. This gives you a weighted
• 19:08
  average of neighbor features, where the weights Alfie are
• 19:11
  learned by the model itself. So it's like mean pulling, but
• 19:15
  instead of a simple average, it's a weighted average where
• 19:17
  the weights are learned attention scores.
• 19:19
  Precisely this give the model much more flexibility.
• 19:22
  It can learn to pay more attention to relevant neighbors
• 19:24
  and ignore irrelevant ones for a specific task.
• 19:27
  That sounds really powerful. Is it as computationally
• 19:29
  expensive as MPN NS? Generally less so than the most
• 19:33
  complex MPN NS. While it computes an attention
• 19:36
  score per edge, that score is usually just a scalar value, not
• 19:40
  a high dimensional vector like some MPN messages.
• 19:43
  This makes it more scalable. Often people use multi head
• 19:46
  attention, computing several sets of attention scores in
• 19:49
  parallel and combining them, which helps stabilize training
• 19:53
  and capture different aspects of neighborhood importance.
• 19:55
  Multi head attention just like in Transformers.
• 19:58
  That connection seems really strong now.
• 20:00
  It is in fact you can view a transformer as a special kind of
• 20:03
  GNN operating on a fully connected graph.
• 20:06
  Fully connected, meaning every input token like a word is
• 20:09
  connected to every other token. Yes, if you think of words as
• 20:13
  nodes, a transformer uses attention as the mechanism to
• 20:16
  aggregate information from all other words nodes.
• 20:19
  The main difference is that in text the order matters, but a
• 20:23
  basic, fully connected graph doesn't have inherent order.
• 20:26
  So Transformers need something extra.
• 20:28
  Right, they need those positional embeddings added to
• 20:31
  the input features to give the model information about the
• 20:34
  sequence order GN NS operating on.
• 20:37
  Naturally, sparse grass often don't need explicit positional
• 20:40
  encoding because the structure itself provides context.
• 20:44
  That's a fascinating link. It really frames Transformers
• 20:46
  within this broader graph perspective.
• 20:49
  OK, so we've covered a good range of architectures.
• 20:52
  Let's shift gears slightly and talk about some cutting edge
• 20:54
  research. NEC Labs America has been doing
• 20:57
  some significant work in GN NS, right?
• 20:59
  What are some of their contributions?
• 21:00
  Yes, they've been tackling some really important practical
• 21:03
  challenges. One area is making GN NS more
• 21:06
  robust. Robust to what?
• 21:08
  Noisy data attacks. Both real world graphs often
• 21:13
  have noisy or even irrelevant edges.
• 21:15
  Sometimes connections might be actively manipulated in
• 21:18
  adversarial settings like finance or security gene and
• 21:21
  performance can degrade significantly with this noise.
• 21:24
  So how do you make them tougher? NEC labs develop methods like
• 21:27
  PTD NET, which stands for a Parameterized Topological
• 21:30
  Denoising Network. That's a mouthful, but the idea
• 21:32
  is clever. It essentially trains an
• 21:34
  auxiliary network alongside the main GNN to learn which edges
• 21:38
  are likely to be irrelevant or noisy for the task at hand.
• 21:42
  It then learns to effectively down weight or even drop those
• 21:45
  edges during the GN NS message passing.
• 21:47
  So it learns to ignore the noise in the graph structure itself.
• 21:50
  Exactly. Wei Chang at NEC Labs put it
• 21:53
  nicely, saying something like by focusing on the most relevant
• 21:57
  connections, we've been able to significantly enhance the
• 22:00
  robustness of GN NS, especially against noisy data, which is
• 22:04
  common in the real world. That makes sense.
• 22:06
  If the model can clean up its own input graph it should
• 22:09
  perform better. What about understanding why a
• 22:11
  GNN makes a prediction? Explain ability is huge in AI
• 22:15
  now. Absolutely critical, especially
• 22:17
  if you're using GNNS for high stakes decisions.
• 22:19
  NEC Labs has also focused heavily on advancing
• 22:22
  explainability. How do you explain AGNN
• 22:24
  prediction? Is it about which nodes or edges
• 22:27
  were most influential? That's the usual approach,
• 22:30
  identifying important subgraphs or features, but a key challenge
• 22:33
  is reliably evaluating whether an explanation method is
• 22:37
  actually faithful to what the model is doing.
• 22:39
  Right. How do you know the explanation
• 22:40
  is correct? NEC Labs developed new ways to
• 22:42
  measure this fidelity using ideas from information theory.
• 22:46
  They've worked on creating robust metrics to ensure that
• 22:48
  when an explanation method highlights certain parts of the
• 22:51
  graph, those parts are genuinely driving the GN NS prediction.
• 22:56
  Hyphen Chen at NEC Labs emphasized their goal is to make
• 22:59
  GN NS not just powerful, but also transparent and
• 23:02
  trustworthy. Trustworthy.
• 23:04
  Is the keyword there. OK, Robustness.
• 23:06
  Explainability. What about graphs that change
• 23:09
  over time, like social networks or communication networks?
• 23:12
  That's another area anomaly detection and dynamic graphs.
• 23:15
  Networks aren't static. They evolve new connections form
• 23:18
  old ones break. Detecting unusual changes can be
• 23:21
  critical for things like network intrusion detection or
• 23:24
  identifying emerging fraudulent patterns.
• 23:26
  Standard GN NS might struggle with the time aspect, right?
• 23:30
  They often process static snapshots, so NEC Labs develops
• 23:34
  TRGNN Structural Temporal Graph Neural Networks.
• 23:37
  These models are specifically designed to learn from sequences
• 23:40
  of graph stamp fats over time. So they look at both the
• 23:42
  structure and how it changes. Exactly.
• 23:45
  They integrate both the structural information from the
• 23:47
  graph at each time step and the temporal dynamics of how edges
• 23:50
  appear or disappear. Zheng Zheng Chen from the lab
• 23:54
  highlighted that this integration allows CIGNN to
• 23:58
  effectively pinpoint anomalies and dynamic systems, making it
• 24:01
  valuable for proactive threat detection and security systems
• 24:04
  where it's actually been deployed.
• 24:05
  Wow. Deployed in real systems?
• 24:07
  That's significant. One more challenge.
• 24:09
  What happens when a trained GNN encounters types of nodes it
• 24:13
  never saw during training? Say a new type of device
• 24:16
  connects to your network. That's the outer distribution
• 24:19
  problem, and it can really hurt reliability.
• 24:21
  The GNN might make unpredictable or poorly calibrated predictions
• 24:25
  for these unfamiliar nodes. NEC Labs tackled this with
• 24:28
  calibration for out of distribution nodes.
• 24:30
  How do you calibrate for something you haven't seen?
• 24:32
  They developed a framework called GRDQ Graph Edge re
• 24:36
  Weighting via Deep Queue Learning.
• 24:38
  It uses reinforcement learning concepts, specifically queue
• 24:41
  learning, to learn how to adjust the importance the weights of
• 24:44
  messages coming from potentially out of distribution neighbors.
• 24:47
  So it learns to dynamically adapt the message passing when
• 24:50
  it encounters unfamiliar nodes. Essentially, yes.
• 24:53
  It tries to mitigate the negative impact of these novel
• 24:56
  nodes types, making the GNN's predictions more reliable and
• 24:59
  calibrated even when the graph environment changes.
• 25:02
  OK, so NEC Labs is pushing on robustness, explainability,
• 25:06
  dynamic graphs and out of distribution handling.
• 25:09
  That really covers a lot of the practical hurdles.
• 25:11
  Now to tie all these architectural ideas, some mean
• 25:15
  GCN attention together. Can we look at a concrete
• 25:18
  example? Maybe a benchmark data set?
• 25:21
  Absolutely. The Core Data set is a classic
• 25:23
  benchmark, perfect for illustrating these differences.
• 25:25
  Cora, what is it? It's a citation network data
• 25:27
  set. Each node represents a
• 25:29
  scientific paper. An edge exists from paper A to
• 25:31
  paper B if paper A cites paper B.
• 25:33
  OK, a network of papers and citations.
• 25:35
  What's the task? Each paper comes with some
• 25:38
  features, typically a bag of words vector representing the
• 25:40
  words in the papers abstractor text.
• 25:42
  The task is node classification, predict the research topic like
• 25:46
  neural networks, reinforcement learning, probabilistic methods
• 25:49
  etcetera. There are 7 topics in Cora for
• 25:51
  each paper. So classify papers based on
• 25:54
  their content and who they cite or get cited by.
• 25:57
  What makes Cora interesting for GNNS?
• 25:59
  Two things mainly. First, the connections clearly
• 26:02
  matter. Papers on similar topics tend to
• 26:04
  cite each other. Second, it's often used in a
• 26:07
  semi supervised or low data setting.
• 26:10
  You might have thousands of papers nodes, but maybe only
• 26:13
  labels for a very small fraction, like 20 papers per
• 26:16
  topic. So just 140 labeled nodes in
• 26:18
  total for training. Only 140 training examples for
• 26:22
  thousands of nodes? That's tiny.
• 26:23
  Exactly. It forces the model to rely
• 26:26
  heavily on the graph structure to propagate information from
• 26:28
  the few labeled nodes to the many unlabeled ones.
• 26:31
  You can't possibly succeed just using the node features alone
• 26:35
  with so little labeled. Data OK So what happens if you
• 26:37
  try what's the baseline performance using only the paper
• 26:41
  features the bag of words with a standard classifier like a
• 26:43
  simple multi layer perceptron MLP ignoring the citations if.
• 26:48
  You do that, Just use the node features.
• 26:49
  Your accuracy is typically pretty low, maybe around 5055%.
• 26:53
  It struggles because it doesn't have enough labels and ignores
• 26:56
  the rich relational information in the citations.
• 26:58
  Right, just over chance basically.
• 27:00
  So now let's bring in the graph structure.
• 27:01
  What if we use that simple some pulling GNN layer we discussed
• 27:05
  first re Lu, A+, IHW? OK, if you build a simple two
• 27:10
  layer GNN using some pulling, the performance jumps
• 27:12
  significantly. You'd likely see accuracy jump
• 27:15
  up to around say 77.5%. Big improvement, but you
• 27:19
  mentioned some pulling can be unstable.
• 27:21
  Does that show up? It can, yes.
• 27:23
  Depending on the initialization and hyper parameters, some
• 27:26
  pulling results on CORIC and sometimes vary a bit more,
• 27:29
  reflecting that potential instability from unnormalized
• 27:32
  feature aggregation. OK, So what about mean pulling
• 27:35
  where we normalize by no degree real UD1A plus IHW?
• 27:39
  Mean pulling generally gives a more stable result and often a
• 27:42
  slight boost in performance. You might get around 78.5% or
• 27:46
  maybe close to 79% accuracy. It clearly shows the benefit of
• 27:50
  that simple normalization for stability and performance.
• 27:52
  Better and more reliable. Now what about the star player?
• 27:55
  The GCN layer with symmetric normalization?
• 27:58
  Real you D12A plus ID 12HW. This is where Core really shines
• 28:02
  as a benchmark for GCNS. A standard two layer GCN model
• 28:07
  often pushes the accuracy significantly higher, achieving
• 28:10
  results consistently above 80%, sometimes reaching 81% or even
• 28:14
  slightly higher depending on the specific setup and data split.
• 28:18
  Wow, so going from mean pulling 79% to GCNS 81% plus is a
• 28:23
  noticeable step up just from changing the normalization
• 28:26
  scheme? It is on this benchmark, yes.
• 28:28
  It really highlighted the empirical effectiveness of that
• 28:31
  symmetric normalization. When the GCN paper came out, it
• 28:33
  became the number to beat for a while.
• 28:35
  What about GAA? The graph Attention network?
• 28:37
  Does learning neighbor importance help further on
• 28:40
  Quora? Yes, GAT models also perform
• 28:42
  very well on Quora, often achieving similar or even
• 28:45
  slightly better results than GCM, perhaps getting into the
• 28:47
  8283% range sometimes. It shows that allowing the model
• 28:51
  to learn which citations are more important for classifying a
• 28:53
  papers topic can provide an additional edge.
• 28:56
  So that Quora example perfectly illustrates the journey.
• 28:59
  Ignoring the graph gives poor results. 50% simple summing
• 29:03
  helps. 77.5% averaging more of lane pulling is better and
• 29:07
  stabler. 79% sophisticated normalization like GCN pushes it
• 29:11
  higher. 81% plus and learn at attention like geek can
• 29:15
  potentially refine it even further. 82% plus.
• 29:17
  Exactly. It makes the abstract concepts
• 29:19
  very concrete. The way you leverage the graph
• 29:22
  structure, especially how you normalize or weight the
• 29:24
  information flowing between nodes, really matters for
• 29:27
  performance. It's not just that nodes are
• 29:29
  connected, but how the GNN processes those connections.
• 29:33
  This has been incredibly insightful.
• 29:34
  We've gone from the basic idea of graphs meeting neural
• 29:36
  networks through the core mechanism of message passing,
• 29:39
  explored a whole spectrum of architectures from simple sums
• 29:42
  to sophisticated attention, seen real world applications changing
• 29:45
  industries, touched on cutting edge research making them more
• 29:48
  robust and trustworthy, and walk through a concrete example
• 29:51
  showing how these design choices impact performance.
• 29:53
  It really covers a lot of ground.
• 29:55
  GNS provides such a powerful lens for looking at
• 29:57
  interconnected data, moving beyond tables and sequences.
• 30:00
  Absolutely. So thinking about this deep, the
• 30:03
  big take away seems to be that connections matter, and GNS give
• 30:07
  us the tools to finally learn from them effectively.
• 30:10
  From finding new medicines to getting you places faster, to
• 30:13
  suggesting things you might like, to even securing networks,
• 30:17
  it's all about understanding relationships.
• 30:19
  Well said. The shift towards analyzing
• 30:21
  relational data is a major trend, and GN NS are at the
• 30:25
  forefront of it. So for everyone listening, the
• 30:27
  final thought perhaps is this. Start looking for the graphs
• 30:31
  hidden in plain sight. Think about your own work, your
• 30:35
  own data, even just your daily life.
• 30:37
  Where are the hidden networks? Yeah, customer interactions,
• 30:40
  process flows, social dynamics, biological pathways.
• 30:42
  Exactly identifying those relationships, those nodes and
• 30:45
  edges, could be the first step towards unlocking completely new
• 30:48
  insights or building entirely new applications using the power
• 30:51
  of graph neural networks. This area is moving so fast too.
• 30:55
  What we covered is a solid foundation, but there's constant
• 30:57
  innovation happening. It's a really exciting feel to
• 31:00
  follow or get involved in. Definitely, hopefully this deep
• 31:03
  dive has given you that foundation and maybe sparked
• 31:06
  some curiosity to explore GNS further.
• 31:09
  It's a complex but incredibly rewarding area of AI.
• 31:12
  Agreed. Thanks for joining us on the
• 31:14
  deep dive. We'll catch you next time.