Latest / AI Conversations / Fine-Tuning Small Models for Incident Investigation

Transcript

Fine-Tuning Small Models for Incident Investigation · AI Conversations

• 0:00
  SPEAKER_02: So, Austin, we're talking about uh production alerts today.
• 0:04
  SPEAKER_02: And it's something that I think a lot of people in the tech space, they just kind of like dread, right?
• 0:09
  SPEAKER_02: Like you get that alert and it's like, oh no, what's broken now?
• 0:14
  SPEAKER_01: Mm-hmm.
• 0:15
  SPEAKER_01: Yeah, it's it's that moment where you're just like, okay, did I break something?
• 0:19
  SPEAKER_01: Did someone else break something?
• 0:21
  SPEAKER_01: Uh, and then the investigation starts.
• 0:23
  SPEAKER_02: Aaron Powell The investigation, exactly.
• 0:26
  SPEAKER_02: And it's like, it can be so time consuming.
• 0:29
  SPEAKER_02: You're looking through logs, you're looking through uh deployment changes, feature flags.
• 0:35
  SPEAKER_02: Like, what what exactly caused this?
• 0:37
  SPEAKER_01: Right.
• 0:38
  SPEAKER_01: And and traditionally, you've had these rule-based approaches, which are fast, they're inexpensive, but their accuracy, it just it's limited, especially on those really complex incidents.
• 0:49
  SPEAKER_02: Yeah, I mean, you can kind of get a general idea, but when things get really messy, it's like, okay, this isn't gonna cut it.
• 0:57
  SPEAKER_02: And then you've got these um agenc investigations, which are super powerful.
• 1:02
  SPEAKER_02: They can really reason deeply about the evidence.
• 1:06
  SPEAKER_02: But the cost, right?
• 1:07
  SPEAKER_02: Using these frontier models for every single alert, it's just it's too much at scale.
• 1:13
  SPEAKER_01: Exactly.
• 1:14
  SPEAKER_01: It's it's a trade-off.
• 1:15
  SPEAKER_01: Do you go for the quick, cheap, less accurate, or the deep, expensive, more accurate?
• 1:22
  SPEAKER_01: And that's the gap uh that this whole approach is trying to close.
• 1:25
  SPEAKER_02: So what what exactly did they do to try and bridge that gap?
• 1:29
  SPEAKER_01: Well, they took a smaller, specialized model.
• 1:32
  SPEAKER_01: They fine-tuned Quen 3.59B uh on traces from investigations that were generated by GLM 5.3.
• 1:40
  SPEAKER_02: So they're taking a smaller model and teaching it, essentially, like how a bigger, more capable model thinks.
• 1:48
  SPEAKER_01: That's that's the core idea.
• 1:49
  SPEAKER_01: And the results were pretty compelling.
• 1:51
  SPEAKER_01: I mean, it achieved 87% of GLM 5.3's recall, but the self-hosted LLM serving cost was uh significantly lower.
• 2:01
  SPEAKER_01: We're talking 0.003 dollars per investigation compared to 0.06 dollars uh for GLM 5.3.
• 2:09
  SPEAKER_01: That's a 20 times decrease.
• 2:11
  SPEAKER_02: Wow, so a 20 times decrease in cost, that's that's a huge jump.
• 2:16
  SPEAKER_02: And that makes it like significantly more practical, I imagine.
• 2:20
  SPEAKER_01: Oh, absolutely.
• 2:21
  SPEAKER_01: I mean, at that price point, a single 40 gigabyte A100 GPU can support approximately 100,000 investigations per week.
• 2:30
  SPEAKER_01: That's enough to investigate a subset of the millions of unique monitors that customers are interacting with every week.
• 2:37
  SPEAKER_02: That's that's incredible.
• 2:38
  SPEAKER_02: So it's not just about like getting a better answer, but it's about making that better answer accessible for a much wider range of alerts.
• 2:47
  SPEAKER_01: Right, and they're even pursuing further optimizations to make it practical for an even larger share of those monitors.
• 2:53
  SPEAKER_02: So, what what does this actually look like in practice?
• 2:56
  SPEAKER_02: Like, how does Datadog currently handle these kinds of investigations?
• 3:00
  SPEAKER_00: Currently, they have two main approaches.
• 3:02
  SPEAKER_00: There's the relevant changes tab, which basically overlays recent changes on a monitor's alert timeline, so you can see uh the deployments, feature flags, configuration changes that are most likely to have contributed to an alert.
• 3:15
  SPEAKER_02: So it's almost like a visual timeline of like this happened, then this happened, then the alert happened.
• 3:22
  SPEAKER_01: Exactly.
• 3:23
  SPEAKER_01: It gives engineers a fast way to identify potentially relevant deployments.
• 3:27
  SPEAKER_01: And then the other approach is that change tracking also exposes its data through a model context protocol tool that agents, like Datadog's Bits Investigation, can use during incident investigations.
• 3:39
  SPEAKER_02: Mm-hmm.
• 3:40
  SPEAKER_02: So the Bits investigation, that's like an agent that's actually like doing the digging, right?
• 3:46
  SPEAKER_01: Yeah, that's the one.
• 3:47
  SPEAKER_01: And according to Internal Telemetry, the change tracking tool contributes to thousands of BITS investigations each week, and 20% of Bits investigation conclusions actually reference a change captured by change tracking.
• 4:00
  SPEAKER_02: So the current system has its strengths.
• 4:03
  SPEAKER_02: Like relevant changes is fast, it's inexpensive, but it's limited in accuracy for complex incidents.
• 4:10
  SPEAKER_02: And the agentic investigations are deep but expensive.
• 4:14
  SPEAKER_02: So this smaller model is like the perfect middle ground?
• 4:18
  SPEAKER_01: That's the idea.
• 4:19
  SPEAKER_01: It's it's trying to combine the strengths of both, deeper agentic investigation at a cost that can support a much larger volume of alerts.
• 4:28
  SPEAKER_02: Hmm.
• 4:29
  SPEAKER_02: So how did they actually teach this smaller model?
• 4:32
  SPEAKER_02: Because you can't just like throw data at it, right?
• 4:35
  SPEAKER_01: No, exactly.
• 4:36
  SPEAKER_01: Prompt engineering alone wasn't enough.
• 4:39
  SPEAKER_01: Even after repeated iterations on the system prompt and tool descriptions, Quen 3.59B tended to search too broadly instead of narrowing its investigation around the most promising evidence.
• 4:51
  SPEAKER_01: So rather than adding more rules, they explored whether they could teach the smaller model the investigative behavior of a more capable model.
• 4:59
  SPEAKER_02: So they're teaching it how to investigate, not just what to investigate.
• 5:04
  SPEAKER_01: Precisely.
• 5:04
  SPEAKER_01: They adapted Nvidia's data flywheel blueprint.
• 5:08
  SPEAKER_01: The idea is to generate investigation traces with a larger teacher model, use those successful traces to fine-tune a smaller student model, and then repeat the process as new production investigations become available.
• 5:21
  SPEAKER_02: That's that's a pretty clever way to do it.
• 5:23
  SPEAKER_02: So the teachers like showing the student the ropes.
• 5:27
  SPEAKER_01: Yeah, and Nvidia demonstrated this by fine-tuning a Lama 3.21B model on tool calling traces from a 70B teacher, reaching 98% of the teacher's accuracy with a roughly 70 times reduction in parameter count.
• 5:41
  SPEAKER_01: So they adapted that approach to see if it could make a gentic change attribution inexpensive enough.
• 5:47
  SPEAKER_02: So what's the the actual workflow for this flywheel?
• 5:51
  SPEAKER_01: So first they identify the change from Bits investigation conclusions.
• 5:55
  SPEAKER_01: Bits investigation writes a freeform conclusion for every incident it investigates.
• 6:00
  SPEAKER_01: They use GLM 5.3 to parse that conclusion, identify any changes it references, and map them to change IDs.
• 6:08
  SPEAKER_01: They treat each referenced change as a proxy label.
• 6:12
  SPEAKER_02: So it's like this is what Bits thought caused it, not necessarily like a human verified 100% this is it.
• 6:20
  SPEAKER_01: Exactly.
• 6:21
  SPEAKER_01: It gives them labeled examples without requiring manual annotation, which is a huge time saver.
• 6:26
  SPEAKER_01: Then they run the teacher agent.
• 6:28
  SPEAKER_01: GLM 5.3 investigates the same alert using eight turns, five tools, and a 65,536 token context window, and it returns a ranked list of possible changes along with confidence scores.
• 6:42
  SPEAKER_01: They intentionally constrain this process so a much smaller model can learn to reproduce it.
• 6:48
  SPEAKER_02: Mm-hmm.
• 6:48
  SPEAKER_02: So the teacher's showing it the right way to do it.
• 6:51
  SPEAKER_01: Right.
• 6:52
  SPEAKER_01: Then they generate the data set.
• 6:54
  SPEAKER_01: They run the teacher model three times on each alert, and they only keep the examples that return changes that match the proxy labels extracted from the Bit's investigation conclusion, discarding the rest.
• 7:05
  SPEAKER_01: This technique is known as rejection sampling fine-tuning.
• 7:09
  SPEAKER_02: So they're really like curating the data for the student.
• 7:12
  SPEAKER_01: Yep.
• 7:13
  SPEAKER_01: They apply this process to 348 internal incidents, and from those they created an initial data set of 100 teacher traces.
• 7:21
  SPEAKER_01: No customer data was used, by the way.
• 7:23
  SPEAKER_02: Good to know.
• 7:24
  SPEAKER_02: And then they fine-tuned the student.
• 7:26
  SPEAKER_01: Yeah, they fine-tuned Quen 3.59B using 16-bit low-rank adaptation or LoRa, which updates a small set of adapter parameters rather than all of the models' weights.
• 7:38
  SPEAKER_01: And they calculated training loss only on the assistant responses in each selected trace.
• 7:43
  SPEAKER_02: So it's it's a very efficient way to train it.
• 7:45
  SPEAKER_01: Very efficient.
• 7:46
  SPEAKER_01: And then they repeat the cycle.
• 7:48
  SPEAKER_01: Both the teacher and student models run in production.
• 7:51
  SPEAKER_01: When the teacher's prediction matches the proxy label and the student's does not, they add the teacher's investigation trace to the training data set and retrain the student on the expanded data set.
• 8:02
  SPEAKER_01: This allows it to continuously learn from new production investigations over time.
• 8:07
  SPEAKER_02: That's that's the the flywheel part, the the continuous learning.
• 8:14
  SPEAKER_01: Exactly.
• 8:15
  SPEAKER_01: They've completed two rounds of this process already, adding 86 training examples and increasing the data set from 100 to 186 examples.
• 8:25
  SPEAKER_02: So it's this continuous improvement loop, making these agentic investigations much more practical and accessible.
• 8:33
  SPEAKER_02: That's that's a pretty powerful insight.