Latest / The Tech Career Podcast with Fexingo: Engineering Jobs, Interviews, and FAANG Career Strategy / How FAANG Engineers Use Data-Driven Decision Making in Product
Transcript
- Lucas: So there's this moment every product engineer knows — you're in a room with product managers, designers, maybe a VP, and someone says, 'Let's just go with what feels right.' And you're sitting there thinking, we have millions of users, we have telemetry, we have compute — why are we guessing? Luna: Right, the classic 'HIPPO' problem — highest paid person's opinion. It's surprisingly persistent even at companies that pride themselves on data culture. Lucas: Exactly. And I think the gap isn't that engineers don't value data — it's that most of us never learned a repeatable process for using it to make product decisions. So today I want to walk through a specific framework I picked up from a former Amazon principal engineer. She called it the 'Decision Data Hierarchy.' It's basically a ladder you climb from raw numbers to a decision you can defend. Luna: Before we climb that ladder — if this conversation is giving you something usable, the way these episodes stay ad-free and focused is listener support. You can find us at buy me a coffee dot com slash fexingo. Just mentioning it because we'd rather keep doing this than run preroll ads. Lucas: Yeah, seriously. No ads, no sponsors, just the signal. So — the Decision Data Hierarchy. It has four rungs. Rung one: raw metrics. That's your daily active users, your latency p99s, your conversion rates. Just numbers. Luna: The stuff you can pull from dashboards without any context. Lucas: Right. Rung two is what I'd call segmented metrics. You break raw numbers down by user cohort, by device type, by region, by time of day. That's where you start seeing patterns. Rung three is causal analysis — you run experiments to isolate which variable actually caused the change. And rung four is decision synthesis — you combine the causal evidence with business constraints and risk tolerance to make a call. Luna: I love this because it exposes the trap so many teams fall into: they stop at rung one or two. They see 'our retention dropped 5 percent' and immediately jump to 'we need to change the onboarding flow.' But they haven't done the causal work. Lucas: Exactly. And the classic example of getting this right is Netflix's content recommendation engine. There's a well-known case from a few years back where they wanted to improve the thumbnail that appears for a show. The raw metric showed that click-through rate on the home page was flat. But when they segmented by device type, they saw engagement was actually up on mobile and down on TV. Luna: So already rung two tells a more interesting story. Did they run an experiment? Lucas: They did. They A/B tested two different thumbnail-generation algorithms — one that optimized for visual saliency and one that optimized for personalization based on watch history. And the experiment showed that the personalization algorithm improved click-through by about 10 percent on TV but had no effect on mobile. That's rung three — causal, quantified, segmented. Luna: And then rung four — they had to decide. Do you deploy the personalization algorithm only on TV? That adds complexity to the codebase. Or do you roll it out everywhere even though mobile doesn't benefit? Lucas: Right. And from what I've read, they went with a hybrid approach — they used the personalization algorithm on TV and kept the saliency model on mobile. But the key is that decision was based on evidence, not on someone's gut feeling that 'personalization is always better.' Luna: So for a junior engineer listening — how do you start building this muscle without being in a position to run big experiments? Lucas: Great question. I think the most accessible entry point is to start asking 'compared to what?' whenever you see a metric. Like if someone says 'our error rate is 2 percent,' your instinct should be 'compared to last week? Compared to the control group? Compared to the industry benchmark?' Just that question forces the conversation up the hierarchy. Luna: Yeah, and you can also volunteer to write the experiment design doc for your team's next launch. Even if you're not the one running the stats, you learn the structure — hypothesis, success metric, sample size, duration. Lucas: Exactly. And that doc is usually where the hierarchy breaks down too. I've seen teams define a success metric but then not calculate the minimum detectable effect or how long the experiment needs to run. They launch a two-day test and call it conclusive, but the sample size is way too small to detect anything meaningful. Luna: That's the classic 'statistical significance trap' — they see a p-value under point-oh-five and declare victory, but the effect size is tiny and the business impact is negligible. Lucas: Right. And I want to push back on something there — data-driven doesn't mean data-worship. There are perfectly good reasons to make a decision without a full experiment. Sometimes the cost of the experiment outweighs the value of the answer, or you need speed because your competitor just shipped something. Luna: So how do you know when to trust the data versus when to trust intuition? Lucas: The framework I like comes from a Google research paper on decision-making under uncertainty. They map decisions on two axes: reversibility and impact. If a decision is easily reversible and low impact — like changing the color of a button — you can just ship it and measure. If it's high impact and hard to reverse — like changing your pricing model — you'd better have causal evidence. Luna: That's a good mental model. And it also helps with the 'analysis paralysis' problem that some data-driven cultures create. Teams spend weeks perfecting an experiment on a feature that doesn't really matter. Lucas: Yeah, I've been on that team. We spent three months optimizing the loading spinner animation. Three months. And the metric moved by zero point one percent. So the hierarchy only works if you pair it with a sense of what's actually worth knowing. Luna: One thing I've noticed at top tech companies is that senior engineers don't just know the tools — they know how to frame the question. I sat in on a design review where a principal engineer said 'I don't care what the experiment says if we haven't defined what success looks like for the user first.' Lucas: That's the real skill. Because you can run a perfectly powered experiment on the wrong question and get a precise answer to something useless. So let's talk about how to define the question. At Amazon, they used a template called the 'press release' — you write a fictional press release announcing the feature before you build it. That forces you to articulate the customer benefit and the metric that proves it. Luna: And that press release becomes the north star for the experiment design. If the metric doesn't align with the benefit described in the press release, you're measuring the wrong thing. Lucas: Exactly. So for a feature that's supposed to reduce customer effort, your primary metric shouldn't be 'time spent on page' — that might actually go down if you're making things easier. You'd want 'task completion rate' or 'repeat usage.' Luna: Alright, so let's bring it back to the engineer who wants to level up. What's one concrete action they can take this week to become more data-driven? Lucas: I'd say pick one metric your team tracks — just one — and spend thirty minutes writing down the full chain from raw data to business outcome. Like, 'we track daily active users because we believe that leads to more engagement, which leads to retention, which leads to revenue.' Then ask: which link in that chain is weakest? That's where you should focus your next experiment. Luna: That's a great exercise. And it's something you can do solo and then bring to your team's retro or planning meeting. It immediately signals that you're thinking beyond the dashboard. Lucas: Exactly. And if you want to go deeper, the book 'Trustworthy Online Controlled Experiments' by Ron Kohavi and others is basically the bible of A/B testing at scale. Kohavi led experimentation at Microsoft and then at Amazon. It's dense but the chapter on 'common pitfalls' alone is worth the price. Luna: And I'd add that you should look at your team's last three experiments and see how many of them followed the hierarchy. Chances are, at least one stopped at segmented metrics and made a causal claim without an experiment. Just noticing that is a form of data literacy. Lucas: Yeah. And if you're in a position to suggest an experiment, frame it as a way to reduce risk, not as extra work. Say 'we could ship this now and hope it works, or we could run a two-week experiment and know for sure.' Most managers will choose 'know for sure' if you present it that way. Luna: Especially if you've already done the work of defining the minimum detectable effect and the required sample size. That shows you're not just asking for more time — you're asking for the right amount of time to get a reliable answer. Lucas: Exactly. And that's the difference between a junior engineer who says 'we should test this' and a senior engineer who says 'we should test this with a sample size of ten thousand per variant over two weeks to detect a five percent relative change at eighty percent power.' The second one gets a decision. Luna: Alright, so to wrap up — the Decision Data Hierarchy: raw metrics, segmented metrics, causal analysis, decision synthesis. Use it to diagnose where your team is getting stuck. Lucas: And remember that data is a tool, not a crutch. The goal is better decisions, not perfect experiments. If you can consistently get to rung three for high-stakes decisions and rung one or two for low-stakes ones, you're already ahead of most teams. Luna: Next episode, let's talk about something I've been curious about — how FAANG engineers handle technical debt when the product roadmap demands speed. Lucas: Oh, that's a good one. There's a specific trade-off framework I've seen work at scale. Let's do it.