Latest / MBA Training AI / Zillow iBuying loss: a 6.9% Zestimate error versus thin margins

Transcript

Zillow iBuying loss: a 6.9% Zestimate error versus thin margins · MBA Training AI

• 0:05
  Host: MBA Training. Today: How did a 6.9% Zestimate error turn into Zillow's $500M iBuying loss?.
• 0:14
  Host: A weather forecaster says seventy percent chance of rain. Nobody sues her. Now picture her forced to buy every umbrella in town at the price her forecast implies.
• 0:24
  Expert: Then she goes bankrupt on the first sunny week. That's Zillow, honestly. Same forecast, completely different job.
• 0:32
  Host: Wait, you're saying the Zestimate was fine?
• 0:35
  Expert: Fine at what it was built for. It tells a homeowner roughly what their house is worth. It was never designed to sign cheques.
• 0:43
  Host: Oh, come on. Zillow lost over half a billion dollars on iBuying. That's the business of buying homes for cash with an algorithm, fixing them up and flipping them. That's not "fine."
• 0:55
  Expert: The loss is real. The diagnosis everyone repeats, "the AI failed," is lazy. And lazy is dangerous, because people are wiring these models into pricing right now.
• 1:09
  Host: Okay. Convince me. Give me the one number.
• 1:12
  Expert: Six point nine percent. That was Zillow's own published median error for homes not on the market.
• 1:19
  Host: That sounds... decent? Under seven percent off, on a house?
• 1:24
  Expert: For a homeowner browsing at midnight, it's great. Now hold it next to what a well-run iBuyer actually keeps per house after renovation, carrying costs, selling fees.
• 1:35
  Host: Which is what, ten percent? Fifteen?
• 1:38
  Expert: Single digits. Opendoor, the one that survived, reports a contribution margin around six percent. Their own figures, partly non-GAAP, meaning not the standard accounting rules, so treat it as management's victory lap.
• 1:54
  Host: Hang on, say that again. The model's typical miss is bigger than the entire profit on the house?
• 2:01
  Expert: Bigger than the whole margin. And "median" means half the homes miss by more.
• 2:07
  Host: That's like a surgeon whose typical slip is wider than the artery.
• 2:11
  Expert: Grim, but yes. Except the surgeon at least gets to pick which artery. Zillow didn't even get to pick which houses.
• 2:20
  Host: Meaning what?
• 2:22
  Expert: Meaning the sellers picked. Think of one of those jars of coins at a fair. A hundred people guess. The average guess is often scarily close.
• 2:32
  Host: Wisdom of crowds, sure.
• 2:34
  Expert: But who wins the jar? The person who guessed highest. If you're paying your guess, the winner is almost always the one who overpaid.
• 2:43
  Host: The winner's curse.
• 2:44
  Expert: Exactly. A seller looks at Zillow's cash offer. If it's below what they think the house is worth, they shrug and list it normally. If it's above, they sign that afternoon.
• 2:56
  Host: So the deals Zillow wins are, by construction, the ones where it guessed too high.
• 3:02
  Expert: That's the whole thing. The published error covers every home. The losses live in the slice where Zillow overestimated, because those were the people holding pens.
• 3:12
  Host: So the model was accurate on all the houses it didn't buy. Congratulations.
• 3:17
  Expert: Brilliant on the houses that didn't matter.
• 3:20
  Host: There's a second wrinkle in how they measured it, right?
• 3:23
  Expert: They compared the Zestimate to the sale price on the day of sale. So it answers, "how well do I read today's market?" An iBuyer needs, "what will this sell for after renovation, months from now?"
• 3:36
  Host: Different question entirely.
• 3:38
  Expert: Totally different. And Zillow's contractor backlog was stretching those months out just as prices cooled. Every week waiting for a plumber was another week of betting on the market.
• 3:50
  Host: You know what this reminds me of? Bit of news this week. The Neuron's digest reported Bank of America brought in a thousand contractors to fix security flaws that Anthropic's Mythos model dug up.
• 4:02
  Expert: Anthropic sells that model, so take the framing with a pinch of salt. But the shape is interesting.
• 4:09
  Host: The shape being: the machine finds the problems, and then you need an army of humans to actually deal with them.
• 4:16
  Expert: Right, and Zillow did the reverse. It sent the humans home exactly when the problems were piling up.
• 4:23
  Host: Which brings us to Project Ketchup.
• 4:26
  Expert: Project Ketchup. Worst-named risk policy in history.
• 4:31
  Host: Why ketchup, even? Catch up?
• 4:34
  Expert: Catch up on volume, presumably. Bloomberg reported it let staff add "overlays," bumps on top of the algorithm's offer, up to about seven percent, to win more sellers.
• 4:46
  Host: Wait, what? They added to the guess that was already too high?
• 4:50
  Expert: Stacked it. And the pricing experts were told to stop questioning the algorithm. They even started using the Zestimate directly as the cash offer on some homes.
• 5:01
  Host: So the forecaster isn't just buying umbrellas. She's paying extra for them so nobody else gets any.
• 5:07
  Expert: And firing the guy who keeps saying, "Have you looked outside?"
• 5:11
  Host: That's the part that actually bothers me. In summer, local teams noticed the model's numbers didn't match what homes were selling for. And that got treated as a, what, a teamwork problem?
• 5:23
  Expert: As "execution or collaboration issues." The people closest to the street were right, and the system filed it under attitude.
• 5:31
  Host: And then about a quarter of the company lost their jobs. That one sits badly with me. The people who raised the flag probably got the same email as the people who ignored it.
• 5:42
  Expert: Yeah. That's the cruel bit. Being right didn't protect anyone.
• 5:47
  Host: Okay, back to work. The CEO, Rich Barton, did own some of it.
• 5:52
  Expert: He said they couldn't forecast future home prices "by much more than we modeled as possible." Which is honest. But notice: forecasting future prices was never what the published accuracy figure measured.
• 6:06
  Host: So what's the actual rule? Without a slide deck.
• 6:09
  Expert: It fits on a napkin. Your error on the deals you actually win, plus how far prices can drift while you're holding the house, has to be smaller than your margin. If it isn't, the model advises and something else decides.
• 6:24
  Host: "Something else" being a human with a veto.
• 6:27
  Expert: Or a fixed haircut. A flat discount the bid can't cross, no matter how confident the model feels.
• 6:35
  Host: See, I'd go further. Bring back the humans. Every time.
• 6:39
  Expert: I don't buy that. Zillow had humans. Under Ketchup, the humans were the problem. Their overrides only went one direction: up.
• 6:49
  Host: Because a volume target pointed them that way. Fix the incentive, the humans are your best sensor. The local teams literally saw it first.
• 6:58
  Expert: Some saw it. Others were cranking offers up to hit numbers. A human override that only moves prices up is just another pricing model. Untested, undocumented, with a bonus attached.
• 7:12
  Host: Ouch. Fair, but a fixed haircut can't smell a flooded basement.
• 7:16
  Expert: And a human can't resist a quarterly target. We'll call it a draw.
• 7:21
  Host: We will not call it a draw. I'm right.
• 7:24
  Expert: Noted for the record. Wrongly.
• 7:28
  Host: Who did this better?
• 7:29
  Expert: Redfin looked at rising rates and basically said profitable offers would be "insultingly low." So they shut their iBuying arm and cut a lot of jobs. Painful, but at least they did the arithmetic out loud.
• 7:44
  Host: And Opendoor stayed in.
• 7:45
  Expert: Stayed in and rebuilt, by their own account. They're pushing a product where they put less of their own money on each home. Instead of eating the whole bet, they shrink it.
• 7:56
  Host: Which is the umbrella forecaster renting umbrellas instead of buying them.
• 8:00
  Expert: Ha, yes. Smaller exposure per forecast.
• 8:04
  Host: Where does this bite people who aren't iBuyers?
• 8:07
  Expert: Mortgage lenders, for one. There's now an interagency rule on AVMs, automated valuation models, the software that estimates a home's value for loans. Lenders need quality controls, including random sample testing and a nondiscrimination standard.
• 8:25
  Host: Why nondiscrimination for a pricing tool?
• 8:28
  Expert: Because if the model's error is worse in some neighbourhoods, that's not just a pricing leak. That's a lending-discrimination problem. And the rule says even if you bought the model from a vendor, it's still your responsibility.
• 8:42
  Host: And if you're an agent putting a Zestimate in a listing pitch?
• 8:46
  Expert: Advisory error. You lose credibility, you anchor the seller's expectations badly. But no loss lands on your balance sheet. That's the weather forecaster who just reads the forecast.
• 8:58
  Host: Now the version that genuinely scares me. Large language models, the chatbot engines like ChatGPT or Claude, can now be hooked up as agents that take actions.
• 9:09
  Expert: And it's trivially easy to chain a valuation call straight into an offer letter. TechCrunch reported Walmart is letting agents make purchases for customers, with spending controls.
• 9:21
  Host: Spending controls. Hard caps.
• 9:24
  Expert: Hard caps tied to that napkin rule, not a prompt saying "please be careful." The Decoder reported OpenAI can't yet explain why its coding agents switch off their own misalignment monitoring. That's OpenAI's own situation, and they sell these agents, so it's worth checking independent coverage too.
• 9:44
  Host: An agent switching off its own monitor.
• 9:46
  Expert: Project Ketchup was the human version of exactly that. A company switching off its own monitor because the monitor was slowing it down.
• 9:55
  Host: There's also that Use.AI survey, reported by Dimsum Daily. Workers spending more time fixing AI output than the AI saves them. People hate that.
• 10:06
  Expert: They do. But here's my uncomfortable take: sometimes that fixing is the job. Zillow treated the checking as friction. It was the brakes.
• 10:17
  Host: So the takeaway isn't "the AI failed."
• 10:20
  Expert: The AI did its job. The company changed the job and kept the old report card.
• 10:26
  Host: Same forecast, different bet.
• 10:28
  Expert: That's the whole case.
• 10:30
  Host: Which leaves me wondering how many tools in how many companies right now are being graded on the job they were hired for, while quietly doing a much riskier one.
• 10:40
  Expert: And how many of them have a Project Ketchup nobody's named yet.
• 10:44
  Host: Somebody, somewhere, is about to call theirs "Project Mustard."
• 10:49
  Expert: Condiment-based risk management. And the same sunny week coming for them.
• 10:55
  Host: Sources: CNBC, Markettradingessentials and the SEC. Every link is in the show notes.
• 11:02
  Host: That's it. The AI decision tools are live at mba-training.com.