Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Duolingo Rebuilt Its AI Tutor for 100 Million Learners
Transcript
- Lucas: Duolingo has over 100 million monthly active users, and every single one of them expects their lesson to feel tailored to exactly where they are. That's not easy when you're generating exercises in real time across 40 languages. Luna: And they've been iterating on their AI tutor, Birdbrain, for years. But I heard they recently did a pretty significant rebuild of how it works under the hood. Lucas: Right. The old Birdbrain was essentially a giant set of rules — if a learner got this word wrong, show it again in three questions. If they got it right three times, move it to spaced repetition. It worked, but it was brittle. Every new language meant manually writing thousands of rules. Luna: So they moved to a model-based approach. What was the trigger? Lucas: Two things. First, they realized their rule engine couldn't keep up with the complexity of 100 million learners — the edge cases were exploding. Second, they had a massive dataset of learner interactions, and they weren't using it to actually improve the system dynamically. They were storing it but not learning from it in real time. Luna: So they turned to transformers. But that's a big leap from a rules engine. How did they start? Lucas: They started with a relatively small model — about 50 million parameters — trained on learner session data. The goal wasn't to replace everything at once. They wanted to predict the probability that a user would get a given exercise correct. That's a binary classification problem, but it had to run in milliseconds. Luna: And that prediction feeds into which exercises the user sees next? Lucas: Exactly. The model outputs a score for every possible exercise in the user's current unit — thousands of them — and the system picks the one where the predicted difficulty is closest to the user's current proficiency. If you're at 80 percent mastery, it gives you exercises you have an 80 percent chance of getting right. Not too easy, not too hard. Luna: That's the zone of proximal development. Vygotsky would be proud. Lucas: Exactly. And the results were striking. After rolling out the model to a subset of users, they saw a 40 percent increase in overall retention — people came back more often — and a 15 percent improvement in learning outcomes measured by standardized tests. Luna: But 50 million parameters — that's still a lot to serve at scale. How are they handling inference? Lucas: They run inference server-side using optimized TensorFlow serving on GPUs. But here's the interesting part: they actually experimented with on-device models using Core ML and TensorFlow Lite. The latency was better, but the model quality suffered because they couldn't update it as frequently. They update the server model every two weeks with fresh training data. Luna: So they traded a bit of latency for better personalization. Makes sense for a learning app where a few hundred milliseconds isn't going to break the experience. Lucas: Right. And they found that they could actually shrink the model quite a bit without hurting accuracy. They did pruning and quantization, got it down to about 15 million parameters with less than a one percent drop in AUC. Luna: That's significant for cost. Fewer parameters means cheaper inference at scale. Lucas: Huge. They estimate they're saving about 60 percent on compute costs compared to the original 50-million-parameter model. And the smaller model actually runs faster, which improved the user experience. Luna: What about the new exercises themselves? Are they also generated by AI? Lucas: Partially. They have a separate system that uses a larger language model — something like GPT-3.5 — to generate candidate exercises, but then human experts review and curate them. The generative model is more of a suggestion engine. The actual selection is driven by Birdbrain's prediction model. Luna: That's a smart split. Use the big model for creativity, the small model for personalization. Lucas: Exactly. And they also built a feedback loop: when a user gets an exercise wrong, that signal goes back into the training data for the next model update. So the system learns from every mistake. Luna: One challenge I imagine is cold start — new users or new languages where you don't have much data. Lucas: They handle that with a hierarchical model. For a new language, they seed the model with data from similar languages — romance languages share a common prior, for example. And for new users, they use a brief onboarding quiz to establish baseline proficiency, then the model kicks in after about 10 exercises. Luna: So the architecture is really modular. There's the prediction model, the exercise generation pipeline, the feedback loop, and the cold-start handling. Lucas: And all of it sits on top of their existing infrastructure — the same database, the same session management. They didn't have to rebuild everything from scratch. That's a lesson in itself: you can modernize your AI stack incrementally. Luna: I want to go back to the results. A 40 percent boost in retention — that's enormous. How much of that do you attribute to the model versus other product changes? Lucas: They ran A/B tests specifically isolating the Birdbrain model change. The 40 percent lift was measured against a control group that still used the old rules engine, with all other features held constant. So it's causal. Luna: That's a rare number in edtech. Usually you see single-digit improvements. 40 percent is almost unheard of. Lucas: Yeah, it's a big deal. And it shows that personalization at scale, when done right, has real impact. But it took a lot of experimentation to get there. They tried more complex models — transformers with attention — and found they didn't improve much over a simpler feedforward network with embeddings. So they went with the simpler architecture. Luna: That's a good reminder that more parameters isn't always better. Sometimes the business constraint — cost, latency, interpretability — should drive the model choice. Lucas: Absolutely. And they made sure the model was interpretable. They can look at which features — like time since last session, number of correct answers in a row, word difficulty — are driving the prediction. That helps the team trust the system. Luna: Speaking of trust, how did they roll this out? Did they just flip a switch for all 100 million users? Lucas: No, very cautious. They started with a 1 percent traffic experiment in one language — Spanish for English speakers. Monitored for two weeks. Then expanded to 10 percent, then 50, then full rollout over about three months. They had kill switches at every stage. Luna: Smart. And now it's live for all languages? Lucas: Yes. Every language now uses the same Birdbrain model, though each has its own trained weights. The architecture is shared, which makes maintenance easier. Luna: If today's episode was worth a coffee to you, there's a link for that — buy me a coffee dot com slash fexingo. It's how we keep this show ad-free and focused on the engineering decisions that actually matter. Lucas: Yeah, listener support is what lets us spend time on deep dives like this one. No corporate sponsors, no filler. Just the details that operators and builders care about. Luna: Exactly. So back to Duolingo — what's next for Birdbrain? Are they working on anything new? Lucas: They've hinted at adding a reinforcement learning layer that optimizes not just the next exercise but a sequence of exercises over a session. So the model would plan a few steps ahead to maximize long-term retention, not just immediate correctness. Luna: That's fascinating. It's like a chess engine for learning. Lucas: Right. And they're also exploring using the model to generate personalized explanations for wrong answers — not just telling you the right answer, but explaining why you got it wrong based on your specific error patterns. Luna: That would be a game-changer. Today's systems mostly give generic feedback. Lucas: Exactly. So the Birdbrain rebuild isn't just about smarter exercises — it's laying the groundwork for a much more adaptive tutoring experience. And they've done it in a way that's practical, cost-conscious, and backed by real experiments. Luna: It's a great example of how a focused AI investment, tied directly to a user-facing metric, can deliver outsized returns. Lucas: And it all started with a simple question: can we predict whether a user will get this question right? Sometimes the best AI systems are the ones that solve a narrow problem really well.