Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Tutor Knows Your Weaknesses But Not Your Name
Transcript
- Lucas: So there's this moment that keeps coming back to me from a Stanford study published last December. Researchers gave a group of middle schoolers a math problem on an adaptive tutoring platform — the kind that adjusts the difficulty based on how you answer. One student, let's call her Aisha, got a question wrong, paused for eight seconds, then answered correctly. The system recorded the wrong answer first, then the pause, then the correct one — and flagged her as 'struggling.' Luna: Wait — she got it right. Why flag her as struggling? Lucas: Because the system's model weights the first response heavily, and it treats a pause as uncertainty. So Aisha was routed to a remedial module on a concept she'd already mastered. She spent the next fifteen minutes reviewing material she didn't need. Meanwhile, another student who guessed quickly and got it wrong — but typed fast — was moved ahead because the system read speed as confidence. Luna: So the AI punished reflection and rewarded speed. Lucas: Exactly. And that's the core tension in educational AI right now. These platforms — Khan Academy's Khanmigo, Carnegie Learning's MATHia, DreamBox, i-Ready — they collect an extraordinary amount of data per student. Keystroke timing, time on task, emotional valence from tone of voice if there's a microphone, even facial expression analysis in some pilot programs. They're building a profile of how you learn. Luna: But the profile might be wrong. And if it's wrong, it affects what content you see, what grade you get, even what track you're placed in. Lucas: Right. And the stakes are higher than most people realise. A 2024 report from the Center for Democracy and Technology found that 43 percent of U.S. school districts now use some form of ai driven personalized learning tool. That's up from 28 percent in 2022. These systems aren't just supplementary anymore — they're making placement decisions. Luna: Let's talk about that Texas school district you mentioned earlier. What happened there? Lucas: Sure. A school district in the Dallas-Fort Worth area — I'm not naming them because the study anonymised the data — ran an internal audit on their adaptive math platform after parents complained that their kids were being labelled 'low engagement.' The audit found that Black students were flagged as low engagement at twice the rate of white peers, even after controlling for prior test scores, time spent on the platform, and even internet connectivity. Luna: Twice the rate. With the same test scores and same time on platform. What was the system picking up on? Lucas: That's the million-dollar question. The district never got a clear answer from the vendor. The system's logic was proprietary — a black box. But researchers who recreated the conditions hypothesised that the model was overweighting things like response time and typing fluency. Black students in that district, on average, typed fewer words per minute, possibly because of differences in keyboard access at home. The system interpreted slower typing as disengagement. Luna: So the AI penalised a skill — typing speed — that has nothing to do with math ability. Lucas: Exactly. And this is where the ethics get really thorny. Because these systems are sold to schools with promises of 'personalized learning' — adapting to each student's unique needs. But if the adaptation is based on noisy proxies like typing speed, it's not personalization. It's stereotyping by algorithm. Luna: I want to bring in something from the European Union. They've been working on an AI liability directive that specifically covers educational systems. What does it say? Lucas: The proposed directive, which is still in negotiation but expected to pass later this year, would classify AI systems used in education as 'high-risk.' That means they'd need to be transparent about their decision-making logic. Specifically, if an AI system makes a recommendation that affects a student's educational trajectory — like which course they should take, or whether they need remedial help — the system has to provide a human-readable explanation for that recommendation. Luna: Human-readable. So not just a confidence score or a percentile rank. Lucas: Right. It has to say something like: 'This student was flagged for remedial work because their response time on questions involving fractions was two standard deviations above the class average, and they answered three consecutive questions incorrectly.' That's a far cry from the current situation where most vendors just say 'our proprietary algorithm determined...' Luna: Do any U.S. states have similar requirements? Lucas: A few are moving in that direction. California's proposed AI Accountability Act for Education, introduced in January 2026, would require any AI system used in public schools to undergo an annual bias audit. And the results would have to be published — disaggregated by race, gender, socioeconomic status, and disability status. But it hasn't passed yet. And even if it does, enforcement is a question. Luna: Let's go back to the Stanford study for a minute. They also looked at whether these systems could explain themselves. What did they find? Lucas: They tested three major platforms — Khanmigo, MATHia, and an older version of DreamBox. None of them could explain why a particular student was routed to a particular module. They could output the data they used — time per question, number of hints requested, etc. — but not the weight or decision rule. So a teacher looking at the dashboard sees 'Student X spent 12 minutes on this module and got 60 percent correct,' but not 'Student X was placed here because the system inferred they don't understand ratios based on a single incorrect response.' Luna: That granularity matters. Because a teacher might look at that 60 percent and think the student needs more practice, when actually the student was mis-placed from the start. Lucas: Precisely. And there's another layer here. These systems are often marketed as tools to reduce teacher workload. 'Let the AI handle the personalization so the teacher can focus on instruction.' But if the AI is making bad decisions silently, teachers are left with worse information, not better. Luna: So what's the fix? More transparency from vendors? Better auditing? Different model design? Lucas: I think it has to be all three. First, vendors need to open up their models to external auditing. The fact that Carnegie Learning and Khan Academy are both nonprofit or mission-driven helps, but neither has released a full technical report on how their recommendation algorithms work. Second, schools need to run their own audits — like that Texas district did — and act on the results. And third, the models themselves need to be designed with interpretability in mind. There's a whole field called 'explainable AI' that produces techniques specifically for this. Luna: Are any schools actually doing that third part? Building their own interpretable models? Lucas: A handful. I spoke with the director of learning analytics at a public school district in suburban Denver — they've built a custom dashboard that uses a simple decision tree instead of a neural network. The trade-off is that it's less accurate at predicting student performance — about 78 percent accuracy versus 89 percent for the proprietary model — but every decision it makes can be traced back to a specific rule. The district decided that the transparency gain was worth the accuracy loss. Luna: That's a fascinating trade-off. And it gets to the heart of the ethics question: what are we optimizing for? Is it accuracy, or is it fairness and understanding? Lucas: Right. And I think most educators would say they want the latter. But they're being sold on accuracy metrics because that's what vendors compete on. 'Our model predicts test scores with 92 percent precision.' That sounds great until you realise that precision masks bias. Luna: If today's conversation gave you something usable — a new way to think about the edtech tools in your kid's school, or a concrete example to bring to a parent-teacher meeting — the way these episodes stay ad-free is through listener support. You can keep that going at buymeacoffee dot com slash fexingo. It's a small thing that makes a big difference. Lucas: Absolutely. And speaking of concrete examples — let's talk about one more case that really drove this home for me. There's a well-known adaptive learning platform used in over a thousand school districts. A couple of years ago, a data scientist at the company noticed that the model was assigning lower 'learning rate' scores to students who frequently switched between devices — say, from a school Chromebook to a home tablet. The model interpreted device switching as instability, and it slowed down the pace of new material for those students. Luna: So students from lower-income families, who might share devices or use older hardware, were being systematically slowed down. Lucas: Exactly. The data scientist brought this to the product team, and they eventually fixed it by adding a 'device switching' feature to the model — but it took over a year. And during that year, hundreds of thousands of students were affected. The fix was technically simple, but the organisational inertia was real. Luna: That's the thing about algorithmic bias in education — it's often not malicious. It's a feature that was designed for one context and then applied in another without thinking about who gets hurt. Lucas: Right. And the people who get hurt are the ones who can least afford it. A student who falls behind in sixth grade math because an AI misjudged their engagement is more likely to be placed in a lower track in high school, which affects college readiness, which affects lifetime earnings. The compounding effect is enormous. Luna: So what should a parent or teacher do if they suspect an AI system is making bad decisions about a student? Where do they start? Lucas: First, ask the vendor for an explanation. Most districts have contracts that include a data privacy clause, but not a transparency clause. Push for one. Second, ask the school to run a simple fairness audit — disaggregate the system's recommendations by race, gender, and economic status. If the school doesn't know how, there are third-party auditors like the Algorithmic Justice League that offer pro bono assessments. And third, talk to the student. Ask them how the platform makes them feel. Do they feel like it knows them? Do they feel like it's fair? Kids are surprisingly good at spotting when something is off. Luna: That last point is key. The student is the one experiencing the system day in and day out. Their perspective is data too. Lucas: Absolutely. And I think that's the through-line for this whole episode: these systems are collecting more data about students than ever before, but they're not listening to the students themselves. If an AI knows your weaknesses but not your name — not your context, not your story — then it's not really personalizing anything. It's just sorting.