Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / How AI Models Learn Bias From Reddit Upvotes
Transcript
- Lucas: So there's this dataset that's quietly shaped a huge chunk of the language models we use today. It's called Reddit—specifically, the public comment corpus from 2005 through 2023. Luna: Right, the one that's been used to train everything from chatbots to sentiment classifiers. It's like the default community dialogue sample. Lucas: Exactly. And it's popular because it's massive, it's conversational, and it's free. But a new paper from the University of Washington, published in May of 2025, points out a pretty fundamental problem: the model isn't learning the truth. It's learning what gets the most upvotes. Luna: So it's inheriting Reddit's own popularity bias. That feels like one of those obvious-in-retrospect things, but the paper quantifies it. Lucas: It does. They fine-tuned a standard open-source language model—like a LLaMA-class model—on subsets of Reddit comments, then tested it across 50 benchmark tasks. The result: a 12 percent drop in answer diversity compared to the base model. The model started converging on the most upvoted take, even when that take was factually wrong. Luna: And this is a specific kind of bias we haven't covered on the show before. It's not demographic bias or historical redlining—it's popularity bias. The tyranny of the majority, algorithmically amplified. Lucas: You beat me to it. And the paper has a particularly striking example from r/askscience. They took a thread where the top-voted answer was confidently incorrect about a basic physics question—it was about why the sky is blue, actually—and the fine-tuned model reproduced that wrong answer with high confidence. The correct, lower-voted answer was essentially erased. Luna: And this is a dataset that's been used to train models that then go into products. If you're building a customer service bot or a medical Q&A system, you're inheriting that popularity filter. Lucas: Right. And the scale here is huge. The Reddit corpus is about 1.7 billion comments. When you train on that, you're effectively saying: 'the crowd is the ground truth.' But the crowd is often wrong, or at least, not optimizing for accuracy. Luna: So what does this mean for reinforcement learning from human feedback, or RLHF? Because that's the dominant method for aligning models—you get humans to rank outputs, and the model learns to produce more of the preferred ones. Lucas: That's the connection. RLHF is supposed to inject human values, but if the 'human' is just a Reddit user upvoting a comment, you're just injecting popularity. The UW paper found that models trained with RLHF on Reddit-derived preference data showed a 23 percent increase in false-positive safety warnings—the model became overly cautious, flagging benign statements as toxic, because the most upvoted comments on Reddit tend to be conflict-averse and self-censor. Luna: If today's tech conversation gave you something usable, whether it's a new angle on AI bias or just a clearer picture of how your chatbot works, consider supporting the show. It's listener-supported, ad-free, and every bit helps keep the research flowing. Lucas: Yeah, absolutely. And if you're inclined, you can toss something in the jar at buy me a coffee dot com slash fexingo. No pressure, but if it's useful, it's appreciated. Luna: Exactly. And back to the data itself — there's another dimension: the temporal bias. Reddit's upvote patterns shift over time. A comment from 2014 about, say, cryptocurrency might be downvoted then, but the same sentiment in 2021 would be upvoted. Lucas: Great point. The model doesn't know the timestamp. It just sees all comments as equally weighted by final score. So you get a weird time-collapsed consensus that mixes early-adopter enthusiasm with late-stage skepticism. The paper calls this 'temporal homogenization.' Luna: And that's a problem for any model that's supposed to be up-to-date or context-aware. A financial advice bot trained on Reddit might give you 2018 advice with 2024 confidence. Lucas: Right. So the question becomes: should we stop using Reddit data entirely? The authors don't go that far. They suggest a few fixes: weighting comments by a credibility score, or using a separate model to fact-check top-voted answers before training. But those are expensive and introduce their own biases. Luna: And there's also the question of whether upvotes actually represent genuine consensus. There's vote manipulation, brigading, bots. The dataset itself is noisy. Lucas: Exactly. The paper estimates that about 7 to 10 percent of Reddit comments in the dataset are from accounts that exhibit bot-like behavior. So you're not even learning from humans in some cases. You're learning from automated upvote cascades. Luna: So what's the takeaway for someone building a product with a language model? If they're using a base model that was fine-tuned on Reddit, what should they check? Lucas: First, they should look at the training data card. Most model providers now disclose data sources. If Reddit is in there, they should probe for popularity bias—run a few tests where the correct answer is not the most popular one, and see what the model outputs. Second, they should consider fine-tuning further on a curated dataset that prioritizes accuracy over consensus. Luna: And for the everyday user—the listener who's using a chatbot or a writing assistant—is there anything they can do? Lucas: Be skeptical. If the answer sounds like a Reddit top comment—confident, pithy, maybe a bit smug—that's a red flag. Ask for sources, or rephrase the question to see if the model gives a different answer. The diversity drop we mentioned means the model might just be echoing the hive mind. Luna: I think there's a deeper issue here, too, which is that we've built a system where 'democratic' AI is seen as inherently good. But democracy in training data doesn't always mean accuracy. Lucas: That's the philosophical punchline. The paper's authors are careful not to say 'all popularity bias is bad.' In some domains, like entertainment recommendations, what's popular is exactly what you want. But in medicine, law, finance, science—you want the truth, not the upvotes. Luna: So the question becomes: how do we separate the two in a scalable way? And do we need to rethink RLHF entirely? Lucas: A few labs are working on 'de-biasing' RLHF by using expert panels instead of crowd workers. But that's slow and expensive. Others are exploring meta-learning approaches where the model is trained to recognize when it's being influenced by popularity. We're early days. Luna: One number from the paper that stuck with me: the model's perplexity actually improved after fine-tuning on Reddit. So it became more fluent, more confident—but less accurate. The fluency masked the bias. Lucas: That's the scariest part. The model sounds better while being wrong more often. It's like a charismatic but unreliable expert. And if you're not testing for accuracy, you'd never know. Luna: So maybe the next frontier in AI ethics is not just about removing hate speech or demographic bias, but about disentangling popularity from truth. That's a much harder problem. Lucas: It is. And it's not just a Reddit problem—it's any platform with a vote or like system. Twitter, Facebook, YouTube. Anywhere the crowd's opinion is used as a training signal. We've only just started to understand the implications. Luna: And on that note, I'm curious: if you had to redesign Reddit's data to make it more useful for AI training, what would you change? Lucas: I'd add a credibility layer—maybe weight upvotes from users with a history of accuracy, like a reputation system. And I'd include the timestamp as a feature so the model can account for temporal context. But those are technical fixes. The deeper fix is cultural: we need to value accuracy over agreement. Luna: That's a nice place to leave it. Thanks, Lucas. Lucas: Thanks, Luna. And thanks to our listeners for thinking through this with us.