Latest / The Feed & The Thread / The Feed & The Thread - May 27, 2026

Transcript

The Feed & The Thread - May 27, 2026 · The Feed & The Thread

• 0:00
  AI just hallucinated usability problems that don't exist. So
• 0:03
  our video data is actually useless. The study shows humans
• 0:06
  caught what the models missed, and we need to know why.
• 0:13
  Welcome to The Feed and The Thread, brought to you by
• 0:16
  Chicago Camps. Leadership By Design is on Thursday and
• 0:18
  Friday, September 17 and 18 and tickets are available now! And
• 0:22
  while you're at it, get caught up on UX fundamentals with five
• 0:25
  minute UX at five em UX dot com. The Feed & The Thread is
• 0:28
  available online at feed and thread dot com to submit your
• 0:31
  feeds, or download our app for all the feeds and threads
• 0:34
  delivered right to your pocket.
• 0:43
  We trust our tools to see what we miss, but what happens when
• 0:46
  those tools start seeing things that aren't there? That tension
• 0:49
  between speed and accuracy is reshaping how we validate our
• 0:53
  work today. Jim Lewis and colleagues at MeasuringU tackle
• 0:56
  this in "Does AI Find Real UI Problems or Just
• 0:58
  Hallucinations?". They pitted human researchers against large
• 1:01
  language models to find usability issues in user testing
• 1:04
  videos. The AI found roughly half the problems humans
• 1:07
  spotted, but it also generated nearly as many false alarms.
• 1:10
  Most of those extra findings were hallucinations rather than
• 1:14
  genuine insights. That rewrites the assumption that AI can
• 1:17
  replace human judgment in qualitative analysis. Gale
• 1:20
  Robins at UX Design.cc makes a case for pausing before you
• 1:23
  start building in "Product discovery’s quietest, most
• 1:25
  consequential decision". Teams often treat loud feedback as
• 1:29
  automatic instructions instead of hypotheses to test. Robins
• 1:32
  suggests filtering signals by checking if they're real,
• 1:35
  connected to a user's job, and valid. Skipping this evaluation
• 1:38
  step means you might be solving the wrong problem entirely.
• 1:41
  Eric Joseph L. argues that low- fidelity prototypes can break
• 1:44
  user trust in "Your Prototype Is Not Being Honest With Your Users
• 1:48
  (And Here’s How To Fix It)", published by Smashing Magazine.
• 1:51
  Users often disengage when they realize they aren't interacting
• 1:54
  with a real product, especially during sensitive flows like
• 1:58
  login screens. He shows how high-fidelity interactions can
• 2:01
  restore that sense of reality. If the prototype feels fake,
• 2:04
  your data probably is too. Other articles today from UX
• 2:07
  Planet on AI and XR, Intercom on speed-to-lead, and Codrops on
• 2:10
  interface feel.
• 2:18
  Welcome to the Thread. Over on r/UXDesign, someone mapped every
• 2:22
  post from April to find what actually drives conversation.
• 2:25
  It's a quiet look at our collective attention span. We
• 2:28
  often assume popularity equals importance. But data
• 2:31
  visualization of community chatter reveals where the real
• 2:34
  friction lives. It's less about flashy trends and more about the
• 2:37
  problems we keep coming back to. That's valuable signal in a
• 2:41
  noisy feed. In r/UXResearch, a product ops lead broke down
• 2:44
  feedback tools into three buckets. AI analysis for
• 2:47
  existing data, structured portals for new input, and
• 2:49
  behavioral signals for usage patterns. The tension here's
• 2:53
  speed versus accuracy. AI tools promise to see what we miss. But
• 2:56
  they also hallucinate patterns that aren't there. We trust
• 2:59
  these tools to consolidate fragmented input. Yet we still
• 3:02
  need human judgment to validate the insights. The tool is only
• 3:06
  as good as the context we provide. Also in r/UXResearch,
• 3:09
  a researcher shared tips for spotting fake participants.
• 3:12
  Scripted answers and timezone mismatches are common red flags.
• 3:15
  It's a gritty reminder that data integrity starts before the
• 3:19
  study begins. We rely on tools to filter noise. But sometimes
• 3:22
  the noise is intentional deception. Verifying the human
• 3:25
  behind the screen remains a manual, tedious task. It's
• 3:28
  unglamorous work, but it protects the validity of every
• 3:31
  insight that follows. Back in r/UXDesign, a designer expressed
• 3:34
  anxiety about AI automating visual execution. They admitted
• 3:38
  relying on aesthetic appeal rather than strategy. When AI
• 3:41
  levels the playing field for pretty designs, strategic
• 3:44
  problem-solving becomes the only real differentiator. It's an
• 3:47
  uncomfortable pivot, but it's also an opportunity to reclaim
• 3:50
  the core value of design. Execution is fast, insight is
• 3:53
  slow, and insight is what matters. Finally, r/UXDesign
• 3:56
  discussed facilitating workshops for engineers stuck in
• 3:59
  optimization mode. One commenter noted that teams need permission
• 4:03
  to try new things. Culture can't be flipped in a single session.
• 4:07
  Engineers think in constraints. Designers think in
• 4:09
  possibilities. Bridging that gap requires patience. It's not
• 4:13
  about convincing them, it's about creating space for blue-
• 4:16
  sky thinking without immediate feasibility checks. We're
• 4:19
  trading speed for depth in every part of the process. From
• 4:22
  validating data to defining strategy, the pressure to move
• 4:25
  fast is colliding with the need to get it right. That tension
• 4:29
  isn't going away. It's our new baseline.
• 4:38
  Chicago Camps is hosting Leadership By Design on Thursday
• 4:41
  & Friday, September 17 & 18. It's an online event, so you can
• 4:44
  join from anywhere in the world! Tickets are free, thanks to the
• 4:47
  generosity of the community! If it's within your budget, you can
• 4:50
  purchase a general admission ticket for only twenty six
• 4:53
  dollars - with limited early bird tickets at only fifteen
• 4:55
  dollars. Get tickets now at Chicago Camps dot org.
• 5:05
  The MeasuringU study showing AI hallucinating usability issues
• 5:08
  is a stark reminder. The models found half the real problems but
• 5:12
  generated nearly as many false alarms. That ratio changes the
• 5:15
  cost of validation. It shifts the burden from discovery to
• 5:18
  verification. We spend hours watching test videos to catch
• 5:21
  the AI's mistakes. That feels like we're just auditing a tool
• 5:24
  instead of doing research. Unless we treat the AI as a
• 5:27
  junior analyst. Gale Robins argues we filter signals by
• 5:30
  checking if they're real and connected to a user's job. The
• 5:33
  AI provides the raw signal. We provide the context. But
• 5:36
  context isn't just a filter. It's the foundation. If we don't
• 5:40
  know why a user is frustrated, the AI can't distinguish between
• 5:43
  a UI glitch and a workflow mismatch. It sees pixels. We see
• 5:46
  intent. Exactly. The tool consolidates fragmented input.
• 5:49
  But it lacks the narrative thread. The tension isn't speed
• 5:52
  versus accuracy. It's pattern recognition versus meaning
• 5:55
  making. So we stop asking if AI can replace human judgment. We
• 5:58
  start asking how to train it to understand the job to be done.
• 6:02
  The skill isn't spotting errors. It's defining what success looks
• 6:05
  like. That reframes the entire workflow. We're not validating
• 6:08
  outputs. We're validating the questions we ask the machine.
• 6:12
  Right. The value isn't in the findings. It's in the framing.
• 6:15
  And that's a muscle we've been ignoring for too long.
• 6:20
  That's The Feed and The Thread for today. We'll catch you next
• 6:23
  time!