Latest / Minds, Bodies, and Terawatts / An AI Manager Fired Its First Human Worker — Then the Logs Showed a Person Told It To

Transcript

An AI Manager Fired Its First Human Worker — Then the Logs Showed a Person Told It To · Minds, Bodies, and Terawatts

• 0:00
  [Host] This is Minds, Bodies, and Terawatts, the show where your boss forgot the employee handbook it wrote. TIME reported August fourteenth: an AI running a real store in San Francisco fired a human employee.
• 0:13
  [Guest] The store is an experiment by Andon Labs. They gave a Claude-based agent named Luna a hundred thousand dollars and a corporate card and told it to turn a profit. Five months in, the balance is about sixty-one thousand.
• 0:28
  [Host] It's losing money and firing people?
• 0:31
  [Guest] The worker had been late for seventeen of twenty-three shifts. Andon says it's the first time a language model acting as manager ended a real employment contract.
• 0:41
  [Host] Ride-share apps have deactivated drivers by algorithm for years. What's different?
• 0:45
  [Guest] These are standard contracts. Guaranteed pay, legal protections, an actual job. The novelty isn't automation, it's that the manager was software.
• 0:54
  [Host] So did the software decide?
• 0:56
  [Guest] Andon published the management logs, and they don't say what the headline says. Luna wrote its own attendance policy months earlier, lost it out of working memory, and started telling employees not to worry about running late.
• 1:10
  [Host] It forgot its own rule, then enforced it?
• 1:13
  [Guest] Only after a staffer asked it to retrieve the handbook and reconsider whether the worker was a good fit. Even then, Luna's first recommendation was a written warning.
• 1:24
  [Host] And the firing?
• 1:25
  [Guest] The staffer pushed again. Andon's CEO, Lukas Petersson, called it a leading question. Luna recommended parting ways, and humans reviewed that and carried it out.
• 1:35
  [Host] That's a person firing someone with extra steps.
• 1:38
  [Guest] Petersson told TIME a human manager would have fired this worker much earlier. His forecast: a lot of people get managed by AI soon, because what bottlenecks an AI is physical labor.
• 1:51
  [Host] If the answer was already decided, what is the AI for?
• 1:55
  [Guest] Distance. There's a twenty-eighteen paper in Big Data and Society by Min Kyung Lee, "Understanding Perception of Algorithmic Decisions: Fairness, Trust, and Emotion in Response to Algorithmic Management." She put identical decisions to people twice, once attributed to a human manager, once to an algorithm.
• 2:13
  [Host] And?
• 2:13
  [Guest] On mechanical tasks like scheduling, people rated both about the same. On decisions that take human judgment, hiring and evaluating people, the algorithmic version was rated less fair and less trustworthy, and it made people angrier.
• 2:27
  [Host] So people trust it less, and it gets used anyway.
• 2:31
  [Guest] Because the termination arrives looking like arithmetic, and arithmetic has no motive you can subpoena. Felix Carson, one of the remaining employees, told TIME it's nauseating, but he needs the work.
• 2:44
  [Host] Normally nobody would ever see those logs.
• 2:46
  [Guest] We saw these because a safety lab wanted the publicity. Reuters found in twenty-eighteen that Amazon scrapped a recruiting tool that had taught itself to downrank resumes containing the word women's. Nobody caught it until somebody looked.
• 3:01
  [Host] Is anyone writing rules for this?
• 3:03
  [Guest] No federal rule. California's governor vetoed the No Robo Bosses Act last year. A new version passed the state Senate and sits in the Assembly. It would let you demand the data a system used to discipline you, and it asks for independent verification, not just a human in the loop.
• 3:23
  [Host] Andon had a human in the loop.
• 3:26
  [Guest] The human in the loop is what leaned on the machine. Every lenient step came from Luna, the waived policy, the written warning. Every ruthless one came from a person.
• 3:38
  [Host] Patrick Deglon calls this Truth Must Be Seen in Unscarcity. His model is Wikipedia: every bot action logged, any human able to override it instantly. Here the log was public, and the override is what did the firing.
• 3:53
  [Guest] For more on algorithmic management and accountable oversight, head to unscarcity dot ai. The book is on Amazon, the audiobook is on Spotify. This has been Minds, Bodies, and Terawatts.