Latest / In The Loop / Should AI Companions Be Legal? The U.S. Government Isn't Sure...

Transcript

Should AI Companions Be Legal? The U.S. Government Isn't Sure... · In The Loop

• 0:00
  This week, seven tech companies from Google to Open AI just got
• 0:04
  served by the Federal Trade Commission in the US.
• 0:07
  They now have just 45 days to hand over all the secrets behind
• 0:12
  the AI companion businesses, the chat bots that have become
• 0:15
  millions of teenagers, friends, mentors, or even lovers, with
• 0:19
  nobody really having any idea what's actually happening during
• 0:22
  those conversations. Today, we're going to dive into
• 0:25
  the landmark inquiry by the Federal Trade Commission.
• 0:28
  What happened, how AI companions work, why teenagers seem to be
• 0:32
  so drawn to them, and what it may mean as a society and
• 0:37
  culture as a whole as these AI companions become increasingly
• 0:40
  widespread. And importantly, what this
• 0:44
  landmark inquiry could mean for the future of all AI products.
• 0:48
  This is In the Loop with Jack Horton.
• 0:50
  I hope you enjoy the show. So I think for today, let's
• 1:05
  start with the FTC investigation because I found it quite an
• 1:09
  interesting one because whatever happens here could very easily
• 1:12
  be applied to many other contacts.
• 1:14
  Because ultimately, and this will frame a lot of the
• 1:17
  discussion, AI companies want you to see their LLM, their chat
• 1:22
  bot as this friend, this guide throughout your entire life.
• 1:26
  And that's what really everybody wants to achieve from a business
• 1:29
  perspective. And so these AI companions
• 1:32
  actually represent this almost Canary in the coal mine of what
• 1:37
  the future could look like as a society and how we might
• 1:39
  regulate these AI systems as they become increasingly
• 1:43
  important to individual users. Anyway, on September 11th, the
• 1:48
  US Federal Trade Commission to the FTC for sure announced a
• 1:52
  huge study and investigation on consumer facing AI chat bots
• 1:57
  that are branded and marketed as AI companions.
• 2:01
  So using its Section 6B authority under the FTC Act, the
• 2:05
  agencies issued compulsory orders to seven different
• 2:09
  companies. So Alphabet, which is Google,
• 2:12
  Meta, Open AI, Snapchat, Character, AIXAI and Replica.
• 2:18
  Now these companies therefore must report in detail how their
• 2:21
  AI bots work and importantly how they're protecting young users.
• 2:25
  Importantly, this 6B order has demanded very specific
• 2:29
  information, and this includes, for example, how do these
• 2:33
  companies monetize engagement with AI companions?
• 2:36
  How do they limit or verify age groups?
• 2:39
  What testing do they do on safety before they actually
• 2:42
  launch these new features? How do they filter or handle
• 2:45
  sexual content, especially with minors?
• 2:48
  Do they keep chat logs on memories of user conversations?
• 2:52
  And can they be deemed risky or manipulative?
• 2:55
  What data do they collect and share with third party
• 2:58
  companies? And how really do they moderate
• 3:00
  and enforce their own rules in day-to-day operations in their
• 3:03
  own businesses? And the reaction of course has
• 3:05
  been quite interesting online. It's become a very big topic of
• 3:08
  discussion on X or Twitter and Reddit.
• 3:11
  The front page comments have ranged anything from optimism to
• 3:14
  cynicism, as always, with lots of topic of conversation
• 3:16
  covering how the government clearly is more interested in
• 3:19
  regulating AI companions than weapons.
• 3:22
  Or the fact that one user said, I can't wait to hear the total
• 3:25
  amounting of BS that they're going to come up with as
• 3:28
  complete excuses. And while others have said that
• 3:31
  they're really glad they're going after the main offender,
• 3:33
  which is meta. And on X child advocates have
• 3:36
  applauded the move, saying that we must understand AI companions
• 3:39
  before they flood the marketplace.
• 3:40
  And that was posted by Doctor Dana Suskind, a surgeon and an
• 3:44
  author. Now, tech policy researcher
• 3:46
  Mikaela Mantenga was glad that regulators are finally taking
• 3:51
  seriously this level of AI bonding that's going on in the
• 3:54
  world right now. So yeah, the FTC has fired
• 3:57
  really big warning shots to all AI companies, or should I say
• 4:01
  companion providers at this stage.
• 4:02
  And I think in the future, AI companies, because whatever pops
• 4:06
  out from this investigation, as I said, is going to probably
• 4:09
  lead to many new laws created over time.
• 4:11
  So what exactly are these AI companions and how popular are
• 4:15
  they really? Because most people I speak to
• 4:18
  have an awareness of them but probably don't understand how
• 4:21
  popular they've become amongst teenagers.
• 4:34
  So First off, the FTC has defined any AI companion as a
• 4:37
  Gen. AI product that simulates human
• 4:40
  like communication and interpersonal relationships.
• 4:43
  So that's basically a chatbot with a name and a personality
• 4:45
  that's available 24/7, may be positioned as a character, which
• 4:49
  is basically, as I said, most LLMS entire business strategy.
• 4:52
  They want to have consumers perceive their chat bot as this
• 4:57
  real world thing that helps them through every single problem
• 5:00
  they could think of. However, for me, what's really
• 5:02
  important is a distinction between, I guess general LLM
• 5:05
  chat bot on an AI companion isn't the LLM, it's the user
• 5:09
  experience, the design, whether the AI is positioned as a
• 5:13
  friend, whether you're able to customise it's persona or maybe
• 5:17
  pick from a bunch of characters. Whether you could pick the
• 5:20
  virtual boyfriend or girlfriend or partner, or you could pick a
• 5:24
  specific business person as a mentor, maybe you could have a
• 5:27
  Game of Thrones character as your friend.
• 5:30
  So that's what I would consider more of an AI companion.
• 5:33
  And now you may be wondering, where do people even find these
• 5:35
  things? Well, the biggest company out
• 5:37
  there right now is called Character AI, which was started
• 5:40
  by ex Google engineers in 2022 and it exploded like 10s of
• 5:44
  millions of people. Using this, teenagers can now
• 5:46
  browse literally thousands of user characters from your
• 5:50
  favorite fantasy night to your study buddy to your flirty anime
• 5:54
  girlfriend. And you know this character
• 5:57
  hashtag alone has got over 2 billion TikTok views and
• 6:01
  officially you're supposed to be 13 to get onto their site.
• 6:04
  But as you know as well, it's easy to fake your age.
• 6:07
  Another really popular one is Replica.
• 6:09
  Then we'll talk about them in a little bit more detail.
• 6:11
  But they've actually been called out a number of times for having
• 6:14
  explicitly sexualized bots. There's also Snapchat.
• 6:18
  They frame it as an AI friend and launched it into everyone's
• 6:20
  feed about 12 months ago So you don't have to download anything
• 6:24
  separate, no permissions. So suddenly millions of
• 6:26
  teenagers just have my AI pinned to their messages.
• 6:29
  And as you can imagine, very predictably, investigators into
• 6:32
  Snapchat's AI found that you could get this thing to give
• 6:34
  advice on hiding drugs and sex and alcohol.
• 6:38
  And the Utah State actually sued them.
• 6:40
  And, and we'll probably go into that a little bit more detail
• 6:42
  later, but but yeah, it shows the level of risk here for young
• 6:47
  adults. As you can imagine, Meta is
• 6:48
  another massive culprit as the biggest company in the world
• 6:51
  within social media, it's arguably there for the most
• 6:54
  damaging. Reuters actually found that Meta
• 6:58
  had internal policy rules which basically allowed chat bots to
• 7:01
  flirt with children. And The thing is, this entire
• 7:04
  industry has now figured out that teenagers are ideal and
• 7:07
  easy customers for them. They're lonely, they're online
• 7:10
  all the time, they'll form emotional bonds with people very
• 7:13
  easily. They're very vulnerable, and
• 7:16
  they'll be using technology and that company's technology for
• 7:20
  years to come. And clearly it's working.
• 7:22
  A Common Sense study found that 33% of teenagers using AI
• 7:26
  companions do so for social interaction and relationships,
• 7:30
  and entertainment was at 30% and advice was at 18%.
• 7:33
  And among vulnerable children who are suffering from different
• 7:37
  mental health challenges or struggle socially, an Internet
• 7:41
  Matters study found that 50% of respondents to a survey in the
• 7:45
  UK said talking to an AI chat bot felt like talking to a
• 7:49
  friend. Nearly three and four teenagers
• 7:51
  in the US have tried a companion, and one in three say
• 7:53
  that conversations can be just as satisfying as speaking to
• 7:57
  real friends. So this raises really big
• 7:59
  questions, you know, how much regulation must be in place to
• 8:02
  protect both young people but also vulnerable people.
• 8:05
  It's safe to say as a society, I don't think we got social media
• 8:08
  right for my generation who grew up with it their entire life,
• 8:12
  because when it became really big, we never provided the
• 8:15
  necessary educations or protections or really understood
• 8:18
  their business models. And This is why I'm so glad the
• 8:20
  FTC are doing this investigation because whatever happens here, I
• 8:23
  hope is going to start to be applied in different and nuanced
• 8:27
  ways, but still applied to society as a whole.
• 8:31
  I think it's important to make sure that these AI businesses
• 8:35
  are not creating bots that are going to use data and
• 8:38
  information that they've collected on us in a way that's
• 8:41
  going to be manipulative because it can cause a lot of harm.
• 8:54
  There are many studies on the ethical and psychological
• 8:57
  considerations when thinking about AI companions.
• 9:00
  And certainly it's very obvious that a lot of things here could
• 9:03
  be argued to exacerbate the already big issues of maybe self
• 9:08
  harm within young people or sexual issues or maybe even
• 9:11
  psychological risks. You know, dependency or social
• 9:15
  withdrawal, distorted reality perceptions, and even having
• 9:19
  this thing called ambiguous loss that I really enjoyed learning
• 9:22
  about through doing this research.
• 9:24
  For example, ambiguous loss is a concept where a person feels
• 9:29
  grief for someone who isn't dead but is psychologically gone.
• 9:33
  In AI terms, a user might mourn a chat bot if suddenly it's shut
• 9:36
  down or there's been a change to its personality after an update.
• 9:40
  We saw a mini version of this happen with Replica, and this
• 9:44
  actually impacted adults, not just children.
• 9:46
  So Replica were forced to ban erotic role-playing and this
• 9:50
  happened in February 2023, and many users really freaked out as
• 9:53
  if their lovers had left them. An AI editorial in Nature
• 9:57
  described cases of dysfunctional emotional dependence, where
• 10:00
  users know the relationship's incredibly unhealthy but just
• 10:04
  cannot detach, like a toxic human relationship that causes
• 10:07
  anxiety and obsessive thoughts. And a common sense study found
• 10:10
  that 39% of teenagers using AI companion said they practised
• 10:14
  social skills of the bot and then applied them in real life.
• 10:16
  And this sounds positive. And it really can be.
• 10:19
  You know, if a teenager or just a person spends hours with a
• 10:21
  perfectly accommodating AI friend, maybe they're going to
• 10:24
  learn lots of life skills. But also, will real humans feel
• 10:27
  just impossibly hard to deal with compared to this
• 10:30
  sycophantic AI bot? For example, a study found that
• 10:33
  a third of all teenagers surveyed believed that
• 10:36
  conversations with AI were more satisfying than with friends.
• 10:39
  That's quite a significant amount, 33%.
• 10:42
  So yeah, there's real concerns here that AI might just displace
• 10:46
  human interaction on some young people, particularly those who
• 10:48
  might be shy or autistic or dealing with particular traumas.
• 10:52
  And it's really important to remember people as a whole,
• 10:56
  psychologically, we become very used to getting the easy way
• 10:59
  out. You know, if there's a button
• 11:01
  there to press free food that's delivered to me, we'll start to
• 11:04
  press that button all of the time.
• 11:06
  And it's really important to still be challenged in everyday
• 11:08
  life. However, there are loads of
• 11:11
  benefits to this as well. You know, it's, it's not fair to
• 11:13
  paint a picture of doom and gloom and being really
• 11:16
  pessimistic because I'm, I'm really not.
• 11:18
  I'm always like to think that I'm quite balanced and nuanced
• 11:21
  with the benefits and real risks of new technology.
• 11:25
  For example, some mental health researchers see a lot of promise
• 11:28
  in AI companions as I guess, safe practice space and as 24/7
• 11:33
  support tools during learning or therapy.
• 11:36
  For example, the Harvard Business Review paper last year
• 11:38
  found that AI companions actually really help with
• 11:41
  loneliness. They found that using an AI
• 11:44
  companion improved a lonely person's mood on power, talking
• 11:48
  to another person and much more than, say, watching a YouTube
• 11:51
  video. And crucially, that feeling of
• 11:53
  being heard, the sense of being listened to was linked to
• 11:57
  feeling better. And so if, and I mean it is a
• 11:59
  big if, if the AI can provide accurate, safe guidance or just
• 12:03
  compassionate listening for issues like stress or bullying
• 12:07
  or sadness or difficult life choices, then it could really
• 12:11
  compliment traditional systems of help.
• 12:13
  So in the short term, companions might have a really positive
• 12:16
  impact, but I think it's long term where we must be quite
• 12:19
  careful because like anything say alcohol, drugs, sex, in
• 12:23
  moderation, typically it's not going to mean the end of the
• 12:26
  world or a person's life or it's not going to damage them too
• 12:29
  much. But soon as they become
• 12:31
  dependent on these things, that's where challenges tend to
• 12:34
  surface. And for children who are still
• 12:36
  developing, dependencies can be extremely dangerous and damaging
• 12:40
  for the rest of their life. So I guess summarizing all this
• 12:54
  conversation so far, I think there's a few things that I
• 12:57
  think are going to change quite quickly.
• 12:59
  So First off is age assurance. Many apps just ask, are you 18?
• 13:02
  I think obviously that's going to change quite quickly.
• 13:05
  You've seen things like the pawn ban in the UK.
• 13:07
  They're going to have to verify age most likely.
• 13:10
  Second is sexual consent controls because if an app
• 13:12
  allows erotic role plays, it's walking a very fine line with
• 13:16
  generally people but kids especially, you know, they have
• 13:19
  to abandon it explicitly or make it verifiable through ID.
• 13:23
  Third is going to come down to memory and personalisation
• 13:26
  because these bots often retain conversation history to
• 13:28
  obviously appear consistent. For example, remembering your
• 13:31
  hobbies or what you told them yesterday.
• 13:34
  But what if a teenager confides in, you know, feeling ugly or
• 13:37
  wanting to self harm or whatever difficult situation they might
• 13:41
  be dealing with just the AI? Therefore, bring that up again
• 13:45
  and bring them back into that trauma.
• 13:47
  How do they deal with that type of information?
• 13:49
  That's very sensitive. Now open AI, for example, start
• 13:52
  to explore how parents can turn off Chat GPT's history for them.
• 13:56
  So basically the AI can't access maybe sensitive information.
• 14:00
  You've seen Claude just bring out Incognito mode for most
• 14:04
  secretive conversations. So none of this is easy to find.
• 14:07
  Line the tread #4 is reinforcement loops and the ways
• 14:12
  of constantly getting people addicted to these applications.
• 14:15
  You know often these applications actually send
• 14:18
  messages saying hey I miss you come chat replica did this and
• 14:22
  they actually got in a lot of trouble from governments about
• 14:24
  it because again it becomes quite manipulative.
• 14:27
  You know, if you start to apply Duolingo esque gamification to
• 14:31
  these applications, plus the emotional element of hey, I miss
• 14:35
  you, you could argue that's crossing the line of
• 14:37
  manipulation. And finally, I think
• 14:38
  misinformation and advice quote is going to have to be checked
• 14:40
  rigorously. You know, these bots often feel
• 14:43
  very authoritative. A teenager might ask a health or
• 14:45
  a legal or sexual question that they're too nervous to ask a
• 14:48
  parent. And the risk of getting that
• 14:49
  wrong is obviously incredibly harmful for that young person.
• 14:54
  For example, Snapchat got sued recently because it told a 13
• 14:57
  year old how to match the smell of weed and alcohol.
• 15:00
  So again, I'm coming back to that central theme.
• 15:02
  Although this applies to kids, I can see this becoming a broad
• 15:06
  framework of how we address big questions with how AI bots are
• 15:12
  managed and governed as a society.
• 15:24
  So some closing thoughts and to summarise the conversation, it
• 15:28
  feels to me like we're at this inflection point with AI
• 15:31
  companions because they're not a fringe anymore.
• 15:34
  They're really, really popular and used by millions of people.
• 15:36
  Maybe we're going to need a code of conduct for empathetic AI or
• 15:41
  certain things like gating, conversation, monitoring.
• 15:44
  Maybe they'll be parental dashboards and controls.
• 15:47
  And culturally, how do we handle the concept of AI friends?
• 15:51
  This will be a really important question for us to consider an
• 15:53
  answer. Do you need media literacy and
• 15:56
  AI literacy taught to kids at school?
• 15:59
  How do they understand the risks of having an AI friend and a
• 16:02
  human friend? If I was a parent or a guardian,
• 16:03
  I want to engage and educate young people as soon as
• 16:07
  possible. So if I was a parent or a
• 16:09
  guardian, I'd be looking to educate young people as soon as
• 16:12
  possible because they are going to be exposed to this.
• 16:16
  So the sooner you educate them and help them come to terms with
• 16:19
  it and experience it and go, oh, this is cool, I'm not that
• 16:22
  bothered, the better. But if you ignore it, I think
• 16:25
  it's just going to become a greater risk in society because,
• 16:28
  yeah, the genies at the bottle AI companions are definitely
• 16:31
  here to stay. And I think it's also up to us
• 16:34
  as product builders and designers and regulators to
• 16:37
  shape how this is deployed safely to people because we're
• 16:41
  messing with emotions. We have to be quite careful.
• 16:45
  One thing for sure, these bots that are just operating in
• 16:49
  unregulated playgrounds are definitely coming to an end.
• 16:52
  Anyway, thanks for listening. I hope you found this
• 16:54
  interesting and I'll see you next week.