Latest / Mini Scroll / Meta’s AI feature missed the mark | How YouTube & X enable deepfakes + Character.ai’s microdramas

Transcript

Meta’s AI feature missed the mark | How YouTube & X enable deepfakes + Character.ai’s microdramas · Mini Scroll

0:00
Coming up today on Mini Tech Scroll, Meta introduces and
0:03
backtracks on a new AI feature. How mainstream social media
0:07
platforms are promoting nudify tools and character AI's foray
0:11
into micro dramas. Hey everyone, and welcome to
0:14
Mini Scroll, the daily Internet culture podcast bringing you the
0:17
biggest stories from social media, the creator economy and
0:20
the digital space every Monday through Thursday and sometimes
0:24
Fridays. My name is Annabelle and I'm
0:26
your Wednesday Mini Tech Scroll host and also the deputy editor
0:29
of Australian in youth digital media brand Centennial World.
0:33
Welcome back to Maytag Scroll for another Wednesday.
0:36
I don't really think I have too many updates at the top of this
0:39
week's episode, apart from the fact that probably the biggest
0:43
story in tech this week was actually covered by Lauren on
0:47
Monday. If you want to hear everything
0:49
that has been going on with Thea, which is the platform that
0:52
Phoebe Gates Co founded and how they are currently in the midst
0:56
of a scandal and possible fraud, definitely check that episode
0:59
out. We thought it would be better to
1:01
cover it there since it was like super timely, but yes, other
1:04
than that, Please remember to rate review and subscribe.
1:07
OK, let's get into the content for this week.
1:09
So we first really need to talk about everything that went on
1:13
with Instagram and muse image last week.
1:16
This story really hit the headlines on Thursday last week,
1:19
and I was actually so annoyed that I didn't get a chance to
1:22
cover it on mini tech. But there have also been some
1:24
updates since then. So I am actually glad that I
1:27
didn't cover it last week. So in case you have haven't
1:30
heard anything about this, about a week ago Meta announced Muse
1:33
Image, which is the first media generation model developed by
1:37
Meta Superintelligence Labs, which is the AI division of
1:40
Meta. So Muse Image, according to a
1:43
blog post was described as quote our most advanced image
1:46
generation model yet. But one of the capabilities that
1:49
had been announced alongside Muse Image soon became the
1:52
centre of controversy and it was absolutely everywhere.
1:55
Basically, this feature allowed Meta users to create AI
1:59
generated pictures of public profiles from Instagram and they
2:03
could do this by simply tagging them in a Meta AI prompt.
2:07
I should know that private accounts and accounts belonging
2:10
to mine is what automatically excluded from the feature, but
2:12
the rest of us had to opt out. They tried to sell this feature
2:16
to users in the blog post by saying that Muse image was
2:19
rooted in your own world and honestly, that positioning
2:22
didn't really work. As the blog post reads quote,
2:25
you can at mention Instagram accounts in the Meta AI app to
2:28
bring specific Instagram profiles right into your images.
2:31
Whether you want to design a custom event invitation, mock up
2:34
a collaborative creative concept, or generate A
2:37
personalized graphic tagging a username, let's Meta AI use
2:41
public photos to build a visual that's ready to post.
2:44
You have control over how your content can be tagged for AI
2:47
creation with an easy setting to turn off this feature at any
2:50
time. While there is an option to turn
2:52
off the feature, as they noted in the blog post, there are so
2:55
many issues with a feature like this.
2:57
First and foremost is the fact that it is opt out as opposed to
3:00
opt in. Considering there would be a
3:03
minimal control over how your likeness could be used.
3:06
It really feels like users should have to actively agree to
3:09
this, and the fact that Meta did opt out kind of suggests to me
3:13
that they were trying to force people to engage with this
3:15
feature in one way or another. Another issue of course is that
3:19
Instagram users could not give their consent to the AI
3:22
generated content of them that would be using their likeness.
3:25
Users who hadn't heard about this update would firstly have
3:28
no idea that it exists and even still have their content used by
3:34
AI, and they also wouldn't have been notified when someone
3:36
reused it. This is obviously a major
3:39
privacy threat and it really opens the door to potential
3:42
harassment and impersonation for any public account.
3:46
And it also just kind of feels like quite a jump.
3:48
Like just because someone posts their pictures publicly, it
3:51
doesn't mean that they should have their content used like
3:54
that. Like obviously in this day and
3:55
everyone is aware that that is a possibility, but a platform
3:59
making it that accessible is another level.
4:02
When I first started hearing about all this it definitely
4:05
reminded me of Soras cameo feature from Open AI, but
4:08
somehow this was even worse. If you guys don't remember, the
4:13
cameo feature basically allowed users to quote UN quote drop
4:16
themselves into any Sora generated scene, but users also
4:20
were in control of their likeness to some degree as it
4:23
was written in the Open AI blog post at the time.
4:26
Quote with cameos you are in control of your likeness end to
4:29
end with Sora. Really.
4:31
You decide who can use your Cameo, and you can revoke access
4:34
or remove any video that includes it at any time.
4:38
Videos containing cameos of you, including drafts created by
4:41
other people, are viewable by you at any time.
4:44
Even though there seemed to be more tools in place when it came
4:46
to Cameo, people were obviously still very stressed about this
4:50
feature before Sora shut down. Obviously in saying that, there
4:53
seemed to be even less oversight with the Muse image feature.
4:58
Quickly, publications, creators and even parliamentarians here
5:01
in Australia started posting information about this new
5:03
feature and how to opt out of it if you are a public account.
5:07
SAG AFTRA was one pretty vocal critic.
5:10
The union issued a statement recommending that their members
5:12
turn off this feature, but also questioning the opt out element
5:16
as their statement read quote. Anything other than a clear and
5:19
conspicuous opt in for these types of uses of Instagram users
5:23
accounts is unacceptable and an utter miscalculation of public
5:27
sentiment regarding the obvious dangers and harms inherent in
5:30
such use. After days of this mounting
5:32
backlash, Meta has since pulled the feature.
5:35
They issued an edit to the blog post announcing news image,
5:38
including an update on July 10. This is what I read quote.
5:42
Earlier this week, we announced that one way for people to
5:44
generate images in Meta AI is by at mentioning public Instagram
5:48
accounts that they want to reference.
5:50
Our intent was to provide a useful creative tool and to give
5:54
people control over whether their public content could be
5:56
referenced in this way. We've had the feedback that this
5:59
feature missed the mark, so it's no longer available.
6:02
I mean, I don't really know if AI at a base level is equal to a
6:07
quote UN quote useful creative tool.
6:09
And I also don't think it's a valid reason for putting out a
6:12
feature like this. It really just goes to show how
6:14
out of touch these tech companies are as the sag after a
6:18
statement commented on because I feel like they're all suffering
6:21
from groupthink at this point. Assuming that everyone wants AI
6:24
everything, which couldn't be further from the truth.
6:26
You can still upload images of someone and ask Meta AI to
6:30
generate an image of them, which is obviously similar to other AI
6:34
chat bots. But this is also something that
6:36
Meta told me in a chat when I was trying to test this out
6:39
while I was researching for this story.
6:41
As the chat log read quote, you can just upload any photo you
6:45
have and say use this person slash make this person do X and
6:49
I'll generate from that. So while you can't tag them, I
6:52
guess you still can be making unconsensual AI generated images
6:57
of someone. Nevertheless, some social media
7:00
users are touting Meta's decision to pull the feature as
7:03
a win, as at Kate McGill vocals said on threads quote.
7:07
It proves fighting back actually works and I definitely think
7:10
there is some truth to that. But we can't be too naive
7:14
because at the same time, we can't forget that this comes
7:17
only like 2 weeks after the launch of Meta glasses, which is
7:21
another tech product that completely undermines our
7:24
ability to control how our likeness is used and is honestly
7:28
just another Ave. for harassment, especially for
7:30
women. I spoke about that in mini tech,
7:33
I think from 2 weeks ago. So while Meta admitted that they
7:36
quote UN quote missed the mark, it doesn't seem like this is
7:39
something that they genuinely believe or care about.
7:43
Considering they continue to promote products like Meta
7:46
Glasses, it's clear where their priority lies, and it's not
7:49
always with user safety. OK, let's get into the next
7:52
story. In more grim news, a new report
7:56
reveals that mainstream social media platforms are leading
7:59
users to tools that generate non consensual intimate images, or
8:03
in other words, nudify apps. We have had many conversations
8:08
around platforms and features that generate not safer work or
8:11
explicit content of social media users without their consent, and
8:15
this did really pick up at the beginning of this year with X's
8:18
whole undressing scandal. I did cover this whole thing in
8:22
multiple episodes of Mini Tech Scroll, but as a quick
8:25
refresher, similar to Muse AI, X rolled out a feature that
8:28
allowed users to edit a photo, any photo, without permission
8:33
from its owner or say a person in the photo.
8:35
Unfortunately, this feature was quickly used for nefarious and
8:39
disgusting reasons. Users started asking Grok to
8:42
remove clothing from people featured in various images, and
8:46
this was absolutely anyone ranging from women to minors.
8:49
Now X kind of operates in this weird space in the online world,
8:53
I would say these days. It obviously is increasingly
8:57
right wing, but you do have niches like Stan Twitter that
9:00
are still alive and well. In that sense, it is definitely
9:03
a more mainstream platform. We're allowing its associated AI
9:07
tool to do something like this is worrying because it's
9:10
obviously not some random website or platform on the dark
9:14
web or something. But this is actually part of a
9:16
bigger trend because X isn't the only pathway to this kind of non
9:21
consensual intimate image generation tool.
9:24
A new report titled Mapping the New Defied Tools Ecosystem
9:28
explored this phenomenon further.
9:30
So for a bit of context, the report was published on Monday
9:33
by the Institute of Strategic Dialogue, which is an
9:35
independent research organization based in London.
9:38
And the research they do is typically around individual
9:41
rights, democratic process, national security and public
9:45
safety. In this report, researchers
9:47
explored the pathways users take to find nudified tools, which is
9:52
obviously apps platforms features that allow users to
9:56
generate non consensual intimate images.
9:59
They also explored how the ecosystem operates more broadly,
10:02
as well as how this content is circulated.
10:05
One of the more concerning, yet perhaps unsurprising findings is
10:09
that a lot of this takes place on mainstream social media
10:12
platforms as opposed to more fringe platforms like 4 Chan.
10:17
So I thought I would recap some of the key findings.
10:20
After identifying a total of 181 websites offering tools that
10:25
allow users to generate nudified images or videos of real people,
10:29
the researchers found that these sites hit approximately 40
10:33
million unique visitors per month.
10:35
And they also found that only two of them included measures to
10:38
verify that users were over 18 years of age.
10:42
While it's not a majority, more than 5.7 million visits to these
10:46
sites and apps containing notified tools came from
10:50
mainstream social media platforms.
10:52
YouTube X, Reddit, and Facebook are the top four, with both
10:56
YouTube and X facilitating over 1,000,000 visits each.
10:59
I'm curious if you guys are surprised about these platforms
11:03
being the top four. At first glance, I was a little
11:06
bit surprised by Facebook, but then the more I thought about it
11:09
and considering everything that just happened with Muse AI that
11:11
I just covered, it's actually not that surprising.
11:14
And also, I do feel like I could see Facebook's user base going
11:18
to platforms where you can adjust an image or generate an
11:22
image using AI. Nevertheless, YouTube was the
11:26
highest on this list and as the report reads, quote the top
11:30
platform identified in terms of referrals to the top ten
11:33
websites analyzed with YouTube, with links to the site on the
11:37
platform resulting in over 1.82 million visits.
11:40
Manual searches on YouTube identified numerous videos
11:43
promoting specific Notify apps and even providing information
11:47
on how to get Quote Undress AI promo codes.
11:51
This seems in direct conflict with Youtubes policy restricting
11:54
the promotion or linking of harmful external content.
11:57
And as far as how these websites operate, a lot of them use a
12:01
freemium model which is a token based or tiered subscription
12:05
monetization model. So people also would have to pay
12:08
to use these tools, whether that be through PayPal, Stripe,
12:12
crypto, etcetera. The researchers even identified
12:15
another layout to this monetization strategy quote in
12:19
which individual non consensual intimate image content creators
12:23
or resellers use third party creator economy platforms such
12:26
as Patreon to sell access to deepfake content or generation
12:31
services. Which is obviously very
12:33
disturbing that they are making a business out of this.
12:35
While YouTube had the most referral links in the data set
12:39
for the report, X was the most prevalent social media platform
12:42
when it came to the distribution of gendered deepfake related
12:46
posts as well. As they explained, X was quote
12:50
used for direct content creation request via in platform croc
12:53
function and for promotion of notification sites.
12:57
I presume Crocs undressing scandal is a major reason behind
13:01
such a high number of direct content creation requests.
13:04
At first the feature was available to everyone, then it
13:08
was restricted, but not entirely.
13:10
It was first limited to premium paid users, which was disturbing
13:15
for many reasons because Gruck was still able to create the
13:18
images and then X was essentially profiting off it.
13:21
But since then it seemed to have been restricted further.
13:25
X has also faced multiple lawsuits over the generation of
13:30
these sexual non consensual images.
13:32
And while X has said that more restrictions are in place, it
13:36
does seem that users continue to bypass them.
13:39
According to an investigation by Wired from just last month.
13:42
The publication found quote dozens of nudified deepfake
13:46
images and videos on Grok's website, including non
13:49
consensual depictions of celebrity and at least one
13:52
prominent US politician. So it's clear that it is an
13:55
ongoing issue on X and I'm not sure if it will ever be fully
13:59
resolved at this stage or at least as with X as we currently
14:03
know it. But actually quick side note on
14:05
X. The platform is also making
14:07
headlines this week because they did I recently announced A tweak
14:10
to its algorithm to boost the visibility of your mutuals, and
14:15
many users have taken this as a win.
14:16
But anyway, it is obviously really important to call
14:20
attention to reports like this and findings like this because
14:24
it does continue to be an ongoing form of harassment that
14:27
disproportionately impacts and harms women.
14:30
And I will also link the report in the description if you're
14:32
interested in reading more about it.
14:34
OK that brings me to my last story for today which is
14:37
character AI is dabbling in micro dramas.
14:41
We have covered micro dramas here on mini tech but I don't
14:43
actually think we've covered character AI.
14:46
Micro dramas are basically like mini soap operas which are
14:50
essentially short form video clips that are part of a bigger
14:53
series. The AI generated fruit.
14:55
Love Island would probably be a UTC version I guess of this but
15:00
it honestly spans way more content than that.
15:03
For a little bit more context, it has been gaining popularity
15:06
in China for years and only now is making its way into other
15:09
markets. And I've also seen claims that
15:12
micro dramas or micro series like these are the future of
15:15
streaming. And I'd definitely be curious to
15:17
hear what you guys think about that industry wise.
15:19
I've also seen actors note how this kind of work is much easier
15:24
to kind of get your foot in the door because there are so many
15:26
more opportunities. But at the same time, I did
15:29
actually see a TikTok earlier this week from an actor saying
15:32
that she feels like AI micro dramas becoming increasingly
15:36
popular because she's singing less and less, costing calls for
15:39
these types of roles. I will inside that TikTok here
15:42
if I can find it, you know? I was like, yeah, I'm not
15:43
worried about AI taking our jobs.
15:45
They're taking them. They're taking our jobs because
15:47
there used to be 15 casting calls a day for vertical micro
15:51
dramas, whatever. Now there's one, maybe 2, and
15:55
that's my main form of income. So I'm a little bit.
15:58
Nervous, I can't lie. It's been it's been fast, but
16:00
also slow. Like I'm getting auditions for
16:03
bigger stuff, which is exciting, except for that isn't not
16:08
consistent income. Like yes, a big movie audition
16:11
is like so exciting. The likelihood of booking that.
16:15
Or like. That paying my next month's rent
16:19
low, so that's a little bit scary.
16:23
Now Character AI is an AI platform where you can build,
16:26
customize, and chat with different characters, and they
16:29
may be original characters, they could be historical figures or
16:32
anything in between. I remember Brittany Broski made
16:36
a couple videos using Character AIA few years ago.
16:38
That was really when it came under my radar.
16:41
Personally. I remember she even made a
16:43
character on Character AI but I can't remember what it was
16:46
called. If anyone remembers please let
16:50
me know. But Character AI has faced
16:53
significant controversy since then.
16:55
In 2024, a mother filed a civil suit against Character AI,
16:59
alleging that her 14 year old son died by suicide after
17:03
communicating with the chatbot. The family has since settled the
17:06
lawsuit, and Character AI has also announced new safety
17:09
features over the years. And now it seems like they're
17:12
pivoting a little bit and trying their hand at micro dramas.
17:15
So last week Character AI announced its CAI series which
17:20
is quote a new micro dramas offering of original short form
17:24
vertical series created by our in house studio team.
17:27
Per the platform's blog post, they also add that the team is
17:31
made-up of people with credits across Netflix, Nickelodeon,
17:34
DreamWorks and more. Though I should also say that
17:37
Character AI does note that these dramas were also created
17:40
with AI production tools, so it does seem to be a mixture of
17:43
both human and AI. To start up this offering, there
17:46
are three shows. The 1st is called last summer
17:49
which is a romance anime, the second one is called the
17:52
nighttime game which is a quote UN quote Gen.
17:55
Z paranormal horror and the last one is Eden Fall which is a
18:00
survival drama on why character AI is kind of going down this
18:04
path. The Hollywood Reporter
18:06
hypothesizes that quote. The move into micro drama seems
18:09
like an attempt to diversify a business that is facing
18:11
significant headwinds, moving it beyond the kinds of intimate
18:15
companion style relationships that give rise to many problems.
18:19
And while that might be part of the reasoning, I should note
18:21
that they aren't just producing these micro dramas, they are
18:24
doing things a little bit differently.
18:26
After watching over 18, users can chat with characters from
18:30
each series on character AI, and it seems to be that they can
18:33
really talk about anything, whether that be the characters
18:36
relationship with other characters, talk about specific
18:39
moments, or even role play storylines from the series.
18:44
It's kind of giving interactive fan fiction in that respect.
18:47
As the blog post continues, quote that creates A deeper
18:50
connection between fans and characters and gives our studio
18:53
team a new feedback loop as we learn which characters, worlds,
18:56
and stories people want to spend more time with.
18:59
I will say this does feel very 2026, but better off or worse
19:03
with the crossover of AI chat bots and micro dramas.
19:06
I do think there is quite a slippery slope here because
19:09
character AI has already been through a wrongful death lawsuit
19:13
as I've covered, which is tied to the parasocial relationship
19:18
ship someone develops with a chat bot.
19:20
I don't even know if parasocial is the right word here, but it
19:23
is some kind of deep emotional attachment.
19:26
And this new format and the ability to chat with the
19:29
characters after engaging with the series is arguably built on
19:33
that same dynamic and engagement strategy.
19:36
For a platform that maybe should try to limit these relationships
19:40
a little bit. That format doesn't really seem
19:42
to do that at first glance. I'd be curious to see how people
19:46
interact with it if they do. I also do think this
19:49
conversation is related to a bigger conversation we're
19:52
constantly having around AI companions, like AI girlfriends
19:55
that a lot of young men are building unhealthy relationships
19:59
with. But I'm definitely curious to
20:00
hear what you guys think about this and the connection of micro
20:03
dramas and talking with those characters.
20:05
Whether you think that is something that we could see down
20:08
the line with more mainstream TV shows or is that just a very
20:12
like chronically online thing? OK, that brings us.
20:14
To the end of Mini Tech Scroll for today.
20:17
I really hope you guys enjoyed. Lauren will speak to you
20:20
tomorrow and I will be back next week.
20:22
Thanks for listening. Bye.
20:24
Centennial World Podcast Network operates on Gadigal land.
20:27
We would like to acknowledge the Gadigal of the Aura Nation, the
20:30
traditional custodians of this land, and pay our respects to
20:33
elders both past and present.