Latest / kill switch / The Watchmen
Transcript
- 0:04Speaker 1: Sleepwalkers is a production of our heart Radio and unusual productions.
- 0:13Speaker 1: Every single thing that you do on a social network
- 0:17Speaker 1: or on a website is essentially recorded. How many pages
- 0:22Speaker 1: you visited, what did you click on, how did you
- 0:25Speaker 1: get to that website? What page did you leave on?
- 0:28Speaker 1: How many photos have you ever uploaded? Where were those
- 0:30Speaker 1: photos uploaded? How many places have you checked into? Who
- 0:33Speaker 1: have you tagged? What photographs have you been tagged in with? Whom?
- 0:37Speaker 1: Where were those photographs taken? Who's in your friendship circle?
- 0:41Speaker 1: Who did you go to school with? There are algorithms
- 0:44Speaker 1: and machine learning technologies that connect all of that data
- 0:49Speaker 1: together and start to find patterns. That's Lisa italian Aretti speaking.
- 0:54Speaker 1: She's a digital sociologist and tech ethics advocate with a
- 0:58Speaker 1: big focus on data. I think some of the data
- 1:02Speaker 1: that's quite concerning is facial recognition technology, where machines are
- 1:07Speaker 1: being fed huge amounts of images pictures of people's faces,
- 1:13Speaker 1: and that can come from dating websites, from photographs of
- 1:17Speaker 1: you on Facebook or Instagram or anywhere on the Internet.
- 1:21Speaker 1: You don't even have to upload it yourself. It could
- 1:23Speaker 1: be somebody else that's uploaded it for you. With all
- 1:26Speaker 1: that data out in the wild. All it takes is
- 1:29Speaker 1: for somebody to suck it up and they can start
- 1:31Speaker 1: connecting the dots. So there were researchers last year from
- 1:35Speaker 1: Stanford University who without the user's permission, scraped thirty thousand
- 1:41Speaker 1: photographs from a dating website that was public rights. So
- 1:46Speaker 1: they took it that they could just use those photographs
- 1:48Speaker 1: for research. Because it was a dating site, people will
- 1:52Speaker 1: also asked the data points, um, what is your sexual orientation?
- 1:56Speaker 1: So gay, straight, by right, and so they had all
- 2:00Speaker 1: of that data connected, so they could then connect faces
- 2:03Speaker 1: to sexual orientation and essentially they built an algorithm that
- 2:08Speaker 1: they said could detect just by somebody's face if they
- 2:11Speaker 1: were gay or straight or by the algorithm worked, it
- 2:15Speaker 1: could predict sexual orientation from photographs with accuracy for men
- 2:21Speaker 1: and accuracy for women based on just five photographs. And
- 2:26Speaker 1: Stanford isn't the only place using facial recognition to categorize people.
- 2:30Speaker 1: Governments are too all around the world. This is Sleepwalkers Welcome.
- 2:48Speaker 1: I'm Azveloshin. In the last episode, we looked at what
- 2:51Speaker 1: happens when artificial intelligence digests huge data sets to find
- 2:56Speaker 1: patterns and make predictions, like what's the most typical move synopsis?
- 3:01Speaker 1: Or is that moral cancerous? But advances in machine learning
- 3:05Speaker 1: are also making us more and more legible. In today's episode,
- 3:10Speaker 1: we ask what happens when we become the data set
- 3:13Speaker 1: and the power to predict is turned on us. Here's
- 3:16Speaker 1: Lisa again. The extra terrifying thing is why creates this
- 3:21Speaker 1: kind of technology? Like? Who does it serve? Why do
- 3:26Speaker 1: we need this? If you take this type of technology,
- 3:30Speaker 1: feed it to a citywide CCTV surveillance system and say
- 3:35Speaker 1: that you go to a place like Saudi Arabia where
- 3:38Speaker 1: being gay is considered a crime, and suddenly you're just
- 3:42Speaker 1: what pulling people off the street and arresting them because
- 3:45Speaker 1: you're gay because the computer said so, so like now
- 3:49Speaker 1: you're going to prison. This may sound like a terrifying
- 3:54Speaker 1: Minority Report style future, but actually it's here today. I
- 4:00Speaker 1: Cara Highs. So Lisa was talking about what happens when
- 4:04Speaker 1: governments start to connect the dots of all this data,
- 4:07Speaker 1: but it's already happening with private enterprise. For example, insurance
- 4:10Speaker 1: companies can now see someone who joins a Facebook group
- 4:14Speaker 1: about a genetic mutation and use that data to guess
- 4:18Speaker 1: that that person may have the genetic mutation and the
- 4:20Speaker 1: condition that's associated, and then the computer says, let's raise
- 4:24Speaker 1: their premium or let's deny them care. And so we
- 4:27Speaker 1: can use this peroxy data to do something The New
- 4:30Speaker 1: York Times has recently called proxy discrimination. And I think
- 4:34Speaker 1: something that's even more widely applicable is this definition of
- 4:39Speaker 1: surveillance capitalism, which is essentially that data is not just
- 4:43Speaker 1: data anymore, it's money. Companies can use data to make
- 4:48Speaker 1: predictions about future behavior and that can make them profit. Right,
- 4:53Speaker 1: that surveillance capitalism. I using information about joining a Facebook
- 4:57Speaker 1: group to make an insurance decision, and it's I don't know,
- 5:00Speaker 1: it's a little bit scary. In the US, it's all
- 5:02Speaker 1: about capitalism, But in other countries surveillance is used for
- 5:06Speaker 1: other purposes. For example, in China, it's about social control.
- 5:11Speaker 1: But in China, this same massive ingestion of data and
- 5:14Speaker 1: statistical modeling is used for governance. So let's take a
- 5:18Speaker 1: closer look at China and how they're using technology to
- 5:21Speaker 1: amplify the power of the state. They have this notion
- 5:26Speaker 1: of a social credit score, which is how good of
- 5:28Speaker 1: a citizen you are according to the government. They have
- 5:30Speaker 1: literally hundreds of millions of cameras around and they can
- 5:32Speaker 1: basically do things like you've been out at the bar,
- 5:35Speaker 1: you know, until to the last couple of nights. That's
- 5:37Speaker 1: not really what a good upstanding citizen would do. So
- 5:39Speaker 1: your social credit score can get ding because you've been
- 5:41Speaker 1: spending too much time at night at a bar. That's
- 5:44Speaker 1: Dr Alex Kilpatrick. He and Mary Haskett co founded Blink Identity,
- 5:48Speaker 1: a facial recognition company that can recognize the users face
- 5:52Speaker 1: in yes, the blink of an eye about no point
- 5:54Speaker 1: four seconds. Here's Blink co founder Mary on Chinese surveillance.
- 5:59Speaker 1: They have camera, so you're recorded jaywalking, and so your
- 6:02Speaker 1: score goes down, you know, and you automatically get a ticket,
- 6:04Speaker 1: which again doesn't sound like that they give a deal
- 6:07Speaker 1: if you really believe in law and order, except if
- 6:09Speaker 1: your score isn't high enough, you can't buy a plane ticket,
- 6:11Speaker 1: you have to travel by bus, you know, you can't
- 6:13Speaker 1: live in certain areas. And you can obviously see how
- 6:16Speaker 1: this could be abused. I mean, it doesn't take much
- 6:18Speaker 1: of an imagination. Other factors that can bring down your
- 6:21Speaker 1: social credit score include what you buy at the store,
- 6:24Speaker 1: your online browsing, and even having a friend with a
- 6:27Speaker 1: low score. This use of generalized surveillance can keep a
- 6:30Speaker 1: whole population in check, which is more or less the
- 6:33Speaker 1: explicit goal of the Communist Party of China. This can
- 6:36Speaker 1: be stifling for the average hand Chinese citizen. For minorities,
- 6:40Speaker 1: it can be much much worse, right, you know, more
- 6:43Speaker 1: specifically and more dauntingly, it can be used for internment,
- 6:47Speaker 1: you know, in the case of the weaker minority in China,
- 6:50Speaker 1: using data that is being created by this minority to
- 6:54Speaker 1: just communicate, you know, one person communicating with another person,
- 6:57Speaker 1: those who are connected that both wagas right, five weeks
- 7:00Speaker 1: are gathering the same place. Now that we know that,
- 7:03Speaker 1: you know, what are we going to do with that data? Oh,
- 7:05Speaker 1: We're going to send these people to re education camps.
- 7:08Speaker 1: And as technology improves, so does the state's ability to
- 7:12Speaker 1: project power. China today is cleaning the floor with the
- 7:16Speaker 1: Americans on voice and facial recognition technology. That's the Embrema,
- 7:22Speaker 1: an expert on global political risk and the founder of
- 7:25Speaker 1: the Eurasia Group. The Chinese have much more data. You
- 7:29Speaker 1: also have a government that is consolidating the data and
- 7:32Speaker 1: allocating it for different types of purposes. And you have
- 7:35Speaker 1: no presumption of privacy whatsoever. With no presumption of privacy,
- 7:39Speaker 1: the amount of data you can collect from your citizens
- 7:41Speaker 1: grows exponentially, and that actually gives you a huge technological advantage.
- 7:47Speaker 1: This is how Kai Fu Lee explains it. What makes
- 7:49Speaker 1: sense AI algorithm work better is how much data you
- 7:53Speaker 1: used to train it. And that's the beauty of deep learning.
- 7:56Speaker 1: You just keep throwing data at it and it just
- 7:58Speaker 1: performs better. And Kai fully understands this world better than most.
- 8:03Speaker 1: His fund, Signivation Ventures, has invested in meg v, a
- 8:06Speaker 1: facial recognition company valued at four billion dollars. Kaifu also
- 8:11Speaker 1: around Google China, so he knows the landscape. China simply
- 8:15Speaker 1: has more data than the US due to not only
- 8:19Speaker 1: the large number of users, but also the depth in
- 8:22Speaker 1: which Chinese users use the Internet for ordering food, for
- 8:26Speaker 1: shared bicycles, for mobile payments. So the AI will actually
- 8:30Speaker 1: just perform better because it's trained out more data. Some
- 8:33Speaker 1: of that data is taken from citizens using surveillance, but
- 8:37Speaker 1: according to Kaifu, much is freely given. I think the
- 8:42Speaker 1: Chinese culture and the Chinese people are more pragmatic, so
- 8:46Speaker 1: that if the software delivers pragmatic value, they ask fewer questions.
- 8:53Speaker 1: For example, you know, we've funded an app that loans
- 8:57Speaker 1: money to ask you a couple of questions and takes
- 9:00Speaker 1: data from your phone at the same level as the
- 9:02Speaker 1: Facebook would take data from your phone, and it's SAPs
- 9:05Speaker 1: the money to you instantly if it decides to lend
- 9:08Speaker 1: money to you. I think in the US people my
- 9:10Speaker 1: question do I really want to give my data to
- 9:12Speaker 1: a lending application? And is it appropriate to consider the
- 9:17Speaker 1: makeup my phone as a part of that give me
- 9:19Speaker 1: giving me money or my zip codes? Because that might
- 9:22Speaker 1: reflect certain things about me. The breadth and depth of
- 9:27Speaker 1: data in China, both from a larger population and a
- 9:30Speaker 1: much deeper integration with technology, gives China a serious competitive advantage.
- 9:35Speaker 1: Where the Americans have much better scientists, the Chinese were
- 9:39Speaker 1: able to buy a lot of science. Now, when you
- 9:41Speaker 1: talk to specialists in this field, they will tell you
- 9:44Speaker 1: that in many parts of Ai, great data and okay
- 9:48Speaker 1: scientists will frequently beat great scientists and okay data. Character
- 9:54Speaker 1: scary thing is that China is using that technological superiority
- 9:57Speaker 1: to build a very different kind of state, you know,
- 10:00Speaker 1: one in which the price of descent is intolerably high.
- 10:03Speaker 1: You know. So according to Ian public demonstrations have fallen marketly, right,
- 10:08Speaker 1: because if you know you're being watched, you're probably less
- 10:11Speaker 1: likely to commit public displays of civil disobedience. Right. We
- 10:14Speaker 1: talked about the vigas earlier in Cashgar, which is a
- 10:17Speaker 1: Muslim city in China. Vigas have to register to go
- 10:20Speaker 1: into the mosque, and once they're inside, they face a
- 10:24Speaker 1: bank of cameras like many many surveillance cameras, and go
- 10:28Speaker 1: figure Muslims stopped going to mosque voluntarily well, because going
- 10:33Speaker 1: to mask is an act of civil disobedience where they are.
- 10:37Speaker 1: Even if it's not explicitly stated, it's it's heavily implied, right.
- 10:41Speaker 1: I mean, I think about it for myself, Like if
- 10:42Speaker 1: I were living in an area in the United States
- 10:45Speaker 1: where going to temple was going to land me in
- 10:47Speaker 1: an internment camp, I would not be going voluntarily if
- 10:51Speaker 1: I knew there were security cameras all over my temple. Absolutely,
- 10:54Speaker 1: And the crazy thing is you wouldn't even have to
- 10:57Speaker 1: know if those security cameras work. It's like on the
- 11:00Speaker 1: o TO in England, we have a lot of speed
- 11:01Speaker 1: cameras and no one knows that they're actually doing it.
- 11:03Speaker 1: It's staying apart from once every five years you've got
- 11:05Speaker 1: a ticket. But it still slows people down the panopticon
- 11:09Speaker 1: even if they can't do that. But people think they
- 11:12Speaker 1: can do that, right. I mean, you don't need a
- 11:14Speaker 1: hundred percent certainty. You just need a government that is
- 11:17Speaker 1: starting to get that capacity and make it known and
- 11:19Speaker 1: have a few people that are sort of strung up
- 11:21Speaker 1: as examples, and suddenly everyone's scared. And this isn't only
- 11:26Speaker 1: happening in China. According to Ian, it was a key
- 11:29Speaker 1: part of Bashar al Assad's strategy in the Syrian Civil War.
- 11:33Speaker 1: Assade got some help from the Russians, who gave them
- 11:36Speaker 1: a couple of hundred computer scientists to go in work
- 11:39Speaker 1: with the Syrian military and identify on social media and
- 11:43Speaker 1: on text messaging who wore those Syrian citizens that were
- 11:47Speaker 1: nodes of dissent and within six months no more moderate
- 11:52Speaker 1: opposition in Syria. They specifically, we're looking into individual Syrian
- 11:58Speaker 1: citizens that we're saying things about the regime that we're untoward,
- 12:02Speaker 1: that we're connected to influencers that were helping to organize protests,
- 12:06Speaker 1: and suddenly, you know, a bunch of those people were
- 12:10Speaker 1: rounded up and some were never heard from again. And
- 12:13Speaker 1: as I mentioned in terms of China, you don't have
- 12:16Speaker 1: to do that with many people before people start ratting
- 12:19Speaker 1: out their friends, being scared of talking to anyone not
- 12:22Speaker 1: going out. The system worked. We may feel comfortably far
- 12:28Speaker 1: from the battlefields of Syria here in the US, and
- 12:31Speaker 1: from the overwhelming number of surveillance cameras on practically every
- 12:34Speaker 1: street corner in China. But the more effective these technologies are,
- 12:38Speaker 1: the more likely they are to be adopted by others. Now,
- 12:41Speaker 1: in other countries, you're going to have a confluence of
- 12:44Speaker 1: both them liking the model and the Chinese directly exporting it.
- 12:49Speaker 1: Who were those countries, Well, look at One Belt, One Road,
- 12:52Speaker 1: the you know, trillion plus dollar investments that the Chinese
- 12:55Speaker 1: are making all over the world in Pakistan, Southeast Asia,
- 12:58Speaker 1: you know, Cambodia, laughs, a whole bunch of countries. And
- 13:01Speaker 1: when you look at those countries and you see that
- 13:04Speaker 1: the Chinese are providing the money in this conditionality in return,
- 13:08Speaker 1: some that conditionalities use Chinese standards for technology that's in
- 13:11Speaker 1: many of these contracts. And with the spread of Chinese
- 13:14Speaker 1: standards of technology comes the spread of Chinese style surveillance,
- 13:18Speaker 1: which could ultimately make the whole world trend more authoritarian.
- 13:22Speaker 1: So as that happens, these governments are going to say,
- 13:26Speaker 1: ah ha, we get the money from China, we use
- 13:29Speaker 1: their technology. We're stuck with their system, but we can
- 13:33Speaker 1: use it to ensure that our people stay in power. Again,
- 13:37Speaker 1: it's easy to let all of this feel comfortably far away,
- 13:41Speaker 1: but remember the Internet doesn't have borders, so you don't
- 13:44Speaker 1: have to be in China for the Chinese state to
- 13:46Speaker 1: access your data. Yeah. I don't know how many people
- 13:49Speaker 1: know this, but Grinder is actually owned by a Chinese company,
- 13:52Speaker 1: Grinder the dating MP. Yes, and actually there have been
- 13:55Speaker 1: articles about the fact that the US government is trying
- 13:58Speaker 1: to force China's hands so that we can buy it
- 14:01Speaker 1: back because there's so much user data that this company
- 14:05Speaker 1: now owns. Well, that Grinder use a data is basically
- 14:08Speaker 1: being seen by the US government as a strategic asset.
- 14:11Speaker 1: It is a strategic asset. I mean, if you think
- 14:13Speaker 1: about it, if somebody is on a military base or
- 14:16Speaker 1: in a barrack and trying to connect with someone on Grinder,
- 14:21Speaker 1: they're turning their location services data on because they want
- 14:24Speaker 1: to see people in their area and if they're turning
- 14:26Speaker 1: that location services data on, they're basically making themselves vulnerable
- 14:31Speaker 1: to the company that owns the data, because it's basically
- 14:34Speaker 1: saying here I am, here, I am and not only that,
- 14:36Speaker 1: here I am, here, I am, and I'm gay, and
- 14:39Speaker 1: that can lead to some possibilities of blackmail. Even today, exactly,
- 14:43Speaker 1: there was an article in the interface that was written
- 14:45Speaker 1: by this guy, Casey Newton, chat history, photos, videos, real
- 14:49Speaker 1: time location. All of that is connected to a user's
- 14:52Speaker 1: email address, and that means that the user's identity can
- 14:56Speaker 1: be very easily learned. That's pretty scary. And even if
- 15:00Speaker 1: don't use Grinder, you might use Flash of Clans or Fortnite,
- 15:03Speaker 1: which are very popular gaming apps also owned by the Chinese.
- 15:07Speaker 1: Now we keep saying the Chinese. To be clear, these
- 15:09Speaker 1: apps aren't owned by the Chinese government, that owned by
- 15:13Speaker 1: Chinese companies. What the actions of the US government imply
- 15:16Speaker 1: by trying to force this company to sell Grinder back
- 15:20Speaker 1: is that they don't believe in the distinction. And do
- 15:23Speaker 1: you think the US government would really be working that
- 15:25Speaker 1: hard to get back a gay dating app if they
- 15:29Speaker 1: didn't think that there was not a murky separation between
- 15:32Speaker 1: the government and companies in China. Right, So every time
- 15:36Speaker 1: we give data away, I mean, we're aware that it
- 15:39Speaker 1: opens us up to targeted ads on Facebook. We talked
- 15:41Speaker 1: about those with Gillian. We're not aware that that data
- 15:45Speaker 1: may end up in the hands of potentially hostile foreign government.
- 15:50Speaker 1: So once again, Sleepwalking. We've been talking about how foreign
- 15:55Speaker 1: governments are using AI, but when we come back, we'll
- 15:58Speaker 1: look at how the police and courts are using it
- 16:00Speaker 1: at home in America. Kara, it's easy to look at
- 16:09Speaker 1: China and to see the big bad wolf. They're using
- 16:12Speaker 1: surveillance technology for the wholesale suppression of an ethnic minority.
- 16:16Speaker 1: They have a social credit score that con emit access
- 16:18Speaker 1: to opportunities and even travel, But algorithms also determined outcomes
- 16:23Speaker 1: here in the US exactly if you think about it,
- 16:26Speaker 1: we do use social ratings. If you use Uber and
- 16:29Speaker 1: have a rating lower than four out of five stars,
- 16:31Speaker 1: you can't get a car right and you can't get
- 16:33Speaker 1: alone if you have a low FICO credit score. And
- 16:36Speaker 1: our criminal justice system also uses algorithmic ratings to decide
- 16:40Speaker 1: people's fate. When I got arrested at sixteen, I was
- 16:45Speaker 1: in high school and John F. Kennedy High School. That's
- 16:48Speaker 1: Glenn Rodriguez. When he was a baby, Glenn's mother was murdered,
- 16:51Speaker 1: and when he was three, his father committed suicide. From
- 16:54Speaker 1: then on, Glenn was raised by his grandmother and he
- 16:58Speaker 1: searched for belonging. The kid who wants to feel accepted,
- 17:02Speaker 1: wants to feel a part of something right. Whatever the
- 17:05Speaker 1: group was up for, I was down. And if they
- 17:08Speaker 1: were going one step forward, I would take two. We
- 17:11Speaker 1: pretty much planned a robbery at a car dealership in
- 17:14Speaker 1: Queen's and we entered the premises. We took three cars.
- 17:20Speaker 1: There was a twenty five year old men in there,
- 17:24Speaker 1: and he initially pulled a gun, and so I had
- 17:27Speaker 1: a gun and I shot him. Glenn was arrested and
- 17:32Speaker 1: convicted of second degree murder. He was sentenced to twenty
- 17:35Speaker 1: five years in jail, and he was still a high schooler.
- 17:38Speaker 1: You feel powerless, you feel hopeless, especially at that age.
- 17:41Speaker 1: So the way I saw it was, this is my life.
- 17:44Speaker 1: You know, I'm probably gonna die in jail, and so
- 17:46Speaker 1: whatever it is that I have to do, I need
- 17:48Speaker 1: to survive. One of the things that I learned very
- 17:50Speaker 1: quickly is that in prison, one of the only things
- 17:53Speaker 1: that is respected is violence, and so in order for
- 17:56Speaker 1: you to survive in there, you have to be violent,
- 17:58Speaker 1: because otherwise you'd be come pray. In time, Glenn established
- 18:02Speaker 1: his reputation and started to feel safer. With that security
- 18:06Speaker 1: and getting older, his thinking began to change. And it
- 18:09Speaker 1: wasn't until later, to like my mid twhenies, when I
- 18:12Speaker 1: started saying, you know what, I need to reverse this
- 18:15Speaker 1: trend if I want to have any chance at parole.
- 18:18Speaker 1: Glenn had to reverse thirteen years of behavior to survive
- 18:22Speaker 1: in prison. He had learned to behave one way, but
- 18:24Speaker 1: to get out he had to behave another. I reveiled
- 18:27Speaker 1: myself of the Puppies behind Bars program, so I was
- 18:30Speaker 1: training service dogs for wounded war veterans for five years.
- 18:33Speaker 1: That was an amazing experience, right because throughout incarceration it's
- 18:36Speaker 1: almost like you build a wall around yourself with the dogs.
- 18:41Speaker 1: You can't fake it with the dogs. If you're trying
- 18:43Speaker 1: to like teach them a command, sometimes you may have
- 18:45Speaker 1: to be silly, which guess what, in prison, being silly
- 18:48Speaker 1: is not acceptable. That's perceived as a weakness. But with
- 18:51Speaker 1: that program, often times you had the resort of being
- 18:53Speaker 1: silly and throwing yourself on the floor and giggling loud
- 18:57Speaker 1: and making all kinds of crazy sounds to try to
- 18:59Speaker 1: get the dogs attend gin Right. To a very large extent,
- 19:01Speaker 1: I believe that that program kind of helped me regain
- 19:04Speaker 1: my humanity as well as helping Glenn. Personally. Taking part
- 19:08Speaker 1: in prison programs for the public good is looked upon
- 19:11Speaker 1: favorably by parole boards. So everything I did, I wanted
- 19:14Speaker 1: to document the kind of showcases what I've done, this
- 19:17Speaker 1: is who I am today. As part of the process,
- 19:20Speaker 1: there's also the Campus Risk Assessment COMPUS stands for Correctional
- 19:25Speaker 1: Offender Management Profiling for Alternative Sanctions. It's an algorithm that
- 19:30Speaker 1: claims to be able to predict how likely a defendant
- 19:33Speaker 1: is to commit another crime based on a list of
- 19:36Speaker 1: a hundred and thirty seven questions. Since being developed in
- 19:41Speaker 1: its estimated COMPASS has been used to assess more than
- 19:43Speaker 1: one million defendants, including Glenn. You meet with this person
- 19:48Speaker 1: a few months before your schedule parole board day, and
- 19:52Speaker 1: they asked you a series of questions. And so when
- 19:54Speaker 1: he got to the disciplinary history section of the Campus
- 19:58Speaker 1: Risk Assessment, there was a list of offenses right for
- 20:02Speaker 1: him to check off. Yes or no for the past
- 20:03Speaker 1: twenty four months, and it was all known. And anyone
- 20:06Speaker 1: who has any experience with prison would tell you that
- 20:08Speaker 1: that is almost impossible to do, right, because misbehavior reports
- 20:12Speaker 1: can be for something as simple as having too many pillows,
- 20:15Speaker 1: something as simple as your parents hanging off your bot,
- 20:18Speaker 1: your sneaker is untied. It takes a lot of energy
- 20:21Speaker 1: to dodge a misbehavior report during the course of a year,
- 20:25Speaker 1: let alone ten, and in my case, it had been eleven.
- 20:29Speaker 1: And then I heard him read the question and he says,
- 20:31Speaker 1: does this person appear to have notable disciplinary issues? And
- 20:36Speaker 1: he says yes, And I was like, hold up, wait
- 20:39Speaker 1: a second, did I just hear you right? Because I
- 20:41Speaker 1: just heard you say that I have notable disciplinary issues?
- 20:45Speaker 1: Do you do realize that I haven't had a misbehavior
- 20:47Speaker 1: report in over a decade? Right? And his answer was well,
- 20:50Speaker 1: I was told that if there's any instance of misbehavior
- 20:54Speaker 1: at any point, I have to check yes for this answer.
- 20:56Speaker 1: So I was like, okay, So at that point there
- 20:58Speaker 1: was nothing I could do. I'm appearing before the parole
- 21:04Speaker 1: board panel. I presented to them a portfolio that was
- 21:07Speaker 1: approximately one hundred pages, had letters to support Now the
- 21:12Speaker 1: Compass is saying that I'm a disciplinary issue and so
- 21:16Speaker 1: I shouldn't be released. But I was denied because of
- 21:19Speaker 1: the fact that I scored high on Compass. They played
- 21:21Speaker 1: a safe and kept me in. It may have been
- 21:24Speaker 1: less than five minutes the hearing. I waited twenty six
- 21:27Speaker 1: years to sit in front of a panel of three
- 21:30Speaker 1: people for less than five minutes. No one wants to
- 21:33Speaker 1: be the one to go against Compass, and next you know,
- 21:35Speaker 1: something goes wrong and now your job is on the
- 21:37Speaker 1: line because you departed from Compass, which is taken as
- 21:42Speaker 1: factual and scientific. In time, Glenn went before another parole
- 21:48Speaker 1: board and this time they freed him against the recommendation
- 21:52Speaker 1: of Compass. And now Glenn has built a life for
- 21:54Speaker 1: himself working with teenagers at risk, giving conceration at the
- 21:58Speaker 1: Center for Community Alternatives. But he's still being affected by
- 22:02Speaker 1: the algorithm. Compass does not end upon your release because
- 22:06Speaker 1: the same Compass risk assessment that's considered for your release.
- 22:10Speaker 1: The term is how you're going to be supervised upon release.
- 22:14Speaker 1: There's a number of restrictions that I have. I have
- 22:16Speaker 1: a curve you. I'm still haunted by a Compass. Despite
- 22:21Speaker 1: turning his life around Compass is still limiting Glen's freedom,
- 22:26Speaker 1: and that should haunt all of us. According to propublic A,
- 22:29Speaker 1: Compass inaccurately labels black defendants as likely to reoffend twice
- 22:34Speaker 1: as often as white defendants. Algorithmic discrimination isn't government policy
- 22:39Speaker 1: here in the US like it is against the weak
- 22:41Speaker 1: as in China, but it still exists. There's this issue
- 22:46Speaker 1: where you can have computer scientists building a more accurate algorithm,
- 22:50Speaker 1: but on account of dubious input factors like gender or
- 22:55Speaker 1: race or religion, you've created something that's unconstitutional. That's Jason
- 22:59Speaker 1: tchet It. He introduced us to Glenn, and he's the
- 23:02Speaker 1: founder of Justice Codes and a legal affairs writer for
- 23:05Speaker 1: the American Bar Association Journal. There's this predisposition to believe
- 23:10Speaker 1: that math doesn't carry all of the biases that humans do.
- 23:16Speaker 1: It's an objective science. I think we need to dispel
- 23:19Speaker 1: that idea. Jason is describing the very human habit of
- 23:23Speaker 1: taking computer output as gospel truth. It's called automation bias,
- 23:28Speaker 1: and it's why parole boards often don't feel comfortable over
- 23:31Speaker 1: writing algorithms like Compass, and why some people follow their
- 23:35Speaker 1: GPS even when it has them driving into the ocean.
- 23:38Speaker 1: This idea that somehow, because math is an underlying force
- 23:43Speaker 1: to these tools, makes them more objective or beyond certain
- 23:46Speaker 1: types of scrutiny is wrong. Computer algorithms are being used
- 23:51Speaker 1: to determine human fate today, whether it's Compassed in the
- 23:55Speaker 1: US or the social credit score in China, So we
- 23:58Speaker 1: have to scrutinize them and understand that their output is
- 24:01Speaker 1: not necessarily neutral. The foundational principle of AI is using
- 24:06Speaker 1: historical data to predict what will happen next, and that
- 24:10Speaker 1: in itself is a challenge to our culture because the
- 24:13Speaker 1: American dream is built on the idea that we have
- 24:16Speaker 1: a capacity to change, that we can move from rags
- 24:19Speaker 1: to riches, from the penitentiary to the boardroom. And it's
- 24:22Speaker 1: not just an American narrative. It's Scrooges change of heart
- 24:26Speaker 1: delivering the turkey to the Cratchets on Christmas Day. It's
- 24:29Speaker 1: Saint Paul's conversion on the Road to Damascus. It's at
- 24:33Speaker 1: the very heart of Western culture. But algorithms like Compass
- 24:38Speaker 1: aren't built to see the potential in people. They're designed
- 24:41Speaker 1: to calculate risk based on past actions, and Compass isn't
- 24:46Speaker 1: the only example of algorithms being used in our criminal
- 24:48Speaker 1: justice system. When we come back, we go right inside
- 24:52Speaker 1: the NYPD to understand how new technology is powering policing.
- 25:03Speaker 1: It's a freezing cold day in New York City when
- 25:05Speaker 1: we arrive at the NYPD headquarters, and before we even
- 25:08Speaker 1: get into the main building, Julie and I have to
- 25:11Speaker 1: pass through airport style security and naturally give up some data,
- 25:15Speaker 1: including submitting a selfie in a kiosk. Yes, right now,
- 25:21Speaker 1: our society is holding big conversations about body cameras, police accountability,
- 25:26Speaker 1: and government monitoring, so we had to ask how does
- 25:30Speaker 1: one of the most recognizable police forces in the world
- 25:33Speaker 1: handle our data. My name is Benjamin Singleton, director of
- 25:37Speaker 1: Analytics at the NYPD, so I probably spend half my
- 25:41Speaker 1: day writing code and half my day and meetings. The
- 25:44Speaker 1: police Department collects records as a regular course of our business.
- 25:47Speaker 1: We respond to nine when one calls, We take crime reports,
- 25:51Speaker 1: we make arrests, We issue moving summons. Is when you
- 25:54Speaker 1: you know, speed in the city. These are examples of
- 25:57Speaker 1: the kind of data that we collect. You know, I
- 25:59Speaker 1: think there's probably some sentiment that there are back doors
- 26:03Speaker 1: into various systems UM, but the NYPD is governed by
- 26:06Speaker 1: the same legal processes as any other UM law enforcement agency.
- 26:11Speaker 1: If we want data from an outside company or vendor,
- 26:14Speaker 1: we get a search warrant from a judge or through
- 26:16Speaker 1: a d a's office issue with subpoena, and that's how
- 26:19Speaker 1: we collect our data. Their cameras throughout the subways, white
- 26:22Speaker 1: corset turnstyles, and easy pause readers on the roads and
- 26:26Speaker 1: more in New York. So what might the NYPD know
- 26:29Speaker 1: about me? If you haven't sort of stood in front
- 26:32Speaker 1: of a police officer who hasn't taken a report by hand,
- 26:35Speaker 1: we probably don't have records on you. That being said,
- 26:38Speaker 1: we do collect data through sensors like license plate readers UM,
- 26:42Speaker 1: and we do have data sharing agreements with some other
- 26:44Speaker 1: criminal justice agencies like corrections, like the courts, and so
- 26:49Speaker 1: there's obviously opportunities for that kind of data to enter
- 26:53Speaker 1: our realm. But one thing that's built into every single
- 26:56Speaker 1: NYPD application is an auditing track. So anytime you look
- 27:00Speaker 1: at any piece of information, no matter what system you're in,
- 27:03Speaker 1: that's being audited. And so we have a very large
- 27:07Speaker 1: internal affairs bureau, and people have gotten in trouble before
- 27:10Speaker 1: for misuse of computer systems, and so I think that
- 27:13Speaker 1: that's an important check that's reassuring to hear. But why
- 27:16Speaker 1: collects so much data in the first place. I think
- 27:19Speaker 1: that the next frontier of machine learning in policing is
- 27:23Speaker 1: bringing decisions and information into the hands of cops who
- 27:27Speaker 1: need to make decisions quickly. We recently rolled out tens
- 27:31Speaker 1: of thousands of mobile phones to all of our cops,
- 27:34Speaker 1: and putting a computer in their hands has really changed
- 27:37Speaker 1: the way that the police. When you have more information,
- 27:39Speaker 1: you can make better decisions. So we could be responding
- 27:43Speaker 1: to a job at a specific location in a building,
- 27:46Speaker 1: and we know what's happened at that building before we
- 27:49Speaker 1: responded to nine. When one calls their last week in apartment,
- 27:52Speaker 1: you know for see and in that interaction it led
- 27:55Speaker 1: to some sort of altercation or we found out that
- 27:58Speaker 1: that person that we interacted have had some sort of issue. Well,
- 28:01Speaker 1: the cop who's working today might not be the same
- 28:03Speaker 1: cop who is working a week ago. And so how
- 28:05Speaker 1: do I convey that information? Maybe in a phone as
- 28:08Speaker 1: a as a pop up, as a notification that tells
- 28:10Speaker 1: you take extra time, take caution. Um, this sort of
- 28:14Speaker 1: incident happened. Using data to give office as context is
- 28:18Speaker 1: hard to argue against if it can lead to safer
- 28:21Speaker 1: interactions for everyone. But of course, what many people find
- 28:25Speaker 1: more concerning is ambient surveillance. Surveillance that happens all the time,
- 28:29Speaker 1: and despite much pressure, the NYPD has yet to release
- 28:32Speaker 1: an explicit facial recognition policy. And where do the effort
- 28:36Speaker 1: stand on facial recognition technology. Our Facial Identification Section, which
- 28:41Speaker 1: sits under the Detective Bureau, is a group of trained
- 28:45Speaker 1: detectives investigators. They use a tool and algorithm that compares
- 28:50Speaker 1: faces that we might get from a surveillance photo, and
- 28:53Speaker 1: they run that algorithm, get potential matches and then conduct
- 28:57Speaker 1: an investigation. But it's not as simple as you know,
- 29:00Speaker 1: a facial recognition hit occurs and that's suddenly licensed to
- 29:04Speaker 1: make an arrest. It doesn't generate probable cause for us.
- 29:08Speaker 1: We still require much more evidence in order to make
- 29:11Speaker 1: a determination that that hit is truly viable and something
- 29:15Speaker 1: we can act on. But there are cases where that
- 29:18Speaker 1: technology has been used as part of an arrest or prosecution.
- 29:25Speaker 1: In the absence of an explicit policy, Ben wasn't able
- 29:28Speaker 1: to answer the question live in the room, but we
- 29:31Speaker 1: did get a statement from the m I p D.
- 29:34Speaker 1: The NYPD has moved deliberately and responsibly in the use
- 29:37Speaker 1: of facial recognition software. There is no NYPD case where
- 29:41Speaker 1: an arrest or prosecution was brought on the basis of
- 29:44Speaker 1: facial recognition. The NYPD uses it on a case by
- 29:47Speaker 1: case basis, and the case must always be supported by
- 29:50Speaker 1: further investigation before any arrest is made. The NYPD has
- 29:55Speaker 1: absolutely no interest in wholesale surveillance, which would be an
- 29:58Speaker 1: enormous and entirely pointless task. We have little choice but
- 30:04Speaker 1: to trust. But that said, Bend is beat convincingly about
- 30:07Speaker 1: how the NYPD actually uses technology to police themselves. There's
- 30:11Speaker 1: also statistical tools around fairness that can actually measure whether
- 30:15Speaker 1: an algorithm is fair, whether it's causing bias, etcetera. And
- 30:19Speaker 1: so we're very interested in utilizing these metrics and we
- 30:23Speaker 1: fully embrace them. We we want to get better, and
- 30:26Speaker 1: we're taking a conservative approach because we know how high
- 30:31Speaker 1: stakes this is. The stakes are high and the path
- 30:35Speaker 1: is murky. I didn't know what to expect at the NYPD.
- 30:39Speaker 1: Would they optimize purely for reducing crime or they take
- 30:42Speaker 1: a broader view of justice. Personally, I found ben reassuring,
- 30:47Speaker 1: but the potential for abuse remains. So how do we
- 30:51Speaker 1: here in America god against that abuse. Well, let's return
- 30:55Speaker 1: to Mary Haskett who found a blink identity with Alex Kilpatrick.
- 31:00Speaker 1: Anytime you're using face wreck without consent, it's going to
- 31:02Speaker 1: get abused because why wouldn't it. And and here's the problem.
- 31:06Speaker 1: I don't think it's appropriate to ask a police department
- 31:09Speaker 1: to just voluntarily not use a tool that's awesome for them.
- 31:13Speaker 1: I mean you need to have a different level. You
- 31:15Speaker 1: need to have your governor, federal states, some government governing
- 31:19Speaker 1: body needs to be saying sorry, this is not appropriate.
- 31:23Speaker 1: This is violating people's rights. The difference between what is
- 31:26Speaker 1: happening in China and in the US is not technological,
- 31:30Speaker 1: it's cultural and political. Edward Snowden had a phrase for this,
- 31:35Speaker 1: turnkey tyranny, meaning that the technical infrastructure of mass surveillance
- 31:39Speaker 1: already exists, and that we're only protected by our values
- 31:43Speaker 1: and our laws. And thinking of China, I think there
- 31:46Speaker 1: are some profoundly creepy things that we are right on
- 31:49Speaker 1: the edge of starting to see. There's cameras everywhere. If
- 31:52Speaker 1: you add face recognition. It's not just oh, they saw
- 31:55Speaker 1: my face, they saw that I went to Starbucks. It's
- 31:59Speaker 1: where you were every day, every time, all of her
- 32:02Speaker 1: history and all that gets saved. It's my pattern of
- 32:05Speaker 1: where I go when I'm outdoors forever. Five years ago,
- 32:09Speaker 1: I would have said that could never happen here. Part
- 32:12Speaker 1: of the reason Mary has so much about privacy is
- 32:15Speaker 1: that she knows how quickly facial recognition is spreading. In fact,
- 32:18Speaker 1: in twenty eight Blink Identity raise money from Live Nation
- 32:22Speaker 1: ticket Master's parent company to allow future concert goers to
- 32:27Speaker 1: use their faces instead of their tickets. We wanted to
- 32:30Speaker 1: be a case study of how to do this in
- 32:32Speaker 1: a way that preserves individual privacy and respects the individual,
- 32:36Speaker 1: and maybe that will help set a precedence and maybe
- 32:38Speaker 1: some of these other objectionable use cases just won't be
- 32:41Speaker 1: able to take off. Facial recognition and other AI technologies
- 32:45Speaker 1: are being developed all over the world, and we can't
- 32:48Speaker 1: trust everyone to be as conscientious as Mary and Alex
- 32:52Speaker 1: In America, the liberty we take for granted is hard
- 32:54Speaker 1: one and fragile, and cases like Glens show what can
- 32:58Speaker 1: happen when algorithms are blind trusted to determine outcomes. So
- 33:03Speaker 1: much hangs in the balance right now about our technological future,
- 33:07Speaker 1: and the decisions we take will affect our lives profoundly
- 33:10Speaker 1: and echo through the lives of our children too. I
- 33:13Speaker 1: mentioned Charles Dickens Christmas Carol earlier to me. One of
- 33:16Speaker 1: the most powerful scenes in the book is Scrooge seeing
- 33:19Speaker 1: for the first time the chains he has made for
- 33:21Speaker 1: himself through his own decisions. Nowadays, we would call those
- 33:25Speaker 1: decisions and those chains longitudinal data, and they'd be very
- 33:29Speaker 1: hard to get rid of. They're the record that Lenn
- 33:32Speaker 1: couldn't shake, that might deny a Chinese citizen a plane ticket,
- 33:36Speaker 1: or deny you health insurance because of your social media activity.
- 33:41Speaker 1: But data can also set us free. In the next episode,
- 33:44Speaker 1: we investigate what's possible when our data is used to
- 33:47Speaker 1: help us. From a dying man brought back to his youth,
- 33:50Speaker 1: to movies and music that read our bodies while they play,
- 33:54Speaker 1: and what happens when Alexa becomes part of the family.
- 33:58Speaker 1: How long that Giuliana dived with m hmmm, I don't
- 34:03Speaker 1: know that one. I am still learning more about dinosaurs
- 34:07Speaker 1: us some dinosaur trivially. Sleepwalkers is a production of I
- 34:24Speaker 1: Heart Radio and Unusual productions. For the latest AI news,
- 34:29Speaker 1: live interviews, and behind the scenes footage, find us on Instagram,
- 34:32Speaker 1: at Sleepwalker's podcast or at Sleepwalker's podcast dot com. Special
- 34:37Speaker 1: thanks this episode to Laurie Arlam and Lucy Brady. Sleepwalkers
- 34:41Speaker 1: is hosted by me Osbaloshen and co hosted by me
- 34:44Speaker 1: Kara Price, with produced by Julian Weller with help from
- 34:47Speaker 1: Jacopo Penzo and Taylor Koin. Mixing by Tristan McNeil and
- 34:51Speaker 1: Julian Weller. Our story editor is Matthew Riddle. Recording assistance
- 34:54Speaker 1: this episode from Dina Bridgett, Rachel London, and Phil Bodger.
- 34:58Speaker 1: Sleepwalkers is Exactly You, produced by me Osloschen and Mangesh Hattikiler.
- 35:04Speaker 1: For more podcasts from my Heart Radio, visit the I
- 35:06Speaker 1: Heart Radio app, Apple Podcasts, or wherever you listen to
- 35:09Speaker 1: your favorite shows.