Latest / kill switch / could you be arrested by a computer?
Transcript
- 0:13Speaker 1: December of twenty twenty, during the pandemic cold winter morning,
- 0:18Speaker 1: the security guard on a train platform and the outskirts
- 0:21Speaker 1: of Saint Louis was working his job and suddenly saw
- 0:27Speaker 1: a man in a security booth who wasn't supposed to
- 0:31Speaker 1: be there. So he approached this man and told him
- 0:34Speaker 1: to leave, and then the verbal altercation ensued and another
- 0:39Speaker 1: man walked up behind him, so he was surrounded by
- 0:41Speaker 1: these two guys who were getting more and more agitated
- 0:44Speaker 1: with him. The next thing he knew, he was assaulted
- 0:47Speaker 1: by these two guys, struck in the head, and when
- 0:51Speaker 1: he was on the ground, they continued to beat him
- 0:53Speaker 1: and hit him in the head and hit ananimous chest
- 0:56Speaker 1: and his ribs. The guys ran off and Michael and
- 1:00Speaker 1: the victim suffered concussion that day and he couldn't remember anything.
- 1:05Speaker 1: So what the police were really left with to solve
- 1:08Speaker 1: this case is the same thing that they're left with
- 1:12Speaker 1: in a lot of cases where there's no witnesses.
- 1:15Speaker 2: A random crime, a victim with no memory, and nobody
- 1:19Speaker 2: seems to have seen anything.
- 1:22Speaker 1: A video camera was the only witness to this crime.
- 1:25Speaker 1: This kind of fuzzy, blurry surveillance video captured a still
- 1:31Speaker 1: of these perpetrators, and that was basically the only thing
- 1:35Speaker 1: that they had to go on.
- 1:36Speaker 2: Douglas McMillan is a reporter for the Washington Post, So Doug,
- 1:40Speaker 2: first off, for real, thanks for being down to talk
- 1:43Speaker 2: with me about all this. Of course, Doug doesn't usually
- 1:46Speaker 2: cover cops or the justice system. His usual beat at
- 1:49Speaker 2: the Post is corporate accountability and technology. But there was
- 1:53Speaker 2: something about this case in Saint Louis that caught us attention.
- 1:56Speaker 1: There's a lot of crimes, hundreds or thousands of solved
- 2:00Speaker 1: cases out there where the only evidence was a camera,
- 2:05Speaker 1: The only evidence was a photo of this in perpetrator.
- 2:09Speaker 1: And the potential of this technology is we have this
- 2:14Speaker 1: seemingly magical tool to help solve those crimes.
- 2:18Speaker 2: The technology he's talking about here is AI facial recognition.
- 2:22Speaker 2: Police around the country are increasingly using this on cases
- 2:25Speaker 2: that they can't find a witness for and this was
- 2:28Speaker 2: one of those cases. The investigation into who had assaulted
- 2:32Speaker 2: Michael Feldman had pretty much gone cold.
- 2:35Speaker 1: Until about eight months after the incident. We believe the
- 2:41Speaker 1: police officers they decided, oh hey, we have these grainy
- 2:45Speaker 1: surveillance images, let's try to run them through the facial
- 2:50Speaker 1: recognition system and see if that will give us any matches.
- 2:53Speaker 1: And so they did that and they got some matches back.
- 2:57Speaker 1: We believe it was between five and ten maybe of
- 3:00Speaker 1: different possible matches. And the one they chose, the one
- 3:04Speaker 1: that they thought looked the most like the perpetrator of
- 3:08Speaker 1: the crime, or one of the two perpetrators of the crime,
- 3:11Speaker 1: was this gentleman named Christopher Gatlin. And then they kind
- 3:14Speaker 1: of were off and running. In this case, they had
- 3:16Speaker 1: a suspect. Then this was the only thing that police
- 3:21Speaker 1: had to arrest this man, take away his freedom, you know,
- 3:26Speaker 1: came down to this AI program.
- 3:29Speaker 2: I mean, because the thing here that of course we're
- 3:32Speaker 2: sort of circling around, is that that's not the guy,
- 3:36Speaker 2: Christopher Gatlin's innocence.
- 3:46Speaker 3: I'm afraid.
- 3:51Speaker 2: From Kaleidoscope in iHeart Podcasts. This is kill Switch. I'm
- 3:57Speaker 2: Dexter Thomas.
- 3:58Speaker 3: I'm sorry, I'm sorry. Good bye.
- 4:40Speaker 1: So they went out investigating Christopher Gatlin and the first
- 4:45Speaker 1: thing they did was print out a picture of his mugshot,
- 4:50Speaker 1: and they took the picture of Chris Gatlin to the
- 4:54Speaker 1: apartment building where Michael Feldman lives and they conducted a
- 4:56Speaker 1: photo lineup with him, and they basically sat Michael Feldman
- 5:01Speaker 1: down and gave him the photos, put them on the
- 5:04Speaker 1: table in front of him, and instructed him to kind
- 5:06Speaker 1: of spend time looking at each one and going through
- 5:09Speaker 1: these photos, take.
- 5:12Speaker 4: A look through all of them and see if anything.
- 5:15Speaker 5: Rings a bell.
- 5:17Speaker 2: Eight months after the assault, the cops visited the victim,
- 5:20Speaker 2: Michael Feldman's house, and they brought some pictures with them.
- 5:24Speaker 2: One of those pictures was of Chris Gatlin. And we
- 5:27Speaker 2: know exactly what happened next because it's all on record.
- 5:31Speaker 1: So we got the body camera a video of this
- 5:35Speaker 1: photo lineup, and the officers who conducted this photo lineup
- 5:40Speaker 1: did not follow the proper procedures for basically getting an
- 5:44Speaker 1: affair and impartial identification of this witness. And you can
- 5:48Speaker 1: see throughout the process when Michael Feldman's going through these photos,
- 5:53Speaker 1: these police officers, who are really supposed to be just
- 5:55Speaker 1: standing off to the side and not saying anything, are
- 5:58Speaker 1: kind of coaching him and giving him kind of little
- 6:02Speaker 1: clues throughout the process on which guy they want him
- 6:06Speaker 1: to pick.
- 6:06Speaker 4: Okay, let's think about before the incident, he went up
- 6:10Speaker 4: and he talked to him a little bit right, Let's
- 6:14Speaker 4: start to focus on things before you actually got any
- 6:17Speaker 4: altercation of it, maybe even before you actually initially went.
- 6:20Speaker 3: To talk to him. Just take a minute kind.
- 6:22Speaker 4: Of ponder, think.
- 6:24Speaker 1: And at one point he actually picks up the picture
- 6:26Speaker 1: of Chris Gatlan and puts it aside, and he keeps
- 6:29Speaker 1: on looking through these photos, and at one point he says,
- 6:34Speaker 1: I want to say it's him pointing to a different guy.
- 6:38Speaker 3: I want to say, jan I remember my memory is
- 6:44Speaker 3: I want to say to him.
- 6:47Speaker 1: When he does that, one of the detectives steps forward
- 6:50Speaker 1: and says, okay. Instead of, you know, accepting that as
- 6:53Speaker 1: an identification, the detective encourages him to continue going and
- 6:57Speaker 1: continue thinking about the interaction. Think about, you know, was
- 7:01Speaker 1: a guy wearing anything, Was he wearing a hat? Did
- 7:03Speaker 1: you recognize anything about his face, his eyes, their clothing
- 7:07Speaker 1: he had.
- 7:10Speaker 3: I don't remember really.
- 7:11Speaker 1: I thought he had like a hat on or something, or.
- 7:14Speaker 4: Stocking something on. So let's picture the picture of these
- 7:17Speaker 4: two guys wearing that, you know, a stocking cap or something.
- 7:20Speaker 4: If you need to use your hands, if you've got
- 7:22Speaker 4: to put hands on the papers.
- 7:23Speaker 3: That's okay.
- 7:24Speaker 1: And so you know, he kind of like, you know,
- 7:26Speaker 1: prods Michael into thinking that he didn't pick the right
- 7:30Speaker 1: person and then going back and kind of changing his mind,
- 7:33Speaker 1: And eventually he does go back and he pulls the
- 7:35Speaker 1: picture of Chris Gatland out and he points to that
- 7:37Speaker 1: and he says it was him, kid.
- 7:42Speaker 3: All right.
- 7:43Speaker 4: I just remember he getting really angry quick.
- 7:46Speaker 2: Just the mannerism I'm just trying to and the eyes probably.
- 7:53Speaker 4: How pissed off he got.
- 7:55Speaker 3: As not to be in that.
- 7:57Speaker 1: You couldn't stand there.
- 7:59Speaker 3: I think.
- 8:01Speaker 5: Okay.
- 8:03Speaker 2: So police take that identification and they're able to get
- 8:06Speaker 2: a warrant to arrest Chris Gatlin. They have no other
- 8:09Speaker 2: evidence other than that an algorithm picked him out. He
- 8:13Speaker 2: spent seventeen months in jail. So, Doug, I've read about
- 8:18Speaker 2: Chris Gatland's case, but you've met him in person. What's
- 8:21Speaker 2: he like.
- 8:22Speaker 1: He's a very sweet, charming guy, father of four kids.
- 8:26Speaker 1: He did not have a criminal record, so the reason
- 8:29Speaker 1: that he was in this system. So the fish recognition
- 8:35Speaker 1: database in Saint Louis has over two hundred and fifty
- 8:38Speaker 1: thousand mug shots of people, including people who were pulled
- 8:44Speaker 1: over for traffic stops and arrested for speeding or arrested
- 8:48Speaker 1: for very minor things. Saint Louis has a long history
- 8:52Speaker 1: of over policing, especially black minority, low income populations, handing
- 8:57Speaker 1: out tickets to people just so that police can meet
- 9:00Speaker 1: their quotas. So Chris, he had had a few different
- 9:04Speaker 1: traffic infractions, and he had one burglary charge from a
- 9:08Speaker 1: few years earlier. It was ultimately dropped right prosecutors. But
- 9:12Speaker 1: that's the reason that he came out, and that's the
- 9:13Speaker 1: reason he was in this system.
- 9:15Speaker 2: It's incredible that you could be in a system that
- 9:19Speaker 2: would lead you to get arrested based off of a
- 9:21Speaker 2: traffic ticket. I have a completely clean record, with the
- 9:25Speaker 2: one exception that a literal decade ago, I was speeding
- 9:30Speaker 2: to work on the freeway. I was late for work.
- 9:32Speaker 2: I got pulled over. Now, I didn't get arrested, but
- 9:35Speaker 2: had somebody decide to go a little harder on me? Sure,
- 9:37Speaker 2: I suppose they could have. And I'm just imagining myself
- 9:42Speaker 2: being at home one day and police coming to my
- 9:46Speaker 2: door and saying, Hey, we're arresting you because we think
- 9:51Speaker 2: you did something that I've never heard of in a
- 9:53Speaker 2: place that I don't.
- 9:54Speaker 1: Go, because you were in this database.
- 9:56Speaker 2: Yeah, because I was five miles over the speed limit
- 9:58Speaker 2: one day going for work. Yeah, so maybe this is
- 10:03Speaker 2: a good time to say that all facial recognition technology
- 10:06Speaker 2: isn't the same, or maybe more importantly, that it all
- 10:10Speaker 2: doesn't come from the same place. The program that the
- 10:13Speaker 2: Saint Louis Cops use searches the internal databases of the
- 10:16Speaker 2: police department. But there's another very popular tool. It's able
- 10:20Speaker 2: to pull data straight from the Internet. It's called clearview AI.
- 10:25Speaker 1: Clearview scrapes together images from Facebook, from LinkedIn, from Venmo accounts,
- 10:33Speaker 1: from news articles, from all of the public web sources
- 10:37Speaker 1: that it can find images with people's face.
- 10:39Speaker 2: Clearview is wild, and they've got billions of photos in there.
- 10:44Speaker 1: Yeah, they say they have billions of images. And there's
- 10:47Speaker 1: a question about whether it's legal for them to scrape
- 10:50Speaker 1: these images and put them in a database, because they
- 10:52Speaker 1: do not have they never really got permission from anybody
- 10:56Speaker 1: to get these images.
- 10:57Speaker 2: Clearview AI's website boats that they've scraped over fifty billion images.
- 11:02Speaker 2: And again this is just from the general Internet. So
- 11:05Speaker 2: when the police run a search, you could be in there,
- 11:08Speaker 2: even if you've never had a run in with law enforcement.
- 11:12Speaker 2: Do we even know how many police departments are using this.
- 11:16Speaker 1: No, it's very hard to tell because they're not required
- 11:20Speaker 1: to disclose this anyway. In most places, they're not required
- 11:23Speaker 1: to disclose that they're using these tools. So the best
- 11:26Speaker 1: proxy for that number is what the companies who make
- 11:31Speaker 1: this software have announced. Clearview AI has said that it
- 11:35Speaker 1: has thousands. I think they have said over three thousand
- 11:38Speaker 1: police customers. I did a pretty exhaustive public record search
- 11:44Speaker 1: for police departments that are using this tools. We only
- 11:47Speaker 1: came up with a list of about seventy that we
- 11:50Speaker 1: could verify that are using it in some way, and
- 11:54Speaker 1: of those, we sought public records on individual cases, and
- 11:58Speaker 1: we could only get individual case on something like thirty
- 12:02Speaker 1: or forty of them. Getting to the bottom of you know,
- 12:05Speaker 1: which police departments are using this, how they're using it
- 12:07Speaker 1: is very difficult because for the most part, there's no
- 12:10Speaker 1: requirement for them to make public how and when they're
- 12:13Speaker 1: using the tools, and that is allowing police to mostly
- 12:18Speaker 1: use these tools in secrets.
- 12:20Speaker 2: But sometimes it's not the machines making the mistakes, it's
- 12:24Speaker 2: the cops. More on that after the break, it's not
- 12:40Speaker 2: too hard to understand why a police department would be
- 12:42Speaker 2: interested in facial recognition tools. It seems pretty logical. I mean,
- 12:46Speaker 2: you're having a hard time crack in a case and
- 12:48Speaker 2: you either don't have any eyewitnesses, or you can't rely
- 12:51Speaker 2: on the witnesses you have, or maybe, like with Michael Feldman,
- 12:54Speaker 2: the victims suffered an injury that affected their memory of
- 12:57Speaker 2: the event, or maybe too much time has passed for
- 13:00Speaker 2: somebody to really remember the details. Eyewitness accounts are full
- 13:04Speaker 2: of problems for law enforcement because human memory is unreliable
- 13:08Speaker 2: and it can be influenced by people's biases. So there's
- 13:11Speaker 2: this potential magical solution. You in put a face into
- 13:15Speaker 2: the system, It searches a database and spits out a match.
- 13:18Speaker 2: All you gotta do is go find that person and boom.
- 13:21Speaker 2: Case closed.
- 13:23Speaker 1: So we know that there are at least eight Americans
- 13:27Speaker 1: who have been wrongly arrested due to police misuse of
- 13:32Speaker 1: facial recognition.
- 13:34Speaker 2: And just to be clear here, when he says wrongly arrested,
- 13:37Speaker 2: that's not just his opinion. What he means is that
- 13:40Speaker 2: the police admitted afterwards that they got the wrong person.
- 13:43Speaker 1: Interestingly, one of those cases this man named Jason Vernaw
- 13:47Speaker 1: who is based in Miami. In that case, the computer
- 13:50Speaker 1: got the right person, but the police had fed the
- 13:54Speaker 1: computer an image of the wrong person. So they fed
- 13:57Speaker 1: it an image who they thought this was the perpetrator
- 14:00Speaker 1: of somebody in a bank committing fraud. But it turns
- 14:02Speaker 1: out they were mistaken and the surveillance image that they
- 14:06Speaker 1: put in the system was just the wrong person to
- 14:08Speaker 1: begin with.
- 14:09Speaker 2: So that was the one time they got it right.
- 14:11Speaker 1: Well, yeah, so the computer actually picked the right person,
- 14:15Speaker 1: but the police were relying on this image they got
- 14:18Speaker 1: from the bank and they just pulled an image from
- 14:20Speaker 1: the wrong time.
- 14:22Speaker 2: So the bank pulled an image from their security cameras
- 14:25Speaker 2: and sent it to the police, and this image was
- 14:28Speaker 2: Jason Vernow, but it was from a completely different time
- 14:31Speaker 2: of the day. The facial recognition software had correctly identified Jason,
- 14:36Speaker 2: but the police failed to make some obvious checks.
- 14:40Speaker 1: What this case shows was that the police put the
- 14:44Speaker 1: image in their system and it popped up Jason Vernow
- 14:47Speaker 1: and they went out and arrested the guy, rather than
- 14:50Speaker 1: taking a second and saying, okay, is we have a suspect.
- 14:53Speaker 1: Let's see if, like, can we prove that Jason ver
- 14:56Speaker 1: Now was tied to this crime. And in the case
- 14:58Speaker 1: of a financial there is a lot of potential evidence there.
- 15:02Speaker 1: There's a check that he supposedly cashed, a faulty check. Well,
- 15:06Speaker 1: this check of his signature matches. There are witnesses, there's
- 15:09Speaker 1: a bank teller who took let's see if the bank
- 15:11Speaker 1: teller can confirm that was him. They didn't do any
- 15:13Speaker 1: of that. They apparently didn't do any real investigative steps.
- 15:17Speaker 1: They just took the word of the machine and went
- 15:19Speaker 1: not arrested him. It speaks to kind of this phenomenon
- 15:24Speaker 1: that seems to be happening with police as they are
- 15:27Speaker 1: trusting this kind of they're kind of imbuing this software
- 15:32Speaker 1: with this magical ability to lead them to the right suspect.
- 15:36Speaker 1: Some researchers call this automation bias, that when a computer
- 15:40Speaker 1: is telling you an answer, you're more likely to believe
- 15:44Speaker 1: that that is the correct answer, even at the expense
- 15:47Speaker 1: of your kind of typical due diligence, your typical you know,
- 15:51Speaker 1: just rational brain and just normal steps that you should
- 15:54Speaker 1: think through and follow before you go out and arrest
- 15:57Speaker 1: somebody take away somebody's freedom.
- 16:00Speaker 2: Automation bias is just one kind of bias here, though,
- 16:04Speaker 2: I mean, there was something unique about this particular person though, right,
- 16:07Speaker 2: compared to the other cases. Yeah, he was white, and
- 16:11Speaker 2: that's the one that they got the computer got right. Yeah.
- 16:14Speaker 3: Yeah.
- 16:15Speaker 2: The facial recognition has been shown to often get certain
- 16:23Speaker 2: people wrong. Right.
- 16:24Speaker 1: Yeah. There's two things going on here. One, facial recognition
- 16:27Speaker 1: has struggled with darker complexions, and part of this is
- 16:32Speaker 1: due to how these programs are trained. So when the
- 16:36Speaker 1: first face rerecation algorithms were trained, they were trained on
- 16:39Speaker 1: lighter skin tones. There's a federal research lab it's called
- 16:44Speaker 1: NIST NIST that does testing into how well facial recognition
- 16:49Speaker 1: performs in laboratory settings, and they found in twenty nineteen
- 16:53Speaker 1: that certain demographic groups, including African Americans, and certain facial
- 16:59Speaker 1: rection algorithms could be up to one hundred times more
- 17:02Speaker 1: likely to be mismatched than lighter skin tones. That was
- 17:05Speaker 1: about six years ago, and the industry says, and clear
- 17:09Speaker 1: View in other companies says that their algorithms have improved
- 17:12Speaker 1: dramatically over that time, and there's some evidence that they
- 17:15Speaker 1: have gotten better. However, we ultimately don't know how accurate
- 17:20Speaker 1: and how reliable this software is in the way that
- 17:23Speaker 1: law enforcement use it because it's never been tested in
- 17:27Speaker 1: the way that law enforcement use it, the way that
- 17:29Speaker 1: that federal lab tests things. It's looking at basically perfect lighting,
- 17:34Speaker 1: a perfectly framed profile photo in perfect conditions. How does
- 17:38Speaker 1: that algorithm do in those settings? Unlike how police use
- 17:42Speaker 1: these tools, which is, we're going to use this grainy
- 17:45Speaker 1: surveillance image shot at a weird angle from above, usually
- 17:50Speaker 1: usually if very poor lighting, and usually the face is
- 17:53Speaker 1: partly obscure or it could be. In the case in
- 17:56Speaker 1: Saint Louis, the guy was actually wearing a COVID mask
- 17:58Speaker 1: that was covering part of his and he was wearing
- 18:00Speaker 1: a hood that covered another part of his face. So
- 18:03Speaker 1: this is the typical setting where police are using these tools.
- 18:06Speaker 2: There's a case where a cartoon came up in a
- 18:11Speaker 2: match at one point, right.
- 18:13Speaker 1: Yeah, So because clear VII just has these I guess
- 18:16Speaker 1: web crawlers that are just scraping the Internet, and so
- 18:19Speaker 1: the things that come up and the Clearview results sometimes
- 18:23Speaker 1: are bizarre. So in this one case in Ohio, two
- 18:26Speaker 1: of the search results that I think we're in the
- 18:28Speaker 1: top ten of the search results were Michael Jordan just
- 18:32Speaker 1: a picture of Michael Jordan and I by the way,
- 18:35Speaker 1: I reached out to Michael Jordan's rep when I published
- 18:38Speaker 1: this in a story, and she didn't have any comment
- 18:41Speaker 1: on whether he might have been implicated into crime in Ohio.
- 18:45Speaker 1: And then the other one was yeah, just like it
- 18:48Speaker 1: illustrated a cartoon image of a black man. So it
- 18:51Speaker 1: just makes you like, kind of makes it you scratch
- 18:54Speaker 1: your head as to you know, what is this tool
- 18:56Speaker 1: that police are relying on if it's feeding them, you know,
- 18:59Speaker 1: garbage as results.
- 19:01Speaker 2: And giving them Michael Jordan committed this crime in the
- 19:04Speaker 2: top ten.
- 19:04Speaker 1: Right, which right, Yeah.
- 19:06Speaker 2: Again a big allegedly here, but I feel pretty sure
- 19:09Speaker 2: they'd have a tough time connecting him to the scene
- 19:12Speaker 2: of whatever that that particular crime was.
- 19:15Speaker 1: Yeah, And if this is such an advanced algorithm, why
- 19:18Speaker 1: can't their computer figure out that's Michael Jordan and just
- 19:20Speaker 1: take him out? And why can't a computer figure out
- 19:23Speaker 1: that that's a cartoon black man and not a real person.
- 19:26Speaker 1: I just ca't like a lot of the people who
- 19:28Speaker 1: are who put forward facial recognition as this science, they
- 19:34Speaker 1: have sort of cast it in the kind of the
- 19:36Speaker 1: cloak of this is a scientific tool. The police are
- 19:40Speaker 1: now using a lot of those arguments kind of break
- 19:42Speaker 1: down when you see stuff like this, when you see, oh, well,
- 19:45Speaker 1: your scientific, highly advanced tool, you know, brought back a
- 19:49Speaker 1: picture of you know, cartoon black Man. How do the
- 19:51Speaker 1: science arrive at that?
- 19:56Speaker 2: Facial recognition as a technology is probably a whole other episod.
- 20:00Speaker 2: But if you've ever used face ID to unlock your iPhone,
- 20:03Speaker 2: you've already used the version of facial recognition on yourself.
- 20:07Speaker 2: In some ways, the technology isn't all that different from
- 20:09Speaker 2: how it started in the nineteen sixties, when researchers were
- 20:12Speaker 2: trying to figure out how to identify facial features like
- 20:15Speaker 2: the bridge of the nose, the edge of the nose,
- 20:17Speaker 2: or the eyes and measure the distance between them. They'd
- 20:21Speaker 2: store all those measurements as data for a single face.
- 20:25Speaker 2: Back in the sixties, this stuff was pretty rudimentary, but
- 20:28Speaker 2: AI's made it more accurate because you can train the
- 20:31Speaker 2: systems on more faces, which means theoretically it's more likely
- 20:35Speaker 2: to tell your face from someone else's. But you've probably
- 20:38Speaker 2: had your face ID fail on you before, and that's
- 20:41Speaker 2: under good conditions. It's just trying to match your face
- 20:44Speaker 2: with well your face. If you add a few million
- 20:48Speaker 2: other faces into the mix, the potential for mistakes multiplies
- 20:52Speaker 2: by a lot. But even with those issues, police are
- 20:56Speaker 2: increasingly leaning on this technology. It seems like police are
- 21:01Speaker 2: presenting the fact that facial recognition software has returned a match,
- 21:07Speaker 2: whether it's true or false, as evidence. But that seems
- 21:12Speaker 2: it seems backwards, right.
- 21:13Speaker 1: Yeah, Well, not only that, but it's on the one hand,
- 21:17Speaker 1: they actually acknowledge that facial recognition is not enough to
- 21:21Speaker 1: make a case. They say this in their police rules
- 21:25Speaker 1: and their public statements over and over and over. If
- 21:28Speaker 1: you hear police talk about facial recognition, they say to
- 21:30Speaker 1: the public, this is only an investigative tool. We are
- 21:34Speaker 1: only going to use this to find investigative leads that
- 21:36Speaker 1: we then go out and corroborate. And in some places
- 21:39Speaker 1: that's actually the law. There's six states where actually police
- 21:42Speaker 1: are required to corroborate any leads that they given facial recgnition.
- 21:46Speaker 1: What we found in looking at cases all around the
- 21:48Speaker 1: country was that the police were not doing that. Sometimes
- 21:51Speaker 1: they do, but often they will just take it at
- 21:54Speaker 1: face value that this software hit is enough, and then
- 21:58Speaker 1: they will use that to go out and arrest the
- 21:59Speaker 1: per I encountered this really surprising thing, which is I
- 22:03Speaker 1: talked to a number of prosecutors and police who told
- 22:07Speaker 1: me that, well, yeah, we did corroboration. As soon as
- 22:10Speaker 1: we got the name from the software, we went and
- 22:13Speaker 1: we looked at that person's other photos and it visually
- 22:16Speaker 1: looked like the same person. And they said that that
- 22:19Speaker 1: is corroboration, visual corroboration of a match.
- 22:22Speaker 2: That sounds like you're corroborating evidence on your own, like
- 22:27Speaker 2: you're becoming a witness at that point.
- 22:28Speaker 1: As a police officer, this thing comes up over and
- 22:31Speaker 1: over is that many people look alike, and humans are
- 22:36Speaker 1: very bad at distinguishing between two people who are similar.
- 22:42Speaker 2: You would think that this is where technology could help,
- 22:46Speaker 2: but AI isn't much better at telling people apart. The
- 22:49Speaker 2: first known case of a wrongful arrest made by AI
- 22:52Speaker 2: was Robert Williams in Detroit in twenty twenty. Doug was
- 22:56Speaker 2: able to get the police interrogation video. William's case is
- 23:02Speaker 2: really interesting because we have the interrogation video of this
- 23:05Speaker 2: and I was just watching this and it's incredible. So
- 23:10Speaker 2: this is you I know you've seen this video. So
- 23:12Speaker 2: this is where they're you know, these two detectives are
- 23:16Speaker 2: sitting in this interrogation room. Rob Williams is sitting across
- 23:20Speaker 2: the table from them, and he's clearly confused. And they
- 23:23Speaker 2: bring out these papers and I'll play it from here.
- 23:29Speaker 5: December twenty sixth, around one pm? Where were you.
- 23:37Speaker 3: Home? Home?
- 23:39Speaker 5: Can you do me a favorite? Is that you?
- 23:43Speaker 3: You know? No?
- 23:45Speaker 5: Not even closed it? No, I'm pissed.
- 23:53Speaker 3: Keep going.
- 23:55Speaker 5: Is that you? No, that's not you at all? Not
- 24:01Speaker 5: there either. You can't y'all can't tail that. I'm one hundred,
- 24:15Speaker 5: so I'm one of the pictures. We actually got facial recognition.
- 24:20Speaker 5: I heard that it was probably favorite recognition.
- 24:23Speaker 3: That is not me.
- 24:24Speaker 2: That's me on my d right, you're you're smiling and
- 24:28Speaker 2: I'm smiling too. As bleak as this is. But there's
- 24:33Speaker 2: something almost funny in this exchange.
- 24:37Speaker 1: Well, they because they present the pictures as if it's
- 24:40Speaker 1: evidence in their favor, and then as soon as he
- 24:43Speaker 1: sees them and reacts to them, suddenly it becomes evidence
- 24:48Speaker 1: in his favor because he's like, are you looking at
- 24:50Speaker 1: the same picture I'm looking at because it's clearly not me?
- 24:53Speaker 1: And it's funny. You can kind of hear in the
- 24:54Speaker 1: officer's voice, well, well, it's we have facial recognition, as
- 25:00Speaker 1: if you know, as if that's somehow, you know, bolsters
- 25:02Speaker 1: his case that this is this is accurate, as if
- 25:05Speaker 1: that's going to change his mind that the picture that
- 25:08Speaker 1: he's looking at is actually him when he knows it's
- 25:10Speaker 1: not him, As if the phrase facial recognition is itself
- 25:15Speaker 1: kind of evidence of the police being correct.
- 25:19Speaker 2: You know, he holds a picture up to his face, like, look,
- 25:23Speaker 2: of course, this isn't me. What are you talking about, Kate?
- 25:26Speaker 2: Forget the software, look at me. The thing that hits
- 25:29Speaker 2: you about this interaction is that when the cop finally
- 25:32Speaker 2: looks at the picture, you could hear the room go
- 25:34Speaker 2: quiet for a second, and nobody's arguing with Robert Williams,
- 25:38Speaker 2: and it kind of sounds like he's agreeing, like it's
- 25:41Speaker 2: the first time he's actually looking at it, like he
- 25:44Speaker 2: had so much confidence in the machine that he didn't
- 25:47Speaker 2: bother to look with his own eyes.
- 25:49Speaker 1: They by and large are not telling even the people
- 25:52Speaker 1: that they identify and arrest using his tools that they
- 25:55Speaker 1: use them, and that's leading to people either finding out
- 26:01Speaker 1: kind of in offhand ways in interrogation that oh, well,
- 26:05Speaker 1: you know the computer picked you. Well, what do you
- 26:07Speaker 1: mean by the computer. A lot of times, they're probably
- 26:09Speaker 1: not finding out at all, which is concerning because one
- 26:12Speaker 1: of the pillars of our courts and our justice system
- 26:16Speaker 1: is that you need to be able to face your accuser.
- 26:20Speaker 1: And so if your accuser is this algorithm, this computer program,
- 26:25Speaker 1: but you're not even being told that it was used,
- 26:28Speaker 1: let alone given any of the details about how it works.
- 26:31Speaker 1: So all of these things are being kind of kept
- 26:34Speaker 1: vague or sometimes completely kept hidden from the people that
- 26:37Speaker 1: these tools are being used to investigate and ultimately arrest.
- 26:41Speaker 2: So what does this mean for the future of policing
- 26:44Speaker 2: and how should we be using these technologies if at all?
- 26:47Speaker 2: That's after the break. People are being investigated and arrested
- 27:07Speaker 2: and not being told that police are using these facial
- 27:10Speaker 2: recognition tools to find them. You don't need a law
- 27:13Speaker 2: agree to feel that something about that is just kind
- 27:16Speaker 2: of off, and you wouldn't be wrong. Here in the US,
- 27:19Speaker 2: there's a specific set of rules that might apply here,
- 27:25Speaker 2: the Brady rules. I want to talk about that. So
- 27:29Speaker 2: can we talk about how Brady rules can can potentially
- 27:33Speaker 2: figure into an arrest that is made with AI facial
- 27:38Speaker 2: recognition as a part of that or is the sole
- 27:40Speaker 2: part of it?
- 27:41Speaker 1: Yeah, So basically, when you prosecute somebody, the prosecution is
- 27:47Speaker 1: required by the courts and required by the Constitution to
- 27:52Speaker 1: share any evidence that speaks to the guilt or the
- 27:56Speaker 1: innocence of the person that they're prosecuting, so whired to
- 28:00Speaker 1: share that with the defense. There's a question now the
- 28:03Speaker 1: courts are grappling with, which is does somebody's identification by
- 28:08Speaker 1: an AI software? Should that be brought into the courtroom?
- 28:11Speaker 1: And should the should the defendant be given a chance
- 28:15Speaker 1: to know everything about that software? And right now this
- 28:19Speaker 1: is playing out in courtrooms across the country.
- 28:21Speaker 2: Basically putting the AI in a manner of speaking, putting
- 28:25Speaker 2: the AI on the witness stand and saying how reliable
- 28:29Speaker 2: really are you? Yeah.
- 28:31Speaker 1: A very good indicator of how that's going to go
- 28:34Speaker 1: is the companies themselves have disclaimers saying this does not
- 28:40Speaker 1: hold up in courts. Clearview AI is the one that
- 28:43Speaker 1: I'm very familiar with. They have language and all of
- 28:46Speaker 1: their contracts with police departments saying this is not admissible
- 28:49Speaker 1: evidence in court. And also basically, please don't ask us
- 28:53Speaker 1: to come into court, because nobody from Clearview AI is
- 28:55Speaker 1: going to come and sit in court and defend this software.
- 28:58Speaker 1: The very companies who make it and advertise it and
- 29:00Speaker 1: market it as this great, amazing technology tool, they would
- 29:04Speaker 1: not even stand behind it.
- 29:05Speaker 2: In your research, in your reporting, you've seen what looks
- 29:09Speaker 2: like and what defendants are saying wrongful arrest based on AI.
- 29:13Speaker 2: Have the police who made these arrests or who made
- 29:18Speaker 2: those decisions. Has there been any repercussions, any discipline, any
- 29:22Speaker 2: guidance for them that you've seen.
- 29:27Speaker 1: I mean, the answer is no, as far as I
- 29:29Speaker 1: can tell, the individual officers have not publicly faced any punishment,
- 29:35Speaker 1: although you know, you never know what happens behind closed doors.
- 29:39Speaker 1: Typically police unions forbid police from publicly talking about punishment
- 29:43Speaker 1: of individuals. Now, we've seen settlements paid out of three
- 29:47Speaker 1: hundred thousand dollars to a few of these people who
- 29:49Speaker 1: are wrong fully arrested. Is this going to change the
- 29:51Speaker 1: behavior of these individuals. There's only six states that have
- 29:55Speaker 1: laws mandating specific disclosures and mandate specific things about how
- 30:01Speaker 1: these tools cannot be used. So you know, the vast
- 30:04Speaker 1: majority of the country police are kind of still just
- 30:07Speaker 1: kind of figuring out. It's really early days.
- 30:10Speaker 2: So just to back up here, we've been talking about
- 30:12Speaker 2: the problems with this technology, what about the upsides? And
- 30:17Speaker 2: it's interesting because if you ask people how facial recognition
- 30:20Speaker 2: could be used to catch criminals, there's not really any
- 30:23Speaker 2: high profile cases we could point to. And before you
- 30:27Speaker 2: say January sixth, well that's complicated.
- 30:31Speaker 1: A lot of people attribute facial recognition with helping to
- 30:34Speaker 1: I identify these people and to bring them to justice.
- 30:39Speaker 1: And yes, facial recond did play a role. However, on
- 30:43Speaker 1: that day, there was a lot of evidence being collected
- 30:45Speaker 1: through social media posts that these people were making on
- 30:48Speaker 1: their own videos that these people were taking and posting.
- 30:51Speaker 1: I think the federal investigators even use of high tech
- 30:54Speaker 1: methods to grab the cellular location data of everybody who
- 30:59Speaker 1: is in the capital that day, and I think they
- 31:00Speaker 1: were able to identify a lot of people that way.
- 31:02Speaker 1: So facial actation in that case was one of many
- 31:05Speaker 1: tools that were used, which is sort of how it's
- 31:08Speaker 1: designed to be used, is you know, not as kind
- 31:10Speaker 1: of the sole investigative lead, but as one of several. So,
- 31:14Speaker 1: you know, other than that, there aren't that many well
- 31:17Speaker 1: known cases out there that we can point to of
- 31:20Speaker 1: you know, this was the thing that brought down, you know,
- 31:23Speaker 1: a murder case that we've been trying to solve for
- 31:25Speaker 1: many years. But I bet we will start to hear
- 31:27Speaker 1: about some of that, and I would love to kind
- 31:29Speaker 1: of have more of those case studies we can pick apart.
- 31:32Speaker 1: I think the public deserves to know about both. The
- 31:34Speaker 1: public should also be aware of the good and the
- 31:37Speaker 1: benefit because as we kind of make laws or think
- 31:41Speaker 1: about regulating how to ring this in, it's going to
- 31:44Speaker 1: be important to understand that balance and to get it right.
- 31:48Speaker 1: I think that there's a danger and rushing to a
- 31:52Speaker 1: blanket ban on this technology.
- 31:56Speaker 2: I want to play some for you really quick. This
- 31:59Speaker 2: is a really I think powerful moment from your podcast,
- 32:03Speaker 2: and I was hoping I could have have you speak
- 32:05Speaker 2: to this a little bit here.
- 32:07Speaker 5: They lean on the technology because I think we are
- 32:11Speaker 5: taught that computers don't make mistakes.
- 32:13Speaker 4: Humans do.
- 32:15Speaker 2: Now I'm sitting there like, how can you do this?
- 32:18Speaker 3: You can't do this?
- 32:19Speaker 1: Like, no way, how can you just put these charges
- 32:22Speaker 1: on them?
- 32:22Speaker 3: And I'm telling you that that's not me.
- 32:25Speaker 2: I know I was in this, So how do I
- 32:27Speaker 2: beat a machine?
- 32:29Speaker 3: So?
- 32:30Speaker 2: I mean, I guess I guess my question is here
- 32:34Speaker 2: these I'm just gonna call them AI arrests. How do
- 32:36Speaker 2: these arrests affect the people who are being accused.
- 32:41Speaker 1: That was a clip of our episode of Post Reports,
- 32:44Speaker 1: and we heard these patterns of experience which were very notable.
- 32:49Speaker 1: So a few things. Wrongful arrests are not a new thing.
- 32:52Speaker 1: The technology did not bring about wrongful arrests, but the
- 32:55Speaker 1: idea of you know, usually there's some other incidental relationship
- 32:59Speaker 1: some somebody has to a crime before they get into
- 33:03Speaker 1: a wrongful orest. In these cases, the AI is plucking
- 33:06Speaker 1: you out of thin air in some cases. And in
- 33:09Speaker 1: one case, this man, Karan Reid, he had never actually
- 33:11Speaker 1: even been to the state of a Louisiana and he
- 33:13Speaker 1: was resident in Atlanta, so he's literally been plucked from
- 33:17Speaker 1: the other side of the country and brought into this
- 33:20Speaker 1: crime that he had literally no proximity to whatsoever. So
- 33:24Speaker 1: this what does that do to the human brain when
- 33:26Speaker 1: you are just suddenly poof inside of a criminal investigation
- 33:30Speaker 1: that you have no connection to whatsoever. It's like baseline
- 33:34Speaker 1: is a word that came up over and over. A
- 33:35Speaker 1: lot of people just kind of got stuck on that idea,
- 33:38Speaker 1: even when they sat in jail for one or two
- 33:40Speaker 1: or three weeks or months in the case of Chris Gatlin,
- 33:44Speaker 1: who was in jail for seventeen months. Some of them
- 33:47Speaker 1: had this feeling of and I think you just heard
- 33:49Speaker 1: it in the clip. If the computer is telling you
- 33:52Speaker 1: that I did it, then how am I going to
- 33:54Speaker 1: convince you? Otherwise? How am I going to beat a computer?
- 33:57Speaker 1: Because you know, it goes back to this thing that
- 33:59Speaker 1: we were talking about before, this automation bias of you know,
- 34:04Speaker 1: he never heard of that term, but he instinctively realized
- 34:09Speaker 1: that he was going against a power that was much
- 34:12Speaker 1: greater than him and in some ways probably much greater
- 34:14Speaker 1: than if a witness had picked you for a crime.
- 34:17Speaker 1: It's like it's a computer picked you for a crime
- 34:20Speaker 1: and put your name on the line and put your
- 34:23Speaker 1: name into this investigation.
- 34:24Speaker 2: So it's our belief in the computer that's what you're
- 34:27Speaker 2: up against.
- 34:28Speaker 1: You know, many of the people that we talked to
- 34:30Speaker 1: and the cases that are public, many of them feel
- 34:32Speaker 1: like they kind of were the lucky ones. And maybe
- 34:36Speaker 1: they got out because one of them had a mole
- 34:39Speaker 1: on his face and his lawyer discovered that the guy
- 34:42Speaker 1: and the surveillance image didn't have a mole. Or there
- 34:46Speaker 1: was one woman, Portia Woodriff who was eight months pregnant
- 34:49Speaker 1: at the time she has arrested, and you know, there
- 34:51Speaker 1: was nothing in the interviews with the victims that said
- 34:54Speaker 1: there's a pregnant woman involved. So that's ultimately one of
- 34:56Speaker 1: the reasons that she got out. But are there many
- 34:59Speaker 1: other people who didn't get lucky and might might be
- 35:02Speaker 1: behind bars still because of facial recognition, you know, wrongfully
- 35:05Speaker 1: put them there.
- 35:06Speaker 2: I mean, when you put it like that, you start
- 35:08Speaker 2: to make me think that maybe we're not hearing horror stories.
- 35:12Speaker 2: Those are the success stories.
- 35:14Speaker 1: This was pretty horrible for these people, not just them
- 35:18Speaker 1: in some of the cases. You know, their their children,
- 35:20Speaker 1: their young children watch them get arrested in their front lawns,
- 35:24Speaker 1: and that's trauma.
- 35:25Speaker 3: Yeah.
- 35:26Speaker 1: One of the one of the men that we've talked
- 35:27Speaker 1: about before, Robert Williams, who's arrested in Detroit. You know,
- 35:29Speaker 1: his kids watched them get arrested in the front lawn
- 35:32Speaker 1: and he says, to this day, they still talk about
- 35:36Speaker 1: that incident. One one of his daughters. This is kind
- 35:39Speaker 1: of heartbreaking. One of his daughters every once in a
- 35:41Speaker 1: while comes up to him and says, Daddy, I think
- 35:44Speaker 1: I've solved it. I found the real man who went
- 35:47Speaker 1: and stole those watches, and she pointed to like a
- 35:49Speaker 1: cartoon character as like that was the one that was
- 35:52Speaker 1: the real guy who stole the watches and not daddy.
- 35:54Speaker 2: Oh my god.
- 35:55Speaker 1: So these, I mean, these experiences are now baked into
- 35:57Speaker 1: their lives and their childhoods. And then that's you know,
- 36:00Speaker 1: I'm a parent of small kids, and that's a very
- 36:02Speaker 1: sad thing. And yes, they were lucky, probably luckier than
- 36:06Speaker 1: other people whose stories we don't know about, but also
- 36:09Speaker 1: these are horrible experiences.
- 36:14Speaker 2: I'm not sure what y'all think about this. I'm still
- 36:16Speaker 2: trying to figure that out myself, and I know a
- 36:19Speaker 2: lot of what we've heard sounds pretty terrifying. So for
- 36:22Speaker 2: one of my last questions, I asked Doug what he
- 36:25Speaker 2: thought the next few years were going to look like
- 36:27Speaker 2: with this technology, and he answered in a way that
- 36:30Speaker 2: I didn't really expect.
- 36:33Speaker 1: We're still at a point of fascination with this technology,
- 36:38Speaker 1: and in places around the country, maybe not in the
- 36:42Speaker 1: big cities, where I mean you do see kind of
- 36:44Speaker 1: a skepticism of tech tools and concerned about privacy and surveillance,
- 36:50Speaker 1: but in many of the places in this country, you know,
- 36:53Speaker 1: in smaller towns around the country, police are adopting these
- 36:56Speaker 1: things and are excited and there's a fascination, and they're
- 36:59Speaker 1: being sold to the public as this kind of magical tool,
- 37:03Speaker 1: like in Evansville, Indiana, in Florence, Kentucky, in Saint John's County, Florida,
- 37:08Speaker 1: these smaller town places where you know, the local population
- 37:12Speaker 1: may not know that the police have started to like
- 37:15Speaker 1: rely on these tools quite a bit. So a simple
- 37:19Speaker 1: goal for my reporting has just been to just kind
- 37:21Speaker 1: of encourage people to ask questions to your go to
- 37:24Speaker 1: your city council meeting. You'll actually learn a lot. You
- 37:27Speaker 1: don't have to sit on the sidelines of this discussion
- 37:28Speaker 1: because you're part of it. You are a person that
- 37:31Speaker 1: is in front of these surveillance cameras too. So now
- 37:34Speaker 1: there's a risk of you, everybody in this country of
- 37:37Speaker 1: getting pulled into an investigation due to, you know, a
- 37:40Speaker 1: misuse of this technology. So you should everybody should think
- 37:43Speaker 1: about themselves as being part of this discussion about whether
- 37:46Speaker 1: and how we use this technology going forward.
- 38:00Speaker 2: Thank you so much for listening to our first episode
- 38:03Speaker 2: of The New kill Switch, and a very special thanks
- 38:07Speaker 2: to our friends over the Washington Post. Podcast Post reports
- 38:10Speaker 2: who helped us with this episode. You can check out
- 38:12Speaker 2: more Doug's reporting over there and stay with us. We
- 38:15Speaker 2: have so much more ahead. This show is all about
- 38:18Speaker 2: what's happening right now between humans, between machines, how we're
- 38:22Speaker 2: being shaped by our own creations, and a whole lot
- 38:24Speaker 2: beyond that. So let us know what you think, and
- 38:27Speaker 2: even if there's something you want us to cover, you
- 38:29Speaker 2: can hit us up at kill Switch at Kaleidoscope dot NYC,
- 38:33Speaker 2: or you could hit me at dex Digi on the
- 38:35Speaker 2: Gram or Blue Sky if that's more your thing. Kill
- 38:39Speaker 2: Switch is hosted by me Dexter Thomas. It's produced by
- 38:42Speaker 2: Shena Ozaki, Daryl Potts, and Kate Osbourne. Our theme song
- 38:47Speaker 2: is by Kyle Murdoch, who also mixed the show. From Kaleidoscope.
- 38:50Speaker 2: Our executive producers are Ozwa Washin, Mangesh Hutti Goodur, and
- 38:55Speaker 2: Kate Osborne from iHeart, our executive producers are Katrina Norvil
- 38:59Speaker 2: and Nick Etur. That's it for me, catch on the
- 39:03Speaker 2: next one. H