Latest / Statecraft / How to Fix Risk Assessment in Child Welfare
Transcript
- 0:04SPEAKER_00: Hi, I'm Santi Ruiz.
- 0:05SPEAKER_00: You're listening to Statecraft.
- 0:07SPEAKER_00: And today we've got a very special guest on,
- 0:09SPEAKER_00: Alex Jutca,
- 0:11SPEAKER_00: Deputy Director of the Allegheny County Department of Human Services.
- 0:14SPEAKER_00: Before you, tune out because that's a bunch of bureaucratic jargon.
- 0:19Don't do that because the Allegheny County Department of Human Services is,
- 0:23SPEAKER_00: to my eye,
- 0:23SPEAKER_00: one of the most interesting state level departments out there.
- 0:27SPEAKER_00: Alex, thank you for coming on.
- 0:30SPEAKER_01: Do you want to take that again?
- 0:31SPEAKER_01: It's a county level department.
- 0:33county level department yeah yeah here we go perfect stay local stay and
- 0:39localyou
- 0:41can say one of those interesting interesting government entities no i'm
- 0:45justi'm
- 0:45just period globally yeah you were so you were so close on to you had it the
- 0:50thing
- 0:51is now you get in your own head so you just gotta like that's right you're good
- 0:54now
- 0:54you'reaproatthisi'mi'mlearningbyrunningalotofrepsalotofthisyou
- 0:58SPEAKER_00: know it's a whole bunch of this exactly
- 1:01SPEAKER_00: Maybe we'll leave this in, you know.
- 1:03SPEAKER_00: Okay.
- 1:05SPEAKER_00: Don't tune out.
- 1:05The Allegheny County Department of Human Services is one of the most
- 1:08interesting
- 1:09SPEAKER_00: state and local departments out there,
- 1:11SPEAKER_00: in my estimation.
- 1:12SPEAKER_00: Alex, thank you for coming on.
- 1:15SPEAKER_01: Thanks for having me.
- 1:15SPEAKER_01: I'm a big fan of IFP and Statecraft.
- 1:18SPEAKER_00: Oh, thank you.
- 1:19SPEAKER_00: We wanted to have you on to talk about data at the state and local level.
- 1:23Will you say a little bit before we get into this about just what you're proud
- 1:27of
- 1:28having worked on in the Allegheny County Department of Human Services from a
- 1:32data perspective?
- 1:33SPEAKER_01: Yeah,
- 1:33sotheDepartmentofHumanServicesinAlleghenyCountyisarguablythebest
- 1:37SPEAKER_01: integrated data system of any state or local agency in the country.
- 1:41And so we've had a two and a half decade sort of run of using integrated data
- 1:46that
- 1:46started with really taking a bunch of disparate government agencies that in
- 1:51many
- 1:51other jurisdictions are siloed and separate and putting them all under one umbrella,
- 1:56SPEAKER_01: the Department of Human Services here in the county.
- 1:58SPEAKER_01: And that spans operations from child welfare,
- 2:01SPEAKER_01: through homeless housing,
- 2:02SPEAKER_01: through the behavioral health treatment system for Medicaid patients.
- 2:06And while many of those systems are separated and sort of siloed in other jurisdictions,
- 2:10SPEAKER_01: they brought them under one umbrella agency.
- 2:13SPEAKER_01: And that really created the initial conditions for integrated data.
- 2:17SPEAKER_01: So by that I mean,
- 2:20SPEAKER_01: Suppose we see Santi Ruiz in the jail.
- 2:22HowdoweknowthatthatSantiRuizisthesameSantiRuizthatwe'reservingin
- 2:26SPEAKER_01: child welfare or in the behavioral health treatment system?
- 2:29And as you know,
- 2:30many of these systems really don't operate independent of one another.
- 2:33There are tremendous spillovers and cross coordination that needs to be done.
- 2:38And so the department really took data as a way not to do the type of research
- 2:42that
- 2:43SPEAKER_01: people like me get really fired up about,
- 2:45SPEAKER_01: but really for sort of meat and potatoes service coordination.
- 2:49SPEAKER_01: And that was an awesome way to start to say,
- 2:51SPEAKER_01: hey,
- 2:51SPEAKER_01: look,
- 2:52SPEAKER_01: you want these things not just for academic purposes,
- 2:54SPEAKER_01: but really to make our clients' lives better.
- 2:58There's this saying in the tech industry and maybe just in business
- 3:02generallythat
- 3:02SPEAKER_00: you ship your org chart,
- 3:04that the way your organization is structured is the way you actually produce
- 3:08the
- 3:09SPEAKER_00: thing you're producing.
- 3:10SPEAKER_00: Will you talk a little bit more about A,
- 3:13SPEAKER_00: what's under the umbrella of the Department of Human Services and B,
- 3:19SPEAKER_00: what are other ways of structuring it?
- 3:20SPEAKER_00: Why is this so interesting or unique?
- 3:22SPEAKER_00: Lots of counties do it other ways.
- 3:25SPEAKER_01: Right, that's right.
- 3:25So we have five main areas within the Department of Human Services in
- 3:29Allegheny County.
- 3:30SPEAKER_01: We have an aging office,
- 3:32SPEAKER_01: an office of intellectual disability,
- 3:34an office that's focused on behavioral health treatment for Medicaid
- 3:37patientsand
- 3:38SPEAKER_01: the uninsured.
- 3:39SPEAKER_01: So by behavioral health,
- 3:40SPEAKER_01: by the way,
- 3:41SPEAKER_01: for people who don't know,
- 3:42SPEAKER_01: I mean substance abuse and mental health services.
- 3:45We have an Office of Community Services that's focused on early childhood supports,
- 3:50SPEAKER_01: family strengthening,
- 3:51SPEAKER_01: and homeless and housing services.
- 3:53SPEAKER_01: And then finally,
- 3:54we have a large child welfare operation that investigates claims of abuse
- 3:58and
- 3:58SPEAKER_01: neglect against children.
- 4:00SPEAKER_00: And all of this lives in one department.
- 4:02SPEAKER_00: I think a lot of folks will listen to that and think, okay, cool.
- 4:05SPEAKER_00: Doesn't everybody do that?
- 4:06SPEAKER_00: It's like, this is all this bundled stuff.
- 4:08SPEAKER_01: That's right.
- 4:08Solving the org design challenge here is just massive because what exists in
- 4:13many
- 4:14other jurisdictions might be a strict subset of that compilation of
- 4:18services where...
- 4:20SPEAKER_01: you sort of have individuals running these different departments.
- 4:23Maybe there are sort of fiefdoms where they don't sort of cooperate very well.
- 4:28Maybe there are difficulties for things that we care a lot about in terms of
- 4:31data sharing.
- 4:32SPEAKER_01: So for example,
- 4:32getting lawyers from different county departments or different local
- 4:35agencieson
- 4:36the same page could be quite difficult and intractable in other places that
- 4:39I've seen.
- 4:40SPEAKER_01: And by just resolving that one sort of org design issue,
- 4:44you've sort of set in motion much greater fostering of collaboration in ways
- 4:50that
- 4:50SPEAKER_01: just really is challenged,
- 4:52SPEAKER_01: I think,
- 4:53SPEAKER_01: in many other jurisdictions.
- 4:55Will you talk a little bit about the kinds of problems that you get in
- 4:59Allegheny County?
- 5:00SPEAKER_00: And for listeners who don't know,
- 5:01SPEAKER_00: Allegheny County is Pittsburgh,
- 5:03SPEAKER_00: Pennsylvania,
- 5:03SPEAKER_00: and the surrounding environs.
- 5:06What do you guys face that's the same as plenty of other mid-sized cities in
- 5:09the country?
- 5:10SPEAKER_00: And what's unique or what stands out?
- 5:12I think that's one of the coolest parts about working in local government is
- 5:16while
- 5:16SPEAKER_01: it feels like you're only serving a small,
- 5:19SPEAKER_01: very delimited number of people,
- 5:21because the problems that you're facing are quite similar to problems that
- 5:25you're
- 5:25SPEAKER_01: seeing across the country,
- 5:27SPEAKER_01: there's an opportunity,
- 5:28SPEAKER_01: I think,
- 5:28SPEAKER_01: to really build evidence and solutions here that scale nationally.
- 5:32Soyoucouldhavedifferencescertainlyintermsoftheexactcompositionof
- 5:38SPEAKER_01: refugee populations,
- 5:39SPEAKER_01: the exact composition of the drug supply.
- 5:41Thosethingsaregonnabedifferentacrossthecountryandthoseallpresent
- 5:44SPEAKER_01: different challenges to different departments.
- 5:47SPEAKER_01: But at the end of the day,
- 5:48you have many common features and many common challenges that I think make
- 5:51thisjob
- 5:52SPEAKER_01: incredibly exciting.
- 5:53SPEAKER_01: So for example,
- 5:54SPEAKER_01: figuring out how best to address child maltreatment,
- 5:57SPEAKER_01: how to really effectively address serious mental illness,
- 6:00SPEAKER_01: downtown homelessness,
- 6:02SPEAKER_01: street encampments,
- 6:03those sorts of things are incredibly important and I think quite relevant as
- 6:07Ilook
- 6:08SPEAKER_01: across the national landscape.
- 6:10SPEAKER_00: Will you just say a little bit more about the specifics, though?
- 6:13SPEAKER_00: I mean,
- 6:14SPEAKER_00: to take the drug point,
- 6:15what's the kind of mix of substance abuses that you guys get in particular in Pittsburgh?
- 6:21SPEAKER_01: Yeah,
- 6:21SPEAKER_01: so Allegheny County,
- 6:23we have an overdose death rate that looks a whole lot like our peer counties as
- 6:28far
- 6:28SPEAKER_01: as what's happening in Philadelphia,
- 6:31SPEAKER_01: Baltimore,
- 6:32places that you would sort of suspect based off of our Rust Belt heritage and
- 6:37the
- 6:37fact that some of these problems in terms of the opioid epidemic
- 6:40specifically
- 6:41SPEAKER_01: really washed ashore in some of these communities quite early.
- 6:45So our rate doesn't look a whole lot different than many of those other jurisdictions.
- 6:51SPEAKER_01: The through line,
- 6:53SPEAKER_01: though,
- 6:53for many of the services that we run are those types of problems where you sort
- 6:57of
- 6:57SPEAKER_01: think about them as confined to the behavioral health system,
- 6:59SPEAKER_01: for example,
- 7:00SPEAKER_01: or problems that are confined to private individuals.
- 7:03But really,
- 7:04they drive a tremendous amount of criminal justice involvement.
- 7:09So they represent a disproportionate share of people booked into the jail.
- 7:12They represent a disproportionate share of the people involved in child welfare.
- 7:17SPEAKER_01: In the homeless housing situation that we've got.
- 7:20SPEAKER_01: And so in many different settings,
- 7:22SPEAKER_01: these things,
- 7:22SPEAKER_01: I think,
- 7:23have common underlying features that make it important for us to really
- 7:26thinkabout
- 7:26SPEAKER_01: common drivers and how best to solve them.
- 7:30SPEAKER_00: Will you put some numbers on those?
- 7:32SPEAKER_00: I've got a talk that you gave at AEI here in front of me,
- 7:34SPEAKER_00: and I'd just love more context from you.
- 7:37SPEAKER_00: You say in this talk in Allegheny County,
- 7:40aboutafifthofparentswhoarechildwelfareinvolvedhaveadiagnosisofan
- 7:44SPEAKER_00: opioid use disorder,
- 7:46and individuals with serial mental illnesses are hospitalized at rates
- 7:50similarto
- 7:50SPEAKER_00: incarceration at state and federal prison.
- 7:53Will you give me a little bit more context on the kind of revolving door of
- 7:56services here?
- 7:57SPEAKER_00: Yeah.
- 7:57SPEAKER_01: That's right.
- 7:58SPEAKER_01: So in Allegheny County,
- 7:59SPEAKER_01: which is a county of 1.2 million people,
- 8:01we get somewhere on the order of 15,000 referrals of child maltreatment to
- 8:07our
- 8:08SPEAKER_01: child welfare hotline.
- 8:10SPEAKER_01: Every year?
- 8:11SPEAKER_01: Every year.
- 8:12SPEAKER_01: About half of those will be screened in for investigation.
- 8:16SPEAKER_01: And so if we look specifically among that group,
- 8:19SPEAKER_01: you start to see very high rates of indications of addiction.
- 8:23SPEAKER_00: Will you just explain screened in versus screened out?
- 8:27SPEAKER_01: Yeah.
- 8:27SPEAKER_01: So in the child welfare system,
- 8:28there's a sort of a front door where referrals for children who are suspected
- 8:33of
- 8:34SPEAKER_01: being maltreated are first processed.
- 8:37And they're the initial decision that's facing the child welfare agencies
- 8:41is
- 8:41SPEAKER_01: whether I should screen a child in for investigation or screen them out.
- 8:45SPEAKER_01: If I screen them out,
- 8:46that means that I've sort of declined to investigate the family or the claim
- 8:50and
- 8:50SPEAKER_01: the
- 8:51SPEAKER_01: things just sort of go along as you would suspect.
- 8:54SPEAKER_01: But if I then,
- 8:55SPEAKER_01: as the agency,
- 8:56SPEAKER_01: choose to investigate that referral,
- 8:59I will eventually send out child welfare caseworkers who will go and conduct
- 9:04home visits.
- 9:05SPEAKER_01: They'll call the person who
- 9:08SPEAKER_01: made the referral to understand a little bit more details.
- 9:10We'll dig into some of our data to understand a little bit of the history of the family.
- 9:15SPEAKER_01: And this is also where in Allegheny County,
- 9:18we've been really pioneers for developing what's known as the Allegheny
- 9:21Family
- 9:22SPEAKER_01: Screening Tool,
- 9:23which is a machine learning algorithm that sits on the front door of child welfare,
- 9:27SPEAKER_01: because you might be asking yourself at this point,
- 9:29SPEAKER_01: but with 15,000 referrals,
- 9:32SPEAKER_01: How do people make informed decisions about which referrals to go out and
- 9:37SPEAKER_01: investigate and which ones not?
- 9:39SPEAKER_01: And it's an incredibly high stakes decision.
- 9:42Anditcanbedifficulttocallscreeningstaffbecauseyoumightonlysee
- 9:48SPEAKER_01: a little bit of sort of an anecdote, a narrative.
- 9:52The person who made the referral might have just seen something that really
- 9:55sortof
- 9:55SPEAKER_01: worried them,
- 9:56SPEAKER_01: but might not have the full story on the family.
- 9:59SPEAKER_01: And it's also really difficult for humans to think in terms of risk.
- 10:02SPEAKER_01: It's just not a sort of thing that we're quite native at.
- 10:07There's a great book by Philip Tetlock called Expert Political Judgment
- 10:11that'son
- 10:12SPEAKER_01: the sort of limited ability for intelligence analysts,
- 10:14SPEAKER_01: for example,
- 10:15tobeatoutpeoplewhoaregeneralistreadersoftheNewYorkTimesorsimple
- 10:19SPEAKER_01: algorithms in terms of forecasting geopolitical risk.
- 10:23I think you face a very sort of similar set of circumstances here in child welfare.
- 10:28And so what the algorithm is trying to do is to predict over the next two years
- 10:32what
- 10:32will concerns of maltreatment rise to the level such that a judge would order
- 10:37the
- 10:37SPEAKER_01: child to be removed from the home?
- 10:40And so really sort of using that risk score as a way to think about where we
- 10:46should
- 10:46be allocating scarce caseworker resources and really where we have the
- 10:50greatest
- 10:50SPEAKER_01: concerns is super important.
- 10:52SPEAKER_01: And we take that machine learning information,
- 10:54SPEAKER_01: those risk scores,
- 10:55SPEAKER_01: and we pair it with the insights from the call screening staff.
- 10:58So that is the model doesn't totally substitute for human decision making.
- 11:03They work in close tandem with one another to actually make more informed
- 11:07decisions overall.
- 11:09So just to map this out,
- 11:0915,000 referrals in a year is something like 40 a day.
- 11:15Andyou'vegotafewpeopleatCPSwhotakethesecallsandthecallsarecoming
- 11:20from some cross section of neighbors or the preschool teacher or someone on
- 11:25the street.
- 11:26SPEAKER_00: Will you just flesh out for me?
- 11:28I'monthehotlineatCPSandI'msittinginfrontofaterminalandIgetacall
- 11:33SPEAKER_00: from a preschool teacher worried about a kid in their class.
- 11:36SPEAKER_00: What happens next?
- 11:38SPEAKER_01: Yeah,
- 11:38sotypicallywhatwewilldoisstarttofilloutinformationaboutthevictim
- 11:45childandtheallegedperpetratortoreallysortofunderstandwhatarethe
- 11:49SPEAKER_01: dynamics here.
- 11:50So we'll often make calls back to the referral source to get more information,
- 11:55SPEAKER_01: understand the story a little bit better.
- 11:58SPEAKER_01: At that point,
- 11:59we'll sort of understand whether we can put together sort of a complete
- 12:03pictureof
- 12:05SPEAKER_01: the referral.
- 12:06Andthatreallystartstothengeneratethedownstreamprocessesintermsof
- 12:11getting enough information to then make an informed decision on screening a
- 12:16child
- 12:16SPEAKER_01: in or out.
- 12:18And what's that process difference between how somebody working at CPS in
- 12:23Allegheny
- 12:24SPEAKER_00: County would go through this and somebody in,
- 12:26SPEAKER_00: I don't know,
- 12:27SPEAKER_00: Idaho?
- 12:27Are folks at most counties not able to cross-check this data to see if parents
- 12:33have
- 12:33SPEAKER_00: been in jail recently,
- 12:34SPEAKER_00: if they've been in the hospital recently?
- 12:36So my sense is that what's unique is maybe the sheer breadth of information
- 12:41thatwe
- 12:41SPEAKER_01: have available to us.
- 12:44I don't know every jurisdiction in terms of their child welfare processes
- 12:47exactly
- 12:48SPEAKER_01: what information they'll all be looking at.
- 12:50I suspect many of them are going to do exactly what you were sort of talking about.
- 12:55But I think what's unique for us is just the sheer amount of information and
- 12:59the
- 12:59SPEAKER_01: historical capture of that data.
- 13:01SPEAKER_01: So, for example, you could see educational information about the child.
- 13:06SPEAKER_01: You could see across system involvement.
- 13:09SPEAKER_01: You could see,
- 13:10SPEAKER_01: for example,
- 13:11if there's an alleged perpetrator who themselves was in child welfare historically,
- 13:15SPEAKER_01: you start to really sort of build out,
- 13:17SPEAKER_01: I think,
- 13:18a very rich understanding about who the people that you're talking about on a
- 13:22given
- 13:23SPEAKER_01: referral really are.
- 13:25SPEAKER_00: And just to clarify for me,
- 13:27SPEAKER_00: the CPS folks on the end of the call,
- 13:30SPEAKER_00: they're making their own call about like,
- 13:32SPEAKER_00: what's the risk to this kid?
- 13:34And at the same time,
- 13:36the Allegheny Family Screening Tool is generating its own score.
- 13:39And that may or may not line up with the judgment of the person taking the call.
- 13:44SPEAKER_01: That's exactly right.
- 13:45SPEAKER_01: And so ultimately the discretion exists with the staff,
- 13:49even when there's a sort of defaulted option that the child should be
- 13:54screenedin
- 13:55SPEAKER_01: or should be screened out,
- 13:56SPEAKER_01: that they can always override.
- 13:57SPEAKER_01: So that always exists at their discretion.
- 14:00And I think it's an important sort of set of check and balances against that.
- 14:05against algorithms where these things can really sort of complement each other.
- 14:08SPEAKER_01: Because there's,
- 14:08SPEAKER_01: of course,
- 14:09things that the algorithm won't pick up that might be gleaned from contacts
- 14:14with
- 14:14SPEAKER_01: referral sources,
- 14:15SPEAKER_01: for example.
- 14:16SPEAKER_00: Can I get a couple examples from you?
- 14:18I'dlovetogetanexampleofwhenmightahumanscreeninsomeonethatthe
- 14:23SPEAKER_00: algorithm says at low risk?
- 14:25SPEAKER_00: And when might a human screen out somebody who the algorithm is saying,
- 14:29SPEAKER_00: ah,
- 14:29SPEAKER_00: it seems like a high risk case?
- 14:31There can be a set of children who might be at high risk of adverse outcomes
- 14:37whose
- 14:37SPEAKER_01: primary concern is in child maltreatment.
- 14:39SPEAKER_01: So think about a youth,
- 14:41SPEAKER_01: for example,
- 14:42SPEAKER_01: who might be at risk of juvenile probation involvement,
- 14:46SPEAKER_01: for example.
- 14:47That's the sort of thing where somebody might call child welfare about it.
- 14:52SPEAKER_01: But unless we think that there's a child maltreatment risk at issue,
- 14:56SPEAKER_01: we wouldn't want to screen that in.
- 14:59They may be at risk of all kinds of stuff,
- 15:01but it's not danger in the home.
- 15:03SPEAKER_01: Exactly.
- 15:04A low risk situation where you might want to screen in is that maybe the
- 15:08historical
- 15:09SPEAKER_01: data around the family isn't pointing to strong risk factors,
- 15:13SPEAKER_01: but the specific nature of the allegations could be quite substantial.
- 15:18And so you could think about ad hoc one-off events where something terrible happens,
- 15:24SPEAKER_01: where there was...
- 15:26SPEAKER_01: It sort of came out of the blue, for example.
- 15:28SPEAKER_01: And those are why you have call screening staff,
- 15:31especially to really do that due diligence and make sure that we're thinking
- 15:34about
- 15:35SPEAKER_01: these things clearly.
- 15:36And talk to me about the effect of using something like the Allegheny Family
- 15:39Screening Tool.
- 15:40SPEAKER_00: I mean, what's the pre-post effect of using these algorithms?
- 15:44SPEAKER_01: So we've had a couple of evaluations of this work done.
- 15:49I should say that just at the beginning,
- 15:51people might be skeptical about...
- 15:53SPEAKER_01: why is child removal from the home the right set of outcomes?
- 15:57Isn't that something that you'd be worried about given that it's connected
- 16:02withall
- 16:02sorts of system involvement and it's the decision of a judge and that could be biased?
- 16:08And I think that those are the types of problems we've tried to think really
- 16:10deeply about.
- 16:11And so one of the initial studies that was done on the tool was really this kind
- 16:14of validation.
- 16:15SPEAKER_01: So
- 16:16SPEAKER_01: If you wanted to,
- 16:17SPEAKER_01: for example,
- 16:18see how there are predictions of home removal measured against
- 16:22hospitalization for maltreatment,
- 16:23SPEAKER_01: for example,
- 16:24SPEAKER_01: we've sort of done that comparison.
- 16:27SPEAKER_01: And those two predictions are very highly correlated.
- 16:29And so that gives us some comfort that this isn't just capturing this one set
- 16:34of
- 16:35harms that we think aren't really mapping to other ways to think about maltreatment.
- 16:41SPEAKER_01: But these things seem highly correlated, which is great.
- 16:43SPEAKER_00: Will you just explain maybe a fifth grade level?
- 16:47SPEAKER_00: What are you measuring with that correlation?
- 16:49The fact that home removal is highly correlated with hospitalization for maltreatment.
- 16:54SPEAKER_00: Give me that again in layman's terms.
- 16:56SPEAKER_01: Yeah,
- 16:56SPEAKER_01: so I think what you might be concerned about is that the concept of child
- 17:00SPEAKER_01: maltreatment is sort of amorphous.
- 17:03SPEAKER_01: So how do you actually measure that?
- 17:07And home removal is one very solid way because you have to go through a
- 17:11judicial process.
- 17:12SPEAKER_01: The judge ultimately has to make that determination.
- 17:15SPEAKER_01: They're making this determination based off of child welfare.
- 17:19SPEAKER_01: But there are other ways that you could possibly measure this.
- 17:21SPEAKER_01: One other way would be,
- 17:22SPEAKER_01: for example,
- 17:23if you show up to a hospital and you need to receive services for injuries that
- 17:27are
- 17:27SPEAKER_01: consistent with maltreatment.
- 17:29And so that's what this comparison really got to is how does the prediction
- 17:33andthe
- 17:34risk for individuals who are at high risk of home removal compared to their
- 17:37risk
- 17:38SPEAKER_01: for showing up in the hospital due to maltreatment-related injuries.
- 17:42And those things go together quite well,
- 17:45andthatgivesussomecomfortthat
- 17:47SPEAKER_01: We're not just capturing some very idiosyncratic,
- 17:51very specific effect of the removal decision and the definition of the
- 17:56removalas
- 17:57SPEAKER_01: the outcome,
- 17:57but rather sort of broadly capturing what we would think about as child maltreatment.
- 18:02because hospitalization for physical injuries is hard to introduce bias there.
- 18:07SPEAKER_01: It's hard to disagree that it is in some measure of something.
- 18:11SPEAKER_01: It has its imperfections as well.
- 18:13SPEAKER_01: But again,
- 18:13I think that we're trying to sort of argue that we shouldn't let the imperfect be,
- 18:18SPEAKER_01: or the perfect be the enemy,
- 18:19SPEAKER_01: the good here,
- 18:20SPEAKER_01: and rather let's just be our own best critic.
- 18:23SPEAKER_01: Let's take sensible approaches.
- 18:25SPEAKER_01: And then as your question sort of asked earlier,
- 18:28you can get into the impact of these things,
- 18:30which I think is super exciting.
- 18:31SPEAKER_01: So there's two different concerns with algorithms.
- 18:33One is how is the algorithm actually performing in terms of out-of-sample prediction?
- 18:38Sothat'salwaystheproofpointforalgorithmsisyouwanttoknowwhenthe
- 18:44SPEAKER_01: algorithm sees data that it hasn't seen before,
- 18:47that it's actually pretty accurate in terms of forecasting what's going to
- 18:50happen
- 18:50SPEAKER_01: to those individuals or those referrals.
- 18:53SPEAKER_01: So that's one test.
- 18:54SPEAKER_01: I think we've
- 18:55SPEAKER_01: got a lot of data that sort of gives us comfort there.
- 18:58But then the other thing is relative to this counterfactual world in which
- 19:01the
- 19:01SPEAKER_01: algorithm doesn't exist,
- 19:03SPEAKER_01: did we actually improve children and family outcomes?
- 19:08And here we've got some very interesting new data by Katherine Rittenhouse
- 19:12and
- 19:12SPEAKER_01: co-authors that looks at the introduction of the AFST.
- 19:17SPEAKER_01: and finds that conditional and being screened in,
- 19:20SPEAKER_01: it eliminates the black-white gap in removal rates.
- 19:24So this is an issue of racial bias that has been incredibly politically sensitive,
- 19:30but incredibly salient to people working in child welfare that these
- 19:34algorithmic
- 19:34SPEAKER_01: tools could exacerbate those sorts of problems.
- 19:36SPEAKER_01: And in fact,
- 19:37the researchers here found some quite comforting and quite encouraging
- 19:41evidence
- 19:41SPEAKER_01: that these things are eliminating the black-white gap.
- 19:44SPEAKER_00: This is really interesting,
- 19:45SPEAKER_00: and I want to just push a little harder because,
- 19:49SPEAKER_00: like you say,
- 19:50SPEAKER_00: when these tools were first being rolled out,
- 19:52and the Allegheny Family Screening Tool was in the middle of the last decade,
- 19:56SPEAKER_00: I think in 2016,
- 19:58SPEAKER_00: there was all this debate over algorithmic bias,
- 20:00SPEAKER_00: and the idea was exactly what you're describing,
- 20:04SPEAKER_00: that
- 20:05SPEAKER_00: oh no,
- 20:05SPEAKER_00: these tools are going to sample on,
- 20:07SPEAKER_00: they're going to incorporate data that we already have is full of racial
- 20:10SPEAKER_00: disparities and bias,
- 20:12and it's going to aggravate whatever problems we already have in the sample.
- 20:16And so I just want to kind of double down on this because it's really striking.
- 20:19SPEAKER_00: Exactly the opposite happened with this tool.
- 20:22SPEAKER_00: You had less racial bias by this measurement in outcomes.
- 20:28SPEAKER_01: And our department director now, Aaron Dalton,
- 20:31along with our then department director,
- 20:33Mark Turner,
- 20:34really spearheaded this initiative.
- 20:36And I think it's great that they were proven out by the data and that the team
- 20:40has
- 20:40SPEAKER_01: been proven out by the data.
- 20:42But one of the things that really struck me coming into my role and stepping into,
- 20:48SPEAKER_01: you know,
- 20:49SPEAKER_01: Aaron's big shoes to fill in this position was
- 20:52SPEAKER_01: that they took incredible pains to,
- 20:56SPEAKER_01: I think,
- 20:56reallyengagecommunityandheartheirconcernsandunderstandhowbestto
- 21:02SPEAKER_01: incorporate that into the tool.
- 21:03SPEAKER_01: And they,
- 21:04SPEAKER_01: I think,
- 21:05SPEAKER_01: painstakingly went through the process of writing an RFP,
- 21:07getting external researchers with impeccable credentials to come in and advise.
- 21:12SPEAKER_01: They went through an ethical review.
- 21:14We've tried to make every analysis transparent that we've done on the tool.
- 21:19SPEAKER_01: And so I think it's great to be sort of validated after the fact,
- 21:23but I think that Aaron and Mark and the team really went through the right
- 21:28process
- 21:28SPEAKER_01: steps here,
- 21:29which were important to even earning the trust from the community and from
- 21:33staffto
- 21:35SPEAKER_01: implement this to begin with.
- 21:36I feel like this is a useful case study for a broader dynamic that you kind of
- 21:41want
- 21:41SPEAKER_00: to see more of in public policy,
- 21:43SPEAKER_00: which is there was a very plausible story about how this could go wrong.
- 21:47SPEAKER_00: Algorithmic bias was this very easy to kind of describe.
- 21:52It was the conventional wisdom in some ways about why these tools wouldn't work.
- 21:56It just wasn't true when you guys went with proper care and attention to detail.
- 22:02SPEAKER_01: That's right.
- 22:02And so many of these things,
- 22:03I think,
- 22:04come down to process and are somewhat empirical questions.
- 22:07Like there are certain circumstances under which building algorithms
- 22:11wouldbethe
- 22:12SPEAKER_01: wrong thing to do.
- 22:13ButIthinkthereisaplaybookherethatonecouldtakeandapplyindifferent
- 22:18SPEAKER_01: settings and give themselves that comfort.
- 22:20SPEAKER_01: And I do think that that measurement after the fact
- 22:23SPEAKER_01: is incredibly important because if you see,
- 22:26SPEAKER_01: for example,
- 22:26SPEAKER_01: that this is having adverse effects,
- 22:29SPEAKER_01: then you then have an opportunity to actually iterate.
- 22:32AndsomuchofwhatIthinkthepublicsectorisreallychallengedbyisthatwe
- 22:36SPEAKER_01: don't have an objective function.
- 22:37SPEAKER_01: Like if I'm an ice cream shop or whatever and I make terrible ice cream,
- 22:41SPEAKER_01: I'm going to go out of business real quick.
- 22:43SPEAKER_01: I'm going to figure that out very, very, very soon.
- 22:45SPEAKER_01: In child welfare, that's not true.
- 22:48Andsobeingabletoreallybetransparentwithourselvesaboutreallycrisp
- 22:52feedback loops in terms of what impact the decisions we're making or having
- 22:56inthe
- 22:56community and on the children that we're responsible for from the
- 23:00maltreatment
- 23:01SPEAKER_01: protection perspective is incredibly important.
- 23:04SPEAKER_00: Yeah.
- 23:05For those of us who did not study linear programming,
- 23:07what is an objective function?
- 23:11SPEAKER_01: Right.
- 23:11I would say it's the thing that you are trying to maximize or minimize within
- 23:18any setting.
- 23:19SPEAKER_01: So in business,
- 23:20what you're trying to do is to typically maximize something like free cash
- 23:24flow or profits.
- 23:25And that sort of crisp statement that provides the kind of the North Star
- 23:29vision
- 23:29SPEAKER_01: for what you're trying to accomplish,
- 23:32SPEAKER_01: or at least one plausible measurement of it,
- 23:34SPEAKER_01: then just really doesn't exist in much of human services.
- 23:39SPEAKER_00: I'm looking at this report about Mark Cherna,
- 23:42SPEAKER_00: who was the director of the Department of Human Services.
- 23:46And he had an independent ethics review conducted of this predictive
- 23:51program before it began.
- 23:52SPEAKER_00: And it had this takeaway that I thought was well put.
- 23:55SPEAKER_00: It's not just that implementing the program is probably ethical,
- 23:59SPEAKER_00: but also that not using it might be unethical.
- 24:01SPEAKER_00: That is, if you have
- 24:03the most accurate and predictive instrument available to you to assess risk
- 24:07for children.
- 24:08SPEAKER_00: It's a strong argument that it's unethical not to use it in your work.
- 24:13SPEAKER_00: In early 2023,
- 24:15therewereacoupleofreportsthatthentheBideneraJusticeDepartmentwas
- 24:20looking at or making inquiries into the way this tool worked and the concern
- 24:25from
- 24:25some outside groups was that it could result in discrimination against
- 24:28families
- 24:29SPEAKER_00: with disabilities.
- 24:30SPEAKER_00: How did you guys think about that risk?
- 24:34I won't comment on the specifics of the inquiry or the sort of the nature of
- 24:38what
- 24:38SPEAKER_01: is kind of a fluid back and forth with us in the Department of Justice.
- 24:42But in general,
- 24:43I think what people are concerned about is discrimination against individuals.
- 24:48SPEAKER_01: So for example,
- 24:49SPEAKER_01: there have been cases in the past where people have been concerned about
- 24:53SPEAKER_01: discrimination against individuals in recovery from being foster parents,
- 24:57SPEAKER_01: for example.
- 24:58That'sthesortofthingthatIthinktheDepartmentofJusticeisprobably
- 25:03SPEAKER_01: concerned about here.
- 25:04SPEAKER_01: And on our side,
- 25:07SPEAKER_01: the way that that manifests is things like,
- 25:09SPEAKER_01: what are the features that are feeding the model?
- 25:11SPEAKER_01: So for example,
- 25:12do you have things like diagnostic history and those sorts of things that I
- 25:17think
- 25:17SPEAKER_01: could be sensitive to individuals?
- 25:20SPEAKER_01: In our setting,
- 25:21what I think we're interested in is thinking about all of that information
- 25:26together
- 25:26SPEAKER_01: and what's actually predictive of risk.
- 25:28And there are certain sort of patterns of behavioral health utilization
- 25:33thatare
- 25:35SPEAKER_01: consistent with higher risk.
- 25:36SPEAKER_01: It doesn't mean,
- 25:38SPEAKER_01: for example,
- 25:38SPEAKER_01: that these things are presumptive on any level,
- 25:41SPEAKER_01: but there are patterns there that the algorithm learns over time.
- 25:46SPEAKER_00: Sure.
- 25:47And I'm assuming even with the other example you gave of people in recovery
- 25:51being
- 25:52SPEAKER_00: foster parents,
- 25:52SPEAKER_00: there probably is a population level,
- 25:54a higher risk to the child if your foster parents have had a lot of encounters
- 25:59with
- 26:00SPEAKER_00: substance abuse in the recent past.
- 26:04SPEAKER_01: Yeah, right.
- 26:05Ithinkthatwhatthey'reconcernedaboutissortofbroadblanketsortof
- 26:08restrictions and rules rather than something in that case that would take
- 26:12advantage
- 26:13SPEAKER_01: of more nuanced information.
- 26:15SPEAKER_00: Sure.
- 26:16SPEAKER_00: You guys built this tool, the Allegheny County screening tool, in-house.
- 26:20It's software developed by the county rather than taking it off the shelf
- 26:24from elsewhere.
- 26:25SPEAKER_00: I think if people are familiar with this sort of thing,
- 26:27they'll be familiar with outside contractors that the government is trying
- 26:31totry
- 26:31SPEAKER_00: to bring in.
- 26:32Talk to me about the decision to build it and to sustain these systems internally.
- 26:36So we had competitively bid for and solicited for external help on
- 26:44developing the tool,
- 26:45but you're totally right that the tool is internal in the sense of using our
- 26:48data
- 26:48SPEAKER_01: and being owned by the Department of Human Services.
- 26:51SPEAKER_01: But we wanted, and where I think that there's often quite
- 26:54goodvalueintermsoftappinginsomeexternalhelpwewantedsortofthebest
- 26:58available expert guidance that we could possibly have on developing this
- 27:03algorithm
- 27:03inchildwelfareintherightwayandsoyouknowovertimeithinkwhatthe
- 27:08departmenthasreallytriedtoputanemphasisonisinternalstatecapacity
- 27:12SPEAKER_01: building so we have a 40 person analytics team we have a
- 27:16SPEAKER_01: 10 person data science team.
- 27:18We have a 10 person engineering team that's developing our own software now.
- 27:22And what I see often is that because there's a lack of skill
- 27:26complementarities
- 27:27between internal government folks and external advisors or consultants or vendors,
- 27:33SPEAKER_01: you often get into situations in which timelines get really lengthened
- 27:37The ambition can sometimes be incredibly lopsided in terms of coming from
- 27:41the
- 27:41SPEAKER_01: outside versus the inside.
- 27:42SPEAKER_01: That can lead to all sorts of problems.
- 27:45SPEAKER_01: It can really result in,
- 27:46SPEAKER_01: I think,
- 27:47SPEAKER_01: not only much slower,
- 27:48but much lower quality software and data science and analytics being
- 27:52delivered because,
- 27:53SPEAKER_01: for example,
- 27:54the agency hasn't taken control and adequately specified what its vision is,
- 27:58SPEAKER_01: how it wants to use this,
- 27:59SPEAKER_01: what the context is,
- 28:00SPEAKER_01: what are the goals.
- 28:01And so I think it really puts government agencies often in a very lopsided
- 28:05position
- 28:06where it feels like an efficient use of government resources to say we're not
- 28:09going
- 28:09SPEAKER_01: to hire internal staff.
- 28:10SPEAKER_01: But in fact, it becomes costly in a million different ways.
- 28:14SPEAKER_00: Right.
- 28:16SPEAKER_00: Let me change gears.
- 28:17I want to talk about something else that the Department of Human Services
- 28:20focuses on,
- 28:20SPEAKER_00: and that's folks with serious mental health issues.
- 28:23SPEAKER_00: These things interact, but you also have a totally different stream.
- 28:26I was really struck by the number of involuntary commitments every year in
- 28:30the U.S.
- 28:30SPEAKER_00: People involuntarily committed into psychiatric or medical treatment.
- 28:34SPEAKER_00: One million commitments.
- 28:35SPEAKER_00: I assume a lot of those are repeat customers.
- 28:38SPEAKER_00: Who is being involuntarily committed?
- 28:40SPEAKER_00: I'm assuming it's not 1 million people a year.
- 28:42SPEAKER_00: I'm assuming the 80-20 rule applies here a little bit too.
- 28:45SPEAKER_00: But yeah,
- 28:46talk to me a little bit about the population that you guys work with and how
- 28:48that
- 28:48SPEAKER_00: compares nationally.
- 28:50SPEAKER_01: Yeah.
- 28:50SPEAKER_01: So every state in the U.S.
- 28:52has a law in the books that is designed to help individuals who are at risk of
- 28:58harm
- 28:58SPEAKER_01: to themselves or others of receiving treatment.
- 29:01And one of the things that we've really dug into over the last couple of years
- 29:06asa
- 29:06team is how to really think about what's going on with this group and how to
- 29:12really
- 29:13SPEAKER_01: improve their health outcomes downstream.
- 29:15And so this has been one of my favorite projects because you very rarely get to
- 29:18see
- 29:18the end-to-end picture of how do we sort of describe what people are going through?
- 29:23SPEAKER_01: How do we sort of think about underlying causal mechanisms?
- 29:26SPEAKER_01: And then how do we propose some solutions to the problem?
- 29:29SPEAKER_01: So a couple of years ago,
- 29:29SPEAKER_01: we sort of sat down as a team and we said,
- 29:31SPEAKER_01: okay,
- 29:31SPEAKER_01: great,
- 29:31SPEAKER_01: we're going to take a deep dive into this data.
- 29:33And a side note here is that Allegheny County might be the only place in the
- 29:37entire
- 29:37SPEAKER_01: country where you could pull this type of analysis off.
- 29:40SPEAKER_01: But we,
- 29:40SPEAKER_01: for example,
- 29:41really looked at what's going on in terms of this group's labor market outcomes.
- 29:45SPEAKER_01: How is their medication adherence?
- 29:48SPEAKER_01: What does their mortality rate look like?
- 29:50SPEAKER_01: And one of the awesome things that this analysis facilitated was this,
- 29:55SPEAKER_01: I think,
- 29:56quite disturbing finding that there's an 8% mortality rate for these
- 30:00individualsin
- 30:01Allegheny County a year after being examined for involuntary hospitalization.
- 30:05SPEAKER_01: So just for comparison,
- 30:07SPEAKER_01: for example,
- 30:08if we look at that against the mortality rate on an age and gender adjusted
- 30:11basis
- 30:12for people who are coming out of our jail or who are going into homeless shelters,
- 30:17SPEAKER_01: or the general population of low-income individuals in the county,
- 30:20SPEAKER_01: for example,
- 30:21SPEAKER_01: this is a rate that just far exceeds that.
- 30:24And so while I think everybody really understands that individuals with
- 30:28serious
- 30:28SPEAKER_01: mental illness might have higher mortality rates,
- 30:32I think being able to put a specific number on it and to go to our partners,
- 30:36SPEAKER_01: go to ourselves and say,
- 30:38SPEAKER_01: it's not some general conceptual problem that's going on.
- 30:42SPEAKER_01: It's an 8% mortality rate.
- 30:43SPEAKER_01: It looks like some pretty serious cancers.
- 30:47SPEAKER_01: in one year, what are we going to do about this?
- 30:50SPEAKER_01: So that's really a credit to my teammate,
- 30:52SPEAKER_01: Pinwelli,
- 30:53SPEAKER_01: who's our chief data scientist here,
- 30:55SPEAKER_01: really sort of fleshed all of that out.
- 30:57And then what you actually see in the data is this incredibly cool kind of
- 31:01natural
- 31:02SPEAKER_01: experiment that's going on,
- 31:03SPEAKER_01: which is that different clinicians view risk quite differently.
- 31:07SPEAKER_01: So if you're a clinician,
- 31:10SPEAKER_01: Santi,
- 31:10SPEAKER_01: and I'm a clinician,
- 31:12we might have different propensities to actually involuntarily commit people.
- 31:17SPEAKER_01: So you might be quite lenient and say,
- 31:19SPEAKER_01: you know,
- 31:20SPEAKER_01: that doesn't meet my bar,
- 31:21SPEAKER_01: that's not that concerning,
- 31:22SPEAKER_01: and I might be quite conservative and so really commit many people.
- 31:26Those differences across positions really presents this really
- 31:29interesting
- 31:29opportunity because then we can not only study the fact that downstream
- 31:34peoplehave
- 31:34SPEAKER_01: poor outcomes,
- 31:35but we can then look at to what extent are these being caused by being upheld in
- 31:41an
- 31:41SPEAKER_01: involuntary commitment
- 31:43SPEAKER_01: And this is a place where data,
- 31:44SPEAKER_01: I think,
- 31:44matters a ton because people who have diametrically opposed views on this
- 31:48whoare
- 31:49SPEAKER_01: quite informed,
- 31:50SPEAKER_01: people who think involuntary hospitalization is terrible,
- 31:53SPEAKER_01: it's incredibly disruptive,
- 31:55SPEAKER_01: it's traumatic to the person,
- 31:56SPEAKER_01: the way that we do in the United States,
- 31:59SPEAKER_01: maybe it's not that effective.
- 32:00SPEAKER_01: You have people who look at the other side of it and say,
- 32:04I really don't know how to get help for my loved one or the police saying,
- 32:06I don't know how to get help for this individual other than for psychiatric care.
- 32:11And so being able to provide evidence on that is something that we're
- 32:14steppinginto
- 32:15SPEAKER_01: now with not just PIM,
- 32:16SPEAKER_01: but a couple of our research partners,
- 32:18SPEAKER_01: Natalia Emanuel and Val Velotny.
- 32:21And what we're finding is that actual involuntary commitment for people
- 32:26thatare
- 32:26SPEAKER_01: sort of uncertain,
- 32:27SPEAKER_01: where different clinicians could possibly make different judgments,
- 32:30seems to actually be exacerbating the negative outcomes that we're seeing.
- 32:34Soitactuallyseemstobemakingthingsliketheriskofharmtoselforothers
- 32:40SPEAKER_01: increase, not decrease.
- 32:41And just to be clear,
- 32:42this is for people who are in that kind of gray area category.
- 32:46SPEAKER_00: One clinician might commit, one clinician might not.
- 32:48SPEAKER_01: That's right.
- 32:48Soyoucanthinkaboutthesetofclientswhoarecomingtotheattentionofan
- 32:53SPEAKER_01: involuntary commitment petition as sort of falling into three buckets.
- 32:57SPEAKER_01: One is people for sure that you would never commit because they're fine.
- 33:03SPEAKER_01: You have people who are incredibly severe.
- 33:05And so no clinician is ever going to think that it would be a good idea not to
- 33:09hospitalize them.
- 33:10Andthenyouhaveagroupinthemiddlewherethere'sgenuineuncertaintyand
- 33:14SPEAKER_01: different clinicians might feel differently.
- 33:16SPEAKER_01: And that's really the group that the team has been studying.
- 33:19SPEAKER_01: And for that group,
- 33:21the way that involuntary hospitalization has been performed thus far does
- 33:25notseem
- 33:25to be benefiting clients in terms of health outcomes and criminal justice involvement.
- 33:31SPEAKER_01: And that's incredibly,
- 33:32SPEAKER_01: I think,
- 33:33important because then it poses a bunch of questions to us about how we can do
- 33:36this
- 33:37SPEAKER_01: work better.
- 33:39SPEAKER_00: Can you give me a sense of the scale?
- 33:41SPEAKER_00: If I see a clinician about whether I should be involuntarily committed,
- 33:45SPEAKER_00: before you know anything about me,
- 33:46SPEAKER_00: what are my odds of being committed?
- 33:49SPEAKER_01: So on average, the upheld rate for a petition is about 80%.
- 33:53But it varies significantly across the types of clinicians and the type of
- 33:59people
- 33:59SPEAKER_01: who could be making that determination.
- 34:02SPEAKER_01: So for example, psychiatrists tend to uphold the
- 34:06SPEAKER_01: at a much lower rate.
- 34:07SPEAKER_01: So that might make sense.
- 34:08Theymightbemuchmorecomfortableintermsofthetypesofrisksthatget
- 34:12presented to them of distinguishing between what's a substance use problem
- 34:15and
- 34:16SPEAKER_01: what's serious mental illness,
- 34:18whereas others who might not have that training might feel less comfortable.
- 34:21And so they might just uphold more of the petitions that they're seeing in a
- 34:25given year.
- 34:26SPEAKER_00: What other kinds of clinicians might see me?
- 34:28SPEAKER_00: A psychiatrist or...
- 34:30SPEAKER_01: Like an emergency room doctor,
- 34:31SPEAKER_01: for example,
- 34:32SPEAKER_01: who might not have specific psychiatric training,
- 34:34SPEAKER_01: but who could be a site that actually sees the petition.
- 34:38SPEAKER_00: Got it.
- 34:39SPEAKER_00: This 80-20 rule seems to apply in so many places.
- 34:42SPEAKER_00: I mean,
- 34:43your colleague Pim has a thread talking about how these people who are
- 34:46getting
- 34:47SPEAKER_00: involuntarily committed make up a tiny fraction of Medicaid enrollees,
- 34:52SPEAKER_00: but they account for 25% of behavioral health spending.
- 34:55SPEAKER_01: Well,
- 34:55and it's fascinating because I think every jurisdiction in the country
- 34:58wantsto
- 34:58SPEAKER_01: know this frequent utilizer problem.
- 35:00SPEAKER_01: And the way that they try to do that, to me, is very top down.
- 35:03SPEAKER_01: They try to sort of say,
- 35:05whatarethesortofthepatternsofusewhereyou'refrequentlycominginto
- 35:10SPEAKER_01: contact with the emergency room or whatever?
- 35:12This was a very sort of organic way of coming across that group where we said,
- 35:16SPEAKER_01: hey,
- 35:17for the 5,000 people in Allegheny County who are being involuntarily committed,
- 35:21SPEAKER_01: they represent about 2% of our Medicaid population.
- 35:26SPEAKER_01: They represent a quarter of our behavioral health spending on Medicaid.
- 35:31And so if you really want to bend the healthcare cost curve in behavioral health,
- 35:35you really have to figure out better solutions to that group in addition to
- 35:38allof
- 35:39the human welfare concerns that you might have just for the fact of their
- 35:41mortality
- 35:42SPEAKER_01: and their criminal justice involvement,
- 35:44SPEAKER_01: for example.
- 35:45I'm not going to ask you to solve the mental health problems in the country or
- 35:48even
- 35:48SPEAKER_00: in the county entirely,
- 35:49SPEAKER_00: right?
- 35:50SPEAKER_00: But what kinds of solutions do you look at?
- 35:52SPEAKER_00: Is the population,
- 35:52SPEAKER_00: like you said,
- 35:53it's like about 5,000 people who are soaking up a ton of the care that you guys provide.
- 35:57SPEAKER_00: How do you think about that population?
- 35:59SPEAKER_01: Well,
- 35:59and this is where I think that the example of going from problem identification,
- 36:03SPEAKER_01: description,
- 36:04causal mechanisms to solution is super powerful because now you're
- 36:08startingtoput
- 36:08SPEAKER_01: all the pieces together.
- 36:09SPEAKER_01: You're starting to say, this group is incredibly high acuity.
- 36:13The current way that we're doing in voluntary hospitalization doesn't seem
- 36:15tobe
- 36:16SPEAKER_01: that beneficial.
- 36:17SPEAKER_01: What can we do about it?
- 36:19And so to the team's credit,
- 36:20wejuststarteddigginginandaskingsomequestionsabout
- 36:24SPEAKER_01: What could be going on?
- 36:27Onethingthatyouseeinthedatathatseemstobeimportanthereisthat
- 36:31adherence to medication for individuals with bipolar disorder or schizophrenia,
- 36:35SPEAKER_01: for example,
- 36:35SPEAKER_01: which is
- 36:37SPEAKER_01: keystone to good care here is just incredibly low.
- 36:41SPEAKER_01: It's about 20%, right?
- 36:42SPEAKER_01: It's about 20%, both before and after hospitalization.
- 36:46And so that's a really good indicator that these people aren't getting
- 36:52connectedto
- 36:52SPEAKER_01: care effectively either before or after.
- 36:56SPEAKER_01: And so
- 36:57It presents a challenge to us in terms of what are the different mechanisms
- 37:01that
- 37:01SPEAKER_01: could lead to higher adoption of behavioral health services.
- 37:04One thing that was really interesting at the time is we were reading through
- 37:08the
- 37:09SPEAKER_01: sort of literature and thinking about what could be effective here.
- 37:12SPEAKER_01: And the literature around medication adherence is incredibly bleak.
- 37:15SPEAKER_01: It's all like, hey, this is very context specific.
- 37:19SPEAKER_01: It works in this developing country.
- 37:21SPEAKER_01: It doesn't work so well here in this richer country.
- 37:23But one thing that seems to be robust across jurisdictions is this idea of
- 37:28using
- 37:28SPEAKER_01: financial incentives to encourage the adoption of different behaviors.
- 37:34SPEAKER_01: And so here,
- 37:35what we're thinking about is the fact that you can imagine if you're already
- 37:39ina
- 37:40SPEAKER_01: situation where your health is deteriorated,
- 37:42SPEAKER_01: maybe taking a daily medication is incredibly difficult.
- 37:46But what if I could give you a once monthly injectable form of that medication
- 37:51that
- 37:51SPEAKER_01: makes you not have to do something 365 times,
- 37:53SPEAKER_01: but just makes you do something 12 times?
- 37:56And what if I can tie financial incentives to that so that I can encourage you
- 37:59to show up?
- 38:00And what's really interesting when you talk to individuals with
- 38:03schizoaffective disorder,
- 38:04SPEAKER_01: for example,
- 38:05Ihaddonethiswithafewofourpeersupportspecialiststhatworkatour
- 38:10SPEAKER_01: psychiatric hospital down the street.
- 38:12SPEAKER_01: And it's amazing to me how many of them said to me something like,
- 38:16SPEAKER_01: hey,
- 38:16it took me eight years and three involuntary hospitalizations to realize
- 38:20thatit
- 38:20SPEAKER_01: was really important for me to keep on my medication.
- 38:23SPEAKER_01: And even if I was going to gain 50 pounds of weight,
- 38:26SPEAKER_01: for example,
- 38:26SPEAKER_01: which is the side effect that sometimes comes with the medications,
- 38:29SPEAKER_01: I had to make the adult decision.
- 38:30SPEAKER_01: do that.
- 38:31And I think for us,
- 38:33the call to arms was really,
- 38:35how do we speed up that cycle?
- 38:37Because there is some sort of utilitarian cost benefit calculus that's
- 38:40going on there.
- 38:41SPEAKER_01: So anything that we can do to shorten that gap and to make it to say,
- 38:46SPEAKER_01: hey,
- 38:46SPEAKER_01: look,
- 38:47SPEAKER_01: this is a non-pecuniary cost to you.
- 38:49SPEAKER_01: It's costly to show up to an appointment.
- 38:51SPEAKER_01: It's costly to take on the risks of side effects.
- 38:54SPEAKER_01: It's painful to get a shot sometimes.
- 38:56SPEAKER_01: It's not the thing that I think anybody looks forward to doing.
- 39:00SPEAKER_01: But it's super important, we hypothesize, to the stability of their care.
- 39:05SPEAKER_01: And so what we're doing right now,
- 39:07SPEAKER_01: we're funded generously through Stanford Impact Labs,
- 39:09SPEAKER_01: where we're now getting to run that in a randomized control trial.
- 39:13SPEAKER_01: And it'll be the first test of that type in the U.S.
- 39:16SPEAKER_01: And so coming out of it,
- 39:17we'll really understand how do people respond to financial incentives in
- 39:20termsof
- 39:20SPEAKER_01: medication adherence.
- 39:23And then how does medication adherence then translate into better outcomes
- 39:27interms
- 39:27SPEAKER_01: of mortality,
- 39:28SPEAKER_01: lower utilization of behavioral health system,
- 39:31SPEAKER_01: better labor market outcomes,
- 39:33SPEAKER_01: and so on?
- 39:34SPEAKER_00: What would constitute success for you?
- 39:36What's the percentage of people taking their medication that would be like,
- 39:40SPEAKER_00: wow,
- 39:40SPEAKER_00: this is amazing?
- 39:41Because I'm guessing that we're not getting 100% adherence to medication no
- 39:45matter
- 39:45SPEAKER_00: what financial incentives you tie to this book.
- 39:48SPEAKER_01: No, I think that's an amazing question.
- 39:51SowhatmostoftheworkthathadbeengoneoninthisspaceinEuropehadsortof
- 39:57found treatment effects on the order of 10 to 15 percentage points as far as adherence.
- 40:04SPEAKER_01: We think that we might be starting with a more acute population.
- 40:07SPEAKER_01: And so we're hopeful that these things could be even bigger.
- 40:11So I don't know,
- 40:12something like doubling the amount of people would be great.
- 40:16SPEAKER_01: But I think this is the sort of problem where
- 40:21SPEAKER_01: you have to pick the low-hanging fruit.
- 40:23SPEAKER_01: And to me, paying people is a low-hanging fruit.
- 40:25And then there's people who yet still are resistant or for whom this
- 40:29treatment
- 40:29SPEAKER_01: doesn't work super well.
- 40:31SPEAKER_01: I don't think you then therefore give up on them.
- 40:34But I also don't think that you're totally discouraged by the treatment if
- 40:36that's
- 40:37SPEAKER_01: what the incentives sort of show.
- 40:38SPEAKER_01: I think you just sort of have to continue to iterate.
- 40:41SPEAKER_01: And this is where I think rapid iteration cycles in government
- 40:45SPEAKER_01: are sort of super critical.
- 40:47SPEAKER_01: You can't just be on this timeline of like,
- 40:48SPEAKER_01: I'm going to try one thing and then pack up and go home.
- 40:51SPEAKER_01: It has to be sort of a way that you stack wins on top of each other.
- 40:55SPEAKER_00: Yesterday, I spoke to an expert on policing.
- 40:59I'mnotgoingtosayhisnamebecauseIthinkwe'llpublishthisconversation
- 41:02SPEAKER_00: before we publish his.
- 41:04But we were talking about a version of this 80-20 problem where Jordan Neely
- 41:09was
- 41:09SPEAKER_00: killed on the subway last year,
- 41:11SPEAKER_00: last year,
- 41:11SPEAKER_00: I believe,
- 41:12SPEAKER_00: in New York,
- 41:13SPEAKER_00: had been on the city's 50 people most in need of help for a long time.
- 41:18SPEAKER_00: And it just kind of highlighted the...
- 41:21SPEAKER_00: there's a specific population of people and often you can really ID,
- 41:24SPEAKER_00: like it's these 5,000 people in the case that you're talking about,
- 41:28SPEAKER_00: or it's these 50 people.
- 41:29SPEAKER_00: And then the challenge is just like,
- 41:31how do you deliver services or help those folks or make sure they don't hurt
- 41:35other people?
- 41:36But it's like a remarkably small population that you really need to focus on.
- 41:40SPEAKER_00: And as you're saying,
- 41:41ifyoucangeta10to15%improvementoradoublingclosetohalfofthosefolks
- 41:45SPEAKER_00: are now not off their meds all the time,
- 41:48SPEAKER_00: that's a big win.
- 41:50SPEAKER_01: That's right.
- 41:51And I think some of it is going to be that you have to pick the low-hanging fruit,
- 41:56andsomeofitisgoingtobethatyouhavetothinkaboutothermechanismsto
- 42:01SPEAKER_01: encourage adoption of care.
- 42:01SPEAKER_01: So,
- 42:02SPEAKER_01: for example,
- 42:02one of the things that we're doing in Allegheny County is opting into the
- 42:05state's
- 42:06SPEAKER_01: assisted outpatient treatment law,
- 42:08whichisawaytogetindividualswhoaredeterioratingintheirconditionin
- 42:14SPEAKER_01: community
- 42:15SPEAKER_01: into care without needing to be involuntarily hospitalized first.
- 42:19So much of the way the behavioral treatment system often sort of takes action
- 42:23is
- 42:24SPEAKER_01: once people have absolutely hit bottom.
- 42:26So think about somebody with substance use disorder who is seeking
- 42:30treatmentfor
- 42:31SPEAKER_01: withdrawal management,
- 42:32SPEAKER_01: for example,
- 42:32SPEAKER_01: or who has lost their family,
- 42:33SPEAKER_01: lost their job,
- 42:34SPEAKER_01: and then they're willing to go into treatment.
- 42:36A similar thing happens with serious mental illness where people
- 42:39deterioratetothe
- 42:40SPEAKER_01: point where they need to be involuntarily hospitalized.
- 42:42What if we could go upstream and offer support and care to individuals before
- 42:46that
- 42:47becomes an issue and leverage some potential legal options to assist
- 42:52individualsto
- 42:52SPEAKER_01: really get treatment and stick on this?
- 42:54SPEAKER_01: And New York City has been very innovative in this space.
- 42:58SPEAKER_01: And I think it's something we have to continue to invest in and monitor.
- 43:01SPEAKER_00: The challenge for you,
- 43:03SPEAKER_00: or one of the challenges at least,
- 43:04SPEAKER_00: is what sociologist Peter Rossi called the iron law of evaluation.
- 43:08There's this famous 1987 paper talking about the fact that social programs
- 43:14thatare
- 43:15SPEAKER_00: rigorously studied generally have disappointing results.
- 43:17The iron law is something like the expected value of any net impact
- 43:22assessmentof
- 43:22SPEAKER_00: any large-scale social program is zero.
- 43:25How do you think about that reality that it's really hard to build really good,
- 43:29SPEAKER_00: effective,
- 43:29SPEAKER_00: measurable social programs?
- 43:32SPEAKER_01: Yeah.
- 43:32SPEAKER_01: I mean,
- 43:33one of my takeaways from working in human services for a few years has just
- 43:37been
- 43:37that if we took a little index card and we wrote down everything that we know
- 43:40works
- 43:40SPEAKER_01: really well,
- 43:41SPEAKER_01: we'd have a whole heck of a lot of white space.
- 43:43SPEAKER_01: So I think some people look at that and they are very discouraged by it.
- 43:47SPEAKER_01: And I look at it,
- 43:48SPEAKER_01: especially in Allegheny County,
- 43:49SPEAKER_01: with the leadership we have,
- 43:50SPEAKER_01: with the data we have,
- 43:51SPEAKER_01: with the partners in the community that we have.
- 43:53SPEAKER_01: And I say,
- 43:54whycan'twebethecountry'sleadingresearchanddevelopmentlabforlocal
- 43:57SPEAKER_01: government that allows us to really
- 44:00SPEAKER_01: quickly build evidence on things.
- 44:02I think part of what that's getting to is that it's really actually quite
- 44:07difficult
- 44:07SPEAKER_01: to even stand up things to test.
- 44:09SPEAKER_01: It takes quite a long time.
- 44:11SPEAKER_01: It's hard to do it at significant scale and efficacy.
- 44:14SPEAKER_01: It's hard to find the right implementation partners.
- 44:17But I think that we really have a world-class opportunity here in Pittsburgh
- 44:21to do that.
- 44:22And so the work that we're talking about with serious mental illness is one of them.
- 44:26We'vegotworkinthepipelinearoundaddictionwhereIthinkthatthere's
- 44:30SPEAKER_01: opportunities for us to innovate both in terms of
- 44:33the medications that are being used for individuals with addictive diseases,
- 44:37SPEAKER_01: as well as the way that we deliver those services.
- 44:40SPEAKER_01: So I'm certainly not discouraged by that.
- 44:42SPEAKER_01: I look at it and I say, that's a great challenge.
- 44:45SPEAKER_01: And it doesn't surprise me too.
- 44:46SPEAKER_01: I think you have to really start digging.
- 44:48But I think all those failures also really help us to get closer to the
- 44:52solutionby
- 44:53SPEAKER_01: ruling out all the things that we know can't plausibly be that important.
- 44:57The Department of Human Services tackles a lot of these kind of famously
- 45:01difficult
- 45:02SPEAKER_00: to deal with social problems.
- 45:03I'm curious just from your personal experience or your read of the things
- 45:08thatyou
- 45:08SPEAKER_00: guys tackle,
- 45:09SPEAKER_00: getting people into housing,
- 45:10SPEAKER_00: mental health,
- 45:11SPEAKER_00: child welfare,
- 45:12SPEAKER_00: opiate addiction,
- 45:13SPEAKER_00: other substance abuse disorders.
- 45:15Give me kind of a spectrum of which ones you think are the most tractable and
- 45:18which
- 45:19SPEAKER_00: ones you're like,
- 45:20SPEAKER_00: this is even harder than the other really hard stuff.
- 45:24I love working in human services because we really do work on some of the
- 45:29hardest
- 45:29SPEAKER_01: problems on earth.
- 45:30SPEAKER_01: I mean, they are incredibly difficult.
- 45:33SPEAKER_01: Community violence is difficult.
- 45:34SPEAKER_01: Child-bound treatment is difficult.
- 45:36SPEAKER_01: Addiction is difficult.
- 45:37SPEAKER_01: So it's hard to give a rank ordering among the impossible.
- 45:43SPEAKER_01: I will say that the thing...
- 45:45SPEAKER_01: I have been,
- 45:47SPEAKER_01: I think,
- 45:47particularly driven by increasing the rate of innovation in behavioral
- 45:51healthin
- 45:52part because I think that there's a large number of individuals who are
- 45:56suffering
- 45:57SPEAKER_01: and there is such an opportunity here to do better.
- 46:03SPEAKER_01: And if I look at, for example...
- 46:05SPEAKER_01: the rate of approval for novel drugs to treat addiction, for example.
- 46:13It's been 20 years since we've had a new drug approved for the treatment of
- 46:17alcohol
- 46:17SPEAKER_01: use disorder.
- 46:18It's been 20 years since a new drug has been approved for opioid use disorder.
- 46:25There still is no FDA approved medication for individuals who are addicted
- 46:29to stimulants.
- 46:30SPEAKER_01: And so all of the work that has been done more recently on this is,
- 46:35SPEAKER_01: to me,
- 46:36SPEAKER_01: felt maybe important,
- 46:38SPEAKER_01: but incremental.
- 46:39SPEAKER_01: So things like injectable formulations of existing medications.
- 46:43SPEAKER_01: But in order to get out of the situation that we're in,
- 46:46I think we're going to have to innovate on treatment efficacy and treatment delivery,
- 46:51SPEAKER_01: or we're at risk of just watching the addiction problem burn out here.
- 46:56And I don't think that that's something that I or anybody else who's working
- 47:00here
- 47:00or any other government agency or any other concerned citizen really wants
- 47:04to see.
- 47:06SPEAKER_01: I think that there's an opportunity really to do much better.
- 47:09SPEAKER_00: And just to define terms for folks,
- 47:12SPEAKER_00: the addiction problem to burnout,
- 47:13SPEAKER_00: for that to happen,
- 47:14that means enough addicts die of their addiction that it's no longer a
- 47:19problemthat
- 47:19SPEAKER_00: shows up on our members.
- 47:21SPEAKER_01: Right.
- 47:21And I sort of think about this as what's happened in this country with alcohol
- 47:26use disorder,
- 47:26where it's sort of a background hum of suffering and despair that has
- 47:33tremendous
- 47:33SPEAKER_01: cost to people and to society and to kids and
- 47:38SPEAKER_01: on all sorts of levels.
- 47:40SPEAKER_01: And we haven't raised the federal alcohol tax in three decades.
- 47:46SPEAKER_01: Not a whole lot of policy innovation.
- 47:47SPEAKER_01: There's no drug innovation here.
- 47:49SPEAKER_01: And I worry deeply about capitulation for opioids,
- 47:53SPEAKER_01: even though it's been such a massive challenge.
- 47:56SPEAKER_01: And you're seeing,
- 47:58SPEAKER_01: I think,
- 47:58some worrying signs that maybe this cycle is repeating in maybe less fatal ways,
- 48:03SPEAKER_01: but with sports gambling.
- 48:04SPEAKER_01: And so I think that those are all really concerning areas.
- 48:08OneotherthingI'llsayaboutbehavioralinnovationthatIthinkissuper
- 48:12SPEAKER_01: important here is,
- 48:13SPEAKER_01: you know,
- 48:13SPEAKER_01: the National Institutes of Health funded a $350 million,
- 48:17SPEAKER_01: four-state,
- 48:1967-county randomized controlled trial called the Communities That Heal project.
- 48:23And this was using the best evidence-based strategies that the behavioral
- 48:27health
- 48:27SPEAKER_01: experts think about when they think about combating opioid use disorder.
- 48:32SPEAKER_01: and they found no results.
- 48:33So that is,
- 48:34they didn't find strong enough results that were distinguishable from zero.
- 48:40SPEAKER_01: And even if you take their point estimate at face value,
- 48:43so even if you ignore the statistical uncertainty and just take that number
- 48:47of face value,
- 48:48it suggests something on the order of a 10 to 15% drop in opioid overdose deaths.
- 48:54SPEAKER_01: So
- 48:55SPEAKER_01: a win, we would take that.
- 48:57But certainly when you're thinking about really making a big impact on
- 49:03opioid
- 49:03SPEAKER_01: overdose deaths,
- 49:05it's just clear that the strategies that we have been pursuing aren't good enough.
- 49:10SPEAKER_00: Right.
- 49:11You've been advising an organization,
- 49:12Casper,
- 49:13weco-publishedsomethingwiththematour
- 49:15SPEAKER_00: home institution, IFP, on an innovation agenda for addiction.
- 49:19SPEAKER_00: And I just want to flag a couple of projects.
- 49:21There's a phase three clinical trial testing semaglutide on alcohol use
- 49:25disorder with the VA.
- 49:27SPEAKER_00: And there's an experiment in Pittsburgh.
- 49:28SPEAKER_00: Maybe you want to tell us a little bit about that one.
- 49:30SPEAKER_01: Yeah,
- 49:31so I've been really fortunate to partner with Nick Reville and his organization,
- 49:35SPEAKER_01: CASPER,
- 49:35SPEAKER_01: which is the Center for Addiction Science Policy and Research.
- 49:39And he and his co-founder,
- 49:42Lindsay Holden,
- 49:43have been total forces of nature here.
- 49:46SPEAKER_01: So he wrote a really awesome substack post called Addiction Needs,
- 49:51SPEAKER_01: and I think what was the title of it?
- 49:52SPEAKER_01: It's something like, we need an Operation Warp Speed, but for addiction.
- 49:57And it was one of those things where like the magic of the internet comes
- 50:00together
- 50:00SPEAKER_01: when you read it and you say,
- 50:01SPEAKER_01: man,
- 50:02SPEAKER_01: this is so deeply resonant with me.
- 50:03And I think he was sort of really responding to some of the nascent evidence
- 50:06around
- 50:07SPEAKER_01: the so-called GLP-1 drugs for addiction.
- 50:10SPEAKER_01: And so for people at home, the GLP-1 drugs are
- 50:14Things that become blockbuster hits such as Ozempic and Wagovi and ZepBound.
- 50:20Sothingsthathavebeenfoundtoreallycurbfoodnoiseforpeoplehavealso
- 50:25started to show signs that they could be curbing other sides of craving and addiction.
- 50:30SPEAKER_01: And that's incredibly important.
- 50:32SPEAKER_01: And so what I've been working with them on is exactly as you said,
- 50:36sort of the traditional phase three clinical trial that would happen in the
- 50:40VA
- 50:40SPEAKER_01: health system.
- 50:42And then more of a public health focused trial that would focus on
- 50:47introducingthis
- 50:47SPEAKER_01: drug for people in high acuity states.
- 50:50So think about individuals after they leave the jail with alcohol use disorder.
- 50:54SPEAKER_01: This group has a two and five risk of going back to jail in a year.
- 50:58SPEAKER_01: They're really having poor health outcomes.
- 51:01SPEAKER_01: Addiction is a major driver of criminal justice involvement.
- 51:04SPEAKER_01: If these things could be effective for that type of a population,
- 51:09SPEAKER_01: that could be a total game changer for society,
- 51:12SPEAKER_01: I think.
- 51:12SPEAKER_01: And that's exactly what we want to go after.
- 51:15So I think reasonable people can disagree in terms of their view on whether
- 51:19that
- 51:19SPEAKER_01: would happen or not.
- 51:20SPEAKER_01: But in my view,
- 51:23what's important is that we just go really fast and try to seize the
- 51:26opportunityas
- 51:27SPEAKER_01: quickly as we can.
- 51:28SPEAKER_01: And I think Pittsburgh,
- 51:29SPEAKER_01: with our unique healthcare resources,
- 51:31SPEAKER_01: plus the ability to really weave in administrative outcomes,
- 51:35SPEAKER_01: something that's not really ever done in these sorts of clinical trials,
- 51:38SPEAKER_01: could be incredibly important.
- 51:40So we're in the middle of talking to some great philanthropic partners who
- 51:43are
- 51:43SPEAKER_01: interested in supporting us on that,
- 51:45SPEAKER_01: and we're hopeful about getting that off the ground this year.
- 51:49There's obviously been a big federal level push for government efficiency
- 51:52inthe
- 51:52SPEAKER_00: news these past few weeks.
- 51:54AndIwantedtoaskyouabouthowyouguyshavethoughtaboutthecountylevel
- 51:57SPEAKER_00: version of this problem in Allegheny.
- 51:59SPEAKER_00: The Department of Human Services runs the mental health housing system.
- 52:03And one of your colleagues wrote about how a few years ago was providing
- 52:07housing
- 52:08SPEAKER_00: for around 500 people at a total annual cost of over 40 million.
- 52:11SPEAKER_00: So that's
- 52:12SPEAKER_00: over $80,000 a person per year for housing alone.
- 52:15And this is for people with serious mental health issues who are also getting
- 52:18often
- 52:19SPEAKER_00: county funded treatment or other help from the county.
- 52:23SPEAKER_00: It's obviously a crazy number.
- 52:24How did you guys think about bringing that down or delivering services for a
- 52:29tenable cost?
- 52:31SPEAKER_01: Well,
- 52:31Ithinkonethingthat'strueaboutthedepartmentisthatwealwayswanttobe
- 52:36scrutinizing the way that we're using our funding to make sure that it's
- 52:39goingto
- 52:39SPEAKER_01: the highest value uses.
- 52:41And one thing that can sometimes be true is that how services are priced over
- 52:47time
- 52:47SPEAKER_01: just needs to be constantly revisited.
- 52:49SPEAKER_01: Is this in line with actual costs?
- 52:51Are there ways to actually gain efficiencies by using Medicaid fee for services,
- 52:55for example?
- 52:56And this is one of those areas where I think because of the budget constraints
- 53:00that
- 53:00SPEAKER_01: we face for those funding sources,
- 53:02wejusthadahighdegreeofinterestinmakingsurethatwewereusingthat
- 53:07SPEAKER_01: funding through its highest value use cases.
- 53:09SPEAKER_01: And so this is an initiative that's been spearheaded by our director,
- 53:12SPEAKER_01: Aaron Dalton,
- 53:13really sort of thinking about what makes sense here as far as right sizing
- 53:17these contracts.
- 53:18How do we make sure that these individuals are receiving the right mental
- 53:24health care,
- 53:24SPEAKER_01: that they're getting supportive services?
- 53:26SPEAKER_01: that the staffing levels are right.
- 53:29ButthosesortsofdecisionssometimesIthinkcomewithshort-termpainto
- 53:34SPEAKER_01: providers as changes are made.
- 53:37But I think thinking about pricing efficiencies more broadly here is
- 53:40somethingthat
- 53:40SPEAKER_01: we're incredibly interested in doing.
- 53:43SPEAKER_01: you do see really interesting incentives in some of these systems.
- 53:46SPEAKER_01: So for example,
- 53:47SPEAKER_01: if you have a homogenous flat pricing for a given service,
- 53:52SPEAKER_01: but people differ in terms of their cost to serve,
- 53:55then you start to give providers incredibly strong incentives to take the
- 54:00lowest risk,
- 54:01SPEAKER_01: least costly to serve individuals.
- 54:04SPEAKER_01: Um,
- 54:04And those aren't things that people do because they're nefarious or because
- 54:07they're
- 54:07SPEAKER_01: trying to gain the system.
- 54:08I think that they sort of organically happen over time if they're not kind of
- 54:12managed well.
- 54:13And that's where things like algorithms and type management really come
- 54:17into play,
- 54:18I think.
- 54:18SPEAKER_00: Alex,
- 54:19SPEAKER_00: you came into public service from the outside,
- 54:21SPEAKER_00: from the tech industry,
- 54:23and you came into a department that's already one of the best in the country,
- 54:27SPEAKER_00: I would say,
- 54:27from a data management perspective and thinking from first principles
- 54:31abouthowyou
- 54:32SPEAKER_00: provide these services.
- 54:34SPEAKER_00: I'm curious, what's the secret sauce?
- 54:36SPEAKER_00: Tyler Cowen would ask,
- 54:37what's the Allegheny County Department of Human Services production function?
- 54:41SPEAKER_00: How does it generate its results?
- 54:44SPEAKER_01: That's right.
- 54:45I think it's been a combination of great infrastructure and partners and
- 54:50really
- 54:51SPEAKER_01: strong culture,
- 54:52SPEAKER_01: especially at the leadership level.
- 54:53SPEAKER_01: So you can have the best data in the world and really great talent,
- 54:57SPEAKER_01: but if your leadership isn't willing to take some risks,
- 55:00if you're not willing to be courageous in certain settings where things
- 55:03mightnot
- 55:04SPEAKER_01: be politically popular,
- 55:06SPEAKER_01: you won't make as much progress.
- 55:07And I think the department through Aaron Dalton now and Mark Turner then has
- 55:12really
- 55:14SPEAKER_01: not just sort of thought about data in smart ways,
- 55:17SPEAKER_01: but really thought about it in bold ways.
- 55:20SPEAKER_01: So that's been incredibly empowering.
- 55:22And when I came into the role,
- 55:23I said,
- 55:24well,
- 55:24you guys have done this so well.
- 55:26SPEAKER_01: Why not double down on it, triple down on it?
- 55:28SPEAKER_01: Why rely on anybody externally?
- 55:31Just hire world-class researchers,
- 55:32world-class engineers to come work for you.
- 55:36And so I've been blessed to sort of get that buy-in and start to build out the team.
- 55:40And Pim on the data science side has done an awesome job of taking that baton
- 55:45and
- 55:45SPEAKER_01: building out a machine learning team plus economist.
- 55:48SPEAKER_01: And they're really sort of speeding up the research production function.
- 55:52SPEAKER_01: And then his counterpart,
- 55:54SPEAKER_01: Rachel Silver,
- 55:54SPEAKER_01: on the technology side has built out an engineering team.
- 55:58Andthisisallowingustodothingslikesavecostsfromvendorswhilealso
- 56:02SPEAKER_01: building high quality software.
- 56:04So I think we'll always have this important relationship with external
- 56:08academics
- 56:08SPEAKER_01: and external vendors in certain cases.
- 56:11SPEAKER_01: But what we're finding is that we're really loving
- 56:14building on top of DHS's incredibly rich legacy with internal state capacity.
- 56:19SPEAKER_01: And I think just as Jen Palka has highlighted at the federal level,
- 56:23SPEAKER_01: these things matter a great deal at the state and local level.
- 56:26There's this concept for Justice Brandeis that state and local government
- 56:29isa
- 56:29SPEAKER_01: laboratory of democracy.
- 56:31And this is where we can try and iterate on a whole lot of really important
- 56:34social problems.
- 56:35SPEAKER_01: And so having that state capacity,
- 56:36SPEAKER_01: I think,
- 56:37SPEAKER_01: to actually execute on really important problems matters a great deal.
- 56:41Let me close with something you mentioned just before we started recording,
- 56:44which was just that based on when predictive algorithms were introduced in
- 56:49these
- 56:49SPEAKER_00: contexts in the early to mid 2010s,
- 56:54a neutral observer might have expected that they'd be far more common in all
- 56:58kinds
- 56:58SPEAKER_00: of sectors.
- 56:59SPEAKER_00: Now,
- 56:59because you've had fantastic results in outcomes for kids and all these
- 57:02obvious
- 57:03SPEAKER_00: clear wins where you're in some cases saving the lives of children,
- 57:06SPEAKER_00: right?
- 57:06SPEAKER_00: So
- 57:07What's your read on why we don't see more adoption at the federal or the state
- 57:12and local level?
- 57:13SPEAKER_00: And are you optimistic over the next 10 years that we'll get more?
- 57:17People often reach out to Allegheny County Department of Human Services
- 57:21because
- 57:21SPEAKER_01: they want to talk about integrated data.
- 57:23SPEAKER_01: And they want to ask my boss,
- 57:25SPEAKER_01: Erin,
- 57:25about how she built the data warehouse and how they pulled off all of these things.
- 57:29And what's so interesting to me is that very quickly people get into the
- 57:32technical
- 57:32details of how that's gone without asking what I think is a more fundamental question,
- 57:37SPEAKER_01: which is,
- 57:38SPEAKER_01: what is it about her and the prior department leadership
- 57:41SPEAKER_01: that's really enabled this to flourish.
- 57:44And so I think some of it is about the uniqueness of the data warehouse and
- 57:48having
- 57:48SPEAKER_01: administrative data.
- 57:49SPEAKER_01: Frankly, I think that that's often a poor excuse, though.
- 57:52I think many jurisdictions could build totally capable data science and
- 57:57algorithmic
- 57:57SPEAKER_01: operations based off of the resources that they have.
- 58:01and sometimes i think that there could be technical skill gaps there but i
- 58:05think
- 58:05oftentimes these things come down to culture and desire and we just have not
- 58:11had
- 58:11that at a large scale and i think we're always interested here the department
- 58:16of
- 58:16thinking about ways that we could be part of a movement to do things that are
- 58:19not
- 58:20just relevant to our clients here but to solving real sort of national
- 58:23problems
- 58:24SPEAKER_01: including capacity building
- 58:27SPEAKER_00: I think the capacity building you guys are doing is great.
- 58:29SPEAKER_00: Alex, thank you for joining.
- 58:31SPEAKER_01: Thanks for having me.