Latest / 7am / The aged care algorithm replacing human judgement
Transcript
- 0:01Speaker 1: I'm Daniel James, and you're listening to 7am. There can
- 0:08Speaker 1: come a point in growing older when staying in the
- 0:11Speaker 1: home you love depends on someone helping you out of bed,
- 0:13Speaker 1: making a meal, or making sure you're taking your medication.
- 0:17Speaker 1: That care can mean holding on to your independence, to
- 0:20Speaker 1: familiar surroundings, and to a life that still feels like
- 0:23Speaker 1: your own. But now, the government is using an algorithm
- 0:26Speaker 1: to determine home care funding, replacing assessor's judgement with a
- 0:30Speaker 1: formula They can overlook the complexities of someone's life, but
- 0:34Speaker 1: with no ability for humans to override the decision, even
- 0:38Speaker 1: when they know it's wrong. Today, aged care expert Professor
- 0:41Speaker 1: Cathy Eger on the human consequences of handing those decisions
- 0:44Speaker 1: to an algorithm and the people whose needs are not
- 0:47Speaker 1: being met as a result. It's Friday, October 30th. Cathy,
- 1:01Speaker 1: thanks for joining us. The government is using an algorithm
- 1:04Speaker 1: to determine how much funding older Australians receive for care
- 1:08Speaker 1: at home. You've managed to reverse engineer that algorithm to
- 1:12Speaker 1: understand how it makes those decisions. What did you discover?
- 1:17Speaker 2: What I discovered is two things, really. One is something
- 1:22Speaker 2: I already knew before I did the engineering bit, which
- 1:25Speaker 2: is we should be using these sort of tools to
- 1:29Speaker 2: as decision support tools, not as decision-making tools. And the
- 1:35Speaker 2: second is that the algorithm itself is fundamentally flawed from
- 1:40Speaker 2: a technical perspective because it was designed by people who
- 1:45Speaker 2: don't understand algorithms.
- 1:48Speaker 1: Right.
- 1:48Speaker 3: And what kind of decisions is the algorithm making? So
- 1:52Speaker 3: a clinical assessor assesses somebody. And in the past, the
- 1:56Speaker 3: clinical assessor said, well, I've assessed Kathy Eager.
- 1:58Speaker 2: This is what she needs. And these are the services
- 2:01Speaker 2: she's going to need. And the cost of those services
- 2:04Speaker 2: is X. The process now reverses that logic and it assesses,
- 2:10Speaker 2: this is what the assessor says you need. This is
- 2:15Speaker 2: how much the algorithm says that translate to in dollars.
- 2:19Speaker 2: And then the assessor has to work out what services
- 2:22Speaker 2: you can afford to from the dollars you've been given.
- 2:26Speaker 2: Things like whether the person has dementia, whether they live alone,
- 2:30Speaker 2: whether they're homeless and live in a car. None of
- 2:33Speaker 2: those things are taken into account. The other thing that's
- 2:37Speaker 2: missing is that it doesn't actually take into account whether
- 2:41Speaker 2: they have a carer, and if so, what their carer needs,
- 2:46Speaker 2: what support they need to continue being carers. There's thousands
- 2:51Speaker 2: of factors involved that an experienced clinical assessor will take
- 2:57Speaker 2: into account about a person that no algorithm is ever
- 3:01Speaker 2: going to be able to take into account.
- 3:04Speaker 1: So why did the government introduce this algorithm or these algorithms?
- 3:09Speaker 2: It's a very good question and it doesn't make any
- 3:12Speaker 2: sense to use an algorithm in aged care because aged
- 3:15Speaker 2: care is a care and support scheme. It's not a
- 3:18Speaker 2: financial entitlement scheme. In my view, the government's never been
- 3:23Speaker 2: transparent about why it introduced the algorithm and it said
- 3:27Speaker 2: things like, oh, assessors gave people too much before, which
- 3:31Speaker 2: is not true.
- 3:32Speaker 4: Under the previous system, we had $ 4 billion of value
- 3:37Speaker 4: allocated to people incorrectly and therefore unable to be deployed
- 3:42Speaker 4: to look after older people.
- 3:45Speaker 2: There's no evidence for the claims that they've given. I
- 3:49Speaker 2: think they've introduced the algorithm in support at home because
- 3:52Speaker 2: this is the prototype that they're intending for the National
- 3:56Speaker 2: Disability Insurance Scheme, which has been full of cost blowouts
- 4:01Speaker 2: and scams and all those sorts of things that we've
- 4:04Speaker 2: seen in the media. Aged care has not experienced those problems,
- 4:09Speaker 2: but the government has put the two together. The NDIS
- 4:13Speaker 2: is an insurance scheme. But the aged care system is
- 4:17Speaker 2: a care and support scheme, and I don't think the
- 4:19Speaker 2: government understands the difference between those two schemes.
- 4:23Speaker 1: An algorithm, of course, can't take into a factor the
- 4:27Speaker 1: important nuanced health information of individuals and support needs of
- 4:33Speaker 1: their family. How badly can that distort the funding someone receives?
- 4:39Speaker 2: It can distort the funding in two ways. Obviously... the
- 4:43Speaker 2: cases that are speaking out are the people who don't
- 4:45Speaker 2: get the funding that they need. But it's also the
- 4:49Speaker 2: case that some people are getting too much funding because
- 4:52Speaker 2: the algorithm doesn't take into account so many other factors.
- 4:57Speaker 2: So two people might look exactly the same in the algorithm,
- 5:02Speaker 2: but one person lives in a home they own that's
- 5:05Speaker 2: all on one level, that's got ramps and rails, that
- 5:09Speaker 2: has safety devices, et cetera. The other person lives on
- 5:13Speaker 2: the second floor of a block of flats that they
- 5:15Speaker 2: rent and they're not allowed to put a nail in
- 5:18Speaker 2: a wall. Even though they look the same on paper
- 5:21Speaker 2: or in the algorithm, their need for care is really
- 5:25Speaker 2: different and an assessor can take that into account and
- 5:29Speaker 2: an algorithm can never take that into account. We will now.
- 5:34Speaker 5: Recommence the discussion inquiry into support at home. And I
- 5:38Speaker 5: now welcome Professor Cathy Eager. Professor Eager, thank you so
- 5:43Speaker 5: much for joining us.
- 5:44Speaker 1: And Cathy, you presented to a Senate inquiry and demonstrated
- 5:48Speaker 1: that a single point on the algorithm scoring system could
- 5:51Speaker 1: mean the difference between someone getting $ 40, 000 or $ 80, 000 in
- 5:53Speaker 1: annual care funding.
- 5:57Speaker 2: And on the basis of a one-point difference in a
- 6:00Speaker 2: 48-point scale is a $ 40, 000 difference for the person.
- 6:05Speaker 1: How is that even possible?
- 6:07Speaker 2: Yes, it's crazy. This is due to the fundamental errors
- 6:11Speaker 2: in the design of the algorithm. If somebody gets a
- 6:14Speaker 2: score of 24 or less, they can actually be allocated
- 6:19Speaker 2: up to 80,000. But if they get a score of 25,
- 6:22Speaker 2: a one-point difference.
- 6:25Speaker 2: Then they might actually get 41,000. That's crazy. And it's
- 6:30Speaker 2: because of technical design flaws in the algorithm.
- 6:35Speaker 1: We'll get to those design flaws in a moment. But
- 6:38Speaker 1: our understanding is that the algorithm itself can't be overruled
- 6:42Speaker 1: by the person doing the assessment. Is that correct?
- 6:44Speaker 2: Yeah, it's true. So what happened was the people who
- 6:48Speaker 2: designed the system and the assessors were trained to believe
- 6:53Speaker 2: that that in the event of an error in the algorithm,
- 6:58Speaker 2: that the assessor can override the algorithm. And all the
- 7:03Speaker 2: manuals were said that, the training was done on that basis,
- 7:07Speaker 2: and the week before the system went live, everybody was told, well, actually, no,
- 7:12Speaker 2: assessors are not able to override the algorithm. So even
- 7:17Speaker 2: though the IT system allows for override, They are not
- 7:21Speaker 2: allowed to override it, even when they know that it's
- 7:27Speaker 2: resulting in huge fundamental errors.
- 7:30Speaker 1: Now, Cathy, you've designed healthcare funding algorithms yourself. How did
- 7:35Speaker 1: the government end up with a system that, as you say,
- 7:37Speaker 1: has such fundamental problems? Was it even tested before it
- 7:41Speaker 1: was rolled out?
- 7:43Speaker 2: The process was long and drawn out. And the first
- 7:46Speaker 2: part involved engaging consultants, a consultancy firm, not a research group,
- 7:52Speaker 2: but a consultancy firm, to design the assessment tool and
- 7:57Speaker 2: the algorithm. So there were two parts. One is design
- 8:01Speaker 2: what's called the integrated assessment tool that assessors use, and
- 8:06Speaker 2: then that's fed into the computer algorithm. It appears that
- 8:10Speaker 2: the public servants managing the project didn't like the algorithm
- 8:14Speaker 2: and that the consulting firm developed. So they then put
- 8:19Speaker 2: that aside and a group of them formed a little
- 8:22Speaker 2: working party and they designed the algorithm in-house.
- 8:25Speaker 1: Do we know how much expertise these public servants had
- 8:28Speaker 1: when they were redesigning this algorithm?
- 8:30Speaker 2: Yeah, well, the public servants are very good expert public servants,
- 8:34Speaker 2: but they're not clinicians and none of them have got
- 8:38Speaker 2: a background in algorithms or or in funding system design,
- 8:43Speaker 2: they are career public servants. That's what a public servant does.
- 8:48Speaker 2: So they were doing something well beyond the expertise of
- 8:54Speaker 2: a public servant. And it's not the fault of the individuals.
- 8:59Speaker 2: The public service should never have taken on that job.
- 9:03Speaker 2: It did not have the skills for it. And even now,
- 9:08Speaker 2: A year later, it still can't admit, the public service
- 9:12Speaker 2: can't admit, that they got it wrong.
- 9:18Speaker 1: Still to come, the damning new report taking the aged
- 9:21Speaker 1: care system to task.
- 9:32Speaker 5: So critically, we think that the legislation needs to be
- 9:35Speaker 5: amended to ensure that there can be a change to
- 9:38Speaker 5: an outcome if it's clinically not appropriate.
- 9:41Speaker 1: Cathy, this week the Commonwealth Ombudsman released a damning report
- 9:45Speaker 1: into the assessment system. He found a reasonable person would
- 9:48Speaker 1: lack confidence that the system was fair or consistent. Having
- 9:52Speaker 1: investigated this yourself, what do you make of those findings?
- 9:56Speaker 2: Oh, look, I'm delighted by the Ombudsman and what he's
- 9:59Speaker 2: come out and said in that review. The ombudsman didn't
- 10:05Speaker 2: look at the algorithm per se. He rightly said that
- 10:09Speaker 2: the ombudsman's office does not have the clinical expertise. So
- 10:14Speaker 2: the ombudsman recognised that you need clinical expertise even if
- 10:18Speaker 2: the department didn't. The ombudsman assessed how the system operated,
- 10:25Speaker 2: and the most important recommendation from my perspective is the
- 10:29Speaker 2: first one, which is that assessors should be allowed to
- 10:34Speaker 2: override the algorithm if the algorithm gets it wrong. The
- 10:39Speaker 2: ombudsman also made a really interesting recommendation. which is that
- 10:44Speaker 2: the department should undertake a root cause analysis to identify
- 10:49Speaker 2: how it is that it made so many mistakes and
- 10:52Speaker 2: where we've got such a flawed system. And that would
- 10:55Speaker 2: be extraordinarily valuable if the department does it, which it
- 11:00Speaker 2: has actually agreed to do. The last one is a
- 11:04Speaker 2: recommendation that the ombudsman made, which is that the Australian
- 11:08Speaker 2: National Audit Office should now do a performance audit of
- 11:13Speaker 2: the whole system and assess whether the department's use of
- 11:18Speaker 2: public resources was efficient, economical and ethical. And I'm taking
- 11:24Speaker 2: that to mean how is it that you could bring
- 11:28Speaker 2: in a consulting group, pay them millions, and then on
- 11:31Speaker 2: the most important bit, which is the algorithm, a group
- 11:34Speaker 2: of public servants completely ignored it and made up their own.
- 11:38Speaker 1: Yeah. The government has defended the algorithm, arguing that it's
- 11:42Speaker 1: delivering faster and more consistent assessments and that relatively few
- 11:46Speaker 1: people have challenged their results. What do you make of
- 11:49Speaker 1: that defence?
- 11:50Speaker 2: There are two parts to that. One is that it's
- 11:52Speaker 2: more consistent results. And what I said at the Senate
- 11:56Speaker 2: about this is very true. If you want to consistently
- 11:59Speaker 2: get things wrong, use an algorithm that's the wrong algorithm
- 12:03Speaker 2: for the job. And I think the consistency argument is
- 12:07Speaker 2: being used... in an incorrect way to imply that somehow
- 12:11Speaker 2: that's equitable. But the algorithm does work for some people
- 12:16Speaker 2: and it doesn't work for others. It's not equitable. It's
- 12:20Speaker 2: just the same. And the second issue is that the
- 12:24Speaker 2: system is so complex. We are talking about frail, older people,
- 12:29Speaker 2: a sizeable percentage of whom have dementia and Most people
- 12:36Speaker 2: will be in this program for three years before they
- 12:39Speaker 2: either die or go to residential aged care. Many people
- 12:43Speaker 2: are just giving up because the only way you can
- 12:47Speaker 2: appeal is to write a really complicated, fill in a
- 12:51Speaker 2: really complicated form. When they went live, you had to
- 12:56Speaker 2: write a letter, put a stamp on it, take it
- 13:00Speaker 2: and post it to a PO box in Adelaide. It
- 13:04Speaker 2: was designed to make it almost impossible for older people
- 13:09Speaker 2: to appeal, and so hardly anybody did. But that's not
- 13:13Speaker 2: a sign that the algorithm's working. It's a sign that
- 13:17Speaker 2: the system is designed to alienate a very frail population.
- 13:24Speaker 1: Cathy, what does it say about the way we're approaching
- 13:26Speaker 1: aged care, that we're replacing the judgment of experienced assessors
- 13:31Speaker 1: with an algorithm that rather than using technology to help
- 13:34Speaker 1: them do their jobs better.
- 13:36Speaker 2: Well, what we're happening at the moment too, and I
- 13:38Speaker 2: should just add this, is that a lot of very
- 13:41Speaker 2: experienced assessors are leaving. They feel like data entry clerks
- 13:46Speaker 2: and they hate the idea that they assess somebody and
- 13:50Speaker 2: they know that they should be getting level six and
- 13:52Speaker 2: they're getting level two and they're the ones who are
- 13:55Speaker 2: getting the flack. We are losing experienced assessors in favour
- 14:03Speaker 2: of an algorithm that is not fit for purpose. And
- 14:08Speaker 2: what government is doing is using an algorithm to ration
- 14:13Speaker 2: support at home. And they think they're making a cost
- 14:18Speaker 2: saving to government. But a person sitting at home not
- 14:22Speaker 2: getting the support they need is not just quietly sitting
- 14:26Speaker 2: there saving the Commonwealth money. They're ending up in emergency
- 14:31Speaker 2: departments and in hospitals. They're going to residential aged care
- 14:36Speaker 2: because they've had a fall. They've forgotten to take their
- 14:38Speaker 2: medication when they need to. They've got malnutrition. All these
- 14:43Speaker 2: other things that go wrong when you leave an old,
- 14:46Speaker 2: frail person at home without the support they need. So
- 14:50Speaker 2: it's penny wise, pound stupid. And I think there are
- 14:56Speaker 2: two issues. One is Government has adopted this new technology
- 15:01Speaker 2: without actually understanding how to use it. And the second
- 15:07Speaker 2: is that they're using this new technology well beyond what
- 15:12Speaker 2: the technology is capable of doing.
- 15:17Speaker 1: Well, Cathy, it sounds like a fundamentally flawed system. Thank
- 15:20Speaker 1: you for bringing it to our attention.
- 15:23Speaker 2: Thank you.
- 15:37Speaker 1: Also in the news, a Darwin teenager will face court
- 15:41Speaker 1: after being charged with terrorism-related offences related to possessing violent
- 15:45Speaker 1: extremist material and instructions for making 3D-printed guns. The charges
- 15:50Speaker 1: follow a joint investigation by Federal and Northern Territory Police
- 15:53Speaker 1: and ASIO after a tip-off about extremist material being shared online.
- 15:58Speaker 1: The Australian Federal Police says it's increasingly concerned about the
- 16:01Speaker 1: speed at which young Australians are becoming radicalised online. And
- 16:07Speaker 1: the Albanese government's free trade agreement with the European Union
- 16:11Speaker 1: is facing growing opposition, with farming groups urging Parliament to
- 16:14Speaker 1: reject the deal. The National Farmers Federation says the agreement
- 16:18Speaker 1: fails to deliver meaningful improvements for Australia's beef, dairy and
- 16:23Speaker 1: sugar exporters, despite government claims it will boost the economy.
- 16:28Speaker 1: The Coalition is also opposed to the deal in its
- 16:30Speaker 1: current form, raising questions about whether it can secure the
- 16:33Speaker 1: parliamentary support needed for the agreement to take effect. I'm
- 16:37Speaker 1: Daniel James. You've been listening to 7am. We'll be back tomorrow.