Latest / HFL Education / The Datafication of Education: Are we measuring what matters?

Transcript

The Datafication of Education: Are we measuring what matters? · HFL Education

• 0:01
  Hello and welcome to the HFL Education Business Services podcast.
• 0:05
  I'm Catherine Loake, the director of business services here at HFL Education and your host.
• 0:09
  We decided to commission this series of podcasts because we recognise that the role of school leadership grows ever wider and ever deeper, and finding the time to unpack some of the key issues away from frontline teaching and learning is ever more challenging.
• 0:20
  So that's where we come in.
• 0:21
  And this week I'm delighted to be joined by Ben Fuller, who leads our assessment teams at HFL Education.
• 0:27
  Hello Ben.
• 0:27
  Good morning Catherine.
• 0:28
  So we're going to be unpacking the datafication of education.
• 0:31
  This this could be more timely, Ben, because my LinkedIn feed this week has blown up around people reflecting on, asking questions about the DfE data spine.
• 0:41
  And it just strikes me that data and MIS were once the sleepy backwater of school offices.
• 0:46
  But suddenly, over recent years, they've come to prominence for lots of different reasons.
• 0:50
  I think this is going to be a really interesting discussion.
• 0:53
  Data's definitely very much back on the agenda.
• 0:56
  I think it was very much a prominent thing, particularly around assessment data, achievement data was very prominent kind of pre 2019 really.
• 1:07
  It kind of went into the background a little bit when that Ofsted framework changed in 2019 and Ofsted became a lot less focused on data and more interested in the qualitative information that they could see in the classroom.
• 1:21
  But it very much feels back on the agenda.
• 1:25
  There's a lot of data that's out there.
• 1:28
  There's a lot of information in the IDSR there's a lot more scrutiny on things like attendance and exclusions and all that sort of thing.
• 1:37
  Not just in the IDSR, but obviously Ofsted want to know what's that looking like at the moment?
• 1:42
  So schools need to have a good handle on their up to the minute data in terms of how they're doing against things like pupil attendance and exclusions and things like that.
• 1:53
  I think the most important thing about data is what's it telling us and what can we do with it.
• 1:58
  What actions does it lead to in terms of decisions that school leaders need to make.
• 2:04
  And I guess that's where I was going to start, really.
• 2:07
  Are we measuring what matters?
• 2:09
  Because it strikes me that more and more schools are incentivised to look at what's measurable rather than what's valuable.
• 2:16
  Is that fair to say?
• 2:17
  I think it's always been a challenge, really, for as long as I can remember that clearly, some some things are easier to measure than others.
• 2:25
  That doesn't mean that the other things aren't valuable, but some things are clearly harder to measure.
• 2:31
  And if we look at the curriculum, if we look at the way that SATs tests work, for example, certain things are easier to quantify in terms of asking questions that children can give an answer to that can be marked right or wrong, some things are much more sort of nebulous in a way and are harder to measure.
• 2:49
  But that doesn't mean they're not important.
• 2:51
  So there's always that danger.
• 2:53
  If we look at it from a curriculum point of view, that schools might narrow their focus.
• 2:58
  Just recently, you know, the department saying that they want to kind of make oracy as important as reading and writing and absolutely brilliant.
• 3:08
  We know that oracy is important, but that's probably a harder thing to measure in a way of saying, has this child achieved an objective or not?
• 3:18
  So we've got to be aware of the limitations of data that it can't always capture everything we want to know.
• 3:25
  It only gives us a bit of a snapshot and a snapshot on a small kind of aspect of the curriculum and pupils experience, and that it's not the complete picture.
• 3:36
  And obviously you just highlighted that some things are difficult, if not impossible to measure.
• 3:42
  And something that I've been really interested in is this concept of belonging in schools and the fact that schools are seeking to measure belonging.
• 3:49
  But how do you actually do that?
• 3:50
  So do we sort of run the risk Ben, we measure around the periphery almost to try and measure something and completely miss the point?
• 3:57
  It's a real risk, I think.
• 3:58
  I think it's useful to kind of realise that what data might do is give us some sort of an insight into something.
• 4:06
  For example, we can look at attendance data and we might say, if we want to know whether a child experiences belonging in their school, then they're going to turn up every day.
• 4:15
  So attendance data is going to suggest that there's a sense of belonging, but it doesn't necessarily mean that, it might correlate but attendance data doesn't, for example, give you any information on children who are physically present in class, but not emotionally present in class.
• 4:34
  So the data can only ever give some possible lines of inquiry.
• 4:40
  But you've always got to dig beneath the surface and you've got to do things like talk to the learners themselves, experience the more qualitative data by getting that pupil voice by dropping into classrooms by just generally experiencing what school looks like through their lens.
• 4:58
  Because data always has limitations.
• 5:01
  And attendance is a classic one because two schools might have very similar attendance figures.
• 5:06
  One school, there might have been a big raft of illness running through the school, meaning that several children missed a week and that's going to bring attendance down.
• 5:17
  apart from that illness, attendance is good.
• 5:20
  Another school might have exactly the same figure, but for different reasons, because there's kind of a low level of number of children who are kind of consistently missing Friday afternoon or whatever it might be.
• 5:32
  And I guess that's the point, isn't it?
• 5:34
  There's a difference between being data led and being data driven.
• 5:39
  And I suppose one of the things I've picked up on in the week, and one of the things probably that concerns me, you know, having the data management services team within business services is, is there a risk that we do become data driven as opposed to data led?
• 5:54
  And I suppose this is the whole concept of the DfE data spine.
• 5:57
  If we're going to move from observability of the data that if the DfE’s got live data to accountability, it's natural, isn't it, human behavior you're going to switch.
• 6:06
  If suddenly there's a massive focus on attendance from central government you're being held to account and you're going to switch all of that, that focus to it, to accountability.
• 6:14
  So are we at risk in education of moving from being data led, as you've just described, which is really key to being data driven?
• 6:20
  I think it's a risk, and I think it's something we need to try and fight, being data driven can send you down the wrong path.
• 6:29
  And it's one of the dangers of high stakes accountability that if you feel highly accountable to certain outcomes, whether that be to Ofsted or because something has to be published on your website or whatever, that it can change your behavior, it can have consequences, and not always in an appropriate way.
• 6:49
  So I think we need to be quite mindful of of that and think, yes, data is useful for making us kind of sit up and take notice about certain things and ask questions, and school leaders and Ofsted inspectors, etc.
• 7:04
  can use data to say, well, that's interesting.
• 7:07
  Let's dig beneath the surface.
• 7:09
  Let's find out what's really going on here, because the data implies X, but we actually need to do a bit of digging to see if that is what's going on or if there's something else happening.
• 7:21
  It's really interesting.
• 7:22
  And I think thing that's really struck me is this concept of balance that you've talked about and I guess, Ben, what's the difference between a useful accountability and burdensome data collection?
• 7:33
  Because we're talking about data, we're talking about attendance, all sorts of things.
• 7:36
  But at the end of that attendance data is a child.
• 7:40
  So how do we ensure we don't lose sight of the learner in the center?
• 7:45
  and it is all about data collection.
• 7:47
  How? That's a big question.
• 7:50
  I don't exactly know how.
• 7:52
  I think it's something that we need to be very mindful of.
• 7:55
  I think school leaders need to think very carefully about what data do we need to collect, and we're not collecting data for the sake of it.
• 8:04
  So we're collecting data because it's data on something that we as a leadership team have decided we need to know about, because it will inform our strategic thinking in some way.
• 8:18
  But we don't want to be asking all of our teachers to be logging all sorts of information into MIS systems that it just is a burden that isn't kind of worth doing.
• 8:32
  And I think, you know, going back a number of years, There used to be from an attainment tracking point of view, many schools were going down the route of trying to track pupils attainment on a half termly basis.
• 8:45
  And we were advising not to do that because actually it's that old adage, if the more time you spend weighing the pig, weighing the pig doesn't make the pig any fatter, And actually, the more time you spend weighing the pig, the less time you actually nurturing and feeding the pig you know, you need to kind of get that right balance of not going down this route of very burdensome data collection activity, which takes time and takes teachers focus away from their core purpose of teaching.
• 9:13
  But nonetheless, a certain amount of data can be useful, provided there's a really good rationale for collecting that data.
• 9:22
  And you've been doing this for a long time, so there's not an awful lot you haven't seen, Ben.
• 9:27
  And are there times at which schools risk, again, that balance tipping.
• 9:33
  So you've talked about high stakes accountability and the Ofsted framework.
• 9:37
  And it strikes me again that if you've had a difficult Ofsted outcome and you have monitoring visits, it becomes very much about the data.
• 9:45
  And I know there's obviously conversations with pupils, but are there points in the system or times in a schools journey where the risk of tipping the balance between data driven and data led are more at risk? Definitely.
• 9:59
  And I think from an achievement point of view, if we're thinking about learner outcomes, this has always been the case, whether we're looking at, let's say GCSE or Key Stage 2, accountability tends to focus on thresholds.
• 10:13
  So at GCSE what proportion of your learners have achieved a grade four in both English and maths.
• 10:18
  So that threshold becomes a bit of a cliff edge.
• 10:21
  And that's what can sometimes drive behavior to the extent that if you've got students who are kind of at risk of not quite getting grade four in both English and maths, they could become your focus group and you're pouring all of your resource into trying to help those students get that threshold.
• 10:41
  You might not need to worry so much about the higher attainers who have already secured it, or the lower attainters who you think, well, they're so far away from reaching that threshold that they're never going to make it.
• 10:53
  Now, I know I'm kind I'm very much, you know, putting that into extreme terms morally, no one would completely neglect those other students.
• 11:00
  But you can see how there could be a focus towards those.
• 11:03
  Same will happen at primary school.
• 11:05
  And this is partly why many, many years ago, you'll remember that we used to have national curriculum levels in primary and Key Stage three, and then decision was made to get rid of levels.
• 11:16
  And it was partly because of that cliff edge system that you could be pouring all of your resource into getting children to that level four the risk of, other children making progress.
• 11:29
  So they got rid of levels, but it hasn't really changed because what they replaced levels with was another system of categorisation, the expected standard or greater depth.
• 11:39
  And actually, I think if we really wanted to try and change that mindset, we would need to remove the idea of categories and labels.
• 11:51
  We could have a sliding scale, if you like, of how well each child is achieving or even a sort of a competency framework that says these are the skills that this child has achieved, which is kind of a more formative assessment.
• 12:07
  But it's when start putting children into categories, which is really driven by the accountability system.
• 12:12
  You know, the DfE want to be able to publish data that say, in this school, X% achieved this standard.
• 12:18
  So you have to define a standard.
• 12:20
  And that's always going to create these perverse incentives to focus on getting children to those standards.
• 12:27
  That's just within achievement data.
• 12:28
  But you could take anything, whether it's attendance or anything like that.
• 12:31
  it can drive behaviour in ways which aren't necessarily the most desirable way.
• 12:39
  We would want people to be acting.
• 12:40
  And that's kind of, I guess, where I see one of the risks of the data spy, and that essentially the tail wagging the dog, because suddenly the DFB has all of this rich data in real time, and we end up shifting educational policy based on the data as opposed to the story and the learner behind that data.
• 13:00
  And as you've said, it's that balance between quantitative and qualitative.
• 13:03
  Yeah, data.
• 13:04
  And it just strikes me that we are moving away from that balance and that.
• 13:10
  Yeah, the real risk is we seem to have come so far in education policy, energy said.
• 13:14
  We now seem to have a focus on every learner as opposed a few that, as you say, perhaps were at risk of not achieving certain grades it just strikes me we're in quite a dangerous place at the I think we could be.
• 13:25
  And coupled on top of that is I think that there are issues around people's data literacy.
• 13:31
  To what extent do they really understand the data that we're dealing So, for example, when the DFA analysing the data, they tend to be looking at things in terms of what is above average and what's below average.
• 13:44
  So they're looking at regional inequality for example.
• 13:46
  They can see that certain parts of the country are below average.
• 13:49
  But of course that average is not an absolute thing.
• 13:51
  It's the average is the average of that year.
• 13:54
  So standards can be going up and up and up and up.
• 13:57
  But they'll always be above and they'll always be below.
• 13:59
  in terms of absolute standards.
• 14:02
  We know, for example, what achievement looked like pre-COVID and then post-COVID it dropped and coming back up in some places kind of back where it was pre-COVID again.
• 14:13
  So from an absolute standard, we can kind of see what it looks like.
• 14:17
  But then when you start looking at the above and below average, that different sorts patterns come out.
• 14:25
  and groups of children, I mean, on the whole, I think it's fair to say that students are achieving better now than they were probably two decades ago.
• 14:37
  So you could look at that and say, standards in schools are rising, but you don't normally hear that message because the focus is always on, well, this is the average and these schools are below average and these schools are above average.
• 14:49
  it's a bit of a minefield.
• 14:50
  And I think also a lot of data is produced to put into the public domain for parents to understand.
• 14:56
  But again you've got that issue around data literacy.
• 14:58
  So some of the data is kind of reduced to a simpler form so that the general public can consume it.
• 15:07
  But that loses the nuance around, well, okay, you're telling me this is above average, but is it statistically significant or not?
• 15:15
  Well, that requires a bit of data literacy to kind of dig into that.
• 15:19
  And that's just reminded me actually, something slightly off beam I wanted to ask you, so you've talked about parents and data, I can remember every time I used to get pinged by the school app, your child's behaviour status has changed.
• 15:33
  On the behaviour dashboard, I used to have like a, you know, a mini, you know, what have they done now?
• 15:38
  Like to much information But sometimes it was they’ve done a brilliant thing and they've got a merit.
• 15:44
  But it was still almost your sat at work as a parent.
• 15:46
  There's not an awful lot you can do to parent your child when they're sat in Year 11 maths class and you're getting, again real time data on how they're behaving in school.
• 15:54
  I found that really anxiety inducing.
• 15:56
  Are we at risk again in terms of datafication of turning parents off education because we are then feeding them this real time information as well?
• 16:05
  That could absolutely happen.
• 16:07
  I think the data anxiety is a real thing, and the more you get real time updates, I guess it's almost like the kind of the dopamine hit of social media, isn't it?
• 16:17
  It's like you could become dependent on the what's happening now, what's happening now.
• 16:22
  And actually sometimes no news is good news.
• 16:25
  You know, you just need to know the extreme things, but you don't need to know minor fluctuations on a day to day basis.
• 16:33
  But it's also context, isn't it?
• 16:34
  Because again, one of my kids had a, you know, a card system at school.
• 16:37
  So you very occasionally obviously, you might get a text to say your child has received a yellow card, but no other context to that.
• 16:45
  So then, okay, what conversation do I have with the child when they get home?
• 16:48
  So I think context is king as well, isn't it, when it comes to to data.
• 16:52
  And I guess back to that learner and understanding again, what drove that behaviour as opposed to it being punitive Again, so as I say, the data is really a flag if you like.
• 17:03
  But there's a story that's always a story behind the data.
• 17:06
  You've always got to dig beneath it that's the case with any data set, really.
• 17:10
  It might be something that sparks conversation, but the data on its own should never be seen as the conclusion.
• 17:19
  I think that's always the danger.
• 17:20
  When you look at any data on a school, there's a danger to look at the data and think, okay, well, I conclude X, the data on its own should never be the conclusion.
• 17:31
  It just sparks that conversation to think, I need to dig into this to understand what's going on and find out why that result is the way it is.
• 17:40
  Sort of wrapping up, then, how would you advise school leaders to get the balance?
• 17:46
  Because that's something that's come across really clearly been it's about balance.
• 17:49
  It's about context.
• 17:51
  It's about qualitative and quantitative.
• 17:54
  So how do leaders get the balance between measuring and then those I suppose those conversations, those contextual conversations and ensuring that staff and governors parents understand actually the story behind the data, I think a really useful starting point is for leaders to think carefully about what data do we want to collect and why do we want to collect it?
• 18:19
  What are we going to use it for?
• 18:21
  Whether it be something that governors want to know about or it's going to inform some decisions.
• 18:28
  But be really clear about why we're collecting this how is it going to shape our thinking.
• 18:32
  But also, I think, to take a step back and start off with just thinking, well, as a school, what is our vision?
• 18:39
  What is our mission?
• 18:40
  What is it we want to achieve and be and really have that at the centre and be aware of and comfortable with the fact that some of those things aren't easily measured, but we still want them to be front and centre of what we believe our school to be about.
• 18:54
  And the evidence for those things is not going to exist in a spreadsheet, but the evidence for those things will exist when you talk to our children or you talk to the staff, or you just go for a learning walk around the school, you will see that our vision is alive, and we need to just be comfortable with the fact that we can't measure everything, and we shouldn't just pour our energy into those things which are easy to measure because the risk is that we lose sight of other things which are important.
• 19:25
  And it comes down to reliability and validity of data.
• 19:30
  Data can be very reliable if we focus on the things which are very easy to measure.
• 19:36
  But that would mean it's not necessarily valid in the sense of it doesn't tell us everything that we're really interested in, and we want to make sure that we are keeping the focus on the important the mission statement, if you like.
• 19:54
  We're keeping our focus on what do we want our school to feel like to look like?
• 20:00
  How do we want our learners to experience this school?
• 20:03
  And like you said earlier, belonging.
• 20:06
  You know, we want children to feel like they belong.
• 20:09
  Well, does that mean we should try to measure it?
• 20:12
  Well, I mean, you could set up surveys, but that would become a data collection exercise just for the sake of it.
• 20:19
  You can just talk to children and get a sense of belonging.
• 20:23
  So I don't think we should try to measure everything.
• 20:27
  We just need to be comfortable with the fact that not everything can be measured, and attempts to measure it are probably going to be flawed, the evidence, will exist in more qualitative sources of data.
• 20:38
  Wow, Ben, I don't.
• 20:40
  You've actually blown my mind.
• 20:42
  I think you're so right that I think that the rise in standards, of course, in part it's down to the quality of teaching and advancements in in training and techniques.
• 20:53
  But also, you're absolutely right.
• 20:54
  It's down to us measuring and understanding what's happening.
• 20:59
  But there's a tipping point at which we can go too far.
• 21:02
  And yeah, that balance is so fine, isn't it, between data lead and data driven.
• 21:09
  So yeah, I think you've given me lots to reflect on, think I'm going to be spending half term thinking about data.
• 21:16
  So thank you ever so much for giving up your time.
• 21:18
  I think leaders would have found that really valuable.
• 21:20
  If you did find it valuable, please do like and subscribe to the podcast and we will be back next week with another episode.
• 21:26
  Bye for now.