Latest / Azadeh and Guy on AI / Double Expert: AI-Assisted Survey Development with Dr. Jordan Wheeler

Transcript

Double Expert: AI-Assisted Survey Development with Dr. Jordan Wheeler · Azadeh and Guy on AI

• 0:00
  SPEAKER_00: Hello, everyone.
• 0:01
  SPEAKER_00: We are recording another episode of Azadeh.
• 0:04
  SPEAKER_03: And Guy.
• 0:05
  SPEAKER_00: On AI.
• 0:06
  SPEAKER_03: And today we have a surprise.
• 0:08
  SPEAKER_00: Yes, we do.
• 0:09
  SPEAKER_03: And so we have Dr. Jordan Wheeler here in the studio, in the flesh.
• 0:15
  And he is a colleague here at UNL,
• 0:19
  but he's also working for Pearson Education right now.
• 0:21
  SPEAKER_00: Yeah, that's correct.
• 0:22
  SPEAKER_03: So he's a man of many talents.
• 0:25
  SPEAKER_00: Yeah, yeah.
• 0:25
  SPEAKER_00: Dr. Wheeler, do you want to introduce yourself?
• 0:28
  SPEAKER_01: Yeah, I can talk a little bit about myself.
• 0:30
  SPEAKER_01: First, thanks for having me on.
• 0:31
  I'm excited to talk a little bit about Gen AI and how we use it
• 0:35
  in our research and
• 0:36
  SPEAKER_01: how we're using it in our collaboration.
• 0:39
  SPEAKER_01: A little bit of background for me.
• 0:41
  SPEAKER_01: So I went to undergrad at Nebraska Wesleyan University,
• 0:45
  SPEAKER_01: studied math,
• 0:47
  SPEAKER_01: then got into statistics for my master's.
• 0:49
  SPEAKER_01: And then when I was doing master's,
• 0:51
  I worked as a data scientist and started to realize I wanted to
• 0:55
  have a bigger
• 0:55
  SPEAKER_01: impact on the educational realm.
• 0:57
  SPEAKER_01: And so I started to Google
• 0:59
  how can I combine stats with education and stumbled upon the
• 1:02
  field of measurements
• 1:04
  SPEAKER_01: and psychometrics.
• 1:05
  And so I applied to different programs and got into the
• 1:07
  University of Georgia,
• 1:09
  where basically I studied different psychometric models focused
• 1:13
  on item response
• 1:13
  SPEAKER_01: theory.
• 1:14
  And a lot of that research focused on how we could integrate
• 1:19
  text analysis models,
• 1:21
  SPEAKER_01: topic models,
• 1:23
  SPEAKER_01: large language models into the measurement process.
• 1:26
  SPEAKER_01: And that led me to where I'm at now.
• 1:28
  SPEAKER_01: I've been at UNL for two and a half years.
• 1:38
  SPEAKER_01: for the educational and psychology.
• 1:40
  SPEAKER_03: For which we are very thankful because we need all the expertise,
• 1:44
  especially because you bring that very specific knowledge base
• 1:49
  around measurement
• 1:50
  SPEAKER_03: and especially AI,
• 1:52
  SPEAKER_03: which is why we have you here.
• 1:54
  SPEAKER_00: And Dr. Vila has also been part of our research project.
• 1:58
  SPEAKER_00: I'm talking about the ACT-E.
• 2:00
  which is about exploring the use of Gen.AI for engineering and
• 2:05
  dissertation.
• 2:05
  SPEAKER_00: Do you like to talk about that as well?
• 2:08
  SPEAKER_01: Yeah, we can talk a little bit about that.
• 2:10
  So that started off,
• 2:11
  I think,
• 2:12
  when you were taking my survey methods course.
• 2:17
  SPEAKER_01: And so in that course,
• 2:18
  I have students develop measures to measure something that they
• 2:21
  want to measure.
• 2:22
  And so I try to tell them it should be focused maybe on
• 2:24
  something that they're
• 2:25
  SPEAKER_01: thinking about for the dissertation.
• 2:26
  SPEAKER_01: And so in that class, we really talk about how do we develop measures.
• 2:31
  SPEAKER_01: A lot of survey method courses focus on sampling,
• 2:33
  SPEAKER_01: and that's an important part of survey research.
• 2:36
  But what I've often found is that writing the items and
• 2:39
  creating the measures is
• 2:41
  SPEAKER_01: often overlooked.
• 2:43
  SPEAKER_01: assume that's the easiest step.
• 2:45
  SPEAKER_01: And to me, that's the hardest step.
• 2:47
  SPEAKER_01: What are you wanting to measure?
• 2:48
  SPEAKER_01: How do you define it?
• 2:50
  SPEAKER_01: What does it look like to be high and low on that construct?
• 2:53
  SPEAKER_01: And then how do you get items that elicit that construct?
• 2:55
  And so that course really focused on the item development
• 2:59
  process and how we can
• 3:00
  SPEAKER_01: actually create measures.
• 3:01
  And so Azadeh took that course and wanted to kind of focus a
• 3:08
  little bit on how
• 3:10
  SPEAKER_01: faculty members are using Gen AI
• 3:13
  SPEAKER_01: in the classroom and their teaching and whatnot.
• 3:16
  SPEAKER_00: Very true, in the College of Engineering.
• 3:18
  SPEAKER_01: Yes, focused in the College of Engineering.
• 3:21
  And so we started to work a little bit together on the item
• 3:25
  development process.
• 3:26
  And when the semester ended,
• 3:28
  I think you reached out to me and asked if I wanted to
• 3:31
  continue.
• 3:32
  SPEAKER_01: And when a student is that eager and motivated,
• 3:36
  SPEAKER_01: I've always found that it's good to cultivate that.
• 3:38
  SPEAKER_01: And those are the people I want to be around, is people who
• 3:42
  SPEAKER_01: are excited about what they're doing.
• 3:44
  And so that's where we started,
• 3:46
  I guess,
• 3:47
  with this collaboration with Guy and then Tarek.
• 3:50
  SPEAKER_03: Yeah,
• 3:50
  and the wonderful thing about that is we started here at UNL
• 3:54
  and with faculty,
• 3:55
  SPEAKER_03: and now we're expanding in two ways.
• 3:57
  SPEAKER_03: We're expanding and working with students,
• 4:00
  SPEAKER_03: and at the same time,
• 4:01
  SPEAKER_03: we're expanding to other universities.
• 4:04
  SPEAKER_03: So this is really exciting.
• 4:06
  SPEAKER_03: And if you're interested and you're listening, please give us a call.
• 4:09
  SPEAKER_00: Yeah, sure.
• 4:10
  SPEAKER_00: It's getting more excited, yes.
• 4:12
  SPEAKER_00: So we have a couple of questions, Dr. Willers.
• 4:15
  So how did you start using generative AI for developing survey
• 4:20
  items or for
• 4:21
  SPEAKER_00: developing measures in educational psychology?
• 4:25
  SPEAKER_01: Yeah, that's a good question.
• 4:27
  Let me take a step back and talk about how I even just started
• 4:30
  to use Gen.AI in my
• 4:31
  SPEAKER_02: day-to-day life.
• 4:33
  So when I was getting my master's,
• 4:36
  computer programming is a huge part of the statistical field.
• 4:41
  SPEAKER_01: Well, Gen.AI wasn't around at that point.
• 4:43
  SPEAKER_01: or it was in its infancy of being used.
• 4:47
  SPEAKER_01: And so when we were coding and we had problems, we had to go Google it.
• 4:51
  And we had to go on Stack Overflow and find a solution and then
• 4:55
  adapt it to our solution.
• 4:56
  And you learn how to write really good questions to Google to
• 5:00
  get to the Stack
• 5:01
  SPEAKER_01: Overflow page that you want to get to.
• 5:04
  SPEAKER_01: And then when ChatGPT started coming around,
• 5:06
  SPEAKER_01: it was in,
• 5:07
  SPEAKER_01: I want to say,
• 5:08
  SPEAKER_01: the last year of my graduate program at Georgia.
• 5:11
  SPEAKER_01: And so I used it a lot first for coding.
• 5:14
  SPEAKER_01: So if I had a coding problem, I would just use that.
• 5:16
  SPEAKER_01: And then I started to think a little bit more about,
• 5:19
  SPEAKER_01: well,
• 5:20
  SPEAKER_01: how can we use this more in the test development process?
• 5:22
  And this is a huge area of research within educational
• 5:25
  psychological measurement is
• 5:26
  SPEAKER_01: how do we use these to develop measures,
• 5:28
  SPEAKER_01: create items,
• 5:29
  SPEAKER_01: those sort of things.
• 5:31
  SPEAKER_01: And so I just started to use it more and more,
• 5:33
  and I found different ways and techniques to improve the
• 5:36
  results I was getting.
• 5:37
  and so it started with um can you help me really like this is
• 5:41
  the contract i want
• 5:42
  to measure this is the theory behind it this is how i'm
• 5:45
  conceptualizing it this is
• 5:46
  how i'm operationalizing can you help me develop items to get
• 5:50
  at that and so
• 5:51
  through that process i've learned how to probe and ask better
• 5:55
  questions to these
• 5:55
  generative models um and that's where we've worked with on the
• 6:00
  student side of the
• 6:00
  SPEAKER_01: survey for yeah
• 6:04
  SPEAKER_03: So how do you start?
• 6:06
  SPEAKER_03: When you get into a new area and you're thinking about the construct,
• 6:10
  what are the first few steps you take as you think about where
• 6:14
  is generative AI
• 6:15
  SPEAKER_03: going to be helpful?
• 6:17
  It also helps our listeners if you talk a little bit about your
• 6:20
  prompting strategies as well.
• 6:23
  SPEAKER_01: Yeah, so that's a really good question.
• 6:25
  SPEAKER_01: And we can talk a little bit about our experience with the students.
• 6:29
  SPEAKER_01: And so we want to understand the landscape of generative AI for
• 6:34
  SPEAKER_01: students, undergrad students in colleges of engineering.
• 6:37
  SPEAKER_01: And so the first thing we want to do is interview those students, right?
• 6:41
  SPEAKER_01: We want to understand how are they thinking of generative
• 6:45
  how is it impacting their lives to be able to develop measures
• 6:48
  that aligns to that
• 6:49
  SPEAKER_01: target population.
• 6:51
  SPEAKER_01: And so you all started to do the cognitive interviews, right?
• 6:54
  You collected transcripts,
• 6:55
  and then you did analysis on those transcripts.
• 6:58
  SPEAKER_01: And you can use generative AI in that process, right?
• 7:00
  You can help it make better sense of the themes that are
• 7:04
  emerging from those transcripts.
• 7:07
  SPEAKER_01: And then once you have those transcripts,
• 7:08
  SPEAKER_01: you can start to kind of develop those themes,
• 7:11
  SPEAKER_01: and those themes become kind of the main pillars of your construct.
• 7:14
  SPEAKER_01: And so from those transcripts,
• 7:17
  you get a better understanding of how students are thinking
• 7:19
  about the construct.
• 7:20
  SPEAKER_01: And so in our case,
• 7:21
  we had students thinking about their perceptions of how other
• 7:25
  students and peers
• 7:25
  SPEAKER_01: are using it,
• 7:26
  SPEAKER_01: how their faculty members are using it,
• 7:29
  SPEAKER_01: their perceived impacts on their own learning.
• 7:31
  SPEAKER_01: We saw that there were some concerns about generative AM.
• 7:35
  SPEAKER_01: Challenges, yes.
• 7:36
  SPEAKER_01: Challenges, right?
• 7:37
  SPEAKER_01: And so these themes emerge,
• 7:39
  and that starts to constitute the landscape of generative AI
• 7:45
  and how students are
• 7:46
  SPEAKER_01: thinking about that,
• 7:47
  SPEAKER_01: especially in undergrad colleges of engineering.
• 7:52
  SPEAKER_03: And I think that what I'm hearing you saying,
• 7:55
  SPEAKER_03: connecting it to what you said before,
• 7:58
  this is really the heart of if you don't define your constructs
• 8:03
  really carefully,
• 8:04
  SPEAKER_03: it's going to be messy.
• 8:05
  And that's a messiness that's really hard to get out of on the
• 8:09
  statistical side,
• 8:10
  right?
• 8:10
  SPEAKER_03: We can't clean up bad questions.
• 8:13
  SPEAKER_01: Yeah, yeah.
• 8:14
  SPEAKER_01: And so in the development process,
• 8:17
  SPEAKER_01: right,
• 8:17
  SPEAKER_01: if you go and you don't have good measures,
• 8:19
  SPEAKER_01: you don't really think about how...
• 8:21
  SPEAKER_01: how you define them,
• 8:22
  SPEAKER_01: how you conceptualize them,
• 8:23
  SPEAKER_01: how you ground them in theory,
• 8:25
  SPEAKER_01: how you operationalize them,
• 8:26
  SPEAKER_01: and you just write items,
• 8:27
  SPEAKER_01: and you think that they're getting at it,
• 8:28
  SPEAKER_01: but you don't do the front-end work of it,
• 8:32
  SPEAKER_01: you go and collect data,
• 8:33
  SPEAKER_01: and then you realize these items are terrible,
• 8:35
  SPEAKER_01: and you just wasted money,
• 8:37
  SPEAKER_01: typically it's pretty expensive to,
• 8:39
  SPEAKER_01: get people to respond to surveys.
• 8:41
  You often have to offer some sort of incentive to get them to
• 8:47
  respond and whatnot.
• 8:48
  And so if you spend all that money to do that and you collect
• 8:51
  it and it's garbage,
• 8:53
  SPEAKER_01: then you have to go back and you're going to have to collect data again,
• 8:55
  SPEAKER_01: and now you're basically
• 8:57
  SPEAKER_01: doubling the cost of collecting that data.
• 8:59
  And part of it can be counted in actual money if you're paying
• 9:04
  them or you're
• 9:05
  SPEAKER_03: giving them donuts,
• 9:06
  SPEAKER_03: which I often do,
• 9:07
  SPEAKER_03: or whatever it is.
• 9:08
  SPEAKER_03: Especially with students, it gets cheap.
• 9:10
  SPEAKER_03: A slice of pizza would get you a long way.
• 9:12
  SPEAKER_03: But it's also your time.
• 9:15
  And your time is super valuable because if you're a graduate
• 9:17
  student or if you're a
• 9:19
  SPEAKER_03: faculty member and you need to publish,
• 9:22
  SPEAKER_03: missing on a semester is a huge cost in time.
• 9:26
  SPEAKER_03: So even if you have money,
• 9:28
  SPEAKER_03: let's say,
• 9:28
  SPEAKER_03: you're still spending time that you're never going to get back.
• 9:34
  And so thinking about what are the things I can do a priori so
• 9:39
  when I go and
• 9:40
  SPEAKER_03: collect robust data,
• 9:41
  SPEAKER_03: it'll come out in a positive way is critically important.
• 9:46
  SPEAKER_01: In that first step, what is the purpose of developing the measure?
• 9:51
  SPEAKER_01: If there is no purpose, then you don't need to be developing it, right?
• 9:55
  SPEAKER_01: What is your target population?
• 9:58
  SPEAKER_01: What are the constructs?
• 10:00
  SPEAKER_01: How do those manifest in your target population?
• 10:03
  SPEAKER_01: How do you define them?
• 10:05
  SPEAKER_01: If you get all of those things really well-defined at the beginning,
• 10:09
  SPEAKER_01: writing items become a lot easier.
• 10:11
  And then you're setting yourself up for success on the back end
• 10:14
  when you go and collect data.
• 10:16
  SPEAKER_01: You could be less worried about problematic items.
• 10:18
  SPEAKER_01: You're still going to have problematic items.
• 10:20
  SPEAKER_01: You're still going to have error.
• 10:22
  SPEAKER_01: You're never going to measure anything without error.
• 10:24
  And but the point of really defining these things at the
• 10:27
  beginning is to reduce
• 10:28
  SPEAKER_01: that error on the back end.
• 10:30
  SPEAKER_01: So you don't have to go back and do it and waste time and resources.
• 10:35
  SPEAKER_03: And especially if you're a graduate student.
• 10:37
  And the reason I'm saying that is because as graduate students,
• 10:40
  you have less time.
• 10:41
  But more than that,
• 10:42
  you will never have a huge sample that can overcome extra
• 10:46
  error.
• 10:47
  SPEAKER_03: And so you're going to be collecting,
• 10:50
  SPEAKER_03: you know,
• 10:50
  SPEAKER_03: if you're lucky,
• 10:51
  SPEAKER_03: 100 and something or so,
• 10:53
  SPEAKER_03: because that's the capacity of a graduate student.
• 10:55
  SPEAKER_03: No offense.
• 10:57
  SPEAKER_02: It's the reality.
• 10:58
  SPEAKER_03: It's the reality.
• 10:59
  SPEAKER_03: And you're not going to have,
• 11:01
  SPEAKER_03: you know,
• 11:02
  SPEAKER_03: $500,000 to conduct research and pay people and all of that.
• 11:06
  SPEAKER_03: And so it's even more important to get as clean a measure as you can.
• 11:12
  So you will get the most out of that smaller sample of 100,
• 11:15
  150,
• 11:15
  whatever you can get to.
• 11:18
  SPEAKER_03: which is challenging.
• 11:19
  SPEAKER_03: It's still challenging.
• 11:23
  SPEAKER_03: What are the advantages you see to using AI in the development process?
• 11:29
  Let me take a step back and talk just briefly about the
• 11:32
  prompting that I do to
• 11:33
  SPEAKER_01: develop the items with the constructs.
• 11:37
  Generally,
• 11:38
  when I use generative models to create items,
• 11:42
  I want to first let them know
• 11:44
  SPEAKER_01: hey, I'm measuring this construct in this target population.
• 11:47
  SPEAKER_01: This is how I'm defining this construct.
• 11:50
  SPEAKER_01: I have a really strong definition of what that construct looks like.
• 11:54
  These are maybe the different subdomains within or the
• 11:57
  different domains within in
• 11:59
  SPEAKER_01: that construct and the pillars of that construct.
• 12:01
  And then how do I operationalize the construct as a whole and
• 12:04
  then all of the
• 12:05
  SPEAKER_01: subscales within it?
• 12:07
  SPEAKER_01: So what does it look like for people to be low, to have a small amount
• 12:13
  SPEAKER_01: high or have a large amount of that construct.
• 12:16
  That helps the generative models write items so that when they
• 12:20
  have a Likert scale
• 12:21
  SPEAKER_01: to them,
• 12:22
  the lower end of that Likert scale aligns with the lower end of
• 12:25
  how you define and
• 12:26
  SPEAKER_01: operationalize your construct,
• 12:28
  and the higher end of that Likert scale aligns with how you
• 12:31
  operationalize the
• 12:33
  SPEAKER_01: upper end of your construct.
• 12:35
  SPEAKER_01: When you prompt it and you give it as much information as possible,
• 12:38
  SPEAKER_01: I start to see that it produces better items.
• 12:41
  SPEAKER_01: You're still going to have to be tweaking to those items.
• 12:44
  SPEAKER_01: You still have to modify them, change them.
• 12:49
  You never just want to take the output of a generative model
• 12:51
  and just copy-paste it.
• 12:52
  SPEAKER_01: You want to think through it.
• 12:54
  That's one of my concerns with generative AI is the blind just
• 12:59
  accepting what it's
• 13:00
  SPEAKER_01: outputting without critically thinking about what it's producing.
• 13:04
  SPEAKER_03: And I find that sometimes it's easier
• 13:08
  because you're not the author to be the editor and the person
• 13:12
  who's critiquing
• 13:13
  SPEAKER_03: because when you're creating the items,
• 13:15
  SPEAKER_03: you need to create some kind of distance.
• 13:17
  SPEAKER_03: And one of the ways to do that,
• 13:18
  SPEAKER_03: of course,
• 13:19
  is to just let it go for a while and then revisit it and it
• 13:23
  looks a little bit
• 13:24
  SPEAKER_03: foreign.
• 13:25
  SPEAKER_03: But that means more time.
• 13:27
  SPEAKER_03: Whereas if AI creates the items,
• 13:30
  SPEAKER_03: we can then be the editors and say,
• 13:32
  SPEAKER_03: that's not what we meant or that's not what we saw in the interviews.
• 13:37
  So you still need to read the interviews,
• 13:39
  conduct the interviews,
• 13:40
  all of these things.
• 13:41
  SPEAKER_03: You can't avoid it.
• 13:42
  SPEAKER_03: But yes, it is that extra step.
• 13:46
  SPEAKER_03: You cannot skip it.
• 13:47
  SPEAKER_03: You cannot cut and paste.
• 13:49
  SPEAKER_03: It's going to be a mess.
• 13:51
  SPEAKER_03: It's a different kind of error introduced into the data.
• 13:54
  SPEAKER_00: So I have a question.
• 13:56
  SPEAKER_00: I'm curious.
• 13:57
  Have you ever had this experience that Gen AI looks at a
• 14:00
  different perspective to
• 14:03
  SPEAKER_00: the construct that you are trying to develop a measure for?
• 14:06
  SPEAKER_00: I mean,
• 14:07
  SPEAKER_00: you have some items that you develop yourself,
• 14:09
  SPEAKER_00: and then you ask AI with that detailed prompt that you just explained,
• 14:13
  and then you see AI looks at this kind of thing from a
• 14:17
  different perspective.
• 14:19
  SPEAKER_01: Yeah, that certainly can happen.
• 14:22
  SPEAKER_01: When I've used it for item development,
• 14:24
  I usually give it maybe some sample items that I created to
• 14:27
  kind of lead it onto
• 14:28
  the path that aligns with what we're seeing in the cognitive
• 14:31
  interviews and sort of
• 14:33
  the understanding of how that construct looks on the front end
• 14:38
  of the test
• 14:38
  SPEAKER_01: development or the skill development process.
• 14:42
  SPEAKER_01: But that certainly could happen.
• 14:43
  And maybe that's an interesting insight to look at when these
• 14:46
  things do have
• 14:47
  SPEAKER_01: different perspectives.
• 14:53
  but i think when you have a really well-defined construct that
• 14:56
  is grounded in the
• 14:57
  cognitive interviews of people from your target population
• 15:00
  right i think and you
• 15:02
  you tell generative ai models like this is how i define it
• 15:06
  usually i think it stays
• 15:07
  within those parameters of how you define it yeah give you
• 15:10
  items um but it's an
• 15:12
  iterative process right like so they'll they'll output you know
• 15:16
  multiple items and
• 15:17
  then you might say well there's this sort of like style that's
• 15:20
  coming off of
• 15:21
  of how you're writing these items i don't really like it maybe
• 15:24
  can you adjust those
• 15:25
  and address it and give it an example of a hey this item here
• 15:29
  that you wrote i
• 15:30
  don't like how you phrased it i would phrase it this way can
• 15:33
  you do that to the
• 15:33
  other items that you generated and then it's this iterative
• 15:36
  process of kind of
• 15:37
  SPEAKER_01: getting and editing to the items themselves
• 15:39
  SPEAKER_00: So there is also a lot of back and forth between you and AI.
• 15:44
  And the other thing I think we've done in our iteration of
• 15:48
  doing that was we had it
• 15:50
  SPEAKER_03: overproduce and then we could select the ones that we liked the most,
• 15:55
  the ones that seem to align better and not have too much
• 15:59
  redundancy in the items
• 16:00
  SPEAKER_03: because often it would say the same exact thing in three different,
• 16:05
  SPEAKER_03: slightly different ways.
• 16:06
  SPEAKER_03: And you're like, okay, I need only one or two of those.
• 16:08
  SPEAKER_03: Yeah.
• 16:09
  Yeah,
• 16:09
  and that's pretty common within skill development prior to even
• 16:12
  generative AI.
• 16:13
  You would write a whole host of items,
• 16:16
  and you know that some of them are going to be bad.
• 16:18
  SPEAKER_01: And then you would invite content experts,
• 16:20
  SPEAKER_01: people in the field,
• 16:21
  SPEAKER_01: review the items,
• 16:22
  and then you compare the ratings of what those experts say
• 16:25
  about the items.
• 16:26
  And you would select the best items that the raters all agreed
• 16:30
  were good items and
• 16:31
  SPEAKER_01: were getting at the construct.
• 16:33
  SPEAKER_01: Now we can put that before that step, right?
• 16:35
  So now we have kind of this double content expert approach to
• 16:39
  item development.
• 16:40
  SPEAKER_01: So first it's from the generative AI and then the actual experts.
• 16:44
  SPEAKER_03: And what is important to say here, we're not skipping the expert phase.
• 16:47
  SPEAKER_03: We're not saying the chat GPT can function as an expert on all of this.
• 16:52
  SPEAKER_03: We're also using...
• 16:54
  our partners across many institutions to give us feedback and
• 16:57
  to tell us which
• 16:58
  items are better and sometimes it's just a relative better
• 17:02
  there's no perfect item
• 17:03
  but they're definitely better items so it it doesn't take away
• 17:08
  steps it's just more
• 17:11
  maybe quicker ways to develop items but hey we're not taking
• 17:16
  the human out of that
• 17:17
  SPEAKER_03: loop at all yeah
• 17:18
  SPEAKER_01: I think that's an important thing to highlight.
• 17:20
  SPEAKER_01: We're not just, again, blindly taking what the generative models produce.
• 17:25
  SPEAKER_01: We're reviewing them.
• 17:26
  SPEAKER_01: We're having content experts review them.
• 17:28
  SPEAKER_01: And we're making sure that when we create our initial pilot form,
• 17:33
  SPEAKER_01: we have the best items,
• 17:34
  SPEAKER_01: at least from a content perspective,
• 17:37
  SPEAKER_01: going on that.
• 17:38
  SPEAKER_01: And then we'll collect the pilot data with it.
• 17:41
  SPEAKER_01: We'll do psychometric analyses.
• 17:42
  SPEAKER_01: And then we'll probably reduce the number of items even more after that.
• 17:46
  SPEAKER_01: We'll get a final scale.
• 17:49
  SPEAKER_03: And so just to remember,
• 17:50
  SPEAKER_03: if you're a graduate student,
• 17:52
  SPEAKER_03: the first time,
• 17:53
  SPEAKER_03: first of all,
• 17:53
  take Jordan's class before you start or an equivalent class if
• 17:57
  you don't have the
• 17:59
  SPEAKER_03: joy of being here.
• 18:00
  SPEAKER_03: And really learn the process before you're studying the thing itself.
• 18:07
  SPEAKER_03: You need to pilot this.
• 18:08
  SPEAKER_03: This is a process.
• 18:09
  SPEAKER_03: Yes.
• 18:10
  SPEAKER_03: And it's worth doing because you get better results at the end.
• 18:14
  Because you don't want to be the person who says at the end,
• 18:16
  well,
• 18:16
  I've done all of this.
• 18:17
  And if I would have asked better questions,
• 18:20
  I would have been able to say something.
• 18:23
  And so a little bit more development time will go a long way
• 18:27
  towards a quality
• 18:28
  SPEAKER_03: dissertation study,
• 18:29
  SPEAKER_03: research,
• 18:30
  SPEAKER_03: pilot study,
• 18:31
  SPEAKER_03: whatever it may be.
• 18:32
  SPEAKER_03: All right, last question.
• 18:35
  SPEAKER_03: What is your advice for graduate students or people who are starting,
• 18:39
  they don't necessarily have a lot of experience of working with
• 18:42
  generative AI,
• 18:44
  SPEAKER_03: and they do want to develop measures?
• 18:47
  SPEAKER_03: Words of advice.
• 18:48
  SPEAKER_03: Well,
• 18:49
  SPEAKER_01: I think you hit on a point I would emphasize,
• 18:51
  SPEAKER_01: which is learn the process before getting into generative AI.
• 18:56
  SPEAKER_01: Go and write the items yourself.
• 18:58
  SPEAKER_01: Learn, experience what that looks like, what that feels like,
• 19:02
  SPEAKER_01: to spend that time to struggle, right?
• 19:03
  SPEAKER_01: Look at a blank page and realize,
• 19:05
  SPEAKER_01: right,
• 19:05
  SPEAKER_01: I have to develop this first,
• 19:07
  SPEAKER_01: right,
• 19:07
  SPEAKER_01: and then I can edit it.
• 19:09
  SPEAKER_01: I think that's super important for learning.
• 19:10
  SPEAKER_01: And,
• 19:11
  SPEAKER_01: you know,
• 19:12
  SPEAKER_01: as a faculty member,
• 19:13
  SPEAKER_01: that's something I'm concerned with students right now is I want them,
• 19:19
  SPEAKER_01: right,
• 19:19
  SPEAKER_01: I want them to use Genitive AI,
• 19:20
  SPEAKER_01: right,
• 19:20
  SPEAKER_01: because it's not going anywhere.
• 19:22
  SPEAKER_01: They're going to be using it in their everyday life.
• 19:24
  SPEAKER_01: But I also want them to appreciate it, right, and appreciate using it
• 19:29
  SPEAKER_01: in the correct manners.
• 19:30
  And I think the only way to do that is to actually experience
• 19:33
  doing the thing without that tool.
• 19:37
  SPEAKER_01: So it's hard when you're writing a paper,
• 19:40
  SPEAKER_01: you're staring at a blank page,
• 19:43
  SPEAKER_01: it's hard to get started.
• 19:45
  Once you get started and you have a draft editing,
• 19:47
  to me,
• 19:48
  that's the fun part.
• 19:48
  SPEAKER_01: I get to kind of tweak around.
• 19:50
  SPEAKER_03: I have a few papers I can give you, if that's the fun part.
• 19:54
  SPEAKER_03: But sure.
• 19:56
  SPEAKER_01: But if you're starting with generative AI outputs,
• 20:00
  then I worry about that a little bit because you're not
• 20:04
  appreciating what it's
• 20:06
  SPEAKER_01: producing.
• 20:08
  SPEAKER_01: And so I think,
• 20:09
  SPEAKER_01: yeah,
• 20:09
  SPEAKER_01: first,
• 20:09
  SPEAKER_01: before you even get started using it,
• 20:11
  SPEAKER_01: learn how to develop measures on your own.
• 20:14
  SPEAKER_01: Take either an intricate measurement course,
• 20:17
  SPEAKER_01: take a classroom assessment course,
• 20:19
  SPEAKER_01: take a survey methods course,
• 20:22
  SPEAKER_01: any test development course.
• 20:23
  SPEAKER_01: Take any course that's related to developing measures and scales.
• 20:28
  SPEAKER_01: Learn that process.
• 20:30
  SPEAKER_01: And then once you get that experience,
• 20:32
  SPEAKER_01: then start to kind of see how you might use generative AI.
• 20:35
  And you'll probably be much better at prompting now that you've
• 20:39
  grappled with this
• 20:40
  SPEAKER_03: for a while because you know what to ask for.
• 20:42
  SPEAKER_03: Otherwise,
• 20:42
  SPEAKER_03: it becomes this,
• 20:43
  SPEAKER_03: I don't know what to ask for because I'm not sure how it's constructed.
• 20:48
  SPEAKER_03: And then you get lousy items, let's be honest.
• 20:52
  SPEAKER_01: You want to play around with it.
• 20:53
  SPEAKER_01: So back to when I was learning to program,
• 20:57
  SPEAKER_01: when I first started to use Google and use Stack Overflow,
• 21:00
  SPEAKER_01: I'm sure the questions I asked to solve my problems were terrible.
• 21:03
  SPEAKER_01: right?
• 21:04
  But after practice with it and struggling with it,
• 21:06
  you learn how to ask better questions.
• 21:08
  SPEAKER_01: You learn to think about what do I want from a very general viewpoint,
• 21:13
  SPEAKER_01: not just a very specific to the problem I have,
• 21:15
  SPEAKER_01: but how might this be addressed in a more general sense?
• 21:19
  SPEAKER_01: And I think with that struggle and with that process,
• 21:21
  SPEAKER_01: you get a better sense of kind of how to prompt it,
• 21:25
  SPEAKER_01: what are the sort of things you have to focus on,
• 21:27
  SPEAKER_01: what are the things you have to look at on the output side of the thing,
• 21:30
  SPEAKER_01: and how might you
• 21:33
  kind of interact with it in this iterative way to get to the
• 21:37
  point where you have
• 21:38
  items that are at least good to go and for you to start editing
• 21:42
  and then send you a
• 21:43
  SPEAKER_01: content team to look at it.
• 21:46
  SPEAKER_03: All right.
• 21:46
  SPEAKER_03: So thank you.
• 21:47
  SPEAKER_03: Yeah.
• 21:47
  SPEAKER_01: Thank you so much.
• 21:48
  SPEAKER_03: This was lots of fun.
• 21:50
  SPEAKER_03: And we'll see you all next time on Azadeh.
• 21:54
  SPEAKER_00: And Zai.
• 21:55
  SPEAKER_03: On AI.