Latest / Azadeh and Guy on AI / AI Meets Quantitative Analysis

Transcript

AI Meets Quantitative Analysis · Azadeh and Guy on AI

0:01
SPEAKER_01: Hi, and welcome to episode five.
0:03
SPEAKER_01: And this time we're talking about quantitative analysis with AI.
0:09
SPEAKER_01: And so welcome.
0:11
SPEAKER_02: Hello, everyone.
0:12
SPEAKER_02: I'm Azadeh.
0:14
SPEAKER_02: I'm a third year PhD student at UNL.
0:18
SPEAKER_02: And my specialization is Innovative Learning Technologies.
0:21
And I'm Guy Trainier,
0:22
a professor here at UNL,
0:24
also Innovative Learning Technologies.
0:26
SPEAKER_01: And as I said, today we're talking about quantitative analysis with AI.
0:30
SPEAKER_01: I love AI.
0:31
SPEAKER_01: AI is everything I think about except my granddaughter.
0:35
So I'm delighted to be here and talk about this,
0:39
especially since while we talked before...
0:43
SPEAKER_01: About quantitative and qualitative analysis.
0:46
SPEAKER_01: And we talked a lot about qualitative analysis.
0:49
SPEAKER_01: I'm a quantitative researcher.
0:50
So this is very close and dear to my heart and also a place
0:54
where my expertise kind
0:56
SPEAKER_01: of bangs against the jagged edge of AI,
1:01
SPEAKER_01: as Ethan Malik likes to say.
1:04
SPEAKER_02: So the things that we are going to cover today will be,
1:08
SPEAKER_02: first we are going to listen to Guy when he talks about Julius AI,
1:13
SPEAKER_02: and I may ask some questions.
1:15
SPEAKER_00: Please.
1:16
SPEAKER_02: Yeah, yeah, sure.
1:17
SPEAKER_02: And then I will talk about a paper that we both read recently.
1:21
SPEAKER_02: It's been published in 2025,
1:23
SPEAKER_02: so it's a fairly recent paper,
1:26
SPEAKER_02: and it's about the use of Gen AI in a statistical data analysis.
1:29
SPEAKER_02: Okay.
1:30
SPEAKER_01: So Julius AI, let's get straight to it.
1:33
Julius AI is an AI that is designed to do a quantitative
1:39
analysis of statistical
1:41
SPEAKER_01: analysis inside a shell that does two things.
1:45
SPEAKER_01: There's a component that helps interpret what you want.
1:47
So you can use normal language and say,
1:49
I want to compare two things on all of that.
1:52
SPEAKER_01: And then there's the analytic part.
1:54
SPEAKER_01: For the analytic part, it actually writes a Python program.
1:59
SPEAKER_01: Why do I love it?
2:00
SPEAKER_01: I love it for a few reasons.
2:03
SPEAKER_01: First of all,
2:03
it means I don't have to learn code or to remember code or to
2:07
open one of my many
2:08
books that have code in them to remind me how to write a
2:12
specific program.
2:14
SPEAKER_01: It writes it for me.
2:15
SPEAKER_01: I love it also because it works.
2:18
does everything in python that means that even large data sets
2:21
and i have quite a
2:22
few of those with million lines of data the usual applications
2:27
like spss cannot
2:28
handle that because you can't transfer it through an excel
2:32
sheet and all of that
2:33
creates a cascade of problems that we're trying to solve and
2:37
they are solvable
2:38
SPEAKER_01: inside julius ai but it is far from perfect
2:41
SPEAKER_02: Yeah, yeah.
2:42
SPEAKER_02: And I have a question.
2:43
SPEAKER_02: And it's a very,
2:44
SPEAKER_02: I don't know,
2:45
simple question in terms of I remember that I created an
2:50
account with Julius AI,
2:51
SPEAKER_02: I don't know how many months ago,
2:54
SPEAKER_02: and I never get to try it really.
2:57
SPEAKER_02: So how is the user interface of that AI tool compared to ChatGPT?
3:01
SPEAKER_02: Yeah.
3:02
SPEAKER_02: which everybody believes it's very straightforward.
3:04
SPEAKER_01: Yeah, so it's just as straightforward.
3:07
SPEAKER_01: So you get this window, you get the plus button where you can add.
3:11
SPEAKER_01: It reminds you of ChatGPT.
3:13
SPEAKER_01: I mean, it's really, really similar.
3:16
SPEAKER_01: You can add tools,
3:17
SPEAKER_01: you can make some adjustments,
3:19
SPEAKER_01: and you can bring your data in,
3:21
SPEAKER_01: or you can ask it actually to go and find data,
3:23
SPEAKER_01: which is really curious.
3:25
SPEAKER_02: So I don't need to go through many options or features as I do with many
3:29
SPEAKER_02: statistical platforms or software.
3:32
SPEAKER_01: That's very true.
3:33
SPEAKER_01: So it is very user-friendly to begin with.
3:38
And that's part of its allure,
3:40
but also part of the potential warning sign.
3:44
And that is the fact that it's simple does not necessarily mean
3:47
you're going to get
3:47
what you want because you still have to understand what you're
3:50
asking it to do.
3:51
SPEAKER_01: But it does do interesting things.
3:54
and as it allows you to think through a problem and one of the
4:01
things that i love
4:02
about all ai and definitely this ai is that you can iterate
4:06
quickly so even if
4:08
you're analyzing a lot of data you give it a command it works
4:12
great it doesn't
4:14
SPEAKER_01: great work you just try it a different way and you
4:17
there's a way to make sure that you're getting better and
4:20
better quality as you move down.
4:22
SPEAKER_01: The other thing that it does really well is with one command,
4:25
SPEAKER_01: or even sometimes without a command,
4:27
SPEAKER_01: it'll produce a figure.
4:29
SPEAKER_01: And so you can actually see the results,
4:32
SPEAKER_01: which is really,
4:34
SPEAKER_01: really helpful for anybody who's doing a quantitative analysis,
4:37
SPEAKER_01: being able to see a box and plot or a scatter plot or...
4:44
a time series or any of those figures really helps you figure
4:48
out what's going on
4:49
SPEAKER_01: and to quickly have a sense.
4:50
Do you need to prompt Julius AI to get a figure or are you
4:54
talking about the
4:56
figures like Boxplot and everything that we normally get from
4:59
statistical software?
5:01
SPEAKER_02: Yes.
5:01
SPEAKER_01: So sometimes it's a classic AI.
5:04
SPEAKER_01: Sometimes it gives you them and sometimes not.
5:06
SPEAKER_01: It's hard to know.
5:07
SPEAKER_01: My advice is always,
5:09
SPEAKER_01: actually based on conversations we've had about prompting,
5:11
SPEAKER_01: is just be clear.
5:13
I want to see a boxplot or I want to see a time series or
5:18
whatever it is that you want.
5:20
SPEAKER_01: It's much easier if you just say it.
5:22
SPEAKER_01: The advantage here is that you can do it in plain language.
5:26
And if you missed it the first time,
5:27
you don't have to run the whole thing.
5:29
SPEAKER_01: You can just say, please add a boxplot.
5:31
SPEAKER_01: So even if you forgot it in the initial prompt,
5:33
SPEAKER_01: you can catch yourself saying,
5:36
SPEAKER_01: oh,
5:36
SPEAKER_01: I'd like to see that or that's not so good.
5:39
And does it give you any options for customizing those visuals
5:44
like the thing that SPSS does?
5:45
SPEAKER_02: Yeah.
5:46
SPEAKER_01: It does.
5:48
SPEAKER_01: Because it's more open-ended,
5:49
SPEAKER_01: you have to think about and you have to know what you're looking for.
5:52
SPEAKER_01: So,
5:53
SPEAKER_01: for example,
5:54
SPEAKER_01: just recently this weekend,
5:56
SPEAKER_01: I played with some data and I asked for a histogram.
6:01
SPEAKER_01: And it produced a histogram,
6:02
SPEAKER_01: which was okay,
6:04
SPEAKER_01: but the bins were not working for me,
6:08
SPEAKER_01: which means the ranges within the boxes were not working for me.
6:11
SPEAKER_01: So I just said, hey, I want a different bin.
6:14
SPEAKER_01: Mm-hmm.
6:15
SPEAKER_01: So you can re-prompt and you can ask more specifically.
6:20
If not,
6:21
much like some statistical software does,
6:23
it'll give you an answer.
6:26
But you can then sharpen it and make it better and tell it to
6:29
do it in a different way.
6:31
SPEAKER_01: Or you can just leave it the way it is.
6:34
SPEAKER_01: But you have a lot of freedom to make it better over time.
6:37
SPEAKER_01: And again, that's where the iteration comes in.
6:40
SPEAKER_01: So prompting and then prompting.
6:44
SPEAKER_01: finding the way to make it the best representation that you can.
6:47
SPEAKER_01: Sure.
6:48
SPEAKER_02: Yes, yes.
6:49
SPEAKER_02: Always helps.
6:51
SPEAKER_01: And so I've been using it extensively.
6:55
Part of the comparison for me is how certain are you that what
6:58
you're getting is quality?
6:59
SPEAKER_01: And the answer is fairly certain.
7:03
Although I have to,
7:05
and this is one of the advantages,
7:06
is it tells you exactly what it did.
7:08
SPEAKER_01: And if...
7:09
SPEAKER_01: You're going through the description.
7:11
SPEAKER_01: It tells you, this is what I thought to do.
7:14
SPEAKER_01: This is what I did with the data.
7:16
SPEAKER_01: This is what I did next.
7:17
SPEAKER_01: This is what I did next.
7:18
SPEAKER_01: And then you can say, no, no, no, no, no, no, no.
7:21
SPEAKER_01: This is not what I wanted.
7:22
I did not want you to choose only the first 15 things or
7:25
something like that.
7:26
SPEAKER_02: So it's not just short comments or short phrases.
7:29
SPEAKER_02: It actually tells you in complete sentences what it does with the data.
7:33
SPEAKER_02: Yes.
7:33
SPEAKER_01: Yes.
7:34
SPEAKER_01: So it does.
7:35
So it shows you a brief one and you can click on show more and
7:39
it says,
7:39
this is what I did.
7:40
SPEAKER_01: First I loaded this,
7:41
SPEAKER_01: then I identified,
7:42
SPEAKER_01: then I converted,
7:43
SPEAKER_01: then I created,
7:44
SPEAKER_01: then I added new column,
7:46
SPEAKER_01: then I saved it all.
7:47
Because every time it does something with your data,
7:49
it also saves everything.
7:50
And you can practically stop the AI if you feel that it's doing
7:54
something that you
7:54
SPEAKER_02: are not intending to do and then change that.
7:57
SPEAKER_02: Okay.
7:57
SPEAKER_01: So you can stop it.
7:59
SPEAKER_01: And you can also plan the analysis and then run it at different times.
8:05
SPEAKER_01: So you can have an agent running in the background,
8:07
SPEAKER_01: especially if you've got large data sets with complicated calculations.
8:12
You can kind of let it go,
8:14
go do something else,
8:15
open a different window and all of that.
8:17
SPEAKER_01: It'll run in the background and it'll let you know when you're done.
8:19
SPEAKER_00: Yeah.
8:20
SPEAKER_01: So this is fantastic.
8:22
SPEAKER_01: And they have templates.
8:25
SPEAKER_01: They have a community Slack if you want to get answers from other users.
8:29
SPEAKER_01: And you can actually create a team and create notebooks.
8:32
SPEAKER_01: So for each project that I have,
8:34
I actually create a notebook where you store everything so it's
8:38
easy to see.
8:39
SPEAKER_01: Because in the beginning, when you use anything new,
8:42
SPEAKER_01: You can find anything.
8:44
But then over time,
8:45
if it really becomes a tool you use,
8:48
it's going to be challenging.
8:49
SPEAKER_01: So you want to make sure that you're having this ability to organize the
8:55
information within the app and not just if you download it to
8:59
your computer.
9:00
I think this categorizing thing is becoming a common feature
9:03
among many AI tools,
9:05
SPEAKER_02: like the thing that we saw with perplexity last time.
9:09
You can also create labs or projects to keep everything related
9:13
in just one place.
9:15
SPEAKER_01: And also, I think, I mean, let's talk about the flip side.
9:19
SPEAKER_01: They want you to stay with their product.
9:21
SPEAKER_01: If they can make you create a complicated data structure and all of this
9:26
SPEAKER_01: information everywhere,
9:27
SPEAKER_02: you're much less likely that you migrate to other tools.
9:30
SPEAKER_02: Yeah.
9:31
SPEAKER_02: Makes a lot of sense.
9:31
SPEAKER_01: So.
9:33
SPEAKER_01: And my advice is actually not to migrate.
9:34
SPEAKER_01: Get really good with a reasonably good tool and stay with it.
9:39
SPEAKER_01: Don't change.
9:40
SPEAKER_02: Yeah.
9:41
SPEAKER_02: Okay.
9:41
SPEAKER_02: Makes sense.
9:42
SPEAKER_02: And I know that you analyze a huge data set with that.
9:46
So the advantage always for me working with Python because
9:49
we've had a large data
9:51
set that we tried to analyze and we realized that neither of us
9:54
knows Python.
9:55
SPEAKER_02: Yeah, unfortunately.
9:57
SPEAKER_02: I know a little bit.
9:58
SPEAKER_01: Yeah, a little bit.
10:00
SPEAKER_01: I know nothing.
10:01
SPEAKER_01: I know it's a snake.
10:03
SPEAKER_02: But it doesn't count, to be honest.
10:06
SPEAKER_01: It counts, but not enough to make a difference right here, right now.
10:10
SPEAKER_01: And also, even if you knew it once...
10:15
SPEAKER_01: The problem is that if you don't remember,
10:17
you need books and you need to remember and it's going to be a
10:19
struggle in the
10:20
SPEAKER_01: beginning.
10:20
So we looked for somebody to do the work with us and eventually
10:23
we found somebody.
10:24
SPEAKER_01: But when you have something like Julius Seha,
10:26
you can actually figure it out yourself because it is a huge
10:30
help as you're doing
10:33
SPEAKER_01: that kind of work.
10:34
SPEAKER_02: I think you also,
10:35
SPEAKER_02: when we were talking,
10:36
SPEAKER_02: you were also mentioned about Julius AI categorizing the data.
10:40
SPEAKER_02: Could you explain about that?
10:41
SPEAKER_01: So when you have a data structure and there are too many groups,
10:45
SPEAKER_01: one of the things you can ask it to do is to categorize within...
10:51
SPEAKER_01: try to find categories.
10:53
And that's where it leans on the classic generative AI and
10:57
creating semantic
10:58
SPEAKER_01: categories,
10:59
SPEAKER_01: bunching them together,
11:00
SPEAKER_01: they will tell you.
11:01
SPEAKER_01: You can ask exactly how the categories were created.
11:05
SPEAKER_01: But this is really efficient.
11:06
SPEAKER_01: Instead of creating a script that recodes everything,
11:10
SPEAKER_01: it does this very quickly,
11:12
SPEAKER_01: and then we're moving on.
11:15
Something I don't appreciate is sometimes it translates the
11:17
whole data sets
11:19
to a new data set that is the summary of the original data set
11:24
and sometimes you
11:25
want to stay with the original data set but that's a that's a
11:29
really it's about me
11:30
not prompting well enough and we've talked about this i try to
11:33
prompt as little as
11:34
SPEAKER_01: possible and to make ai
11:37
SPEAKER_01: Figure out out of the thin air what I wanted to say.
11:40
SPEAKER_01: Read my mind.
11:42
SPEAKER_01: And that doesn't work very well.
11:44
SPEAKER_01: So you have to prompt.
11:46
SPEAKER_01: And this is where the difference is.
11:47
SPEAKER_01: You cannot use Julius AI without knowing quite a bit about statistics.
11:52
SPEAKER_01: So whatever you're trying to figure out how to analyze,
11:54
SPEAKER_01: you have to at least have a sense of that,
11:56
SPEAKER_01: how that works.
11:58
And what are the data structures and what are the expectations
12:02
and what are the
12:02
SPEAKER_01: rules about that?
12:04
SPEAKER_01: If you do, you will get greater results.
12:06
If you're just asking AI to do this for you with no supervision
12:11
and no ability to
12:12
SPEAKER_01: see critically if it's doing the right thing,
12:15
SPEAKER_01: you might get something useful,
12:17
SPEAKER_01: but you will not know if it's useful or not.
12:19
SPEAKER_01: And that's exactly where we're having a difficulty.
12:22
SPEAKER_01: And that brings us to the paper we're reviewing because...
12:27
SPEAKER_01: That's exactly what they looked at.
12:28
SPEAKER_01: So tell us a little bit more about that.
12:31
So the title of the paper that I'm going to talk about is The
12:34
Use of Generative AI
12:36
in Statistical Data Analysis and Its Impact on Teaching
12:40
Statistics at Universities
12:42
SPEAKER_02: of Applied Sciences.
12:43
The author of that paper is Joachim Schwartz,
12:47
and I think they did this study in Germany.
12:50
SPEAKER_02: So if I want to summarize the study,
12:53
the study suggests that the focus of statistics education
12:56
should shift from
12:57
SPEAKER_02: teaching software-specific skills to fostering a deeper understanding of
13:02
SPEAKER_02: statistical concepts,
13:04
SPEAKER_02: which I find really helpful and useful,
13:07
SPEAKER_02: both for also faculty and also for students.
13:10
So this shift will prepare students to effectively use AI tools
13:15
while maintaining
13:16
SPEAKER_02: rigorous standards of data analysis.
13:20
So this was trial with students or were they doing a benchmark
13:28
on how well AI
13:32
SPEAKER_01: responds to prompts?
13:33
SPEAKER_01: Yeah.
13:33
SPEAKER_02: They did not have actual participants.
13:36
SPEAKER_02: So they mostly, as you said, tried to benchmark AI tool.
13:41
SPEAKER_02: And the AI tool that they did the study about is GPT Data Analyst.
13:46
It's one of those customized GPTs that when you log into GPT,
13:50
you can have those things.
13:51
SPEAKER_02: And this one is also available through ChatGPT Plus version.
13:56
SPEAKER_02: So it's not free.
13:57
SPEAKER_02: You cannot use it for free.
13:59
SPEAKER_02: Sure.
13:59
So it's basically one of the specific GPTs that were created
14:04
out of the large scheme.
14:05
SPEAKER_01: Very true, yes.
14:07
So it's their effort to be a store where you can get everything
14:11
instead of having
14:12
SPEAKER_01: to jump to something like Julius.
14:15
SPEAKER_02: Very true, yes.
14:17
SPEAKER_02: So I have some information about the methods of the study,
14:20
SPEAKER_02: about their findings,
14:21
SPEAKER_02: and also about the key ideas.
14:24
SPEAKER_02: So which part are you more interested in?
14:27
SPEAKER_01: Well, let's start with the research question and then move to methods.
14:31
SPEAKER_01: I mean, I think it's important to know where we're going with this.
14:34
SPEAKER_02: Yeah.
14:35
The research question was mostly checking the ability of chat
14:40
GPT data analysts in
14:42
SPEAKER_02: analyzing the data.
14:43
SPEAKER_02: They tried different kinds of data sets,
14:46
SPEAKER_02: and I need to mention that all of them were artificially created,
14:50
SPEAKER_02: so they are not real data.
14:52
But the thing that they did is that they changed the data sets
14:57
in terms of the
14:58
SPEAKER_02: number of observations and also violations.
15:01
Because we know that violations are an important thing when you
15:06
are doing the
15:06
SPEAKER_02: quantitative data analysis in terms of choosing the statistical tests,
15:10
SPEAKER_02: interpreting the data,
15:12
SPEAKER_02: and they wanted to see how AI can deal with those kind of things.
15:17
SPEAKER_02: So they had four artificially generated data sets.
15:21
SPEAKER_02: For ANOVA test cases,
15:23
they had one data set with slight violations and the other data
15:27
set with serious
15:28
SPEAKER_02: violations.
15:28
SPEAKER_02: By violations, I mean missing values and also outliers.
15:33
SPEAKER_02: For other tests,
15:34
one of the data sets was suitable for a two-sample t-test and
15:37
the other one was
15:38
SPEAKER_02: suitable for a linear regression.
15:40
SPEAKER_01: Okay.
15:41
So basically they've done something similar to the work we've
15:44
done with qualitative.
15:46
SPEAKER_01: Sure.
15:46
SPEAKER_01: And that is they produced conditions.
15:50
So they're not talking about the user yet because they want to
15:54
know what's
15:54
SPEAKER_01: happening with the machine.
15:55
SPEAKER_00: Yeah.
15:56
SPEAKER_01: Right.
15:57
SPEAKER_01: And they're testing it on things where they know the properties.
16:02
SPEAKER_01: So they know what's in the data.
16:05
SPEAKER_01: They know what it looks like and what they should get, which...
16:08
It strengthens their ability to say,
16:11
oh,
16:11
it can deal with this and not that.
16:13
SPEAKER_01: And there's,
16:14
SPEAKER_01: by the way,
16:14
a long history of these kind of studies about usually it used
16:20
to be the sensitivity
16:21
SPEAKER_01: of specific patients.
16:24
analyses to violations so for example they would you do an
16:28
ANOVA and say what
16:29
happens if you have one big outlier what happens if you have 10
16:33
those kind of
16:33
things and they would try it repeatedly to get the result and
16:38
know what are the
16:38
boundaries of statistical analysis and when it completely
16:42
breaks down yeah yeah and
16:44
so this is a similar approach and I love that because it does
16:46
tell us what it can
16:47
SPEAKER_01: and can't do so what did we find
16:49
SPEAKER_02: So one more thing.
16:51
SPEAKER_01: Sorry.
16:52
SPEAKER_01: What did Joachim find?
16:55
SPEAKER_02: Very true.
16:55
SPEAKER_02: But I need to mention one thing before I move to the findings.
16:58
SPEAKER_02: So they also had test user assumptions.
17:02
SPEAKER_02: So based on their assumption,
17:03
SPEAKER_02: the user is a person with no formal statistics training,
17:06
SPEAKER_02: which is very important.
17:07
SPEAKER_00: Yeah.
17:08
SPEAKER_02: Many students,
17:09
SPEAKER_02: I'm talking about undergrad students,
17:12
SPEAKER_02: at the beginning,
17:13
SPEAKER_02: they have no formal statistical knowledge.
17:16
SPEAKER_02: So it's hard for them to understand many things.
17:19
SPEAKER_02: And going to AI nowadays is a solution that everybody...
17:24
So this study and the findings have important implications from
17:29
that perspective as
17:30
SPEAKER_02: well.
17:30
SPEAKER_02: So findings...
17:34
SPEAKER_02: So in terms of guidance ability,
17:37
SPEAKER_02: when asked,
17:38
SPEAKER_02: ChatGPT could outline general analysis steps,
17:41
SPEAKER_02: just very similar to what you said about Julius AI.
17:45
SPEAKER_02: So it can talk about hypothesis formulation,
17:48
SPEAKER_02: descriptive statistics,
17:50
SPEAKER_02: visualization,
17:51
SPEAKER_02: and also hypothesis testing.
17:53
SPEAKER_02: But the thing that they found is that this advice was incomplete, right?
17:59
And usually mentioned normality checks,
18:01
but not equal variances,
18:03
outliers or missing values.
18:05
So these concepts are very important when you are doing a
18:08
statistical analysis.
18:09
SPEAKER_02: However, AI is not able to notice all of those violations.
18:14
SPEAKER_01: At least not consistently.
18:15
SPEAKER_02: Yeah, yeah, very true, yes.
18:17
SPEAKER_02: So I will talk about the ANOVA data set.
18:21
SPEAKER_02: ChatGPT performed full analysis,
18:23
SPEAKER_02: descriptive statistics,
18:25
SPEAKER_02: box plots,
18:25
SPEAKER_02: and ANOVA,
18:26
SPEAKER_02: and it correctly found no statistical significance.
18:31
But at the same time,
18:32
it ignored issues like unequal variances and also a small
18:36
sample size.
18:37
SPEAKER_01: Okay.
18:38
SPEAKER_01: Something that gets me really riled up is we were doing some reviews for
18:45
SPEAKER_01: publications this morning,
18:46
SPEAKER_01: and there was a very small sample size that got me hot and bothered.
18:51
SPEAKER_01: So I apologize to whoever wrote that review for.
18:54
But you cannot run very small samples with any kind of
19:00
reliability or generalizability and...
19:03
SPEAKER_01: And we have been doing research for a long time.
19:06
SPEAKER_01: So we know how this should be done well.
19:09
SPEAKER_01: And now we're asking, can ChatGPT help us or will it tell us?
19:12
SPEAKER_01: And obviously it does not tell us we are going wrong.
19:15
SPEAKER_02: Yes.
19:16
And for example,
19:17
for linear regression data set,
19:20
it always produced regression results.
19:22
SPEAKER_02: And it only once mentioned assumptions of regression,
19:25
SPEAKER_02: which are linearity and also almost cadasticity.
19:29
SPEAKER_02: But it never tested them.
19:31
SPEAKER_02: And what was the name of that?
19:35
GPT analyst also once added correlation checks before
19:41
regression,
19:42
but inconsistently.
19:44
So really what I'm hearing is what we've been discussing in
19:47
other things.
19:48
SPEAKER_01: And that is we've got to prompt it more accurately.
19:51
SPEAKER_01: If we just say test this thing, then it's going to be all over the place.
19:58
So just like writing a program,
19:59
we still have to write a program,
20:01
but we do it in plain English.
20:03
SPEAKER_01: But you can't assume that the AI, at least at this point, right?
20:07
SPEAKER_00: Yeah.
20:08
we can't assume that it'll know by itself what are all the
20:12
steps that need to do to
20:15
SPEAKER_01: create a quality product.
20:16
Yeah,
20:17
it doesn't do the mind reading that everybody expects from it,
20:20
to be honest.
20:22
So the key ideas that we get from the paper,
20:24
AI makes data analysis accessible.
20:26
SPEAKER_02: Even people with little statistical knowledge can run analysis now,
20:31
SPEAKER_02: but ChatGPT can create code and explain results.
20:35
so it's not flawless the most important thing is that it's not
20:39
flawless and also
20:41
human oversight remains essential without a statistical
20:45
knowledge users may not
20:46
even notice these gaps or yeah errors in the data and it has
20:51
also some implications
20:52
for teaching so instead of focusing so much on learning
20:56
software like r and excel
20:58
SPEAKER_02: or python
21:00
SPEAKER_02: Courses should emphasize on statistical concepts and critical thinking.
21:05
So it would be great if they could enable students to just
21:10
analyze and interpret
21:11
SPEAKER_02: the results instead of just running tests.
21:15
SPEAKER_01: And the thing that I love is...
21:18
SPEAKER_01: that because the output is a little bit uncertain,
21:22
SPEAKER_01: a little bit all over the place,
21:23
SPEAKER_01: it forces us to be more critical,
21:25
SPEAKER_01: which gives a great opportunity for us to see,
21:29
SPEAKER_01: for example,
21:30
SPEAKER_01: for everybody to run it three times,
21:32
SPEAKER_01: look at the differences,
21:34
SPEAKER_01: and think about what does that mean,
21:36
which run was with the more information and more appropriate
21:40
and which run wasn't.
21:41
Yeah,
21:41
and last but not least,
21:42
generative AI won't replace a statistics education.
21:46
SPEAKER_02: It shifts its purpose.
21:48
SPEAKER_02: So students still need a strong statistical reasoning so they can use AI
21:52
SPEAKER_02: responsibly,
21:53
SPEAKER_02: check its work,
21:55
SPEAKER_02: and avoid blind trust in its outputs.
21:58
SPEAKER_01: Yeah, so do we have new tools that we can play with?
22:02
SPEAKER_01: Yes.
22:03
SPEAKER_01: Does it mean we don't have to work anymore?
22:06
SPEAKER_02: No.
22:07
SPEAKER_01: No,
22:07
and which is very good because we still need to get a paycheck
22:10
and feed children
22:12
SPEAKER_01: and grandchildren and all of that,
22:15
SPEAKER_01: or whoever.
22:17
SPEAKER_01: And so it is really important to know that at this point,
22:23
SPEAKER_01: the human is absolutely essentially in the loop,
22:26
SPEAKER_01: both from an interpretive but also from a supervisory point of view,
22:29
SPEAKER_01: making sure everything is taken into account.
22:32
SPEAKER_01: And so can you, again, bottom line, great tools.
22:37
SPEAKER_01: We have to be diligent.
22:39
SPEAKER_02: Yeah.
22:39
SPEAKER_02: We need to check.
22:40
SPEAKER_02: You need to run the tests several times to make sure they are consistent.
22:46
SPEAKER_02: And you need to have foundational knowledge.
22:49
SPEAKER_01: And we'll see you next time.