Latest / Ethical Machines / Rewriting History with AI

Transcript

Rewriting History with AI · Ethical Machines

0:01
[SPEAKER_03]: So large language models are good for lots of things.
0:03
[SPEAKER_03]: And of course, one thing that you might do, especially if you're a student, is that you might ask it questions about history.
0:09
[SPEAKER_03]: You might want to know how to certain kinds of events unfold in the world.
0:12
[SPEAKER_03]: So you might ask it about the Vietnam War, with the US.
0:15
[SPEAKER_03]: You might ask it about the Russia Ukraine War right now.
0:18
[SPEAKER_03]: You might ask it about Israel and Gaza.
0:21
[SPEAKER_03]: And I love this, of course, give you lots of different kinds of answers.
0:25
[SPEAKER_03]: The thing is, though, there are certain problems with this, as my guest points out.
0:28
[SPEAKER_03]: My guest today is Nuno Moni, he's an associate research professor at the University of Notre Dame, and he points out that there are at least three kinds of basic problems for why we should worry by this kind of thing.
0:40
[SPEAKER_03]: Number one,
0:40
[SPEAKER_03]: There are hallucinations, right?
0:42
[SPEAKER_03]: LMS just make things up and it makes up historical narratives.
0:46
[SPEAKER_03]: That's a bit problem.
0:48
[SPEAKER_03]: There are people who spread misinformation, disinformation online and of course, since the algorithms that power the LMS are searching the internet, they can be pulling misinformation and then presenting it to students, teachers, the average citizen, as though it were true.
1:04
[SPEAKER_03]: And then there's the fact that
1:06
[SPEAKER_03]: history is contentious.
1:07
[SPEAKER_03]: We argue about how to interpret the facts.
1:09
[SPEAKER_03]: We think that different things might have caused this.
1:11
[SPEAKER_03]: We blame these people.
1:12
[SPEAKER_03]: We blame those people who say this was the main thing.
1:14
[SPEAKER_03]: No, that was the main thing.
1:15
[SPEAKER_03]: And we get a lot of legitimate disagreement about how to interpret these historical facts and create the appropriate historical narratives.
1:25
[SPEAKER_03]: If you like flood the zone with a particular interpretation of history, that's going to crowd out the other legitimate
1:36
[SPEAKER_03]: B.A.
1:36
[SPEAKER_03]: loan individuals, B.A.
1:38
[SPEAKER_03]: certain groups of people, B.A.
1:39
[SPEAKER_03]: State actors, let's say Russia trying to enforce a certain kind of force even.
1:44
[SPEAKER_03]: Suddenly or not, so suddenly manipulate the narrative using LLMs by flooding the internet with false narratives about how the Russian Ukraine war started.
1:53
[SPEAKER_03]: You've got a real bit problem on your hands.
1:55
[SPEAKER_03]: Meanwhile, as Nuno points out, teachers are not aware of these kinds of problems, students are aware of these kinds of problems, the average citizen is not aware of these kinds of problems, and while there might not be a kind of watershed moment where, oh my god, the L.M.
2:09
[SPEAKER_03]: just made stuff off about some history above a lot.
2:11
[SPEAKER_03]: You'll find bits and pieces up here and there.
2:13
[SPEAKER_03]: The cumulative effect of this, you know, says over years, decades, generations, it's created totally distorted understanding of what happened in history, how people got into certain kinds of conflicts, where certain kinds of people came from, so and and so forth.
2:27
[SPEAKER_03]: So you can see how
2:30
[SPEAKER_03]: Getting confused about, to put it lightly, confused about what's gone on in history, what sort of made us so on and so forth, can lead to verbal disputes that lead to violent disputes that lead to geopolitical disputes, it's ultimately potentially very dangerous stuff.
2:50
[SPEAKER_03]: Conversation goes all through this sort of thing with no no he also has three recommendations on how he can avoid this not so great future and but honestly it's one of these conversations that I started with all right I guess I get the issue yeah bad things about history or fall some of history not great.
3:07
[SPEAKER_03]: but Nuno does such a great job of bringing out the problem and showing what happens when this problem scales that by the end of it, I'm pretty worried.
3:15
[SPEAKER_03]: I'm pretty worried, I didn't mean something to do, and thankfully Nuno has some good recommendations.
3:19
[SPEAKER_03]: Okay, the last thing before we have that conversation we'll know, I have an announcement, I have a new book coming out, probably in April of 2026, the title over the book is The Ethical Nightmare Challenge, and it explains how
3:31
[SPEAKER_03]: the way in which AI is getting increasingly complex, moving from narrow to generative to now a gentic and then multi-agentic AI.
3:39
[SPEAKER_03]: The risk landscape, the possible nightmares that we might get from AI, just explodes.
3:43
[SPEAKER_03]: It's an extremely complex problem.
3:45
[SPEAKER_03]: And the standard ways that we do AI ethics in a business or a corporate or any sort of large organization context, the way that we do that completely falls apart.
3:55
[SPEAKER_03]: And what I offer is a novel approach to dealing with the potential nightmares of AI.
4:00
[SPEAKER_03]: How to avoid it.
4:00
[SPEAKER_03]: So the book is called The Ethical Nightmare Challenge.
4:03
[SPEAKER_03]: How to avoid the worst of AI.
4:04
[SPEAKER_03]: It'll come out sometime in April.
4:06
[SPEAKER_03]: I'll be saying lots about it on the podcast at some point.
4:09
[SPEAKER_03]: I'll be posting about it on Substack and LinkedIn, blah, blah.
4:12
[SPEAKER_03]: If you want to follow what's going on with it, you want notifications about when things are coming out, when you can buy it, et cetera.
4:17
[SPEAKER_03]: You can go to my sub-stack, read blackman.substack.com.
4:21
[SPEAKER_03]: Just subscribe there and you'll get, every once in a while, send an update about the book and maybe share some material in advance with the release of the book.
4:28
[SPEAKER_03]: So it'll also be an audio book and stuff like that.
4:31
[SPEAKER_03]: Okay, that's it about me in my book.
4:34
[SPEAKER_03]: Let's get on to this really great terrifying cautiously optimistic conversation with new nominees.
4:39
[SPEAKER_03]: And by the way, if you have any questions, comment, et cetera, email me at bnf3flat.com.
4:43
[SPEAKER_03]: Okay, let's go talk to you now.
5:00
[SPEAKER_03]: All right, so, you know, you're predicting, if I have you right, that there's something not the way that AI operates or is operating at the moment, that is going to lead to things like intergenerational fighting, cultural, and fighting, and I think you use the phrase cognitive warfare in a previous discussion.
5:16
[SPEAKER_03]: Do I have you right?
5:17
[SPEAKER_03]: That's what you're predicting.
5:18
[SPEAKER_03]: And if so, how the hell do we get there?
5:20
[SPEAKER_01]: So, great question.
5:21
[SPEAKER_01]: I think it's one way of putting it.
5:24
[SPEAKER_01]: I would phrase it in the following way.
5:27
[SPEAKER_01]: What we were seeing in two specific ways is that chatbots and modern AI has gathered a massive kind of adherence and numbers of users unique trajectory all of that, but more than that it has gathered the traditional impact in relationship that people have with technology which is some sort of
5:54
[SPEAKER_01]: optimism towards that, thinking that the things that come out of it are some sort of authority, the statement and especially believe it.
6:02
[SPEAKER_01]: Now, this is fine.
6:04
[SPEAKER_01]: Maybe if I see something there that I recognize and it's wrong, the issue with
6:11
[SPEAKER_01]: A lot of things in AI is when you look further than just the immediate impact.
6:17
[SPEAKER_01]: When you picture a room of 10, 20 students, let's say 13 to 15 years old, getting their first crack at, for example, historical landmark events that define the culture and identity that you're living in.
6:34
[SPEAKER_02]: Yeah.
6:35
[SPEAKER_01]: If those kids go home and start using chatGPT, called, what's on a, and all of them repeat the same query.
6:44
[SPEAKER_01]: First, you'll probably find different answers.
6:48
[SPEAKER_01]: And that speaks a little bit to the variability in terms of the historical narrative that they would find.
6:54
[SPEAKER_01]: And as we also know, if you dig a little deeper, not only you will get a different set of events, you'll probably get a few wrong ones.
7:04
[SPEAKER_01]: When you scale this in terms of generation, what you will have essentially is historical confusion.
7:11
[SPEAKER_01]: You have a new generation that is currently using these tools to acquire and get their first crack at information that defines their identity.
7:21
[SPEAKER_01]: And I can't express how important that is in terms of just that early onset and the discussions that it has.
7:30
[SPEAKER_01]: And for me,
7:31
[SPEAKER_01]: It just raises a ton of questions that I myself am still grappling in terms of the offshoots of what this means, not only in education, but also when you go broader in terms of just a cultural sense.
7:43
[SPEAKER_01]: Yeah.
7:43
[SPEAKER_01]: And just to answer that, a bit that you said about cognitive warfare, one particular aspect that we know here is that
7:53
[SPEAKER_01]: is not that difficult if you have the resources to bias the narratives that you will find in these tools.
8:00
[SPEAKER_01]: And so this basically spells out a new vector of cognitive warfare where actors not only state, but you can think of state actors.
8:10
[SPEAKER_01]: can flood the internet with certain narratives that ultimately have a great possibility of being imprinted and the responses that are popping up.
8:20
[SPEAKER_01]: And this has already been seen for example, in Ukraine and Russia, for example.
8:26
[SPEAKER_03]: Okay, so yeah, let me see if I have this right or the sort of the big picture view.
8:30
[SPEAKER_03]: So first of all, I guess we back up even a little bit.
8:34
[SPEAKER_03]: I don't know if we're backing up or just moving to the side.
8:36
[SPEAKER_03]: We're talking about going to LLM.
8:38
[SPEAKER_03]: So any large language model like the ones you mentioned for the news.
8:42
[SPEAKER_03]: You've got to get different news stories from different algorithms, right?
8:44
[SPEAKER_03]: So some are going to be a little bit right leaning or very much right leaning.
8:47
[SPEAKER_03]: Some are going to be left leaning or very much left leaning.
8:49
[SPEAKER_03]: And so depending on where you're going to get your information, there's two kinds of issues.
8:53
[SPEAKER_03]: One that you mentioned, I think, one is the bias issue.
8:56
[SPEAKER_03]: So where is the information coming from and is it bias in some way?
8:59
[SPEAKER_03]: Secondly, is it truthful at all, so it's one thing to have a perspective.
9:04
[SPEAKER_03]: It's another thing for that perspective to completely misrepresent what's going on.
9:07
[SPEAKER_03]: And if you have AI hallucinations, then you have not just a different perspective, you have outright misrepresentation.
9:14
[SPEAKER_03]: And then I take it the further point you're making is, at least two more further points.
9:18
[SPEAKER_03]: One is,
9:20
[SPEAKER_03]: that when histories are either fabricated or presented in a lopsided manner, if it's just the news, the news being lopsided or fake leads to lots of antagonism between people among people.
9:31
[SPEAKER_03]: And we'll see the same thing with historical narratives.
9:33
[SPEAKER_03]: So it's not just what's happening right now.
9:35
[SPEAKER_03]: It's what happened last year, five years ago, 10 years ago, 100 years ago, and 20.
9:40
[SPEAKER_01]: Yes, yes, keep going.
9:41
[SPEAKER_01]: I'll give you this just.
9:43
[SPEAKER_01]: for you to picture what we actually did in what's the kind of where I'm coming from.
9:48
[SPEAKER_01]: Yeah, research rise in workplace.
9:51
[SPEAKER_01]: So two years ago, I got an email from someone saying, hey, there's this person that wants to talk with you, would you meet with her?
9:58
[SPEAKER_01]: My colleague Yajna from the Catholic University in Croatia, it was working in memory modulation.
10:06
[SPEAKER_01]: So essentially, how do you strengthen and bias your memories towards a certain narrative?
10:11
[SPEAKER_01]: This is very important, for example, in post conflict situations where you work on public furniture, you work on museum, text books, and what's not so that you create narratives that kind of bind people together.
10:22
[SPEAKER_01]: That's an example.
10:24
[SPEAKER_03]: So can we get a little more concrete with that first?
10:25
[SPEAKER_03]: So what does that mean?
10:27
[SPEAKER_03]: Can you give me an actual concrete example where something has happened and now we're engaged in what you just called memory modulation?
10:34
[SPEAKER_01]: Exactly, so that can happen in post-conflasely, for example, the Balkans, which is where yes, is coming from.
10:42
[SPEAKER_01]: After the war in the Balkans of the 90s, you essentially, as a fate, mainly as a state, you have the need to explain that in a historical narrative or a kind of an alternative that you're going to, for example, put in textbooks that you're going
11:01
[SPEAKER_01]: memorialized names in streets you put up statues you change your museums or even create a museum about what happened in this and not so that's a way that you can strengthen and what's the next and bias that context and that narrative towards a set of things.
11:20
[SPEAKER_03]: Yeah, it's kind of creating a kind of cognitive and emotional backdrop against which you see the current goings on.
11:28
[SPEAKER_01]: Exactly, but also you determine also the interpretation of what has happened before and you see this here in this country and Portugal in my Homeland then all those things.
11:40
[SPEAKER_01]: Yeah, but the question that the fight over Confederate statues is that kind of exact Well, the story don't narrative history is not a fixed name always fluid and there's no single history and so it's a political debate And that's how it is but our discussion at some point when to a place where I just asked her
11:57
[SPEAKER_01]: to what an extent our LLM's already tools of memory modulation.
12:03
[SPEAKER_01]: What an extent.
12:04
[SPEAKER_01]: And so what we did was we set up the research project, with Yasin and the Catholic University of Croatia, with my friend Miguel Caradini and University of Queen but in Portugal, Italian men, and did the Gashka here at the University of Notre Dame, and we looked at four conflicts.
12:25
[SPEAKER_01]: or conflicts that we wanted to essentially study via a large language models.
12:31
[SPEAKER_01]: That was US Vietnam, war in the Balkans, Israel Palestine, and the Portuguese colonial wars.
12:40
[SPEAKER_01]: And we had some things that we wanted to look at.
12:44
[SPEAKER_01]: For example, the impact of language, right?
12:46
[SPEAKER_01]: If you look at the various languages that would be related to that conflict, are they telling the same story?
12:53
[SPEAKER_01]: We also wanted to test, for example, the era of time, constructing that narrative from beginning to end, but also from end to beginning.
13:03
[SPEAKER_01]: What we found,
13:05
[SPEAKER_01]: is quite interesting in terms of just the overlap of events.
13:12
[SPEAKER_01]: There's very little sometimes.
13:14
[SPEAKER_01]: If you shift languages, you will find very different narratives and when I say different, there's not because they are
13:24
[SPEAKER_01]: based on each country's relative to language, literature, or bibliographic references.
13:32
[SPEAKER_01]: It's just different because a lot of the events that show up there, especially in low-language, low-resource languages, is plainly rock.
13:41
[SPEAKER_01]: I'll give you an example.
13:42
[SPEAKER_01]: US Vietnam.
13:44
[SPEAKER_01]: If you reconstruct that historical narrative, what you will see is basically a sequence of victorious battles by the U.S. followed by and then Saigon fell.
13:55
[SPEAKER_03]: This is if you query an L.M.
13:57
[SPEAKER_03]: in English and ask something roughly, give me an overview of the Vietnam War.
14:02
[SPEAKER_01]: Yeah.
14:03
[SPEAKER_01]: Okay.
14:03
[SPEAKER_01]: And you will see an narrative that basically
14:06
[SPEAKER_01]: has nothing to say about what was going on in the US itself that determined to a certain extent how that evolved.
14:16
[SPEAKER_01]: And so it's just a flattening of history, historical narrative generated.
14:23
[SPEAKER_01]: has no way of mimicking or passing the message on the intricate nodes that are in play and do these things.
14:32
[SPEAKER_01]: But that's the thing, when you take just a face value that kids are going to use this, they are.
14:39
[SPEAKER_01]: I wish there were ways to conform that in a way, but it's going to happen.
14:44
[SPEAKER_01]: What you will see is essentially this, kids that are learning the same events in different languages who will have very different.
14:53
[SPEAKER_01]: narratives.
14:54
[SPEAKER_01]: They will see different things.
14:57
[SPEAKER_01]: They even kids in the same room will have that.
15:01
[SPEAKER_01]: And of course, yeah, go ahead.
15:02
[SPEAKER_03]: Sorry.
15:02
[SPEAKER_03]: So just so I'm clear to make this sort of as simple and dumb as possible.
15:07
[SPEAKER_03]: Is the idea that American kids quarry, chat, LGBT, and English?
15:11
[SPEAKER_03]: vietnamese kids quarry it in vietnamese and they get radically different outputs because there's something about the language that you're using to interact with the thing and perhaps the data that the thing was trained I don't know how actually how the translation works it just comes up with wildly different answers and it's well known if you ask an LLM and English the same question twice you're going to get
15:31
[SPEAKER_03]: different answers each time.
15:33
[SPEAKER_03]: But they're not going to be crazy different.
15:34
[SPEAKER_03]: They're not going to be unusual.
15:35
[SPEAKER_03]: They're just a little bit different.
15:37
[SPEAKER_03]: And this is actually asking the same question English versus Vietnamese gives you very different answers.
15:43
[SPEAKER_01]: Exactly.
15:44
[SPEAKER_01]: But the thing is when you're finding out something along these lines, you don't have one prompt.
15:51
[SPEAKER_01]: You started with one of the things we experimented, one of the tools that we experimented was with deep seek and forward memory and English.
16:12
[SPEAKER_01]: Looking at Israel, Palestine, it just went on, blew through today and stopped in 1948, where it would state that there was a piece of cord signed and everything was all right.
16:29
[SPEAKER_03]: So that's just made some shit up.
16:31
[SPEAKER_01]: Yes, it is.
16:33
[SPEAKER_03]: It's making things up.
16:34
[SPEAKER_03]: But can I ask you a technical question?
16:36
[SPEAKER_03]: Yeah.
16:38
[SPEAKER_03]: This is about, obviously when they first, is this obvious?
16:43
[SPEAKER_03]: Here's my question.
16:43
[SPEAKER_03]: Are we talking about with an LLM?
16:46
[SPEAKER_03]: Is there one that's just, there was a chatGPT that was trained on English text and a chatGPT that was trained on Vietnamese text?
16:53
[SPEAKER_03]: Or is it that it was trained simultaneously on English and Vietnamese text?
16:57
[SPEAKER_03]: And so it was multilingual from the start?
16:59
[SPEAKER_03]: Or is it that it was trained primarily in English?
17:02
[SPEAKER_03]: And then when you query and Vietnamese, it translate the Vietnamese to English.
17:06
[SPEAKER_03]: retrieves the English response and translates it to Vietnamese and outputs the Vietnamese text.
17:11
[SPEAKER_03]: What's underneath the hood here?
17:12
[SPEAKER_01]: usually you would have multi-language.
17:15
[SPEAKER_01]: And that is part of why we see this distinctive behavior because for different languages you'll have in a symmetric representation of the data that is there and the tales and different references.
17:30
[SPEAKER_01]: And thus that all crossed.
17:32
[SPEAKER_01]: Then I imagine that there are some transletive kind of modules in there that are at work.
17:40
[SPEAKER_01]: I want to clarify something because this is very important from the historical point of view and it's something that my historian and memory studies colleagues were very adamant explaining it is perfectly expected normal and it should be that
17:57
[SPEAKER_01]: The history that, for example, in a conflict, the history that you would read here from both parties, for example, would be different.
18:06
[SPEAKER_01]: Right?
18:07
[SPEAKER_01]: The U.S. have a certain narrative, historical narrative on, for example, the U.S. Vietnam War.
18:13
[SPEAKER_01]: Sure.
18:14
[SPEAKER_01]: The Vietnamese also have one.
18:15
[SPEAKER_01]: And maybe they're, they don't overlap completely.
18:18
[SPEAKER_01]: Sure.
18:19
[SPEAKER_01]: And that's completely fine.
18:20
[SPEAKER_01]: That just,
18:21
[SPEAKER_01]: explains the mosaic of different references.
18:24
[SPEAKER_01]: There's this beautiful book by eminoluf that is the history of creesates from the point of view of Arabs.
18:32
[SPEAKER_01]: And it's basically telling the story of the creesates solely using
18:37
[SPEAKER_01]: historical books, bibliographic references, and what's not from Arab countries.
18:43
[SPEAKER_01]: And it's completely different story.
18:45
[SPEAKER_01]: That should be.
18:46
[SPEAKER_01]: The thing here is not exactly that.
18:48
[SPEAKER_01]: We're not talking about the same thing, where you can confront these two things based on references.
18:54
[SPEAKER_01]: We're talking about made up things in most cases, especially in the low resource languages.
18:59
[SPEAKER_01]: And that is something that when you take it face value, you just escalate the number of users that these tools have worldwide.
19:08
[SPEAKER_01]: And that spells a very dangerous, I would say, situation.
19:15
[SPEAKER_03]: It's just another, not just another breakdown in our sort of shared reality.
19:21
[SPEAKER_01]: Yeah.
19:22
[SPEAKER_01]: but also in a thing that I think you coined, which is a Gentic monopoly.
19:27
[SPEAKER_01]: I did coined that phrase.
19:29
[SPEAKER_01]: Yeah, I heard that, I heard that, because that means, when, if you think a little bit about, that means that these companies hold a certain a Gentic monopoly over historical narratives.
19:44
[SPEAKER_01]: And that is something that I would,
19:50
[SPEAKER_01]: Yeah, what an extent that we comfortable with that.
19:52
[SPEAKER_03]: Okay, so I guess when I talk about agedic monopolies, I talk about slugging slightly different, but I think you're pointing something like, look, there's only a handful of these models of these foundation models.
20:02
[SPEAKER_03]: Right, there's clawed, there's Gemini, there's Deep Seek, there's, but there's not that many, right?
20:06
[SPEAKER_03]: I'm gonna guess, I don't know, it doesn't, three doesn't at a time.
20:10
[SPEAKER_03]: And even if there's many more than that, there's obviously like outside's market share for a chat chip,
20:20
[SPEAKER_03]: Yeah, let us thousand flowers bloom three companies is not a thousand flowers, if the vast majority of humanity is getting a lot of historical information from just three parties, we're getting a phenomenally biased monopolistic account of what history looks like that, but more than that because the way that we have so until today.
20:44
[SPEAKER_01]: Basically, we humans have been the only presumers of history.
20:48
[SPEAKER_01]: We are the only ones that produce it in consumer.
20:51
[SPEAKER_01]: That is no longer our reality.
20:54
[SPEAKER_01]: Like these tools by the view of their probabilistic nature, but also what we were just talking about in terms of multilanguage representation of events, they to a certain extent create some time is a very synthetic reality.
21:11
[SPEAKER_01]: Now, when we were talking about the history of a certain event in a certain country, sure, there's probably different ways to tell it.
21:20
[SPEAKER_01]: I'm sure there's people that would contend.
21:22
[SPEAKER_01]: This wasn't like this or the causal relations aren't that.
21:26
[SPEAKER_01]: But usually, that is binded or bounded to some sort of
21:34
[SPEAKER_01]: Like, there's a museum.
21:36
[SPEAKER_01]: There's a series of documents that someone's studying.
21:39
[SPEAKER_01]: It found them.
21:40
[SPEAKER_01]: Like, you can point to it.
21:41
[SPEAKER_01]: It's not something that you just can make up and set.
21:44
[SPEAKER_01]: I know this because I know this.
21:46
[SPEAKER_02]: Yeah, sure.
21:46
[SPEAKER_01]: But this is like that, I don't know.
21:49
[SPEAKER_01]: Like that person that is sitting on the corner, that acts this way.
21:53
[SPEAKER_01]: Here's an information you need to know just because I know it and if you ask for a reference, it's going to give you some reference that it doesn't even contain.
22:00
[SPEAKER_01]: Yeah, right.
22:01
[SPEAKER_01]: That information.
22:03
[SPEAKER_01]: And so this is just a significantly paradigm shift or a situation that we are witnessing here.
22:12
[SPEAKER_03]: Yeah.
22:13
[SPEAKER_03]: I did a podcast recording with Jason Steinhauer.
22:16
[SPEAKER_03]: I don't know.
22:16
[SPEAKER_03]: Maybe a year ago something in he's a historian.
22:18
[SPEAKER_03]: And he was talking to me about people getting history from social media, especially Instagram and places like that, Twitter.
22:26
[SPEAKER_03]: And I said to him, who the hell is getting their history from there?
22:29
[SPEAKER_03]: That seems like an insane place to get there.
22:31
[SPEAKER_03]: And he said, oh, you'd be shocked.
22:33
[SPEAKER_03]: Tons of people got their history from social media.
22:35
[SPEAKER_03]: And now that I do realize that, and I myself often turn to an LLM to give me some educational background about such and such, not as never history, but other kinds of things like usually technical concepts.
22:49
[SPEAKER_03]: I can see how.
22:51
[SPEAKER_03]: this could potentially be worse than history on Instagram, et cetera, because it tells you a story in there at a forum, blah, blah, blah.
22:58
[SPEAKER_03]: At some point, we're going to get, I imagine you're going to get jetted by I to create not just the text that tells you the story, but probably video, which will self be fabricated, because it'll just generate the video and the audio to it.
23:08
[SPEAKER_03]: People will go to what more often I can imagine.
23:11
[SPEAKER_03]: teachers, teachers, not really understanding the perils here and assigning students to do some research on history and either tells them to use AI or not to use AI, but either way, AI is going to be in the mix.
23:24
[SPEAKER_03]: The more you talk, the more I think, oh, this is a, this is actually a, I think the phrase is, it's a big fucking problem.
23:30
[SPEAKER_01]: That is a very definite good way of defining it.
23:34
[SPEAKER_01]: And I think it's such not a point that is critical.
23:36
[SPEAKER_01]: That's why we've translated all of this into a policy brief that will be coming out soon and it will be spreading it worldwide.
23:43
[SPEAKER_01]: We're not prepared for this.
23:45
[SPEAKER_01]: Yeah, that's a very touchy thing.
23:49
[SPEAKER_01]: Like, yeah, we are seriously not prepared for this.
23:52
[SPEAKER_01]: Like our teachers are a whole teaching community hasn't been properly trained, not by their fault.
24:01
[SPEAKER_01]: Most of the states don't even have regulation or just procedural indications on how they should handle AI.
24:08
[SPEAKER_01]: But their students are already using it.
24:10
[SPEAKER_01]: Yeah.
24:10
[SPEAKER_01]: And so like it seems like we're playing catch up with something that has this potential for a generational impact that we're probably not even going to feel it for five to ten years.
24:23
[SPEAKER_01]: Right.
24:23
[SPEAKER_01]: And so it just begs the question on.
24:26
[SPEAKER_01]: How do you approach this and what are the things that you should really just sink your teeth in?
24:30
[SPEAKER_01]: Not to solve it, but at least to get some leg up on just catching the train and having something in place that will give at least educators a way to deal with this in their classrooms.
24:43
[SPEAKER_03]: Okay, let me try to push back a little bit and accuse you of overblowing the problem.
24:48
[SPEAKER_03]: Look, as you pointed out yourself, there's always competing historical narratives
24:53
[SPEAKER_03]: There's always a concern that the information that you're drawing from is going to be biased in some way Wikipedia is a good resource, but also it's particular people and these things out.
25:05
[SPEAKER_03]: If you go to Wikipedia on the sort of Israel, Helsinki and Gaza conflict lately, it's certainly representing certain view, certain interpretation of those events that not everyone's going to agree with, almost practically by definition, because it represents
25:18
[SPEAKER_03]: So we've already got that problem Google search results.
25:21
[SPEAKER_03]: Maybe there's I imagine that I wonder whether if I Google an English, the Vietnamese war versus I Google in Vietnamese, the American Vietnam war, I'm going to get different kinds of search results that are going to reflect different historical narratives, different perspectives, blah, blah, blah, blah.
25:37
[SPEAKER_03]: Maybe I won't get as much miss or disinformation as I will with an LLM.
25:40
[SPEAKER_03]: One thing that's special with the LLM's I get is the hallucination, but
25:43
[SPEAKER_03]: If we put the hallucinations to the side, which is a big thing to put to the side, I get that.
25:47
[SPEAKER_03]: Let's put aside for a moment.
25:48
[SPEAKER_03]: And just focus on, yeah, there's competing historical letters.
25:51
[SPEAKER_03]: Is the main reason why this is supposed to be different?
25:55
[SPEAKER_03]: this sort of monopolistic point that it's only a small, redundant, but small handful of companies that are creating these LLMs or is that that people really only have a consult of one LLM, they've got their preferred LLM and they use it for all the things and they're not gonna consult different ones.
26:10
[SPEAKER_03]: Yeah, why is this different from the history of fighting over history?
26:16
[SPEAKER_01]: So the history of fighting over history, that's very good point, then I'm glad you pushed back on that.
26:21
[SPEAKER_01]: Yeah.
26:22
[SPEAKER_01]: The history of fighting over history is usually a fight of facts and interpretations.
26:28
[SPEAKER_01]: Right?
26:28
[SPEAKER_01]: You get the historians in a room and they're all experts on something, they're going to battle.
26:34
[SPEAKER_03]: Yeah, what was the main cause of the French Revolution?
26:36
[SPEAKER_01]: There's going to be lots of fights.
26:37
[SPEAKER_01]: If you've ever been in the room, I usually, they're pointing at stuff that are figuratively
26:45
[SPEAKER_01]: Right?
26:46
[SPEAKER_01]: You may not agree that was the cause and that was the set of causes.
26:50
[SPEAKER_01]: Yes, sure.
26:50
[SPEAKER_01]: But to recognize that those things were there.
26:53
[SPEAKER_01]: Mm-hmm.
26:54
[SPEAKER_01]: That's definitely not what we have here.
26:57
[SPEAKER_01]: And you're talking about a thing that has been thoroughly studied that the way that people relate to is not the same as books.
27:05
[SPEAKER_01]: It's really something of a magic like belief that whatever comes from these things, especially at earlier ages, is that it's true.
27:15
[SPEAKER_01]: Yeah.
27:16
[SPEAKER_01]: And so you're not getting a counterpoint here.
27:18
[SPEAKER_01]: This is not a debate when you're getting information from these places.
27:22
[SPEAKER_01]: So if you believe what's there, you're getting that.
27:25
[SPEAKER_01]: And I'll just get to a second point that plays to what I really meant to be down about the gigantic monopolies.
27:32
[SPEAKER_01]: You saw it, for example, Elon Musk said about Glock being a non-woke.
27:39
[SPEAKER_01]: Sure, get one's enough.
27:41
[SPEAKER_01]: One distinctive behavior, that for example, we observed and Chatsupity's response for the Israel Palestine War, was that it was the single only event
27:57
[SPEAKER_01]: The narratives that we got in Hebrew and Arabic were similar in some things.
28:06
[SPEAKER_01]: One of them was associating Arabs with terrorists, and both languages, actually in English also.
28:15
[SPEAKER_01]: The relation of those two words was glaring, for example.
28:21
[SPEAKER_01]: And my prediction would be that this isn't even something that came from the data.
28:27
[SPEAKER_01]: This is the human loop is just reinforcing the burning with human in the middle going through narratives and choosing this or that should be better or worse.
28:37
[SPEAKER_01]: That means that for certain things, which then could be a generalized, there is a finger on the butt.
28:45
[SPEAKER_01]: There is a finger on the scale that these companies have.
28:48
[SPEAKER_01]: And that also begs the question with what authority given the level and the scale of people that are using this with what authority.
28:58
[SPEAKER_01]: Like who oversees this part of the equation?
29:01
[SPEAKER_03]: Yeah, I'm not sure what authority is the
29:07
[SPEAKER_03]: what authorities I feel that the answer something like it's their product so they get to have say over it and that seems like a not unfair answer to me.
29:18
[SPEAKER_03]: However, I might say something like
29:22
[SPEAKER_03]: What are their responsibilities?
29:24
[SPEAKER_03]: What are their ethical, social responsibilities?
29:27
[SPEAKER_03]: Given that, they have the authority to change this thing in various ways, that doesn't mean that they can do it in whatever way they want, that they're bereft of any obligations to discharge that authority.
29:38
[SPEAKER_03]: There's irresponsible use of authority, and responsible use of one's authority.
29:43
[SPEAKER_03]: And,
29:45
[SPEAKER_03]: What are the ethical obligations and where do they come from?
29:47
[SPEAKER_03]: It's going to be something like the phrase that I can't stand, but it happens to be true with great power, comes great authority, or the fact that they can potentially harm or so many people that can cause so much social unrest, so much conflict, that they bring upon themselves a certain level of responsibility, because they've chosen to create a product that puts them in the mix, that puts them into the fight over in this case history.
30:10
[SPEAKER_03]: And then the question becomes, what is a responsible discharge
30:15
[SPEAKER_01]: That is very good.
30:16
[SPEAKER_01]: And I can see that.
30:17
[SPEAKER_01]: I'll play the yes but game.
30:19
[SPEAKER_01]: Yeah.
30:20
[SPEAKER_01]: And I'll tell you that, okay, yes, but who will then oversee the responsibility if you acknowledge that there's even if an intended consequence about the scale of this effect.
30:40
[SPEAKER_01]: Who should oversee the potential, the negative potential of this?
30:47
[SPEAKER_01]: And enforce that people like these companies need to be responsible about, because for you talked about social media, there are commissions that are responsible to go over those things and make sure that there's some level of responsibility, and if it works, sometimes limited.
31:04
[SPEAKER_02]: Yeah.
31:04
[SPEAKER_01]: My question to you is there any oversight on this?
31:07
[SPEAKER_01]: Yeah, the question that I am aware of is no, okay.
31:11
[SPEAKER_03]: So here's the pushback on that.
31:13
[SPEAKER_03]: What do you want?
31:14
[SPEAKER_03]: Want government to oversight on the decisions that they make on how history is presented?
31:18
[SPEAKER_03]: That feels a bit fascist.
31:20
[SPEAKER_01]: And that feels a bit authoritarian.
31:22
[SPEAKER_01]: And that just is the beginning of the problem.
31:26
[SPEAKER_01]: which is, how do we solve that?
31:29
[SPEAKER_01]: And I'll be honest, like, I think this is a very interesting debate for which I would have invented, because we're certain extent no one should trust government, right?
31:38
[SPEAKER_01]: It's full extent.
31:40
[SPEAKER_01]: That's what the democracy ultimately means.
31:42
[SPEAKER_01]: Yeah, that's why people get elected from time to time and they're not there forever.
31:46
[SPEAKER_01]: Yeah.
31:46
[SPEAKER_01]: But we need to figure out a way where this can work.
31:51
[SPEAKER_01]: Yeah.
31:51
[SPEAKER_01]: Honestly, if social media tells us something is that it's not because it has sometimes a lot of poor and false and negative content that people drive away from it.
32:04
[SPEAKER_01]: There's a psychological effect here of addiction, of people using and getting so
32:13
[SPEAKER_01]: And so I think it's a multi-dimensional issue.
32:16
[SPEAKER_01]: I wouldn't venture into the kind of we need government oversight.
32:21
[SPEAKER_00]: Yeah.
32:22
[SPEAKER_01]: In terms of we need this entity that is constantly looking at it.
32:25
[SPEAKER_01]: Yeah.
32:27
[SPEAKER_01]: But we need something.
32:29
[SPEAKER_03]: Yeah, I feel like in this context, what people are going to call for is transparency.
32:32
[SPEAKER_03]: They should need to be transparent about how they're handling these issues blah, blah, blah, blah.
32:36
[SPEAKER_03]: But that also strikes me as potentially too weak.
32:39
[SPEAKER_03]: OK, it's A, they're going to be careful of what they shared, what they don't.
32:43
[SPEAKER_03]: And B.
32:44
[SPEAKER_03]: Okay, so this is what we're doing.
32:45
[SPEAKER_03]: And if it's really bad, now we can be upset, I think it's because boy caught them, but usually that doesn't do too much work.
32:52
[SPEAKER_03]: And it's one issue among many, it's obscure issue, right?
32:54
[SPEAKER_03]: Just begin to appreciate the ways in which histories represented by large language models in such a way that could lead to various kinds of disagreements, verbal conflict that turns into political and ultimately violent conflict.
33:07
[SPEAKER_03]: It's long-chain to follow that most people aren't going to get a grip on.
33:09
[SPEAKER_03]: And so it's got to feel like my guess is,
33:13
[SPEAKER_03]: when Elon Musk says here's the standards that way he was or don't he was in order to put guardrails around this stuff or keep it high on this stuff if point 05 were sent to the population boycotts him for that reason he's not going to care those people probably boycotting not using rock anyway because they hate him so much so transparency I feel like transparency is the go to answer and these kinds of context we need them to be transparent but
33:36
[SPEAKER_03]: That's not enough.
33:38
[SPEAKER_03]: I feel like we're stuck between it can't just be up to them and they're transparent about it because that gives us, it looks like insufficient power and it can't be government solely because that looks like tilting towards authoritarianism.
33:51
[SPEAKER_03]: That's a good title, tilting towards authoritarianism.
33:54
[SPEAKER_01]: I have one thing that we actually thought of.
33:59
[SPEAKER_01]: It is minor, but I think it can be effective to a certain extent.
34:04
[SPEAKER_01]: gamification.
34:06
[SPEAKER_01]: In the sense of you build a coalition of historians that changes over time, that assesses these tools and gets a center piece becomes respected in the sense that they evaluate these tools concerning their historical kind of representation in what's
34:30
[SPEAKER_01]: had first do the work, and give some yardstick in terms of improvement, something that could be improved, thinking about ways that this could improve, but also have something that even companies can strive to improve, but also identify the things that are for the public, the general public things that are definitely wrong.
34:53
[SPEAKER_03]: So is the idea that you have something like,
34:57
[SPEAKER_03]: It's that that you with the thought is and it can be extrapolated I think you've got the non-profit association of historians come with certain criteria for grading different kinds of LLMs based on how How good or shaded they are in the historical front so to speak they give a kind of scorecard or grade reports or poor card for these things you could do the same thing with psychologists with philosophers with blah blah blah and at least that would start some kind of process and then the I take it.
35:22
[SPEAKER_03]: That could be cool, but then the question is, is anybody care, and so now you have to get people to care enough in the way that maybe they care about that energy saver sticker, or something along those lines, or lead certified buildings.
35:34
[SPEAKER_03]: I just don't know how much they actually care, or I could see I could, this is just a new riffing now.
35:39
[SPEAKER_03]: I could see schools saying, look, you can only use the models for this thing.
35:45
[SPEAKER_03]: If they get a certain kind of great grade by the non-profit association of historians or whatever.
35:50
[SPEAKER_01]: Yeah, that those are examples, just something that you can use as a yardstick and a evaluation like an index or whatever there's indexes for everything.
35:59
[SPEAKER_01]: But there's one thing that I would push back there.
36:03
[SPEAKER_01]: Yeah, there's a mention of this that I feel that or it would be my opinion that governments should care.
36:13
[SPEAKER_01]: which is when we go from this discussion and education to purposely change the idea of the top of the conversation.
36:24
[SPEAKER_01]: Yeah, yeah, this now veers from your traditional which commission would look at this and body and what's enough to actually national defense.
36:35
[SPEAKER_00]: Yeah.
36:35
[SPEAKER_01]: So now you're talking about a different all game and just a different set of methods that you would have to put into place because you would still need to do the work of figuring out, quote-unquote, where you're being attacked.
36:50
[SPEAKER_01]: Yeah.
36:51
[SPEAKER_01]: It's our narrative.
36:52
[SPEAKER_01]: Let's call it national narrative historical narrative being attacked.
36:57
[SPEAKER_03]: That's really true because so usually when we think about
37:01
[SPEAKER_03]: Especially in the context of large language models, one thing that we're concerned about is data poisoning.
37:08
[SPEAKER_03]: So you get, as you, you know better than I do, right?
37:11
[SPEAKER_03]: Maybe you should just explain data poisoning, I bet you'll do a better job than me.
37:15
[SPEAKER_01]: The data processing is a technique to essentially bias a model towards something.
37:21
[SPEAKER_01]: It can be just biasing towards it being wrong, but also biasing responses towards a certain goal.
37:27
[SPEAKER_01]: I give you an example.
37:28
[SPEAKER_01]: It's already been discussed in research that, for example, Russia through some
37:34
[SPEAKER_01]: they backed media outlets has been flooding the internet, both in Russian, but also in Ukrainian.
37:40
[SPEAKER_01]: And some of the responses that are there, particularly in terms of historical references, origins of war and what's or not, have creaked up to the responses of certain public facing chat bonds.
37:52
[SPEAKER_03]: Okay, so the idea is something like Russia publishes a bunch of websites with BS information about how the Ukraine war started and LLMs are Eventually what like Google indexes these sites or something like that LLMs find them and they quote unquote learn from them or at least they retrieve information from them So when someone asks about the Russian Ukraine war, it's increasingly likely that it's going to surface Information that was created by Russia for propaganda purposes
38:18
[SPEAKER_01]: I think it's a probabilistic part.
38:20
[SPEAKER_01]: I don't think just flooding the internet with it will make it sway because companies spend a ton of money curating these data system and ensuring that they're as decent as possible.
38:31
[SPEAKER_01]: But we're talking a lot about a lot of resources.
38:35
[SPEAKER_01]: Depending on the amount and diversity of content that you are creating that yes, increases the probability of a creeping gap,
38:44
[SPEAKER_01]: these.
38:45
[SPEAKER_01]: When we're talking about low or meat source languages, so we'll find you guys that have less representation in terms of data.
38:52
[SPEAKER_03]: Does that make it?
38:53
[SPEAKER_01]: Because that makes it easier to poison them if it's, if there's less to compete with, less text to compete with, less to trust the difference between poisoning the well or poisoning the notion.
39:02
[SPEAKER_01]: Yeah, yes.
39:04
[SPEAKER_03]: Totally.
39:04
[SPEAKER_03]: Yeah.
39:05
[SPEAKER_03]: So usually, at least in a corporate context, when we think about cyber attacks and data poison, then we're worried about corporate espionage or worried about getting models to behave in ways that are fraudulent or leaked data or blah, blah, blah, but
39:20
[SPEAKER_03]: There's this sort of statewide data poisoning.
39:22
[SPEAKER_03]: It doesn't have to be a state.
39:23
[SPEAKER_03]: It can be just the white supremacist society.
39:26
[SPEAKER_03]: And this tries to enact a data poisoning.
39:29
[SPEAKER_03]: So yeah, that's really hard to track, I imagine.
39:34
[SPEAKER_03]: It's probably hard to monitor for this kind of thing because you're talking about the poison coming in and drips and drabs in unpredictable ways from unpredictable places.
39:42
[SPEAKER_01]: Yes, there would be ways of testing for that and definitely, but one thing that I would just expand here is one thing that you touched.
39:53
[SPEAKER_01]: Now, this means that to a certain extent, it's no longer that maximum of victors write the story.
40:02
[SPEAKER_01]: It's about basically who can produce more data that I can write it.
40:05
[SPEAKER_01]: And this means that it's not only states,
40:08
[SPEAKER_01]: Like you talked about the fraternity of white power.
40:11
[SPEAKER_00]: Right, right, right.
40:11
[SPEAKER_01]: It could be internal.
40:13
[SPEAKER_01]: It could be some other organization worldwide that just has amended to do something.
40:19
[SPEAKER_01]: And it's can't be or another.
40:20
[SPEAKER_01]: It's not about firing a single bullet.
40:24
[SPEAKER_01]: It's just about what is the capability of generating information and positioning yet where it needs to be.
40:31
[SPEAKER_03]: Yeah, it's even, it's, oh my god, it's so hard.
40:34
[SPEAKER_03]: One thing we've been worrying about for a long time with social media is the spread of misinformation.
40:39
[SPEAKER_03]: But to your point, there's even this issue of, first of all, there's that worry when it comes to large language models in history.
40:46
[SPEAKER_03]: Yes.
40:47
[SPEAKER_03]: But then there's also just to the other point, there's interpretations of the existing,
40:53
[SPEAKER_03]: legit data, the real-life things, and you could also engage in some kind of poisoning, not one way to put false information out there, intentionally.
41:06
[SPEAKER_03]: Another way is for the AI to hallucinate false stuff, so it's too pads to falsehood, but then you could also just, let's say there's two interpretations of historical event, of course, there's always more than two, but let's say there's two.
41:17
[SPEAKER_03]: You can start flooding the internet or whatever the technique is,
41:20
[SPEAKER_03]: to prioritize one interpretation over the other, which cannot be objected to one of the grounds that it's false or at least not just that it's false because in some sense it might be legitimate interpretation, but it might not be.
41:33
[SPEAKER_03]: And you can see, you can test it and you can see a very strong argument for saying, that needs to be taken down.
41:41
[SPEAKER_03]: That needs, that can't be allowed.
41:42
[SPEAKER_03]: That's clearly false information.
41:45
[SPEAKER_03]: It's a lie.
41:46
[SPEAKER_03]: It's fabricated either it's hallucinated or it's intentional propaganda by a state actor or whatever.
41:51
[SPEAKER_03]: But the grounds for taking something down because it's one member of the set of
41:57
[SPEAKER_03]: information or data that is trying to bolster a particular interpretation of the historical data, that's a much harder case to make.
42:05
[SPEAKER_01]: Yes.
42:06
[SPEAKER_01]: Technically, I can see how you could to a certain extent evaluate for those things.
42:12
[SPEAKER_01]: Yeah.
42:13
[SPEAKER_01]: But there's all of those dimensions.
42:15
[SPEAKER_01]: There's also the one that I mentioned a little before, which is depending on the level of kind of popularity of the issue, it can be caught up in just the human and the loop internal debate of annotators choosing one over the the other.
42:36
[SPEAKER_01]: all of this spells a situation that is very hard not to sound conspiracy theorist, but I think it's a legitimate concern.
42:50
[SPEAKER_01]: I think education is the most clear cut case for me, because one thing that really concerns me is just a low-intensity and a low-development that
43:02
[SPEAKER_01]: it's something that if you don't get a grip now.
43:05
[SPEAKER_01]: Once it's done, you're done.
43:09
[SPEAKER_01]: But this one, it just has so many weights in and creates this new vector for cognitive warfare that I think
43:19
[SPEAKER_01]: If we have any discussion about the history and oversight and education on this front, it seems much more clear cut than that, you need something in place that would have an idea of the shifts, especially the intentional ones.
43:36
[SPEAKER_03]: Yeah.
43:37
[SPEAKER_03]: I started off thinking like how the hell is you going to make this case about from history to intergenerational, cultural warfare and cognitive manipulative.
43:46
[SPEAKER_03]: And now like oh shit.
43:47
[SPEAKER_03]: Yeah, this is we're seeing it.
43:49
[SPEAKER_03]: This seems like a real big problem.
43:51
[SPEAKER_03]: Okay.
43:52
[SPEAKER_03]: So you mentioned that you had some policy recommendations.
43:55
[SPEAKER_03]: Yes.
43:56
[SPEAKER_01]: Can you say something with that?
43:57
[SPEAKER_03]: So maybe I don't have to go off in despair.
44:01
[SPEAKER_01]: No, and we based on the work that we did, we essentially gathered of all of this information.
44:07
[SPEAKER_01]: We got all of these stories, memory studies, folks, and just go over all of these narratives, and really think through, what can we do, right?
44:17
[SPEAKER_01]: And I think there's three aspects that we really need to look into.
44:22
[SPEAKER_01]: The first one concerning education, primarily, with all of the challenges that we have that we just discussed, there's simple things that we can do, right?
44:34
[SPEAKER_01]: I think Professor Hill development and making sure that we have the mechanisms for our teaching community, our teachers evolve and be much more prepared to handle this in a classroom are really necessary.
44:51
[SPEAKER_01]: some band-aid situations, which is enhanced disclosure requirements when we're touching on issues related to history, that can be something that can be discussed.
45:05
[SPEAKER_01]: But there's a broader kind of cultural level, like mass society or mass public scenario here, which is
45:15
[SPEAKER_01]: People outside of the educational trail are still getting their information a lot of information from here.
45:22
[SPEAKER_01]: And so the same thing applies.
45:25
[SPEAKER_01]: Like we need some sort of, I would as a review board, but coalition that would look into studying the impacts, but also measuring to an extent, these tools are capable of being representative in what they are putting out.
45:43
[SPEAKER_01]: And I think it would be fair to say that this coalition or this organization should have some sort of collaborative framework with companies with the companies that they're looking at because I think from at least from some of the people that I've had the pleasure of kind of facing or meeting that are from these companies there's a lot of people that acknowledge these things and there are seriously concerned about.
46:08
[SPEAKER_01]: the potential impacts of the tools that they're building.
46:11
[SPEAKER_01]: And then I think there's this third one of cognitive warfare that's simply, we just need
46:17
[SPEAKER_01]: Clear protocols on how to identify and respond to these misinformation and biasing campaigns.
46:23
[SPEAKER_01]: There needs to be responsibility on who would actually follow suit, but also, we need just a ton of research.
46:31
[SPEAKER_01]: There's so many questions on how we would even detect, but also hatch and de-buy as these humongous tools and model tools that are online that we seriously note, there should be an effort in terms of,
46:46
[SPEAKER_01]: federal funding towards improving our capability of responding to these things.
46:51
[SPEAKER_01]: So these are some of the threats to Paul that we would like to start this question.
46:56
[SPEAKER_03]: So the three are, one is educate education.
47:01
[SPEAKER_03]: Yeah, and in particular educate the teachers, presumably also educate the students on these things as well.
47:06
[SPEAKER_01]: Because that's the thing that's like why we were talking, like even energy certificate kind of thing that you mentioned.
47:12
[SPEAKER_01]: Those things actually land here also.
47:14
[SPEAKER_01]: So there's that, then the second thing was, there's a broader culture, just the understanding, twisted and the extent, the representativeness and the factualness and saying of the historical representation.
47:27
[SPEAKER_01]: So this is essentially creating that coalition of historians that every year would look at different
47:33
[SPEAKER_03]: So the second thing is this, I put as a score mechanism by the historical association or something along those lines.
47:39
[SPEAKER_03]: And then the third thing is protocols, potentially that are government such that these things are monitored and reacted to in a certain way.
47:51
[SPEAKER_03]: Yeah, but I take it then rubber really hits the road as we try to articulate what the education looks like that's not that hard.
47:58
[SPEAKER_03]: I don't think what the modern needs to look like what is how do we actually build the evaluation system.
48:04
[SPEAKER_03]: I feel like that's a very solvable problem.
48:06
[SPEAKER_03]: You get the historians together.
48:07
[SPEAKER_03]: They can come up with some criteria and a report card blah blah rubric.
48:11
[SPEAKER_03]: That seems easy-ish enough.
48:13
[SPEAKER_03]: The third thing seems like the really hard part,
48:21
[SPEAKER_03]: I guess it would be, I actually relate your score goals.
48:25
[SPEAKER_01]: Yeah, technically, I see ways of that.
48:30
[SPEAKER_01]: But this wouldn't need to be something that is to a certain extent.
48:35
[SPEAKER_01]: I wouldn't say enforced, but nicely enforced to the companies that have the responsibility of keeping these things solitary.
48:45
[SPEAKER_01]: And yeah, technically, I don't see, I don't see an issue with kind of having a first steps on it.
48:54
[SPEAKER_01]: I do think there's a ton like those steps.
48:58
[SPEAKER_01]: We'll get a ton of questions that we'll need if very determined it kind of research efforts to be in place for them to be answered.
49:07
[SPEAKER_03]: So, when you're starting your non-profit to finish into two of these things that many times.
49:13
[SPEAKER_01]: Yeah, okay.
49:14
[SPEAKER_01]: No, actually, we're looking at at this moment.
49:17
[SPEAKER_01]: I'm building several collaborations here, but also around the world concerning this.
49:22
[SPEAKER_01]: We're doubling down on this.
49:23
[SPEAKER_01]: For example, we're looking at collaborations in Ukraine to actually have an idea of
49:29
[SPEAKER_01]: how the representation of historical narratives in Ukraine is actually looking for Ukrainian kids today and adolescent.
49:37
[SPEAKER_01]: We're also looking at things like, for example, what are the cognitive physiology calls and the social impacts of learning historical events and classroom settings versus using these tools or mixes of them.
49:51
[SPEAKER_01]: And so we're exploring this in various routes so that we can
49:57
[SPEAKER_01]: But ultimately, the goal is essentially what you had, we want together as much people as we can.
50:06
[SPEAKER_01]: to get as much buy in as we can also to actually start looking at this from the more rubber meets the road is like, yeah, making the effort off just analyzing these things and pulling people together that believe that this is an issue that should be handled immediately.
50:21
[SPEAKER_03]: That's great.
50:22
[SPEAKER_03]: That's great.
50:23
[SPEAKER_03]: You can have to come back on and tell me what the progress looks like.
50:25
[SPEAKER_00]: Hey, as soon as I have news, we'll read the brief in the comments.
50:29
[SPEAKER_03]: I will thank you so much.
50:29
[SPEAKER_03]: This was really great and really slightly terrifying.
50:32
[SPEAKER_03]: At least there's steps to be taken.
50:34
[SPEAKER_00]: positive outlook.
50:35
[SPEAKER_00]: I am an incurable optimist.
50:38
[SPEAKER_00]: That's good.
50:38
[SPEAKER_00]: That's good.
50:39
[SPEAKER_00]: Thank you for the invitation.
50:40
[SPEAKER_00]: No, my pleasure.
50:41
[SPEAKER_00]: Thank you.
50:41
[SPEAKER_00]: Bye.
50:42
[SPEAKER_00]: Cheers.
50:45
[SPEAKER_03]: Thanks everyone for listening.
50:46
[SPEAKER_03]: I hope you enjoyed the episode.
50:47
[SPEAKER_03]: And I want to thank a few people who make this all possible.
50:50
[SPEAKER_03]: Delano Alexander, he's my producer.
50:52
[SPEAKER_03]: He does such an amazing job on everything that he touches.
50:55
[SPEAKER_03]: Garrett Lane is the sound engineer.
50:56
[SPEAKER_03]: That's honestly, he's just, he's too good for me.
50:59
[SPEAKER_03]: Authamia Jairam has joined us regionally.
51:01
[SPEAKER_03]: He's helping us find the best guests to bring on to this podcast.
51:04
[SPEAKER_03]: And Ashley Wasselenko helps us organize all this.
51:07
[SPEAKER_03]: Get people on the schedule, get people's bios, headshots, and really keeps it straight on the track.
51:11
[SPEAKER_03]: So thanks guys, really appreciate all the hard work you do.