Latest / Why We Do What We Do / The Importance of Peer Review

Transcript

The Importance of Peer Review · Why We Do What We Do

• 0:03
  Speaker 1: You're listening to Why we do what we do. Welcome
• 0:15
  Speaker 1: to Why we do what we do. I am I
• 0:19
  Speaker 1: am the host who's master's thesis is still sitting in
• 0:22
  Speaker 1: a file drawer in.
• 0:23
  Speaker 2: My name is Abraham, and i'm reviewer to host Shane
• 0:28
  Speaker 2: Oh good one.
• 0:29
  Speaker 1: We are a psychology podcast. We talk about the things
• 0:32
  Speaker 1: that humans and non human animals do and we try
• 0:35
  Speaker 1: to bring a nice skeptical scientific lens to our understanding
• 0:39
  Speaker 1: of behavior that shapes the world and which we live.
• 0:42
  Speaker 1: And thank you so much for joining us. We're happy
• 0:44
  Speaker 1: to have you here.
• 0:45
  Speaker 2: Yeah, for sure. We're looking forward to hearing your feedback
• 0:47
  Speaker 2: on this one, because this episode is about the peer
• 0:50
  Speaker 2: review process. For my science nerds out there and for
• 0:54
  Speaker 2: my researchers, and for all the people who have written
• 0:56
  Speaker 2: a manuscript and turned it in and then got it rejected.
• 0:58
  Speaker 2: This episode is for you.
• 1:01
  Speaker 1: That's right, or got accepted, that's.
• 1:02
  Speaker 2: Not accepted, that's cool too, Yeah, yeah.
• 1:04
  Speaker 1: But this is about the process. You know, sometimes we
• 1:06
  Speaker 1: do that, we talk about processes that we put in
• 1:08
  Speaker 1: place because it helps us be better as a species,
• 1:11
  Speaker 1: and this is one of those things. And if you're
• 1:14
  Speaker 1: joining us for the first time, we hope that you
• 1:16
  Speaker 1: enjoy this. But as we said, we thrive on feedback.
• 1:18
  Speaker 1: You go ahead and leave us that review, because we're
• 1:20
  Speaker 1: talking about reviews today. You can also leave us five stars.
• 1:23
  Speaker 1: You can just write to us. That's all good. If
• 1:25
  Speaker 1: you're a returning listener, then welcome back. You may also
• 1:28
  Speaker 1: leave us a review, give us some feedback, tell us
• 1:30
  Speaker 1: what you think right to us, leave us a rating,
• 1:32
  Speaker 1: go tell a friend. We'll talk more about the ways
• 1:34
  Speaker 1: you can support us. Either way, we're happy that you
• 1:36
  Speaker 1: found your way here, and we hope that you enjoy
• 1:38
  Speaker 1: what you hear in this episode today I started alluding
• 1:41
  Speaker 1: to this but didn't get through it. You can join
• 1:42
  Speaker 1: us on Patreon, pick up some merch like subscribe, tell
• 1:45
  Speaker 1: a friend. But no matter how you got here, and
• 1:47
  Speaker 1: no matter what you choose to say in your review,
• 1:49
  Speaker 1: we are going to start by wishing you all a
• 1:52
  Speaker 1: happy Banana Lover's Day, because it is August twenty seventh,
• 1:55
  Speaker 1: and you know everyone knows the August twenty seventh is
• 1:57
  Speaker 1: Banana Lover's Day.
• 1:58
  Speaker 2: Obviously, also just because day, why just because.
• 2:04
  Speaker 1: Just because bananas bananas. It is kiss me Day with
• 2:08
  Speaker 1: consent please, But if that's your thing, then sure, yeah.
• 2:10
  Speaker 2: Yeah, yeah, absolutely, It's National Petroleum Day.
• 2:14
  Speaker 1: It is National Pots de Creme Day, sure, which I
• 2:19
  Speaker 1: think that means something to do with potatoes.
• 2:22
  Speaker 2: I don't know, maybe that's what I would assume, okay,
• 2:26
  Speaker 2: or just milk pots.
• 2:28
  Speaker 1: It is, uh sure.
• 2:30
  Speaker 2: It is Slow Art Day as opposed to fast art.
• 2:33
  Speaker 1: Right, yeah, as opposed to fast art exactly, so, Bob Ross,
• 2:36
  Speaker 1: I think is where that is headed. It is Tarzan Day.
• 2:40
  Speaker 1: Not sure exactly what that means here, and I'm not
• 2:42
  Speaker 1: gonna look it up, but if you celebrate, then we acknowledge.
• 2:44
  Speaker 2: You sure get your loin cloths out. I guess it
• 2:47
  Speaker 2: is the Duchess who wasn't day?
• 2:50
  Speaker 1: Lots of those? Sure, most people. Maybe it is tug
• 2:56
  Speaker 1: of War Day. So if you're into that sort of thing,
• 2:59
  Speaker 1: go pull on a rope.
• 3:00
  Speaker 2: Yeh, go pull on a rope. Maybe that's gonna be
• 3:02
  Speaker 2: my new insult. Eh yeah, You're go pull on a
• 3:04
  Speaker 2: rope and see what people say. It's willing to lend
• 3:07
  Speaker 2: a hand Wednesday.
• 3:09
  Speaker 1: Oh that's so nice. I love that that that's a
• 3:12
  Speaker 1: holiday that clearly has to fall on a day of
• 3:14
  Speaker 1: the week. What in fact, we're probably gonna say this
• 3:16
  Speaker 1: one every year because we published on Wednesdays. Yeah, exactly,
• 3:20
  Speaker 1: and this one has to fall on a Wednesday or
• 3:22
  Speaker 1: else it only happens every seven years. Yeah. It is
• 3:25
  Speaker 1: World Rock paper Scissors Day. Oh man, competition.
• 3:29
  Speaker 2: Get ready the great equalizer that is pop rock paper scissors.
• 3:33
  Speaker 1: We should do a mini on rock paper scissors. That
• 3:35
  Speaker 1: never occurred to me, But there's like I know that
• 3:38
  Speaker 1: there there's like actual math and science behind, like what
• 3:40
  Speaker 1: throws are most likely to happen, and like where this originated. Yeah,
• 3:43
  Speaker 1: I'm gonna write it down.
• 3:44
  Speaker 2: In variations because then you could play rock paper scisors
• 3:47
  Speaker 2: spock lizard stuff like that. Wow.
• 3:49
  Speaker 1: I did not know that you can actually also do
• 3:51
  Speaker 1: multiplayer rock paper scis. I don't know if you know this.
• 3:53
  Speaker 2: I didn't know that. There's also there's a version of
• 3:55
  Speaker 2: rock paper scissors that I believe has like twenty different
• 3:58
  Speaker 2: signs you could throw.
• 3:59
  Speaker 1: Whoa, oh yeah, okay, many it is.
• 4:02
  Speaker 2: And it's like it all works anyway, all right. It
• 4:04
  Speaker 2: is a Be Kind to Humankind week.
• 4:07
  Speaker 1: Ooh, I love that it's so much. Yes, please be
• 4:10
  Speaker 1: kind to humankind. That's great. Mostly it is break safety week,
• 4:15
  Speaker 1: and this is not break as in shatter something. It's
• 4:17
  Speaker 1: breaks in b R A K E. So break doesn't stop,
• 4:20
  Speaker 1: And I guess it's saying stop yeah.
• 4:22
  Speaker 2: Stop week, yeah, just yeah stop. Well, I guess like
• 4:25
  Speaker 2: make sure you stop. Well yeah, make sure the equipment
• 4:27
  Speaker 2: you have Yeah, make sure the equipment do you have?
• 4:29
  Speaker 2: Stops like it's supposed to. It's a Health Unit Coordinators week.
• 4:33
  Speaker 1: Good good. It's National Composites Week.
• 4:36
  Speaker 2: It is. It's National Safe at Home Week.
• 4:39
  Speaker 1: It's Robbie a'l awaal week. I hope I said that.
• 4:42
  Speaker 2: Right, Yeah, And it is World water Week like water
• 4:46
  Speaker 2: World Week is like yeah, Kevin country movies, specifically water World,
• 4:50
  Speaker 2: and not any of his other movies, which he was
• 4:52
  Speaker 2: much better in All the Man, not the Postman, not Yellowstone,
• 4:55
  Speaker 2: the Field of Dreams, Field of Dreams. He was a
• 5:00
  Speaker 2: Field dream That's that was Kevin Costner, right now, That
• 5:01
  Speaker 2: was Kevin Costner, not okay, Man of Steel.
• 5:05
  Speaker 1: Oh yeah, I forgot he was in that. Yeah, wow, okay,
• 5:12
  Speaker 1: all right, great. So anyway, those are those are the
• 5:15
  Speaker 1: days of the week that well, I guess the holidays
• 5:17
  Speaker 1: that land on the day or the week that that
• 5:20
  Speaker 1: this publication comes out. Wow, that was a weird stream
• 5:22
  Speaker 1: of conscience. Consciousness ramble that this went on. Those are holidays, Yeah,
• 5:27
  Speaker 1: those are holidays that happen to occur. When we publish
• 5:29
  Speaker 1: these episodes, we're acknowledging them. Yes, we're actually here to
• 5:32
  Speaker 1: talk about other things. As we said, our primary target
• 5:35
  Speaker 1: today is the process by which we evaluate one another's
• 5:38
  Speaker 1: work when we're disseminating information, a process I think we
• 5:41
  Speaker 1: could probably use in many, many, many more spaces than
• 5:44
  Speaker 1: is currently being used, because we seem to live in
• 5:46
  Speaker 1: the world of dumb right now. Yes, so I really
• 5:49
  Speaker 1: love it if we could return to the world of
• 5:50
  Speaker 1: less dumb than currently exists.
• 5:53
  Speaker 2: Yes, so I think we're going to talk about this,
• 5:56
  Speaker 2: and I will say too, like, we're going to talk
• 5:58
  Speaker 2: about this from the perspective of like submitting scientific manuscripts, right,
• 6:02
  Speaker 2: and what this does. Yeah, So just so everybody's got
• 6:04
  Speaker 2: that too.
• 6:05
  Speaker 1: So we always need to start by defining our terms.
• 6:08
  Speaker 1: Perhaps you've heard this sane peer review. I know we've
• 6:11
  Speaker 1: mentioned it on here before, So if you are a
• 6:13
  Speaker 1: regular listener, you've heard us say it almost certainly, and
• 6:17
  Speaker 1: probably outside of the context of this episode you have
• 6:20
  Speaker 1: heard it as well. But we definitely want to define
• 6:23
  Speaker 1: it so that we're all on the same page before
• 6:25
  Speaker 1: we jump into this. So, peer review is the evaluation
• 6:28
  Speaker 1: of scientific, academic, or professional work by others working in
• 6:33
  Speaker 1: the same field.
• 6:34
  Speaker 2: Yes, and so for those who are uninitiated, the peer
• 6:37
  Speaker 2: review process is an evaluation process that ensures that research
• 6:41
  Speaker 2: meets a certain level of rigor and quality and validity
• 6:44
  Speaker 2: prior to publication. That's the goal of it at least, okay,
• 6:47
  Speaker 2: And this is often used in the research space where
• 6:50
  Speaker 2: new research is submitted to a journal for review or
• 6:53
  Speaker 2: a board for review before it is accepted for publication.
• 6:57
  Speaker 2: And usually it involves some reviewers that takes some time
• 6:59
  Speaker 2: to view it and provide feedback, comments and decide whether
• 7:03
  Speaker 2: it's accepted or rejected or accepted with revisions, or whatever
• 7:07
  Speaker 2: it might be.
• 7:07
  Speaker 1: And this is a process. The whole point of what
• 7:10
  Speaker 1: we're going to be discussing today is to scrutinize one
• 7:14
  Speaker 1: another's work such that we feel like we have the
• 7:17
  Speaker 1: most rigorous way of interpreting and disseminating information. Yes, so
• 7:24
  Speaker 1: that those who then read and access that information will
• 7:28
  Speaker 1: have the clearest, most accurate picture of what we now
• 7:32
  Speaker 1: know as a field. And that happens every single time
• 7:34
  Speaker 1: one of these manuscripts is submitted. So this is part
• 7:37
  Speaker 1: of a self regulation process within certain fields that ensures
• 7:43
  Speaker 1: that a good research is published and b bad research
• 7:47
  Speaker 1: does not make it to publication. So this is we're
• 7:51
  Speaker 1: trying to like put in a filter here, and I
• 7:55
  Speaker 1: think this is why I say we could use this
• 7:57
  Speaker 1: in more spaces in our world today, as like where
• 8:00
  Speaker 1: is the filter that allows it so that we get
• 8:03
  Speaker 1: as much correct information as possible and we do not
• 8:06
  Speaker 1: get in correct information and so and we want to
• 8:09
  Speaker 1: talk about like what kinds of things peer reviews are
• 8:12
  Speaker 1: looking for, like how this process works. But essentially that's
• 8:15
  Speaker 1: the thing to know is like this is the filter
• 8:17
  Speaker 1: that prevents the crap from getting out there and like
• 8:21
  Speaker 1: has people wasting time on stuff that is ultimately proves
• 8:24
  Speaker 1: to be incorrect. And it doesn't always work, but it
• 8:27
  Speaker 1: works a lot better than just saying like publish whatever
• 8:29
  Speaker 1: you want. Nobody cares like it'll be great.
• 8:31
  Speaker 2: Yeah, yeah, yeah exactly. And we're gonna talk about that too,
• 8:34
  Speaker 2: Like we're talking about like you know, yeah, sure it's
• 8:36
  Speaker 2: not a perfect process, but it is better. It's the
• 8:38
  Speaker 2: best thing we got right now given the circumstances. So
• 8:41
  Speaker 2: but we'll we'll talk more about that too. So the
• 8:44
  Speaker 2: peer review process is, in its modern iteration, is a
• 8:47
  Speaker 2: fairly new thing, but the history of the peer review
• 8:50
  Speaker 2: process may be as old as the nine hundred cell,
• 8:54
  Speaker 2: specifically where a physician named Ali al Rawi of Syria
• 8:59
  Speaker 2: describe the process in his book Ethics of the Physician.
• 9:02
  Speaker 2: He explained that quote, physicians must take notes describing the
• 9:06
  Speaker 2: state of their patients medical conditions upon each visit. Following treatment,
• 9:09
  Speaker 2: the notes were scrutinized by a local medical council to
• 9:12
  Speaker 2: determine whether the physician had met the required standards of
• 9:16
  Speaker 2: medical care. So this was the end quote, and that
• 9:18
  Speaker 2: was like the official the first description of what a
• 9:21
  Speaker 2: peer review process might actually look like.
• 9:24
  Speaker 1: Man, we talked about this before, but it would be
• 9:27
  Speaker 1: so interesting to do some history around the scholarly work
• 9:31
  Speaker 1: of the Middle East. Yes, because before they entered what
• 9:37
  Speaker 1: I would describe as sort of the dark ages of
• 9:40
  Speaker 1: like religious zealousness, they were like the world hub of
• 9:45
  Speaker 1: scientific and academic achievements. So much of what the world
• 9:49
  Speaker 1: has gained and learned came out of the brilliant scientists
• 9:52
  Speaker 1: that lived in that part of the world, who were
• 9:54
  Speaker 1: doing good work before the sort of religious zealots took
• 9:57
  Speaker 1: over and plunged it into the dark ages they seem
• 9:59
  Speaker 1: to be in right now. Yeah, yeah, and yes I
• 10:01
  Speaker 1: am firing shots at you people over there who are
• 10:04
  Speaker 1: like deliberately censoring information and making it impossible for people
• 10:08
  Speaker 1: to do research and things like that. Yeah, because you
• 10:10
  Speaker 1: are standing in the way of the progress of mankind,
• 10:12
  Speaker 1: and therefore you are my enemy. You're not Miami or
• 10:14
  Speaker 1: the enemy of mankind, but as an arborer of people
• 10:18
  Speaker 1: speaking on behalf of the process of bettering the human condition,
• 10:22
  Speaker 1: I do not salute you. You're a butt.
• 10:24
  Speaker 2: Yeah, yeah, yeah. Anybody who suppresses knowledge is a butt.
• 10:27
  Speaker 1: Yes, yes, you're a butt. Anyway, we need to continue
• 10:31
  Speaker 1: expanding upon the point that you were making here, which
• 10:33
  Speaker 1: was that this old this came from nine hundred CE possibly,
• 10:37
  Speaker 1: and that's amazing, like really incredible that they were engineering
• 10:40
  Speaker 1: that level of brilliance that long ago. That's really awesome. Again,
• 10:44
  Speaker 1: just speaks to the quality of the people that were
• 10:46
  Speaker 1: doing that kind of work and which was really awesome.
• 10:49
  Speaker 1: And it goes on to describe so you were describing
• 10:51
  Speaker 1: this this medical text, right, the doctor who would write
• 10:54
  Speaker 1: these notes, and then those notes would be scrutinized by
• 10:56
  Speaker 1: a panel to sort of give quality assurance, if you will.
• 10:59
  Speaker 1: And it also went on to describe what happens if
• 11:02
  Speaker 1: treatment is deemed unacceptable, which might be one of the
• 11:05
  Speaker 1: first descriptions of malpractice lawsuits as well. Yeah, speaking to
• 11:09
  Speaker 1: like you are in fact making decisions that are on
• 11:12
  Speaker 1: the best interests of the client based on current knowledge
• 11:15
  Speaker 1: and that like understanding that that knowledge is likely going
• 11:17
  Speaker 1: to change, and that's again that's part of the process.
• 11:20
  Speaker 2: Yeah, it's super interesting at this point in time. You know,
• 11:23
  Speaker 2: we've got history going on. There's a lot of things happening,
• 11:25
  Speaker 2: and then comes the printing press, and so the printing
• 11:28
  Speaker 2: press shows up and documents are made more accessible than
• 11:32
  Speaker 2: they ever have been to the public, and that includes ads.
• 11:43
  Speaker 1: All right, we're back. And I don't know if people
• 11:46
  Speaker 1: can tell from listening to this, but part of the
• 11:49
  Speaker 1: joy of these recordings is us trying to surprise each
• 11:52
  Speaker 1: other with ad segues that are like not scripted or planned.
• 11:55
  Speaker 1: I mean, we know, like about when we want ads
• 11:57
  Speaker 1: to happen. But anyway, that that one got me.
• 12:01
  Speaker 2: Yeah, it was good stuff.
• 12:03
  Speaker 1: Which is good. Yeah, Okay, anyway, so we were talking
• 12:05
  Speaker 1: about the printing press, so please continue.
• 12:07
  Speaker 2: Yah, yeah, yeah, So the printing press shows up and
• 12:09
  Speaker 2: documents are made more accessible to the general public. But
• 12:11
  Speaker 2: as a result, peer review and editing became increasingly more
• 12:14
  Speaker 2: common because you were starting to see more publications in
• 12:17
  Speaker 2: scientific journals and scientific research and dissemination. So this peer
• 12:20
  Speaker 2: review process in the form of editing was it started
• 12:24
  Speaker 2: to become more common. The evaluation of science wasn't formally
• 12:27
  Speaker 2: written about or codified in any sort of way as
• 12:29
  Speaker 2: we understand it today, Like, it wasn't really systemized just
• 12:31
  Speaker 2: yet until Francis Bacon shows up with Novum Organum, and
• 12:36
  Speaker 2: so he starts to kind of outline yeah, exactly. He
• 12:40
  Speaker 2: starts to outline what a peer review process could and
• 12:42
  Speaker 2: should look like on its like most rudimentary terms awesome.
• 12:46
  Speaker 1: So in this work he described what has since become
• 12:49
  Speaker 1: the very first universally accepted method for new science, shaping
• 12:53
  Speaker 1: the scientific method as we know it today. In this process, though,
• 12:57
  Speaker 1: the peer review system was only thought to be an
• 13:00
  Speaker 1: editor's job. So the person there's like, you still have
• 13:03
  Speaker 1: a filter process, but it's you have someone who's more
• 13:06
  Speaker 1: or less in charge of I guess you would say,
• 13:10
  Speaker 1: just like the quality of like the writing and everything. Yeah,
• 13:12
  Speaker 1: it was not really a validity process. It was like,
• 13:15
  Speaker 1: let's make sure that this is clear. It'll make it
• 13:17
  Speaker 1: out to the audience, like it's written in such a
• 13:20
  Speaker 1: way that like it makes sense to me, and like
• 13:22
  Speaker 1: it's having someone look over it and just say like okay, yeah,
• 13:24
  Speaker 1: I get it, yeah that makes sense, or maybe fix
• 13:26
  Speaker 1: this part here I don't really understand what's being said.
• 13:29
  Speaker 1: Did not really include at that point still the level
• 13:32
  Speaker 1: of scrutiny of like bringing in other experts to really
• 13:35
  Speaker 1: help hone it. But that's okay, like this is this
• 13:37
  Speaker 1: is first steps here that are really important.
• 13:39
  Speaker 2: Yeah, for sure. So this goes on for a while
• 13:41
  Speaker 2: and then World War Two shows up and disrupts everything.
• 13:45
  Speaker 2: World War two the worst of the world wars. So, yeah,
• 13:50
  Speaker 2: nobody likes a sequel.
• 13:52
  Speaker 1: The think, particularly the current single we find ourselves.
• 13:55
  Speaker 2: In single right now is not great right now?
• 13:57
  Speaker 1: Yeah?
• 13:58
  Speaker 2: Yeah. No, So here's the thing, World War two. Everybody
• 14:00
  Speaker 2: talks about how disruptive it was on industries, on innovation,
• 14:03
  Speaker 2: on all this stuff, and specifically you know, like military
• 14:07
  Speaker 2: expertise and whatnot, But they don't really talk about how
• 14:10
  Speaker 2: World War two was significant in that it increased the
• 14:14
  Speaker 2: amount of scientific research that we were doing, and in
• 14:17
  Speaker 2: lots of different areas, not just social sciences, but in physics,
• 14:21
  Speaker 2: in engineering, and all these different areas that we really
• 14:24
  Speaker 2: need to innovate, and so we see this huge boom
• 14:27
  Speaker 2: of new research in not only engineering and physics, but
• 14:32
  Speaker 2: in social sciences. In the medical field, you start seeing
• 14:34
  Speaker 2: the emergence of psychological services start to take place in
• 14:38
  Speaker 2: like in better ways. You start seeing medical care improve,
• 14:40
  Speaker 2: you start seeing nursing improved. So you see all this
• 14:42
  Speaker 2: stuff kind of happen. And the peer review process was
• 14:45
  Speaker 2: further systematized during this time due to the increase in
• 14:48
  Speaker 2: research to give it the much needed critiques and validity
• 14:52
  Speaker 2: that it was that we were looking for for scientific
• 14:54
  Speaker 2: We started seeing science become more rigorous during this time.
• 14:58
  Speaker 1: It is an unfortunate reality of war and kind of
• 15:02
  Speaker 1: aggression more broadly, that it entails innovation. That is often
• 15:09
  Speaker 1: I think there are military leaders who very pragmatically understand
• 15:12
  Speaker 1: the value of science as it relates to technologies of war,
• 15:16
  Speaker 1: and so a lot of times when wars happen, innovation
• 15:20
  Speaker 1: can really boom, which means there is a huge boom
• 15:23
  Speaker 1: in scientific process because they're really interested in pushing the
• 15:27
  Speaker 1: next super deadly weapon.
• 15:29
  Speaker 2: You know.
• 15:29
  Speaker 1: That's how we ended up with things like atomic weapons
• 15:32
  Speaker 1: and nuclear weapons, lasers and lasers, yeah, bombs that could
• 15:35
  Speaker 1: completely destroy the entire planet. Essentially was scientists were pushed
• 15:41
  Speaker 1: to really figure out how these processes work and use
• 15:44
  Speaker 1: and leverage that knowledge to engineer greater and greater weapons.
• 15:47
  Speaker 1: And so it's unfortunate. I do not think that it
• 15:51
  Speaker 1: was worth it to have science be pushed that far
• 15:53
  Speaker 1: that fast, Like I would rather have said, let's not
• 15:55
  Speaker 1: have the war and not gain that scientific sure innovation,
• 15:58
  Speaker 1: but we did get like it's complicated, it's nuanced. We
• 16:01
  Speaker 1: do now have that scientific innovation. So there's really no
• 16:03
  Speaker 1: putting the genie back in the bottle, as they say, right, right,
• 16:06
  Speaker 1: But it happens a lot like pretty much every time
• 16:08
  Speaker 1: there's been some major war in the last several hundred
• 16:12
  Speaker 1: years at least, it has come with major technological revolutions
• 16:16
  Speaker 1: that have been spurred by employing people who use science
• 16:20
  Speaker 1: and scientific methods to innovate in that space.
• 16:23
  Speaker 2: Right, So that brings us to at current day. Right,
• 16:27
  Speaker 2: the current peer review system is pretty similar to what
• 16:30
  Speaker 2: we started to see back then, it's become more refined
• 16:33
  Speaker 2: and more polished, and maybe there's a little bit more
• 16:35
  Speaker 2: nuanced and with technology systems, it's made it a little
• 16:38
  Speaker 2: bit easier to be anonymous. But like, what we're kind
• 16:41
  Speaker 2: of discovering now is that the peer review process is
• 16:43
  Speaker 2: not that dissimilar to what we were starting to find
• 16:46
  Speaker 2: to merge during and after World War Two. But the
• 16:49
  Speaker 2: question that everybody has, because most of us are anonymous,
• 16:54
  Speaker 2: is who are the reviewers? So I wrote this as
• 16:57
  Speaker 2: a kindergarten cop arnold sort who.
• 17:00
  Speaker 1: Are the reviewers?
• 17:01
  Speaker 2: What did they do? So I think it's important to understand,
• 17:05
  Speaker 2: like how how reviewers show up, what they do, who
• 17:08
  Speaker 2: they are, because they are kind of like the crux
• 17:11
  Speaker 2: of the peer review process, right, they're the first part.
• 17:14
  Speaker 1: Yeah. I mean if you hear the term peer review
• 17:16
  Speaker 1: of the first word and there is peer like it
• 17:18
  Speaker 1: implies that the reviewers are peers of the person who's
• 17:21
  Speaker 1: submitting the manuscripts. Right most of the time, yes, but
• 17:25
  Speaker 1: there are some interesting critiques about that. Most of the
• 17:28
  Speaker 1: time they call them experts in the field, but they're
• 17:31
  Speaker 1: not actually are peers. These might be folks who are
• 17:34
  Speaker 1: highly qualified researchers. You got to imagine like they're even
• 17:38
  Speaker 1: though they're not necessarily peers. There may be really good
• 17:41
  Speaker 1: at some particular aspect of science or this method or
• 17:44
  Speaker 1: something that's related, but not necessarily directly in the same
• 17:47
  Speaker 1: line of work. Maybe they're not directly colleagues, and actually
• 17:49
  Speaker 1: it's better that they don't know each other. We'll get
• 17:52
  Speaker 1: into that in a little bit. But those are sort
• 17:54
  Speaker 1: of the peers. But of course we then must ask,
• 17:56
  Speaker 1: all right, so you decide you're going to get these
• 17:58
  Speaker 1: experts to take a look at this manuscript and provide
• 18:01
  Speaker 1: their own critique of that manuscript. What defines an expert though.
• 18:06
  Speaker 2: Yeah, and that is a great question that we are
• 18:08
  Speaker 2: not going to answer today. It's a long standing discussion overall.
• 18:12
  Speaker 2: But let's talk about like this, what the peer reviewer does, right,
• 18:16
  Speaker 2: like in kind of like what they are supposed to do.
• 18:18
  Speaker 2: Like you, it's important to understand and not to expect
• 18:21
  Speaker 2: that the peer reviewer knows every single thing about a subject,
• 18:24
  Speaker 2: right like we should. They should have a general understanding,
• 18:26
  Speaker 2: they should have something like that. But like what they're
• 18:28
  Speaker 2: reviewing is new research, so they should at very least
• 18:31
  Speaker 2: have some information about the topic at hand. But generally speaking,
• 18:36
  Speaker 2: a peer reviewer is anyone who has competence and expertise
• 18:38
  Speaker 2: in the subject that the journal covers, not necessarily the
• 18:43
  Speaker 2: article itself, but whatever scope of the journal it is. Like, so,
• 18:47
  Speaker 2: for example, journal behavior analysis is going to have behavior
• 18:50
  Speaker 2: analysts generally speaking as the reviewers. They're not going to
• 18:53
  Speaker 2: have social workers or social psychologists as the reviewers, or
• 18:58
  Speaker 2: a physicist be the reviewer, right like, not generally speaking,
• 19:02
  Speaker 2: Like there may be some exceptions.
• 19:03
  Speaker 1: To the rule, or conspiracy theorists or.
• 19:05
  Speaker 2: Conspiracy theorists, we certainly won't have those all the time. Yeah,
• 19:08
  Speaker 2: unless the behavior analyst is a conspiracy theorist. And that's
• 19:10
  Speaker 2: kind of a fun that's a fun can unpack. But
• 19:14
  Speaker 2: generally speaking, like a journal will have a particular viewpoint
• 19:17
  Speaker 2: in a particular scope, and the reviewers at that journal
• 19:20
  Speaker 2: will meet that scope in some way.
• 19:23
  Speaker 1: And reviewers, I mean, who makes up this pool of people.
• 19:26
  Speaker 1: They can it can be quite variable. They can vary
• 19:28
  Speaker 1: in age and experience to some degree. And it is
• 19:31
  Speaker 1: reported that young reviewers actually often provide better, higher quality feedback.
• 19:35
  Speaker 1: It's not always the case, but a lot of times
• 19:37
  Speaker 1: these are people who are fresh out of classes. They
• 19:40
  Speaker 1: are like really in contact with a lot of the
• 19:43
  Speaker 1: recent work that's been done. They often are like motivated
• 19:47
  Speaker 1: and excited and like ready to like lend some time
• 19:49
  Speaker 1: to these things, so they might provide higher quality feedback.
• 19:54
  Speaker 1: They might also not provide verigh quality feedback because they
• 19:58
  Speaker 1: don't necessarily know how to They don't know what to
• 20:00
  Speaker 1: look for, they don't know how to comment about the
• 20:02
  Speaker 1: thing that they're looking for. So it is actually a
• 20:04
  Speaker 1: process should also entail some amount of mentorship so that
• 20:08
  Speaker 1: you can become a better reviewer. But the point being
• 20:11
  Speaker 1: is that, like you, just because someone is relatively new,
• 20:14
  Speaker 1: it does not necessarily mean that they will be bad
• 20:16
  Speaker 1: at reviewing. It also doesn't mean that they'll be good,
• 20:18
  Speaker 1: but like it does seem to be that they may
• 20:21
  Speaker 1: provide higher quality, better feedback.
• 20:23
  Speaker 2: Yeah, and also spoiler alert, older reviewers also provide bad feedback,
• 20:28
  Speaker 2: So like it happens too, like not always, but like
• 20:31
  Speaker 2: it is it is incumbent on the reviewer, and we'll
• 20:34
  Speaker 2: talk about kind of the process with that too. So yeah,
• 20:36
  Speaker 2: now reviewers will review approximately eight papers a year. That's
• 20:40
  Speaker 2: usually like as somebody who is like a consistent reviewer
• 20:43
  Speaker 2: will review about eight papers a year on average. Each
• 20:46
  Speaker 2: reviewer usually belongs to a pool of researchers or reviewers
• 20:49
  Speaker 2: for a designated journal or multiple journals, and may be
• 20:52
  Speaker 2: selected based upon their area of competence or availability. Oftentimes,
• 20:57
  Speaker 2: when I get selected to review, it's based on what
• 20:59
  Speaker 2: my research topics are, right, Yeah, so like so somebody
• 21:02
  Speaker 2: else said, it's like you were selected as a possible
• 21:05
  Speaker 2: reviewer because you have studied this or you have talked
• 21:08
  Speaker 2: about the subject. Would you be interested in reviewing? And
• 21:10
  Speaker 2: so that's usually when I get those I don't get
• 21:12
  Speaker 2: it because I'm just an available.
• 21:13
  Speaker 1: Reviewer, right, And generally speaking, the process goes like this.
• 21:18
  Speaker 1: You will receive them manuscript, usually deidentified, that's more and
• 21:22
  Speaker 1: more common anymore, which means that you don't know who
• 21:24
  Speaker 1: the authors of that manuscript are. You often don't know
• 21:26
  Speaker 1: the affiliation of those authors, like do they belong to university?
• 21:30
  Speaker 1: Like what grip they belong to you? You won't get
• 21:32
  Speaker 1: that information. You'll just have the title and the rest
• 21:35
  Speaker 1: of the paper and no names. And then what you
• 21:37
  Speaker 1: do is you review the full document and they're going
• 21:40
  Speaker 1: to look for things such as alignment, quality of writing, design, selection,
• 21:45
  Speaker 1: treatment effects are going to look at the methods and
• 21:46
  Speaker 1: scrutinize those to ensure that the methods would allow for
• 21:49
  Speaker 1: an analysis inter interpretation of the data. Some very ambitious
• 21:53
  Speaker 1: reviewers might even like crunch the numbers themselves, the double
• 21:56
  Speaker 1: check the numbers against what the authors purport. They if
• 21:59
  Speaker 1: this is one that has a lot of numbers in it,
• 22:01
  Speaker 1: which is not always the case. They'll look for coherence.
• 22:04
  Speaker 1: They'll look for just basically anything that is. They look
• 22:08
  Speaker 1: for consistent, high quality information to validate the manuscript and
• 22:13
  Speaker 1: the things that are being said in that manuscript. And
• 22:16
  Speaker 1: they're thinking about if this goes to publication and this
• 22:20
  Speaker 1: ends up in a journal, will the readers of this
• 22:22
  Speaker 1: journal be able to walk away with new useful information?
• 22:26
  Speaker 1: Right because it is conveyed in such a way that
• 22:28
  Speaker 1: it is it is as correct as we can as
• 22:30
  Speaker 1: we understand it to be based on current information. It
• 22:33
  Speaker 1: is clearly written, and it has all of the support
• 22:36
  Speaker 1: needed for someone to read this and walk away with
• 22:38
  Speaker 1: something of value. And in many cases, the point of
• 22:41
  Speaker 1: the reviewer might also be oriented to can this be replicated?
• 22:46
  Speaker 1: Like is this written in such a way that someone
• 22:48
  Speaker 1: else can take this and that will further inspire additional
• 22:51
  Speaker 1: research to be concluded, such as research into high quality ads. Okay,
• 23:03
  Speaker 1: we're back. Ads are not. I mean, people do actually
• 23:06
  Speaker 1: do research into ads, but that's not what's happening in
• 23:10
  Speaker 1: the peer review. Yeah, that wasn't what I was gonna say.
• 23:11
  Speaker 1: That's just a way to segue.
• 23:13
  Speaker 2: Yeah, no, no, no, it was good. It was good.
• 23:14
  Speaker 2: That was good. I like it. So another thing that
• 23:17
  Speaker 2: you need to know about peer reviewers, because I think
• 23:18
  Speaker 2: this is something that always comes up as part of
• 23:20
  Speaker 2: the conversation, is like, you know, what are people's affiliations,
• 23:23
  Speaker 2: what are their motivations? Peer reviewers don't get paid to review.
• 23:27
  Speaker 1: Yeah, it's volunteer.
• 23:28
  Speaker 2: It's volunteer. It's completely free labor. Also, I think it's
• 23:31
  Speaker 2: really important to note and it point out that for
• 23:34
  Speaker 2: the most part, research is also a free labor exercise.
• 23:37
  Speaker 2: You might work at a university that does pay for
• 23:40
  Speaker 2: your role, and your role is to publish research and
• 23:43
  Speaker 2: write research and stuff like that. Yeah, but like generally speaking,
• 23:47
  Speaker 2: like if as me as like working in the private sector,
• 23:49
  Speaker 2: if I wanted to do research, I would not get
• 23:52
  Speaker 2: paid for it. Like I would have to go do
• 23:53
  Speaker 2: the research and it would be free labor right, so
• 23:55
  Speaker 2: like kind of a unique thing. But peer reviewers specifically
• 23:58
  Speaker 2: what we're talking about, they don't get paid to review
• 24:00
  Speaker 2: any articles, and so it's a service that is offered
• 24:03
  Speaker 2: to the field by the field to self regulate the field,
• 24:06
  Speaker 2: and at least in behavior analysis, that's really all we do. Yeah,
• 24:09
  Speaker 2: so it's kind of a fun it's kind of an
• 24:11
  Speaker 2: interesting thing. So you have to be like really invested
• 24:13
  Speaker 2: in wanting to do this.
• 24:14
  Speaker 1: It's important if you think about, like what could happen
• 24:19
  Speaker 1: in sort of the space of corruption for people who
• 24:21
  Speaker 1: are getting paid to do this. They could feel like
• 24:23
  Speaker 1: if they're getting paid, they're obligated to say yay or
• 24:26
  Speaker 1: nay to greenlighting something as being published. Sure, they could
• 24:29
  Speaker 1: feel obligated to produce a certain amount, like not very
• 24:34
  Speaker 1: much or not very little, like it will alter the
• 24:37
  Speaker 1: kind of work that they'll be doing. So in order
• 24:40
  Speaker 1: to avoid any complication that comes from specifically skewing motivation.
• 24:45
  Speaker 1: By offering financial incentives, you get people who just want
• 24:50
  Speaker 1: to help the progress of science. And that's pretty much
• 24:52
  Speaker 1: the entire process as far as I understand it. In
• 24:55
  Speaker 1: the majority of cases, editors do not get editors or
• 24:58
  Speaker 1: journals do get paid. Sorry, But like the editors, the
• 25:01
  Speaker 1: action editor who oversees the review does not get paid,
• 25:04
  Speaker 1: and the reviewers themselves do not get paid. And in
• 25:06
  Speaker 1: many cases the authors do not necessarily get paid for
• 25:09
  Speaker 1: the publication. And actually many cases they have to pay
• 25:12
  Speaker 1: if they want it to be like open access, yeah,
• 25:14
  Speaker 1: that other people can read it and not have to
• 25:16
  Speaker 1: pay for it.
• 25:16
  Speaker 2: They don't even own their own publication.
• 25:18
  Speaker 1: Yeah yeah, yeah, they don't own it anymore. They basically
• 25:20
  Speaker 1: give it up to the publisher. In that particular instance.
• 25:24
  Speaker 1: There are what are called pay to publish journals. They're
• 25:28
  Speaker 1: highly suspect. And this is the reason that like this
• 25:30
  Speaker 1: isn't a pay thing is because there are journals where
• 25:33
  Speaker 1: like if you pay them, they will publish your work
• 25:36
  Speaker 1: and it might go through like the veneer of like
• 25:41
  Speaker 1: the image the process of what looks like peer review.
• 25:43
  Speaker 1: But if you pay them to publish it, they're going
• 25:45
  Speaker 1: to publish it. And you got to believe that, like,
• 25:47
  Speaker 1: if you paid a journal to publish your work and
• 25:49
  Speaker 1: it's not going to receive the level of scrutiny that
• 25:52
  Speaker 1: like a scientific journal should be receiving and would otherwise
• 25:55
  Speaker 1: be receiving, that that is something where you are clearly
• 25:59
  Speaker 1: motivated to get that work out there without it being scrutinized,
• 26:02
  Speaker 1: which means that again, your work is suspect. So we
• 26:05
  Speaker 1: should be pretty distrusting of pay to print journals instead
• 26:10
  Speaker 1: of like a scientific space.
• 26:11
  Speaker 2: Yeah, we are going to talk about predatory journalis in
• 26:13
  Speaker 2: a little bit too because of that exact issue. Right.
• 26:15
  Speaker 2: Another thing that I think is really important and I
• 26:17
  Speaker 2: think you and I could probably speak on this a
• 26:19
  Speaker 2: little bit. When it comes to peer reviewers and the
• 26:22
  Speaker 2: obligation or the task of reviewing a manuscript. You might
• 26:25
  Speaker 2: be going, sure, like, there's probably a criterion that you follow,
• 26:29
  Speaker 2: or you know, some kind of checklist that you might
• 26:31
  Speaker 2: look for, And the answer is no, there's not.
• 26:35
  Speaker 1: Nope.
• 26:35
  Speaker 2: I have collaborated on one journal that had something like that,
• 26:38
  Speaker 2: and I was also we started the null Hypothesis Journal
• 26:42
  Speaker 2: seven Dimensions, and we had set up some criterion for
• 26:45
  Speaker 2: what to look for too, But for all the reviewers,
• 26:47
  Speaker 2: we had like some checklists to kind of look for
• 26:49
  Speaker 2: certain things that fit the value of the journal and
• 26:51
  Speaker 2: stuff like that. But yeah, for the most part, I've
• 26:54
  Speaker 2: never reviewed for a journal, at least in behavior analysis,
• 26:57
  Speaker 2: that has had any sort of review criterion, any sort
• 27:01
  Speaker 2: of even training on how to do it. Like it's
• 27:05
  Speaker 2: just kind of like, hey, you can review this, and
• 27:06
  Speaker 2: we go okay, and then that's that's usually the extent
• 27:09
  Speaker 2: of what we what we're trained to do in that.
• 27:12
  Speaker 2: But I don't know, do you ever have you ever
• 27:13
  Speaker 2: seen any sort of formal criterion in your review processes
• 27:16
  Speaker 2: at all? I never have.
• 27:17
  Speaker 1: And actually I was asked to do a review before
• 27:20
  Speaker 1: I had ever had any kind of like, as you said,
• 27:23
  Speaker 1: training is not something that is typically offered. And so
• 27:26
  Speaker 1: I did my best, but I went it sort of
• 27:28
  Speaker 1: opened my eyes to again I didn't know what I
• 27:31
  Speaker 1: didn't know, and I didn't realize how like little I
• 27:33
  Speaker 1: understood about that process. And so I went to other
• 27:37
  Speaker 1: professors and sort of mentors in grad school and I said,
• 27:40
  Speaker 1: the next time that you are reviewing or you're an
• 27:43
  Speaker 1: editor for an article, would you let me do a
• 27:45
  Speaker 1: mock review and you can help teach me what I
• 27:48
  Speaker 1: should be doing in my review. And so I got
• 27:51
  Speaker 1: like three or four opportunities where they basically gave me
• 27:53
  Speaker 1: a real manuscript that was it either had already gone
• 27:57
  Speaker 1: through the whole process or it was actually go through
• 28:00
  Speaker 1: the process, and then I would write my own review,
• 28:03
  Speaker 1: and then they would give me feedback on my review
• 28:06
  Speaker 1: and help sort of shape for me. They would show
• 28:08
  Speaker 1: me their reviews that they had done, so I had
• 28:10
  Speaker 1: an idea of like where to start, and they would
• 28:12
  Speaker 1: give me feedback on what I had done, so that
• 28:14
  Speaker 1: I knew better how to submit better reviews. And that
• 28:18
  Speaker 1: was enormously enormously helpful, and I actually started to whenever
• 28:21
  Speaker 1: I have been a PCBA supervisor, I have when I've
• 28:25
  Speaker 1: given had the opportunity to review articles, I offered to
• 28:28
  Speaker 1: them say would you like to do this with me?
• 28:30
  Speaker 1: And like, you won't be a reviewer, but like, if
• 28:33
  Speaker 1: you would like to see this and write your own review,
• 28:35
  Speaker 1: I'll give you some reviews I've done. Then you can
• 28:39
  Speaker 1: have an opportunity to like practice this and get some experience,
• 28:42
  Speaker 1: and I'll do my best to give you some feedback.
• 28:44
  Speaker 2: And I'm like, I'm not sure that I'm the best
• 28:45
  Speaker 2: reviewer of the world. I tried.
• 28:46
  Speaker 1: I try to be, like, I really try to do
• 28:48
  Speaker 1: my best, but I think it was enormously helpful to
• 28:51
  Speaker 1: have practice opportunities with people who had done many of
• 28:54
  Speaker 1: these to help them sort of shape my sort of
• 28:56
  Speaker 1: repertoire around this.
• 28:57
  Speaker 2: Yeah, I love that. I love that so much. Yeah,
• 29:01
  Speaker 2: now we know who's the who, who's the who's who,
• 29:04
  Speaker 2: Who's the who's who? We know the who's who? We
• 29:07
  Speaker 2: know we know this is an episode about owls now Surprise. No,
• 29:11
  Speaker 2: so we know who's who that's involved in doing this.
• 29:15
  Speaker 2: You've got editors, you've got the journal, you get the
• 29:16
  Speaker 2: peer reviewers. But let's talk about the types of peer
• 29:18
  Speaker 2: review that are done, because I think this is the
• 29:20
  Speaker 2: part that people don't realize that there are multiple variations
• 29:24
  Speaker 2: of this theme of how we can review stuff. And
• 29:26
  Speaker 2: the first one is called a single anonymized review. And
• 29:30
  Speaker 2: in this type of review, the author does not know
• 29:33
  Speaker 2: who the reviewers are. So the author will submit their
• 29:36
  Speaker 2: manuscript they don't know who's going to review it. And
• 29:39
  Speaker 2: this is actually pretty commonly used generally speaking. But the
• 29:43
  Speaker 2: thing that's kind of unique about this is that the
• 29:45
  Speaker 2: authors don't know who the reviewers are, but the reviewers
• 29:48
  Speaker 2: know who the author is, right, Because that is that
• 29:52
  Speaker 2: is the difference here, Like, that's what the single anonymized
• 29:55
  Speaker 2: part of this is the only person that's anonymized is
• 29:58
  Speaker 2: the author that the reviewers. I'm sorry is the reviewers.
• 30:02
  Speaker 2: And so this is great for reviewers because they don't
• 30:04
  Speaker 2: have to worry about softening their feedback, but they are
• 30:07
  Speaker 2: going to maybe provide bias feedback as a result of
• 30:09
  Speaker 2: knowing who the authors are exactly. So there is like
• 30:11
  Speaker 2: a little bit of a conundrum there.
• 30:13
  Speaker 1: That actually was a lot of my early experience where
• 30:15
  Speaker 1: with single anonymized reviews, where if I was a reviewer,
• 30:18
  Speaker 1: I knew who the author was, or if I was
• 30:20
  Speaker 1: the author of the reviewers knew who I was, or
• 30:22
  Speaker 1: the other people who are also authors with me on
• 30:24
  Speaker 1: those and yeah, it does, I think have an influence
• 30:27
  Speaker 1: because if you see a name of someone who you
• 30:29
  Speaker 1: really respect who is an author on a paper, and
• 30:32
  Speaker 1: you're like, well, man, I really want to I really
• 30:33
  Speaker 1: want to I acknowledge this person who's like an expert,
• 30:36
  Speaker 1: they really know what they're talking about. I really want
• 30:38
  Speaker 1: to make sure that I can walk away having said like,
• 30:42
  Speaker 1: let's accept this for publication, even if that's not like
• 30:44
  Speaker 1: an implicit or an explicit thought that you're having, that
• 30:48
  Speaker 1: bias can exist. And I'm certain certain that it does
• 30:52
  Speaker 1: because I had that experience as a reviewer when I
• 30:54
  Speaker 1: saw a paper and I got to see how the
• 30:55
  Speaker 1: authors on who are on that, and I was like, wow, this,
• 30:58
  Speaker 1: like I know these people there are like really respected
• 31:00
  Speaker 1: people in the field.
• 31:01
  Speaker 2: Right.
• 31:01
  Speaker 1: Alternatively, you might see one where you see an author
• 31:03
  Speaker 1: who you really don't respect and say like, I really
• 31:06
  Speaker 1: don't want this to go to publication because I hate
• 31:08
  Speaker 1: that guy. Yeah, or person you.
• 31:11
  Speaker 2: Had somebody in mind. You're like, oh, I know who
• 31:13
  Speaker 2: that is. I know who I would not like the
• 31:15
  Speaker 2: good of.
• 31:15
  Speaker 1: A life, And so it's it does have an opportunity
• 31:17
  Speaker 1: in there, which means that you then get double or
• 31:20
  Speaker 1: triple anonymized reviews. And this type of review means that
• 31:23
  Speaker 1: nobody knows who anyone is. The author doesn't know the reviewers,
• 31:26
  Speaker 1: the reviewer doesn't know the author. This is the most
• 31:29
  Speaker 1: common in social science spaces, and in the event of
• 31:32
  Speaker 1: the triple anonymized review, the author's reviewers, and even the
• 31:35
  Speaker 1: editors identities are all hidden from one another. And again,
• 31:39
  Speaker 1: the idea is to reduce bias as much as possible
• 31:42
  Speaker 1: in those spaces.
• 31:43
  Speaker 2: Sure you've got open peer reviews. Open peer reviews are
• 31:46
  Speaker 2: super interesting. So for an open peer review, all identities
• 31:49
  Speaker 2: are known by all participants. The authors know, the reviewers, know,
• 31:53
  Speaker 2: and this is really good for transparency, Like when you
• 31:56
  Speaker 2: talk about kind of like building credibility of peer reviews,
• 31:59
  Speaker 2: like it's good to know who's reviewing what so that
• 32:02
  Speaker 2: you can identify possible conflicx of interest or what their
• 32:05
  Speaker 2: general expertise is. That's really helpful. And reviewers may not
• 32:08
  Speaker 2: actually prefer this though, because their comments can be directly
• 32:12
  Speaker 2: linked to them, which you know, I translate that as
• 32:15
  Speaker 2: they can't be mean. Yeah, yeah, because you always hear
• 32:18
  Speaker 2: about like reviewer two it's like kind of a jerk,
• 32:20
  Speaker 2: and I had that experience. I had the experience with
• 32:22
  Speaker 2: my first paper. Okay, the reviewer to was a complete asshole,
• 32:26
  Speaker 2: but we'll talk about what that looks like later too.
• 32:28
  Speaker 2: But like, open peer reviews are really interesting for that
• 32:30
  Speaker 2: reason that everybody knows who everybody is in that review process.
• 32:33
  Speaker 1: I think you could reasonably argue, like the transparency piece
• 32:36
  Speaker 1: of this is really good and you're not wrong, but
• 32:38
  Speaker 1: I think it's just nuanced because yeah, as you said,
• 32:40
  Speaker 1: there then becomes the opportunity for someone to think like
• 32:44
  Speaker 1: I want these authors to think well of me, or
• 32:47
  Speaker 1: I don't want them to be mad at me, or yeah,
• 32:49
  Speaker 1: or I don't like them, and I want to make
• 32:51
  Speaker 1: my opinion of them known. But it changes the dynamic
• 32:56
  Speaker 1: of how you're going to interact with that work if
• 32:58
  Speaker 1: you're thinking more about the people involved than not about
• 33:00
  Speaker 1: necessarily the work itself. Sure, exactly Relatedly, there is transparent
• 33:04
  Speaker 1: peer review. This type of review includes the publication of
• 33:07
  Speaker 1: decision letters along with the article, and while it aligns
• 33:11
  Speaker 1: more with the single or double anonymized reviews, it does
• 33:13
  Speaker 1: include the full publication of the results of the peer
• 33:16
  Speaker 1: review for public review, meaning that they can see what
• 33:19
  Speaker 1: decisions went into allowing that manuscript to make it to publication,
• 33:24
  Speaker 1: like sort of how it was discussed and shaped among
• 33:27
  Speaker 1: the people who looked at that article.
• 33:29
  Speaker 2: Right, you'll see the article comes out, it'll come out
• 33:31
  Speaker 2: in the journal. In some of the attachments you'll see
• 33:33
  Speaker 2: like the decision letters, the feedback that was provided by
• 33:36
  Speaker 2: all the reviewers, what edits were made, and any sort
• 33:39
  Speaker 2: of ads that are related to that article. Two.
• 33:48
  Speaker 1: Okay, we're back. The next type we want to talk
• 33:50
  Speaker 1: about is called a collaborative review. This sounds kind of neat.
• 33:54
  Speaker 1: I've never done it, but I like the idea here.
• 33:56
  Speaker 1: So this is most of the time you like get
• 34:00
  Speaker 1: and a manuscript I'm imagining back in like the eighties
• 34:03
  Speaker 1: and nineties, you are like mailed physically in an envelope,
• 34:07
  Speaker 1: like a hard copy of yes, But in the digital age,
• 34:11
  Speaker 1: you mostly get like an email, but you don't see
• 34:14
  Speaker 1: who the other reviewers are. Sometimes you don't even know
• 34:15
  Speaker 1: how many other reviewers there are. And you just do
• 34:18
  Speaker 1: that document and then you submit it. You work on
• 34:20
  Speaker 1: the document. You don't need any that's all anyway, In
• 34:23
  Speaker 1: the collaborative review, rather than having reviewers independently review the manuscript,
• 34:28
  Speaker 1: collaborative reviewers may include a team of people who work
• 34:30
  Speaker 1: together and then together they review and approve or reject
• 34:35
  Speaker 1: those papers, and this usually results in a unified report
• 34:38
  Speaker 1: from the collaborative team. And in some cases, the team
• 34:42
  Speaker 1: of reviewers may work directly with the author to provide
• 34:44
  Speaker 1: feedback and support until the manuscript is ready for publication.
• 34:49
  Speaker 1: I do kind of see some benefit in that. I
• 34:51
  Speaker 1: do think that, like it's almost like you have like
• 34:55
  Speaker 1: review of the reviewers. In a way, you have sort
• 34:58
  Speaker 1: of some oversight of the the people who are participating
• 35:01
  Speaker 1: in that review process, which I could see a lot
• 35:04
  Speaker 1: of benefit in.
• 35:04
  Speaker 2: Yeah, I think you get like some some like ioa
• 35:07
  Speaker 2: across reviewers, right, like you get some like at least
• 35:09
  Speaker 2: in some discussion too, Like you get like, can we
• 35:12
  Speaker 2: confirm that we all agree on this? Yes, A noe?
• 35:14
  Speaker 2: Why don't we agree on this? Let's go ahead and
• 35:16
  Speaker 2: look at evidence, Like there's some cool stuff that could
• 35:18
  Speaker 2: happen in those spaces exactly, Like I feel like I
• 35:21
  Speaker 2: would love that, and like I feel like everybody that
• 35:23
  Speaker 2: is in that space too, Like the reviewers learn more, Yeah,
• 35:26
  Speaker 2: because you're learning from other people and like seeing their viewpoints,
• 35:29
  Speaker 2: like you just mentioned all the feedback that you got
• 35:30
  Speaker 2: on the other stuff, Like I think that's a cool
• 35:31
  Speaker 2: way to go about that.
• 35:33
  Speaker 1: Well, most of the time that I've done a review,
• 35:35
  Speaker 1: I later got to see the other reviewers review. Yes,
• 35:39
  Speaker 1: like the other people involved their reviews, and I always
• 35:41
  Speaker 1: find it really useful to look over the things that
• 35:44
  Speaker 1: they said, what the action editor may be glommed onto
• 35:47
  Speaker 1: and like they said, like this reviewer made this point
• 35:50
  Speaker 1: I think is really important for the development of this manuscript,
• 35:53
  Speaker 1: and that all like you kind of do all of
• 35:55
  Speaker 1: it in a collaborative review, you would get that whole
• 35:58
  Speaker 1: process all at once, right, where like I really honed
• 36:01
  Speaker 1: in onto this one piece that to me was really
• 36:03
  Speaker 1: important in how we move forward with shaping this manuscript,
• 36:06
  Speaker 1: and someone else actually really glommed onto a different piece
• 36:10
  Speaker 1: that they thought to them was important, and I was like, oh,
• 36:12
  Speaker 1: I hadn't considered that angle before, and likewise they may
• 36:15
  Speaker 1: have said, like, I wasn't considering your angle before. So
• 36:17
  Speaker 1: I do kind of like that idea, and I think
• 36:20
  Speaker 1: it'd be kind of cool. And it seems like you
• 36:21
  Speaker 1: sort of get a version of that a little bit.
• 36:23
  Speaker 1: And the way that at least I've participated in these,
• 36:26
  Speaker 1: which is as I said, you often get to see
• 36:28
  Speaker 1: like the action editor's summary of all the reviewers and
• 36:31
  Speaker 1: their sort of final recommendations, and you also get to
• 36:34
  Speaker 1: see the other reviewers recommendations as well.
• 36:37
  Speaker 2: Yeah, absolutely, I love it. There's also post publication reviews,
• 36:41
  Speaker 2: and so this review is not like the initial peer review,
• 36:43
  Speaker 2: but this is an additional review that can be included
• 36:46
  Speaker 2: once a manuscript has been submitted and improved for publication.
• 36:49
  Speaker 2: So generally speaking of what happens is it allows additional
• 36:52
  Speaker 2: reviewers to come in and provide feedback regarding designs or
• 36:55
  Speaker 2: results or the writing, you know, the methods, and then
• 36:58
  Speaker 2: offer guidance on next step with an identified research line.
• 37:02
  Speaker 2: So what might happen is like, maybe an article is
• 37:04
  Speaker 2: published and they share some some cool insights. There are
• 37:07
  Speaker 2: some really great effects in their data, and then in
• 37:10
  Speaker 2: their results section they talk about a couple things, but
• 37:12
  Speaker 2: they don't talk about really much more like they're missing
• 37:15
  Speaker 2: some things. So an additional reviewer could come in and go,
• 37:17
  Speaker 2: these are all really great ideas. Also, if you're going
• 37:20
  Speaker 2: to do this, consider this, or maybe consider this X
• 37:23
  Speaker 2: question or why question, or you know, if you're going
• 37:26
  Speaker 2: to do this design again for replication purposes, maybe consider
• 37:29
  Speaker 2: this DV versus that DV, and so you can do
• 37:32
  Speaker 2: like a kind of an additional review that would be
• 37:34
  Speaker 2: like almost like supplemental or like an appendix to the
• 37:38
  Speaker 2: article the original article.
• 37:39
  Speaker 1: I don't think I've seen that happen, but it seems
• 37:41
  Speaker 1: like it's kind of like polishing it up to sort
• 37:43
  Speaker 1: of like, Yeah, if you're in the post publication review part,
• 37:47
  Speaker 1: then really at the point where like you're not making
• 37:49
  Speaker 1: any substantive changes, just like how can we really make
• 37:52
  Speaker 1: this thing shine? Like whereas yeah, where we can sort
• 37:55
  Speaker 1: of augment what currently exists a little bit. Yeah, then
• 37:58
  Speaker 1: the final one here is a transferable peer review. In
• 38:01
  Speaker 1: some instances, a manuscript may be deemed incompatible with a journal,
• 38:05
  Speaker 1: and it makes a lot of sense. You'll actually see
• 38:07
  Speaker 1: this where there are journals that they have a specific mission,
• 38:10
  Speaker 1: like their goal is to public types of articles because
• 38:13
  Speaker 1: they know who their audience is, and they're catering to
• 38:16
  Speaker 1: an audience as looking for specific types of articles. So,
• 38:19
  Speaker 1: for example, if you're a journal who caters to clinicians
• 38:23
  Speaker 1: who go to this journal because they want practical, real
• 38:26
  Speaker 1: world solutions that they can put right into their daily practice,
• 38:30
  Speaker 1: then you don't want that journal to be publishing a
• 38:33
  Speaker 1: bunch of sort of whimsical conceptual articles that reflect on
• 38:37
  Speaker 1: things that don't really have any practical application, right, And
• 38:40
  Speaker 1: I do think that those are papers that can have
• 38:42
  Speaker 1: a lot of merit and value, but they would not
• 38:44
  Speaker 1: necessarily be appropriate for that journal. So in those cases,
• 38:47
  Speaker 1: the journal may elect to transfer the review to a
• 38:50
  Speaker 1: more compatible journal. So if I wanted to publish a
• 38:52
  Speaker 1: paper and behavior and social issues, but the theme of
• 38:55
  Speaker 1: the manuscript was more experimental, they could have the option
• 38:58
  Speaker 1: to transfer my review to the Journal of the Experimental
• 39:01
  Speaker 1: Analysis of Behavior, which is a much more experimentally oriented
• 39:06
  Speaker 1: publication outlet.
• 39:07
  Speaker 2: Right, right, right, and nothing wrong with either of those journals,
• 39:10
  Speaker 2: just like just understanding that the purpose and the scope
• 39:12
  Speaker 2: of those journals. Right.
• 39:14
  Speaker 1: Yeah.
• 39:14
  Speaker 2: So both of us have published, and both of us
• 39:17
  Speaker 2: have been reviewers. I want to talk about that a
• 39:19
  Speaker 2: little bit because I think that it's valuable to have
• 39:21
  Speaker 2: that experience. So as an author, I have submitted I
• 39:25
  Speaker 2: believe six research journal research articles, and two of them
• 39:30
  Speaker 2: were accepted. The other four were rejected with feedback, okay,
• 39:32
  Speaker 2: And part of that is because my writing I've just
• 39:34
  Speaker 2: never been a very academic writer, Like I'm very like
• 39:37
  Speaker 2: aformal writer and just I've got to pull them up.
• 39:39
  Speaker 2: So I've gotten good feedback on those. But I will
• 39:42
  Speaker 2: say on the first article that I published, reviewer to
• 39:46
  Speaker 2: I want to bring this up because I think it's
• 39:47
  Speaker 2: important to understand again, there's no criterion for how reviewers
• 39:51
  Speaker 2: review papers. So the feedback that we got from this
• 39:54
  Speaker 2: reviewer was that we didn't understand behavior analysis wow, and
• 40:00
  Speaker 2: that they had cited a couple of things early in
• 40:03
  Speaker 2: the paper that they were like they never brought this
• 40:05
  Speaker 2: back up when we did, Like, so they were like
• 40:07
  Speaker 2: errors in their review. Yeah, And we were just kind
• 40:11
  Speaker 2: of like no, like I did talk about this, like
• 40:13
  Speaker 2: here's on this page. And actually our response to the
• 40:15
  Speaker 2: editor was we did respond to this, we did address this,
• 40:18
  Speaker 2: here's where they do this, this is this, And the
• 40:20
  Speaker 2: feedback was so blatantly bad that we went in and
• 40:25
  Speaker 2: we requested a new reviewer. Wow, we went, no, this
• 40:29
  Speaker 2: is not valid, Like, we are going to go to
• 40:31
  Speaker 2: the editor and we're going to ask for a new
• 40:32
  Speaker 2: reviewer because they they clearly did not read the paper
• 40:36
  Speaker 2: or they had a clear bias in their feedback. Yeah,
• 40:39
  Speaker 2: the editor granted us a new reviewer, and that new
• 40:42
  Speaker 2: reviewer went no, this is great, and and we were
• 40:44
  Speaker 2: able to publish. So like that was my first experience
• 40:47
  Speaker 2: as an author, wow, dealing with like a journal And
• 40:50
  Speaker 2: I was kind of like, wow, what an interesting thing,
• 40:52
  Speaker 2: like to find that like a peer is either biased
• 40:54
  Speaker 2: or incompetent or just didn't do the work right, like
• 40:57
  Speaker 2: a peer reviewer who's supposed to be a reviewer. So
• 40:59
  Speaker 2: like that, how was my experience there as an author,
• 41:01
  Speaker 2: but as a reviewer, Like, I've had the pleasure of
• 41:04
  Speaker 2: reading some journal articles that were like incredible topics, super intering,
• 41:08
  Speaker 2: interesting stuff, and they never get published because they got
• 41:11
  Speaker 2: rejected because of some missing element or because of something
• 41:14
  Speaker 2: like that. So like I keep all those because I
• 41:16
  Speaker 2: like to see, like if it shows up somewhere else eventually.
• 41:19
  Speaker 2: But right, you know, it's a bummer to read some
• 41:22
  Speaker 2: really great work and go oh, like another reviewer didn't
• 41:25
  Speaker 2: agree with me, and the editor ultimately didn't agree with
• 41:27
  Speaker 2: me on the review. So yeah, I don't know, have
• 41:28
  Speaker 2: you had any experiences like that?
• 41:30
  Speaker 1: Yeah, yeah, So I think it's probably fairly common. I
• 41:33
  Speaker 1: could be wrong, but I think it's fairly common to
• 41:36
  Speaker 1: be an author before you're a reviewer. Yeah, And I
• 41:39
  Speaker 1: think it makes a lot of sense because once you
• 41:40
  Speaker 1: have become an author in a published space, then they
• 41:44
  Speaker 1: can look at that. The editors of the journal can
• 41:46
  Speaker 1: look at that and say, like, well, they know, like
• 41:49
  Speaker 1: they've already met the criteria to be published in this journal,
• 41:52
  Speaker 1: so they already have some familiarity at least with the
• 41:55
  Speaker 1: kinds of things that we're looking for that would make
• 41:57
  Speaker 1: a manuscript publishable. And so my first one was an author,
• 42:01
  Speaker 1: and I found the reviews to be extremely helpful, the
• 42:05
  Speaker 1: overwhelming majority. I will say that, like the worst reviews
• 42:08
  Speaker 1: are as you said, when the reviewers don't do the work.
• 42:12
  Speaker 1: The worst reviews are the ones that lack critical analysis.
• 42:16
  Speaker 1: I've seen like terrible, terrible reviews, and it was because
• 42:20
  Speaker 1: it was like a paragraph where the reviewer was like,
• 42:22
  Speaker 1: this was awesome, no notes, publish as is.
• 42:24
  Speaker 2: Yeah.
• 42:25
  Speaker 1: I was like, hold on, like, let's do some critical
• 42:28
  Speaker 1: analysis here, like what worked, what didn't, Like there's got
• 42:32
  Speaker 1: to be at least a little bit more besides saying
• 42:34
  Speaker 1: everything's great, please publish right, elaborate please. And I've also
• 42:38
  Speaker 1: seen terrible reviews similar to yours, where they were like
• 42:41
  Speaker 1: this didn't happen, this didn't happen, this didn't happen. This
• 42:44
  Speaker 1: was bad, like don't publish, And I was like, well,
• 42:46
  Speaker 1: those things did happen. I could point to you where
• 42:48
  Speaker 1: they happened and.
• 42:49
  Speaker 2: Show you the specific evidence.
• 42:50
  Speaker 1: Yeah, both where I was one of the reviewers and
• 42:52
  Speaker 1: I got to see like that reviewers like once that
• 42:55
  Speaker 1: it was in the sort of decision process, and where
• 42:58
  Speaker 1: I was the author and I had like reviewer who
• 43:00
  Speaker 1: was saying things like you didn't mention this, and I
• 43:02
  Speaker 1: was like, except for the three times I did here
• 43:04
  Speaker 1: on this page to this on page of this right,
• 43:06
  Speaker 1: And they're like, well and this doesn't make any sense
• 43:08
  Speaker 1: how you talk about this, And I was like, how
• 43:09
  Speaker 1: would you talk about it?
• 43:10
  Speaker 2: Then?
• 43:10
  Speaker 1: Could you please go provide some like I would like
• 43:13
  Speaker 1: to know how I could write this differently, and like
• 43:15
  Speaker 1: I've also man, I've read papers where I remember one
• 43:19
  Speaker 1: time I wrote to one of my sort of mentors
• 43:22
  Speaker 1: who was helping to guide me through this process. I
• 43:25
  Speaker 1: was like, can you reject an article on bad writing alone?
• 43:28
  Speaker 1: She was like, no, you really shouldn't. I was like,
• 43:32
  Speaker 1: all right, fine, but I actually so I wanted to share,
• 43:37
  Speaker 1: Like one of my experiences with getting my mentorship on
• 43:40
  Speaker 1: this is they oriented me as a reviewer to the
• 43:44
  Speaker 1: value that our job as the reviewer is to try
• 43:47
  Speaker 1: to help shepherd this manuscript to publication it can be yes,
• 43:52
  Speaker 1: And I love that that approach is to think like, Okay,
• 43:56
  Speaker 1: you went through the time and effort to create this
• 43:58
  Speaker 1: document that you think should be out there in the world.
• 44:02
  Speaker 1: What do I need to look for to ensure that
• 44:05
  Speaker 1: this document can meet all the criteria that it would
• 44:08
  Speaker 1: make sense to be published? And I'm thinking about as
• 44:11
  Speaker 1: the reviewer, what's the mission of the journal That is
• 44:13
  Speaker 1: actually always forefront of my mind, right, And I'm thinking
• 44:16
  Speaker 1: like does everything that happen in this manuscript make sense?
• 44:20
  Speaker 1: And if not, is where the things that are missing?
• 44:23
  Speaker 1: Things that can be repaired? And so when I write
• 44:26
  Speaker 1: my review, I want my feedback to be as kind
• 44:30
  Speaker 1: as possible. These are people who often have sunk years
• 44:34
  Speaker 1: into delivering this manuscript, right, Like, they have put a
• 44:38
  Speaker 1: lot of time and effort into making this, and so
• 44:41
  Speaker 1: like you want to be considered of the fact that
• 44:43
  Speaker 1: they put so much work into this, Like you don't
• 44:46
  Speaker 1: have to be a dick.
• 44:47
  Speaker 2: Yeah, exactly, exactly exactly, don't be a jerk.
• 44:50
  Speaker 1: You don't have to be Yeah, just like think about
• 44:54
  Speaker 1: like if you have constructive criticisms, say that. And so
• 44:57
  Speaker 1: like when I've gone and I've read papers where I
• 44:59
  Speaker 1: was like, this is almost unreadable. But the feedback I
• 45:02
  Speaker 1: put was like, there's the sentences that like you didn't
• 45:05
  Speaker 1: have a period. I'm like, I'm not sure where this
• 45:07
  Speaker 1: sentence ended. It seemed like it start it's got went here,
• 45:11
  Speaker 1: and then it went a totally different direction, and they
• 45:13
  Speaker 1: seem like there are two topics. I'm like, I'm thinking,
• 45:15
  Speaker 1: probably what happened is you moved a section of the
• 45:17
  Speaker 1: paper around and you just missed that there was a
• 45:19
  Speaker 1: thing that sort of then left not connected, and like,
• 45:22
  Speaker 1: I want to give you as much benefit of the
• 45:23
  Speaker 1: doubt as I can, but I'm like, I don't know
• 45:25
  Speaker 1: how to read this part and have it make sense
• 45:27
  Speaker 1: to me, and so I'm like, I'm just not sure
• 45:29
  Speaker 1: where that thing would go. So I would often offer, like,
• 45:32
  Speaker 1: this is if this is what you meant, this is
• 45:34
  Speaker 1: how I would suggest writing it, so that you can
• 45:37
  Speaker 1: communicate that idea clearly.
• 45:38
  Speaker 2: Well. If I see that, I usually go like, the
• 45:40
  Speaker 2: entire paper may benefit from like spending a little bit
• 45:43
  Speaker 2: of time inform and style, like like making sure that
• 45:45
  Speaker 2: you have with somebody else review and catch these errors
• 45:48
  Speaker 2: just in case, to kind of clean that up a
• 45:49
  Speaker 2: little bit. Like I'll always give people a direction.
• 45:52
  Speaker 1: Yeah, And at the very least you can always find
• 45:53
  Speaker 1: like here's a space if you're not sure what to do.
• 45:55
  Speaker 1: Put in an ad. We're back. Don't put ads in manuscripts.
• 46:05
  Speaker 2: Don't put bad Yeah, don't put add in mainuscripts.
• 46:07
  Speaker 1: That's bad news, They're bad. I think there's a lot
• 46:10
  Speaker 1: to say about it. Honestly, I mostly find the process
• 46:13
  Speaker 1: extremely helpful. I like being a reviewer. I really like
• 46:17
  Speaker 1: getting feedback from reviewers, Yes, but it is. I have
• 46:20
  Speaker 1: had some experiences sometimes where, you know, it always feels
• 46:23
  Speaker 1: good to me as a reviewer if when I write
• 46:25
  Speaker 1: my review and then the action editor, who basically is
• 46:28
  Speaker 1: like they're overseeing all the reviewers, they know who the
• 46:31
  Speaker 1: reviewers are, They're like responsible for managing their deadlines, and
• 46:34
  Speaker 1: then they take the reviews and they synthesize that into
• 46:37
  Speaker 1: a feedback document. They always add their own feedback as well,
• 46:40
  Speaker 1: assuming that there's anything to add. The action editor does
• 46:43
  Speaker 1: a lot of work. And when I'm one of like
• 46:46
  Speaker 1: two reviewers and the action editor primarily cites my review
• 46:50
  Speaker 1: in their like synthesis of the reviews and their feedback,
• 46:53
  Speaker 1: I always feel really good about that. Yeah, as a reviewer,
• 46:55
  Speaker 1: I'm like, I feel like I'm on the right track
• 46:57
  Speaker 1: with that. I've had some reviews where I read a manuscript.
• 47:01
  Speaker 1: This actually happened I think within the last year, and
• 47:03
  Speaker 1: I remember reading it and I was like, there's some
• 47:05
  Speaker 1: really cool ideas in here, and I think this could
• 47:08
  Speaker 1: be a really cool paper. I'm like, there are just
• 47:10
  Speaker 1: some parts of it where I'm like it just it
• 47:12
  Speaker 1: cant There's so many missing pieces that I can't synthesize
• 47:15
  Speaker 1: it into a hole, and I'm like, if you can
• 47:17
  Speaker 1: just fill in those gaps, you'll be great. And the
• 47:19
  Speaker 1: other person literally wrote there like, this is amazing, best
• 47:22
  Speaker 1: idea anyone's ever had. I think you might maybe had
• 47:26
  Speaker 1: a typo don't fix anything, publish as is.
• 47:28
  Speaker 2: Wow.
• 47:29
  Speaker 1: The action editor came back and basically just agreed with
• 47:32
  Speaker 1: me completely, And actually, I actually ed are greatly elaborated
• 47:35
  Speaker 1: on the points that I made. I felt good about
• 47:37
  Speaker 1: that because I was like thinking that, I'm like, I
• 47:38
  Speaker 1: really tried to like scrutinize this. I'm like, I wanted
• 47:41
  Speaker 1: to bring a level of analysis to this is something
• 47:43
  Speaker 1: that could end up in people's hands where they're reading
• 47:46
  Speaker 1: it and they're trying to use this to do something
• 47:48
  Speaker 1: with their field, their practice, their thoughts, they're going to
• 47:51
  Speaker 1: write about it as well or something. And I'm like, so,
• 47:54
  Speaker 1: if I'm really clear on this, then I feel good
• 47:57
  Speaker 1: about this going out in the world. But if I
• 47:58
  Speaker 1: feel like there's pieces missing, I'm going to suggests that,
• 48:00
  Speaker 1: like they you know, they bring some additional level of
• 48:04
  Speaker 1: clarity to the writing and whatnot. And that doesn't necessarily
• 48:07
  Speaker 1: always mean reject either, Like it could mean this is
• 48:10
  Speaker 1: like you're on the right path, please revise and submit again.
• 48:13
  Speaker 1: It will get accepted. And I would say that, like,
• 48:16
  Speaker 1: I think I've also submitted maybe six or seven somewhere
• 48:19
  Speaker 1: ballparking there, it's not very many. Like a lot of
• 48:21
  Speaker 1: people who are professionals, they published dozens of manuscripts, they
• 48:24
  Speaker 1: have a lot of practice. But like, I don't think
• 48:26
  Speaker 1: I've I've never had a manuscript get totally rejected. I
• 48:30
  Speaker 1: have had in the total summer I've had, I've had
• 48:33
  Speaker 1: reviewers recommend that, but the the overall output has always
• 48:37
  Speaker 1: been something like either revise and resubmit or accept with
• 48:42
  Speaker 1: revisions right those but that always means like they've always
• 48:46
  Speaker 1: made it to publication. It just took some time to
• 48:49
  Speaker 1: work into a better place.
• 48:51
  Speaker 2: Yeah, yeah, let's talk about this then, Like so, so
• 48:54
  Speaker 2: here we are at the stage where you know, we
• 48:56
  Speaker 2: understand the peer review, we kind of heard the different types,
• 48:58
  Speaker 2: and the question that we would ask you at this
• 49:00
  Speaker 2: point in time is is this the best way to
• 49:02
  Speaker 2: go about actually scrutinizing and approving scientific research? And I
• 49:07
  Speaker 2: think there's probably some empirical question in there somewhere, but
• 49:11
  Speaker 2: I do think that like at the very least we
• 49:13
  Speaker 2: can talk about what it looks like scientifically and what
• 49:16
  Speaker 2: we can do going forward. So Accel and friends in
• 49:19
  Speaker 2: twenty twenty five, a very recent article actually discuss some
• 49:23
  Speaker 2: important points related to the current peer review process that
• 49:26
  Speaker 2: I think are worth evaluating. They cite specifically problems related
• 49:29
  Speaker 2: to a lack of reviewers and then further a lack
• 49:33
  Speaker 2: of qualified reviewers. They cite predatory journals, political and administrative
• 49:37
  Speaker 2: biases which we know to be true, and poor reliability
• 49:41
  Speaker 2: as general problems exist in the peer review process. So
• 49:44
  Speaker 2: what I'd like to do is take each one of
• 49:47
  Speaker 2: those and talk about some of those points and kind
• 49:49
  Speaker 2: of unpack them a little bit as we go forward.
• 49:51
  Speaker 2: So that's what we're gonna do right now.
• 49:52
  Speaker 1: Okay, that sounds good. Yeah, So let's start with quality.
• 49:54
  Speaker 1: So apparently eighteen percent of accepted papers, meaning it went
• 49:59
  Speaker 1: through the peer review and made it to publication, they
• 50:02
  Speaker 1: were eventually found to have statistical errors. This was thought
• 50:06
  Speaker 1: to be a result of poor scrutiny and the peer
• 50:08
  Speaker 1: review process. Axl and friends go on to cite that
• 50:11
  Speaker 1: research outcomes were poorly described, conclusions were not accurate, results
• 50:16
  Speaker 1: were not reproducible in some cases a huge issue in
• 50:19
  Speaker 1: many particularly psychological fields. Yeah, and so it's worth asking,
• 50:23
  Speaker 1: like what's the impact here, And if there is an
• 50:26
  Speaker 1: increasing rate of retracted articles after publications, it just means
• 50:32
  Speaker 1: that those errors are making it through a filter that's
• 50:34
  Speaker 1: supposed to stop those errors from getting through, and that
• 50:38
  Speaker 1: does happen. Like the error rate unlike blog posts is
• 50:42
  Speaker 1: through the roof, probably over one hundred percent somehow, Like
• 50:45
  Speaker 1: it's that bad. So like having it down to eighteen
• 50:47
  Speaker 1: percent is definitely a lot better than just let's blow
• 50:51
  Speaker 1: everything up and let people do whatever. But it is
• 50:54
  Speaker 1: an unacceptably high rate. I think for considering that these
• 50:58
  Speaker 1: are intended, these should have gone through a process that
• 51:01
  Speaker 1: prevented those errors from happening in the first place, and
• 51:04
  Speaker 1: that the whole point of that process was to stop
• 51:07
  Speaker 1: these errors from making it to publication. So it's either
• 51:10
  Speaker 1: people were rushed or they just didn't spend the time
• 51:14
  Speaker 1: and do the work to really make sure that everything
• 51:16
  Speaker 1: made sense. But like I have actually gone through sometimes
• 51:19
  Speaker 1: and tried to like recreate the graphs or alternatively reverse
• 51:23
  Speaker 1: engineer the data that would have produced that graph so
• 51:26
  Speaker 1: that I could understand like the numbers that went behind it. Yeah,
• 51:30
  Speaker 1: when I was more ambitious in my younger years, I
• 51:33
  Speaker 1: did that. But like that can be a really great
• 51:37
  Speaker 1: step in ensuring that like no steps were missed in
• 51:40
  Speaker 1: the process.
• 51:40
  Speaker 2: Yeah, yeah, yeah. We brought up predatory journals earlier, and
• 51:43
  Speaker 2: I think this is really important. It was reported that
• 51:45
  Speaker 2: as of twenty fourteen, there were nearly eight thousand predatory journals.
• 51:51
  Speaker 2: Whoa that published nearly half a million articles. Uh oh
• 51:56
  Speaker 2: so yeah, yeah, that's roughly. I did the math. It's
• 51:58
  Speaker 2: about sixty two to sixty three articles per journal.
• 52:01
  Speaker 1: Oh my god, I know.
• 52:02
  Speaker 2: So what does that mean? Well, it doesn't necessarily diminish
• 52:06
  Speaker 2: journals with good reputations, like the ones that have high
• 52:09
  Speaker 2: impact factors, that have good peer review processes, but it
• 52:11
  Speaker 2: does dilute the science pool. To do to researcher stating
• 52:14
  Speaker 2: that they would still and this was in the article,
• 52:17
  Speaker 2: they would still continue to publish in these journals despite
• 52:21
  Speaker 2: knowing they were predatory, and specifically there were researchers that
• 52:25
  Speaker 2: they cited that they were working in a space where
• 52:28
  Speaker 2: publication was a big deal. So one motivation for this
• 52:31
  Speaker 2: was that in the academic space, institutions abide by a publisher.
• 52:34
  Speaker 2: Parish viewpoint, which means that they're looking at the number
• 52:38
  Speaker 2: of publications, not the quality of publications. So predatory journals
• 52:41
  Speaker 2: are a really nice outlet to kind of boost those numbers,
• 52:44
  Speaker 2: like pump up those numbers without actually having the quality
• 52:46
  Speaker 2: behind it. Yeah.
• 52:48
  Speaker 1: I like sort of guarantees that you can publish something
• 52:50
  Speaker 1: because all you have to do is pay they have
• 52:51
  Speaker 1: it published, and then it will be published, and then
• 52:53
  Speaker 1: you can say, look at how many publications I have.
• 52:55
  Speaker 2: Yeah, I agree.
• 52:56
  Speaker 1: You know, I understand why universities want to you have
• 53:00
  Speaker 1: some like I guess criteria for like a minimum threshold
• 53:04
  Speaker 1: of publications to like earn tenure and get raises and
• 53:07
  Speaker 1: sometimes even maintain your job. But I think that sometimes
• 53:12
  Speaker 1: it artificially incentivizes the wrong thing. Like I think what
• 53:16
  Speaker 1: they want to be incentivizing is people doing good scientific work,
• 53:20
  Speaker 1: but they're focusing on the output of that work and
• 53:22
  Speaker 1: not the work itself. And so it's like, rather than
• 53:26
  Speaker 1: you publish an article gets get cited a lot and
• 53:29
  Speaker 1: becomes a major thing that is used by people in
• 53:33
  Speaker 1: the field because it was so influential and so important.
• 53:37
  Speaker 1: You get articles that are just published because they could
• 53:41
  Speaker 1: be published, you know, you get quantity over quality sometimes,
• 53:46
  Speaker 1: And I do think that I most authors, I think
• 53:49
  Speaker 1: most people who are in these spaces are really legitimately
• 53:52
  Speaker 1: trying to do really good work. I think that you,
• 53:55
  Speaker 1: but you do incentivize, I think, toward the wrong thing
• 53:58
  Speaker 1: in a lot of these places. Right, So biases occur
• 54:02
  Speaker 1: within the peer review process as well. As we mentioned earlier,
• 54:05
  Speaker 1: this isn't just reviewer biases. Sometimes there are systemic biases
• 54:09
  Speaker 1: that can favor some authors, some topics, methods, institutions, or groups.
• 54:14
  Speaker 1: If you ever noticed a researcher with an abundance of
• 54:17
  Speaker 1: publications in a certain journal, This can be found in
• 54:19
  Speaker 1: conference presentations as well, simply implying that, like that is
• 54:23
  Speaker 1: a home that the author has found where they can
• 54:25
  Speaker 1: publish a lot. Yeah, because either they it's easy for them,
• 54:29
  Speaker 1: or they're well known because it is one of the
• 54:31
  Speaker 1: single blind review processes, or you know other There might
• 54:35
  Speaker 1: be other things, but important to understand that like systemic
• 54:39
  Speaker 1: biases can favor these and like, let's take for example,
• 54:43
  Speaker 1: there are these journals. Calling them scientific journals is being
• 54:47
  Speaker 1: way more generous than I'm willing to be. But they
• 54:49
  Speaker 1: are journals that publish onlike pseudoscience, things, so you get complementary.
• 54:53
  Speaker 1: In alternative medicine journals, for example, there's several of them.
• 54:56
  Speaker 1: There's there's a whole bunch of them that are out there.
• 54:58
  Speaker 1: They're very likely to accept any publication that comes across
• 55:01
  Speaker 1: their desk that makes complimentary and alternative medicine look good. Yeah,
• 55:05
  Speaker 1: if it is critical of what they're doing, then they're
• 55:08
  Speaker 1: less likely to publish it. And that might be the
• 55:10
  Speaker 1: case for a lot of a lot of journals, like
• 55:12
  Speaker 1: if they're not necessarily going to publish something that is
• 55:14
  Speaker 1: critical of the journal or the mission of that journal,
• 55:17
  Speaker 1: but like they're going to be very friendly. Like you
• 55:20
  Speaker 1: come in and you're like, look, you guys, powdered unicorn
• 55:23
  Speaker 1: horn is like the greatest medicine ever and all you
• 55:27
  Speaker 1: have to do is wish it into existence and then
• 55:29
  Speaker 1: you will cure cancer. And these journals are like, sweet,
• 55:32
  Speaker 1: that sounds really good. Let's publish that right, no notes,
• 55:36
  Speaker 1: you've got it on lock. And like there are a
• 55:38
  Speaker 1: lot of journals I think that have that sort of bias,
• 55:40
  Speaker 1: and so like if you publish something that is in
• 55:45
  Speaker 1: line with things they like to read, then you know
• 55:48
  Speaker 1: it might get published with very little scrutiny.
• 55:49
  Speaker 2: Yeah, yeah, exactly. And the last thing that we'll bring
• 55:52
  Speaker 2: up to here is poor reliability, right, And this is
• 55:55
  Speaker 2: one of the greater concerns related to overall acceptance of
• 55:57
  Speaker 2: papers as poor science maybe get poorer science. Right, So
• 56:00
  Speaker 2: a bad article can go out and then somebody will
• 56:03
  Speaker 2: cite that article and it will continue forward until eventually,
• 56:07
  Speaker 2: now we have kind of like this whole of science
• 56:09
  Speaker 2: that is not actually good science because of that one
• 56:11
  Speaker 2: bad study to begin with. So a bad study can
• 56:13
  Speaker 2: perpetuate ongoing bad science and ultimately dilute the rigor of publication,
• 56:16
  Speaker 2: and rejected papers may get republished in other journals and
• 56:20
  Speaker 2: later win Nobel prizes. So there's another side of this where,
• 56:22
  Speaker 2: like good it's happened multiple times where like good science
• 56:26
  Speaker 2: was rejected as a result of poor peer review reliability. Right,
• 56:30
  Speaker 2: I'm telling you, like my article, like I brought up
• 56:32
  Speaker 2: the article about that reviewer too, that article has been
• 56:35
  Speaker 2: circulated thousands of times now because of the discussion paper.
• 56:39
  Speaker 2: But like we were told that it was going to
• 56:40
  Speaker 2: be rejected, and it was one of those things where
• 56:42
  Speaker 2: it's like it's had a pretty decent impact factor compared
• 56:46
  Speaker 2: to some other works. That I've done, and simply because
• 56:49
  Speaker 2: I think the topic is relevant and whatnot. But it
• 56:51
  Speaker 2: was one of those things that like, if it would
• 56:53
  Speaker 2: have been rejected, it wouldn't have gone out and it
• 56:55
  Speaker 2: wouldn't have influenced folks, right, yeah, yeah, And we also know,
• 56:59
  Speaker 2: according to currencyatistics that the reliability measures of peer reviewers
• 57:02
  Speaker 2: haven't really improved since the seventies. It's been pretty consistently
• 57:06
  Speaker 2: flat lined. And that's not to say that all peer
• 57:08
  Speaker 2: review is bad and that peer reviewers are terrible and
• 57:10
  Speaker 2: anything like that. We're not saying that, not by a
• 57:12
  Speaker 2: long shot. But it does highlight an inherent problem with
• 57:15
  Speaker 2: the system at large in that it is systematized, but
• 57:17
  Speaker 2: it's not structured in the way that it could be
• 57:20
  Speaker 2: to improve these processes. And there's and there's going to
• 57:23
  Speaker 2: be a couple solutions that we're going to talk about
• 57:24
  Speaker 2: in this next section.
• 57:25
  Speaker 1: Well, let's go ahead and make that transition though, And
• 57:28
  Speaker 1: what better way to make it transition than with some
• 57:29
  Speaker 1: ads of course. All right, So, as we said, like
• 57:39
  Speaker 1: the system that we have is a good system, it
• 57:43
  Speaker 1: could be better, though, we think, yeah, it's certainly better
• 57:46
  Speaker 1: than how most information makes it out into the world.
• 57:49
  Speaker 1: It's because I think there is a there's an inherent
• 57:52
  Speaker 1: reason for this, which is that this is a lengthy process.
• 57:55
  Speaker 1: As we said, manuscripts can take years to make it
• 57:59
  Speaker 1: to review and then another year or more to make
• 58:02
  Speaker 1: it to publication. It could be less like sometimes they're
• 58:04
  Speaker 1: really quick turn rounds, but like the process of running
• 58:07
  Speaker 1: a manuscript can take a long time. And like when
• 58:09
  Speaker 1: you've got like information that needs to be disseminated, like
• 58:12
  Speaker 1: right now. This is one of the reasons that like
• 58:14
  Speaker 1: TikTok and Instagram are like some of the places, and
• 58:17
  Speaker 1: like Twitter and blue Sky, like these are some of
• 58:19
  Speaker 1: the places where breaking news happens. Is because it's happening
• 58:22
  Speaker 1: faster than the news can report it. Yeah, and like
• 58:25
  Speaker 1: information is power, and people seek out information and want
• 58:28
  Speaker 1: to know what's going on. And so anyway, this process
• 58:32
  Speaker 1: is very slow, but it is like the best process
• 58:35
  Speaker 1: for filtering out really good information. But as we said,
• 58:39
  Speaker 1: there is a better way, There is a way forward
• 58:41
  Speaker 1: through this current view process. We can do better, and
• 58:44
  Speaker 1: there are some proposed solutions in the current process that
• 58:48
  Speaker 1: might facilitate that.
• 58:50
  Speaker 2: Yeah, So one of the first suggestions is open and
• 58:52
  Speaker 2: transparent peer review. The overarching goal of this type of review,
• 58:56
  Speaker 2: like we mentioned before, is that it can improve the
• 58:58
  Speaker 2: credibility of the review process by showcasing who the reviewers are,
• 59:02
  Speaker 2: what their feedback might have been, the entire decision process,
• 59:05
  Speaker 2: all the way through. It will make it so that
• 59:07
  Speaker 2: it's very clear and transparent on how we arrived this
• 59:10
  Speaker 2: decision and then leave it open for scrutiny from the
• 59:13
  Speaker 2: public or like other peers that might exist in the field.
• 59:15
  Speaker 2: So we do have that opportunity to do that as
• 59:17
  Speaker 2: part of the process.
• 59:18
  Speaker 1: Another one, as we've sort of alluded to, is reviewer training,
• 59:22
  Speaker 1: and this is like getting people ready to be reviewers.
• 59:26
  Speaker 1: We train reviewers how to conduct a review. We can
• 59:29
  Speaker 1: start to establish some criteria by which we do reviews,
• 59:33
  Speaker 1: like what kind of things are you looking for in
• 59:34
  Speaker 1: the review, and that like when you are crafting your
• 59:37
  Speaker 1: review that you are making sure you are accomplishing. Journals
• 59:41
  Speaker 1: not journals, but journals can increase reliability and validity through
• 59:46
  Speaker 1: systematic measures and ongoing oversight of the reviewers. And it's
• 59:50
  Speaker 1: like there are kind of versions of this process now,
• 59:52
  Speaker 1: they're just they're kind of done in whatever style makes
• 59:56
  Speaker 1: sense to the people who oversee those processes now, and
• 59:58
  Speaker 1: it's not very regular it or systematized, and it's also
• 1:00:02
  Speaker 1: a little opaque, like we don't really know how those
• 1:00:04
  Speaker 1: decisions are being made all the time.
• 1:00:06
  Speaker 2: Right right. We could also look at pre print peer review,
• 1:00:10
  Speaker 2: So pre prints are really great opportunities to share into
• 1:00:12
  Speaker 2: so many information about research prior to peer review. So
• 1:00:15
  Speaker 2: this is before that paper enters the peer review process,
• 1:00:18
  Speaker 2: and it's a great opportunity to share a study ahead
• 1:00:20
  Speaker 2: of time, receive public feedback about the manuscript prior to
• 1:00:22
  Speaker 2: formal submissions to publication, and then start kind of looking
• 1:00:25
  Speaker 2: at it from that perspective where you can actually start
• 1:00:27
  Speaker 2: doing the edits and get that feedback prior to any
• 1:00:30
  Speaker 2: sort of formal review from a journal that you're interested
• 1:00:33
  Speaker 2: in publishing in.
• 1:00:34
  Speaker 1: I think we did a mini on pre prints, did we?
• 1:00:37
  Speaker 1: Or did we full a dful episode. We've talked about
• 1:00:39
  Speaker 1: it before.
• 1:00:39
  Speaker 2: We did a mini on preregistration. We might have done, yeah,
• 1:00:44
  Speaker 2: but which is a different We didn't even talk about
• 1:00:46
  Speaker 2: it in this one. But like preprints are cool, like
• 1:00:47
  Speaker 2: you can write the manuscript and publish it and like, hey,
• 1:00:50
  Speaker 2: have at it before it ever goes anywhere.
• 1:00:52
  Speaker 1: Right right, Yeah, yeah, yeah, that's a cool IDEA reviewer
• 1:00:54
  Speaker 1: incentives recognition and availability. There is a severe lack of
• 1:01:00
  Speaker 1: research reviewers, especially in behavior analysis, and it makes sense.
• 1:01:03
  Speaker 1: Again as we mentioned, you don't get paid. There's no
• 1:01:05
  Speaker 1: incentive for review besides love of the game. They could
• 1:01:09
  Speaker 1: consider some monetary compensation. It can take a long time
• 1:01:13
  Speaker 1: to do reviews. You might spend many hours on these.
• 1:01:15
  Speaker 1: One of the early reviews I did, where again I
• 1:01:18
  Speaker 1: was very ambitious, is I read every article in the
• 1:01:21
  Speaker 1: reference list, every one of them. Wow of the manuscript
• 1:01:23
  Speaker 1: because I wanted to understand the history of what research
• 1:01:27
  Speaker 1: line led to that manuscript being done. And I learned
• 1:01:31
  Speaker 1: so much. It was actually one of my favorite ways
• 1:01:33
  Speaker 1: to learn about a topic is I got to see,
• 1:01:35
  Speaker 1: like I read them in chronological order of like when
• 1:01:39
  Speaker 1: things were published. I had to see each new iteration
• 1:01:41
  Speaker 1: on that line of research. It was really cool. Yeah,
• 1:01:43
  Speaker 1: but it took a very very long time. So anyway, sure,
• 1:01:46
  Speaker 1: if journals could give some kind of monetary compensation, this
• 1:01:49
  Speaker 1: is a money free labor of love. Generally speaking, that
• 1:01:53
  Speaker 1: might help. There could introduce some bias and problems there,
• 1:01:55
  Speaker 1: so like it would need to be really careful with
• 1:01:57
  Speaker 1: that system and again, I think it would have to
• 1:01:59
  Speaker 1: be based on merit and quality, but not necessarily quantity
• 1:02:03
  Speaker 1: or result.
• 1:02:04
  Speaker 2: Sure exactly. We can also look at recruitment processes. So
• 1:02:07
  Speaker 2: right now, most journals don't really have a formal process
• 1:02:10
  Speaker 2: for recruiting reviewers. I don't know about you, but I
• 1:02:13
  Speaker 2: got one day somebody went hey, will you review this?
• 1:02:15
  Speaker 2: And I went okay, and so like it, And then
• 1:02:18
  Speaker 2: now I was recruited as a reviewer. So like. It
• 1:02:20
  Speaker 2: wasn't like a process where I applied to be a
• 1:02:22
  Speaker 2: reviewer or had any of those situations I don't like.
• 1:02:25
  Speaker 2: I said, I don't know what your experience was, but
• 1:02:26
  Speaker 2: that was my experience kind of stepping into that space.
• 1:02:29
  Speaker 2: And if we can improve the criterion for who can
• 1:02:32
  Speaker 2: review and ensure there's a diverse reviewer pool, then the
• 1:02:36
  Speaker 2: quality of reviews can also improve. Right, So we have
• 1:02:38
  Speaker 2: people who can dedicate more time to reviews because they're
• 1:02:42
  Speaker 2: not reviewing as many per year. We've got diverse perspectives,
• 1:02:45
  Speaker 2: we've got people who are experts in different areas who
• 1:02:47
  Speaker 2: are willing to give time and expertise, Like, we can
• 1:02:50
  Speaker 2: do a lot of cool stuff if we have a
• 1:02:52
  Speaker 2: decent reviewer pool.
• 1:02:54
  Speaker 1: That diverse perspective thing that might be. That's extremely important piece, Yes,
• 1:02:59
  Speaker 1: extremely important, And I think it's very easy to fall
• 1:03:02
  Speaker 1: into the pattern of like we're all trained the same way,
• 1:03:04
  Speaker 1: we all agree with each other, we're basically in an
• 1:03:06
  Speaker 1: echo chamber. No, not in science. We want a lot
• 1:03:10
  Speaker 1: of people who are thinking about those different ways, and
• 1:03:12
  Speaker 1: we're not necessarily all going to agree, and then they
• 1:03:15
  Speaker 1: might not necessarily make a meaningful contribution every single time.
• 1:03:20
  Speaker 1: But we want the diversity of opinions in there so
• 1:03:22
  Speaker 1: that we can learn from each other.
• 1:03:23
  Speaker 2: Ye.
• 1:03:24
  Speaker 1: That has got to be one of the most important features.
• 1:03:26
  Speaker 2: I love that, Yeah, absolutely, all right.
• 1:03:28
  Speaker 1: The last one we have here is a more transparent
• 1:03:31
  Speaker 1: editorial process right now. Again, the decision making process is
• 1:03:35
  Speaker 1: a little bit opaque. We don't really understand exactly how
• 1:03:37
  Speaker 1: people are making their decisions. In some instances, an article
• 1:03:40
  Speaker 1: may be approved but rejected by the editor, and while
• 1:03:44
  Speaker 1: there is usually some explanation, they don't often give formal criteria,
• 1:03:48
  Speaker 1: although they might offer like they might offer at least
• 1:03:51
  Speaker 1: specific details. Sure, and making the entire process more clear
• 1:03:54
  Speaker 1: and transparent might improve the credibility of the journal and
• 1:03:57
  Speaker 1: the editor and allow some critique of the decision making
• 1:04:00
  Speaker 1: process and reduction in bias by having some more information there,
• 1:04:04
  Speaker 1: Because if we can compare the decision making process against
• 1:04:10
  Speaker 1: like articles that are similar, where one gets published and
• 1:04:13
  Speaker 1: one does not, then we can say, like, where, like
• 1:04:16
  Speaker 1: what was the difference in here that led to one
• 1:04:18
  Speaker 1: being published over the other. Was it in the decision
• 1:04:20
  Speaker 1: making process because of bias? Like if somebody came at
• 1:04:24
  Speaker 1: this from a position they're like, I just don't like
• 1:04:26
  Speaker 1: these people, or I just don't like this topic, or
• 1:04:28
  Speaker 1: I do very much like these people, and I do
• 1:04:30
  Speaker 1: very much like this topic. So I'm willing to like
• 1:04:32
  Speaker 1: let some things slide, Like those are helpful things for
• 1:04:35
  Speaker 1: us to know, right so that we can do better.
• 1:04:37
  Speaker 2: Absolutely absolutely so I think for me to like really
• 1:04:42
  Speaker 2: kind of drive this episode home, Like, I think it's
• 1:04:44
  Speaker 2: important to remember the peer review process is a process
• 1:04:46
  Speaker 2: that is a human invention, and a fairly modern one
• 1:04:49
  Speaker 2: at that, right, Like, it's a fairly new process in
• 1:04:51
  Speaker 2: the scope of science and humanity and all that in society,
• 1:04:55
  Speaker 2: Like all things, we're still learning how it works and
• 1:04:56
  Speaker 2: we're looking at ways to improve it. Thankfully, we have
• 1:04:59
  Speaker 2: the opportunity to be able to do this and to
• 1:05:01
  Speaker 2: improve this through some proposed solutions that we mentioned. And
• 1:05:04
  Speaker 2: we know that the peer review process is not perfect,
• 1:05:06
  Speaker 2: but and it is better than nothing, but it's also
• 1:05:08
  Speaker 2: pretty decent right now. Like if it's done well, it
• 1:05:11
  Speaker 2: can be pretty great. And so, like you know, I
• 1:05:14
  Speaker 2: don't think there's anything wrong with critiquing something that is
• 1:05:17
  Speaker 2: that exists in a field where we want to advance knowledge.
• 1:05:20
  Speaker 2: I think that we always have to be critical about that.
• 1:05:22
  Speaker 2: But I think the peer of view process can be
• 1:05:24
  Speaker 2: pretty good right now comparatively, and I think that we
• 1:05:28
  Speaker 2: can just make some tweaks and changes along the way
• 1:05:30
  Speaker 2: to make it a little bit better as we go.
• 1:05:32
  Speaker 1: Yeah, I think it's reasonable to conclude that the peer
• 1:05:36
  Speaker 1: review is the best process we have right now, yes,
• 1:05:40
  Speaker 1: for disseminating high quality information. And that does not mean
• 1:05:44
  Speaker 1: it could not be better. It could be a better process,
• 1:05:48
  Speaker 1: possibly a lot better of a process, but it is
• 1:05:51
  Speaker 1: a very good process that we have that we we
• 1:05:55
  Speaker 1: scrutinize these works so that the information can be I
• 1:06:00
  Speaker 1: guess I want to say, evaluated for its quality as
• 1:06:04
  Speaker 1: much as possible that by the time it makes it
• 1:06:07
  Speaker 1: to publication is widely available. That we have said, like
• 1:06:10
  Speaker 1: we've done our due diligence here. If you read this,
• 1:06:13
  Speaker 1: you can trust that, like a lot of work has
• 1:06:17
  Speaker 1: gone into ensuring that the information you're reading is as
• 1:06:20
  Speaker 1: clear and as accurate as it could possibly be. Yeah,
• 1:06:22
  Speaker 1: and like that's that's a really important thing, like that
• 1:06:25
  Speaker 1: we can trust where our information is coming from and
• 1:06:28
  Speaker 1: do so without bias against like a lot of things,
• 1:06:32
  Speaker 1: Like it's clear that this is not advancing some specific
• 1:06:35
  Speaker 1: agenda when it's done through good scientific process. It's kind
• 1:06:39
  Speaker 1: of like, you know, we're going through this process right
• 1:06:41
  Speaker 1: now in the United States, where like we can't trust
• 1:06:43
  Speaker 1: government websites anymore. They are deliberately withholding They're like if
• 1:06:48
  Speaker 1: they don't like the numbers, they fire the people in
• 1:06:50
  Speaker 1: charge of those numbers and they get new numbers. They
• 1:06:52
  Speaker 1: are withholding information, they're removing valid information. There are countries
• 1:06:57
  Speaker 1: with dictators that are famously rated as like lacking democracy
• 1:07:02
  Speaker 1: and having violent human rights abuses, who are no longer
• 1:07:06
  Speaker 1: being called like being called out for their human rights abuses.
• 1:07:09
  Speaker 1: They're basically saying everything's hunky dory over there. Nobody looked
• 1:07:12
  Speaker 1: behind the curtain, please, Right, That's the kind of thing
• 1:07:14
  Speaker 1: that's going on the US government so like we can
• 1:07:16
  Speaker 1: no longer trust our government, and that's the thing that
• 1:07:17
  Speaker 1: the people here voted for. We voted so that we
• 1:07:19
  Speaker 1: can put people in charge who will wreck the systems
• 1:07:22
  Speaker 1: that we had in place to allow for accurate dissemination
• 1:07:26
  Speaker 1: of information to happen, and now we can't trust it, right,
• 1:07:28
  Speaker 1: and that sucks, Like that really sucks. So at least
• 1:07:31
  Speaker 1: we have the institutions of science to hopefully continue to
• 1:07:34
  Speaker 1: be the guardrails against these evil forces.
• 1:07:37
  Speaker 2: Yeah, most definitely.
• 1:07:38
  Speaker 1: All right, Well, we've gone very long and we still
• 1:07:41
  Speaker 1: have to recommend some things. In fact, if you're joining
• 1:07:43
  Speaker 1: us for the first time, you may not know this.
• 1:07:45
  Speaker 1: Matter fact, you wouldn't know this. Probably at the end
• 1:07:47
  Speaker 1: of each discussion, we make some recommendations that usually just lighthearted,
• 1:07:50
  Speaker 1: fun things. Maybe they're heavy, I don't know, but it's
• 1:07:52
  Speaker 1: just stuff. It could be books, it could be music,
• 1:07:54
  Speaker 1: it could be games, it could be shows, it could
• 1:07:56
  Speaker 1: be activities.
• 1:07:57
  Speaker 2: Whatever.
• 1:07:57
  Speaker 1: We recommend some things and it's usually not really to
• 1:08:00
  Speaker 1: our topic at hand, and that will be the case today.
• 1:08:02
  Speaker 1: We also have some listener mail to get to because
• 1:08:04
  Speaker 1: we've been having some really cool interactions with our listeners,
• 1:08:07
  Speaker 1: which we very much appreciate, and if you would like
• 1:08:09
  Speaker 1: to write us and tell us your thoughts on peer review,
• 1:08:12
  Speaker 1: your experiences with peer review, we're very happy to read
• 1:08:15
  Speaker 1: those things, and in many cases we're happy to share
• 1:08:17
  Speaker 1: those things in the form of a listener mail. You
• 1:08:19
  Speaker 1: can read us directly by emailing us at info at
• 1:08:22
  Speaker 1: wwdwwdpodcast dot com. You can also reach us on the
• 1:08:25
  Speaker 1: social media platforms. We just recently read a listener mail
• 1:08:28
  Speaker 1: that came to us that way. Man, We look forward
• 1:08:29
  Speaker 1: to hearing from all of you. If you like to
• 1:08:31
  Speaker 1: support us, then you can join us over on Patreon.
• 1:08:34
  Speaker 1: We get ad free episodes, early episodes, all that kind
• 1:08:36
  Speaker 1: of good stuff, and I will shout out your name
• 1:08:38
  Speaker 1: in a list of people who help support us that
• 1:08:40
  Speaker 1: we very much appreciate. So with that being said, here
• 1:08:43
  Speaker 1: are the following people who have earned our thanks. Thank
• 1:08:45
  Speaker 1: you so much to Mike m Megan, Mike T, Justin,
• 1:08:48
  Speaker 1: Kim Brad, Stephanie, Brian, Ashley, Kiara and Charlie. Thank you
• 1:08:53
  Speaker 1: for being there. You guys are really the best.
• 1:08:55
  Speaker 2: You're the best.
• 1:08:57
  Speaker 1: You can also support us by leaving us a rating
• 1:08:59
  Speaker 1: and a review. Lever you listen to podcasts, you can
• 1:09:01
  Speaker 1: do one or the other or both. But we appreciate
• 1:09:03
  Speaker 1: if you have some nice things to say that you
• 1:09:05
  Speaker 1: say them to help other people find us. You can
• 1:09:07
  Speaker 1: tell a friend, you can like or share or do
• 1:09:09
  Speaker 1: whatever that just helps get the word out. That is
• 1:09:11
  Speaker 1: actually one of the most powerful things you can do,
• 1:09:13
  Speaker 1: beyond of course just being a supporter, which is great.
• 1:09:16
  Speaker 1: Also you can go steal someone's phone and subscribe to
• 1:09:19
  Speaker 1: the podcast on their phone. That's also good too, and
• 1:09:21
  Speaker 1: then give it back to them and give it back
• 1:09:23
  Speaker 1: to them. Yeah yeah, I don't just steal things. But
• 1:09:26
  Speaker 1: also also very importantly, thank you to my team of people,
• 1:09:28
  Speaker 1: without whom I could not make this podcast. So writing
• 1:09:31
  Speaker 1: and fact checking from Shane and myself, our social media
• 1:09:35
  Speaker 1: coordinator is Emma Wilson, and our audio engineer who makes
• 1:09:39
  Speaker 1: everything sound like a podcast episode so that you can
• 1:09:42
  Speaker 1: listen to it and hear it and you can critique
• 1:09:44
  Speaker 1: it and review it and go about your day is
• 1:09:46
  Speaker 1: justin So thank you very much to my team of people.
• 1:09:49
  Speaker 2: Y'all are the best.
• 1:09:50
  Speaker 1: You're one of those team of people.
• 1:09:51
  Speaker 2: Hey oh hey, yes, I am so are you look
• 1:09:53
  Speaker 2: at us?
• 1:09:54
  Speaker 1: Is there a thing I'm missing? Before we move on
• 1:09:55
  Speaker 1: to our listener mailn recommendations.
• 1:09:57
  Speaker 2: No, that covers it.
• 1:09:59
  Speaker 1: I think just one ad.
• 1:10:00
  Speaker 2: Oh, always with the ads.
• 1:10:10
  Speaker 1: All right, we're back with Let's start with some listener mail.
• 1:10:16
  Speaker 1: All right, this one comes to us from actually one
• 1:10:18
  Speaker 1: of our patrons. You don't have to be a patron
• 1:10:20
  Speaker 1: to write to us, but one of our patrons did,
• 1:10:22
  Speaker 1: and so thank you to Brian for writing in. This
• 1:10:24
  Speaker 1: is a following up on our episode. We published a
• 1:10:27
  Speaker 1: nostalgia episode on our board games. Brian ran with some
• 1:10:30
  Speaker 1: further recommendations. He said, I wanted to throw some other
• 1:10:32
  Speaker 1: suggestions your way that weren't mentioned in case you haven't
• 1:10:35
  Speaker 1: heard of them or just haven't played. If you have played,
• 1:10:37
  Speaker 1: what are your thoughts? He mentioned a game called Puerto
• 1:10:39
  Speaker 1: Rico Say it's a great combination of mechanics including role selection,
• 1:10:43
  Speaker 1: engine building, resource management, and shipping and trading, all with
• 1:10:46
  Speaker 1: variable turn order and a following mechanic that keeps you
• 1:10:49
  Speaker 1: busy the whole time. You're assuming the role of a
• 1:10:51
  Speaker 1: colonial governor during the age of exploration. In Puerto Rico Septima,
• 1:10:56
  Speaker 1: you take on the role of a witch trying to
• 1:10:58
  Speaker 1: become the next leader of your coven divides simultaneous action selection,
• 1:11:02
  Speaker 1: engine building, area control, and resource management. There's a huge
• 1:11:05
  Speaker 1: emphasis on risk versus reward and social suspicion. There are
• 1:11:09
  Speaker 1: a lot of things going on in this game, but
• 1:11:11
  Speaker 1: definitely a fun one. Dune Imperium blends deck building and
• 1:11:14
  Speaker 1: worker placement, all said on the backdrop theme of Dune,
• 1:11:17
  Speaker 1: the book series Dune Yes, with a lot of the
• 1:11:20
  Speaker 1: sort of character cards are now based off of, like
• 1:11:21
  Speaker 1: the actors who play the various characters in that story.
• 1:11:24
  Speaker 1: He says, this one's great, has so much replayability. The
• 1:11:27
  Speaker 1: rise of Ix expansion is great. Spice must flow, and
• 1:11:31
  Speaker 1: then he says Isle of Cats rescuing cats from a
• 1:11:35
  Speaker 1: doomed island by placing Polyannimo cat tiles onto your boat
• 1:11:39
  Speaker 1: combines card drafting and tile placement as a fun family one. First,
• 1:11:43
  Speaker 1: let's say thank you so much for writing in This
• 1:11:44
  Speaker 1: is a really fun list. I am familiar with Puerto Rico.
• 1:11:47
  Speaker 1: I've never played it, but it looks really good and
• 1:11:49
  Speaker 1: I know people who love it as one of their
• 1:11:51
  Speaker 1: favorite games. Exceptim I've never heard of, so I'm very
• 1:11:53
  Speaker 1: curious about this. This witch is coven one Dune Imperium.
• 1:11:57
  Speaker 1: I have played, Shane, I would be This is definitely
• 1:12:00
  Speaker 1: on the much heavier side of like board games, or
• 1:12:03
  Speaker 1: relative to what you'd be familiar with, But I'd be
• 1:12:05
  Speaker 1: curious to see how you'd enjoy it given your familiarity
• 1:12:08
  Speaker 1: and history with the Dunes series.
• 1:12:10
  Speaker 2: As I say, I just love doing so much that
• 1:12:12
  Speaker 2: I would I would sacrifice so much time to play it.
• 1:12:16
  Speaker 1: It is like it's legitimately a good game.
• 1:12:18
  Speaker 2: I played this game.
• 1:12:19
  Speaker 1: I think that for me, the battle section of the
• 1:12:22
  Speaker 1: game is not the strongest, but the rest of the
• 1:12:25
  Speaker 1: game mechanics work really well. And it's not that it's bad.
• 1:12:28
  Speaker 1: I've just I played other games that I think handled
• 1:12:29
  Speaker 1: it a little better. But it is. I love deck building,
• 1:12:32
  Speaker 1: I love work replacement of those two combined it works
• 1:12:35
  Speaker 1: really well. It is a very very good game, So
• 1:12:38
  Speaker 1: I'm not trying to criticize it. I think it's great.
• 1:12:40
  Speaker 1: Isle of Cats I've heard of, but I've never played
• 1:12:42
  Speaker 1: it seemed like one of those games I didn't have
• 1:12:44
  Speaker 1: much strategy, But the way that Brian described it, I'm
• 1:12:46
  Speaker 1: actually like really curious now because I do like Polyonimo games.
• 1:12:49
  Speaker 1: I like having lighthearted, easy games that are like quick
• 1:12:52
  Speaker 1: and fun to play but don't require a huge amount
• 1:12:54
  Speaker 1: of setup or teardown. So those are those are appreciated.
• 1:12:58
  Speaker 1: So anyway, thank.
• 1:12:59
  Speaker 2: You so much for writing in Yeah, that rules, thank you.
• 1:13:02
  Speaker 1: And those are more recommendations for other people if they
• 1:13:04
  Speaker 1: want to check out those games as well. So Puerto
• 1:13:06
  Speaker 1: Rico and doing Imperium I definitely understand to be heavier.
• 1:13:08
  Speaker 1: I'm not familiar enough with Septima, but it sounds like
• 1:13:10
  Speaker 1: it's on the heavier side as well. I love Cats.
• 1:13:12
  Speaker 1: I think is probably a nice, light, easy chill game
• 1:13:15
  Speaker 1: for people who are not really in the hobby to
• 1:13:16
  Speaker 1: pick up and play. Yeah, yeah, for sure, But I
• 1:13:18
  Speaker 1: think that is what you have to say about that
• 1:13:20
  Speaker 1: for right now, let's go ahead and move on to recommendations.
• 1:13:25
  Speaker 2: Oh so excited recommendations. So I'm going to recommend a
• 1:13:36
  Speaker 2: new zine publication. And the reason I'm recommending it is
• 1:13:40
  Speaker 2: because I wrote it, and so it's a zine. I'm
• 1:13:44
  Speaker 2: going to do a series of zines called Radical Care,
• 1:13:47
  Speaker 2: and it is really I was influenced by the book
• 1:13:50
  Speaker 2: let this Radicalize You about like doing something, and for me,
• 1:13:54
  Speaker 2: I know that my strength in advocacy is dissemination, so
• 1:13:58
  Speaker 2: like I feel like I do a pretty good job
• 1:14:00
  Speaker 2: seventy eighty the information, sharing information, making it palatable, making
• 1:14:02
  Speaker 2: it useful, and giving people direction. And so that's the
• 1:14:05
  Speaker 2: goal of Radical Care is taking some social issue that
• 1:14:08
  Speaker 2: has come up and giving information about that social issue,
• 1:14:12
  Speaker 2: giving direction and resources about that social issue, to really
• 1:14:16
  Speaker 2: paint a picture of why this is a problem, what
• 1:14:19
  Speaker 2: we can do to do better, and some actions to take.
• 1:14:21
  Speaker 2: And so the first issue is about homelessness and houselessness,
• 1:14:25
  Speaker 2: and so it was I started working on it when
• 1:14:27
  Speaker 2: we did our episode on this, and I was like, oh,
• 1:14:30
  Speaker 2: I think we could do something really cool. And so
• 1:14:31
  Speaker 2: finally I was able to finish it up and put
• 1:14:33
  Speaker 2: it together and it's like twenty five pages. It's a quick,
• 1:14:37
  Speaker 2: easy thing, and it's digital. It's free, and so we
• 1:14:40
  Speaker 2: make it so I'm making it so that it will
• 1:14:42
  Speaker 2: forever be free under every circumstances, every circumstance that we
• 1:14:46
  Speaker 2: can think of. So, yeah, we published the first one.
• 1:14:48
  Speaker 2: So Radical Care Issue one is about houselessness. I'm working
• 1:14:52
  Speaker 2: on issue too right now, and it will be about
• 1:14:54
  Speaker 2: the death penalty in capital punishment. So I'm going to
• 1:14:56
  Speaker 2: start tackling like pretty heavy stuff on that on that front.
• 1:14:59
  Speaker 1: So all right, you communist well, that link will be
• 1:15:06
  Speaker 1: the link for this freezine will be in the show
• 1:15:08
  Speaker 1: notes for this. So wherever you're listening to this, if
• 1:15:10
  Speaker 1: you just look at the episode and you scroll down
• 1:15:12
  Speaker 1: to where the notes and section stuff are, I actually
• 1:15:16
  Speaker 1: usually put recommendations right toward the top. There's like the
• 1:15:18
  Speaker 1: description of what the episode is, and right under that
• 1:15:20
  Speaker 1: is are recommendations. So you should have a clickable link
• 1:15:22
  Speaker 1: there that will take you to this very cool I'm
• 1:15:24
  Speaker 1: recommending a board game as I do. This is one
• 1:15:27
  Speaker 1: that I have played fairly recently. It is a style
• 1:15:30
  Speaker 1: of mechanic that's pretty unique. It's called flick and right.
• 1:15:34
  Speaker 1: There are lots of something and right type games, most
• 1:15:37
  Speaker 1: commonly roll and right, and that's where you roll a
• 1:15:40
  Speaker 1: pair of dice or some amount of dice. And there's
• 1:15:42
  Speaker 1: also flip and write or also called draw and right,
• 1:15:45
  Speaker 1: where you have a deck of cards, you'll flip it
• 1:15:47
  Speaker 1: over and then you'll take some things. The way that
• 1:15:49
  Speaker 1: it works is you have essentially everyone has their own
• 1:15:51
  Speaker 1: sort of score pad, and you have dry erase markers,
• 1:15:54
  Speaker 1: and there's a board in the middle of the table
• 1:15:56
  Speaker 1: where the board has sort of big plastic side along
• 1:16:00
  Speaker 1: the edge of it, and you have these little wooden
• 1:16:02
  Speaker 1: discs you flick on to the board and you can
• 1:16:05
  Speaker 1: knock other people's wooden discs. And you're flicking five wooden
• 1:16:09
  Speaker 1: discs that all have a number one through five on them,
• 1:16:12
  Speaker 1: and then depending on where they land, you can use
• 1:16:14
  Speaker 1: them to score different sections of your scoring pad. So
• 1:16:18
  Speaker 1: there's an element of you trying to flick your disc
• 1:16:20
  Speaker 1: into a place where you can get it, to try
• 1:16:22
  Speaker 1: to knock people out of positions that you don't want
• 1:16:24
  Speaker 1: them to score very well, and then for you to
• 1:16:27
  Speaker 1: just strategically use your scoring disc to fill out your
• 1:16:30
  Speaker 1: scoring pad the best that you can. Super easy game.
• 1:16:32
  Speaker 1: You can play it, and once you're familiar with it,
• 1:16:34
  Speaker 1: I think you can play it probably twenty minutes, yeah,
• 1:16:37
  Speaker 1: because you just go a few rounds doing this and
• 1:16:39
  Speaker 1: then that's it. It was a very entertaining little Johns
• 1:16:42
  Speaker 1: where you're flicking discs onto a board and then trying
• 1:16:44
  Speaker 1: to fill out your score sheet the best you can.
• 1:16:47
  Speaker 1: Not particularly complicated. I think most people can pick this
• 1:16:49
  Speaker 1: up and play it. Some of the scoring and symbols
• 1:16:51
  Speaker 1: are a little weird. I think they could have done
• 1:16:53
  Speaker 1: a better job explaining some of the symbols on there,
• 1:16:55
  Speaker 1: but if you read through it and you like double
• 1:16:57
  Speaker 1: check where they reference those things, it'll be come fairly claim.
• 1:17:00
  Speaker 1: In the rule books, I think only four or five pages,
• 1:17:02
  Speaker 1: like for most board game rule books, it's pretty pretty light.
• 1:17:05
  Speaker 2: Yeah yeah, and like in.
• 1:17:06
  Speaker 1: Addition to like one of those pages is credits, and
• 1:17:08
  Speaker 1: one of them is if you want to play like
• 1:17:10
  Speaker 1: a variant of the game or something, so like it's
• 1:17:12
  Speaker 1: probably only three pages of rules, really.
• 1:17:14
  Speaker 2: Yeah, yeah, for sure. All right.
• 1:17:15
  Speaker 1: Anyway, so that is Sonora the Board Game. If you're
• 1:17:19
  Speaker 1: interested in checking out this wild flick and write game,
• 1:17:22
  Speaker 1: and then the new zine and by Shane Spiker, Radical
• 1:17:25
  Speaker 1: Care issue number one, Free to All, Free to All.
• 1:17:29
  Speaker 1: All right, well, we've gone very long, so we should
• 1:17:30
  Speaker 1: probably wrap this one up. I think we've said all
• 1:17:32
  Speaker 1: the thanks and all the credits and done all the
• 1:17:34
  Speaker 1: things that we need to do. Is there anything you
• 1:17:35
  Speaker 1: like to add or anything that I forgot before.
• 1:17:37
  Speaker 2: We move on? Nothing today?
• 1:17:39
  Speaker 1: All right, we've accomplished peer review and reform of peer review.
• 1:17:43
  Speaker 1: Here on why we do what we do. I am
• 1:17:45
  Speaker 1: el imming today. Thank you all for listening. This is
• 1:17:49
  Speaker 1: Abraham and.
• 1:17:50
  Speaker 2: This is Shane. We're out Sea.
• 1:17:53
  Speaker 1: You've been listening to Why We Do What We Do.
• 1:17:56
  Speaker 2: You can learn more about this and other episodes by
• 1:17:58
  Speaker 2: going to ww D w w D podcast dot com.
• 1:18:02
  Speaker 2: Thanks for listening, and we hope you have an awesome day.