Latest / Daniel and Kelly’s Extraordinary Universe / How does scientific peer review work?

Transcript

How does scientific peer review work? · Daniel and Kelly’s Extraordinary Universe

• 0:07
  Speaker 1: Science is supposed to be our best way of revealing
• 0:10
  Speaker 1: the truth about the universe. But science is constantly being
• 0:14
  Speaker 1: updated and corrected, and sometimes we learn after the fact
• 0:17
  Speaker 1: that a study was flawed or even was shoddy to
• 0:20
  Speaker 1: begin with.
• 0:21
  Speaker 2: So how do scientists decide.
• 0:23
  Speaker 1: Whether a new result is robust or not? And how
• 0:26
  Speaker 1: does the general public know when the science is settled
• 0:29
  Speaker 1: or about to be upended. No system is perfect, but
• 0:32
  Speaker 1: it's important that the process is transparent.
• 0:35
  Speaker 2: So today we're.
• 0:35
  Speaker 1: Going to shine a light on the inner workings of
• 0:37
  Speaker 1: one of the crucial parts of the scientific process, peer review.
• 0:41
  Speaker 1: How does it work? Who are these peers? Can peer
• 0:45
  Speaker 1: reviewed papers be wrong? And does that mean that all
• 0:48
  Speaker 1: of science is questionable? Welcome to Daniel and Kelly's peer
• 0:52
  Speaker 1: reviewed universe. Hi.
• 1:07
  Speaker 3: I'm Daniel. I'm a particle physicist and I've published over
• 1:10
  Speaker 3: a thousand papers, but haven't read most of them.
• 1:14
  Speaker 4: Hi.
• 1:14
  Speaker 2: I'm Kelly Wieder Smith and I'm confused. What does that mean? Daniel?
• 1:18
  Speaker 2: How have you published over a thousand papers at the LHC.
• 1:22
  Speaker 1: Does everybody who is reading Oh that's cheating, man, that's cheating.
• 1:27
  Speaker 3: Well, the way the collaboration works is everybody who's contributed
• 1:31
  Speaker 3: to the project is an author. And when you're an author,
• 1:34
  Speaker 3: you get included on every paper that comes out during
• 1:37
  Speaker 3: your authorship period. And ATLAS is a huge collaboration of
• 1:41
  Speaker 3: lots of clever people. We put out about one hundred
• 1:43
  Speaker 3: papers a year. Every single one has my name on it,
• 1:46
  Speaker 3: and I haven't read most of them. I couldn't even
• 1:48
  Speaker 3: explain the titles of some of them to you.
• 1:50
  Speaker 1: So in my so fields have their own cultures. In
• 1:54
  Speaker 1: my field, in my culture, that would ethically not be okay.
• 2:01
  Speaker 1: You should be responsible for what is in every paper
• 2:03
  Speaker 1: that your name is on, in my opinion.
• 2:06
  Speaker 3: And I was in another collaboration previously where things work differently,
• 2:09
  Speaker 3: where you weren't an author on a paper unless you
• 2:11
  Speaker 3: asked to be, and nobody questioned it. If you had
• 2:14
  Speaker 3: the right to be an author, if you were on
• 2:16
  Speaker 3: the list, you could be. But it was an opt
• 2:17
  Speaker 3: in and I thought that was much better. The lengths
• 2:20
  Speaker 3: of all the list was shorter, and authorship meant something.
• 2:23
  Speaker 3: But here you can ask to be removed, but you
• 2:26
  Speaker 3: have to go through this process and it would be
• 2:28
  Speaker 3: like I'd be doing it twice a week every week.
• 2:30
  Speaker 1: So wow, So what does your CV include all of
• 2:35
  Speaker 1: those papers or just the ones that you feel you've
• 2:37
  Speaker 1: actually contributed to, So.
• 2:39
  Speaker 3: That depends the CV I make and share. People only
• 2:41
  Speaker 3: includes papers that I actually did anything on, or had
• 2:44
  Speaker 3: ideas for, or really contributed to. But sometimes an institution
• 2:49
  Speaker 3: wants a list of your official papers, and then I
• 2:52
  Speaker 3: have to like include all I don't even know the
• 2:54
  Speaker 3: number one thousand plus papers at this point.
• 2:57
  Speaker 2: Oh poor dude.
• 2:58
  Speaker 3: Now, so.
• 3:02
  Speaker 2: How do you get them? Ball? Do you just copy
• 3:04
  Speaker 2: them from Google Scholar?
• 3:05
  Speaker 3: Yeah, Particle physics is pretty good at automation and stuff
• 3:08
  Speaker 3: like this, so we have our own databases of papers
• 3:12
  Speaker 3: and it's pretty easy to download that kind of stuff.
• 3:15
  Speaker 3: But yeah, it's pretty tough to have so many papers.
• 3:17
  Speaker 2: Oh, so many pages you have to print out. The
• 3:20
  Speaker 2: file for the PDA for your resume is so big.
• 3:23
  Speaker 3: How about you, Kelly, how many papers do you haven't?
• 3:25
  Speaker 3: Have you finished that paper from your thesis yet?
• 3:27
  Speaker 2: Oh I'm feeling angry now, Dan, I have not.
• 3:32
  Speaker 1: I've been busy with other things, including my new Ducks
• 3:36
  Speaker 1: and geese. But when goats and goats, which I won't
• 3:39
  Speaker 1: mention of.
• 3:40
  Speaker 3: My co authors, do your ducks and geese and goats?
• 3:42
  Speaker 3: Go on your CV. That's my question. No, uh, students supervised?
• 3:46
  Speaker 3: Come on?
• 3:46
  Speaker 2: Oh yeah, no, I've got that list.
• 3:48
  Speaker 1: But in the Great, the Great Goat, I think I've
• 3:53
  Speaker 1: got I haven't I haven't counted. I think I've got
• 3:56
  Speaker 1: something like thirty that. My pace has slowed since I
• 4:00
  Speaker 1: writing POPSI, but you know I still get a couple
• 4:03
  Speaker 1: in there Every year.
• 4:04
  Speaker 3: My CV has something whimsical and unseerious on it. Yours
• 4:07
  Speaker 3: is totally professional.
• 4:09
  Speaker 2: Ah, yes, what whimsical thing do you have on yours?
• 4:17
  Speaker 4: Oh?
• 4:17
  Speaker 3: I'm just gonna leave that as an easter eggs. Oh.
• 4:19
  Speaker 3: I put it in there to see if anybody actually
• 4:21
  Speaker 3: reads my CV, because if they don't respond to that,
• 4:23
  Speaker 3: I'm like, hmm, you didn't really read it. Okay, yeah
• 4:26
  Speaker 3: check it out.
• 4:26
  Speaker 2: All right, Well, I'll be looking up your CV. Do
• 4:29
  Speaker 2: you haven't updated? Here's the question.
• 4:30
  Speaker 1: Most scholars don't update their CV, but like once every
• 4:34
  Speaker 1: half decade.
• 4:35
  Speaker 2: Is your online CV updated?
• 4:38
  Speaker 3: Yeah? I think I updated a couple months ago.
• 4:40
  Speaker 2: Oh all right, bravo, good out.
• 4:43
  Speaker 1: All right, Well today we're not talking about the thousands
• 4:45
  Speaker 1: of publications that Daniel has, but we're all very proud
• 4:48
  Speaker 1: of Daniel for all of his hard.
• 4:50
  Speaker 3: Work, or the tens of geese and ducks that Kelly
• 4:54
  Speaker 3: is currently raising, though we're very proud of her and
• 4:56
  Speaker 3: her literally extended family.
• 4:58
  Speaker 1: Well, I don't have that many, and I wouldn't claim
• 5:01
  Speaker 1: other people's ducks that I didn't help out with as
• 5:03
  Speaker 1: my own ducks.
• 5:04
  Speaker 3: Oh wow, now no.
• 5:09
  Speaker 1: But anyway, today we are answering a question from a
• 5:13
  Speaker 1: superfan and we're going to go into a bit more detail.
• 5:16
  Speaker 1: This question inspired us to make a whole episode about
• 5:19
  Speaker 1: the process of peer reviews. So let's go ahead and
• 5:21
  Speaker 1: hear what Stephen from British Columbia has to say.
• 5:25
  Speaker 4: Hi, Daniel and Kelly, this is show super fan Stephen
• 5:29
  Speaker 4: from British Columbia, Canada. My question today has to do
• 5:34
  Speaker 4: with how scientific studies are peer reviewed?
• 5:38
  Speaker 1: Is it me?
• 5:39
  Speaker 4: Or is science denial spreading these days? And it came
• 5:44
  Speaker 4: to my attention recently that not all peer reviewed studies
• 5:48
  Speaker 4: are replicated, And then it occurred to me, doesn't this
• 5:53
  Speaker 4: automatically cast doubt on a discovery if a study is
• 5:57
  Speaker 4: not replicated but is still considered peer reviewed.
• 6:02
  Speaker 3: I would love to.
• 6:04
  Speaker 5: Hear an explanation about how the peer reviewing process works
• 6:09
  Speaker 5: and how scientists determine when a fact is a fact,
• 6:14
  Speaker 5: a finding is a finding, and how the community comes.
• 6:18
  Speaker 4: To a consensus on new discoveries.
• 6:21
  Speaker 3: I also think your audience.
• 6:23
  Speaker 4: Could use some tools for how to determine what studies
• 6:27
  Speaker 4: are legitimate and how to spot a bad study. How
• 6:33
  Speaker 4: should one respond to someone who claims a scientific study
• 6:39
  Speaker 4: is bogus. Thank you Daniel and Kelly for this great
• 6:43
  Speaker 4: podcast and for explaining science topics to the masses.
• 6:47
  Speaker 3: Thank you Stephen for asking this question. I think it's
• 6:50
  Speaker 3: really the right moment to dig into this. You know,
• 6:52
  Speaker 3: when we see institutions being attacked, when we see science
• 6:56
  Speaker 3: being denied, when we see the whole process of science
• 6:59
  Speaker 3: being q and I think it's important to shine some
• 7:02
  Speaker 3: light on how does this all work, what does it mean,
• 7:04
  Speaker 3: especially for those folks who are not scientists, who don't know,
• 7:07
  Speaker 3: what does it mean for a paper to be peer reviewed?
• 7:09
  Speaker 3: What is bad science? Thank you Stephen for asking this
• 7:12
  Speaker 3: question and giving us the chance to dig into all
• 7:15
  Speaker 3: of this.
• 7:15
  Speaker 1: That's right, and so the question in particular is about
• 7:18
  Speaker 1: peer review for scientific manuscripts. But peer review actually happens
• 7:22
  Speaker 1: at multiple stages in the process of doing science, and
• 7:25
  Speaker 1: so we thought we would take a step back and
• 7:27
  Speaker 1: talk about peer review from the very beginning, which is
• 7:30
  Speaker 1: when your idea becomes a grant. And this is one
• 7:34
  Speaker 1: of the less fun parts about science is writing this
• 7:38
  Speaker 1: grant because peer review for grants is harsh.
• 7:41
  Speaker 3: This is like the science version of when a build
• 7:43
  Speaker 3: becomes a law. We should have written a song for this.
• 7:45
  Speaker 1: No, well we still can. You know, we could write
• 7:47
  Speaker 1: it and then we can add it to the top. No,
• 7:51
  Speaker 1: we won't do that to everyone.
• 7:52
  Speaker 3: So Kelly, tell me what is your process of grant writing?
• 7:55
  Speaker 3: First of all, who do you submit your grants to?
• 7:58
  Speaker 3: And what's it like for you to prepare that?
• 8:00
  Speaker 1: So I submit my grants to the National Science Foundation.
• 8:03
  Speaker 1: And actually it's possible that in the last six months
• 8:06
  Speaker 1: the process has changed a bit because I know the
• 8:09
  Speaker 1: new administration is shaking things up.
• 8:11
  Speaker 3: The agency formerly known as the National Science Foundation.
• 8:15
  Speaker 2: No, what is it called now?
• 8:16
  Speaker 3: I'm joking?
• 8:17
  Speaker 2: Oh god, anything is believable, Daniel.
• 8:20
  Speaker 1: At this point, all right, the institution still known as
• 8:24
  Speaker 1: the NSF or the National Science Foundation. So for this
• 8:28
  Speaker 1: you write a grant, and there's different programs that accept
• 8:31
  Speaker 1: different kinds of grants, but often the grant is like
• 8:33
  Speaker 1: fifteen pages where you go into extreme detail and what
• 8:38
  Speaker 1: we know about the question you want to ask based
• 8:41
  Speaker 1: on the literature that's already out there, extreme detail and
• 8:44
  Speaker 1: the experimental design, and then a lot of detail about
• 8:47
  Speaker 1: what you think the implications will be of your results.
• 8:50
  Speaker 3: So you talk about why this question is interesting, to
• 8:53
  Speaker 3: argue that your work will help answer it, and then
• 8:56
  Speaker 3: you lay out in detail how are you going to
• 8:58
  Speaker 3: spend every dollar, who's going to do what, and when
• 9:00
  Speaker 3: you get like a whole gan chart for like when
• 9:02
  Speaker 3: everything is going to happen yep, And then you have
• 9:04
  Speaker 3: to lay out also like the products for the grant, right, yeah.
• 9:07
  Speaker 1: Yeah, the budgets and so the you're supposed to There's
• 9:09
  Speaker 1: a section called intellectual merit where you talk about how
• 9:12
  Speaker 1: many papers you think you're going to end up producing,
• 9:14
  Speaker 1: what big questions you're going to answer, And then there's
• 9:16
  Speaker 1: also a section called broader impacts, where you talk about
• 9:18
  Speaker 1: how your work is going to benefit society writ large.
• 9:22
  Speaker 1: And you know, for my grants in the past, it's
• 9:24
  Speaker 1: been things like, you know, my information is going to
• 9:26
  Speaker 1: be helpful to fisheries managers who are managing fish populations,
• 9:30
  Speaker 1: which is like a literally a multi billion dollar industry
• 9:34
  Speaker 1: for like people who want to go out fishing and stuff.
• 9:36
  Speaker 1: So most of my research has been on fish and
• 9:38
  Speaker 1: their parasites. Also intellectual merit will include things like how
• 9:42
  Speaker 1: many students you're going to train and what skills you're
• 9:44
  Speaker 1: going to give them that they can then use when
• 9:46
  Speaker 1: they go out in the workforce. And depending on the
• 9:49
  Speaker 1: field you're in, sometimes your broader impacts will be things like, Hey,
• 9:52
  Speaker 1: this could be what we learn about the brains of
• 9:55
  Speaker 1: this fish could one day help us produce a new
• 9:57
  Speaker 1: medication for anxiety or something like. So you talk about
• 10:01
  Speaker 1: how your stuff would impact the broader society.
• 10:04
  Speaker 3: And so for those folks who might think, hey, scientists
• 10:07
  Speaker 3: are just cashing in on the government gravy train, tell
• 10:10
  Speaker 3: us like, is this the kind of thing you throw
• 10:12
  Speaker 3: together in an afternoon and then can confidently think it
• 10:15
  Speaker 3: will be funded?
• 10:16
  Speaker 2: Oh my god? Okay.
• 10:17
  Speaker 1: So one of the reasons, to be honest, one of
• 10:19
  Speaker 1: the reasons I sort of transitioned away from academia in
• 10:23
  Speaker 1: the most purest sense, so I'm like academic adjacent now,
• 10:27
  Speaker 1: but was because I spent too many Christmas Eves working
• 10:31
  Speaker 1: on my grants because they were due like sometime in January,
• 10:35
  Speaker 1: and they just they were like so many I would
• 10:40
  Speaker 1: say months of work. In a lot of cases, if
• 10:41
  Speaker 1: it's a new idea, if you're just cleaning up an
• 10:43
  Speaker 1: old grant, then it can be once of work. And
• 10:47
  Speaker 1: then so many great grants are submitted, and when they
• 10:50
  Speaker 1: go to review, you have three peers of yours who
• 10:54
  Speaker 1: are in closely related fields review it, and then there's
• 10:57
  Speaker 1: a panel discussion where they talk about their views of
• 11:00
  Speaker 1: your grant in front of a bunch of other experts
• 11:02
  Speaker 1: who can also weigh in. And then all those reviews
• 11:05
  Speaker 1: go to program directors who pick out of like the
• 11:09
  Speaker 1: twenty best grants, you know, maybe they can only fund
• 11:12
  Speaker 1: fifteen or something. And then they pick some on the
• 11:15
  Speaker 1: East coast, some on the West coast, some in the
• 11:17
  Speaker 1: center of the country, some from major research institutions, some
• 11:21
  Speaker 1: from institutions that focus a lot on undergraduate research, and
• 11:24
  Speaker 1: they have research programs as well, And so so many
• 11:28
  Speaker 1: really great grants don't make it into the pile that
• 11:32
  Speaker 1: get funded.
• 11:33
  Speaker 3: Meaning that people have read them, have said this is excellent,
• 11:36
  Speaker 3: the science is good, it's interesting, it's important. We would
• 11:39
  Speaker 3: benefit as a society if we did this. It's all
• 11:42
  Speaker 3: well thought.
• 11:42
  Speaker 2: Through, but no, that is exactly right, yes, and it
• 11:46
  Speaker 2: is soul crushing. I know so many people who are like,
• 11:49
  Speaker 2: I gave up Christmas Eve on this grant, and this
• 11:51
  Speaker 2: is the third time I've submitted it, and it just
• 11:54
  Speaker 2: never makes it quite high enough, even though everybody thinks
• 11:56
  Speaker 2: it's amazing. And then quickly just to mention for follow up.
• 11:59
  Speaker 1: If you do get grant, every year you have to
• 12:01
  Speaker 1: write a progress report saying what you've done, have you
• 12:04
  Speaker 1: stuck to your timeline, how have you spent the money
• 12:07
  Speaker 1: that you've been given, are you going to hit your timelines?
• 12:10
  Speaker 1: And what have you done that impacts society? And so
• 12:12
  Speaker 1: every year you have to report on that and then
• 12:15
  Speaker 1: at the end you have to write a bigger report.
• 12:17
  Speaker 1: So you need to be justifying what you're doing with
• 12:19
  Speaker 1: the money every step of the way.
• 12:21
  Speaker 3: And before we turn it around and ask me questions,
• 12:24
  Speaker 3: who are the peers here, Who are folks who are
• 12:26
  Speaker 3: reading your grants and commenting on it.
• 12:29
  Speaker 1: There are folks in the same discipline at other universities.
• 12:31
  Speaker 1: So you also have to fill out a detailed excel
• 12:34
  Speaker 1: sheet where you talk about who in the field who
• 12:37
  Speaker 1: does similar work, has been your student or your postdoc
• 12:40
  Speaker 1: or your mentor or a co author.
• 12:42
  Speaker 3: You might have a conflict.
• 12:43
  Speaker 1: Yeah, and anyone who might like you for you and
• 12:47
  Speaker 1: who might be too nice to you when they review
• 12:50
  Speaker 1: the grant. So it has to be only people that
• 12:52
  Speaker 1: you haven't worked with or haven't co author of paper
• 12:54
  Speaker 1: with for something like five to ten years. I guess
• 12:57
  Speaker 1: they assume that the liking of someone wears off about
• 13:00
  Speaker 1: half a decade. But it can be tough because you know,
• 13:04
  Speaker 1: these fields are kind of small, but you know, usually
• 13:05
  Speaker 1: it's other people at research institutions who do similar work,
• 13:09
  Speaker 1: are familiar with the literature and can tell if what
• 13:12
  Speaker 1: you're proposing makes sense or not and is good or not.
• 13:14
  Speaker 3: And so the thing to take away from this is
• 13:16
  Speaker 3: getting a grand funded is hard, right. Not only is
• 13:20
  Speaker 3: it a huge amount of work, but you have to
• 13:22
  Speaker 3: survive an excruciating process where really only the best are selected.
• 13:27
  Speaker 3: It's sort of like watching the Olympics and you're wondering, like, oh,
• 13:30
  Speaker 3: here's somebody from this country and somebody from that country,
• 13:33
  Speaker 3: and you know that each person has already won some
• 13:36
  Speaker 3: sort of like really competitive national competition just to even
• 13:40
  Speaker 3: be there at the Olympics representing their country. And so
• 13:44
  Speaker 3: like every grant that's funded, even if it seems silly
• 13:46
  Speaker 3: and it's about like the sex lives of ducks, you
• 13:49
  Speaker 3: know that it's been like excruciatingly written, reviewed in detail,
• 13:54
  Speaker 3: questioned by experts without conflicts of interest, and found to
• 13:57
  Speaker 3: be excellent, beaten out lots of other grand These are
• 14:00
  Speaker 3: not slush funds shoveled to scientists, so you know, just
• 14:04
  Speaker 3: like do whatever they want with. These are hard one
• 14:07
  Speaker 3: funds to do science that the agency thinks are a
• 14:10
  Speaker 3: good idea.
• 14:11
  Speaker 1: Yeah, it is absolutely excruciating to do these things. It
• 14:15
  Speaker 1: feels great when you get it funded. And I said,
• 14:17
  Speaker 1: I hate writing grants. Sometimes I enjoy the process of
• 14:20
  Speaker 1: like figuring out just the right experimental design for a question.
• 14:23
  Speaker 1: That's like fun for me. But in general, when the
• 14:26
  Speaker 1: grant doesn't get funded, it sucks. But anyway, all right,
• 14:29
  Speaker 1: so what's your experience.
• 14:31
  Speaker 3: Yeah, my experience is very similar. You know, we come
• 14:34
  Speaker 3: up with an idea, we spend a lot of time
• 14:36
  Speaker 3: polishing it, often doing a lot of the work in
• 14:39
  Speaker 3: advance that we need to demonstrate that the work described
• 14:43
  Speaker 3: it's reasonable. Right. If it's too far forward thinking, then
• 14:48
  Speaker 3: you won't get the money because they're like, well, that
• 14:50
  Speaker 3: might work, but can you really prove it. It's too
• 14:52
  Speaker 3: much of a risk. If you've already done too much
• 14:55
  Speaker 3: of the work, then they're like, oh, this is already done,
• 14:57
  Speaker 3: why would we fund it? So it's really sort of
• 14:59
  Speaker 3: on the edge there of like you've done the initial
• 15:01
  Speaker 3: work for free right or on some other grant or whatever.
• 15:04
  Speaker 3: So you prove that the idea is valid and can fly,
• 15:08
  Speaker 3: but not so much that they're like, why would we
• 15:10
  Speaker 3: fund this? You've already done it, which is often a
• 15:12
  Speaker 3: delicate balance. And my feeling with grants is submit it
• 15:16
  Speaker 3: and forget it, because such a tiny fraction ever come
• 15:19
  Speaker 3: back with money that you just got to like let
• 15:21
  Speaker 3: it go, like, hey, I've submitted it, and I never
• 15:24
  Speaker 3: expect to hear back, you know, anything positive, And so
• 15:28
  Speaker 3: I just sort of like give up emotionally on everyone
• 15:31
  Speaker 3: because otherwise it's too hard, you know, it's too hard
• 15:34
  Speaker 3: to deal with, yea. So my process is very similar
• 15:36
  Speaker 3: to yours, with a couple of exceptions. Sometimes I submit
• 15:40
  Speaker 3: to private foundations, like I recently found a private foundation
• 15:44
  Speaker 3: that likes to fund projects that are too blue sky
• 15:48
  Speaker 3: that were rejected by the NSF for being like way
• 15:52
  Speaker 3: too out there, and that submission process was like write
• 15:56
  Speaker 3: a paragraph, send us your rejected NSF grant with the views,
• 16:00
  Speaker 3: and that's it.
• 16:01
  Speaker 4: Whoa.
• 16:02
  Speaker 3: And that one actually just came back and they just
• 16:03
  Speaker 3: gave me some money to build my cosmic ray smartphone
• 16:07
  Speaker 3: telescope down here at Irvine. So yeah, that was actually
• 16:10
  Speaker 3: a really positive experience graduations.
• 16:13
  Speaker 2: Yeah, that's awesome.
• 16:14
  Speaker 3: Yeah exactly, because you know, private foundations can do whatever
• 16:18
  Speaker 3: they like with your money. What we talked about previously
• 16:20
  Speaker 3: was mostly the process of applying to government institutions, the
• 16:24
  Speaker 3: NIH the NSF. Most of my fund incomes from Department
• 16:27
  Speaker 3: of Energy, Office of Science, which funds particle physics and
• 16:30
  Speaker 3: lots of other stuff in the United States. But private
• 16:34
  Speaker 3: fundations can do anything. Like the Bill Gates Foundation, you
• 16:36
  Speaker 3: can't even apply to. They have to like reach out
• 16:38
  Speaker 3: to you.
• 16:39
  Speaker 2: Wow.
• 16:39
  Speaker 3: And you know Mackenzie Scott just like gives money to
• 16:42
  Speaker 3: people that don't even apply. They just get a phone
• 16:44
  Speaker 3: call saying like, by the way, here comes a million
• 16:46
  Speaker 3: dollars or more.
• 16:47
  Speaker 4: Whoa.
• 16:48
  Speaker 3: So yeah, private foundations are weird.
• 16:51
  Speaker 1: Yeah, so I'm sure that they make solid choices about
• 16:55
  Speaker 1: who they're going to give their money. To But I
• 16:56
  Speaker 1: guess one way to sort of check how much you
• 16:58
  Speaker 1: can trust an article as you can look in the
• 17:01
  Speaker 1: acknowledgment section at the end and say, oh, this was
• 17:03
  Speaker 1: funded by the National Science Foundation, And then you know
• 17:06
  Speaker 1: this went through rigorous peer review. And just because something
• 17:08
  Speaker 1: doesn't go through rigorous peer review at the grant stage
• 17:11
  Speaker 1: doesn't mean it's bad science. But at least you know
• 17:14
  Speaker 1: if it went through a government agency, it has been
• 17:16
  Speaker 1: sort of gone over with a fine tooth coust And the.
• 17:19
  Speaker 3: Thing that's frustrating to me is this process is so inefficient.
• 17:23
  Speaker 3: Scientists spend so much of their time preparing grant proposals
• 17:26
  Speaker 3: and having them rejected. And you know, each of these
• 17:30
  Speaker 3: is a great idea, each of them. If we did
• 17:32
  Speaker 3: it would benefit society in terms of sheer knowledge or
• 17:36
  Speaker 3: you know, new technology or something. We have all these
• 17:39
  Speaker 3: smart people constantly pitching great ideas to the government, the
• 17:43
  Speaker 3: government saying, yeah, that's awesome, we'd love to do it,
• 17:46
  Speaker 3: but we can't. Like, why don't we just double triple
• 17:51
  Speaker 3: funding for science. It would be like better for us
• 17:54
  Speaker 3: every dollar we spend, we know comes back two threefold
• 17:57
  Speaker 3: in terms of like economic output. It just seems crazy
• 18:01
  Speaker 3: to me to reject all of these excellent ideas.
• 18:04
  Speaker 2: I mean, I gotta say, not every idea I've reviewed
• 18:07
  Speaker 2: on a panel has been excellent.
• 18:09
  Speaker 3: No, but there are plenty above threshold.
• 18:10
  Speaker 2: Yes, absolutely, absolutely.
• 18:12
  Speaker 1: There's always a grant every round where my heart breaks
• 18:15
  Speaker 1: a little that it didn't get funded because I thought, oh,
• 18:17
  Speaker 1: that is so cool, but there wasn't enough money to
• 18:19
  Speaker 1: cover it. And so yes, there's a lot of good
• 18:22
  Speaker 1: work that's not getting done.
• 18:23
  Speaker 3: And I agree. I'm a grant reviewer often and I
• 18:26
  Speaker 3: see grants I'm like, no, this is not well thought out,
• 18:28
  Speaker 3: or there's a flaw here, or this isn't cutting edge.
• 18:30
  Speaker 3: You know, somebody did this last year, and it's important
• 18:33
  Speaker 3: that this stuff gets reviewed and gets reviewed in a
• 18:35
  Speaker 3: fair way. But there's so many that are above threshold,
• 18:38
  Speaker 3: and only a fraction of those get funded, and it
• 18:40
  Speaker 3: just seems to me to be a waste. But whatever.
• 18:42
  Speaker 3: I'm not a political person. I understand these things. But
• 18:46
  Speaker 3: I want folks out there to realize that every grant
• 18:48
  Speaker 3: you've seen that's been funded has been reviewed and excruciating
• 18:51
  Speaker 3: detailed before dollar one was even sent to the institution.
• 18:55
  Speaker 2: That's right.
• 18:55
  Speaker 1: I don't think we need to get into lots of
• 18:57
  Speaker 1: detail about this, but every once in a while, my
• 18:59
  Speaker 1: husband and I will have chats about what would be
• 19:01
  Speaker 1: a better system for funding grants, Like maybe every four
• 19:04
  Speaker 1: years you get a chance to submit and that way
• 19:07
  Speaker 1: you don't have to write grants the other three years
• 19:09
  Speaker 1: and there's fewer people in the pool and you're more
• 19:12
  Speaker 1: likely to get it. I think that's not the answer,
• 19:14
  Speaker 1: but I wonder if there's some way we could save
• 19:16
  Speaker 1: scientists from spending all of this time on grants that
• 19:20
  Speaker 1: don't get funded, or maybe we should just throw twenty
• 19:22
  Speaker 1: times as much money at the scientific community. There's our solution.
• 19:25
  Speaker 3: Well, you know, it used to be fifty years ago
• 19:28
  Speaker 3: that the philosophy is a little bit different that the
• 19:30
  Speaker 3: government funded people rather than projects. And they were like, oh,
• 19:34
  Speaker 3: scientist X at this university, you're a smart lady. You've
• 19:38
  Speaker 3: done good work. We'll just keep giving you money and
• 19:41
  Speaker 3: it'll just go for a while. As long as you
• 19:42
  Speaker 3: keep doing something, we'll keep giving you money. And I
• 19:45
  Speaker 3: think that we used to be the major model, and
• 19:47
  Speaker 3: it's just not that way anymore. Now. It's projects, and
• 19:49
  Speaker 3: so if you have a lab at a university, it's
• 19:52
  Speaker 3: sort of like running a small business. You know, you
• 19:53
  Speaker 3: have to constantly be pitching grants to get funds. You know,
• 19:57
  Speaker 3: you're like running a store at the mall. Katrina likes
• 19:59
  Speaker 3: to say, you're constantly looking for new ways to contribute.
• 20:04
  Speaker 3: The one holdover that I'm aware of is actually experimental
• 20:07
  Speaker 3: particle physics is still a little bit of the older model.
• 20:10
  Speaker 3: They run competitions for junior faculty, and if you win
• 20:13
  Speaker 3: one of these awards, like I want a Outstanding Junior
• 20:17
  Speaker 3: Investigator Award when I was a very young professor, then
• 20:20
  Speaker 3: you sort of get in the club and then they
• 20:22
  Speaker 3: mostly fund you, and your find can go up if
• 20:25
  Speaker 3: you do great, and go down if you're less productive.
• 20:27
  Speaker 3: But it's much more stable than typically I think, because
• 20:31
  Speaker 3: these particle physics projects last like twenty years. You know,
• 20:34
  Speaker 3: we build a collider, we expect to use it for
• 20:36
  Speaker 3: twenty five years. That's my thinking for why they still
• 20:39
  Speaker 3: do it in the sort of older model. But it
• 20:41
  Speaker 3: means I don't have to write as many grants because
• 20:43
  Speaker 3: I do have one more stable source of funding.
• 20:45
  Speaker 2: Wow. So even to this day, you still get a
• 20:48
  Speaker 2: chunk of money every year.
• 20:49
  Speaker 3: I have to submit a grant every three years to
• 20:51
  Speaker 3: propose new work and to tell them what I did
• 20:53
  Speaker 3: in the last three years, and so far every time
• 20:55
  Speaker 3: my grant has been renewed and continued. So yeah, I've
• 20:59
  Speaker 3: been funded from the Armament of Energy for a couple
• 21:01
  Speaker 3: of decades now, which is very nice. Yeah, and I'm
• 21:04
  Speaker 3: very grateful to the Department of Energy, thank you very much,
• 21:06
  Speaker 3: and to all the taxpayers who support them.
• 21:08
  Speaker 1: Oh you government, shill, I'm just kidding. I am so
• 21:13
  Speaker 1: happy for you. That's awesome. I know there's a couple
• 21:15
  Speaker 1: grants for people in the medical field that work that
• 21:18
  Speaker 1: way also where they fund your lab and just trust
• 21:21
  Speaker 1: that you're doing awesome stuff and you will continue to
• 21:23
  Speaker 1: do awesome stuff with that money.
• 21:25
  Speaker 2: And that sounds pretty sweet.
• 21:26
  Speaker 3: Yeah, it's pretty nice. Yeah. I think you're talking about
• 21:28
  Speaker 3: like the Howard Hughes Awards, for example.
• 21:30
  Speaker 2: That sounds very yeah. Yeah yeah. All right, Well, my
• 21:33
  Speaker 2: jealousy aside.
• 21:34
  Speaker 1: Let's all take a break, and when we get back,
• 21:37
  Speaker 1: we're going to talk about the next phase when peer
• 21:39
  Speaker 1: review is done, and.
• 21:40
  Speaker 2: That is when you're about to start your experiments.
• 22:03
  Speaker 3: All right, we are back and we are talking about
• 22:05
  Speaker 3: peer review. Thanks to a question from a listener, and
• 22:08
  Speaker 3: we dug into the process of peer review for grant
• 22:11
  Speaker 3: proposals and grant writing. And now let's talk about what
• 22:15
  Speaker 3: it's like to review an experiment while it's running, before
• 22:18
  Speaker 3: the paper is even sent to the journal. Kelly, I
• 22:22
  Speaker 3: mostly do research on particles that don't have rights and
• 22:25
  Speaker 3: don't have institutional review boards protecting them. What's it like
• 22:29
  Speaker 3: to do an experiment on living creatures with emotions that
• 22:32
  Speaker 3: can feel pain and have people looking over your shoulder.
• 22:35
  Speaker 2: Oh, it can be pretty stressful.
• 22:37
  Speaker 1: So say you get that National Science Foundation grant, they
• 22:40
  Speaker 1: won't actually release the funds to you until you've shown
• 22:43
  Speaker 1: that you've acquired certain permits and protocols so that your
• 22:47
  Speaker 1: institution is giving you permission to do the research. So
• 22:50
  Speaker 1: my PhD work required collecting some fish out of estuaries
• 22:53
  Speaker 1: in California. So first I had to talk to the
• 22:56
  Speaker 1: state government and fill out permits to get permission to
• 22:59
  Speaker 1: go collect those fish. So I had to convince the
• 23:02
  Speaker 1: California government that I wasn't going to take too many,
• 23:05
  Speaker 1: that there were plenty of these fish out there, that
• 23:07
  Speaker 1: what I was going to do to them was asking
• 23:09
  Speaker 1: a worthwhile scientific.
• 23:11
  Speaker 3: Question, Kelly, what are you going to do to.
• 23:13
  Speaker 1: Them, I'm going to hear well the questions I'm going
• 23:17
  Speaker 1: to ask of them that they will be contributing in
• 23:20
  Speaker 1: a meaningful way to science. And so first you have
• 23:23
  Speaker 1: to get that permission to take the animals out of
• 23:25
  Speaker 1: the wild, and then you need to fill out a
• 23:27
  Speaker 1: protocol through the Institutional Animal Care and Use Committee, which
• 23:31
  Speaker 1: is iya Cook. It includes five people, and they include
• 23:35
  Speaker 1: folks like veterinarians, an outside member, so like somebody who's
• 23:38
  Speaker 1: part of the community who is just sort of going
• 23:41
  Speaker 1: to weigh in on what the general public feels about
• 23:43
  Speaker 1: the work that's being done. Sometimes you'll also have like
• 23:46
  Speaker 1: an ethicist or a lawyer in there, and often you'll
• 23:49
  Speaker 1: have like a faculty member on there also.
• 23:51
  Speaker 3: And what do these people want? Do they want to
• 23:53
  Speaker 3: say yes? Do they want to say no? Are they
• 23:56
  Speaker 3: just totally disinterested? Like what are their motivations? Why is
• 23:59
  Speaker 3: like a rando from the public doing this? Anyway?
• 24:01
  Speaker 1: So, if I'm being completely honest, my field in the
• 24:04
  Speaker 1: past did some uncomfortable things to animals, and you know,
• 24:10
  Speaker 1: they didn't use, for example, anesthetic when they were doing
• 24:14
  Speaker 1: some surgeries, and they should have things like that. And
• 24:16
  Speaker 1: so this committee is trying to make sure that you
• 24:19
  Speaker 1: are treating the animals as nice as possible and using
• 24:22
  Speaker 1: the fewest animals that you need to get answers to
• 24:25
  Speaker 1: your questions. So you need to convince them that you
• 24:28
  Speaker 1: have read up on the most effective anesthetics for whatever
• 24:32
  Speaker 1: animal it is that you're using. You also often have
• 24:35
  Speaker 1: to take a bunch of online training courses where you
• 24:38
  Speaker 1: show I am well trained and the best way to
• 24:41
  Speaker 1: treat these animals and I know how these anesthetics work.
• 24:44
  Speaker 1: You often have to prove to them that you have
• 24:46
  Speaker 1: like teamed up with someone who can make sure you're
• 24:47
  Speaker 1: doing the process correctly, and you need to convince them
• 24:50
  Speaker 1: that you've thought really hard about the exact number of
• 24:53
  Speaker 1: animals that you're going to use for these experiments. You know,
• 24:56
  Speaker 1: their job isn't to stop science, and they're not necessarily
• 24:59
  Speaker 1: trying to figure you're out of the question that's being
• 25:01
  Speaker 1: asked is good or not. They assume that if you
• 25:04
  Speaker 1: have funding from the National Science Foundation, that has already happened.
• 25:07
  Speaker 1: But their goal is to just make sure that nothing
• 25:09
  Speaker 1: unethical happens and that the animals who are contributing their
• 25:12
  Speaker 1: lives to science have the best life possible.
• 25:14
  Speaker 3: Oh that sounds nice. It is, and it's nice to
• 25:16
  Speaker 3: see the process sort of self correcting, like, yeah, okay,
• 25:19
  Speaker 3: we trust scientists. Actually, maybe they need some more eyeballs
• 25:23
  Speaker 3: on them because they have conflicts of interest, and so
• 25:26
  Speaker 3: it's good to have other people who don't have those
• 25:28
  Speaker 3: conflicts looking over your shoulder and making sure you're doing
• 25:31
  Speaker 3: things with the right way.
• 25:32
  Speaker 1: That's nice, it is, And I also appreciate that there's
• 25:34
  Speaker 1: a veterinarian on there. The veterinarian will like check in
• 25:37
  Speaker 1: pretty regularly. I have training in parasitology, but I don't
• 25:41
  Speaker 1: necessarily have training in care of fish or something.
• 25:44
  Speaker 2: And I've done a bunch of reading.
• 25:45
  Speaker 1: But it's nice to have a veterinarian check in every
• 25:47
  Speaker 1: once in a while and offer their expertise to make
• 25:50
  Speaker 1: sure that these animals really are being treated as well
• 25:52
  Speaker 1: as you can in a lab setting. And I have
• 25:54
  Speaker 1: never worked on humans, but if you are doing things
• 25:57
  Speaker 1: with humans, including just like sending out survey to humans
• 26:01
  Speaker 1: that might ask questions that would make people feel uncomfortable,
• 26:04
  Speaker 1: you have to pass those procedures through an institutional review board.
• 26:07
  Speaker 1: So there's a similar procedure for working through humans. So
• 26:11
  Speaker 1: I guess you don't have anything like that for particle
• 26:14
  Speaker 1: physics because we don't care what you do to the particles.
• 26:17
  Speaker 3: No, we don't. But I did one time come home
• 26:19
  Speaker 3: from work to find Katrina collecting samples for an experiment
• 26:22
  Speaker 3: from our children. Oh, And I was like, hmm, shouldn't
• 26:26
  Speaker 3: you be asking the other parent pofore experimenting on your children,
• 26:30
  Speaker 3: like you have a conflict of interest here? And she
• 26:33
  Speaker 3: was like, it's just a saliva sample. I'm like, the
• 26:35
  Speaker 3: children can't consent to. So we have a lot of
• 26:38
  Speaker 3: fun joking about that.
• 26:39
  Speaker 2: Did you give permission? That's good? That's good.
• 26:43
  Speaker 1: So, Daniel, there's this new thing that's getting sort of
• 26:45
  Speaker 1: big in science that my friends have been talking about.
• 26:48
  Speaker 1: I haven't had a chance to do this yet, but
• 26:50
  Speaker 1: pre registering a scientific study, have you done this yet?
• 26:53
  Speaker 3: We don't actually do this in particle physics, because this
• 26:55
  Speaker 3: is another way that particle physics is weird, is that
• 26:58
  Speaker 3: we publish our studies when we get a negative result.
• 27:02
  Speaker 3: Like lots of fields of science, you might say I
• 27:04
  Speaker 3: have an idea for how we might learn something, and
• 27:06
  Speaker 3: you know, you do this study and then it didn't
• 27:08
  Speaker 3: work or you learned that there's nothing new there like okay,
• 27:13
  Speaker 3: bears eat salmon. We already knew that you know like
• 27:15
  Speaker 3: or whatever. And often those studies don't get published if
• 27:18
  Speaker 3: the result isn't statistically significant or you didn't learn something new,
• 27:22
  Speaker 3: and there's a statistical issue there, which is that this
• 27:25
  Speaker 3: can lead to a bias in our data and our understanding.
• 27:28
  Speaker 3: The effect is called PA hacking, and it means that
• 27:31
  Speaker 3: sometimes things that are not real can appear to be
• 27:34
  Speaker 3: significant just due to random fluctuation. Say, for example, I'm
• 27:38
  Speaker 3: testing a bunch of coins to see if they're fare.
• 27:41
  Speaker 3: I flip each one one hundred times. Then like most
• 27:44
  Speaker 3: of the time, I'm going to get fifty heads, but occasionally,
• 27:47
  Speaker 3: just due to random fluctuations, I'm going to get one
• 27:49
  Speaker 3: that gets like seventy heads right. And the more I
• 27:52
  Speaker 3: do this experiment, the more likely I am to have
• 27:55
  Speaker 3: one that's a fluctuation. And so if I only publish
• 27:58
  Speaker 3: the ones that seem to be sick magnificant, that that
• 28:00
  Speaker 3: crossed some statistical threshold to be weird, it's going to
• 28:03
  Speaker 3: look like these coins are not fair, when in reality
• 28:06
  Speaker 3: they are. That's what PA hacking is. P refers to
• 28:09
  Speaker 3: the probability for the alternative hypothesis that the coin is
• 28:12
  Speaker 3: fair to fluctuate randomly to look like it's not fair.
• 28:17
  Speaker 3: And so the way to counteract this is to publish
• 28:20
  Speaker 3: negative results, to say, look, I did all these experiments
• 28:23
  Speaker 3: and the coin came out fair. We already knew that, yawn.
• 28:26
  Speaker 3: But it's important that we include this context so that
• 28:29
  Speaker 3: if one in a thousand studies says, look I saw
• 28:31
  Speaker 3: one in a thousand times effect, you know what it means.
• 28:35
  Speaker 3: And often we do this by pre registering studies by
• 28:38
  Speaker 3: saying I'm going to go out and do this study
• 28:40
  Speaker 3: and I don't know the results yet, but I'm going
• 28:42
  Speaker 3: to publish it either way, right, And so that's a
• 28:45
  Speaker 3: way to counteract p hacking.
• 28:47
  Speaker 1: So in my field, what you've just described, I think
• 28:50
  Speaker 1: we would call the file draw effect. And so the
• 28:53
  Speaker 1: idea here is that, yeah, when you get a null
• 28:55
  Speaker 1: result or a result that's less interesting, you might try
• 28:58
  Speaker 1: to publish it somewhere, but you're less likely to put
• 29:00
  Speaker 1: in the effort to try to publish it in a
• 29:02
  Speaker 1: lower tier journal that you're going to get less credit
• 29:04
  Speaker 1: from your institution for having published in. And that could
• 29:07
  Speaker 1: make it look like an effect is there, because when
• 29:09
  Speaker 1: you randomly get a positive effect, it gets published and
• 29:11
  Speaker 1: otherwise it doesn't. For us, pe hacking is when you
• 29:14
  Speaker 1: didn't get the result that you wanted, but you're looking
• 29:17
  Speaker 1: at your data and you're like, oh, you know, I
• 29:19
  Speaker 1: wasn't actually asking a question about this other thing, but
• 29:23
  Speaker 1: if I look at my data, there's actually a statistically
• 29:25
  Speaker 1: significant result there, and so I'm going to publish on that,
• 29:29
  Speaker 1: and you know, maybe I'll mention that this wasn't the
• 29:31
  Speaker 1: initial thing, but I found this, you know, significant relationship
• 29:35
  Speaker 1: that's positive, So I'll publish on that. And the deal
• 29:37
  Speaker 1: there is, you know, like you said, randomly, you would
• 29:39
  Speaker 1: expect to get results that look like something is really
• 29:42
  Speaker 1: going on. And if you just are searching through your
• 29:45
  Speaker 1: data set, you're more likely to find something that's significant.
• 29:48
  Speaker 1: And that wasn't the question you're originally asking. And so
• 29:51
  Speaker 1: the reason we do pre registering a scientific study is
• 29:54
  Speaker 1: you'll say, ahead of time, I am specifically looking at
• 29:57
  Speaker 1: the relationship between I don't know, parasite and how often
• 30:01
  Speaker 1: a fish darts. And so if I go on and
• 30:04
  Speaker 1: I publish a paper about parasites and how active a
• 30:07
  Speaker 1: fish is, you could say, hey, did you just find
• 30:11
  Speaker 1: that result? And that you changed your paper to be
• 30:13
  Speaker 1: about that because it looked interesting. And so anyway, this
• 30:15
  Speaker 1: is how we make sure that you're not looking for
• 30:18
  Speaker 1: significant results and just publishing whatever you find.
• 30:21
  Speaker 3: And in particle physics we're lucky enough that we always
• 30:24
  Speaker 3: publish because a negative result is still interesting. Like if
• 30:28
  Speaker 3: you look for a particle because you think it's there
• 30:30
  Speaker 3: and the universe would make more sense for it to exist,
• 30:33
  Speaker 3: like the Higgs boson, then if you see it, great,
• 30:36
  Speaker 3: you publish it. If you don't, that's still interesting. You
• 30:39
  Speaker 3: still want to know, oh, there isn't a Higgs Boson,
• 30:42
  Speaker 3: because if we smash particles together often enough and we
• 30:45
  Speaker 3: don't see it, we can say something about the likelihood
• 30:47
  Speaker 3: of it not existing, because if it existed, we would
• 30:50
  Speaker 3: have seen it. Now. We still are susceptible to p
• 30:53
  Speaker 3: hacking because we will sometimes see fluctuations like data will
• 30:56
  Speaker 3: just like look like a new particle sometimes, and to
• 31:00
  Speaker 3: that we have a very stringent criteria. It's called this
• 31:03
  Speaker 3: five sigma threshold, which means if we only claim the
• 31:05
  Speaker 3: particle is discovered if it's very very very unlikely to
• 31:10
  Speaker 3: come from a fluctuation, but we still in principle could
• 31:12
  Speaker 3: get fooled. And that's why we always have duplicate experiments,
• 31:15
  Speaker 3: like at the largechange in cloud, we have two of
• 31:17
  Speaker 3: these big detectors and they're totally independent, and that's why
• 31:21
  Speaker 3: you expect to see the same physics in both. And
• 31:23
  Speaker 3: if you don't, you know something this squarely.
• 31:25
  Speaker 1: Yeah, And in my field, we're getting better where if
• 31:28
  Speaker 1: you get no result or negative results, as long as
• 31:31
  Speaker 1: you can show that you did a sound experiment and
• 31:34
  Speaker 1: the answer is no, then you can still publish it somewhere.
• 31:38
  Speaker 1: And so Public Library of Science or PLUS is a
• 31:40
  Speaker 1: journal that encourages folks to submit they're null results as
• 31:44
  Speaker 1: long as the science was good and you can defend
• 31:46
  Speaker 1: what you did to try to make sure that you're
• 31:47
  Speaker 1: not getting this file drawer effects stuff.
• 31:49
  Speaker 2: So you get the no answers just as often as
• 31:51
  Speaker 2: the yes answers, and.
• 31:52
  Speaker 3: That's helpful, right. It saves people time because if Kelly
• 31:55
  Speaker 3: had a great idea and it turned out the answer
• 31:57
  Speaker 3: is no, and then she just throws it in the
• 31:58
  Speaker 3: file drawer, then Sam across the country is going to
• 32:01
  Speaker 3: try the same thing someday and waste their time. So
• 32:04
  Speaker 3: it's it's good to get this stuff out there.
• 32:06
  Speaker 1: Right, And you know, in my field, wasting time also
• 32:09
  Speaker 1: often means that some animals may have died in the process.
• 32:12
  Speaker 1: And so to me, I feel like there's this additional
• 32:15
  Speaker 1: ethical requirement that you get anything that you learned from
• 32:18
  Speaker 1: these animals out so that nobody has to go and
• 32:20
  Speaker 1: repeat it and it you know, more animals might have
• 32:23
  Speaker 1: to pass away for science. So anyway, good to get
• 32:26
  Speaker 1: it out there.
• 32:26
  Speaker 3: All right, So then let's talk more about what happens
• 32:29
  Speaker 3: when you think you have an interesting result and you've
• 32:31
  Speaker 3: written it up and you want to publish it.
• 32:33
  Speaker 1: And we're going to leave everybody on a ledge for
• 32:36
  Speaker 1: just a second, and when we get back from the break,
• 32:37
  Speaker 1: we'll talk more about the fascinating world of peer review.
• 33:00
  Speaker 1: All right, Daniel, you have a significant result. You've got
• 33:03
  Speaker 1: a result that looks good, or at least even if
• 33:04
  Speaker 1: it's no, you're sure it was no because the experiment
• 33:07
  Speaker 1: was done.
• 33:07
  Speaker 2: Well, what do you do next?
• 33:09
  Speaker 3: So we write it up, and when all of our
• 33:12
  Speaker 3: co authors are happy with it, the first thing we
• 33:15
  Speaker 3: do is we post it online. So particle physics is
• 33:18
  Speaker 3: a very very online science, and we invented this thing
• 33:22
  Speaker 3: called the archive where scientists post their papers and it's
• 33:26
  Speaker 3: called a preprint because we posted before we submitted to
• 33:29
  Speaker 3: the journal. So for example, I finished a paper this
• 33:32
  Speaker 3: week and yesterday it appeared on the archive, and nobody's
• 33:36
  Speaker 3: reviewed it, nobody's confirmed it. It's just out there for
• 33:39
  Speaker 3: other scientists to look at. And in a week or so,
• 33:41
  Speaker 3: if we don't get like screaming fits from somebody saying
• 33:44
  Speaker 3: you stole my idea or this is deeply wrong, then
• 33:47
  Speaker 3: we'll submit it to a journal. So the process for
• 33:50
  Speaker 3: particle physics is first post it online, then submitted to
• 33:53
  Speaker 3: a journal where it's going to get reviewed, and that's
• 33:55
  Speaker 3: going to take months and months, but it's so slow.
• 33:57
  Speaker 3: The particle physicis just read the pre print server and
• 34:00
  Speaker 3: read those papers, and nobody really cares when it comes
• 34:04
  Speaker 3: out in the journal later because we already have seen
• 34:07
  Speaker 3: the paper months earlier.
• 34:08
  Speaker 1: But what if the peer reviewed version catches an error?
• 34:11
  Speaker 1: Do you go back and update the archive? Versions out?
• 34:14
  Speaker 3: Okay, you can update it. And usually papers in the
• 34:16
  Speaker 3: archive are like version three, version four, and there's comments
• 34:18
  Speaker 3: in between. You can compare them. You can see the
• 34:20
  Speaker 3: mistakes getting fixed, absolutely, And some papers appear in the
• 34:24
  Speaker 3: archive and then are never published and that's fine. I'd
• 34:27
  Speaker 3: actually had one paper like ten years ago put on
• 34:29
  Speaker 3: the archive. It took like more than a year to review,
• 34:33
  Speaker 3: and then they rejected it because the paper already had
• 34:36
  Speaker 3: like thirty citations, and so they were like, why publish this,
• 34:39
  Speaker 3: It's already out there, people are reading it and using it.
• 34:41
  Speaker 2: Huh, that's such a stupid reason.
• 34:44
  Speaker 3: Yeah, it was really dumb. But then we just gave up,
• 34:46
  Speaker 3: and now it's like one of my more cited papers
• 34:48
  Speaker 3: and it was never published. The system is silly. It
• 34:52
  Speaker 3: is silly, but I know that in other field is different.
• 34:55
  Speaker 3: Like my friends in astronomy, they never put papers on
• 34:57
  Speaker 3: the archive until after they've been reviewed, and they think
• 35:01
  Speaker 3: that if they send it to a journal, it's already
• 35:03
  Speaker 3: out in the archive. Those journal referees are going to
• 35:05
  Speaker 3: find that disrespectful like that they didn't wait for them
• 35:09
  Speaker 3: to chime in. So it's definitely a field by field
• 35:11
  Speaker 3: culture kind of thing. How does it work for you guys?
• 35:14
  Speaker 1: Yeah, so y'all were definitely the trailblazers. We subsequently came
• 35:17
  Speaker 1: up with bio Archive, and I remember when I was
• 35:22
  Speaker 1: initially wanting to submit to bio archive, I was hesitant
• 35:25
  Speaker 1: to do it because some of our journals did say
• 35:28
  Speaker 1: your paper should not be available anywhere else, because we
• 35:31
  Speaker 1: want to be the only place where you can find it.
• 35:33
  Speaker 1: And that's because journals make money off of people who
• 35:36
  Speaker 1: download their articles, and so they didn't want someone else
• 35:39
  Speaker 1: to be able to get another version for free somewhere else.
• 35:42
  Speaker 1: And on bio archive or archive, all of the articles
• 35:45
  Speaker 1: are free. But it seems like it's now become acceptable
• 35:48
  Speaker 1: that you can put papers in these pre print servers
• 35:51
  Speaker 1: like bio Archive, and so now a lot of people
• 35:53
  Speaker 1: will do that, and they do it for a variety
• 35:55
  Speaker 1: of reasons. Some of it is they want to get
• 35:57
  Speaker 1: feedback early, they want to get the results out out
• 36:00
  Speaker 1: there early. But sometimes it's also like if you've got
• 36:03
  Speaker 1: a PhD student who wants to go looking for a job.
• 36:07
  Speaker 1: On their resume, if they finished a study, they'll write
• 36:10
  Speaker 1: in preparation next to the manuscript. But in preparation could
• 36:13
  Speaker 1: mean I've got the data and I still need to
• 36:15
  Speaker 1: do all the stats and I haven't even started writing.
• 36:18
  Speaker 1: Or it could mean I'm about to submit it to
• 36:20
  Speaker 1: a journal, and if you've put it on bio archive,
• 36:22
  Speaker 1: you're saying, look, it's done. I just need to start
• 36:25
  Speaker 1: the peer review process, or even it's in peer review,
• 36:27
  Speaker 1: because that process can take a long time. So anyway,
• 36:29
  Speaker 1: it's becoming a lot more common to put your papers
• 36:32
  Speaker 1: on bio archive, just to make them easier to access
• 36:34
  Speaker 1: for other people, and just to show that you really
• 36:37
  Speaker 1: have actually finished that study finally, after a decade or
• 36:41
  Speaker 1: however long it took, the.
• 36:43
  Speaker 3: Clock has still taken on your thesis. There, Kelly, we'll see.
• 36:46
  Speaker 2: I'll get into preparation.
• 36:48
  Speaker 3: In preparation, so then you send it to a journal,
• 36:51
  Speaker 3: and you pick a journal, like if you think it's
• 36:52
  Speaker 3: a really exciting paper, you send it to Nature or Science.
• 36:55
  Speaker 3: If you think it's less exciting, you send it to
• 36:57
  Speaker 3: a journal with a smaller impact factor. So in particle
• 37:00
  Speaker 3: physics we often publish in like Physical Review or in
• 37:04
  Speaker 3: Journal of highind Physics or these kinds of journals, and
• 37:07
  Speaker 3: it goes to an editor, and an editor finds people
• 37:10
  Speaker 3: to review it. These are the peers, and they reach
• 37:12
  Speaker 3: out to folks they know who are experts in the field,
• 37:15
  Speaker 3: who aren't your supervisor or your brother or this kind
• 37:18
  Speaker 3: of thing, and ask them to review these things. And
• 37:21
  Speaker 3: Katrina is a senior editor on a journal, so I
• 37:23
  Speaker 3: see this process all the time, and she has to
• 37:24
  Speaker 3: read the paper, think about who might know about it,
• 37:27
  Speaker 3: and then find people to review it. And usually you
• 37:30
  Speaker 3: have one to three people read this paper and give
• 37:34
  Speaker 3: comments on it. But finding reviewers can be tough.
• 37:37
  Speaker 1: So for our grants that we talked about earlier, you
• 37:39
  Speaker 1: have to make sure that the person reviewing the grant
• 37:41
  Speaker 1: doesn't have a conflict of interest. They're not your brother
• 37:44
  Speaker 1: or your supervisor or something like that. And in my field,
• 37:47
  Speaker 1: the same holds true for when you're reviewing manuscripts. Is
• 37:50
  Speaker 1: that true for your field as well?
• 37:51
  Speaker 3: Yeah? Absolutely, Yeah. You have to say, oh, I can't
• 37:53
  Speaker 3: read this one because that was my student or something
• 37:57
  Speaker 3: like that.
• 37:58
  Speaker 2: Yeah. How long does review usually take for your.
• 38:00
  Speaker 3: I'd say it's you know, like four to eight weeks
• 38:03
  Speaker 3: or something like that.
• 38:04
  Speaker 1: Wow, how about you, guys, I'd be happy four to
• 38:07
  Speaker 1: eight weeks. That sounds good, but it can be you know,
• 38:09
  Speaker 1: sometimes six months, and at six months you start writing
• 38:11
  Speaker 1: the editor being.
• 38:12
  Speaker 2: Like, come on man, this is crazy. But yeah, it
• 38:16
  Speaker 2: can be hard to find reviewers.
• 38:18
  Speaker 3: Because reviewers are just other scientists busy writing their own
• 38:21
  Speaker 3: grants and writing their own papers and doing all their
• 38:23
  Speaker 3: work and picking up their kids from daycare and stuff
• 38:25
  Speaker 3: like this and trying to get through that mountain of laundry.
• 38:28
  Speaker 3: You know. I think that people forget that, like science
• 38:30
  Speaker 3: is just people, right, and people are busy. Yeah, And
• 38:33
  Speaker 3: if you're out there and you've written some treatise on
• 38:35
  Speaker 3: the fundamental physics of the universe and nobody has read it,
• 38:38
  Speaker 3: it's not because we're gatekeeping. Is just because we're busy,
• 38:41
  Speaker 3: you know. And there's lots of those. And I often
• 38:44
  Speaker 3: say no to editors who ask me to review stuff
• 38:46
  Speaker 3: because I'm just too busy to do it in a
• 38:47
  Speaker 3: timely manner.
• 38:48
  Speaker 2: Yeah, no, me as well.
• 38:49
  Speaker 1: And I think it's worth noting that while we're doing
• 38:51
  Speaker 1: this review work, we're doing it for free. Yeah, Like
• 38:54
  Speaker 1: when you're on a grant review panel, you can lose
• 38:58
  Speaker 1: weeks of time to reviewing these things and you don't
• 39:00
  Speaker 1: get reimbursed for that time. And the same thing goes
• 39:03
  Speaker 1: So when I review a manuscript, I read it for
• 39:05
  Speaker 1: one day, takes me a couple hours to read carefully,
• 39:09
  Speaker 1: and then I sit on it for a few days,
• 39:10
  Speaker 1: and that's the main thing that is occupying my brain
• 39:12
  Speaker 1: in the background for those days. And that could be
• 39:14
  Speaker 1: like I could have been thinking about the introduction to
• 39:17
  Speaker 1: my next book or something, but instead I'm thinking about
• 39:19
  Speaker 1: the methods to.
• 39:19
  Speaker 2: Make sure it made sense.
• 39:21
  Speaker 1: And then I spend another three to four hours reading
• 39:23
  Speaker 1: it again and writing in detail my comments. And if
• 39:26
  Speaker 1: I think there's some literature they missed, I go and
• 39:28
  Speaker 1: I find it and like it's a multi day process.
• 39:31
  Speaker 1: When I commit to reviewing a paper, and reviewers don't
• 39:34
  Speaker 1: get paid for that, nor do the editors.
• 39:36
  Speaker 2: How long does it take you?
• 39:37
  Speaker 3: Well, I wish that all of my papers were reviewed
• 39:40
  Speaker 3: by somebody as thorough as you. But my process is similar,
• 39:44
  Speaker 3: like an initial read, some thoughts, and then let it sit.
• 39:48
  Speaker 3: Sometimes I'll discuss it with people in my group. We'll
• 39:50
  Speaker 3: read it together, and then I read it again in detail.
• 39:53
  Speaker 3: And then, especially if the review is negative, I wait
• 39:56
  Speaker 3: and I sit on it, and I come back later
• 39:58
  Speaker 3: I'm like, was this fair? And also, most importantly, was
• 40:01
  Speaker 3: I nice enough on my comments? Constructive and helpful and
• 40:06
  Speaker 3: not like just negative or harsh in any way? I'd
• 40:09
  Speaker 3: like try to take a sandpaper to anything that's negative
• 40:12
  Speaker 3: and smooth it over, because I've been on the other
• 40:15
  Speaker 3: end of harsh reviews and it's it doesn't help anybody
• 40:19
  Speaker 3: to get zingers in there and like especially because a
• 40:21
  Speaker 3: lot of my papers are written by young people and
• 40:23
  Speaker 3: it's often their first paper, and then I have to
• 40:24
  Speaker 3: show them this review that's like, this guy's a jerk.
• 40:28
  Speaker 3: This isn't necessary. I have to explain to them that
• 40:30
  Speaker 3: the anonymity here is protecting them. But you know, most
• 40:33
  Speaker 3: reviewers are fair and are thoughtful and are courteous, and
• 40:37
  Speaker 3: so often we get like helpful feedback, like what about this,
• 40:40
  Speaker 3: or have you thought about this question? Or what would
• 40:42
  Speaker 3: you say to this concern? Do you find your feedback
• 40:45
  Speaker 3: to be mostly useful?
• 40:46
  Speaker 1: Yeah, But most of the time the feedback I get
• 40:48
  Speaker 1: improves the paper. I did have one reviewer when it
• 40:51
  Speaker 1: was one of my first papers who told me they
• 40:52
  Speaker 1: were very disappointed in me.
• 40:54
  Speaker 3: It's like, you're a jerk, disappointed, thanks dead.
• 41:01
  Speaker 2: And that was because my supplemental materials were too long.
• 41:05
  Speaker 2: I was like, God, give me a break.
• 41:06
  Speaker 1: But anyway, so in my field, we're moving towards double
• 41:09
  Speaker 1: blind review, so the reviewers are not supposed to know
• 41:12
  Speaker 1: the names of the authors, and the authors are not
• 41:15
  Speaker 1: supposed to know who reviewed their paper. And that's a
• 41:18
  Speaker 1: really great idea. So that you feel like you can
• 41:20
  Speaker 1: honestly tell someone if their paper was good or not.
• 41:22
  Speaker 1: You don't have to worry about someone getting mad at
• 41:24
  Speaker 1: you in particular, though in practice there's usually like four
• 41:28
  Speaker 1: people who could review your paper, so you know, like
• 41:31
  Speaker 1: you know, it was Joe or Francceine and Franccene tends
• 41:35
  Speaker 1: to be meaner and so.
• 41:36
  Speaker 2: But anyway, so yeah, what about mirror field? Is it
• 41:39
  Speaker 2: blind or double blind?
• 41:40
  Speaker 3: It's only blind to one direction. The review is anonymous,
• 41:43
  Speaker 3: but you see who the authors are. I know, in
• 41:46
  Speaker 3: some other fields, like in top computer science conferences, for example,
• 41:49
  Speaker 3: it's double blind, and that's helpful. But also it's not
• 41:52
  Speaker 3: that hard to figure it out. If you wanted to
• 41:54
  Speaker 3: know who these people are, you could figure it out.
• 41:57
  Speaker 3: But you might also be wondering, like, what is the
• 41:58
  Speaker 3: job of the reviewer exactlytly does reviewer have to make
• 42:01
  Speaker 3: sure the experiment was right? Like if I'm checking Kelly's work,
• 42:05
  Speaker 3: I'm the reviewer, do you have to go out and
• 42:06
  Speaker 3: do the experiment and make sure she was right? And
• 42:08
  Speaker 3: that's the things that peer review is not replication. The
• 42:11
  Speaker 3: job of the reviewer is to ask do the claims
• 42:15
  Speaker 3: of the paper are they supported by the evidence provided,
• 42:18
  Speaker 3: is their logic? There is there mathematical tissue between the
• 42:22
  Speaker 3: work that's been done and the conclusions that are being claimed.
• 42:25
  Speaker 3: And also is it well written in the citations? And finally,
• 42:28
  Speaker 3: is it interesting? Is it important? Does it play a
• 42:31
  Speaker 3: role in the scientific conversation? Which is a little bit
• 42:35
  Speaker 3: subjective of course, but science is by the people and
• 42:37
  Speaker 3: for the people. But if you're working on something nobody's
• 42:41
  Speaker 3: interested in, then nobody's going to read your paper. Then
• 42:43
  Speaker 3: a journal might say this is solid work, but nobody cares.
• 42:46
  Speaker 3: It's a boring question, or you didn't learn anything interesting.
• 42:50
  Speaker 3: The reviewer's job is not to say, hey, I did
• 42:53
  Speaker 3: this experiment also, and I know that this is a
• 42:55
  Speaker 3: feature of the universe. That's not the task of the
• 42:57
  Speaker 3: Reviewer's not their job to do that for you. It's
• 43:00
  Speaker 3: also not the task of the reviewer to say, like, hey,
• 43:03
  Speaker 3: this is a cool idea. I would have done this differently,
• 43:05
  Speaker 3: and so you now need to go back and do
• 43:06
  Speaker 3: all these additional studies that I think are important. And
• 43:10
  Speaker 3: you see this a lot in peer review that reviews
• 43:12
  Speaker 3: come back and say this is cool, but also do
• 43:14
  Speaker 3: xyz and like that's frustrating to me, because that's not
• 43:18
  Speaker 3: the job, but the reviewer.
• 43:19
  Speaker 2: Yes, yep, Nope. That's super frustrating.
• 43:22
  Speaker 1: And I think it's worth noting that sometimes bad papers
• 43:26
  Speaker 1: get through this procedure, and I one of the ways
• 43:29
  Speaker 1: that bad papers get through is that reviewers aren't required
• 43:33
  Speaker 1: or and this makes sense, but they're not responsible for
• 43:36
  Speaker 1: going through, for example, the codes for the statistical models
• 43:39
  Speaker 1: that you ran. And so if somebody you know, check
• 43:41
  Speaker 1: their codes a bunch of times, but they forgot a
• 43:43
  Speaker 1: minus signed somewhere, the result could be wrong and maybe
• 43:47
  Speaker 1: nobody knows. And I have known a couple of people
• 43:50
  Speaker 1: who afterwards have gone to use those models again for
• 43:53
  Speaker 1: some other question and caught their mistake. And then a
• 43:56
  Speaker 1: good scientist will contact that journal and say, I have
• 44:00
  Speaker 1: I have to retract my paper, or I have to
• 44:02
  Speaker 1: submit errata and let everybody know I forgot the minus
• 44:05
  Speaker 1: signed these it's a different result, and that's really painful.
• 44:08
  Speaker 1: But at least they're being honest, and that's great, and
• 44:11
  Speaker 1: I think that's really important. Or some people are dishonest
• 44:15
  Speaker 1: and there's no way for the reviewer to know that
• 44:17
  Speaker 1: my field had somebody where they were one of the
• 44:20
  Speaker 1: biggest names in the field, and there's this new requirement
• 44:25
  Speaker 1: in our field.
• 44:25
  Speaker 2: What's a couple decades old? Now I'm old.
• 44:27
  Speaker 1: But where you have to put the data that you use,
• 44:31
  Speaker 1: that you collected as part of the experiment online, somewhere
• 44:34
  Speaker 1: in a public place where people can download your data.
• 44:36
  Speaker 1: And again, you know, in a field where animals are
• 44:39
  Speaker 1: being used to collect data. It's great to know that
• 44:42
  Speaker 1: those data could be used by other people to ask questions.
• 44:44
  Speaker 1: You can get more information out of them. But it
• 44:46
  Speaker 1: also means that if someone gets suspect results, you can
• 44:50
  Speaker 1: pull their data and rerun the models. And when somebody
• 44:53
  Speaker 1: was looking at the data, it was clear that the
• 44:56
  Speaker 1: numbers just didn't make sense, like one column was always
• 44:59
  Speaker 1: the prior column times three and it anyway, after more scrutiny,
• 45:04
  Speaker 1: it became clear that this person was making up their data.
• 45:07
  Speaker 1: But you know, we have this new check where your
• 45:09
  Speaker 1: data have to be available to everyone else, and that
• 45:11
  Speaker 1: has helped us sort of tease out people who are
• 45:13
  Speaker 1: being dishonest. So the system is evolving and getting better
• 45:16
  Speaker 1: over time. But still sometimes you get stuff through.
• 45:19
  Speaker 3: You certainly do, and there are folks out there who
• 45:21
  Speaker 3: are like combing through papers to find this stuff and
• 45:25
  Speaker 3: there's a scientist, for example, her name is Elizabeth Bick,
• 45:28
  Speaker 3: and this is her passion. She combs through old papers
• 45:31
  Speaker 3: and finds evidence of like photoshopping in biology. You know,
• 45:36
  Speaker 3: you take a picture of your gel and it's supposed
• 45:38
  Speaker 3: to have this blob and she finds, oh, this blob
• 45:40
  Speaker 3: is the same as that blob, or it's been reverted
• 45:42
  Speaker 3: or whatever. And so there are definitely lots of ways
• 45:45
  Speaker 3: that you could find this stuff now that we couldn't
• 45:46
  Speaker 3: have done beforehand. But peer review is not a guarantee
• 45:51
  Speaker 3: that this hasn't been done, like whereviewers should look for
• 45:53
  Speaker 3: this stuff and call it out if you see it.
• 45:55
  Speaker 3: But stuff definitely can get through. It's not a perfect system,
• 45:59
  Speaker 3: you know, it's one of these. It's the terrible system.
• 46:01
  Speaker 3: But it's also the best that we have so far.
• 46:03
  Speaker 2: Just in aside here, I am a massive wimp.
• 46:05
  Speaker 1: If I like purposefully fabricated data in this era where
• 46:10
  Speaker 1: everything is like public and people can peek behind the curtain.
• 46:13
  Speaker 2: I would never sleep again. I'd be like, someone's gonna
• 46:15
  Speaker 2: find me out, and it would be like I would
• 46:18
  Speaker 2: be miserable the rest of my life.
• 46:20
  Speaker 1: I'd rather have a bunch of low impact papers than
• 46:24
  Speaker 1: worry that I was gonna get called out for lying,
• 46:26
  Speaker 1: but anyway, I am a whim.
• 46:28
  Speaker 3: Yeah, And so the highest standard really is not just
• 46:31
  Speaker 3: that something has been pure reviewed, but that something has
• 46:33
  Speaker 3: been independently replicated. Like you might have a scientist who's
• 46:37
  Speaker 3: doing totally solid work and see some effect in their lab,
• 46:41
  Speaker 3: but doesn't realize that it's an artifact due to some
• 46:43
  Speaker 3: condition the humidity or the local gravity or the trains
• 46:48
  Speaker 3: or something are causing some influence on their experiments, And
• 46:51
  Speaker 3: so you want people on the other side of the
• 46:52
  Speaker 3: world who built it differently, who made different assumptions, who
• 46:55
  Speaker 3: sensitive different stuff to reproduce it. You might remember this
• 46:59
  Speaker 3: excitement of about high temperature superconductors. A couple of years
• 47:02
  Speaker 3: ago LK ninety nine. Korean scientists claimed to have created
• 47:06
  Speaker 3: this room temperature, cheap superconductor which would revolutionize the industry,
• 47:11
  Speaker 3: and so very quickly people were out there trying to
• 47:14
  Speaker 3: replicate it, and people were excited. But until another independent
• 47:17
  Speaker 3: group built it and showed that it was real, nobody
• 47:20
  Speaker 3: really accepted it and thought we were in a new era.
• 47:23
  Speaker 3: And personally, for example, when we were discovering the Higgs boson,
• 47:26
  Speaker 3: I saw the data accumulating around a mass of one
• 47:29
  Speaker 3: hundred and twenty five. We were looking at it constantly.
• 47:32
  Speaker 3: It was building up and up and up. But until
• 47:34
  Speaker 3: I heard that my colleagues around the ring also saw
• 47:37
  Speaker 3: an effect at the same place, I didn't personally think, Okay, yeah,
• 47:41
  Speaker 3: we found this thing. And so this sort of independent
• 47:44
  Speaker 3: replication is really sort of the highest standard. What do
• 47:46
  Speaker 3: you think, Kelly, I.
• 47:47
  Speaker 1: Think so, And to me, this is one of the
• 47:51
  Speaker 1: current weaknesses in science, at least in my field. So
• 47:55
  Speaker 1: replicating data is absolutely critical. But if you write to
• 47:58
  Speaker 1: the National Science Foundation, for example, and you're like, oh,
• 48:00
  Speaker 1: I just want to do the same experiment that Maria did,
• 48:03
  Speaker 1: but I'm going to do it on another continent, it's
• 48:05
  Speaker 1: going to be hard to get money for that because
• 48:07
  Speaker 1: it's not a new idea, and it's also probably not
• 48:10
  Speaker 1: going to get published in a top tier journal. And
• 48:14
  Speaker 1: so if you are training PhD students who are going
• 48:16
  Speaker 1: to want to get jobs and they're just replic quote unquote,
• 48:19
  Speaker 1: just replicating somebody else's work, that's not going to help
• 48:22
  Speaker 1: them get a job as much as following up on
• 48:24
  Speaker 1: their own exciting new idea. And so we've got this
• 48:27
  Speaker 1: incentive structure. It's actually not super great for encouraging people
• 48:31
  Speaker 1: to replicate each other's studies, but I do think it's
• 48:34
  Speaker 1: absolutely critical, and I would love to see us sort
• 48:37
  Speaker 1: of work on that incentive structure to make replication just
• 48:40
  Speaker 1: as important as the initial thing that got done.
• 48:43
  Speaker 3: That's interesting. In particle physics, there's maybe a slightly healthier environment.
• 48:47
  Speaker 3: We have a few signals of new physics that we've
• 48:50
  Speaker 3: seen in experiments that we're all curious about, but nobody
• 48:53
  Speaker 3: really accepts because it doesn't quite make sense. And then
• 48:56
  Speaker 3: there's been another generation of experiments to follow up on those.
• 49:00
  Speaker 3: For example, there's a very significant signal of dark matter
• 49:03
  Speaker 3: an experiment in Italy called DOMA, and nobody really believes
• 49:06
  Speaker 3: it because we've never seen it in another experiment. And
• 49:08
  Speaker 3: people have set up other experiments very similar to DAMA,
• 49:11
  Speaker 3: but in another part of the world with different conditions,
• 49:14
  Speaker 3: in a different cave, for example, and not seeing the
• 49:17
  Speaker 3: same effects. And so those were experiments definitely inspired by
• 49:20
  Speaker 3: this signal to test this in other conditions. And in
• 49:24
  Speaker 3: the last few years there was this quote unquote discovery
• 49:27
  Speaker 3: of a fifth force in this experiment in Hungary, and
• 49:31
  Speaker 3: folks in Berkeley are trying to replicate it, and there's
• 49:33
  Speaker 3: an experiment in Italy trying to probe it as well,
• 49:36
  Speaker 3: And so there's definitely like follow up work. But usually
• 49:39
  Speaker 3: those follow up experiments are a little bit broader, and
• 49:41
  Speaker 3: they try to not just check this one new result,
• 49:43
  Speaker 3: but also learn something else along the way to make it,
• 49:47
  Speaker 3: so they're also covering new ground. But I agree we
• 49:50
  Speaker 3: should definitely have replication. But it comes back to funding, right,
• 49:53
  Speaker 3: Like if your reviewer, what would you rather fund, Like
• 49:56
  Speaker 3: let's check Kelly study, or let's do this brand new
• 49:58
  Speaker 3: thing that could tell us something new about the universe.
• 50:01
  Speaker 1: Yeah, And when I write a grant, I try to
• 50:03
  Speaker 1: work my own replication in there, like as part of
• 50:06
  Speaker 1: asking a new question, I'm going to repeat the experiment,
• 50:09
  Speaker 1: but maybe put a little tweak But then I can
• 50:11
  Speaker 1: at least make sure that I'm still getting the same
• 50:13
  Speaker 1: results in you know, a different space or with slightly
• 50:16
  Speaker 1: different lighting or something like that. And so, and you know,
• 50:18
  Speaker 1: especially when to defend my field a little bit, especially
• 50:21
  Speaker 1: when animals need to be euthanized as part of the experiment,
• 50:25
  Speaker 1: you know, you might be a little bit hesitant to
• 50:27
  Speaker 1: be like, well, this is just for replication, Well, if
• 50:28
  Speaker 1: you already got an answer, do animals need to get so? Anyway,
• 50:31
  Speaker 1: So we do also try to ask additional questions while
• 50:34
  Speaker 1: trying to replicate, but it would be nice if we
• 50:36
  Speaker 1: had more incentive for that.
• 50:37
  Speaker 3: And there's folks out there just doing this, Like you
• 50:40
  Speaker 3: might have heard of the replication crisis that comes out
• 50:43
  Speaker 3: of people finding papers in the literature and saying like,
• 50:45
  Speaker 3: all right, let's reproduce this, let's see if it holds up.
• 50:48
  Speaker 3: And this is one reason that like p hacking is
• 50:50
  Speaker 3: a thing people talk about, because we discovered that some
• 50:53
  Speaker 3: of the results in literature are just statistical artifacts that
• 50:57
  Speaker 3: were selected in order to get a paper out there.
• 51:01
  Speaker 3: And so I think what you're seeing in a broader
• 51:03
  Speaker 3: sentence is science is self correcting. Just like we saw
• 51:06
  Speaker 3: that we needed to add external reviewers and members of
• 51:09
  Speaker 3: the public when we're talking about the experiments you do
• 51:12
  Speaker 3: on animals. Now we see like, okay, we need some
• 51:15
  Speaker 3: sort of way to protect against this kind of abuse
• 51:17
  Speaker 3: as well. And so you know, the process is a
• 51:20
  Speaker 3: living thing. It's not like science is a crisp philosophy.
• 51:23
  Speaker 3: What we call science has changed over the last ten years,
• 51:26
  Speaker 3: fifty years, one hundred years and it will continue to
• 51:29
  Speaker 3: evolve and I hope keep delivering great truths about the universe.
• 51:33
  Speaker 1: Yeah, me too, And so Steven asked, how can you
• 51:36
  Speaker 1: know if something is good science or not? I feel
• 51:39
  Speaker 1: like that was the big push behind the question. And
• 51:43
  Speaker 1: so what do you think when you read a paper
• 51:45
  Speaker 1: for the first time?
• 51:46
  Speaker 2: What do you look for?
• 51:47
  Speaker 3: Well, you know, science has to stand on its own,
• 51:49
  Speaker 3: So the thing I look for is like, does the
• 51:51
  Speaker 3: paper make sense? Does it lay out an argument? Do
• 51:54
  Speaker 3: the conclusions follow from the evidence presented? That's the most
• 51:57
  Speaker 3: important thing to me. But before I'm in a person
• 52:00
  Speaker 3: something believe that something is real part of the universe, yeah,
• 52:02
  Speaker 3: I need to see it reproduced by another group. I've
• 52:05
  Speaker 3: often reproduced papers myself, especially if they're like statistical, just
• 52:08
  Speaker 3: to make sure I understand, like, how is this calculation
• 52:10
  Speaker 3: being done? Exactly? How do you get from step three
• 52:12
  Speaker 3: to four? Because I really want to understand what are
• 52:16
  Speaker 3: the assumptions being made and think about whether those are
• 52:19
  Speaker 3: broad enough. So I would say that, yes, peer review
• 52:22
  Speaker 3: papers can be wrong, but that's part of science, and
• 52:26
  Speaker 3: the highest standard I think is independent replication.
• 52:28
  Speaker 1: What do you think, Kelly, Yeah, no, I agree. I mean,
• 52:30
  Speaker 1: when I read a paper, I'll look to see, you know,
• 52:33
  Speaker 1: what journal is it in. So lately there in this
• 52:36
  Speaker 1: day and age, there are some new what we call
• 52:38
  Speaker 1: predatory journals where they will encourage people to submit, but
• 52:41
  Speaker 1: they publish just about anything they get. And peer review
• 52:45
  Speaker 1: isn't so much about peers having a chance to turn
• 52:47
  Speaker 1: down a study, but they'll like give a.
• 52:49
  Speaker 2: Little bit of input. But the paper gets published one
• 52:52
  Speaker 2: way or another.
• 52:52
  Speaker 1: So you need to look to see what kind of
• 52:54
  Speaker 1: journal it was published in, and then you know, if
• 52:57
  Speaker 1: it's a field I know about, I'll look to see
• 52:59
  Speaker 1: did you cite the paper that I would expect you
• 53:01
  Speaker 1: to be citing. Was your literature search deep enough? And
• 53:04
  Speaker 1: where your experiments designed? Well, what else could those results
• 53:08
  Speaker 1: have meant? And then who funded the study and stuff
• 53:11
  Speaker 1: like that. But you know, if you are a member
• 53:12
  Speaker 1: of the general public and you can't do all of
• 53:14
  Speaker 1: that and you're just reading a pop side summary, I
• 53:16
  Speaker 1: would look for again, like where is this being published in?
• 53:19
  Speaker 1: You know, like if it's The Atlantic by ed Young,
• 53:22
  Speaker 1: it's probably great. And if the science journalist had other
• 53:26
  Speaker 1: scientists way in on what might have been wrong about
• 53:29
  Speaker 1: the study, that's a good sign, and so you look
• 53:32
  Speaker 1: for where you're getting the information, how critical they seem
• 53:35
  Speaker 1: to be, and stuff like that. What do you look
• 53:37
  Speaker 1: for in a pop side article.
• 53:39
  Speaker 3: Yeah, in a pop side article, I definitely look to
• 53:41
  Speaker 3: see whether they have talked to people who are not authors,
• 53:46
  Speaker 3: you know, other people in the field who know this stuff,
• 53:49
  Speaker 3: and is it just a press release from the university,
• 53:52
  Speaker 3: or as a journalist who's actually thought about this stuff
• 53:54
  Speaker 3: and written something up. Those are definitely the things I
• 53:57
  Speaker 3: look for in popside press releases, because there's the temptation
• 54:00
  Speaker 3: and the sort of marketplace of ideas to overinflate the
• 54:04
  Speaker 3: meaning of an incremental study.
• 54:05
  Speaker 1: Yep, yep, all right, So in general I would say
• 54:07
  Speaker 1: that you know, we have this process in place to
• 54:10
  Speaker 1: try to make sure that the best ideas are the
• 54:11
  Speaker 1: ones that move forward, and that we're all checking to
• 54:14
  Speaker 1: make sure no one missed anything important along the way.
• 54:17
  Speaker 1: Some stuff gets through either by mistake or because some
• 54:20
  Speaker 1: people are unscrupulous, but hopefully that doesn't happen that often.
• 54:25
  Speaker 1: But you know, the system is evolving at all times.
• 54:27
  Speaker 1: And if you have a study that you're interested in,
• 54:30
  Speaker 1: but you don't know if you can trust the press
• 54:32
  Speaker 1: release or whatever. You can send it to us, and
• 54:34
  Speaker 1: if it happens to me in our wheelhouse, we're happy
• 54:36
  Speaker 1: to weigh in and tell you what sort of you know,
• 54:39
  Speaker 1: set off our alarm bells in our heads, or what
• 54:42
  Speaker 1: we liked about the study, and we're happy to help
• 54:45
  Speaker 1: people figure out what was done well and what was
• 54:48
  Speaker 1: just squeaking through.
• 54:50
  Speaker 3: All right, Thanks very much, Steven for asking this question,
• 54:53
  Speaker 3: if for shining a light on the inner workings of science, and.
• 54:56
  Speaker 1: If you'd like to reach out to us, send us
• 54:58
  Speaker 1: an email at questions at Daniel and Kelly dot org
• 55:01
  Speaker 1: and let's see what Stephen had to say about our answer.
• 55:04
  Speaker 6: Hi, Daniel and Kelly, it's Steve here. Thanks so much
• 55:07
  Speaker 6: for answering my question. I think you shed a lot
• 55:09
  Speaker 6: of light on some topics that most of the general
• 55:13
  Speaker 6: public are not aware of, and I really appreciate that.
• 55:17
  Speaker 6: It's actually really fascinating to learn that just because something
• 55:21
  Speaker 6: is peer reviewed doesn't mean it's one hundred percent fact.
• 55:25
  Speaker 6: And it's definitely a lot to take away here, so
• 55:29
  Speaker 6: I appreciate you guys diving into the topic and looking
• 55:32
  Speaker 6: forward to the next episode.
• 55:40
  Speaker 1: Daniel and Kelly's Extraordinary Universe is produced by iHeartRadio.
• 55:44
  Speaker 2: We would love to hear from you, We really would.
• 55:47
  Speaker 3: We want to know what questions you have about this
• 55:50
  Speaker 3: extraordinary universe.
• 55:51
  Speaker 1: We want to know your thoughts on recent shows, suggestions
• 55:54
  Speaker 1: for future shows.
• 55:56
  Speaker 2: If you contact us, we will get back to you.
• 55:58
  Speaker 3: We really mean it. Answer every message. Email us at
• 56:02
  Speaker 3: Questions at Danielankelly.
• 56:04
  Speaker 1: Dot org, or you can find us on social media.
• 56:06
  Speaker 1: We have accounts on x, Instagram, Blue Sky and on
• 56:10
  Speaker 1: all of those platforms. You can find us at D
• 56:12
  Speaker 1: and K Universe.
• 56:14
  Speaker 3: Don't be shy, write to us