Latest / Geoffrey Challen: Teaching and Technology / Less Peer, More Review
Transcript
- 0:00Geoffrey Challen: Hey everyone, my name is Geoff Challen, and I teach computing at the University of Illinois.
- 0:04Geoffrey Challen: I write about teaching and technology at geoffreychallen.com,
- 0:08Geoffrey Challen: where you can find a written version of the essay you're about to hear.
- 0:11Geoffrey Challen: What follows is the essay being read aloud, but not by me, by a clone of my voice.
- 0:16Geoffrey Challen: I hope you enjoy it.
- 0:17Geoffrey Challen: If you have any feedback, please don't hesitate to reach out to me directly.
- 0:21Geoffrey Challen: I'd love to hear from you.
- 0:22Geoffrey Challen: Less Peer, More Review, an essay by Geoffrey Challen, read aloud by Geoffrey Challen's voice clone.
- 0:30Geoffrey Challen: Before I became a teaching professor, I reviewed for eleven mobile systems conferences, including MobiSys.
- 0:37Geoffrey Challen: Here’s an invite I received for MobiSys 2017:
- 0:40Geoffrey Challen: As Program Committee Co-chairs for ACM MobiSys 2017,
- 0:44Geoffrey Challen: we would be delighted if you would accept our invitation to serve on the Technical Program Committee.
- 0:49Geoffrey Challen: Your technical expertise and wisdom would be instrumental in helping us build a strong and diverse program. […]
- 0:56Geoffrey Challen: Based on last year’s numbers, we anticipate a review load of approximately 22 papers for TPC members,
- 1:02Geoffrey Challen: 15 in the first round and 7 in the second round. […]
- 1:05Geoffrey Challen: The TPC meeting is a single day […].
- 1:08Geoffrey Challen: Attendance is required for all TPC members.
- 1:11Geoffrey Challen: Please note that agreeing to serve means that you attend the TPC meeting in person—
- 1:15Geoffrey Challen: there will be no remote call-in offered. […]
- 1:18Geoffrey Challen: You are expected to read and write reviews for papers yourself.
- 1:21Geoffrey Challen: Delegation of reviews to others is not permitted.
- 1:25Geoffrey Challen: Now I review and submit work on computing education to SIGCSE.
- 1:29Geoffrey Challen: Here’s an invite I received for SIGCSE 2026:
- 1:33Geoffrey Challen: Tl;dr: If you haven’t already, please fill out the SIGCSE TS reviewer form.
- 1:40Geoffrey Challen: We need over 700 volunteers and currently we have 550!
- 1:45Geoffrey Challen: In all my time reviewing in mobile systems I never received a message like this—
- 1:50Geoffrey Challen: a mass appeal to lists reaching thousands of members.
- 1:53Geoffrey Challen: This is a sign that it’s time to revisit how SIGCSE reviews papers.
- 1:58Geoffrey Challen: The good news is that the fix is within reach.
- 2:02Geoffrey Challen: Quality Is Job Number One
- 2:04Geoffrey Challen: Program committees have one job: produce high-quality reviews.
- 2:09Geoffrey Challen: Reviews determine what a community sees and learns from,
- 2:12Geoffrey Challen: they start up or shut down lines of inquiry, and they affect careers.
- 2:18Geoffrey Challen: High-quality reviewing requires people who can and will produce high-quality reviews.
- 2:23Geoffrey Challen: Process matters, but no process can fully compensate for poor reviews,
- 2:28Geoffrey Challen: so the process should be designed to attract, motivate, and support good reviewers.
- 2:33Geoffrey Challen: The need to do better is becoming more urgent.
- 2:37Geoffrey Challen: SIGCSE accepted 32% of papers in 2019, 32% in 2026, and 25.8% in 2027, when submissions grew by 40%.
- 2:49Geoffrey Challen: When NeurIPS had two independent committees review a tenth of its 2014 submissions,
- 2:54Geoffrey Challen: they disagreed on about a quarter of the papers,
- 2:56Geoffrey Challen: against the 37% you’d expect from coin flips,
- 2:59Geoffrey Challen: and that was an invited committee of experts with discussion and area chairs.
- 3:04Geoffrey Challen: No reviewing process is perfectly accurate,
- 3:07Geoffrey Challen: particularly given that different community members would disagree on what accurate means!
- 3:11Geoffrey Challen: But as a conference gets more selective,
- 3:13Geoffrey Challen: more decisions land in the band where even good committees disagree, and process becomes more important.
- 3:20Geoffrey Challen: Here’s how the process is organized for a systems conference like MobiSys.
- 3:24Geoffrey Challen: A steering committee chooses several program committee chairs,
- 3:27Geoffrey Challen: people with an exemplary track record of research and community visibility.
- 3:31Geoffrey Challen: The program committee chairs then invite a certain number of people to join the program committee,
- 3:36Geoffrey Challen: from established highly-visible researchers to promising newcomers.
- 3:40Geoffrey Challen: Participating in a systems program committee is an honor and a responsibility.
- 3:44Geoffrey Challen: Nobody needs to be begged to join.
- 3:47Geoffrey Challen: Compare how the process is organized for SIGCSE.
- 3:50Geoffrey Challen: It starts out the same: with a steering committee and program committee chairs.
- 3:55Geoffrey Challen: But from there SIGCSE recruits both its reviewers and its senior reviewers through a web form that anyone can complete.
- 4:02Geoffrey Challen: SIGCSE calls this a “community reviewing process”,
- 4:06Geoffrey Challen: despite the fact that peer reviewing is always done by community members.
- 4:11Geoffrey Challen: And at least for the past few years, people clearly need to be begged to join.
- 4:16Geoffrey Challen: Too Big to Not Fail
- 4:18Geoffrey Challen: The size of any program committee is a balance.
- 4:21Geoffrey Challen: If the committee is too small, members are expected to do too much work, breadth is limited, and quality suffers.
- 4:28Geoffrey Challen: As it grows it becomes more diverse, but it becomes harder for the group to organize and communicate.
- 4:34Geoffrey Challen: For systems program committees, the goal seems to be just as large as necessary.
- 4:39Geoffrey Challen: SIGCSE’s goal seems to be as large as possible.
- 4:43Geoffrey Challen: MobiSys 2025 had 85 committee members for about 2,800 submitted pages: 33 pages per committee member.
- 4:52Geoffrey Challen: The committee is split into a 25-member heavy PC that follows
- 4:56Geoffrey Challen: papers through every round and attends the committee meeting,
- 4:59Geoffrey Challen: and a 60-member light PC that reviews in the first round only.
- 5:03Geoffrey Challen: Each paper gets roughly three reviews in the first round,
- 5:06Geoffrey Challen: and those that survive an early-reject cut get more in a second round, followed by a rebuttal and shepherding.
- 5:12Geoffrey Challen: Spread evenly, that is about 100 pages for a light-PC member,
- 5:15Geoffrey Challen: and roughly 200 for a heavy-PC member who follows papers into the second round.
- 5:20Geoffrey Challen: The committee is under no illusions about this:
- 5:23Geoffrey Challen: the 2025 self-nomination page warns that “joining the MobiSys PC is an extremely
- 5:28Geoffrey Challen: time-consuming commitment with the expectation of investing 200 or more hours.”
- 5:34Geoffrey Challen: SIGCSE 2027 had 724 committee members for about 4,600 submitted pages: six pages per committee member.
- 5:44Geoffrey Challen: The committee is split into 618 reviewers and 106 senior reviewers, who lead the discussion and write the meta-review.
- 5:51Geoffrey Challen: Each paper gets three or four reviews in a single round; 2027 averaged 3.9.
- 5:57Geoffrey Challen: Spread evenly, that is about 30 pages for a reviewer.
- 6:01Geoffrey Challen: So a MobiSys light-PC member reads more than three times as many pages as a SIGCSE reviewer,
- 6:07Geoffrey Challen: and a heavy-PC member nearly seven times as many, over two rounds rather than one.
- 6:12Geoffrey Challen: I estimate that I spent three to four hours on my reviews this cycle.
- 6:17Geoffrey Challen: SIGCSE Board policy asks for at least four reviews and a meta-review
- 6:22Geoffrey Challen: per paper and suggests no more than three papers per reviewer,
- 6:25Geoffrey Challen: down from a historical four: 18 pages, down from 24.
- 6:30Geoffrey Challen: For 2027’s 4,600 pages, that works out to over a thousand reviewers.
- 6:36Geoffrey Challen: The 2027 chairs found 618, asked each of them for about 30 pages,
- 6:41Geoffrey Challen: and still delivered 3.9 reviews per paper, below the target.
- 6:46Geoffrey Challen: A program committee takes on work on behalf of its community.
- 6:49Geoffrey Challen: They read everything, including papers that aren’t there yet,
- 6:53Geoffrey Challen: so that everyone else can devote their attention to significant results.
- 6:57Geoffrey Challen: I’d argue that this matters even more for SIGCSE.
- 7:00Geoffrey Challen: The community is larger, so it produces more work,
- 7:03Geoffrey Challen: and I suspect that it reads fewer papers than research-focused academics do.
- 7:08Geoffrey Challen: For many SIGCSE members,
- 7:10Geoffrey Challen: it would be better for the community for them to read more accepted papers rather than more submitted papers.
- 7:17Geoffrey Challen: Under Your Own Name
- 7:19Geoffrey Challen: More reviewers means less work for each reviewer.
- 7:22Geoffrey Challen: But more reviewers also means less-experienced reviewers and potentially more lower-quality reviews.
- 7:29Geoffrey Challen: The rest of the process has to be robust to weaker inputs.
- 7:33Geoffrey Challen: MobiSys ends in a program committee meeting—mandatory and in person when I served,
- 7:38Geoffrey Challen: hybrid now—where the heavy PC decides the final program.
- 7:43Geoffrey Challen: Reviews are anonymous to authors but not to other program committee members.
- 7:47Geoffrey Challen: Committee members can read each other’s reviews, discussion happens under your own name,
- 7:52Geoffrey Challen: and you have to defend your review in a room full of colleagues.
- 7:55Geoffrey Challen: A review that affects your reputation is one that you write carefully.
- 7:59Geoffrey Challen: SIGCSE ends in six days of online discussion, led by a senior reviewer who writes the meta-review.
- 8:06Geoffrey Challen: Reviewers are anonymous to authors and to each other.
- 8:09Geoffrey Challen: Only the chairs can connect a review to a name.
- 8:12Geoffrey Challen: There is no meeting.
- 8:14Geoffrey Challen: As far as I can tell, senior reviewers are also anonymous to each other,
- 8:18Geoffrey Challen: although that part of the process is invisible to me.
- 8:21Geoffrey Challen: Neither a poor review nor a poor decision carries any reputational cost,
- 8:25Geoffrey Challen: which makes poor-quality reviews and decisions more likely.
- 8:30Geoffrey Challen: Fewer, Better Reviewers
- 8:32Geoffrey Challen: As described above, SIGCSE already has a two-tier committee.
- 8:37Geoffrey Challen: When submissions grew, the senior tier grew with them, from 36 in 2017 to 121 in 2026.
- 8:46Geoffrey Challen: What’s missing on the papers track is the second round, the meeting, and the names.
- 8:51Geoffrey Challen: So borrow MobiSys’s structure and keep SIGCSE’s people.
- 8:56Geoffrey Challen: Round one stays as it is: three reviews per paper from the full pool.
- 9:00Geoffrey Challen: Then make an early-reject cut, gated by a senior reviewer who has read the paper,
- 9:05Geoffrey Challen: so that three volunteers who misunderstood a submission can’t end it on their own.
- 9:09Geoffrey Challen: The rest, perhaps half, go to a second round in which the senior reviewers read them—not just the reviews,
- 9:15Geoffrey Challen: the papers—and add reviews of their own.
- 9:18Geoffrey Challen: That’s about 45 additional pages per senior reviewer.
- 9:21Geoffrey Challen: Then hold a meeting: online, by track, a few hours, for the papers still in the middle after the second round.
- 9:28Geoffrey Challen: Reviewers stay anonymous to authors, but are identified to each other.
- 9:32Geoffrey Challen: Fewer people reading more pages need more time.
- 9:36Geoffrey Challen: In 2027 papers were due July 3 and decisions went out September 14:
- 9:41Geoffrey Challen: ten and a half weeks, of which reviewing and discussion took about three and a half.
- 9:46Geoffrey Challen: A second round and meeting fit in that window if the notification is split:
- 9:50Geoffrey Challen: round one decisions first and round two following.
- 9:53Geoffrey Challen: And the smaller group has to be assembled for breadth,
- 9:56Geoffrey Challen: with representation from two-year colleges, K–12, and international educators.
- 10:01Geoffrey Challen: Fewer, better does not mean fewer, R1.
- 10:04Geoffrey Challen: SIGCSE also has tracks, which invite a divide-and-conquer approach.
- 10:10Geoffrey Challen: Papers are already split into computing education research,
- 10:13Geoffrey Challen: experience reports and tools, and position and curricula initiatives,
- 10:17Geoffrey Challen: with reviewer instructions for each, and it would be natural to split them further:
- 10:22Geoffrey Challen: a CS1 track, a K–12 track, a tools track of its own.
- 10:26Geoffrey Challen: A track lets you assemble a committee of CS1 experts to review CS1 papers,
- 10:31Geoffrey Challen: and the smaller committee makes communication and coordination easier.
- 10:34Geoffrey Challen: The people who teach the same course are the sharpest readers of each other’s work,
- 10:38Geoffrey Challen: so if CS1 needs its own room, give it one.
- 10:42Geoffrey Challen: This doesn’t need to happen all at once.
- 10:44Geoffrey Challen: The existing tracks also allow experimentation.
- 10:47Geoffrey Challen: Set up one this way for one cycle.
- 10:50Geoffrey Challen: Keep its reviewer pool for round one,
- 10:52Geoffrey Challen: add a senior committee sized so that nobody reads more than 50 pages in the second round,
- 10:56Geoffrey Challen: hold the meeting, and identify the committee to each other.
- 11:00Geoffrey Challen: SIGCSE already collects feedback on reviews, providing a way to evaluate the results.
- 11:05Geoffrey Challen: If it works, consider expanding the model to other tracks.
- 11:09Geoffrey Challen: None of this is radical.
- 11:11Geoffrey Challen: It’s what other conferences already do, adapted in a way that’s sensitive to SIGCSE’s inclusivity goals and history.
- 11:18Geoffrey Challen: SIGCSE has the people, the tracks, and the senior reviewers.
- 11:23Geoffrey Challen: It just needs a better process.
- 11:26Geoffrey Challen: This is real Geoff again.
- 11:27Geoffrey Challen: Thanks for listening.
- 11:28Geoffrey Challen: You'll find more essays at geoffreychallen.com, and you can subscribe to the podcast wherever you're listening now.
- 11:34Geoffrey Challen: Until next time.
- 11:35Geoffrey Challen: Keep teaching with technology.